The Black Box Illusion: ML APIs are an invitation to theft

The Attack Paradigm: Efficiency is Key

Forget those brute-force attacks from 2020. Nobody is going to query your API randomly until they break the bank (well, almost nobody). The game now is budget efficiency. The attacker wants to replicate your function $f_v(x)$ with a substitute model $f_s(x)$ spending the absolute minimum on API calls.

Important pause here: do not confuse Extraction with Inversion. They are completely different attacks:

• Extraction (Stealing): “I want to copy your model’s functionality so I don’t have to pay for it anymore.” This is IP theft, pure and simple.
• Inversion: “I want to reconstruct the faces/data you used in training.” This is a privacy violation.

What I’m discussing here is extraction. And it has become frighteningly efficient.

The AugSteal Revolution

If your API returns only “Hard-Labels” (like “Cat” or “Dog”, without probabilities), you probably think you’re safe. My friend, I have bad news.

The AugSteal framework¹ has completely destroyed that notion of security. And its logic is astute and brilliant at the same time: the attacker doesn’t care about what your model knows. He wants to know what his substitute model doesn’t know.

It works like this:

1. Active Learning (AL): The attacker selects inputs where their clone model has high entropy (uncertainty). You know those cases where the AI isn’t sure? Those decision boundary points? That’s exactly where they attack.

2. Surgical Querying: They only pay to query your API at these critical points. No waste.

3. MixMatch/Data Augmentation: They use rotations, noise, and augmentations to extract consistency from a single “Hard-Label” response. It’s like squeezing blood from a stone, but it works.

The result? Replicas with over 90% fidelity costing a ridiculous fraction of the original training. We are talking about saving millions here.

LLMs: The Mathematics of Extraction

Now things get more interesting (and mathematically elegant, if you’re into that sort of thing). LLMs are, essentially, giant matrix multipliers. And linear algebra is ruthless.

Recent research² has demonstrated that it is trivial to extract the embedding projection layer from models like the GPT family. Trivial, no exaggeration.

# Simplified concept of W extraction
import numpy as np

# The attacker sends queries varying the bias (b)
# and observes the exact changes in the output log-probs (z).
# With N queries, he solves:
# W * h = z_observed - b_injected

def solve_projection_layer(queries_results):
    # Solves linear system Ax = B using least squares
    # This extracts the matrix W the provider wanted to keep secret
    projection_matrix = np.linalg.lstsq(
        queries_results['h'],
        queries_results['z_adjusted']
    )
    return projection_matrix

The Replica is the Attacker’s Dojo

Now you might be thinking: “Alright Patrick, but why would anyone steal my model besides saving on the API bill?”

Good question. The answer is: to break it.

There is a property in Deep Learning called Transferability³. Adversarial examples (inputs surgically designed to fool the AI) created for Model A often fool Model B, if the architectures or datasets are similar.

So the attack flow looks like this:

1. I steal your model and have $f_s(x)$ running locally on my GPU.
2. I can generate millions of adversarial attacks per second (White-Box attack) without paying a cent in API fees and without showing up in your rate limiting logs.
3. I optimize the attack until I hit 100% success on my local model.
4. I fire a SINGLE request against your production API. And it gets through.

Your API becomes a static target. The attacker trains in the dojo (the replica) and wins in the ring (production) with a single punch. It’s like training against a dummy of you before the real fight.

Side-Channels: The Silence that Speaks

Even if you block logprobs, hide confidence scores, and ban suspicious IPs, your physical infrastructure still gives away information. Timing Attacks on NLP APIs are absolutely devastating.

Mixture-of-Experts (MoE) models, like GPT-4 and beyond or Mixtral, have variable inference times depending on how many and which “experts” are activated internally.

If input A consistently takes 200ms and input B takes 250ms, I’ve gained a bit of information about the complexity or routing path of your network. Seems like nothing? Multiply that by 10,000 queries.

In classification systems, if the “Fraud” class goes through 3 extra layers of verification and auditing, the latency reveals the result before the JSON even arrives. It’s like poker, but your server has obvious tells.

For the attacker, your latency jitter is Morse code broadcasting details of your architecture.

Defense: A Game of Cat and Mouse (with State)

Okay, enough with the bad news. How do we defend ourselves?

Passive defense is dead. The only way out is Stateful Detection.

Conclusion

Your model is your product. If you expose it via API, you are selling free samples of your intellectual property with every HTTP 200 request.

Security by obscurity is over. Dead. Gone. Buried. If you aren’t implementing state monitoring to detect Active Learning patterns, or measuring the entropy of incoming queries, your model is likely already:

• Someone else’s training dataset, OR
• The punching bag for an adversarial script running 24/7

The golden rule: Protect the weights, but protect the gradients EVEN MORE.

And if you think this is paranoia, maybe it’s time to check your API logs. Those 50,000 “suspicious” requests from last week? They weren’t scraping bots.

They were someone cloning you.

Further reading:

• OWASP Top 10 for LLM Applications. (2025). Open Web Application Security Project. Available at: https://genai.owasp.org/llm-top-10/
• LLM10: Model Theft. (2023). Open Web Application Security Project. Available at: https://genai.owasp.org/llmrisk2023-24/llm10-model-theft/

References: