Defend your ML APIs against model theft. Interactive tools for rate limit calculation, query pattern detection, and output watermarking simulation. Estimate how long extraction would take and how to detect it.
Estimate how long it would take an attacker to fully extract your model given your rate limiting configuration. Adjust parameters to find the right balance between usability and security.
Watch how different query patterns are detected by an anomaly detection system. Normal usage patterns versus extraction attack patterns. Click to simulate different attack types.
See how imperceptible perturbations in model outputs can fingerprint and later identify stolen models. Compare original vs. watermarked outputs and test detection.
Model extraction is one of the most economically damaging attacks on ML systems. A well-trained model represents months of compute time, expensive training data curation, and proprietary algorithmic insights. An attacker who can replicate your model's behavior by querying its API effectively steals all of that investment. Unlike traditional software theft, model extraction does not require access to source code or weights — only access to the model's input-output interface. This makes any ML model served via an API a potential target.
The attack works by sending carefully chosen inputs to the target model and collecting the corresponding outputs (predictions and confidence scores). The attacker then trains their own "surrogate" or "student" model on this synthetic dataset, using the target model's outputs as ground truth labels. With enough query-response pairs, the surrogate model converges to a functional copy that replicates the target's behavior across the input space. The fidelity of the copy depends on the number of queries, the information content of each response (full probability distributions leak more than top-1 labels), and the complexity of the target model.
Rate limiting is the first line of defense against model extraction. The goal is to make full extraction impractically slow while still allowing legitimate usage. The extraction time calculator above helps you find the right balance. For a 70M parameter image classifier returning full probability distributions, a skilled attacker needs roughly 50,000-100,000 queries for a high-fidelity copy. At 60 queries per minute, that takes 14-28 hours. At 10 queries per minute, it takes 3.5-7 days. At 1 query per minute, extraction takes over a month and becomes economically unviable.
Effective rate limiting uses multiple layers: per-API-key limits (e.g., 100 queries/minute), per-IP limits to prevent key rotation, daily and monthly query budgets per user, and escalating throttling where sustained high-volume usage triggers progressively stricter limits. The key insight is that legitimate users rarely need to make thousands of sequential queries, while extraction attackers always do. The difference in query patterns is detectable.
Beyond simple rate limits, monitoring query patterns can detect extraction attempts with high accuracy. Normal users exhibit natural patterns: variable query rates, diverse input types, browsing pauses, and organic input distributions. Extraction attackers exhibit systematic patterns: constant high query rates, structured synthetic inputs (grid searches, uniform distributions), sequential queries with small perturbations (gradient estimation), and requests for full probability distributions rather than top-1 predictions.
The pattern detection simulator above demonstrates four query types. Normal traffic shows clustered queries with natural pauses and diverse inputs. Grid search attacks show uniform coverage of the input space. Boundary probing concentrates queries near decision boundaries where the model changes its prediction, as these are the most informative for training a surrogate. Gradient estimation uses pairs of similar inputs with tiny perturbations to numerically estimate the model's gradients, which enables more efficient surrogate training.
Detection algorithms include statistical tests on query distributions (Kolmogorov-Smirnov test against expected user distributions), anomaly detection on query timing patterns (constant rate vs. bursty), input similarity analysis (flagging sequences of nearly identical inputs), and information-theoretic measures (detecting queries that maximize information gain about the model). When a suspicious pattern is detected, the system can throttle the user, inject noise into outputs, or alert security teams.
Watermarking provides a post-theft detection mechanism. The idea is to embed an imperceptible signature in the model's outputs that will be learned by any surrogate model trained on those outputs. When a suspected stolen model is discovered, you test it with a set of secret "trigger" inputs and check whether the outputs match the expected watermark pattern. If they do, you have strong evidence that the model was trained on your outputs.
The watermark works by slightly perturbing the model's confidence scores for specific trigger inputs. For example, on a particular trigger image, the model might report 91.3% confidence instead of 91.0%. These perturbations are small enough to not affect prediction quality but large enough to be statistically detectable across multiple trigger inputs. The watermark key (which trigger inputs to use and what perturbation pattern to expect) is kept secret. An attacker cannot remove the watermark without knowing the key, and the perturbation pattern propagates through knowledge distillation to the surrogate model.
The watermark strength parameter controls the tradeoff between detectability and output quality impact. Stronger watermarks are easier to detect but may measurably affect output quality. Weaker watermarks preserve quality but require more trigger inputs for reliable detection. The simulator above lets you experiment with this tradeoff, comparing original and watermarked outputs to see how the perturbations manifest in confidence scores.
No single defense stops a determined attacker. The most effective approach combines rate limiting (slowing extraction), query pattern detection (detecting extraction attempts), output perturbation (reducing information per query), watermarking (enabling post-theft detection), and access controls (limiting who can query the model and what information is returned). Specifically: return top-1 predictions only (not full probability distributions), add random noise to confidence scores, implement per-user query budgets, monitor for anomalous patterns, and watermark outputs.
For high-value models, additional defenses include requiring CAPTCHA for sustained high-volume queries, implementing adaptive rate limits that tighten when suspicious patterns are detected, using proof-of-work challenges to increase the cost of automated querying, and deploying honeypot inputs that trigger alerts if seen in surrogate model training data. The ML Threat Model Generator can help you systematically identify which extraction defenses are most important for your specific deployment, and the ML Security Checklist includes extraction-specific controls to verify.
Model extraction exists in a legal gray area. While terms of service typically prohibit automated querying and reverse engineering, enforcement across jurisdictions is challenging. The EU AI Act and various trade secret laws may provide legal recourse, but technical defenses remain the primary protection. Watermarking creates evidence for legal proceedings by demonstrating that a competitor's model was derived from your outputs. Rate limiting and query budgets also serve a business purpose beyond security: they prevent free-riders from consuming compute resources without contributing revenue, aligning security incentives with business incentives.
A model extraction attack reconstructs a copy of your ML model by systematically querying its API and training a surrogate on the input-output pairs. The attacker sends chosen inputs, collects predictions and confidence scores, and uses them as training data. With enough queries, the surrogate replicates the original model's behavior without ever accessing its weights or code. This effectively steals months of training investment.
Rate limiting restricts queries per time window, making full extraction impractically slow. Since extraction requires thousands to millions of queries, strict limits (10-100 queries/minute) force attackers to spend days or weeks on extraction, making it economically unviable. Multi-layer rate limiting (per-key, per-IP, daily budgets) combined with escalating throttling provides the strongest defense.
Output watermarking embeds imperceptible signatures in predictions. Small, structured perturbations are added to confidence scores that propagate when a surrogate model is trained on those outputs. Testing a suspected stolen model with secret trigger inputs reveals the watermark pattern, proving derivation. Watermarking provides post-theft evidence rather than prevention, complementing rate limiting and query detection.
It depends on model complexity and output type. Linear models need hundreds of queries, neural classifiers need 10K-100K, and LLMs may need millions. Returning only top-1 labels (not confidence scores) increases required queries 5-10x. Model complexity (number of parameters, output classes) scales the requirement. Use the extraction time calculator above to estimate for your specific configuration.
Key indicators: unusually high query volume from one user, systematic synthetic inputs (grid searches, uniform distributions), sequential queries with tiny perturbations (gradient estimation), requests for full probability distributions, constant query rate without natural pauses, and low input diversity compared to legitimate users. Detection systems using statistical tests and anomaly detection can flag these patterns automatically.