Simulate membership inference attacks against ML models and measure defense effectiveness. Configure your model parameters, select attack methods, and compare defenses with precision/recall metrics, ROC curves, and utility-privacy tradeoff analysis.
Set your model parameters and attack configuration. The simulator generates realistic attack metrics based on known vulnerability profiles for each model type.
| Defense | Attack Accuracy | AUC-ROC | Model Utility | Effectiveness |
|---|
Membership inference is the most studied privacy attack against machine learning models. First formalized by Shokri et al. in 2017, it exploits a fundamental property of trained models: they behave differently on data they have seen during training compared to data they have not. A model that has memorized a training example will produce higher confidence predictions, lower loss values, and more deterministic outputs for that example compared to unseen data from the same distribution. This behavioral gap, which increases with overfitting, is the signal that membership inference attacks exploit.
The privacy implications are severe. If an attacker can determine that a specific medical record was used to train a disease prediction model, they learn that the person in that record has the disease, even without access to the medical database directly. Similarly, confirming that a financial record was in the training set of a credit scoring model reveals information about the individual's financial history. Membership inference transforms model access into a side channel for extracting information about training data, making it a direct threat to data privacy regulations including GDPR, HIPAA, and the EU AI Act.
The simplest membership inference attack uses a confidence threshold. The attacker queries the model with a target record and checks whether the model's maximum confidence score exceeds a threshold. Training members typically receive higher confidence because the model has been optimized to classify them correctly with high certainty. The threshold is calibrated using a small reference dataset that includes both members and non-members (or inferred from the model's general behavior). Despite its simplicity, threshold-based attacks achieve 60-75% accuracy on overfitted neural networks, well above the 50% random baseline. Their advantage is that they require no additional model training, making them the lowest-cost attack to mount.
Shadow model attacks, introduced in the original Shokri et al. paper, are more sophisticated. The attacker trains multiple shadow models on data drawn from the same distribution as the target model's training set. For each shadow model, the attacker knows exactly which records are members and which are not, because they control the training process. By querying each shadow model on its members and non-members, the attacker builds a labeled dataset of (model output, member/non-member) pairs. An attack classifier trained on this dataset learns to distinguish the subtle statistical patterns that differentiate member and non-member model outputs. Shadow model attacks typically achieve 65-80% accuracy and are particularly effective against models that expose full confidence distributions (multi-class probabilities) rather than just the top prediction.
Metric-based attacks use the model's loss value (or an approximation of it) rather than raw confidence scores. The intuition is that training members have lower loss because the model was trained to minimize loss on exactly those examples. Loss-based membership inference is often more powerful than confidence-based attacks because loss captures the model's behavior more precisely than the maximum confidence alone. Label-only attacks operate in the most restricted setting: the attacker only sees the predicted class label, not the confidence scores. These attacks use perturbation-based techniques, querying the model with slightly modified versions of the target record to estimate the model's sensitivity around that point. Training members tend to be in stable prediction regions where small perturbations do not change the label, while non-members are closer to decision boundaries. Label-only attacks are weaker but demonstrate that even minimal model access is sufficient for privacy attacks.
Five primary defenses exist against membership inference attacks. Differential privacy (DP-SGD) provides the strongest theoretical guarantee by adding noise during training that bounds the influence of any individual record. At epsilon 1.0, it reduces attack accuracy to near 50% but typically costs 2-10% model utility. Regularization (L2 penalty, dropout, early stopping) reduces overfitting, which directly reduces the member/non-member behavioral gap. Strong regularization can reduce attack accuracy by 5-10% with minimal utility loss. Confidence masking returns only the predicted label or rounded confidence scores, reducing the information available to attackers. This costs no utility in the model's predictions but removes the fine-grained confidence signal that attacks exploit. Knowledge distillation trains a student model on the outputs of the original model, naturally smoothing the confidence distribution and reducing memorization. The student model is typically 3-5% less vulnerable than the teacher. Data augmentation increases the effective training set size, which reduces per-record memorization and makes each individual record less distinguishable. The simulator above models each defense's impact on attack metrics and utility, helping you choose the right combination for your privacy requirements.
The standard metric for membership inference vulnerability is attack accuracy: the fraction of records correctly classified as members or non-members. A perfectly secure model would have an attack accuracy of 50% (random guessing). However, accuracy alone can be misleading because it averages over the entire population. The TPR@FPR metric (true positive rate at a fixed false positive rate) captures the worst-case vulnerability: how many members can the attacker correctly identify while limiting false accusations. TPR@FPR=0.1% is particularly relevant for privacy because it measures the attack's precision in identifying specific individuals. The AUC-ROC (area under the receiver operating characteristic curve) summarizes attack performance across all operating points; an AUC of 0.5 indicates no vulnerability, while 1.0 indicates perfect attack. For deployment decisions, compare the undefended AUC to the defended AUC to quantify the improvement from each defense mechanism.
To audit your model for membership inference vulnerability: (1) Reserve 20% of your training data as a known-member set and hold out an equal-sized non-member set from the same distribution. (2) Train your model on the remaining 80% plus the known-member set. (3) Query the model on both sets, collecting confidence scores and loss values. (4) Run threshold-based and metric-based attacks using the confidence and loss distributions. (5) If shadow model attacks are feasible, train 5-10 shadow models and evaluate the attack classifier. (6) Report attack accuracy, AUC-ROC, and TPR@FPR=0.1% as your vulnerability metrics. (7) Apply defenses (regularization, DP-SGD, confidence masking) and re-run the audit to measure improvement. (8) Document the results for privacy impact assessments and regulatory compliance. The simulator above automates steps 4-6 using realistic statistical models calibrated on published attack results.
A membership inference attack determines whether a specific data record was used to train a model. The attacker queries the model and analyzes output confidence to distinguish members from non-members. Models are more confident on training data due to memorization, which creates an exploitable statistical signal.
Overfitted models are most vulnerable. Small training datasets, complex architectures, many output classes, and language models that memorize rare sequences all increase vulnerability. Tree-based models and linear models are generally less vulnerable than deep neural networks.
DP-SGD clips gradients and adds noise during training, mathematically bounding how much the model depends on any individual record. At epsilon 1.0, it typically reduces attack accuracy from 60-70% to 52-55%, with 2-10% utility cost.
Threshold attacks use a simple confidence cutoff. Shadow attacks train separate models to learn member/non-member patterns. Shadow attacks are more accurate (65-80% vs 60-75%) but require data from the same distribution as the target.
Split data into known-member and non-member sets, query the model on both, and measure how well an attacker can distinguish them. Key metrics: attack accuracy (target: near 50%), AUC-ROC (target: near 0.5), and TPR@FPR=0.1% (worst-case vulnerability).