Score Your Model's Exposure to Membership Inference Attacks
Feed in your model's train/test performance and instantly see how much a membership inference adversary can learn about who was in your training set — expressed as attack advantage, equivalent AUC, and an empirical privacy epsilon.
Recommendations
- Enter values above.
How the risk score is computed
A membership inference attack (MIA) tries to decide whether a specific record was part of a model's training set. The single strongest predictor of MIA success is the generalization gap — how much better the model performs on training data than on unseen data. Yeom et al. (2018) proved that for a threshold attacker, the membership advantage is bounded by exactly this gap:
Advantage ranges from 0 (attacker no better than a coin flip) to 1 (perfect membership recovery). We convert it into two familiar quantities. An adversary's balanced accuracy is 0.5 + advantage/2, which is also the operating point of the equivalent ROC — so we report an attacker AUC = 0.5 + advantage/2. We then derive an empirical, worst-case privacy lower bound in the differential-privacy sense: any mechanism whose attacker can reach advantage adv cannot satisfy an epsilon tighter than
In the loss-distribution mode we instead model members and non-members as two Gaussian loss distributions and compute the advantage as the total-variation distance a threshold attacker achieves, then feed the same conversions. The score is bucketed into Minimal / Low / Elevated / Severe tiers, shifted upward when your data is sensitive or your training set is small (each record then carries more marginal influence and memorizes more easily). This is a fast pre-audit heuristic — a real shadow-model or LiRA attack will give tighter numbers, but the gap bound is a rigorous upper limit that no attacker can exceed.