A/B Testing Dashboard — Sample Quality & Statistical Integrity

Meridian SaaS Platform • Q3 2024 Integrity Audit

Concurrent Experiments
42
Monthly Active Users
280K
SRM Detection Rate
7.2%
Target: <3%
False Discovery Rate
4.8%
Within 5% target
Holdout Validation Rate
84%
Goal: 90%
Peeking Violations
12
Last 30 days
Power Achievement
91%
Above 80%
Network Contamination
2.1%
Low interference

P-Value Distribution Analysis

Histogram of p-values across 126 completed experiments (Q3) • Healthy distribution should be uniform under null

SRM Detection Over Time

Weekly SRM incidence rate • Spike in Week 4 due to SDK v3.2 bug

MDE Calibration Accuracy

Planned vs. achieved minimum detectable effect across experiment types

Sequential Testing Violations (Peeking)

Experiments stopped early without correction • By team

Assignment Mechanism Uniformity

Chi-square test results for randomization quality • Green = uniform

Integrity Audit Table

Recent experiments flagged for statistical integrity issues

Experiment ID Team SRM Status P-Value Sample Size Power Issues
EXP-4721 Growth SRM Detected 0.042 18,420 0.73 Mobile redirect SRM, Underpowered
EXP-4718 Product Warning 0.003 42,100 0.88 Stopped early (peeking)
EXP-4715 Platform Pass 0.156 51,200 0.92 —
EXP-4709 Growth SRM Detected 0.001 29,800 0.81 Assignment skew, Network contamination
EXP-4702 Product Pass 0.089 38,500 0.95 —
EXP-4698 Growth Warning 0.012 22,300 0.68 Underpowered, Collision with EXP-4702
EXP-4691 Platform Pass 0.421 47,900 0.90 —
EXP-4687 Growth SRM Detected 0.028 31,200 0.79 SDK v3.2 bug, Mobile redirect

Multiple Comparison Correction Coverage

Percentage of multi-metric experiments applying Bonferroni/BH correction

False Discovery Rate Trend

Weekly FDR estimate from holdout validation • Target: <5%

Experiment Collision & Mutual Exclusion Heatmap

User overlap between concurrent experiments • Red = high collision risk

Experiment Collision Rate
8.4%
23 overlapping pairs detected in active portfolio
Avg. Experiment Duration
21d
Median runtime to reach 80% power threshold
SDK Health Score
6.8/10
Assignment latency and error rate composite