Email Experiment & Testing Analytics

Brightbox Electronics • Revenue-validated test insights • Target: 8 tests/mo, <7% false winners

North-star: Validated revenue lift per completed test
|
Goal: 8 tests/mo • <7% false winners • ≥80% power
Active Tests
7
2 completed this week
Winner Revenue Lift
+12.4%
vs. control avg
Statistical Power
86%
Above Target
False Winner Rate
4%
Under 7% goal
Learning Velocity
8/mo
On target
SRM Index
1.01
Balanced exposure
False Winner Detected — Week 4 Urgency Subject Test
Opens +11% but revenue -6%. Recommend archiving variant. Power: 91%

Test Outcomes (Last 90 Days)

Revenue Lift vs. Statistical Power

Experiment Results (11 Recent Tests)

Week Test Element Variant Opens Δ Revenue Δ Power SRM Complaint Δ Outcome
W11 Hero + CTA Interaction Variant B +8.2% +15.1% 94% 1.00 -0.02% Winner
W10 Preheader CTA Variant A +5.3% +9.4% 88% 1.02 0.00% Winner
W9 Send Time 9AM PST +3.1% +4.8% 79% 1.01 0.01% Inconclusive
W8 Subject Emoji 🎁 prefix +6.7% +7.2% 91% 0.99 0.00% Winner
W7 Image Placement Hero top -1.2% +2.3% 82% 1.00 0.00% Winner
W6 Personalization Token First name +4.5% +3.9% 76% 1.03 0.01% Inconclusive
W4 Urgency Subject "Last chance!" +11.0% -6.1% 91% 1.01 +0.14% False Winner
W3 CTA Color Orange +2.8% +5.6% 87% 1.00 0.00% Winner
W2 Footer Social Icons visible +0.9% +1.1% 68% 1.01 0.00% Inconclusive
W1 Preview Text Benefit-led +7.3% +8.9% 89% 0.99 -0.01% Winner
W0 Mobile Layout Single column +4.2% +6.7% 85% 1.00 0.00% Winner

Learning Velocity (Tests per Month)

Element Revenue Impact (Treemap)

Hero + CTA
+15.1%
Preheader
+9.4%
Preview Text
+8.9%
Subject Emoji
+7.2%
Mobile Layout
+6.7%
CTA Color
+5.6%
Send Time
+4.8%
Image Place
+2.3%

Complaint Rate Delta by Test Variant