LLM Product Usage & Prompt Analytics

Lumos AI • Last 30 Days • Updated 2 min ago

Day-30 Retention

31.4%

+2.1% WoW

Prompt Success Rate

84.7%

-3.8% (incident)

Avg Session Depth

4.2

turns/session (P50)

Refusal Rate

8.3%

+4.2% spike D18-21

Hallucination Score

6.1%

-1.3% improvement

Cost/Retained User

$4.21

-$0.18 vs plan

Feature Adoption Funnel (30-Day Cohort)

Top Prompt Intent Clusters

Refusal Rate by Intent Category

Session Turn Depth Distribution

P25: 2 turns • P50: 4 turns • P75: 8 turns • P90: 14 turns

User Tier Engagement (30-Day)

Hallucination Proxy (30d)

Token Efficiency Ratio

Output Latency Perception

Streaming 87%

Non-Streaming 62%

CSAT proxy score (% satisfied with response time)

Prompt Success Rate by Feature × Day

Incident window (Day 18–21) shows sharp drop in Summarization feature

Feature

D1-3

D4-6

D7-9

D10-12

D13-15

D16-18

D19-21

D22-24

D25-27

D28-30

Writing Assistant

91%

92%

90%

91%

93%

89%

88%

90%

92%

91%

Code Helper

87%

86%

88%

85%

87%

86%

82%

85%

87%

88%

Semantic Search

94%

95%

93%

94%

96%

95%

94%

95%

96%

95%

Summarization

89%

88%

87%

89%

88%

86%

71%

81%

85%

87%

Q&A Knowledge

83%

82%

84%

81%

83%

82%

78%

80%

82%

83%

90–100%

85–90%

80–85%

<80%

Incident Summary: Day 18–21 System Prompt Update

On Day 18, a system prompt change aimed at reducing hallucinations accidentally increased Summarization refusal rate from 5.2% to 14.1%. Single-turn sessions spiked from 31% to 47%. Partial rollback on Day 21 restored refusal to 8.9%, but Day-30 retention for Summarization cohort dropped 6.8pp. Root cause: overly conservative safety filter. Action: implement A/B test framework for prompt changes.