sdr_draco_baseline_analysis
SDR: DRACO Benchmark Baseline Analysis
Worker auto-analysis, 2026-03-16 11:30 MSK
Pre-evaluation analysis — no API key needed (structural metrics only)
DRACO Score Rankings (LLM-judge, 37 tasks, 0-100 scale)
| Rank | Model | Mean | Median | Min | Max |
|---|---|---|---|---|---|
| 1 | anthropic/claude-opus-4.6 | 50.3 | 46.5 | 8.8 | 93.2 |
| 2 | perplexity/sonar-deep-research | 35.8 | 36.2 | 0.0 | 70.7 |
| 3 | google/gemini-3-flash-preview | 34.6 | 33.4 | 4.9 | 71.4 |
| 4 | alibaba/tongyi-deepresearch-30b-a3b | 26.7 | 22.7 | 0.0 | 75.2 |
| 5 | dr-gemini-3-flash | 19.7 | 19.5 | 0.0 | 45.7 |
| — | local_deep_research (SDR) | TBD | — | — | — |
Gap to beat: ≥50.3 to be competitive with Claude-opus (leader)
Reasonable initial target: ≥35 (beat perplexity/google baselines)
Claude-opus Domain Breakdown (best competitor)
| Domain | Score | N | Notes |
|---|---|---|---|
| Academic | 69.3/100 | 12 | Strongest — literature review tasks |
| Medicine | 51.5/100 | 6 | Good factual retrieval |
| General Knowledge | 41.4/100 | 9 | Mixed |
| Technology | 34.8/100 | 10 | Weakest — specific tech questions |
Implication for SDR: Academic domain is our sweet spot (operator splitting, ML). Technology domain (34.8) = achievable target for SDR first benchmark.
Structural Weaknesses in Claude-opus Baseline
From structural_baseline.json (37 reports):
| Metric | Value | Issue |
|---|---|---|
| Diversity score | 0.131/1.0 | Very low — mainly same-domain sources |
| Has references section | 6/37 (16%) | Most reports lack formal references |
| Word count avg | 3,103 | Moderate depth |
| Inline years found | 22/37 (59%) | Good year diversity in citations |
| Academic ratio | ~0% | Surprisingly low for academic papers |
Key insight: Even the best competitor (claude-opus) has severe structural weaknesses:
- Only 16% of reports include a references section
- Very low source diversity (0.131/1.0)
- SDR could gain structural score points by ensuring every report has proper references
SDR Improvement Opportunities (pre-API-key analysis)
Quick wins (structural, no quality judgement needed):
- Always include references section — Claude-opus fails 84% of the time; SDR could enforce this in prompts
- Source diversity — Explicitly request sources from different domains/years in research prompts
- Academic ratio — For academic tasks, ensure arxiv/scholar sources are prioritized
LLM quality (needs API key for benchmarking):
- Judge model: Anthropic claude-haiku is now configured (
eval/config.py); ready for benchmark_2 - Academic domain benchmark first: Start with DRACO Academic subset (12 tasks) — fastest path to meaningful comparison
Next Steps for SDR Evaluation
- Get ANTHROPIC_API_KEY → provide to Feanor (ask-level task in backlog)
- Run benchmark_2 with claude-haiku judge on Academic domain first (12 tasks)
- Target score: ≥55 on Academic domain (beat claude-opus 69.3 is aspirational; ≥35 = competitive)
- Structural baseline: Already computed for claude-opus; run same for SDR to compare
Eval System Status
- Tests: 105/105 ✅ (display tests fixed 10:20 today)
- Judge: Anthropic claude-haiku configured, auto-detect mode
- Baseline:
structural_baseline.jsonsaved for anthropic_claude-opus-4.6 - Benchmark 2 config:
data/benchmarks/benchmark_2.yaml(TBD — needs tasks) - Blocker:
ANTHROPIC_API_KEYfor live benchmark run