RAG + evaluation platform
SEC-RAG-Eval
May 2026 – present
A retrieval-augmented question-answering service over US SEC 10-K, 10-Q, and 8-K filings, and the evaluation harness that measures it. The RAG system is the subject under test; the harness is the point — it reports which design choice moved which metric, and how much of that was real rather than an artifact of how the answer was scored.
Results
| recall@5 (fuzzy) | 0.44 → 0.64V0 → V2 |
|---|---|
| recall@10 (fuzzy) | 0.54 → 0.74 |
| tables@5 (fuzzy) | 0.32 → 0.70 |
| faithfulness | 0.93 |
| answer accuracy (all 150) | 0.50 |
| cost / query | $0.009 |
The finding
Recall isn't a number — it's a bracket, and the clean validation was an artifact
Scoring the identical retrieval three ways gives 0.09 under strict substring matching, 0.64 under the shipped fuzzy overlap, and 0.81 under embedding similarity — an 8× spread from grading alone. Hand-labeling 50 query-chunk pairs then caught a bigger problem: an automated labeler had “validated” the shipped scorer at 0.67 agreement against a true 0.18. The clean validation was itself the artifact — so the repo shipped it once, then published the correction.
Honest limitations
- recall@5 = 0.64 is the generous end of the bracket. The same retrieval scores 0.09 under strict substring matching; the honest number sits between.
- About 20% of the recall gain was chunk size inflating the overlap metric, not the system improving — measured by crossing embedding model × chunk size × scorer.
- Answer accuracy is 0.50 over all 150 questions; the service declines about a third rather than guessing when the evidence isn't retrieved.
What got built
- Launched an SEC-filings question-answering service on GCP Cloud Run — FastAPI over Postgres pgvector with 15,192 chunks from 84 10-K/10-Q/8-K filings under text-embedding-3-large, Claude Haiku 4.5 generation, 118 tests, and CI that blocks retrieval regressions.
- Raised recall@5 from 0.44 to 0.64 on FinanceBench-150 by measuring and retiring five approaches that regressed — hybrid RRF fusion, a BGE cross-encoder reranker, and dedicated table extraction among them — then shipping the one that worked, an embedding-model swap, at $0.009 per query.
- Extended the service to any of ~10,400 US public companies by fetching, parsing, and indexing a company's newest 10-K, 10-Q, or 8-K from EDGAR on demand, streamed over SSE with section-level citations, per-IP rate limiting, and a Neon cache.
Stack
- FastAPI
- PostgreSQL / pgvector
- text-embedding-3-large
- Claude Haiku 4.5
- GCP Cloud Run
- Neon
- Docker
- CI/CD
- FinanceBench