Home / Rank #7
Context hygiene & token management
Chat histories grow without bound, RAG pipelines stuff 20 chunks where 3 would do, and agents drag full tool outputs through every subsequent turn. Input-side bloat is the quiet half of most AI bills.
Sliding-window histories with periodic summarization, reranked retrieval that keeps only the top few chunks, and tool-output truncation routinely reclaim 30–50% of input spend — and usually improve answer quality, because the model sees less noise.
How to do it
- Cap conversation history; summarize older turns into a compact state instead of replaying them.
- Add a reranker to retrieval and cut passed chunks to the minimum that preserves answer quality.
- Truncate or summarize tool outputs before they enter the context.
- Log tokens-per-request per feature; alert on drift.
Frequently asked questions
Will trimming context hurt quality?
Usually the opposite — models get distracted by irrelevant context ("lost in the middle"). Measured trimming with evals tends to improve both cost and accuracy.
Tools for this method
Mitosis Cortex
The cut-spend-at-the-source option: a persistent memory graph over your email, docs, chats, and tools that agents query instead of…
Helicone
Long the default open-source LLM cost dashboard. Acquired by Mintlify in March 2026 and now maintenance-only — existing deployment…
Next method: #8 Cost attribution & AI FinOps