When the combined total of your system prompt, retrieved RAG context, and conversation history exceeds an LLM’s context window, the system physically cannot process the entire input.
When the combined total of your system prompt, retrieved RAG context, and conversation history exceeds an LLM’s context window, the system physically cannot process the entire input.
What Happens and Symptoms of Exceeding Limits
- Data Truncation and Outdated Answers: The most immediate symptom is that the system will truncate the data. If new data is retrieved successfully but pushed beyond the context window, it gets cut off. Without this new context, the LLM will default to its pre-trained parametric memory, causing your RAG system to confidently return outdated or hallucinated answers.
- Reasoning Degradation: Even if you stay just under the absolute limit, filling a massive context window with too much retrieved data (e.g., injecting 20 document chunks) fills the prompt with noise, which severely degrades the AI’s reasoning quality on what actually matters.
- Out-of-Memory (OOM) Crashes: If you are self-hosting your models, processing long context windows causes the model’s KV cache to grow continuously. If your GPU is running near maximum utilization, this overflow will result in random OOM crashes.
Specific Token Counts and Pricing Impact
Different models possess different maximum context lengths. For example, GPT-3.5 supports approximately 16,000 tokens, while GPT-4 supports up to 128,000 tokens. Many self-hosted models default to a much smaller window, such as 2,048 or 4,096 tokens, which often needs to be manually increased for enterprise workloads.
Expanding your context window has severe cost implications:
- API Costs: Token waste from unclassified or overly large context windows drives up API bills significantly. For example, API inputs for GPT-3.5 cost roughly £0.0004 per 1,000 words, but relying on a larger model like GPT-4 to handle massive contexts is 10 to 20 times more expensive.
- Infrastructure Costs (Self-Hosted): Larger context windows consume proportionally more VRAM. Running a 7B parameter model at a 4K context window requires about 5 GB of VRAM, but expanding that same model to a 32K context window demands 8 to 10 GB of VRAM.
Mitigations for SMB Deployments
To avoid these limits and keep costs under control without sacrificing performance, you must aggressively manage what goes into the prompt:
- Strict Chunking and Capping: Break your enterprise documents into smaller chunks of 400 to 800 tokens with a 10% to 15% overlap. Cap the number of chunks injected into the prompt to a strict limit (e.g., a Top-5 chunks maximum) to prevent overwhelming the model.
- Score-Gating: Implement a minimum relevance score threshold for your vector database retrieval. If a retrieved chunk falls below this score, discard it entirely rather than wasting token space on low-confidence context.
- Summarization and Memory Nodes: Instead of feeding the entire raw conversation history back into the LLM on every turn, use memory management tools. Implement a “Summary Memory” node that maintains a condensed running summary of the chat, or a “Buffer Memory” node that only remembers the most recent messages.
- Enable Flash Attention: If you are self-hosting, ensure “Flash Attention” is enabled in your configuration. This optimization significantly reduces memory usage and improves processing speed, and it is particularly beneficial for context windows larger than 8,000 tokens.