Processing Large Volumes of Documents with LLMs

Large Language Models (LLMs) are becoming the technology of choice when it comes to analyzing, summarizing, and making large amounts of text searchable. In practice, however, these systems quickly reach their limits: context windows are limited, throughput issues slow down processing, and even modern retrieval methods sometimes overlook critical content.

The following section outlines three key pain points in LLM-based document processing—along with the most important lessons learned from real-world projects and the strategies that have proven successful.

Even state-of-the-art long-context models with hundreds of thousands of tokens cannot process everything at once. Studies show that LLMs often fail to consistently utilize relevant information in very long inputs (the “Lost in the Middle²” phenomenon). Anyone attempting to summarize 50 documents at once will inevitably lose important details.

Strategy:

• Multi-stage orchestration using approaches such as Map-Reduce or Refine.
• With Map-Reduce, individual documents or larger blocks of text are processed and pre-aggregated separately (Map) before a consolidation step (Reduce) merges the results into a final output.
• With the Refine approach, a draft is first generated from the first chunk. This draft is then iteratively refined with additional chunks.
• Although both methods increase latency, they enable scalability and greater completeness.

A single request can take seconds or even a minute. While this is often acceptable, users benefit greatly from receiving early confirmation that their request is being processed. More critical, however, is throughput: platforms like Azure OpenAI limit the number of tokens or requests per minute. Once these limits are reached, users often wait 30–90 seconds for processing to begin.

Strategy:

• Use streaming to provide early feedback even with longer latencies.
• Use prompt caching to avoid having to constantly recalculate static instructions or contexts.
• Carefully control request volume and manage budgets for tokens or requests per minute.

Processing additional documents does not necessarily increase accuracy. Even with hybrid search (semantic + keyword), relevant items can slip through the cracks. This gives rise to the classic Top-k dilemma:

• Top 10 → fast, but with the risk of missing half of the relevant information.
• Top 20 → better coverage, but twice the latency.
• Top 30 → even more comprehensive, but three times slower and more expensive.

Strategy:

• Offer clear modes: Fast (Top 10, low latency, streaming) vs. Thorough (Top 20–30, Map-Reduce, slower).
• Incorporate users’ domain knowledge: Offer filters and selection options to allow users to intentionally add relevant documents.

1. Speed often trumps completeness
   ; in many cases, users prioritize a quick response. If this is the default mode, optimization should focus on time to first token.
2. Transparency builds trust
   . If the thorough mode takes longer, this should be clearly indicated in the UI. Users are more likely to accept trade-offs when they are explicitly communicated.
3. Combine
   model sizes: Small, cost-effective models are ideal for preprocessing (classification, ranking). Larger models should be reserved for final responses—this saves costs and improves both quality and speed.
4. Search remains essential
   . Users are familiar with search tools. Integrating filters and selection options increases relevance and reduces unnecessary processing.

• Customer Service Knowledge:
  Agents receive immediate answers based on the top 10 documents. For more complex cases, they switch to Thorough Mode, which processes over 30 documents using MapReduce.
• Legal & Compliance:
  Completeness is crucial here. The default mode is Thorough, supplemented by filters such as time periods or document types.
• Technical Documentation:
  Smaller models first cluster and tag sections before a larger model generates the final summary. This prevents irrelevant content from cluttering the context window.

• Context limits remain the bottleneck—even with long context windows, structured retrieval logic is essential.
• Throughput is the real hurdle—streaming, caching, and queue management are crucial.
• More documents mean higher costs and latency—transparency regarding this trade-off is essential.
• Prioritize user expectations—most users value speed over completeness.

²Source: https://arxiv.org/abs/2307.03172