Optimizing Infrastructure for Large Language Models

While smaller models run on just a few GPUs, state-of-the-art models often require several hundred gigabytes to as much as one terabyte of VRAM, as well as up to 8 GPUs or more. (Example: Qwen3 235B, one of the most popular open-weight models today, requires approximately 600 GB for full performance.)

Added to this is storage space: Storing the models requires at least as much disk capacity as VRAM. Even though storage is cheaper than VRAM, it should not be underestimated in infrastructure planning. A miscalculation leads directly to bottlenecks and prevents efficient fine-tuning or inference.

Recommendation: Create a clear model profile early on, including details on storage requirements, latency requirements, and batch sizes.

Workloads change over time and place new demands on the infrastructure. Optimizations usually require adjustments.

• Conventional GPU servers are often sufficient for smaller LLMs.
• Larger LLMs, on the other hand, require specialized servers capable of hosting many GPUs in parallel.

Although GPUs can be networked across multiple servers, this usually results in significantly higher latency—often a deal-breaker for interactive LLM systems.

Real-world example: Consolidating GPUs onto a single server often significantly reduces latency compared to distributing them across multiple machines.

GPUs are indispensable for production-ready LLMs—they form the core of the system. The choice of hardware and vendor plays a key role here: Using different GPU types in the same setup often leads to compatibility issues and performance losses.

NVIDIA remains the market leader and offers the best combination of raw computing power and a mature software ecosystem (CUDA, TensorRT). While alternative providers may offer cost advantages, they often lag behind in software optimization. For most companies, NVIDIA is therefore the safe choice.

Recommendation: Plan realistically. Complex models typically require multiple GPUs on the same server—not distributed across clusters or even data centers.

Without dedicated test environments, optimizations often go straight into production—a risky strategy.

Best Practice: Set up a sandbox where GPU configurations, memory allocations, and quantization strategies can be safely tested without affecting live systems.

GPUs are expensive—and large models require many of them. In addition to the initial purchase cost, energy consumption, cooling, floor space, maintenance, and potential hardware failures must also be factored in.

In practice, failures do occur: Recently, a customer unexpectedly had to replace a GPU server on short notice—resulting in downtime and additional costs. Even though electricity costs may seem smaller as a percentage, they add up significantly during continuous operation.

Recommendation: Create a Total Cost of Ownership (TCO) model that takes into account acquisition, operation, and scaling.

Failover setups and dedicated development environments require additional hardware—and therefore additional funding. If you don't take this into account from the start, you'll face unpleasant surprises later on.

• Increasing hardware requirements: Each new generation of LLMs requires more VRAM and computing power. Model upgrades almost always involve purchasing new hardware.
• Slow internal processes: Trial and error is part of optimization. Long approval cycles or rigid change management processes slow down innovation.
• Quantization & Compression: You can drastically reduce VRAM requirements—with only a minor loss of quality. However, striking the right balance is complex.
• Data Protection & Location: The location of the hardware is critical for compliance. “On-premises” sounds simple, but for international companies, it does not necessarily mean that data is processed at the location where it was generated.

• Pre-Purchase Analysis: Thoroughly Understand the LLM's Requirements.
• Optimize the GPU setup: Keep GPUs physically close together to minimize latency.
• Invest in test environments: Sandboxes are crucial for secure optimization.
• Take a holistic approach to budgeting: Consider acquisition, operation, and redundancy.
• Use quantization strategically: Reduce memory usage without sacrificing too much model quality.
• Ensure flexibility: Make hardware setups quickly adaptable—especially in large organizations with rigid processes.

Many companies underestimate the complexity of LLM infrastructures until bottlenecks arise in production. A structured approach to hardware planning helps avoid costly mistakes.

If you’d like to validate your setup or explore optimization paths, we’d be happy to share our best practices from real-world projects.