DynoSim: Simulating the Pareto Frontier for LLM Serving Optimization
DynoSim addresses the complexity of tuning modern LLM serving deployments by simulating the Pareto frontier across interacting system choices such as model backend, tensor-parallel shape, and autoscaling thresholds.
Modern LLM serving is hard to tune because each deployment involves a stack of interacting choices such as model backend, tensor-parallel shape, prefill/decode split, worker counts, scheduler settings, routing policy, KV cache behavior, autoscaling thresholds, and topology. These choices interact across layers, and local improvements can shift bottlenecks elsewhere. DynoSim simulates these interactions to help optimize performance trade-offs in large model serving deployments. This approach aids in understanding the complex performance landscape and tuning configurations effectively for better throughput and latency in LLM inference workloads.