Context: Introduction to distributed inference with llm-d

Challenges

I measure the LLM Inference performance by monitoring metrics such as CPU utilization, Memory consumption, HTTP Request counts, etc. This conventional approach do not fully capture LLM behavior, and become inadequate.

The inference system rely on generic routing strategies such as Round Robin and Sticky Sessions, both of which were designed for Web traffic, not LLMs. These strategies do not account for prompt structure, token count, queue depth, and GPU load. Also, These strategies fail to reuse the cache computation efficiently.

llm-d in actions

The three core areas:

  1. Disaggregated inference
  2. Cache-aware routing
  3. Mixture of Experts

Well-lit 1: Disaggregated inference

In LLMs, there are two stages: the prefill phase and the decode phase. The prefill phase processes the input tokens and computes the KV cache. The decode phase generates the output tokens. The disaggregated inference method is able to handle these processes effectively.

Below is an illustration of the LLM stages, where prompt processing produces a KV cache, which is sent to the decode phase to generate the output.

Request -> Prompt processing (prefill) -> KV Cache -> Token Generation (Decode) -> Output

If a GPU is not available, llm-d allows the prefill phase to run on CPUs while the decode phase runs on GPUs. The decode phase is typically more memory-intensive, and it can benefit from GPU acceleration. On the other hand, the prefill phase can often be handled effectively with a CPU.

As a result, llm-d uses disaggregated inference rather than Round Robin to improve LLM performance through routing.

Well-lit 2: Cache-aware routing

llm-d introduce inference gateway for understanding the structure and content of incoming prompts, after that it route them based on cache context and prior requests.

For example, consider user who requests a summary of financial report. If the same prompt has already been processed, the inference gateway will route to the relevant cache. This reduces redundant computation, improve time to first token, and lowers GPU usage. Details below:

image

Well-lit 3: Mixture of Experts

This approach introduces two forms of parallelism: Expert Parallelism (EP) and Data Parallelism (DP).

In short, the expert parallelism able breaks the model into specialized components. This avoids relying on a single large GPU machine. Instead, llm-d makes it possible to use multiple smaller nodes, often with mixed hardware, to scale horizontally across the entire datacenter.

For example, consider we use the GPT-OSS model, which is broken into several experts such as math and biology. If the inference gateway receives an incoming prompt about math, it will be routed to the math expert, which runs on a machine with more GPUs. On the other hand, if it receives a prompt about biology, it will be routed to the relevant expert, and run on a machine with fewer GPUs.

Details below:

image