LLM Fine-Tuning, Compiling and Serving Benchmarks

The challenge

Published benchmarks never matched the team's traffic shape. Long prompts, bursty concurrency, and strict tail-latency targets meant a config that looked fastest in isolation was often the worst choice in production.

Approach

• Built distributed fine-tuning on Ray with Horovod-based data parallelism, LoRA adapters for cheap task iteration, and a full RLHF stage with reward-model training and preference optimisation.
• Wrote custom CUDA C++ kernels for the attention and sampling hot paths, then validated numerical parity against the reference implementation before adoption.
• Compiled each checkpoint through ONNX Runtime, TensorRT-LLM, and vLLM at multiple quantisation levels, holding hardware, tokeniser, and prompt mix constant across runs.
• Replayed production traffic traces instead of synthetic uniform load, reporting throughput against P99 latency so tail behaviour is visible rather than averaged away.
• Published every result as a versioned artefact with the exact commit, driver, and container digest, making regressions attributable to a single change.

Outcome

The selected TensorRT-LLM configuration delivered 5.8x the baseline throughput at the team's P99 target, cutting cost per million tokens by 71% while retaining 99.2% of evaluation quality. The harness is now a merge gate for serving changes.