SGLang's benchmark disagreed with its own engine logs for four months. I found out why on my first day in the repo, along with a segfault and a 1.1-second import.
First day in the repository: pytest ends in a segfault, a torch pin breaks in thirty seconds, and one imported function costs 1.1 seconds. The benchmark has contradicted the engine logs for four months. I start by reading both calculations.
Ten years in data engineering left me with one useful reflex: measure before believing. I chose the serving layer precisely because it is the part of the LLM stack I know least.
This log covers only what I ran, measured or read that day. The claim, first PR, benchmark audit and vLLM validation are public.
Making a CUDA-only repo run on a Mac
SGLang declares CUDA-only dependencies, compiles Rust extensions and pins torch==2.13.0. On an arm64 Mac, pip install -e python/ fails at the first CUDA wheel. Its CPU CI shows the way through: do not install the package; run it from the source tree with PYTHONPATH.
Three pins bit me, in order of embarrassment: a floating torchvision silently upgraded torch to 2.14.0 and broke the repo pin; the latest transformers collides with SGLang's patches (ValueError: 'qwen3_asr' already registered) so 5.12.1 it is; and coverage segfaulted. That one was interesting.
Plain pytest runs were green. The moment I added --cov, the interpreter died with Segmentation fault somewhere inside torch. The stack told the story: pytest-cov's import hook calls find_spec on every module to decide whether to trace it. That call triggered the first import of sglang, which runs its darwin platform stubs, which import torch, whose native extensions were mid-initialization from a different entry point. Import machinery re-entered itself, and the C layer tore. For about a minute I assumed this was the part where Mac users get politely asked to leave.
The fix is one line, outside the repo: a sitecustomize.py that pre-imports sglang before coverage's hooks exist.
The lesson generalizes: when a test tool crashes, read the stack from the bottom up and ask what is being imported from inside someone else's hook.
Finding real work in an already-tested file
The unit-test tracking issue lists srt/parser. Branch coverage immediately disproves my assumption: the module already has 1,600 lines of tests. Yet its largest file remains the least covered. The issue identified an area; measurement found the work.
Branch coverage per file, before the new tests:
| File in srt/parser | Before | After (19 tests) |
|---|---|---|
| code_completion_parser.py | 98% | 98% |
| harmony_parser.py | 96% | 96% |
| conversation.py | 92% | 92% |
| jinja_template_utils.py | 92% | 92% |
| template_detection.py | 89% | 89% |
| inkling_tokenizer.py | 85% | 85% |
| inkling_renderer.py | 81% | 81% |
| reasoning_parser.py | 64% | 77% |
Branch coverage, main @ 99b9109, measured with --cov-branch. The suite total moved from 82% to 86%.
Inside it, one class had no tests at all: MuseGlimmerDetector, the reasoning-side parser for a channel protocol where to=self content is thinking and to=user content is the visible reply. Two hundred lines of state-machine code: sink routing, marker holdback across chunk boundaries, and a finish() path that promotes reasoning to content when a turn produced nothing visible. There is a trap worth naming: a function-call detector with the same name in a different package is well tested. Same name, different class. If I had grepped instead of measured, the claim comment would have been wrong in public.
So I claimed it before writing a line, with the numbers: 64%, 286 uncovered statements, the file-line ranges, the name-collision note, and a promise to paste before and after coverage in the PR. Claiming first is collision insurance and it forces you to say the scope out loud where someone can correct you.
Then I wrote 19 tests, one behavior each: channel routing, chunk-split markers that must be held and never leaked, the function-call interplay that preserves framing, the promotion semantics at finish(). Green in 0.73 seconds. The file moved from 64% to 77%, the suite from 82% to 86%. The PR #37974 is open as I write this.
A four-month-old benchmark bug, read from both sides
Issue #3050 sat dormant since May: the benchmark reports roughly half the decode throughput the engine logs show, someone guessed at causes, and the thread stalled. The maintainer had asked the community for help and nobody had answered with code. Both metrics live in the repo, so I read both.
They are not two measurements of the same thing. The benchmark's number is a TPOT-derived, whole-run, client-side average: it includes queueing, ramp-up, and the tail after the last token. The engine's number is num_generated_tokens divided by the time since the previous log line (metrics_reporter.py:799). One is an average over the whole run. The other is close to an instantaneous rate. The reported inconsistency lives in that comparison, not in either metric.
- itl is recorded per stream event, not per token. Events carrying several tokens produce one latency sample, so ITL percentiles from the completions backend conflate event and token latency. A retokenized correction exists, but only for the sglang-oai backends.
- output_len silently falls back to the requested length when the response lacks usage stats, which makes TPOT too small for early-stopped requests.
- The flush_cache cold-start theory from the thread only applies under CI or an explicit flag (
serving.py:1467-1471). It cannot explain a default run.
I wrote five unit tests that pin the arithmetic with a stubbed clock: exact timestamps in, invariants out. sum(itl) plus the [DONE] tail equals latency - ttft, to the tick. Mean ITL exceeds TPOT for the same request, from denominators alone.
Tests do not fix the bug; they make the next person able to reason about it without trusting anyone's memory, including mine. They are open as PR #37973.
One second of import time for one function call
The last artifact of the day was free. An open PR about import overhead had been sitting with zero comments for two months, so I profiled current main on my machine: import sglang takes 5.3 seconds warm, 6.4 cold, across 3,479 modules. The top of the cumulative column:
| Package | Cumulative (warm) |
|---|---|
| torch | 1.51s |
| transformers | 1.13s |
| torchvision (one function: decode_jpeg) | 1.11s |
The torchvision line is the interesting one. It exists because srt/utils/common.py:96 imports decode_jpeg from torchvision.io at module scope. One function, unconditionally, for 1.1 seconds of every import sglang. I posted the numbers on that PR with the observation and no fix attached. Numbers travel further than suggestions.
The scoreboard, honestly
Nothing was merged at publication time. As of that evening: one claim on the unit-test tracking issue with the test PR behind it, five invariant tests on their own branch, two evidence comments on dormant threads, and a vLLM validation claim for gemma-4-E4B-it. Six threads, two projects, zero merges.
One day of activity does not prove a durable contribution. Worse, AI-assisted newcomers can flood maintainers with changes that look plausible but cost too much to verify. SGLang says it rejects PRs that appear automatically generated. I use an agent for mechanical work — reading code, following imports, running coverage — but I choose every test and must be able to defend it. The merge may come or not. The claims are already falsifiable.
The vLLM lane landed after the first draft of this article: google/gemma-4-E4B-it validated on both execution cells with zero run-to-run variance, results posted and the doc PR #55343 open.
My only scorecard: did a maintainer receive something they can verify faster than they can doubt it? The merge remains their decision, on their schedule.
That is the trade I came to make.