Serving a 4B web extractor at 16.85 valid pages per second

Part one showed that fine-tuning improved extraction. The operational question was whether one consumer GPU could keep up with 900,000 real pages.

The 16.85-pages/s headline is an operator-reported production-shaped observation. Because its workload and exact server manifest were not preserved, I keep it separate from the reproducible long-context benchmark.

TL;DR

1. In a production-shaped API test, one RTX 5090 processed 240 real documents in 14.18 seconds at 48 concurrent requests.
2. 239/240 responses were schema-valid: 16.85 valid pages/s, with about 2.98 seconds p50 and 3.06 seconds p95 request latency.
3. That is API capacity, not crawler throughput. The live ingestion pipeline was persisting about 458 classifications per ten minutes, or 0.76/s. Storage, leases, preprocessing, database writes, retries, and bad inputs now dominate.
4. The accepted path is boring and effective: Markdown, merged weights, vLLM, per-channel FP8, constrained JSON, persistent connections, and 48 independent requests in flight.
5. Patched n-gram speculation, static compilation, alternative kernels, SGLang, and larger scheduler/concurrency settings did not clear the correctness plus 5% throughput gate.

Define the number before optimizing it

I count valid pages per second, not submitted requests per second. A page counts only when the request completes, the output parses as JSON, and it satisfies the supplied JSON Schema. Timeouts, 5xx responses, malformed JSON, schema violations, and missing batch items stay in the denominator.

That exposes a common failure mode: higher concurrency can raise attempted throughput by failing difficult pages faster. JSON Schema checks structure, not truth; validity and extraction quality are separate gates.

The whole path matters

The model sees Markdown because the fine-tune saw Markdown. Raw HTML contains scripts, styles, navigation, templates, and hidden elements that waste context and create a training-serving mismatch.

flowchart LR
  A[Page capture] --> B[Queue + lease]
  B --> C[Gateway
HTML to Markdown]
  C --> D[vLLM
batch + constrained decode]
  D --> E[Validate JSON]
  E --> F[Database completion write]

The gateway accepts HTML or Markdown, handles gzip, builds the training-compatible prompt, and validates requests and results. vLLM schedules each page independently; the gateway never concatenates unrelated documents.

Continuous batching adds and removes requests as GPU slots become available. Prefill ingests the prompt; decode generates the answer. Because pages are longer than JSON answers, extraction is prefill-heavy.

The accepted server

Applying a LoRA adapter dynamically adds avoidable serving work. I merged it into a BF16 checkpoint, then let vLLM quantize linear weights online to per-channel FP8. This is the accepted reproducible configuration; the headline run’s exact manifest was not preserved.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

MODEL_DIR=artifacts/checkpoints/domain-extractor/merged-bf16
MODEL_NAME=qwen35-4b-extractor

.venv-vllm/bin/vllm serve \
  "$MODEL_DIR" \
  --tokenizer models/Qwen3.5-4B \
  --served-model-name "$MODEL_NAME" \
  --language-model-only \
  --dtype bfloat16 \
  --quantization fp8_per_channel \
  --max-model-len 32768 \
  --max-num-batched-tokens 16384 \
  --optimization-level 3 \
  --performance-mode throughput \
  --disable-uvicorn-access-log \
  --host 127.0.0.1 --port 8000

The 32k context is a request ceiling; training used a 16k cap. --max-num-batched-tokens limits scheduler work per iteration, not per-page context.

The gateway connects to vLLM’s OpenAI-compatible endpoint:

1
2
3
4
5

FTE_SERVICE_UPSTREAM_URL=http://127.0.0.1:8000/v1 \
FTE_SERVICE_MODEL=qwen35-4b-extractor \
FTE_SERVICE_MAX_CONCURRENCY=48 \
FTE_SERVICE_MAX_TOKENS=1024 \
scripts/serve_gateway.sh

For one high-throughput client:

• keep 48 documents in flight;
• send one document per request and let vLLM batch internally;
• reuse one HTTP connection pool instead of reconnecting;
• gzip request bodies when bandwidth matters;
• send Markdown when it already exists, otherwise let the gateway clean HTML;
• keep the schema and instruction stable when possible; and
• allow up to 1,024 output tokens rather than silently truncating hard pages.

Two benchmarks, two questions

The frozen workload contains about 2.50 million input tokens, or roughly 10.4k tokens per page. It is a conservative long-document stress test with a saved workload digest.

The newer run is capacity evidence for its production-shaped distribution, not an apples-to-apples 6.2x software optimization: its workload digest, token totals, request mode, warm-up policy, and server manifest were not saved. Future runs should preserve that provenance.

At 16.85 valid pages/s, 900,000 pages represent about 14.8 inference-hours if a client keeps the API saturated.

What helped

The useful serving path came from stacking ordinary decisions:

• Representation parity. Train and serve Markdown, not raw HTML.
• Merged LoRA weights. One matched A/B improved successful throughput by 32.8% and median latency by 36%; dynamic and merged arms completed 39/50 and 40/50 pages. This is a workload result, not a universal LoRA law.
• FP8 serving. On an earlier matched long-page run, online per-channel FP8 was about 1.53x the BF16 throughput. The public quality regression also moved against FP8, so BF16 remains the quality reference.
• Persistent HTTP. Keep-alive added 8% in one matched long-page A/B.
• Enough independent work. Forty-eight in-flight pages kept continuous batching productive. Raising concurrency to 64 was slower on the strict long-page screen.

The common mechanism is higher GPU utilization: less adapter and connection overhead, plus enough independent work for vLLM to schedule well.

What did not help

Each candidate had to preserve request IDs, JSON parsing, schema validity, semantic parity, and concurrency safety, then beat the control by 5%.

N-gram speculation looked ideal because extraction often copies prompt values. But scanning and padding long 32k-context requests cost more than accepted drafts saved; both correctness-patched candidates were slower. Unpatched speculation was never accepted for concurrent schema-guided requests.

Prefix caching can also fake gains on repeated pages by reusing identical prompt prefixes. Candidate screens therefore used fresh processes and cold pages.

The GPU is no longer the bottleneck

The live ingestion system was persisting about 458 classifications per ten minutes, or roughly 0.76 pages/s. At that sustained rate, 900,000 pages would take about 13.6 days—not 14.8 hours.

The API-only benchmark excludes object-storage reads, queue and lease work, normalization, malformed pages, retries, 400-level failures, and database writes.

The model server has headroom. The next target is ordinary distributed-systems work: overlap reads, keep 48 valid requests available, batch safe writes, classify 400 failures, and fail fast on irrecoverable inputs.

What this establishes

One RTX 5090 can serve a fine-tuned 4B extractor at nearly 17 schema-valid real pages per second on the measured production-shaped sample. It does not prove 17 pages/s for every schema, input length, or client, and schema validity does not prove factual correctness.

The durable lesson is to measure the model API and trace through persistence: a fast GPU only moves the bottleneck.

References

• vLLM serve options
• Qwen3.5-4B model card
• Unsloth Qwen3.5 fine-tuning guide
• Part one: fine-tuning the extractor