Teaching a 4B model to extract structured data from web pages

Emitting JSON is easy. The useful result is that supervised fine-tuning made a 4B model materially better at extracting the right values.

TL;DR

1. On 97 held-out pages with human-corrected teacher drafts, fine-tuning Qwen3.5-4B raised page-macro value accuracy from 23.41% to 41.54%. The paired gain was 18.13 percentage points, with a 95% interval from +11.26 to +25.53 points.
2. Schema-valid responses rose from 62.89% to 95.88%.
3. A second, production-domain stage mixed 28,699 private examples with 15,313 public replay examples. It processed 50.07 million tokens in 3 hours 55 minutes on one 32 GB RTX 5090.
4. Most labels came from other models, so this measures the extraction contract, not human-level accuracy or general model superiority.

The contract

The service accepts three inputs:

1

Markdown page + extraction instruction + arbitrary JSON Schema -> JSON

JSON Schema defines the permitted keys, types, arrays, and required fields, and may change on every request. The evaluations below cover public pages and one private corpus, not every schema or domain.

Specialist extractors already exist, including NuExtract 2.0 and Schematron. I fine-tuned a local model for my exact prompt, Markdown representation, schemas, and pages, and to measure what one consumer GPU could buy.

Stage one: did fine-tuning teach the job?

The first stage used scrapegraphai/scrapegraph-100k-finetuning. GPT-5-nano generated its answers: useful pseudo-labels, not human ground truth.

After removing exact and normalized-content overlaps, I sealed 97 usable, human-corrected examples. They never informed checkpoint, prompt, or threshold selection.

The primary metric was page-macro normalized value accuracy: every page weighs equally, whether its schema has one field or fifty. Normalization ignores whitespace, not incorrect values.

Against the untouched 4B, value accuracy gained 18.13 points (paired 95% interval: +11.26 to +25.53) and schema validity gained 32.99 points. Failures stayed in the denominator.

The tuned 4B also beat my local quantized 27B by 7.28 points here, though paired tests were borderline or inconclusive. The narrow conclusion: teaching this task mattered more on this benchmark than adding general-purpose parameters.

Stage two: adapt it to production data without forgetting everything else

The final candidate continued from the public-corpus adapter. Its mixture was deliberately not 100% private data:

DeepSeek produced most private-corpus answers; GPT-5-nano produced the public replay. Replay was intended to reduce catastrophic forgetting, but without a replay ablation that remains a design choice, not a causal result.

Why 16k context?

An 8,192-token pilot retained 97.8% of the public corpus, but production pages were longer. A length audit and the 32 GB memory budget led to a 16,384-token cap; preflight counted 277.6 million input and 7.75 million answer tokens. The service may expose 32k at inference, but training stopped at 16k, so quality beyond it is unproven.

The final recipe

I used LoRA (small trainable updates to a frozen base model) and BF16. The Unsloth Qwen3.5 guide informed the memory choices; the run used Transformers, TRL, and PEFT.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

model: Qwen/Qwen3.5-4B
parent_adapter: qwen35-4b-sgai-sft-small-seed7
dtype: bfloat16
lora: {rank: 16, alpha: 32, dropout: 0.0}

max_seq_length: 16384
microbatch: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
packing: false
loss: assistant_only_chunked_nll
optimizer: adamw_torch_fused
learning_rate: 0.0001
scheduler: cosine
token_budget: 50000000

Gradient accumulation turns 16 one-page microbatches into one optimizer batch; checkpointing trades computation for memory. Assistant-only loss predicts the JSON answer, not every source token. Packing stayed off because it was much slower on this long-sequence workload.

What survived domain adaptation?

On 3,159 source-separated private development rows, both BF16 and online FP8 returned parseable, schema-valid JSON for every row:

These numbers measure agreement with pseudo-labels, not human accuracy. The split separates source/entity groups but lacks reliable template fingerprints, so it cannot establish template generalization.

On a separate 97-page public regression, the final domain-adapted BF16 model reached 38.78% value accuracy and 96.91% schema validity. Online FP8 reached 36.14% and 94.85%. BF16 remains the quality reference; FP8 bought throughput at a small but measurable quality cost.

What this establishes

Supervised fine-tuning improved a frozen, human-corrected benchmark. One RTX 5090 then domain-adapted the 4B extractor at 16k in under four hours while retaining useful public-task performance.

It does not show that a specialist 4B generally beats a 27B, that pseudo-label agreement equals human correctness, or that valid JSON contains the right facts.

Part two is about exactly that: serving the model at queue scale without counting fast failures as progress.

Reproducibility notes

• The 97 labels are human-corrected teacher drafts, not independent ground truth.
• Intervals use paired, group-aware resampling; all failures remain in the denominator.
• Raw pages and predictions stay local; MLflow stores metrics, configuration, digests, and provenance.
• A frozen report preserves the public comparison and its caveats.