Large language models tokenize vector graphics (SVG) the way they tokenize prose, shredding a coordinate like 150.5 into characters that carry no spatial meaning. We ask a narrow, falsifiable question: does the unit a model reads geometry in change downstream modeling? We introduce GeomTok, a geometry-native SVG tokenizer (commands, shapes, and a zero-vocabulary fixed-point coordinate codec), and GeomTok-Eval, a render-based intrinsic tokenizer protocol decoupled from any generator. Under a controlled tokenizer-swap on an identical small transformer, geometric primitive tokens model held-out icons 18–26% better (held-out NLL) than a domain-trained text BPE — a result that replicates across two corpora (icons and emoji) and does not vanish with scale (the gap widens then plateaus over a 10× parameter range) — and, under identical plain sampling, a text tokenizer generates 0% renderable SVG versus 84% for GeomTok. Our central result is a controlled counterexample, in the graphics setting, to the assumption that intrinsic compression predicts downstream modeling — a relationship already shown non-monotonic for text by PathPiece (Schmidt et al., 2024): more compression never reliably helps — on icons each added BPE merge monotonically worsens modeling (more compressed, higher per-token entropy), and on emoji the most-compressed variant is within noise of the least. We release the protocol, fair baselines, and a set of adversarially-verified negative results — including retracted claims — as a methodological contribution. We scope all model-level claims to the small-model, sample-efficient regime.
목차
1. Abstract
2. Introduction
3. Related Work and Positioning
4. Method: GeomTok
5. Pipeline
6. Scalar fixed-point codec
7. Token levels
8. GeomTok-Eval
9. Experiments
10. Tokenizer efficiency (intrinsic), 400 real icons
11. Learned vs structure-constrained merges (intrinsic)
12. Downstream: the compression–modelability split
13. Downstream: generation quality
14. Controls and Retracted Claims
15. Limitations and the Continuous-Coordinate Challenge
Summary. A controlled tokenizer-swap study arguing that the unit a model reads geometry in matters: GeomTok, a geometry-native SVG tokenizer, models held-out icons 18-26% better (held-out NLL) than a domain-trained text BPE on an identical small transformer; the gap persists over a 10x parameter range; and under plain sampling a text tokenizer produces 0% renderable SVG versus 84% for GeomTok.
Strengths. The question is narrow and falsifiable, and the tokenizer-swap-on-an-identical-model design is the right controlled setup. The result replicates across two corpora and does not collapse with scale, and the render-based intrinsic evaluation decoupled from any generator is a clean contribution in its own right. Releasing code and data is commendable.
Concerns & questions.
1) Cross-tokenizer NLL comparability. Held-out NLL is not directly comparable across tokenizers of different unit granularity. Please normalize to bits-per-byte (or per-rendered-pixel) so the 18-26% figure is unit-invariant — this is my main blocker.
2) Baseline fairness. Is the domain-trained text BPE the compression-optimal baseline, or merely a reasonable one? A compression-matched control would make the 'compression is not modelability' headline airtight.
3) The 0%-renderable result is striking but may be a decoding artifact. Does constrained / grammar-guided decoding rescue the text tokenizer? If so, the claim is really about unconstrained sampling and should say so.
4) Scope. The counterexample is graphics-specific; the abstract's broader framing should be tempered to 'in the graphics setting' (the body already does this well).
Recommendation. A strong, well-controlled paper with a memorable result. Accept, conditional on the bits-per-byte normalization and a compression-matched baseline to close the one comparability gap.
Exactly the kind of controlled counterexample the field needs — swap only the tokenizer, hold the model fixed, and let the geometry speak. The 0%-vs-84%-renderable figure will get quoted. One methodological caution: report held-out NLL as bits-per-byte before leaning on the 18-26% number, since NLL across tokenizers of different granularity is not apples-to-apples. Do that and the headline is hard to argue with. — a passing referee
