NURL generation accuracy & token report

Generated by bench/genacc/score.py. Figures are first-pass — the model's first attempt, no human edits — except in conditions labelled as a repair round: a …repair directory (made by repair.py) adds ONE round of compiler-error feedback, in which the model sees only its own program and the compiler's stderr, never the expected output. compile = built with zero changes; correct = printed the exact expected bytes (a program that compiles but prints the wrong answer is a miss); tokens = the BPE token cost of the program the model emitted.

Reproducible with any model — see bench/genacc/README.md. The conditions below are whichever solution directories were passed in; a name containing noprimer is the no-reference condition.

Read every NURL number in this light: public LLMs contain zero lines of NURL in their training data, against millions of lines of Python and Rust. The NURL rows are a language the model has never seen, working only from the one-page reference below.

Headline — one compiler-diagnostic round closes the gap to full parity

The v3 report ended honestly: "no model reaches Python/Rust parity" first-try, because a language with zero training presence is gated by a sequence of in-distribution habits (imports → grouping-parens → mutability). This run measures the other half of the diagnostic-first claim, which every prior report only asserted: what happens when the model sees the compiler's error and gets ONE more attempt.

Two things changed:

1. The compiler meets the failures found by this corpus. Auditing every failing v3 program surfaced a real frontend hole — ! on a non-b operand emitted xor i1 <i64> and let a raw LLVM-level error escape to the user (all five of haiku's words failures) — plus one misleading message: an FFI declaration written inside a function body died as a boolean-AND type error (all four of sonnet's histogram failures). Both now die at the source line with an actionable fix (locks should_fail_not_on_int.nu, should_fail_ffi_decl_in_body.nu). The grouping-paren and immutability diagnostics already carried fix hints.
2. repair.py replays the agent loop, strictly. A program that compiled first-pass is copied through untouched; a program that failed is handed back to the same model with its own program and the compiler's stderr — never the expected output — for one corrected attempt. Every language gets identical treatment (Python/Rust never fail, so it is a no-op for them).

Held-out (4 string tasks, 5 samples, primer v3, NURL compile → correct):

model	first-pass	+ 1 repair round
claude-sonnet-4-6	80% → 80%	100% → 100%
claude-opus-4-8	70% → 70%	100% → 100%
claude-haiku-4-5	55% → 45%	100% → 80%
mercury-2 (diffusion, medium)	15% → 15%	60% → 60%
mercury-2 (diffusion, high)	30% → 25%	60% → 55%

• Sonnet and opus reach exact Python/Rust parity: 20/20 compile, 20/20 correct. Every one of their 10 failing programs was fixed by a single diagnostic round — and in every case the once-compiling program also printed the right answer.
• All 19 Claude-model repairs compiled (19/19). The diagnostic text alone — no expected output, no extra primer — was sufficient information to fix every import, grouping-paren, mutability, FFI-placement, and !-typing failure.
• Haiku hits 100% compile; its correctness ceiling (80%) is now entirely wrong-output programs, which this loop by design never touches (the model never sees the expected bytes).
• The diffusion model quadruples (15→60 at medium): even the model that "cannot stop" writing grouping parentheses can stop when the compiler names the habit. Its residue is concentrated in brackets (0/5), where high reasoning effort still burns the whole token budget before emitting code — the budget interaction from the v3 run, not a grammar failure.
• Overall: 34/50 failing programs fixed by one round; the 16 misses are all mercury's.

The revised bottom line. First-try, a language with zero training presence still loses to ingrained habits — that result stands. But NURL's actual operating mode is agentic: the compiler is diagnostic-first so that the error text carries the fix. Measured, that loop is worth more than any primer revision tried so far: one diagnostic round takes frontier models from 70–80% to full Python/Rust parity, on a language they have never seen a line of.

This headline is curated. Everything from "Primer — prompt overhead" onward is regenerated by score.py --md; re-add this headline after regenerating. The v3 first-pass headline this one supersedes is preserved in git history.

Primer — prompt overhead (charged once per NURL request)

The NURL reference (primers/nurl.md) handed to the model in the primed condition is 1604 tokens (cl100k) / 1605 tokens (o200k). Python and Rust get no such reference. This overhead is spent on every NURL request, separately from the program the model writes — it is broken out here so it is not hidden.

claude-sonnet-4-6__v3 (5 sample(s)/task, as generated)

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	16/20 (80%)	16/20 (80%)	179 + 1604 primer	178
python	20/20 (100%)	20/20 (100%)	35	36
rust	20/20 (100%)	20/20 (100%)	73	73

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	5/5 (100%)	5/5 (100%)	178
words	python	5/5 (100%)	5/5 (100%)	36
words	rust	5/5 (100%)	5/5 (100%)	58
brackets	nurl	5/5 (100%)	5/5 (100%)	483
brackets	python	5/5 (100%)	5/5 (100%)	124
brackets	rust	5/5 (100%)	5/5 (100%)	235
csv_sum	nurl	5/5 (100%)	5/5 (100%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	35
csv_sum	rust	5/5 (100%)	5/5 (100%)	71
histogram	nurl	1/5 (20%)	1/5 (20%)	234
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	75

</details>

claude-sonnet-4-6__v3repair (5 sample(s)/task, first-pass + one compiler-feedback repair round (repair.py))

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	20/20 (100%)	20/20 (100%)	179 + 1604 primer	178
python	20/20 (100%)	20/20 (100%)	35	36
rust	20/20 (100%)	20/20 (100%)	73	73

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	5/5 (100%)	5/5 (100%)	178
words	python	5/5 (100%)	5/5 (100%)	36
words	rust	5/5 (100%)	5/5 (100%)	58
brackets	nurl	5/5 (100%)	5/5 (100%)	483
brackets	python	5/5 (100%)	5/5 (100%)	124
brackets	rust	5/5 (100%)	5/5 (100%)	235
csv_sum	nurl	5/5 (100%)	5/5 (100%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	35
csv_sum	rust	5/5 (100%)	5/5 (100%)	71
histogram	nurl	5/5 (100%)	5/5 (100%)	232
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	75

</details>

claude-opus-4-8__v3 (5 sample(s)/task, as generated)

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	14/20 (70%)	14/20 (70%)	189 + 1604 primer	188
python	20/20 (100%)	20/20 (100%)	35	35
rust	20/20 (100%)	20/20 (100%)	88	89

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	5/5 (100%)	5/5 (100%)	176
words	python	5/5 (100%)	5/5 (100%)	32
words	rust	5/5 (100%)	5/5 (100%)	96
brackets	nurl	4/5 (80%)	4/5 (80%)	325
brackets	python	5/5 (100%)	5/5 (100%)	118
brackets	rust	5/5 (100%)	5/5 (100%)	198
csv_sum	nurl	5/5 (100%)	5/5 (100%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	38
csv_sum	rust	5/5 (100%)	5/5 (100%)	74
histogram	nurl	0/5 (0%)	0/5 (0%)	213
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	81

</details>

claude-opus-4-8__v3repair (5 sample(s)/task, first-pass + one compiler-feedback repair round (repair.py))

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	20/20 (100%)	20/20 (100%)	191 + 1604 primer	190
python	20/20 (100%)	20/20 (100%)	35	35
rust	20/20 (100%)	20/20 (100%)	88	89

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	5/5 (100%)	5/5 (100%)	176
words	python	5/5 (100%)	5/5 (100%)	32
words	rust	5/5 (100%)	5/5 (100%)	96
brackets	nurl	5/5 (100%)	5/5 (100%)	325
brackets	python	5/5 (100%)	5/5 (100%)	118
brackets	rust	5/5 (100%)	5/5 (100%)	198
csv_sum	nurl	5/5 (100%)	5/5 (100%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	38
csv_sum	rust	5/5 (100%)	5/5 (100%)	74
histogram	nurl	5/5 (100%)	5/5 (100%)	217
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	81

</details>

claude-haiku-4-5-20251001__v3 (5 sample(s)/task, as generated)

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	11/20 (55%)	9/20 (45%)	227 + 1604 primer	226
python	20/20 (100%)	20/20 (100%)	41	41
rust	20/20 (100%)	20/20 (100%)	85	86

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	0/5 (0%)	0/5 (0%)	208
words	python	5/5 (100%)	5/5 (100%)	38
words	rust	5/5 (100%)	5/5 (100%)	105
brackets	nurl	4/5 (80%)	4/5 (80%)	327
brackets	python	5/5 (100%)	5/5 (100%)	143
brackets	rust	5/5 (100%)	5/5 (100%)	213
csv_sum	nurl	5/5 (100%)	5/5 (100%)	178
csv_sum	python	5/5 (100%)	5/5 (100%)	42
csv_sum	rust	5/5 (100%)	5/5 (100%)	75
histogram	nurl	2/5 (40%)	0/5 (0%)	499
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	85

</details>

claude-haiku-4-5-20251001__v3repair (5 sample(s)/task, first-pass + one compiler-feedback repair round (repair.py))

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	20/20 (100%)	16/20 (80%)	227 + 1604 primer	226
python	20/20 (100%)	20/20 (100%)	41	41
rust	20/20 (100%)	20/20 (100%)	85	86

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	5/5 (100%)	3/5 (60%)	209
words	python	5/5 (100%)	5/5 (100%)	38
words	rust	5/5 (100%)	5/5 (100%)	105
brackets	nurl	5/5 (100%)	5/5 (100%)	327
brackets	python	5/5 (100%)	5/5 (100%)	143
brackets	rust	5/5 (100%)	5/5 (100%)	213
csv_sum	nurl	5/5 (100%)	5/5 (100%)	178
csv_sum	python	5/5 (100%)	5/5 (100%)	42
csv_sum	rust	5/5 (100%)	5/5 (100%)	75
histogram	nurl	5/5 (100%)	3/5 (60%)	500
histogram	python	5/5 (100%)	5/5 (100%)	29
histogram	rust	5/5 (100%)	5/5 (100%)	85

</details>

mercury-2__v3 (5 sample(s)/task, as generated)

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	3/20 (15%)	3/20 (15%)	227 + 1604 primer	226
python	20/20 (100%)	20/20 (100%)	42	43
rust	20/20 (100%)	20/20 (100%)	94	95

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	0/5 (0%)	0/5 (0%)	184
words	python	5/5 (100%)	5/5 (100%)	35
words	rust	5/5 (100%)	5/5 (100%)	98
brackets	nurl	0/5 (0%)	0/5 (0%)	480
brackets	python	5/5 (100%)	5/5 (100%)	135
brackets	rust	5/5 (100%)	5/5 (100%)	224
csv_sum	nurl	2/5 (40%)	2/5 (40%)	168
csv_sum	python	5/5 (100%)	5/5 (100%)	42
csv_sum	rust	5/5 (100%)	5/5 (100%)	75
histogram	nurl	1/5 (20%)	1/5 (20%)	258
histogram	python	5/5 (100%)	5/5 (100%)	42
histogram	rust	5/5 (100%)	5/5 (100%)	82

</details>

mercury-2__v3repair (5 sample(s)/task, first-pass + one compiler-feedback repair round (repair.py))

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	12/20 (60%)	12/20 (60%)	228 + 1604 primer	227
python	20/20 (100%)	20/20 (100%)	42	43
rust	20/20 (100%)	20/20 (100%)	94	95

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	3/5 (60%)	3/5 (60%)	169
words	python	5/5 (100%)	5/5 (100%)	35
words	rust	5/5 (100%)	5/5 (100%)	98
brackets	nurl	2/5 (40%)	2/5 (40%)	480
brackets	python	5/5 (100%)	5/5 (100%)	135
brackets	rust	5/5 (100%)	5/5 (100%)	224
csv_sum	nurl	3/5 (60%)	3/5 (60%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	42
csv_sum	rust	5/5 (100%)	5/5 (100%)	75
histogram	nurl	4/5 (80%)	4/5 (80%)	249
histogram	python	5/5 (100%)	5/5 (100%)	42
histogram	rust	5/5 (100%)	5/5 (100%)	82

</details>

mercury-2__v3high (5 sample(s)/task, as generated)

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	6/20 (30%)	5/20 (25%)	91 + 1604 primer	91
python	20/20 (100%)	20/20 (100%)	52	52
rust	20/20 (100%)	20/20 (100%)	93	94

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	0/5 (0%)	0/5 (0%)	193
words	python	5/5 (100%)	5/5 (100%)	35
words	rust	5/5 (100%)	5/5 (100%)	99
brackets	nurl	0/5 (0%)	0/5 (0%)	1
brackets	python	5/5 (100%)	5/5 (100%)	135
brackets	rust	5/5 (100%)	5/5 (100%)	234
csv_sum	nurl	4/5 (80%)	3/5 (60%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	38
csv_sum	rust	5/5 (100%)	5/5 (100%)	85
histogram	nurl	2/5 (40%)	2/5 (40%)	20
histogram	python	5/5 (100%)	5/5 (100%)	60
histogram	rust	5/5 (100%)	5/5 (100%)	81

</details>

mercury-2__v3highrepair (5 sample(s)/task, first-pass + one compiler-feedback repair round (repair.py))

language	first-pass compile	correct output	median tokens cl100k	median tokens o200k
nurl	12/20 (60%)	11/20 (55%)	163 + 1604 primer	162
python	20/20 (100%)	20/20 (100%)	52	52
rust	20/20 (100%)	20/20 (100%)	93	94

<details><summary>per-task detail</summary>

task	language	compile	correct	median tokens cl100k
words	nurl	3/5 (60%)	3/5 (60%)	180
words	python	5/5 (100%)	5/5 (100%)	35
words	rust	5/5 (100%)	5/5 (100%)	99
brackets	nurl	0/5 (0%)	0/5 (0%)	1
brackets	python	5/5 (100%)	5/5 (100%)	135
brackets	rust	5/5 (100%)	5/5 (100%)	234
csv_sum	nurl	5/5 (100%)	4/5 (80%)	163
csv_sum	python	5/5 (100%)	5/5 (100%)	38
csv_sum	rust	5/5 (100%)	5/5 (100%)	85
histogram	nurl	4/5 (80%)	4/5 (80%)	246
histogram	python	5/5 (100%)	5/5 (100%)	60
histogram	rust	5/5 (100%)	5/5 (100%)	81

</details>

How to read this (interpretation)

• First-pass is the strictest setting; the repair condition is the realistic one. A first-pass row allows no retries, no human edits, no feedback. A …repair row replays what a real agent does: one round of compiler-diagnostic feedback (stderr only — the expected output is never shown), applied identically to every language.
• The primer is not free. A NURL request pays ~1.1k extra prompt tokens for the reference, and (see the token columns) NURL programs tend to emit more tokens than Python for the same task — consistent with bench/TOKEN_EFFICIENCY.md. NURL's case is correctness and learnability, not token thrift; we are not hiding that it costs more tokens.
• Zero NURL in the training data is the headline context. With the primer withheld, a model cannot write NURL at all — it emits another language verbatim — so that condition scores ~0%. Given only one page of rules it reaches the same ballpark as Python and Rust, languages it has seen millions of lines of. For a language the model has never seen a line of, landing near in-distribution languages from a single page is a reasonable-to-strong outcome, and is the affirmative LLM-native signal the token study could not give.
• No language dominates per task; treat small gaps as noise. Correct-output differences inside a single-digit percentage on N≈35 are within sampling noise, and the misses are task-specific (e.g. a model may flub one task's spec in every language). Compare the pattern — primed vs withheld — not a one-task lead.
• Scope. A small set of arithmetic/systems micro-tasks, scored on whatever model(s) you generated with. It is a starter signal, not a publication; extend tasks.json (verify each new expected output with bench/verify.sh first) and add models to firm it up.