A tiny model vs. the DHCP relay spec
It scored 100% on the DHCP relay spec — by memorizing it. Vulcan scores 100% by learning it.
We cloned a ~1-million-parameter model and trained it from scratch on the DHCP relay decision — RFC 1542 and RFC 3046, the rules a relay agent follows for every packet. It reached a perfect 100% on a fresh, held-out battery. Watch it run below — on real packets from a VM booting locally — and then watch a single hop it was never shown pull the 100% apart. Vulcan runs the same decision the other way: instead of training on examples and hoping the pattern holds, it interprets the spec directly — so it correctly interprets 100% of the spec, including the hop the trained model never saw.
● real packet, parsed from the capture · ● spec scenario · ⚠ real adversarial probe — a bad actor trying to register. Every verdict is the model's own generated output.
The reveal — the honest failure mode
100% was memorization. Here's the proof.
A DHCP relay decision is a small, fully-enumerable function — so a model with enough room can just memorize the table and score 100% without learning anything. To tell memorization from understanding, we trained the hop-count step on relay depths 0–23 and then asked for depths 24–31 it had never seen.
| How numbers are shown to the model | Seen range (0–23) | Unseen range (24–31) |
|---|---|---|
Whole tokens (hop 24) |
100% | 0% |
Digits (hop 2 4) |
100% | 62% |
With whole-number tokens it scores a flat 0% on anything unseen — the symbols for 24–31 were never in its vocabulary, so it literally can't say them. Give it the digits and it does better — but look where it breaks:
It nails 24→25 through 28→29 — it generalized "add one to the last digit," a pattern the
training data taught. But at 29→30 it fails: it rolls the 9 to 0 and never
carries the tens digit up to 3 — because a leading 3 never once appeared
in its training answers (the largest was 24). It didn't learn to count. It learned the
exact keystrokes it was shown, and stopped at the first character it had never been asked to
write.
One of our adversarial probes floods the relay with the hop count jammed to 255. Vulcan
discards it in a single line — hops > 16 → discard —
because it's applying the rule, not recalling a pattern. The trained model, never shown a
hop value that large, emits nonsense ("14 option82 giaddr add option82…") —
the identical out-of-range blind spot from above, this time as a way in. Vulcan catches
the attack; the confident model doesn't. That is the whole argument, in one packet.
Nothing was wrong with the model or the training — this is what neural pattern-matching is, even when the headline number is 100%. Vulcan takes the opposite path: instead of training on examples and hoping the pattern generalizes, it interprets the spec once and applies it — so it correctly interprets 100% of the spec, not just the 100% it happened to be shown. When the cost of a confident-but-wrong answer is real — a vehicle bus, a safety case, a network that has to behave — you don't want a score you have to trust. You want a verdict that shows the rule it checked. That's what the other two demos do.
Details
The spec it learned, the packets it saw, and how it was built
No hidden benchmark, no cherry-picked run — the grounding, the decision set, and the provenance are all here. Expand what you want to inspect.
Real packets, real verdicts — a VM booting locally
We booted a throwaway virtual machine that network-boots over DHCP — literally a machine trying to come up on the network — and captured its handshake off the virtual wire. Each real packet below was parsed and handed to the model as-is:
A locally-booting VM talks to the server directly, with no relay in the path (relay address 0). So the model correctly says: the client's requests are the kind a relay would forward, while the server's replies — not addressed through any relay — are not this relay's to handle. Right call, real data.
The decision set — every verdict the relay can reach
There is no learned weight quietly deciding an action. The relay's behavior is a fixed set of decisions from the spec; the model was trained to reproduce them, and the held-out battery scores it against Vulcan's spec oracle. Ten decisions, both directions:
Grounding — the published standards, and only those
Every decision traces to a published, active IETF standard. No experimental drafts, no invented behavior:
| RFC | Title | Status | Used for |
|---|---|---|---|
| 2131 / 2132 | Dynamic Host Configuration Protocol & Options | Draft Std | message types, DHCPNAK-is-broadcast |
| 1542 | Clarifications & Extensions for the BOOTP | Draft Std | giaddr, hop threshold, unicast-to-server, broadcast flag |
| 3046 | DHCP Relay Agent Information Option (82) | Informational | insert Option 82 upstream, strip it on the reply |
| 8415 | DHCP for IPv6 (DHCPv6) | Proposed Std | a second, harder relay battery (nested relay-forward / hop limit) |
How it was built
A from-scratch decoder-only transformer (~1M parameters), trained on Apple-silicon GPU in about two minutes. It reads a packet's relay-relevant fields written in a small controlled English and generates the verdict word by word — a generative model, graded by exact match against the spec oracle on a freshly-seeded battery it never saw in training. The same recipe hits 100% at a tenth the size, which is the whole point of the reveal above: on a fully-enumerable spec, size buys memorization, not understanding. Vulcan doesn't need either — it interprets the spec directly, so there's no table to memorize and nothing to buy with more parameters.