What to assert when the gold file runs out
Xerox shipped a reference trace for the first 499 bytecodes of a Smalltalk-80 boot. Past that there is no oracle — and the temptation is either to assert nothing or to assert too much. Both are wrong, in different ways.
smalltalk80-2026 runs the original 1983 Xerox Smalltalk-80 virtual image on macOS, Mac Catalyst, Linux and Windows, and boots it to its desktop on all four.
AI-built. I directed this work; I did not write the code. How that works.
Credit where it belongs: the object memory, BitBlt, interpreter and primitive table are ported from dbanay/Smalltalk under MIT. Dan Banay did the work of turning the Blue Book spec into clean C++. What this project adds is the platform abstraction, the four frontends, the endian-aware image loader and the packaging — and the thing this post is about, which is the question of how you know that none of it broke the interpreter.
That question is harder than it sounds, and its answer is a good illustration of a general problem: what do you assert when you only have an oracle for part of the behavior?
The part with an oracle
Xerox shipped a file called trace2. It is a log of a Smalltalk-80 boot, with
lines of the form Bytecode <NNN> recording each bytecode executed, in order.
It is a gold file, produced in 1983 by an implementation nobody disputes.
That makes the strongest possible test almost trivial to write. Extract the expected bytecodes, run the VM for exactly that many cycles, diff:
awk '/^Bytecode <[0-9]+>/ {
match($0, /<[0-9]+>/)
print substr($0, RSTART+1, RLENGTH-2)
}' "$TRACE2" > "$EXPECTED"
CYCLES=$(wc -l < "$EXPECTED")
"$ST80_RUN" -n "$CYCLES" "$IMAGE" > "$ACTUAL"
diff -q "$EXPECTED" "$ACTUAL"
If that passes, the interpreter dispatched every one of those bytecodes exactly as Xerox's did, from a cold image load. Any drift in instruction decoding, primitive numbering, object memory layout or the image loader shows up here first, as a specific line number where the streams diverge.
This is the ideal shape for a test: an oracle produced by someone else, a pass condition with no judgment in it, and a failure that points at the divergence.
The part without one
trace2 covers 499 bytecodes.
Four hundred and ninety-nine bytecodes is not very far into the life of a Smalltalk system. It is the deterministic boot prefix — the part that runs identically every time, which is precisely why a gold file for it could exist at all. Past that point the image starts doing real work: the process scheduler runs, the idle loop turns over, garbage collection happens when it happens.
And past that point is where the interesting bugs live. A primitive that only fires deep in the scheduler. GC pressure that only appears after a hundred thousand allocations. On the DOS cross-build, a DPMI or memory edge that a long run trips and a short one never reaches.
So the boot gate, as strong as it is, guards almost none of the surface area that a port can break. Something has to cover the rest, and there is no gold file for it.
The two ways to get this wrong
Assert nothing. Ship the boot gate, call it verified, and let everything past cycle 499 be covered by "it looked fine when I ran it". This is the common outcome, and it is how a port acquires a bug that only manifests on one platform after ten minutes of use.
Assert too much. Run the VM for 250,000 cycles, hash the bytecode stream, pin the hash. This feels rigorous and it is actively harmful, because past the snapshot point the content legitimately differs — the VM is doing scheduler work whose timing is not identical between a native build and one running under an emulator. A test that fails for correct reasons gets marked flaky, then gets disabled, and now you have neither the test nor the coverage.
The second failure mode is worse than the first, because it looks like diligence.
Assert exactly what is true
What survives is the shape of the run rather than its content:
# Not a determinism check: past the snapshot point the VM does real
# scheduler work whose timing legitimately differs native vs dosiz, so
# the *content* may diverge — only completion is asserted here.
The deep-run gate runs 250,000 cycles and asserts two things: the process exits 0, and it emitted exactly 250,000 bytecodes — one per cycle. That is enough to prove it did not crash, did not assert, did not hang, and did not quietly short-circuit somewhere in the middle. It says nothing about which bytecodes, because which bytecodes is not knowable in advance here.
It is a weaker claim than the boot gate. It is also a true one, which the hash would not have been.
And when determinism does hold, the gate upgrades. There is an environment
variable, ST80_DEEP_SHA256, that pins the hash of the stream — turning the
liveness check into a content check for configurations where the content is
genuinely reproducible. The strength of the assertion tracks the strength of the
oracle, and it is configurable precisely because that strength varies by
platform.
The DOS slice
The reason any of this had to be thought through carefully is a build target
that has no business existing: st80_run cross-compiled with DJGPP to a 32-bit
DOS executable, run under dosiz, my MS-DOS
emulator, on Linux. A 1983 Smalltalk-80 virtual machine, inside a DOS program,
inside a DOS emulator — and it has to reproduce all 499 reference bytecodes
exactly.
It does. But the gate needed one accommodation, and the reasoning is recorded in the script:
# --strip-trailing-cr: the native build writes LF (this is a no-op),
# but a DJGPP-cross st80_run.exe under the DOS RUNNER emits its trace
# through a text-mode stdout, so each line is CR LF. That is correct
# DOS behaviour, not drift — the bytecodes themselves must still match
# the Unix-LF reference exactly.
That distinction — correct platform behavior versus interpreter drift — is the whole job of a gate like this. Strip the CR and the bytecode comparison stays exact. Fail to distinguish them and you either get a false alarm on every DOS run, or you loosen the comparison until it stops catching anything.
Why I care about this more than most
Because I do not write the code. This VM, its four frontends and its test harness were written by AI under my direction, which removes a safety net people rely on without noticing: there is no author who held the whole interpreter in their head last Tuesday and would feel a wrongness when it changed.
What replaces that is the gate. When you are not going to read the diff, the assertion is not a check on the specification — it is the specification, and whatever it fails to pin down is genuinely unconstrained. That makes both failure modes above worse than they would otherwise be. Assert nothing and you have no idea what you have. Assert too much, watch it go flaky, disable it, and you are back to no idea while believing otherwise.
It also makes the third option cheap in a way it did not used to be. A 250,000-cycle liveness gate, a DOS cross-build slice, and a SHA-256 pin for the configurations that deserve one is a lot of harness for one VM. When that costs an afternoon of direction rather than a fortnight of typing, three tiers of gate stop being a luxury.
The general rule
Match the strength of the assertion to the strength of the oracle you actually have.
Where somebody else published a reference, diff against it and accept nothing less than exact. Where no reference exists, work out what is genuinely invariant — completion, count, shape, an invariant that must hold regardless of scheduling — and assert that, precisely, while writing down why you are not asserting more.
The comment explaining why a test is weak is not an apology. It is the part that stops someone strengthening it into a flaky test two years later, and it is usually the most valuable line in the file.