Catacrawl balances itself with a bot. A bot career plays run after run up the difficulty ladder carrying its own levels, coins and gear, and prints how many runs each rung took. Same seed, same code, same answer — that is the whole contract, and every balance decision I have made in four months sits on it.

I ran a 900-second single-seed trace twice, in two separate launches of the game, and diffed them. They agreed for forty-seven seconds and then diverged. On a card.

One loop

When you level up mid-run, Catacrawl deals three cards. The pool is built like this:

for id, lvl in pairs(char.skills) do
  -- a "+N to <skill>" card for every skill this cat has learned
end

and then one random number is drawn and walked against the pool's accumulated weights to pick a card. One random number. Perfectly deterministic — given the pool.

The pool's order decides which card that single number lands on. And pairs() over a string-keyed table has no defined order. In LuaJIT 2.1 it is worse than undefined: the string-hash seed is randomised per process, deliberately, as a defence against hash-collision denial of service. So the iteration order of that table is different in every launch of the game.

Two processes, same seed, same code, different boon on offer at 0:47.

Why it hid for months

Skill cards only enter the pool at magic rarity or better. So a run replays perfectly until its first magic-or-better card, which is often a minute in, and plenty of seeds never diverge at all. Every quick check I had ever done — replay a seed, diff the output — had been done in one process, where the hash seed is fixed for the lifetime of that process and everything agrees with itself beautifully.

The test has to be two processes. A diff of a process against itself passes. That is the part I want to hand to anyone else using a seeded simulation as a measuring instrument: your reproducibility test is probably testing the wrong thing, and it will keep passing.

What it invalidated

Not the game — a player never notices, and it is a roguelike. The instrument. Every "seeded means replayable" claim in my own notes held only within a single run of a tool. So:

• Every A/B I had ever run by comparing today's numbers against a report from last week had been comparing two different games.
• The daily seeded run — one map, one seed, everyone plays the same thing — would have offered two players different boons on the same date. That feature is currently switched off, which is the only reason it never shipped as a visible bug.

The fix is not a new random draw. It is a sort: collect the ids, table.sort them, walk them with ipairs. Same pool, same weights, the same single random number — only its order is now fixed. Three processes now produce identical hands.

The second thing that moves a measurement under you

While confirming the fix I found the other half. My bot re-fits its own decision policy — which skills to buy, which cards to take, what to spend coins on — from my real recorded play sessions, whenever that log has grown past the committed snapshot. It prints one line saying so, and I had stopped reading it.

Which means a career run today and the same career run last week play different policies. I measured it: identical seeds, identical build, 38 / 44 / 51 runs on 17 September and 33 / 36 / 31 on 19 September with no change at all to the thing under test. I had just played some games in between.

So the rule now is: an A/B replays both sides on the same day. An old report is not a "before". I keep a small script that patches the previous values back in and asserts it can reproduce one known number before it trusts itself — because the first version of that script rebuilt a damage formula as a + b*(l-1) when the real one is a + b*l, and it would have cheerfully reported the difference between two wrong numbers as the effect of my change.

None of this was caught by a test. All of it was caught by diffing two things that should have been identical and then refusing to explain the difference away.