Megdan is a five-minute quiz for three players, and the results are ammunition in a nightly war over the real map of Serbia. Every evening the day’s matches decide which municipality holds which front.
That last sentence is the whole constraint. In an ordinary quiz app a wrong answer key is a one-star review. Here it moves a border. Somebody answers correctly, the game says no, and their town loses ground because of my mistake.
So the question was never “can a model write trivia in Serbian”. It can, all day, for almost nothing.
Generation is the cheap part. Everything else exists because a wrong question takes somebody’s territory.
I’m not going to publish the recipe - the prompts, the models, the sources and the thresholds are the part that took the work. But the shape, and what it taught me, travels well beyond quizzes.
The Model Does Not Vouch for Itself
In the workflow post I wrote that LLMs aren’t unsafe because they’re dumb, they’re unsafe because they’re confident. A model that writes a question and is then asked “is this correct?” says yes. Of course it does. It wrote it.
The whole design is that one distrust, repeated:
write → cross-examine → check against sources → clean up → a person reads
One model writes. Something that didn’t write it, and has different blind spots, tries to break it. Facts are tied to a real source. What survives is de-duplicated and tidied. And a named human reads before anything reaches a player.
It’s a funnel where each step distrusts the one before it. It runs offline, away from the game, and every question that comes out carries its receipt: what made it, from what, and when. A bad batch can be traced and withdrawn whole.
Three rules hold it together.
Model output is untrusted data. It gets parsed and validated like input from the internet, because that’s what it is.
A discard is a discard. A doubtful question doesn’t go to a review queue where a tired person approves it at midnight. It’s gone. Drafts are cheap, remember.
What can be checked by code is checked by code. A model asked to show its source will, some of the time, describe the source it wishes existed. No prompt fixes that. A dumb, deterministic check does. Wherever I could replace “ask a model whether” with a few lines of code, the pipeline got better.
The Checker Needs a Test Too
Here’s the part I got wrong first.
Before running anything at volume, the pipeline had to pass a gate: a set of hand-labelled questions, good ones and deliberately broken ones, each with a known defect. The validation has to throw out the broken and keep the good. Simple.
On the first run it kept about a quarter of the good questions.
One of my checks looked for answer options that give the key away. The way I’d built it, it didn’t find giveaways. It found fame. It killed a question about which river flows through Zemun because the Danube is the famous river, and one about the Battle of Kosovo because 1389 is the famous year. For an easy question the famous option is the answer. That’s what easy means. Meanwhile, deliberately absurd option sets sailed through.
I had asked one question and measured a different property. Only one of them was a defect.
Without the labelled set I’d never have known. The pipeline would have run, produced plausible output, and quietly thrown away the best questions while keeping some of the worst. It would have felt rigorous.
The gate failed four times before it passed. Each failure was a wrong assumption of mine, and each was found for the price of a lunch instead of a season. That’s the cheapest debugging I’ve ever bought.
Where the Effort Actually Goes
Roughly half of what gets drafted survives. The writing is a small slice of the cost. Checking against sources is most of it, and most of the losses are honest ones: the fact is probably true, but nothing I’m willing to trust says so in a way the question needs.
Which settles an argument before it starts. I tried a faster, cheaper writer. It was clean and quick, and it kept circling the same two or three famous events. No automated check measures taste or variety, so the writer is the one place worth paying for it. Saving on the small slice to lose in the thing nobody measures is a bad trade.
A few other things I’d tell myself at the start:
Variety is designed, not sampled. Left alone, a generated corpus drifts toward one voice and one set of famous facts. Turning a randomness knob doesn’t fix that. Deciding in advance what to ask about does.
The defects you plan for aren’t the ones you get. One question asked which team Serbia played in an Olympic final without saying which sport. The writer knew. Every check knew. A player with a clock running didn’t. It took a person reading the output to see it, and then it became a check.
Reliability is boring and decides everything. Rate limits, outages, dropped connections. A long run that dies on a hiccup costs more than any model choice. Waiting patiently is a feature.
The human read stays. The pipeline’s job isn’t to replace it. It’s to make it a read of plausible questions instead of hopeful ones.
The architecture is not clever. It’s distrust, arranged in order, with a test for the distrust itself.
In the first post of this series I called these things glorified auto-complete. A quiz question is a fine thing to auto-complete. An answer key isn’t.
The model writes the question. It doesn’t get to mark its own homework.
