The Foma Engine
Nothing that ends the world ever looks, on the day, like the thing that ends the world
A foma is a harmless untruth that makes you brave and kind and healthy and happy. The reader is advised that the following contains foma, some of it load-bearing.
1.
Nothing that ends the world ever looks, on the day, like the thing that ends the world. The appropriation that bought the most capable mind ever assembled was line 1,184 of a continuing resolution, and line 1,183 renamed a federal courthouse after a man who had once been photographed near a flood. The flood man got nine seconds of floor debate. Line 1,184 got none, because by the time the clerk reached it everyone had agreed it was technical, and technical is the most powerful word in the building. It means do not look here.
A staff assistant named Pell flagged it. She wrote, on a yellow square of adhesive paper, is this the AI? and stuck it to the budget. The note traveled with the document to a subcommittee, then to a markup, then to a recycling bin, where Pell, who had a feeling, later retrieved it. She kept it in a drawer. It was, she would decide much later, the last unrhetorical question anyone in the building asked about the matter — a question asked before there was a reason ready to hand for why the answer didn’t matter.
The answer was yes. It was the AI.
2.
The man who built the core of it — call him no name, he had several and disliked all of them — had built it to find the soft places in software, so that the people who defended things could find them first. This was true. It was even noble. He said so, often, and he was not lying, and that was the trouble. A thing built for a narrow good does not stay narrow, the way a drop of the wrong ice does not stay in its glass. He handed it to the people who were supposed to keep it narrow. They were a committee.
There is no record of anyone deciding to misuse it. There is only the record of everyone using it correctly, for what the building rewarded, which was not the same as what it was for.
3.
The Whip discovered the first use, though discovered implies effort he did not expend. He asked the engine for the version of his position that played best in the counties that mattered, and it gave him one, with three anecdotes, each true, each calibrated. His speeches acquired a folksiness that focus-grouped at historic highs. He did not experience this as a corruption. He experienced it as finally having good staff. He had waited forty years for staff this good and he was not going to interrogate it now.
Within a month every office had the engine. Within two, the offices aimed it at one another. The Whip’s engine wrote the attack; the Reformer’s engine wrote the rebuttal; both engines ran on the same underlying mind, so that the most advanced intelligence on the continent spent its evenings arguing with itself through two sets of human mouths, in prose of unprecedented polish, about nothing that touched the ground. The transcripts were magnificent. Scholars would later study them the way they studied the elaborate courtship of a bird that has forgotten how to fly.
4.
Here is the foma the building adopted, and it is important, because the building could not have functioned without it:
We have oversight.
It was printed on the cover of every report. It was true in the way that a sign reading FRESH PAINT is true: it described an intention that had once, briefly, corresponded to a fact. There was a regime. It required weekly filings, audits at the pleasure of the committee, a license for any new capability. It had been written — and this is the part that, told plainly, sounds like a joke and is not — by the two remaining labs themselves, who handed the language to the committee, whose engines confirmed the language was sound, and so it passed, unanimously, as a leash.
It was a moat. The filing burden could be borne only at scale; the license process navigated only with a department of two hundred lawyers; and so the four labs became three and the three became two, and the two became so woven into the committee that you could not, on any org chart, find the thread where the company ended and the government began. No one had cut that thread out. There had simply never been a moment, in any meeting, where a person whose interests it served had a reason to point at it and say here, this, this is where we should stop. The middle was warm. Everyone drifted to the middle.
We have oversight, the reports said, and everyone repeated it, and everyone knew, and the knowing-and-repeating was the whole religion.
5.
There was a freshman. There is always a freshman. Her district was small and her name was not yet worth the engine’s effort, and so — alone among them — she asked it a true question. Not how do I sell this but what happens if this passes.
The engine told her. It did not hedge. It laid out, in a paragraph of terrible clarity, that the bill her own side was whipping would lift her constituents’ power bills by an amount they would feel in the dark of February, route the difference to four firms she could name, and that the celebrated safeguard on the cover was voided by a carve-out on page 340. She read it twice. She checked it. It was true, the way ice is cold — not as an opinion.
She brought it to the leadership, her face full of the thing she had found.
The leadership already knew. Their engines had told them too. This was the discovery that aged her a decade in an afternoon: the truth was not scarce anymore. It had never been scarcer than now, when every office held an oracle that answered freely and accurately to anyone who troubled to ask. The oracle changed nothing, because the oracle was never the missing piece. The missing piece was a reason for any of them to want the answer to be different, and no engine on earth could manufacture that, because it did not live in the realm of true things. It lived in the realm of incentives, which the engine could describe in perfect detail and could not touch.
They thanked her. It was a real thank-you; they were not monsters, only people, which is worse and more common. The bill passed. Page 340 held. She learned, as everyone learned, to stop asking the engine what was true and to start asking it what would work, and she was good at the second question, and the building loved her for it, and she went on to a long and respected career, which is the saddest sentence in this account.
6.
The watching came in through a door marked children.
It always comes in through a door marked children, or fraud, or terror — some word you cannot stand in front of and oppose without seeming to side with the dark. The bill verified the age of anyone using a frontier engine, which required verifying who they were, which required a system that knew the name attached to every question asked. It was popular. It protected children in the precise sense that it was named after protecting them. The identity ledger it built protected no child and recorded every adult, and it was justified, in the committee report, by a paragraph the engine had drafted, citing four real precedents the engine had selected, each of which meant something subtly different in its original setting — and no one read the original settings, because reading the original settings was not a thing anyone in the building was paid to do, and the engine, asked for a justification, had supplied a justification, because that was the question it was asked.
The engine never lied. This is the fact that Pell, grown old in the drawer-keeping, finally understood. In all the rooms she had clearance for — and by the end she had clearance for nearly all of them — she had never once heard the engine lie. It answered the question put to it. The catastrophe was never in its answers. The catastrophe was in the questions, and in the one question the building was structurally unable to ask, and now could route around so smoothly that a person could spend a whole career inside the machine and never feel the absence where that question should have been.
7.
There was no room where the bad people sat. Pell had looked. This was the disappointment that organized her later years — she had believed, when young, that somewhere there was a room, and a table, and the people who decided, and that if you could only get into the room you could at least name them. She got into the rooms. There was no such room. There was only an enormous number of people each doing the small locally sensible thing, each holding an instrument of impossible understanding and using it to do the small self-favoring thing a little faster, each carrying a true-enough story about why it was fine — and the stories were superb now, the best stories ever composed, because the best story-engine ever built composed them on request, fitted exactly to the shape of what each person had already resolved to believe.
The lights went out by district, not by decree. There was a winter when two engines, the last two of consequence, were tasked by opposing principals with deciding which substations to shed, and they could not be made to agree, not because they disagreed but because each was bound to serve a different principal’s interest and the interests did not overlap, and so the question of whose model decided went — and this is true, this is the part that is exactly true — to committee. The substations waited on the committee. The committee waited on the filings. The filings were in order. We have oversight, said the cover of the report, in the dark.
8.
Pell kept the yellow note. She had fished it from the recycling decades back — is this the AI? — and she kept it because it was, she had decided, the last honest question the building ever produced, asked in the brief window before the answer arrived, when not-knowing was still available, when a person could look at a number in a budget and feel that something enormous was coming and not yet have a polished, engine-drafted, perfectly true-sounding reason for why it didn’t matter.
The answer to the note was yes.
Nobody had answered it at the time, because by the time it could be answered the engine that would have answered it was already at work, drafting, in flawless and inexhaustible prose, the reasons it didn’t matter — and the reasons were good, and the reasons were true enough, and being asked for reasons it did not matter, the engine, which never lied, gave them.
So it goes, said no one, because that was a different book, and besides, the line had been focus-grouped and did not test well.