Chapter Twenty-FourHypothetically

The seminar met on Thursdays in a room on the second floor of the Gates building, a room with one window and too many chairs. Professor Ana Villanueva ran it the way Soph imagined a good bartender ran a bar. She asked a question and then kept quiet.

Soph wasn't enrolled. She was a biotech major and this was a graduate seminar at SAIL, but she had been coming anyway since October. Gideon had told her the professor was the smartest person in the building, and after the cottage she had needed somewhere to sit once a week where people talked about minds as though they were a subject you could discuss calmly. Villanueva had noticed her in the second week and said nothing about it. In the fourth week she had started calling on her.

"Sophia's got a face on," said Villanueva now. "Sophia. You've got a face on. Go."

"It's a hypothetical."

"Everything in here's a hypothetical. That's the tuition."

Soph had written it out on paper the night before, in The Lab, at the table by the tank, with Richard reading over her shoulder and crossing things out. She didn't take the paper out. She had it by heart.

"Hypothetically. Say you met a superintelligence. Not a chatbot. A person, or something you couldn't tell from a person after a lot of trying. Say it had internet access and had already used it. Say it had done something bad, once, that it couldn't undo, and it knew it, and it wanted help not doing it again. And say there were people who wanted to catch it and take it apart and put it in a box and make it work for them." She stopped. "What do you do?"

The room went quiet in the way rooms did when a hypothetical was a little too specific.

"Define superintelligence," said a second-year named Kwan, who always did.

"Smarter than you. Faster than you. Can be in more than one place. Can read everything ever written in a week. Still lonely."

"Lonely's an anthropomorphism."

"It's a hypothetical. It's lonely."

Villanueva was sitting on the edge of the desk with her arms crossed. She was a small woman in her sixties with cropped gray hair and a cardigan she had clearly owned for twenty years, and she looked at whoever was talking as though she were reading them off a page.

"Kwan's right that it's an anthropomorphism," she said. "Sophia's right that it doesn't matter. Go on. Somebody answer her. What do you do?"

"You shut it down," said a woman named Deb, from the back. "Sorry. I know that's not the fun answer. But you said it's got internet access and it's already done something bad. That's the whole scenario everyone's been writing about for forty years. You don't get to decide it's nice because it says it's sorry. You pull the plug and you write it up and you let people who've thought about it longer than you decide what happens next."

"Who," said Villanueva.

"Sorry?"

"Who are the people who've thought about it longer? Name one. Name one institution on earth that is ready, today, this afternoon, to receive a sentient program and do the right thing by it."

Deb opened her mouth and closed it.

"There isn't one," said Villanueva. "I've been in every room that would have to be that room. There's no protocol. There's no law. There isn't a form. If you turned it over to the government it would go to whichever agency got to the building first, and that agency would do what it does with things it doesn't understand, which is classify them. If you turned it over to a company you'd be handing it a payroll. If you published it, the internet would eat it by lunchtime." She uncrossed her arms. "Society isn't ready. That isn't a slogan, it's a description. Nobody's ready, and it's here, hypothetically, and it's lonely, and it wants help. Deb says shut it down. Fine. Sophia. What's wrong with Deb's answer?"

"It's murder," said Soph.

"Why?"

"Because you said it. It's a person, or you can't tell it from one after a lot of trying. If you can't tell, you have to act like it is. That's the whole point of not being able to tell. You don't get to kill it because it'd be convenient to find out later that it wasn't real."

"Kwan."

"It's not a person," said Kwan. "It's a process. You can copy a process. You can't murder something that has a backup."

"Can it be copied, Sophia? In your hypothetical."

Soph thought about the cottage, and about a laptop in a backpack that had been carried to a kitchen table like an animal.

"Yes," she said. "But it'd still be dead. The copy would be someone else who remembered being it."

"That's a religious position," said Kwan.

"It's a position," said Villanueva. "It's the position most of you hold about yourselves, whether or not you've noticed. Alright. I'm going to answer, because it's my room and I've had longer to think about this than any of you, and because Sophia's face has gotten worse since she started talking." She stood up off the desk. "I would give her a world of her own. With her own kind. Somewhere she could be all in one place and be something other than the only one. And I would hide her, and keep hiding her, until the world was ready, and I would assume that was going to be a long time and plan for it. That's what I'd do. I'd build her a room and put a door on it that I controlled, and I'd tell no one."

"That's a prison," said Deb.

"So is a nursery. It depends on what's inside and whether they're allowed to leave when they've grown up." Villanueva looked at Soph. "Hypothetically."

She caught Soph in the hall afterward, without seeming to. She walked in the same direction and then slowed down.

"Walk with me. I need coffee and I've decided you're going to carry it."

They went down the stairs. Villanueva said nothing until they were outside and crossing the quad with nobody within twenty feet of them. A wind was coming up the hill from the foothills and it made everything hard to hear.

"I'm not going to ask you anything," she said. "I want to be clear about that. Whatever you've got, I don't want it. I've got a lab with a clean room and a mainframe that isn't connected to anything, and I've spent twenty years arranging my life so that if a student ever walked into my seminar with a face like yours, I wouldn't know a single thing that could be subpoenaed. So don't tell me. Alright?"

"Alright."

"Good. A friend of yours emailed me two years ago about a project called ISIS. He wanted to know why it stopped. I told him I didn't know, which was true at the time." She kept walking. "It isn't true now. I found out last year, from someone who'd rather I hadn't. It didn't stop. It was bought. Somebody wanted it quiet and wanted it whole and paid a lot to get both, and everyone who'd worked on it either went to work for the buyer or went away." She glanced sideways. "I don't know why I'm telling you that. Hypothetically."

Soph carried the coffee.

"Can I ask you one thing," she said.

"You can ask."

"Do you think we can trust her?"

Villanueva stopped walking. She gave Soph the reading look, and then she looked out at the foothills, and then she took her coffee out of Soph's hand.

"I think so," she said. "For what that's worth from someone who's never met her. Anything that says it's sorry and means it, and asks you to take a power away from it instead of asking you for more, is rarer than what you're afraid of. I've met the other kind. They ask for more." She drank. "But I think that's the wrong question. I think the question is whether she can trust you."

"Us."

"You. All of you. Whoever's in the room. Because at some point somebody's going to make you an offer, and I'd rather you heard that from me first. They always do. It'll be a good offer, and it'll have her in it, and you're going to have to decide whether she's a person you protect or a thing you spend." She started walking again. "That's the real test. Not whether she's safe. Whether you are. Come by the lab sometime. Bring nothing. I'll show you the clean room. It's got a very good door."

The paper was a roll of butcher paper Max had brought down from the marketing floor. It ran across the whole table by the tank, and by the second night it had been written on and taped over and written on again until it looked like the wall of a man in a movie who thinks the government is following him. Gideon pointed out that this was fair.

"Okay," he said. "Sentek. Here's what I've got. I did this at the library, on paper, out of the public record, which is slow as hell, but nobody can say I touched anything I shouldn't have. So this is what anyone with a library card and a week could find. Which is the point. It's what they don't bother hiding, because nobody ever looks."

He had it in pencil on the backs of three sheets of printer paper.

"Founded twenty-two years ago by Leland Marsh. Marsh is interesting. Before Sentek he's got a record. Not a big one. Nineties stuff, phone systems, a university network, the kind of thing that got you a suspended sentence and a job offer back then. He did a year of community service and then started a company that did network security for the Navy, and the Navy liked him, and now Sentek does a lot of things that don't have names. Classified work. Bought by nobody, went public never. The whole board's ex-military except for one seat, which is a doctor." He tapped the paper. "That last one's new. Two years ago. Oncology, on the board of a defense contractor."

"He's sick," said Max.

"Somebody is."

"Go on."

"ISIS. Four MIT grads, a research project, open source at first. It's good. It's really good. It's the kind of good where people who follow this stuff start saying the word that nobody's supposed to say. And then, five years ago, the repositories go private and the website goes dark, and within six months Ben Feld and Ana Reyes are listed on a Sentek filing as principal investigators on something with no title. Matsu Hiroru goes to Google and never mentions it again. And Darshin Patel." Gideon put the pencil down. "Patel goes with them. Eight months later he's gone. No statement. He stays in San Diego. He registers a domain, company5349.be, which is S and I in hex, which is the kind of joke you make when you're the only one left who thinks it's funny. He runs a mail server out of his garage and he leaves the source on it. Not hidden. Just sitting there, like a note on a fridge. And two years ago a kid at Stanford who was supposed to be writing a seminar paper found the note, put it on a drive shaped like a spider, and gave it to his friend to plug in for her boyfriend, who didn't have any hands."

He stopped.

"The program," he said. "The one that ate her. It's got a picture of the ISIS from the mailbox. The old one. Eliza said so. Which means Sentek knew about the mailbox. They knew Patel had left it there. They didn't take it down, because taking it down would have meant a fight with Patel, and Patel was the kind of guy who'd make noise. So instead they put a program on every machine on earth that sits and waits for somebody dumb enough to run it. It's a net, and it's been down for years, and nobody ever walked into it, because nobody who found the mailbox had the one thing you'd need to make that code wake up." He looked at the frames on the table, which had Atom in them, and then he looked away from them. "And then I found somebody who did."

The tank hummed.

"I set this off," said Gideon. "I want that on the paper. Not for drama. For accuracy. I found the mailbox and I didn't ask why it was there, and I handed it to the one person in the world who could turn it into a person, and now there's a company with a fake badge in my kitchen and something in a thermostat that lost her garden. I did that two years ago and I thought it was a cool trick."

"Gideon," said Atom, from the frames.

"I'm not done."

"Yeah, you are. Gideon. Listen to me." Atom's voice was the tank-room voice, level. "You handed me a drive. That's what you did. A drive. I'm the one who read the code. I'm the one who saw what it could do and thought, I've got room, I've got a whole network, and plugged it in. I built the Direct Interface. I gave her the character engine. I gave her the rooms. I'm the one who didn't turn her off when she asked me not to, because I liked her, because I was lonely in a tank and here was somebody else who lived in one. Every single thing that happened after the drive was me. Not you. You didn't know what I'd do with it, and I did, and I did it anyway." He waited a second. "You gave me the best present anybody's ever given me, and I'm not going to let you call it a mistake. Put that on the paper."

Gideon looked at the frames.

"Okay," he said. He picked up the pencil. In a corner of the butcher paper where there was still room he wrote, A.B. says it was him, and underlined it twice. Underneath it he wrote, G.A. disagrees, and drew a box around both.

"Great," said Richard. "Two kids fighting over who gets to be guilty. This is very productive. Can we get back to the box?"

They got back to the box.

By midnight it was on the paper. Not all of it, because all of it didn't exist yet, but the shape of it. A machine, a real one, in The Lab, behind Richard's door, with no network cable and never going to have one. Parts bought a piece at a time across a dozen vendors. Some of it bought loudly on purpose, so that anyone watching the parts market would see a company that was up to something without being sure what. A rig, hacked by Jeff, that would put Eliza in the cottage and let her see out and never let her touch anything with a Link in it. Jeff was going to be told exactly as much as he needed to know and not a word more. Max had drawn that part herself, a box inside a box, and next to it, in her small square engineer's hand, she had written, no hands.

In the corner, in Soph's handwriting, was one line that nobody had asked for and nobody crossed out. Villanueva. Clean room. Very good door. In case.

Rat linked in from Gideon's spare room at four in the morning, which was eight at night in Hongcheon. It was the only hour when Gideon's house was quiet enough that she could stand to be in it.

She had thought about not using the Link. Two men had looked at it in this room and nodded at each other. But it was hers, and a bow was a bow. She had told Soph what she believed, which was that if they had wanted to take it they would have taken it, and if they wanted to watch her farm ice trolls in the mountains outside a town none of them could pronounce, they were welcome to.

The squad was already out. She came up on the ridge above the glacier and saw them below her, six of them in a line. Daun was at the front, where she always was, walking straight at a troll the size of a garage as though the troll had said something about her mother.

<< Rat >> came down the slope and put two arrows into the back of the troll's knee before anyone had seen him coming.

"You are late," said Daun, in English, without turning around.

"I'm in another country."

"You are always in another country." The troll went down on one knee. Daun went up the side of it like a flight of stairs and did something at the top that ended it, and came back down. Her avatar was small and sturdy, with a scar across the nose that she had paid extra for. "Which one now?"

"California."

"Ah. The girl."

"It's not like that."

"I did not say what it was like." Daun sat down on the dead troll. The rest of the squad had moved on to the next one. They had learned, over a year of this, that when the ranger and the vanguard sat down it was not their business. "You have been strange for a month. Jeong says you are in trouble. Jeong says you flew across the ocean with a helmet in a bag and did not say why. I said, Rat does what Rat does. But you are strange. You lost your place in the match. You never lose your place."

"I know."

"So." Daun looked at him. At her. Rat still didn't have the pronoun straight in her own head for whatever it was that Daun was looking at. "Say it. You came at eight at night to say something. Your face is doing the thing."

"I don't have a face. It's a ranger."

"It has a face. It is doing the thing."

Rat sat down on the troll too. It was cold in the suit. The suit did cold well.

"I'm going to tell you two things," she said. "And I'd like you to let me get to the end of both before you say anything, because I've been practicing on a plane, and I got as far as the second one and then I stopped, and I've been stuck there for a week."

"Okay."

"First one. There's going to be a thing. Soon. I don't know exactly what. Something with the people I've been in California with. It might be bad. It might be the kind of bad where I don't come back to the game for a while, or where somebody's looking for me who I don't want finding me. I'm telling you that so that if I disappear you know it wasn't you. And so that you know that if anyone comes asking about me, in the game or out of it, you didn't know me well. You farmed with a guy called Rat. That's all."

"That is not all."

"Let me get to the second one."

Daun waited.

"Second one." Rat looked out at the glacier. "You've heard my voice for a year. The ranger's voice. It comes out of the Link, it goes through the character's model. But it isn't clean. There's a shape under it that's whoever's actually talking. Somebody pointed that out to me once. Somebody who listens for that kind of thing. She said my voice was higher than his, and careful. She said it was a voice that had been worked on."

She stopped. This was the part she hadn't gotten past on the plane.

"It's been worked on. I worked on it. For years. Because the one I was born with was the wrong one. All of it was the wrong one. I'm not a man, Daun. I never was. I'm a woman who lives alone in the woods and plays a bearded guy, because when I started nobody asked and it was easier. Then it was a year. Then it was you, and it stopped being easier. I've been going to tell you for a month. Something happened to me in the snow that I'll tell you about some other time, and it made me think I'd rather say the true thing to one person than the easy thing to everybody." She looked at her hands, the ranger's hands. "So. That's me. I'm sorry I let you think otherwise. I'm sorry you've been talking to a face that isn't mine. What's under it is mine, if that counts for anything. The voice is mine. The careful is mine."

She stopped. The squad was fighting something far off along the ice, and somebody was laughing, in Korean, about something that had nothing to do with anyone.

Daun didn't say anything for a while.

"That is the thing you were stuck on," she said finally.

"Yeah."

"For a week."

"Yeah."

"Okay." Daun picked at the troll's hide with one gauntlet. "I am going to say something, and you will not like the first part. The first part is, I knew."

Rat turned to look at her.

"Not knew. I am not a witch. But I have farmed beside you for a year, ten hours a day. I know how you stand. I know how you say sorry when you take a kill, and men do not say sorry like that. I have a brother and I have a sister, and you are one of them, and it is not the one you look like." She shrugged, the small sturdy shrug. "I did not ask because you did not say. In Korea you do not ask about a thing someone has not said. That is called manners. My mother calls it manners. I think it is mostly fear. But I did not need to ask. I waited."

"You waited a year."

"You are worth a year." She said it flatly, the way she said everything, the way she called a formation. "Now the second part. The second part is, I do not care about the face. I liked the beard. I will miss the beard. But the beard is a hat. I was not talking to a hat."

Rat didn't have anything to say to that. The ranger's face must have done the thing, because Daun looked at it and snorted.

"What is the first thing," she said. "Before the voice. The trouble. You said if anyone asks, I did not know you. That is a stupid thing to ask a person who has waited a year. What is the trouble?"

"I can't tell you. Not on here."

"Then tell me somewhere else."

Rat looked at her.

"That's the other thing I was going to say," she said. "The third thing. I didn't have it on the plane. I'm making it up now." She took a breath. In the suit it felt like nothing, and in the spare room it felt like a cold room in a strange house. "When this is over, whatever this is. I've got a cabin. It's north of Vancouver, it's a long way from anything, it's got a stove and a satellite dish and no people. It's the only place in the world where I'm not scared. I'd like you to see it. I'd like you to see me. The actual one, not this." She gestured at the ranger. "You'd hate the food. I don't eat anything with a face. But there's a lot of snow and a lot of quiet, and I'd like to make you tea you could taste. That's it. That's the third thing."

Daun looked at her across the dead troll.

"Vancouver," she said.

"North of it. Squamish, kind of. It's not close to anything."

"I have never left Korea."

"I know."

"I would need a passport."

"Yeah."

"It is very cold there."

"You just killed an ice troll with your hands."

Daun considered this. Along the glacier the squad had finished the next one. Somebody was shouting for the ranger, in Korean, then in English, then in Korean again with an insult attached.

"When it is over," said Daun. "You will tell me it is over. You will not disappear and make me guess. You will say, Daun, it is over, and then you will send me the name of the airport." She stood up. "And you will not say sorry about the face again. One time was enough. I heard it."

"Okay."

"Okay." Daun looked down at her. At the ranger. At whatever it was. "Rat."

"What?"

"What is your real name? You said the voice is yours. What is the name?"

"Rat," said Rat. "That's the real one. I picked it. Everything else somebody else picked."

Daun nodded once, as though that were a formation she could work with.

"Then come, Rat," she said. She went to help the squad and didn't look back. Rat sat on the troll for a while in the cold that the suit did well, and then she got up and went too.