The Human Is Becoming Latency

The world wants humans to remain in control of autonomous weapons. Modern warfare may reward everyone who removes them.

Humanity finally reached a glorious agreement on dreaded killer robots last week. 

In my own backyard over in Geneva!

I think this maybe, just maybe, qualifies as significant news. Don’t know, though; could just be all the Sci-Fi I read…

After more than a decade of negotiations, 128 countries meeting in Switzerland last week agreed on a document that could eventually, possibly, form the basis for international rules governing “lethal autonomous weapons.” 

Broadly speaking, these are weapons capable of selecting and engaging targets with varying degrees of autonomy once activated, although getting governments to agree on exactly where that definition begins and ends has proven to be part of the problem.

The text is unsurprisingly non-binding, fairly typical of bureaucrats who have a knack for emulating snails.

Sadly, the definition of ‘autonomous weapon’ was reportedly weakened during the final stages of those negotiations.

Strange bedfellows emerged, with the United States and Russia continuing to favor national guidelines over legally binding international restrictions.

So after ten whole years of impressively slow deliberation, humanity has finally managed to agree that perhaps we should maybe take the bold step of continuing discussions about whether or not machines should be allowed to decide who dies. 

‘Who’ refers to our fellow humans if that wasn’t clear enough.

In fairness, this is how diplomacy is known to work. Or not work?

Nuclear arms accords weren’t merely decided over a quick and pain-free lunch either.

But there is something almost poetic about our beloved diplomat friends debating the precise meaning of "human control" while defense companies and militaries are busy building super weapons designed to operate an order of magnitude faster than humans can.

Everyone seems quite comfortable with allowing a human to remain somewhere in the process of taking a life. Or many.

So that means we’re allowed to have artificial intelligence. We can still make use of our autonomous drones. 

We can automate targeting, surveillance, navigation, and threat assessment, and most of the decision-making too. 

But somewhere, somehow, there should still be a human exercising judgment.

The trustworthy adult in the room.

The problem is that the pesky human grown-up is slow. Which presents a problem. And modern warfare has famously never been very sentimental about bottlenecks.

PLEASE CONFIRM YOU WOULD LIKE TO KILL THIS PERSON

Imagine for a moment that a drone spots an SUV moving through a ‘contested’ area.

Software identifies its shape and cross-references it with a database replete with other known military vehicle shapes and sizes.

Other sensors analyze its movement, heat signature, and location. A model assesses the probability that it belongs to the enemy. 

In the background, another system layer checks the surrounding area for civilians. Yet another determines which instrument of death is most suitable and immediately available.

By the time a soldier sees anything, the machine has already managed to convert a messy piece of reality into something a whole lot easier to process.

HOSTILE TARGET: 94% CONFIDENCE (shouldn’t it be 100%?)

CIVILIAN RISK: LOW (whatever this means)

ENGAGEMENT RECOMMENDED (by whom?!)

APPROVE?

The soldier clicks yes.

Technically, there was a human in the loop. Just like Italian luxury fashion brands technically “assemble” their accessories in Italy (after having them shipped from Asia), or American car companies “assemble” their cars domestically and definitely not in Mexico. 

But I am not sure I would describe that human as the person who made the decision.

The more of the process we automate, the stranger that distinction tends to become. And these cutting-edge technologies are really starting to make it easy to do something unforgivable with the unconscious click of a button.

A person with a meager several seconds to approve an analysis assembled by systems they cannot possibly reproduce in their own head may have legal authority over the decision without possessing much practical agency over it.

Give them fifty targets instead of one and the problem gets worse. Maybe make the required response time half a second?

Or give them five hundred targets, and now we’re totally raining down chaos.

Eventually, "human control" starts looking a hell of a lot like one of those useless boxes confirming that you have indeed read the 345,000 terms and conditions for your iPhone update.

You haven't, and everyone, including Apple, knows you haven't.

But legally, congratulations, you’ve participated. Here’s your participation ribbon!

There is actually a name for part of this problem: automation bias. Humans have a tendency to place too much faith in recommendations generated by computers simply because a computer generated them. 

In military decision-support systems, that creates the slightly horrifying possibility that the human supposedly exercising judgment becomes little more than a rubber stamp for whatever the machine recommends.

Which seems less than ideal when the rubber stamp launches a missile.

This is the fork in the road where the Geneva debate starts becoming much more interesting than the phrase "killer robots" suggests. And I think we can all agree that “killer robots” already sounds pretty interesting. Proven by the millions of Hollywood movies about killer robots and Terminators.

The reassuring version of human control likes to imagine a soldier looking through a camera, assessing a target, and in turn using their combined experience and wisdom to make an informed and conscious decision to fire. 

In reality, relevant human judgment could and probably should occur much earlier. 

People can determine geographic boundaries, target categories, operating conditions, and rules of engagement governing when autonomous systems are permitted to act.

The human doesn't necessarily disappear but they do take a tremendous step into the background.

First a person chooses the target >>> They approve a target selected by a machine >>> They approve a category or cluster of targets >>> They approve an entire area where a machine can attack that category of target.

This slippery slope eventually leads to a person designing or authorizing the rules governing the system that decides what does or doesn’t count as a target in the first place.

But there is still a human in the loop, despite them being further and further away from the corpse.

THE GENEVA CHECKBOX: WHAT DOES HUMAN CONTROL ACTUALLY MEAN?

This could happen without anyone ever formally deciding to surrender control to machines.

Imagine the progression.

In the first generation, an AI identifies a hostile vehicle and recommends a strike, and then a soldier approves it.

The next iteration, the system identifies ten targets simultaneously. The soldier reviews them quickly and approves the whole package.

Then fifty. A hundred. A thousand. Ten thousand. Keep going.

It ain’t long before an autonomous swarm detects an incoming attack and calculates that it has 0.7 seconds to respond before losing part of the network.

At which point the system politely asking Gary for permission begins to seem less like a critical ethical safeguard feature and more like an inconvenient software bug.

And this isn’t just some meaningless thought experiment…

A U.S. Army Military Review article published this summer describes a growing "velocity gap" between machine-speed sensing and human-paced targeting procedures. AI systems can collect, combine, and analyze battlefield information at mind-blowing speeds we can’t comprehend. But action still proceeds according to human validation and approval. For now.

Major Brodka argues that potential adversaries are actively developing systems and doctrine intended to exploit that gap.

That phrase, velocity gap, is doing an extraordinary amount of legwork there.

Because once speed becomes an advantage, and by all accounts it already is, every country eventually faces a conundrum over whether it wants autonomous killing or not.

Imagine two different militaries.

One requires individual human approval before lethal action.

The other permits a machine to respond automatically once predetermined conditions are satisfied.

The second military gets faster, more tactically efficient, and gains a huge upper hand.

The first responds by automating slightly more of its own process.

Then the second gets faster again. Ad infinitum.

This frantic spiral is basically a non-cooperative game: each side automates because the cost of being slower rises as the other side automates. Nobody has to want that equilibrium. Competitive pressure does the rest.

Modern military game theory defines this as an optimization trap requiring constant automation for survival.

In this strange series of events, nobody ever had to become a fanatical, overzealous advocate for robot murder. They only need to suspect that the other side has shaved another second or two off its murder decision loop. 

Eventually morality encounters latency.

WE HAVE SEEN THIS MOVIE BEFORE, WITH FEWER EXPLOSIONS

Financial markets offer a strange but ultimately useful preview.

Let’s take trading for example.

High-frequency trading did not emerge because society held a grand philosophical debate and decided financial markets would function better if humans were replaced by computers fighting over fractions of mere microseconds.

There was an obvious advantage to being faster than the rest. Makes total sense, right?

Sooner or later, somebody exploited it, which in turn forced someone else to respond.

It wasn’t long before trading firm behemoths were spending small fortunes locating and utilizing computers physically closer in proximity to financial exchanges because tiny differences in transmission time mattered. Every tick on the clock presented an opportunity to grab an edge.

Now this isn’t to suggest that human traders vanished from the system all of a sudden because intelligence started taking a backseat to speed.

They did, however, disappear after the realization arrived that intelligence taking several seconds had become far too slow to permit.

Now turn around and make the stakes literal life-or-death situations. Way more on the line.

Between those systems IDing an object, classifying it, predicting behavior, calculating a response, and ultimately executing… There is no technological reason why that sequence needs to slow down or get less efficient. The need for it to happen at a speed compatible with human reflection is no longer there.

Quite the opposite, in fact.

If autonomous weapons develop the way most technology develops, speed will become a feature. Not just any feature but the paramount feature.

Someone will always advertise lower latency. That’s how capitalism tends to work.

Some other defense company will demonstrate that its swarm reacts 40 percent faster than the competition.

Some overworked defense contractor will eventually have to create a PowerPoint slide titled TIME-TO-KILL OPTIMIZATION and everyone in the room will nod thoughtfully because that is their job.

Suddenly we’re talking about warfare leaving humans behind altogether.

This is what worries me significantly more than the Hollywood version where an artificial superintelligence suddenly develops consciousness and decides humanity has had a good run while it lasted.

The biggest fallacy is that machines need hatred or consciousness or a villainous backstory to present a clear danger to us squishy, fleshy humanoids.

They don't even need much autonomy at first.

They simply need to keep getting faster, and that’ll be enough to surpass us and cut us out of the sequence.

UNFORTUNATELY, THE ROBOTS HAVE A POINT

This would all be easier if humans were skilled at war.

Even after tens of thousands of years honing our craft, we simply aren’t.

Humans are prone to panic. We get tired. We misidentify things constantly. We seek revenge in intergenerational conflicts. We “just follow orders.”

We misunderstand good ones. We rotate through a symphony of emotions, getting scared and angry and confused, occasionally all at the same time.

As far as I know, a machine does not yet become enraged because its friend was killed yesterday. It does not really need to sleep.

It can even theoretically evaluate several sensors simultaneously and apply the same targeting rules every time.

Under sufficiently constrained conditions and with the right training, an autonomous system might become much better than a person at identifying a legitimate military target. It could already be the case.

This creates a problem I don't think the simple "always keep a human in the loop" argument handles particularly well.

Suppose an autonomous defensive system identifies incoming missiles correctly 99.9 percent of the time.

Suppose humans under the same conditions manage 97 percent.

Do we deliberately insert a worse, less efficient, less capable decision-maker because the act feels morally cleaner when a person clicks the confirm button?

Now make it more complex. More morally ambiguous.

Imagine the machine is demonstrably better than a human soldier at distinguishing military vehicles from civilian ones.

Would forcing a soldier to make the final decision actually protect civilians?

Or would we be sacrificing real actual human beings to preserve a philosophical idea about human agency?

I very clearly don't know where that line should be, and I am incredibly suspicious of anyone who claims they do.

The U.S. position has long emphasized that autonomy can, under some circumstances, improve precision and discrimination in the use of force, while Defense Department policy requires systems to allow commanders and operators to exercise appropriate levels of human judgment.

Neither this nor the opposite is an absurd position given what we currently know.

The UN Secretary-General and the president of the International Committee of the Red Cross warned on August 25 that humanity is approaching a "moral red line" in allowing machines to autonomously target human beings. They both called for legally binding rules and restrictions on these weapons systems.

Both sides are really arguing about the same uncomfortable question underneath it all…

If a machine becomes better at making certain lethal decisions than we are, what exactly are we protecting by insisting a human make them?

Human judgment? Dignity? Accountability? Like humanity is some upstanding arbiter of morality without blemish?

Or our own psychological need to believe that somewhere behind every act of organized violence there remains a person consciously choosing to do it?

Why are we compelled to think that way? I don't have a clean answer. And somehow that bothers me more than having an answer.

THE MOST IMPORTANT LATENCY IN HISTORY

There is, however, one fairly compelling historical argument for keeping the unreliable hairless ape somewhere in the kill system.

His name was Stanislav Petrov.

On September 26, 1983, Petrov was the Soviet lieutenant colonel on duty when an early-warning system informed him that the United States had launched a nuclear missile toward the Soviet Union.

Then more appeared. The system was reporting five incoming American missiles headed right for the motherland.

This was not a particularly chill moment in human history, despite occurring in the Cold War.

Relations between the Soviet Union and the United States were atrocious. Only a few weeks earlier, the Soviets had shot down Korean Air Lines Flight 007, killing 269 people. Both countries had enormous nuclear arsenals pointed at each other and elaborate systems designed to make sure that if one civilization ended, the other one would have a fairly bad afternoon too.

Petrov was ordered to report any warning up the chain of command.

His machine and information were all pointing to the conclusion that America had finally made the first move to launch a nuclear attack and eradicate its bitter foe.

Petrov didn't believe it. The whole thing felt off.

Five missiles made little sense as the opening salvo of an American first strike. Ground radar had not confirmed what the satellites were reporting. Petrov decided the computer was probably wrong and waited it out.

He was right, which is potentially why we’re still here today and not fighting for scraps in some dusty, irradiated, apocalyptic wasteland.

The Soviet satellite system had mistaken sunlight reflecting off high-altitude clouds for American missile launches.

The machine looked at the sky and concluded the sky was falling.

It took a human looking at the machine and thinking, essentially:

“Are you sure?”

That hesitation may have been one of the most valuable, if not the most valuable, pieces of human latency in history.

Petrov never particularly liked being described as the man who saved the world. He insisted that he had simply been doing his job.

But years later he said something that, in the context of autonomous weapons, feels considerably more important.

"They were lucky it was me on shift that night."

Which is a pretty haunting sentence the more you think about it, especially in the context of what we’ve already been discussing.

Because Petrov represented almost everything a highly optimized automated military system is designed to eliminate.

He introduced a delay.

He distrusted the data given to him.

He relied partly on intuition.

He deviated from what the system was practically imploring him to do.

He inserted messy human uncertainty and judgment into a process built to detect an attack and respond as quickly as possible.

From the perspective of pure system efficiency, Stanislav Petrov was latency.

Thank God for that.

THE DISTANCE BETWEEN US AND THE DEAD

There is another part of autonomous warfare that may matter even more than whether our machine friends make better targeting decisions.

War has always been politically expensive because humans fight it. Politicians expend tremendous amounts of goodwill and human capital to fight these needless wars, all financial costs aside.

Armies require people. Those people have families, communities, and voters attached to them. Governments asking citizens to fight are asking them to risk becoming maimed, traumatized, broken, or resigned to oblivion altogether. For God and Country!

That ends up creating some friction. Unfortunately, not enough friction, clearly.

Human history is littered with an impressive amount of war.

But friction nonetheless…

Autonomous warfare changes the political equation because it can increase the distance between a society and the violence carried out on its behalf. 

Sort of like how most people nowadays happily eat cheeseburgers and buffalo chicken wings but never need to butcher a cow or slaughter a chicken to enjoy those creature comforts.

A government that can project military force without exposing many or any of its own citizens to danger faces a different political calculation from one that must send thousands of young people overseas to die in strange lands they couldn’t find on a map.

The war in Iran right now is a timely example.

In fact, literally yesterday, Iran’s Revolutionary Guards said they captured an American autonomous underwater vehicle near the entrance to the Strait of Hormuz.

The Pentagon says the Anduril-built Dive-LD had malfunctioned while surveying regional waters. 

Palmer Luckey’s Anduril had an even more interesting description for it: an “attritable autonomous system.” Which is some wonderfully sterile defense-industry mumbo-jumbo jargon for a machine deliberately designed to go somewhere dangerous because losing the machine is an acceptable outcome.

And that is sort of the entire point I’m trying to make here.

You can send the robot into the Strait instead of the sailor. If somebody captures it, shoots it, sinks it, or it simply decides to stop working, nobody’s mother gets a knock on the door and a neatly folded flag.

This is obviously a good thing for the sailor.

But zoom out one level and the political arithmetic starts changing considerably.

The easier it becomes to project force while insulating your own population from the consequences, the easier it potentially becomes to keep projecting force.

We are already beginning to replace the human cost of escalation with hardware costs.

Hardware is considerably easier to reorder.

This is usually discussed as a benefit of drones and autonomous systems, and from the perspective of the soldier who does not have to die, it sure as hell is!

I would think that most people would rather attrit the robot than attrit somebody's son. The fact that the word "attrit" exists is another matter entirely.

The problem appears one step later.

What happens to the threshold for using violence when violence becomes much cheaper for the country deciding to use it? 

No more grieving families, fewer funerals on television, fewer homeless veterans with PTSD.

Less political pressure to explain why any of this is happening at all.

The violence remains extremely physical for the person standing on the other end of the drone.

It just becomes increasingly abstract for everyone else.

This may be the more important divide in the coming age of autonomous warfare.

Not human versus machine.

People who remain exposed to war versus people for whom war has become another remotely administered system.

On September 7, UN human-rights chief Volker Türk condemned Russia's reported use of fully autonomous drones in Ukraine and renewed calls to prohibit lethal autonomous weapons.

So some version of the future we are already fighting about in the halls of Geneva may already be bleeding into the present.

That should probably make the discussions feel slightly less academic and a bit more pressing.

HUMAN-IN-THE-LOOP

Going back to the beginning, the agreement in Geneva definitely matters.

International norms matter. Weapons treaties matter. Even the tedious process of 128 countries arguing until the middle of the night about individual phrases matters.

Pretending otherwise would be cheap and unnecessary cynicism.

What worries me is that the language we are using may conceal the actual nature of the problem.

"Human in the loop" sounds reassuring because the human is doing all the moral work in that there sentence.

Human means judgment and responsibility, and that we are still ultimately steering the ship.

But the loop itself is going to keep changing and getting faster, and adding more machines, and adding more targets, and incorporating more information that we could ever possibly hope to process or understand.

It will increasingly ask humans to supervise decisions rather than originate or inform them.

And perhaps most importantly, militaries have enormous incentives to make that loop move faster than the enemy's. Oh, and defense companies and their investors have the financial incentive to give those militaries what they want.

Humans are known to build technology to remove friction we are tired of facing.

Every so often, we discover some of the friction was unknowingly protecting us.

Hesitation or waiting for authorization before firing is inefficient.

Human uncertainty or asking another person to verify the target is not ideal.

Conscience might be the least scalable part of the entire system.

That does not mean we should go ahead and freeze military technology where it is, even if that were possible in our game-theory-informed system. Nor does it mean autonomous systems are intrinsically more or less ethical than humans. 

History has provided approximately zero evidence that humans possess some magical competence for applying violence responsibly. We’re awful at it.

What it does mean is that the problem is way harder than banning Terminator killbots.

The danger may not be that one morning a general decides machines should be given control over life and death. No, that would be too convenient, and there would be a clear decision to oppose. Or a villain to blame. The reality is that it’s way more banal and mundane than any James Cameron film.

We’ll simply keep on improving our targeting software, shortening response times, automating more steps in the kill chain, and making bigger, badder battle bots for our bottom lines and petty squabbles.

We will scale up and give each operator more systems to supervise because the software makes them more productive. Then more and more and more. Like what Claude Design did for marketing teams or Codex did for product sprints…

The reports will ultimately still contain a human name neatly printed at the top and signed at the bottom.

There will still be rules, commanders, operating parameters, and accountability frameworks we pretend to adhere to and care about.

And one distant day we may look closely at the process and discover something peculiar.

The person legally responsible for the killing will no longer really be deciding whether the target should die or not.

The machine has already identified him, evaluated him, classified him, selected its brutal instrument of death, calculated the odds and risk, and left nothing but a big, bright red button eager to be pushed by a permitting human hand.

The lawyers get a human. The generals get accountability. The diplomats get "meaningful human control." Gary gets a precious 0.7 seconds.

Everyone wins. Well, nearly everyone.

And maybe 99.9 percent of the time, the machine will even be right.

I, unsurprisingly, am writing about and haunted by what happens in the other 0.1 percent.

About a Soviet lieutenant colonel staring at a screen in 1983 while his computers confidently informed him that nuclear missiles were crossing the sky. About the twenty-three minutes of irritating, inefficient human hesitation that followed.

About whether the next Stanislav Petrov gets the same twenty-three minutes' grace period.

Or three seconds. Two. Or none.

Maybe we never remove the last human at all. Somewhere, a human will still be sitting there, technically in charge, with less time to think than Petrov had to blink.

We’ll keep him in the loop, but we won’t leave him enough time to be human.