International Committee for Robot Arms Control opening statement to the Convention on Conventional Weapons Meeting of Experts on lethal autonomous weapons systems, United Nations Geneva 13 April 2015

I am speaking on behalf of the International Committee for Robot Arms Control (or ICRAC as we are known), a founding NGO of the Campaign to Stop Killer Robots. We would very much like to thank Ambassador Biontino for preparations in chairing this second meeting of experts and for inviting our members to share their expertise. And we thank all of the States Parties for their participation.

ICRAC is an international association of scientists, technologists, lawyers, and policy experts committed to the peaceful use of robotics in the service of humanity and the regulation of robotic weapons.

ICRAC members have carried out research on various aspects of autonomous weapons systems and published their results in scientific journals as well as at conferences and in mass media.

We would like to stress that ICRAC experts are available and willing to provide technical expertise to the High Contracting Parties as they engage in discussions about autonomous weapons systems.

ICRAC urges the international community to seriously consider the prohibition of autonomous weapons systems in light of the pressing dangers they pose to global peace and security. We have produced a new leaflet on problems for global security posed by LAWS that is now available.

We fear that once they are developed, they will proliferate rapidly, and if deployed they may interact unpredictably and contribute to regional and global destabilization and arms races.

ICRAC urges nations to be guided by the principles of humanity in its deliberations and take into account considerations of human security, human rights, human dignity, humanitarian law and the public conscience. The Martens Clause reminds us that such moral principles are the basis for international law. Human judgment and meaningful human control over the use of violence must be made an explicit requirement in international policymaking on autonomous weapons.

ICRAC urges a broader discussion about the arming of autonomous systems beyond just lethal weapons, to include so-called “sub-lethal” or “less-than-lethal” weapons. These could still cause unnecessary suffering to humans. We urge nations to consider the human rights implications of the development and potential use of these weapons in any situation, including domestic policing, border control and internal law enforcement.

In conclusion, ICRAC encourages the CCW to move towards a preemptive ban on the development, production and use of autonomous weapons systems. ICRAC urges delegates to build consensus for negotiating a legally binding instrument to ban autonomous weapons systems.

Author information

Mark Gubrud

The short answer is no. But Russia is testing and may deploy at its ICBM bases a lethal mobile system which has “automatic and semi-automatic control modes.” Additionally, Deputy Prime Minister Dmitry Rogozin has recently called for “robotic systems that are fully integrated in the command and control system, capable not only to gather intelligence and to receive from the other components of the combat system, but also on their own strike.”

However, Russia’s statements at the UN have expressed concern about autonomous weapons as a threat to human life and international law, and Russia will be a participant in the Geneva CCW meeting. Moreover, a critical examination of claims that Russia is notably more aggressive in its early deployments of autonomous weapon systems than other nations, let alone that Russia is “leading a new robotic arms race,” shows these claims to be inflated and unwarranted. I have posted a detailed report at 1.0 Human.

Author information

Mark Gubrud

A common remark is that verification would require inspection of software as well as hardware, and that nations will never permit such intrusive inspections—as well as the fact that even clearly-written and well-documented software can be very difficult to read and interpret, let alone the case in which software may be deliberately obscured or encrypted. The physical form of systems that are capable of being autonomous may also not always be readily definable or discernible. Where both a cockpit and communications link to a remote operator are lacking, one may reasonably infer that a system is intended to operate autonomously, but their presence does not necessarily ensure that the system is incapable of operating autonomously.

The solution to this conundrum is already at hand, however, in the increasing emphasis in these discussions on the need for meaningful human control. This approach is increasingly recognized as a conceptual reframing of the problem of banning autonomous weapons, as already proposed in 2010 with the Berlin Statement (originally titled “The Principle of Human Control of Weapons and All Technology”). That statement asserts positively “That it is unacceptable for machines to control, determine, or decide upon the application of force or violence in conflict or war. In all cases where such a decision must be made, at least one human being must be held personally responsible and legally accountable for the decision and its foreseeable consequences.” The emphasis on personal responsibility and legal accountability for the decision to use violent force has become recognized as one of the elements of the concept of meaningful human control, which also emphasizes the role of adequate information and deliberation by the decision maker.

Thus, while it may indeed be impractical to verify compliance with a ban on “autonomous weapons” as such, it may very well be possible to verify compliance with a requirement for accountable and meaningful human control and decision in each use of violent force.

This is not to say that we should not also declaratively ban autonomous weapons–minus a list of exceptions for systems that operate autonomously but do not make significant lethal decisions autonomously, or that are purely defensive in nature and defend human life against immediate threats from incoming munitions, or are to be allowed for other, pragmatic reasons. Certainly, we should ban fully autonomous weapons. But the way to implement and verify such a ban may be better framed in terms of human control.

Two years ago, ICRAC members took part in an effort to consider measures for promoting compliance with an autonomous weapons ban. The result was a working paper, “Compliance Measures for an Autonomous Weapons Convention,” which is posted here. The work has not received wide recognition, but given that the question has begun to arise, it seems appropriate to highlight it now, rather than witness the emergence of  “a ban on killer robots would be nice, but it’s unverifiable” as a persistent canard.

The paper highlights many aspects of ensuring compliance with an autonomous weapons convention, including the enunciation of strong, simple, intuitive principles as the moral foundation for such an agreement, framing in terms of clear definitions, articulation of allowed exceptions, declaration of previously existing autonomous weapon systems, national implementing legislation, and the creation of an international treaty implementing organization (TIO). The role of the TIO in verification is detailed, particularly its support for cryptographic validation of records to tie them to particular weapon systems and the use of force at particular times (and potentially, places). These records, it is proposed, would be held by the compliant States Parties themselves and not released to the TIO or subject to any other possible compromise of military secrets, except in the case of an orderly inquiry into particular suspicious events, and possibly also some quota of routine, random inspections to verify continuous compliance. The cryptographic principles on which the tamper-proofing and time-stamping of such records can be carried out are simple and well-understood, and full encrypted records need not be exposed to the possibility of decryption if only “digital signatures” of the records are supplied to the TIO for archiving.

We believe that a scheme of this type can support rigorous verification of compliance in cases where it is suspected that a fully autonomous weapon system has been used, which should be a sufficient deterrent to their use. If coupled with other transparency and confidence-building measures, including routine on-site inspections of facilities in which remotely operated or nearly autonomous systems are developed, tested, manufactured, stockpiled, deployed or used, and with national means of intelligence which should suffice to reveal any prohibited activities large enough in scale and scope to pose a significant strategic security threat. These measures should suffice to ensure that no State Party will find the risks of non-compliance to be outweighed by uncertain and hypothetical military benefits.

– by Mark Gubrud (@mgubrud) and ICRAC’s Juergen Altmann

Author information

Mark Gubrud

Briefly, Foust argues that teleoperated drones today are too vulnerable to hacking through their communications links, and that the solution is to make them fully autonomous. And just as briefly, I show that this is wrong in a number of ways.

Author information

Mark Gubrud

Actually, Markoff does clearly say that the Pentagon is falling behind Google. In fact, the entire article seems to suggest that the military has neglected land robots, which is simply untrue, as I explain in my response, posted on my own blog.

I felt compelled to respond because, just days before Markoff’s article was posted (it was also included in the Sept. 24 New York print edition), I had published in the Bulletin of the Atomic Scientists an analysis of US policy for autonomous weapons which shows that it is actually an aggressive, “full speed ahead” policy. I don’t know if Markoff read my piece before he wrote his, but if you put them side by side and step back until you can only read the headlines, his looks like a rebuttal of mine, even though he is far too good a writer to say something that is actually wrong, and therefore he doesn’t actually contradict anything I wrote.

Author information

Mark Gubrud

Doomsday Clock

In November 2012, United States Deputy Defense Secretary Ashton Carter signed directive 3000.09, establishing policy for the “design, development, acquisition, testing, fielding, and … application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.”  Without fanfare, the world had its first openly declared national policy for killer robots.

The policy has been widely misperceived as one of caution. According to one account, the directive promises that a human will always decide when a robot kills another human. Others even read it as imposing a 10-year moratorium to allow for discussion of ethics and safeguards. However, as a Defense Department spokesman confirmed for me, the 10-year expiration date is routine for such directives, and the policy itself is “not a moratorium on anything.”

A careful reading of the directive finds that it lists some broad and imprecise criteria and requires senior officials to certify that these criteria have been met if systems are intended to target and kill people by machine decision alone. But it fully supports developing, testing, and using the technology, without delay. Far from applying the brakes, the policy in effect overrides longstanding resistance within the military, establishes a framework for managing legal, ethical, and technical concerns, and signals to developers and vendors that the Pentagon is serious about autonomous weapons.

Did soldiers ask for killer robots? In the years before this new policy was announced, spokesmen routinely denied that the US military would even consider lethal autonomy for machines.  Over the past year, speaking for themselves, someretired and even active duty officers have written passionately against both autonomous weapons and the overuse of remotely operated drones. In May 2013, the first nationwide poll ever taken on this topic found that Americans opposed to autonomous weapons outnumbered supporters by two to one. Strikingly, the closer people were to the military—family, former military, or active duty—the more likely they were to strongly oppose autonomous weapons and support efforts to ban them.

Since the 1990s, the military has exhibited what autonomy proponent Barry Watts has called “a cultural disinclination to turn attack decisions over to software algorithms.” Legacy weapons such as land and sea mines have been deemphasized and some futuristic programs canceled—or altered to provide greater capabilities for human control. Most notably, the Army’s Future Combat Systems program, which was to include a variety of networked drones and robots at an eventual cost estimated as high as $300 billion, was cancelled in 2009, with $16 billion already spent.

At the same time, calls for autonomous weapons have been rising both outside and from some inside  the military. In 2001, retired Army lieutenant colonel T. K. Adams argued that humans were becoming the most vulnerable, burdensome, and performance-limiting components of manned systems. Communications links for remote operation would be vulnerable to disruption, and full autonomy would be needed as a fallback. Furthermore, warfare would become too fast and too complex for humans to direct. Realistic or not, such thinking, together with budget pressures and the perception that robots are cheaper than people, has supported a steady growth of autonomy research and development in military and contractor-supported labs. In March 2012, the Naval Research Lab opened a new facility dedicated to development and testing of autonomous systems, complete with simulated rainforest, desert, littoral, and shipboard or urban combat environments. But the killer roboticists’ brainchildren have continued to face what a 2012 Defense Science Board report, commissioned by then-Undersecretary Carter, called “material obstacles within the Department that are inhibiting the broad acceptance of autonomy.”

The discrimination problem. Navy scientist John Canning recounts a 2003 meeting at which high-level lawyers from the Navy and Pentagon objected to autonomous weapons. They assumed that robots could not comply with international humanitarian law, core principles of which include a responsibility to distinguish civilians from combatants and to refrain from attacks that would cause excessive harm to civilians. These principles, and the military rules of engagement intended to implement them, assume a level of awareness, understanding, and judgment that computers simply don’t have. Weapons are also subject to mandated legal review, and indiscriminate weapons—that is, weapons that cannot be selectively directed to attack lawful targets and avoid civilians—are forbidden. The lawyers did not think they would ever be able to sign off on autonomous weapons.

Georgia Tech roboticist Ron Arkin has argued that unemotional robots, following rigid programs, could actually be more ethical than human soldiers. But his proposals fail to solve the hard problems of distinguishing civilians, understanding and predicting social and tactical situations, or judging the proportionality of force. Others argue, philosophically, that only humans can make such targeting judgments legitimately. In a world getting used to talking about virtual assistants and self-driving cars, it may not be obvious what the limits of artificial intelligence will be, or what people will accept, in 10, 20, or 40 years. But for now, and for the immediate future, the robot discrimination problem is hard to dispute.

To break the legal deadlock, Canning suggested that robots might normally be granted autonomy to attack materiel, including other robots, but not humans. Yet in many situations it might be impossible to avoid the risk—or the intent—of killing or injuring people. For such cases, Canning proposed what he called “dial-a-level” autonomy; that is, the robot might ordinarily be required to ask a human what to do, but in some circumstances it could be authorized to take action on its own.

In recent years, autonomy visionaries have stressed human-machine partnerships and flexibility to decide the level of autonomy a weapon may be allowed, based on tactical needs.  In a 2011 roadmap, for example, the Defense Department envisions unmanned systems that seamlessly operate with manned systems while gradually reducing the degree of human control and decision making required. A 2011 Navy presentationdepicts decisions about autonomy and control as a continuous tradeoff, explaining that while human control minimizes the risk of attacking unintended targets, machine autonomy maximizes the chance of defeating the intended ones. It seems likely that in desperate combat, autonomy would be dialed up to the highest level.

Appropriate levels of human judgment. In the spring of 2011, the Defense Department convened a group of uniformed and civilian personnel to begin developing a policy for autonomous weapons. The directive that emerged 18 months later lists a number of requirements for autonomous systems and draws a line at systems intended to autonomously target and engage humans–or to apply kinetic force (e.g., bullets and bombs) against any targets. But the directive neither states nor implies that this line should not be crossed.

Rather, the line may be crossed if two undersecretaries and the Chairman of the Joint Chiefs of Staff affirm that the listed requirements have been met. In the event of an urgent military need, any of the requirements can be waived—with the exception of a legal review. Furthermore, the line is not as clearly drawn as it may seem to be.

The requirements listed in the directive are not much more stringent than those that apply to any weapon system. Tactics, techniques, and procedures must be developed to specify how an autonomous weapon system should be used. Hardware and software must undergo rigorous verification and validation. Human-machine interfaces must be understandable to trained operators and must provide clear activation and deactivation procedures and have “safeties, anti-tamper mechanisms, and information assurance” that minimize the probability of unintended engagements.

These requirements sound reassuring; they promise to address many of the concerns people have about autonomous weapons. According to the directive, it is Defense Department policy that the measures listed will ensure that the systems will work in realistic environments against adaptive adversaries. But saying it doesn’t necessarily make it so.

In reality, neither mathematical analysis nor field testing can possibly locate every software bug or situation in which such complex systems may fail or behave inappropriately. Adversaries will strive to locate points of vulnerability, and it is terribly hard to anticipate everything that adversaries may do, let alone know how their actions may affect system performance. The notion of information assurance implies a promise to solve problems of software reliability and computer security that bedevil contemporary technology.

The centerpiece of the entire directive is this statement: “Autonomous and semi-autonomous weapon systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.” Although the phrase is never defined, it does not appear that appropriate levels always require at least one human being to make the decision to kill another. Rather, the appropriate level might well be the decision to dispatch a robot on a mission and let it select the targets to engage. In making such decisions, it appears that the burden of ensuring compliance with rules of engagement and laws of war falls on commanders and operators when the robots themselves are incapable of ensuring this. But in practice, it seems likely that unintended atrocities committed by autonomous weapons will be blamed on technical failures.

Semi-autonomy: Smudging the line. In theory, as long as three senior officials withhold their signatures, autonomous weapon systems that are intended to target humans or use kinetic or lethal force would be blocked. But the policy green lights—no extra signatures needed—semi-autonomous weapon systems that may apply any kind of force against any targets, including people. The crucial line that the policy draws between semi- and fully autonomous systems is fuzzy and broken. As technology advances, it is likely to be crossed as a matter of course.

The directive defines a semi-autonomous weapon system as one intended to engage only those targets that have been selected by a human operator. But the system itself is allowed to use autonomy to acquire, track and identify potential targets. It can cue the operator, prioritize targets, and decide when to fire. What the operator must do to select targets is left unspecified. Would a verbal OK, gesture, or even neurological interface be acceptable?

A system with such capabilities may not be intended to function without a human operator, but at most it would require a trivial modification to do so—perhaps a hack.At least three companies already market such systems. The policy clears them for immediate use after acquisition, via standard procedures.

The policy also addresses fire-and-forget or lock-on-after-launch homing munitions, which would include many systems in use today. Such munitions have seekers that autonomously find and home on targets. The directive classifies them as semi-autonomous weapon systems, on the theory that the operator selects targets by using tactics, techniques and procedures that “maximize the probability that the only targets within the seeker’s acquisition basket” will be the intended targets. Yet, upon launch, such munitions become, de facto, fully autonomous.

No restrictions are placed on the technology that a seeker may use to find a target and decide whether that is what it was looking for. This opens a clear path for weapons that can be sent on hunt-and-kill missions, limited only by the ability of their onboard sensors and computers to narrow their acquisition baskets to selected targets.

The way forward—to what? In the mid-2000s, Lockheed Martin was developing a small autonomous drone missile for the Air Force, and a similar system for the Army. Equipped with several types of onboard sensors, the missiles would fly out to designated areas and wander in search of generic targets, such as tanks, rocket launchers, radars, or personnel, which they would autonomously recognize and attack. Both programs were canceled, amid legal, ethical, and technical questions, to be superseded by systems that combine autonomous capabilities with radio links to human operators. Under the new policy, would such wide-area search munitions be classified as autonomous or semi-autonomous? Either way, the policy establishes that weapons like these may be developed, acquired, and used.

Given the long internal debate and general public opposition to killer robots, this is a highly aggressive policy. The US military never intended to replace foot soldiers with autonomous lethal robots during this decade, particularly not where civilians might be at risk. But funding the development and acquisition of systems that have autonomous targeting and fire-control capabilities—even if they are not intended for fully autonomous killing—will spur the weapons industry, in the United States and elsewhere, to accelerate exploration and investment in the technology of autonomous warfare.

The real issue is whether the world needs to go this way at all. The message of this policy is: full speed ahead.

Author information

Mark Gubrud

By now we’ve all become accustomed to the idea of somebody sitting in a trailer, gazing at a screen with a joystick in hand, and pushing a button to blow up a person, or usually more than one, somewhere on the other side of the world. Indeed, when we first heard of this, it was just another bit of disorienting news in the brave new post-9/11 world. George W. Bush was president then, and like it or not, he was going to do it. Then came Barack Obama, the avatar of liberal hopes, to double Pentagon procurement spending for drones in his first year of office, and execute 6 times as many CIA drone strikes in Pakistan as had Bush, with the approval of a majority of Democrats as well as Republicans. So it seems odd for Joshua Foust to be announcing now that he’s found “a liberal case for drones,” as if liberals (as a political demographic) weren’t already solidly on board with the president who seems to have all but defined himself as the man with the drone cojones.

However, the drones that Foust is out to build a case for are drones not as we know them. Autonomous drones, or autonomous weapons in general, are the new shock of the new, and not yet fastened in the hearts of liberals, conservatives, or even military professionals. In fact, not many people are yet aware that in November the United States became the first nation to have an openly declared policy for the development, acquisition and use of autonomous weapon systems (AWS), weapons which “once activated, can select and engage targets without further intervention by a human operator.”

Many of those who have heard of this are under the impression that the policy is a moratorium on systems that kill without a “human in the loop.” It’s not. According to Pentagon spokesman James Gregory, the policy “mandates a review by senior defense leaders [two under-secretaries plus the chairman of the Joint Chiefs] before entering formal development and again before fielding” of systemsintended to kill people by autonomous machine decision. “This review process,” says LTC Gregory, “doesn’t really impose a moratorium on anything.”

In fact, the new policy clears the way for aggressive development of a technology that the military has been reluctant to embrace, and has actually backed away from in recent years.

Older systems such as robotic antisubmarine mines have been phased out, and systems in development such as loitering autonomous hunter-killer missiles have been canceled or redirected to include communications links to human operators, while still retaining capabilities for autonomous operation. In 2005, the US Air Force ended work on LOCAAS, a small air-launched missile that was to hunt for tanks and missile launchers, using its own sensors, and attack them autonomously. It was superseded bySMACM, a larger missile with a two-way link to an operator. The Army canceled its autonomousLoitering Attack Missile, but has recently been experimenting with Switchblade, a small ground-launched missile that can loiter and has both remote-control and semi-autonomous modes.

At the same time, a growing contingent, both inside and outside the Pentagon, has declared the inevitability of AWS and called for “policies that introduce a higher degree of autonomy to reduce the manpower burden and reliance on full-time high-speed communications links while also reducing decision loop cycle time.” The November 2012 Directive, which “Establishes DoD policy and assigns responsibilities for the…. design, development, acquisition, testing, fielding, and employment of autonomous and semi-autonomous weapon systems,” appears to resolve this debate in favor of a commitment to autonomous technology and AWS development, integration of AWS into “operational mission planning” and use in war.

Foust only quotes from the Directive the one line which probably does best summarize its approach:

“Autonomous and semi-autonomous weapon systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.”

The problem with this is that nowhere does the document explain what the “appropriate levels” are, but a strongly implied assumption is that human judgment is not always required at the level of the decision to take a human life—that this is something which can be appropriately delegated to a machine.

While opening the door for delegation of lethal authority to electronic decision makers, the Directive places the burden of responsibility for their decisions on human commanders and operators. If the systems themselves are not able to ensure that they operate “in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable rules of engagement (ROE)” it is up to “Persons who authorize the use of, direct the use of, or operate” AWS to ensure this, despite not being able to know exactly what situations the systems will encounter, let alone how they will behave. The same can be said of a commander sending soldiers into a fight, but in most cases the soldiers themselves are supposed to be responsible for not shooting civilians.

The fact that soldiers often fail in this regard is held by Foust, and many other apologists for killer robots, to deflate the basic moral objection to machines that kill autonomously. After all, going by the record, how moral are human beings, especially in war? Perhaps emotionless robots can be programmed to be more scrupulously faithful to international humanitarian law (IHL) and military ROEs, going one better by exposing themselves to greater risk in, for example, checking whether a wounded opponent is still armed and dangerous, instead of just shooting him again to make sure.

While one may imagine such happy scenarios, in practice, for the immediate future, artificial intelligence (particularly in mobile robots) can’t come close to human performance in the cognition, reasoning and judgment required to distinguish combatants from noncombatants, to understand human behavior in combat situations, or to weigh military gains against harm or risk to civilians before deciding on the use of weapons. As long as this is so, giving machines increasing latitude to decide between one action and another, once the action starts, is only loosening the degree of control by responsible human judgment.

Will emotional humans, under stress and in danger, be as responsible in their use of AWS as the Directive demands, or will using AWS become a way to evade responsibility? Foust suggests that determining why an AWS made a bad decision could be “as simple as plugging in a black box,” and that programmers could be held responsible for war crimes caused by bugs in code. But in reality, what investigation would likely reveal is that the technology encountered a condition that had not been anticipated—the kind of thing that leads to millions of software failures every working day, most of which at least do not have fatal consequences.

Foust has it entirely backwards when he says that the Pentagon’s declaration about “appropriate levels of human judgment” implies that “the U.S. government isn’t looking to develop complex behaviors in drones.” What would we be talking about, if that were true? Increasing levels of autonomy implies increasing levels of complexity and capability of weapon systems to classify objects, interpret situations, predict the next move, and to decide how to act. Saying that it is the commanders’ and operators’ responsibility to understand the limitations of these systems, so that they can be used as they are, is clearing the way for further development of these systems without setting any ultimate limits.

This becomes clear in the Directive’s definition of “semi-autonomous weapon systems” (SAWS). These are fully green-lighted for development, acquisition and use without needing any special signatures or high-level oversight. If something is a semi-autonomous weapon system, it can be developed, funded, purchased, integrated into operational planning and used in combat today, or ASAP.

Two kinds of SAWS are defined. The first are systems with capabilities for

“…acquiring, tracking, and identifying potential targets; cueing potential targets to human operators; prioritizing selected targets; timing of when to fire; or providing terminal guidance to home in on selected targets, provided that human control is retained over the decision to select individual targets and specific target groups for engagement.”

Got that? For example, you might find an operator sitting at a console, and on the screen he sees some blurry figures through a FLIR camera. The system places a cursor over the figures and says “TARGET GROUP IDENTIFIED.” The operator says “Engage,” and the system does the rest. Obviously, it takes nothing more than a trivial software modification, or the throw of a switch, to make such a system fully autonomous. At least three companies already produce sentry systems which have capabilities of this sort. Nor is there any reason it could not be a mobile robot sent out to patrol an urban environment.

The second kind of SAWS, also green-lighted by the Directive, comprises “homing munitions” that are sent out to seek a target rather than being locked onto a particular target before launch. These SAWS clearly include missiles, cruise missiles and robotic torpedoes, and there is no clear exclusion of autonomous drones and ground robots (even if you think self-destruction would be required to meet the definition of a “munition”). Here, the operator is supposed to be responsible for following “tactics, techniques and procedures” which “maximize the probability that the only targets within the seeker’s acquisition basket” are the intended targets. However, most seekers have at least some ability to distinguish intended targets from “clutter,” and as this technology develops, the sophistication of target identification and discrimination is expected to increase. First we rely on the technology to seek out missile launchers within some limited search area, and then it gets good enough to distinguish missile launchers from gasoline trucks or the piping in a water treatment plant, so we can expand the search area. This process can be continued without limit, and eventually we are dispatching highly intelligent combat robots on search-and-destroy missions over a wide area as if they were human commandos.

In effect, the second definition of SAWS has erased any meaningful distinction between this kind of weapon and the fully autonomous ones that are supposed to require senior review (the so-called moratorium), since as soon as the missile is launched it becomes fully autonomous. There is thus no red line left to cross between the weapons that are fully approved under the new US policy, set by this administration, and, to put it baldly, the Terminator—or anything in between. It becomes a matter of technology and time.

It is curious that so many who write on both sides of this question jeer at science fiction, as if its warnings could be dismissed just because they were delivered through expressionistic art and scenarios that don’t pass analytical muster. How can we talk about the future, and about how technology may reshape or destabilize our world, without being subject to this particular form of ad hominem ridicule? Perhaps it may help to draw parallels with the indisputable past.

Obama’s assumption and expansion of the presidential prerogative to drone is eerily reminiscent of Harry Truman’s assumption of the authority to use nuclear weapons and to largely set policy for their further development, production and deployment. Liberals mostly went along with that, too, at least until realizing that the power to initiate nuclear war was the power to end human history. By the time popular (and mostly liberal) dissent against an unbounded nuclear arms race rose to the level of a potent political force, it was almost too late. We were very lucky—perhaps owing largely to themoderating role of human judgment in crises and accidents at the brink of apocalypse—to survive the Cold War, as well as two decades (and counting) of the new world disorder.

Killer robots are not the only element of the global technological arms race, but they are currently the most salient, rapidly-advancing and fateful. If we continue to allow global security policies to be driven by advancing technology, then the arms race will continue, and it may even reheat to Cold War levels, with multiple players this time. Robotic armed forces controlled by AI systems too complex for anyone to understand will be set in confrontation with each other, and sooner or later, our luck will run out.

We can stop this. The line to draw is clear: No autonomous initiation of violence, no machines deciding on their own to kill human beings or to start or escalate conflicts in which people will be killed. It is a bright red line that everyone can understand, and it defines a strong moral principle that we instinctively know is right. To enshrine this principle in a global ban on autonomous weapons is a necessary step in our unending effort to secure the human future.

Author information

Mark Gubrud

The DoD Directive appears at first glance a stony rejection of ICRAC and HRW’s call for a broad AWS ban, setting a presumption that the US will proceed to develop, deploy and use AWS, under certain doctrines and guidelines: “The Commanders of the Combatant Commands shall…Use autonomous and semi-autonomous weapon systems in accordance with this Directive….”

On closer examination, a more complicated picture emerges. The policy defines an AWS as “A weapon system that, once activated, can select and engage targets without further intervention by a human operator.” The ability of the system to select and engage targets autonomously makes it an AWS even if it is human-supervised with a possible human override; thus “human on the loop” is assigned the same status as “human out of the loop,” a conservative (good) policy.

The policy distinguishes AWS from “semi-autonomous weapon system” (SAWS) which it defines as “A weapon system that, once activated, is intended to only engage individual targets or specific target groups that have been selected by a human operator.” The category includes systems that automatically acquire, track, identify and prioritize potential targets or cue humans to their presence, “provided that human control is retained over the decision to select individual targets and specific target groups for engagement.” It also includes weapons with terminal homing guidance, and “fire and forget” weapons where the target has been human-selected.

The policy basically green-lights the development and use of both lethal and nonlethal SAWS for all targets.

Fully autonomous kinetic weapons, however, are only pre-authorized “for local defense” of manned installations and platforms, presumably referring to missile and projectile interception systems. And they have to be human-supervised.

Unsupervised AWS are only authorized for “non-lethal, non-kinetic force, such as some forms of electronic attack, against materiel targets….”

This policy is basically, “Let the machines target other machines; Let men target men.” Since the most compelling arms-race pressures will arise from machine-vs.-machine confrontation, this solution is a thin blanket, but it suggests some level of sensitivity to the issue of robots targeting humans without being able to exercise “human judgment” — a phrase that appears repeatedly in the DoD Directive.

This approach seems calculated to preempt the main thrust of HRW’s report, that robots cannot satisfy the principles of distinction and proportionality as required by international humanitarian law, therefore AWS should never be allowed.

The policy directs that AWS and SAWS “shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.”

Assuming that directive has been followed, responsibility for IHL compliance will fall on those commanders and operators: “Persons who authorize the use of, direct the use of, or operate autonomous and semi-autonomous weapon systems must do so with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable rules of engagement (ROE).”

The policy thus entrenches AWS policy in a strong defensive position against assault on the basis of IHL and the limits of computation.

However, development and fielding of fully autonomous lethal weapons which do engage human targets is not ruled out. It must be approved by three Undersecretaries of Defense and the Chairman of the Joint Chiefs, but a separate set of guidelines is provided for such systems, suggesting that such approval would not be extraordinary.

For the time being, however, DoD can deny that such programs have been approved and point to this policy to deflect questions about killer robots targeting people, and whether that comports with international law.

Author information

Mark Gubrud

Two years ago, in presenting the first version of what became ICRAC’s Berlin Statement , I emphasized that the dictum “machines shall not decide to kill” could serve as the “kernel” of a convention on robotic weapons that would naturally encompass ICRAC’s broader goals for robot arms control.  The idea of machines taking the decision to kill people, or initiating violence that causes death and suffering[1], was abhorrent to almost everyone who considered it. This almost universal repugnance could be harnessed as the “engine” of a global movement to stop killer robots, over the opposition of a minority who would argue the inevitability and military necessity of robotic and autonomous weapons.

I called this the “Principle of Human Control,” and felt it important that this should be declared as a new principle, consistent with just war theory, the laws of war, and human rights law, but not explicitly contained in nor necessarily derived or derivable from existing bodies of philosophy and law.  A new principle was needed for the simple reason that the threat against which the principle was raised had not previously existed, and was only becoming imaginable as the march of technology brought us closer to the day when machines might plausibly be deemed capable, and trustworthy, to make lethal decisions autonomously.

Accordingly, whereas ICRAC’s founding Mission Statement would only “propose… that this discussion should consider” that “machines should not be allowed to make the decision to kill people”; the Berlin Statement declared that “We believe… it is unacceptable for machines to control, determine, or decide upon the application of force or violence in conflict or war. In all cases where such a decision must be made, at least one human being must be held personally responsible and legally accountable for the decision….” [Emphasis added.] No claim was made that our belief could be proven true according to any body of law, philosophy or science. We declared a new principle.

Distinction and Proportionality

Nevertheless, as the debate about autonomous weapons takes shape, many arguments revolve around international humanitarian law (IHL), or the law of armed conflict (LOAC), and the principles of this body of law, as well as the deeper philosophical principles of just war theory.  Some authors argue that these venerated principles have stood the test of time, and have proven adaptable as weapons and modes of warfare have evolved. Others claim that 21^st century weapons and irregular warfare render the existing instruments of LOAC quaint and ripe for revision. Some argue that autonomous weapons cannot fulfill IHL requirements to distinguish between combatants and non-combatants (Principle of Distinction) and to weigh military necessity and objectives against the risk or expectation of collateral harm to non-combatants (Principle of Proportionality). Others maintain that even limited discrimination capabilities might help to reduce harm to non-combatants, and that human commanders, who will decide when and what kinds of AW to use, will remain responsible for the judgment of proportionality.

One argument that AWS are inconsistent with IHL/LOAC rests on the proposition that machines are simply incapable of the judgment required to ensure their compliance.  For example, Protocol I of the Geneva Conventions requires that “Parties to the conflict shall at all times distinguish between the civilian population and combatants and between civilian objects and military objectives and accordingly shall direct their operations only against military objectives.” This duty to “at all times distinguish” embodies the Principle of Distinction.  It is implicitly required of whatever agent “shall direct… operations” so that the operations are directed “only against military objectives.” Therefore, if the agent directing “operations” were to be a machine, it would have to be a machine able to “at all times distinguish” between civilians, combatants, civilian objects and military objectives.

Distinction is clearly a challenge for machines, but human capabilities to “at all times distinguish” also have limits. For example, one may not be able to see clearly in the presence of smoke or other obstructions, or to judge quickly and correctly whether a figure in the shadows is a civilian or combatant, particularly when under fire. If human responsibility for adherence to the Principle of Distinction is limited by human capabilities, in various circumstances, would such limits not apply to machines as well? If we don’t expect perfection in all situations, might there not be circumstances in which a machine’s ability to discriminate would be comparable to, or perhaps better than, a human’s?

Questions of interpretation also challenge the argument. Who bears the responsibility of the “Parties to the conflict” to “at all times distinguish” and to “direct their operations only against military objectives”? “Parties” would normally be interpreted to mean the States involved, a fairly high level of direction.  May such “Parties” not lawfully direct that autonomous weapons be used only against combatants and military objectives in some conflict, subject to technical limitations which may result in unintended harm to noncombatants or civilian objects, which the “Parties… at all times distinguish”?  Alternatively, if responsibility is delegated to a human commander, who directs that an autonomous weapon be used in some tactical situation, and if that person meanwhile upholds the obligation to “at all times distinguish,” can it not be argued that the commander may lawfully direct the operation only against military objectives, subject to technical limitations which might produce an unintended result?

Certainly, the technical limitations of AWS matter, in relation to the circumstances in which they are used. But a reasonable interpretation of the Principle of Distinction cannot demand absolute perfection of machines (since humans are incapable of it) nor forbid a responsible human commander from taking some risk of a mistake. The actual capabilities of machines, and the acceptable level of risk, will become the issues of contention. It is not obvious that the resolution will be to ban autonomous weapons categorically.

Protocol I also requires (in several statements) that “those who plan or decide upon an attack shall… refrain from deciding to launch any attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated.” This is known as the Principle of Proportionality, and it poses an even more formidable challenge to technical capabilities, if machines would bear the burden of judging not only what effects upon civilians and civilian objects “may be expected,” but whether those “would be excessive in relation to the… military advantage anticipated.”

Here again, questions of interpretation, particularly in view of human fallibility, challenge the argument. In practice, human commanders have little objective basis on which to judge what is “excessive” other than, perhaps, some set of examples and judicial precedents – which might very well be coded as a database suitable for machine access. In practice, human judgment of what is “excessive” is externally regulated only by the possibility of later review and adverse judgment. This (in theory) motivates commanders to think carefully about whether their decisions will be perceived as reasonable. Could the same mode of regulation not apply equally to the decisions of machines? Surely a machine could be programmed to seek action plans that it assesses are consistent with the rules and precedents coded in its database. If humans judge that machine decisions are reasonable and proportional, at least as consistently as human decisions, might that not satisfy the Principle?

Alternatively, if an autonomous weapon system is programmed to fire under some conditions, and not under others, subject to known limitations of its capabilities to autonomously distinguish conditions, could a human commander who is aware of those limitations not accept responsibility for his or her own judgment of proportionality, taking some calculated risk that the machine might misclassify the actual situation, just as it might be subject to any other error or malfunction? Is this fundamentally different from the situation in which a commander authorizes an attack by human combatants, knowing there is some risk of their autonomously taking some action which would later be judged excessive, due either to their own misperception of the situation or their own misjudgment of proportionality?

Frames of reference and hard cases

While just war theory separates jus in bello (just conduct in war) from jus ad bellum (justice in going to war), in practice these principles are often entangled. Judgment that some amount of risk or harm to noncombatants is not “excessive in relation to” military objectives likely depends on perceptions or assumptions about the necessity and legitimacy of the conflict itself. Conversely, perceived excesses of violence can help to delegitimize a conflict. This is true whether action is taken on human decision or on the decision of a machine.

Whether the harm caused by a particular “attack” (or weapon, or type of attack) is judged “excessive” will depend on what it is compared with, which may again depend on whether the perceived alternative would also be perceived as plausibly not excessive, given military necessity. High-altitude saturation bombing was largely judged not excessive in World War II, and was far more controversial in Vietnam, but may still set a standard for comparison with the US drone strike campaigns in Pakistan and Yemen, as seen by those who believe the campaigns to be necessary and just. Those less convinced of jus ad bellum in this instance may be more inclined to compare the drone strikes with alternatives such as special operations forces raids on selected high-value targets, or perhaps no military action at all.

If autonomous weapons have only limited capabilities for discrimination and proportionality, they may still be claimed as consistent with jus in bello if they are perceived as alternative to weapons that are completely indiscriminate and almost certain to cost more innocent lives. Compared with a 500-lb bomb dropped on the roof of a building where noncombatants might be found, an autonomous robot that can recognize persons bearing arms, or even (perhaps an easier technical problem) a specific, known individual, could reasonably be expected to reduce the level of harm or risk to noncombatants.  In this scenario, the robot might not have perfect abilities to implement the Principles of Distinction and Proportionality, but a tactical decision to use the robot, as alternative to a blunter weapon, might arguably uphold those principles if they are viewed as practical goals.

A different judgment might be reached if autonomous weapons were compared with armed teleoperated robots, which keep a human “in the loop”, and which might provide another means of assault on a location in which noncombatants may be present.  However, teleoperation might not always be reliable or practical, due to the vagaries and vulnerabilities of communication links, or the need for small size or stealth of the robot. For very small robots, resolution and bandwidth limits in teloperation may mean that fully autonomous systems which seek a specific individual or object might compare favorably in terms of discrimination, and hence proportionality. Another principle is needed if we are to categorically rule out the use of autonomous weapons when the possibility of teleoperation is not available.

Similar arguments can be made in defense of missiles equipped with target recognition and terminal homing capabilities, which arguably can be called robotic weapons, autonomous in the sense that after release they are charged to make at least targeting refinement decisions autonomously. Such weapons already exist and have been deployed, albeit with only rudimentary target recognition capabilities. What would be the objection to improving those capabilities to achieve more precise targeting and a lower risk of collateral harm?  It might be argued that such weapons are sent on a one-way mission and do not make an independent “kill decision,” but what would be the objection if the systems were further developed so that, upon failure to recognize an appropriate target, or upon detection of humans in the vicinity of the target (with or without distinction), the weapon decides to abort its mission? It could then safely self-destruct or divert to an open area while disarming its warhead. Even an imperfect “abort decision” capability would appear to be an improvement from an IHL standpoint, if compared with the use of a missile with no such capability. Yet, the machine would in effect be deciding whether to kill.

Note that teleoperation and even “on-the-loop” human intervention might not be possible in the split second prior to impact during which a missile’s target analysis system might have sufficient information to make a decision. Another case in which events occur too rapidly for meaningful human control or even supervision of target identification and fire decision is that of point defense systems such as the Phalanx guns deployed on ships, and other anti-missile,-mortar and -shell systems employing ballistic or guided interceptor munitions or lasers. Even longer-range missile and air defense systems such as the Patriot and Aegis challenge human capabilities to make crucial target discrimination decisions in seconds. In practice, humans have often failed to exert an “abort” command when operating in an “on-the-loop” role, with deadly results in a number of incidents.

I think that the case for automated fire decision in a point defense system is so compelling that any credible proposal for an autonomous weapons convention will have to carve out an exception for such systems, at least when they are directly defending human lives against immediate threats. Strict and continuous human supervision should of course be required, including accountability for failure to intervene when information indicating a system error is available to those charged to be “on the loop.” Such systems should also be operated in a “normally-off” mode, and only activated upon warning of an incoming attack. But as long as a system is limited to defense against weapons incoming to a given human-inhabited location (including the instantaneous location of an inhabited vehicle), automated fire decisions will likely have to be allowed.

A similar exception was carved out from the Convention on Cluster Munitions in the case of the so-called sensor-fuzed weapon (SFW), or any weapon meeting the CCM’s criteria of having fewer than 10 submunitions, each of which weighs more than four kilograms, is “designed to detect and engage a single target object”, and is equipped with self-deactivating and self-destruct mechanisms. Here again we see the implications of proportionality and distinction as guiding principles for robot arms control. The rationale for excepting SFW-type weapons from the CCM was that such weapons, with their more sophisticated capabilities for target identification, discrimination, self-guidance, selective engagement, and self-deactivation, do not pose the same risk of harm to civilians posed by traditional cluster munitions with their many small and highly unreliable bomblets. Following this logic, further development of the SFW, to incorporate even more sophisticated discrimination and fire/no fire decision making capabilities, would be hard to object to. Yet it is not clear why the SFW would not meet a reasonable, broad definition of “autonomous weapons.”

It is certainly true that early AWS can have only limited capabilities for discrimination, and even less to judge proportionality. Exaggerated perceptions of their precision and selectivity may lead to excesses in their use, as may well be occurring with drones already. Yet this is not, I think, the central concern that is driving either the nascent campaign to ban AWS, nor the broader public’s unease with the rise of killer robots. If it were, it would suggest the need not for a ban but for regulation of these weapons and their use, and for a go-slow approach to their deployment – until the technology can be perfected, or is good enough to be acceptable in well-understood circumstances.

So then, What’s so bad about killer robots?

We were a family. How’d it break up and come apart, so that now we’re turned against each other? … This great evil. Where does it come from? How’d it steal into the world? What seed, what root did it grow from? Who’s doin’ this? Who’s killin’ us? Robbing us of life and light. Mockin’ us with the sight of what we might’ve known.

– Private Witt, in Terrence Malick’s screenplay for The Thin Red Line

The cases just considered may be seen as borderline cases for the prohibition of AWS. Borderline cases always exist; we might say that in the vicinity of every bright red line there is a broad grey zone. Those opposed to drawing a line are fond of citing ambiguous cases. It is true that where you place the line in relation to such cases may be somewhat arbitrary. What is important is to draw the line somewhere. If we stand back from the grey and fuzzy border zones to see the big picture, we can see clearly the difference between violent force in human conflict today, and some future in which decisions about the use of violent force are routinely made by machines.

In such a future, we risk the unleashing of conflict itself as an autonomous force, embodied in technology, and divorced from the body of humanity, within which it first arose.

What is conflict? One perspective on conflict is that it arises because each of us has a unique point of view. We also join in community, but humanity as a whole is spread across the globe. The human community is divided and in conflict because of differing points of view. This is not different from saying we have differing and conflicting interests, e.g. in controlling the same territory or limited resources. However, the willingness of human individuals to sacrifice themselves for families, platoon brothers, tribes, nations… or for a cause, shows that self-interest is only one factor in our point of view about what is good and just, and worthy of fighting for, wrong and unjust, and worthy of anger and violence.

In another perspective, conflict is a process which arises within and between us, and which can consume us and escape our control. Because humans have the capacity for anger and violence, because violence easily becomes lethal, and because life and death transcend in importance, we easily become caught up in emotions that overpower reason. Community fractures, separating us from them, and we are unable to forgive the terrible things that they have done, unable to consider their claims to justice and agree on a compromise with our own. Across the fault lines of love and reason, we and they speak to each other in the language of violence, wear our masks and play our roles in the Greek tragedy of conflict and war.

Yet until now it has always been true that conflict has consisted solely of willful human deeds. When a weapon is fired, one person deliberately unleashes violent, potentially lethal force upon another. It may be irrational, but it is intentional, and essential. We say that weapons are fired “in anger,”  an animal passion that is rooted in mortality and the struggle for survival. I think this is what the warriors mean when they say that war is deeply human (and somehow, in spite of robot weapons, always will be).

Anger humanizes violence, and its apparent absence is part of what makes remote control killing so deeply disturbing. Yet even in the cool detachment of the drone operator’s padded chair, we find one human being accepting the responsibility for the act of killing another, because the human community is divided and the community to which the “cubicle warrior” is loyal has gravely decided that this killing is a necessary burden of evil. That burden is felt strongly by military veterans and professionals, who correspondingly also feel, surprisingly often, that there is something deeply wrong – and terrifying – about the idea of machines that would usurp from us, or to which we would surrender, the heaviest responsibility ever assumed by human beings: that of deciding when, and under what circumstances, we are justified in injuring or killing others.

If the community is democratic, if it is even truly human, the burden is felt. When the enemy hits back, the pain and loss are felt, too. There is always the possibility of saying “Enough.” As long as conflict remains human conflict, it ends when people finally, for whatever reasons, decide to stop fighting.

In making the process of killing fully autonomous, we risk machines no longer under human control pursuing conflict for its own sake, conflict that is no longer human conflict, no longer about right and wrong. We risk machines mercilessly extinguishing human lives according to programs developed to embody only military doctrines and goals, and the laws and logic of states. We risk the dulling or loss of our ability and responsibility to judge when the price or the risk is too great. Or to know when too much blood has been spilled, either because it is our own blood or because in spilling the blood of others we lost our claim of justice. We risk becoming either tyrants who rule through robot soldiers, or peasants who submit to a robotic regime, or perhaps both at once (already the drones are coming home to roost).

Do I mean to invoke here the specter of Skynet, the artificial intelligence that declared war on humanity in The Terminator, reputedly because it feared we’d turn it off? Or Colossus, the military supercomputer that took control of the world in a brutal coup in order to fulfill its mission of ensuring peace? The scorn that “serious” people direct against these tropes from science fiction betrays their own nervousness.  Artists have mined our apprehensions about the world we’re creating, and projected them before us in gaudy masks and cartoonish story lines that beg to be decoded. The military system is already a kind of machine, pursuing its own agenda, just as states, corporations and institutions of all kinds are. These machines are made of people, and their minds are the minds of people – increasingly augmented by information technology, from clay tablets to search engines. We like to think that this augmentation increases our effective intelligence, but as soon as words are written down, thinking rigidifies. Yet one essential fact remains: each human mind, dazzled and lost in the maze of knowledge, of law and the machinery of institutions, remains tethered intimately and existentially to a human heart. It is that tether which the military complex, enabled by technology, now threatens to break.

No claim is made here of the infallibility or even wisdom of human decision making in conflict. On the contrary, we are all familiar with history’s march of folly, hubris, aggression and anger, the tragic farce of bluster, miscalculation and misunderstanding , the tragedy of right pitted against right, outrage responding to outrage and leading to further outrage, escalation and the madness of war. In full view of this, we might wish for an all-wise and all-powerful super-AI to impose a global pax robotica. But apart from the question of how we would ensure the benevolence of an electronic emperor, there is simply no reason to think our present drift into robotic warfare will lead to peace.

The robotic weapons being created today are just that, weapons. They are fitted into increasingly automated, integrated, networked systems which gather and process “intelligence” to produce action orders following plans and doctrines issued from on high. The tactical officer increasingly consults a computer to learn the next objective, estimate weapons effects and perhaps assess the risk of killing civilians; after the action the officer reports to the system, which updates its model of the conflict.

As military systems are increasingly automated, and the human role is progressively atomized, mechanized, and displaced, these systems remain pitted against each other, embodying the same contradictions of reason and purpose. As machines take over from brains, they will sideline the hearts that whispered: Life is precious. Artificial intelligence will know everything about correlations of force, kill probabilities, stealth and counter-stealth, and perhaps also everything about the economic value of resources, the intricacies of laws and treaties, protocols and codes of conduct, the theory of games and the flow of information through networks. Yet it will understand nothing about the purpose of any of this, nothing about simply being human.

Probably the most certain reason why the AI warlords of the future won’t understand what we were trying to protect when we created them is because we won’t tell them. We’ll scrupulously avoid corrupting their military discipline with any hint that, in some cases, we might and should and probably would back down. That we might learn not only that we lack the physical might to impose our will on others, but that we were wrong to even try, or to want to. Would we even know how to tell our machines when and why they should stop fighting, propose or accept a cease-fire, or even withdraw in defeat, rather than lose more, and risk everything? Do we even understand how to tell this to ourselves?

On the contrary, violence and especially war always represent the failure of reason, the tearing and breakdown of community, and negotiating directly with the primal chaos that civilization sought to expel. Great job for a robot!

If ceding control to an automated war machine seems far removed from military robots performing the dull, dirty and dangerous jobs of today, and even from the possible next step of automatic target selection and autonomous fire decision, it nevertheless represents the logical destination of a program to outsource human responsibility for making decisions about the use of violent force. I believe it epitomizes the concern that underlies the widespread and almost universal view that moving from today’s non-weapons military robots and teleoperated lethal drones to fully autonomous weapons is a step that we should never take – or, if ever, only reluctantly, with great care, and only because it might be necessary or unavoidable. We feel instinctively that the killer robot is no longer a human tool, but has become an enemy of humanity; as depicted in Terminator, a gleaming metal skeleton whose eyes glow with the fire of Hell. A whiff of death lurks in every weapon, but the killer robot embodies death as an animated Other, Death that walks, Death that pursues each of us with a determination of its own.

The reason for banning autonomous weapons is to draw a bright red line across a broad grey zone that lies between us and an out-of-control future. Surely the place to draw this line is at the point where a machine is empowered to decide, by a mechanical process not controlled or even fully understood by any of us, the use of force against a human being. Because violence commits us, we must generalize this to a ban on any autonomous initiation of violence, lethal or nonlethal, against humans or against property, including other machines. Finer details do matter, but what is essential is to draw the line.

If we allow ourselves to cross this line, we will find ourselves driven onward by the imperatives of an arms race. This is another type of conflict process which we struggle to control, and crossing the line already means losing that struggle. If we tell ourselves that we will limit lethal autonomy with reasonable demands for Distinction and Proportionality, we will find that the weapons demand to be freed, in order to confront, and if need be to fight, others of their own kind.

We will increasingly risk the ignition of violence by the unanticipated interactions of proliferating systems pitted in confrontation with one another. The ever-increasing complexity of such confrontations will far surpass the Cold War’s already dangerous and unmanageable correlations of forces. Ensuring stability would be a difficult engineering challenge, but the creators of these systems will be teams in different nations working against each other. The further we go, the more control we cede to machines, the harder it will be to turn back, the more timid we will be to even consider dismantling or tampering with the systems that guard us against the other systems.

We risk being seduced, also, by the promise of imperial war without risk to ourselves, punishing or subjugating others with impunity as long as they do not have the same technical means to use against us. If this could be successful, it would be monstrous enough, but history shows that people do find ways of striking back, and the technologies that would enable AWS are widely spread around the globe. History also shows that imperialism leads to the clash of empires. The norms that we set in our dealings with the weak are the norms we will have to live with when we deal with the strong. We can’t expect to enjoy the comforts of home as our robots fight an endless stream of “asymmetrical” conflicts, when that is exactly the sort of behavior which will conjure up a “peer competitor” to say “You do it, why can’t we?”—and draw us into serious confrontation.

The Principle of Humanity in Conflict

We say “No to killer robots” because they pose a threat to humanity. The threat is new and unanticipated in prior law or philosophy.  We cannot derive our opposition from the principles of just war theory or the codes of international humanitarian law.  We need to declare new principles.

Here is an attempt to formulate a set of principles which address the threat from autonomous weapons. I do not claim that this is the final formulation. I do not claim that the wording here is perfect. I do not claim to have distilled a mathematically minimal set of independent principles. Rather, these principles are interlocking, partially redundant, mutually reflecting, mutually referential, and mutually reinforcing. Together, they can be referred to as the Principle of Humanity in Conflict, unless you have a better name.

In the literature on IHL/LOAC, the principle that in the conduct of war we must not be needlessly cruel, inflict unnecessary suffering, or make use of weapons that are inherently inhumane, is sometimes called the Principle of Humanity. This set of principles can be seen as an expanded Principle of Humanity.

Human Control: Any use of violent force, whether lethal or sub-lethal, against the body of a human being, or to oppose the will of a human being, must be fully under human control. If violence is initiated, it must be the decision of a human being.

Hence it is unacceptable for machines to control, determine (e.g. by the narrowing of options), or decide upon the application of violent force. This applies whether the target is a human or another machine; we can’t just turn robots loose to fight other robots as a proxy for our conflicts.

Human Responsibility: We must hold ourselves responsible for the decision to use violent force, and cannot delegate that responsibility to machines. At least one person (a human person), and preferably exactly one (e.g. a commanding officer), must be accountable for each decision to use violent force against a particular person or object at a particular time and place.

Where data has been recorded pertaining to the circumstances of such decisions, it must be retained and made available for judicial review.

A human commander cannot accept responsibility, a priori, for the individual violent actions of autonomous weapons which will occur in circumstances which cannot be fully anticipated; this would be a mere pretense of responsibility, to hide the fact of irresponsibility. To authorize the use of autonomous weapons only within certain boundaries  of space and time, or within a certain conflict, would not create an exception to this principle.

Human Dignity: It is a human right not to be killed on the decision of machines, nor to be subjected to violent force or pain on the decision of machines, nor to be threatened with violent force, pain or death as a form of coercion on the decision of machines, nor to be ruled in the conduct of life through the agency of overseeing machines that may decide the use of force as coercion.

Human Sovereignty: Humanity is, and must remain, sovereign. Threats to human sovereignty, and security, include the process of conflict that arises between us, and with which we struggle to retain or regain control. Externalizing that process in an autonomous technology would make it ever more difficult to control, until finally we would have lost even the ability to exert control.

The Principle of Humanity in Conflict: Here we mean, most fundamentally, the principle that conflict is, and must remain, human. It is between us. When in conflict with one another, we must not lose sight of our humanity (as is the case when we are needlessly cruel or inhumane).

We must always try to resolve conflicts nonviolently. Violent force, when unavoidable, must be used only in full respect for the humanity of opponents and recognition of the gravity of the act of killing (for we are all mortal). Taking arms against an opponent always entails the possibility of killing, and of being killed (for none of us is omnipotent).

Use of violent force can be accepted only in conflict between human beings, and only then because, and only when, we have failed to resolve the conflict nonviolently. Allowing machines to assume control of the conduct of conflict, to use violence autonomously, pitilessly and in contempt of the humanity that we share with our opponents, would be inhumane, and inhuman.

Author information

Mark Gubrud

According to the report, “The U.S. has carried out at least 280 attacks since 2004 in Pakistan’s tribal region.”

What is curious about the AP report is its spin.  Sebastian Abbot’s story starts with the assertion that “American drone strikes inside Pakistan are killing far fewer civilians than many in the country are led to believe,” based on heated statements by Pakistani islamists that drone strikes “are killing nearly 100 percent innocent people” and so on.

In reality, the AP’s findings are fully consistent with reporting by the respected Bureau of Investigative Journalism which has found that the drone war in Pakistan has killed 464-818 civilians (including 175 children) out of  2,413-3,058 total deaths, a 15%-34% civilian fraction of the dead.  Note that the injured, maimed, widowed and orphaned are not counted among these victims.

President Obama recently caused a stir with a public remark that “drones have not caused a huge number of civilian casualties.”  But they are killing thousands, and it appears that at least about a quarter of the victims are not among the “militants” being targeted.  In this light, it seems doubtful that ‘Really, it’s not as many dead kids as you think’ is such a good talking point for proponents of so-called “targeted killing.”

Author information

Mark Gubrud