Guidelines for the human control of weapons systems [PDF]

Authored by Noel Sharkey, chair of ICRAC[1]

Since 2014, high contracting parties to the CCW have expressed interest and concern about the meaningful human control of weapons systems. There is an extensive scientific and engineering literature on the dynamics of human-machine interaction and human supervisory control of machinery. A short guide is presented here consisting of two parts. Part 1 is a simple primer on the psychology of human reasoning. Part 2 outlines different levels for the control of weapons systems, adapted from human-machine interaction research, and discusses them in terms of the properties of human reasoning. This outlines which of the levels can ensure the legality of human control of weapons systems and guarantee that precautionary measures are taken to assess the significance of potential targets, their necessity and appropriateness, as well as the likely incidental and possible accidental effects of the attack.

1. A short primer on human reasoning for the control of weapons

A well-established distinction in human psychology, backed by over 100 years of substantial research, divides human reasoning into two types: (i) fast automatic processes needed for routine and/or well tasks like riding a bicycle or playing tennis and (ii) slower deliberative processes needed for thoughtful reasoning such as making a diplomatic decision.

The drawback of deliberative reasoning is that it requires attention and memory resources and so it can easily be disrupted by anything like stress, or being pressured into making a quick decision.

Automatic processes kick in first, but we can override them if we are operating in novel circumstances or performing tasks that require active control or attention. Automatic processes are essential to our normal functioning, but they have a number of liabilities when it comes to making important decisions such as those required to determine the legitimacy of a target.

Four of the known properties of automatic reasoning[2] illustrate why it is it problematic for the supervisory control of weapons.

• neglects ambiguity and suppresses doubt. Automatic processes jump to conclusions. An unambiguous answer pops up instantly without question. There is no search for alternative interpretations or uncertainty. If something looks like it might be a legitimate target, in ambiguous circumstances, automatic reasoning will be certain that it is legitimate.
• infers and invents causes and intentions. Automatic reasoning rapidly invents coherent causal stories by linking fragments of available information. Events that include people are automatically attributed with intentions that fit a causal story. For example, people loading muckrakes onto a truck could initiate a causal story that they were loading rifles. This is called assimilation bias in the human supervisory control literature.[3]
• is biased to believe and confirm. Automatic reasoning favours uncritical acceptance of suggestions and maintains a strong bias. If a computer suggests a target to an operator, automatic reasoning alone would make it highly likely to be accepted. This is automation bias.[4] Confirmation bias[5] selects information that confirms a prior belief.
• focuses on existing evidence and ignores absent evidence. Automatic reasoning builds coherent explanatory stories without consideration of evidence or contextual information that might be missing. What You See Is All There Is (WYSIATI)[6]. It facilitates the feeling of coherence that makes us confident to accept information as true. For example, a man firing a rifle may be deemed to be a hostile target with WYSIATI when a quick look around might reveal that he is shooting a wolf hunting his goats.

2. Levels of human control and how they impact on human decision-making

We can look at levels of human control for weapons systems by adapting research from the human supervisory control literature as shown in Table 1.[7]

Level 1 control is the ideal. A human commander (or operator) has full contextual and situational awareness of the target area at the time of a specific attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack. There is active cognitive participation in the attack and sufficient time for deliberation on the nature of the target, its significance in terms of the necessity and appropriateness, and likely incidental and possible accidental effects. There must also be a means for the rapid suspension or abortion of the attack.

Level 2 control could be acceptable if it is shown to meet the requirement of deliberating on potential targets. The human operator or commander should deliberatively assess necessity and appropriateness and whether any of the suggested alternatives are permissible objects of attack. Without sufficient time or in a distracting environment the illegitimacy of a target could be overlooked.

A rank ordered list of targets is particularly problematic as automation bias could create a tendency to accept the top ranked target unless sufficient time and attentional space is given for deliberative reasoning.

Level 3 is unacceptable. This type of control has been experimentally shown to create what is known as automation bias in which human operators come to trust computer generated solutions as correct and disregard or don’t search for contradictory information. Cummings experimented with automation bias in a study on an interface designed for supervision and resource allocation of in-flight GPS guided Tomahawk missiles.[8] She found that when the computer recommendations were wrong, operators using Level 3 control had a significantly decreased accuracy.

Level 4 is unacceptable because it does not promote target validation and a short time to veto would reinforce automation bias and leave no room for doubt or deliberation. As the attack will take place unless a human intervenes, this undermines well-established presumptions under international humanitarian law that promote civilian protection.

The time pressure will result in operators neglecting ambiguity and suppressing doubt, inferring and inventing causes and intentions, being biased to believe and confirm, focusing on existing evidence and ignoring absent but needed evidence. An example of the errors caused by demands of fast veto was in the 2004 Iraq war when the U.S. Army’s Patriot missile system shot down a British Tornado and an American F/A-18, killing three pilots.

Level 5 control is unacceptable as it describes weapons that are autonomous in the critical functions of target selection and the application of violent force.

It should be clear from the above that there are lessons to be drawn both from the psychology of human reasoning and from the literature on human-machine interaction. An understanding of this research is urgently needed to ensure that human-machine interaction is designed to get the best level of human control needed to comply with the international law in all circumstances.

Conclusion: Necessary conditions for meaningful human control of weapons.

A commander or operator should

• have full contextual and situational awareness of the target area at the time of initiating a specific attack;
• be able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack, such as changes in the legitimacy of the targets;
• have active cognitive participation in the attack;
• have sufficient time for deliberation on the nature of targets, their significance in terms of the necessity and appropriateness of an attack and the likely incidental and possible accidental effects of the attack and…
• …have a means for the rapid suspension or abortion of the attack.

—

[1] Special thanks to Lucy Suchman, Frank Sauer and Amanda Sharkey and members of ICRAC for helpful comments.

[2] D. Kahneman 2011:, Thinking, Fast and Slow, Penguin Books. He refers to the two processes as System 1 and System 2, These are exactly the same as the terms automatic and deliberative used here for clarity and consistency.

[3] J.M. Carroll and M.B. Rosson, ‘Paradox of the active user’, in J.M. Carroll (eds.), Interfacing Thought: Cognitive Aspects of Human-Computer Interaction (MIT Press, 1987), 80–111.

[4] K.L. Mosier and L.J. Skitka 1996: Human decision makers and automated decision aids: made for each other?, in: Mouloua, M. (Eds.): Automation and Human Performance: Theory and Applications, Lawrence Erlbaum Associates, 201–220.

[5] C.G. Lord, L. Ross and M. Lepper 1979: ‘Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence’, Journal of Personality and Social Psychology, 47, 1231–1243.

[6] Kaheneman ibid.

[7] For a more in-depth understanding of these analyses and references see N. Sharkey 2016: Staying in the Loop. Human Supervisory Control of Weapons, in: Bhuta, Nehal et al. (Eds.): Autonomous Weapons Systems. Law, Ethics, Policy. Cambridge University Press, 23-38.

[8] M.L. Cummings 2006: Automation and Accountability in Decision Support System Interface Design, in: Journal of Technology Studies 32: 1, 23–31.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

In July 2015, an open letter from artificial-intelligence experts and roboticists
called for a ban on autonomous weapon systems (AWS), comparing
their revolutionary potential to that of gun powder and nuclear weapons.

According to a 2012 Pentagon directive, AWS are weapon systems which,
‘once activated … can select and engage targets without further intervention
by a human operator’. Proponents of AWS have suggested that they
could offer various benefits, from reducing military expenditure to ringing
in a new era of more humane and less atrocious warfare. By contrast, critics
– some characterising AWS as ‘killer robots’ – expect the accompanying
political, legal and ethical risks to outweigh these benefits, and thus argue
for a preventive prohibition.

AWS are not yet operational, but decades of military research and development,
as well as the growing technological overlap between the rapidly
expanding commercial use of artificial intelligence (AI) and robotics, and
the accelerating ‘spin-in’ of these technologies into the military realm, make
autonomy in weapon systems a possibility for the very near future. Military
programmes adapting key technologies and components for achieving
autonomy in weapon systems, as well as the development of prototypes
and doctrine, are well under way in a number of states.

Accompanying this work is a rapidly expanding body of literature on the
various technical, legal and ethical implications of AWS. However, one particularly
crucial aspect has – with exceptions confirming the rule – received
comparably little systematic attention: the potential impact of autonomous
weapon systems on global peace and strategic stability.
By drawing on Cold War lessons and extrapolating insights from the
current military use of remotely controlled unmanned systems, we argue
that AWS are prone to proliferation and bound to foment an arms race
resulting in increased crisis instability and escalation risks. We conclude
that these strategic risks justify a critical stance towards AWS.

Read the full article here: Altmann, Jürgen/Sauer, Frank 2017: Autonomous Weapon Systems and Strategic Stability, in: Survival 59 (5): 117-142.

Author information

Frank Sauer

Click here to read this post in braille

Reflections on the Review Conference as a newcomer to CCW

The Fifth Review Conference of the Convention on Conventional Weapons (CCW) was a great success for advocates of a ban on fully autonomous weapons. Held at the United Nations in Geneva in December 2016, the Conference was also an opportunity for me to discover and reflect on the processes and challenges of the CCW, to which I was a newcomer.

.

I became involved when I attended the Conference as part of Harvard Law School’s International Human Rights Clinic (IHRC).  I also contributed to a report that IHRC co-published with Human Rights Watch the week before the Review Conference. Making the Case: The Dangers of Killer Robots and the Need for a Preemptive Banrebuts the major arguments against a prohibition on the development and use of fully autonomous weapons. These weapons, also known as killer robots and lethal autonomous weapons systems, would be able to select and engage targets without human intervention.

The Review Conference was a key step toward a ban because states parties agreed to formalise talks on killer robots by establishing a Group of Government Experts (GGE), which will meet for 10 days in 2017. This GGE creates the expectation of an outcome as past GGEs have led to negotiation of new or stronger CCW protocols. It provides a forum for states and experts to discuss the parameters of a possible protocol which hopefully will take the form of a ban. The Review Conference also showed that support a ban is gaining traction around the world. Argentina, Panama, Peru and Venezuela joined the call for the first time at the Conference, bringing to 19 the number of states in favour of a ban.

The establishment of a GGE was the news I eagerly waited for the entire week. When the Review Conference opened on December 12, this result did not seem guaranteed. Decisions under the CCW are adopted on the basis of the consensus. This means that any state can block progress and the Russian delegation, from the beginning of the week, forcefully opposed the move to set up a GGE. All other countries that addressed killer robots during the Review Conference explicitly supported establishing such a group. There was something strange about the risk of a single state blocking efforts openly promoted by numerous countries, and I wondered whether, faced with the threat of isolation, it would actually do so. Ultimately, this opposition appears to have been overcome by overwhelming support for more formal discussions.

.

I first heard about fully autonomous weapons when I joined IHRC in September. At the Review Conference, I realised how invested I had become in this issue and how relieved I was when, on Friday, it became clear that Russia was not going to block a GGE. Fully autonomous weapons are still only under development. Yet, because they have the potential to dramatically change the way that wars are fought, it is incumbent upon us to address the dangers they pose before they find their way to military arsenals and the battlefield.

Several other points caught my attention throughout the week.

Firstly, I joined the Review Conference as part of the Campaign to Stop Killer Robots, an international coalition of non-governmental organisations (NGOs) working towards a preemptive ban on these weapons. In this capacity, I found it interesting and encouraging to observe the role played by civil society at the Review Conference, including doing advocacy, releasing research publications and making statements during the sessions. In their public remarks, state representatives often explicitly acknowledged the work of specific NGOs and experts and the importance of civil society engagement in the dialogue. Many diplomats also attended side events, organised by the Campaign, such as one on the need to adopt a ban rather than a regulatory approach to deal with the dangers associated with killer robots. In the never-ending discussions about the correct balance to strike between military interests and humanitarian concerns, civil society has a vital role to play in emphasising the importance of humanitarian protection and pushing states to adopt ambitious goals. Civil society’s efforts are all the more important when it comes to killer robots which have the potential to revolutionise warfare and raise deep ethical questions.

.

Secondly, I was surprised and concerned by the limited media coverage of the Review Conference, especially given the fact that a Review Conference happens only once every five years and addresses matters of global concern. Discussions about killer robots should take into account the views of the public at large because delegating decisions about the use of lethal force to machines raises fundamental moral and ethical questions and international law prohibits weapons that run counter to the dictates of the public conscience.  Media coverage is important to raise the public’s awareness and facilitate its involvement in the debate. Civil society can contribute by engaging with the media and disseminating information about emerging weapons technologies that have the potential to affect societies and the world we live in. In so doing, civil society can promote media scrutiny and public participation and thereby put greater pressure on states to be ambitious and adopt encompassing solutions.

Finally, much of the debate at the Conference concentrated on the issue of finances. Financial constraints forced some discussions to take place in an informal setting without the use of official translators. Dozens of countries throughout the week noted their concerns at the financial difficulties facing the CCW. Given the fact that the Conference lasted only five days, it was regrettable that financial discussions took time away from the substantive issues. If this pattern continues, there is a risk that it will undermine the effectiveness and impact of the GGE in 2017 and the CCW as a whole. States parties should therefore take steps to resolve the situation by making their financial contributions as soon as possible.

Author information

Frank Sauer

Sauer, Frank 2016: Stopping ‘Killer Robots’: Why Now Is the Time to Ban Autonomous Weapons Systems, in: Arms Control Today 46 (8): 8-13.

Click here to read the full article.

NEW: Click here for the BRF file of the full article

[F]our possible outcomes can be predicted for the CCW process. The first would be a legally binding and preventive multilateral arms control agreement derived by consensus in the CCW and thus involving the major stakeholders, the outcome referenced as “a ban.” Considering the growing number of states-parties calling for a ban and the large number of governments calling for meaningful human control and expressing considerable unease with the idea of autonomous weapons systems, combined with the fact that no government is openly promoting their development, this seems possible. It would require mustering considerable political will. Verification and compliance for a ban, as well as for weaker restrictions, would then require creative arms control solutions. After all, with full autonomy in a weapons system eventually coming down to merely flipping a software switch, how can one tell if a specific system at a specific time is not operating autonomously? A few arms control experts are already wrapping their heads around these questions.

 

Can #KillerRobots (autonomous weapons systems) work as preventive arms control? More in October’s #ArmsControlToday https://t.co/E7sDVzdmbn pic.twitter.com/LwPSojH9Gr

— Arms Control Assoc (@ArmsControlNow) October 13, 2016

The second outcome would be restrictions short of a ban. The details of such an agreement are impossible to predict, but it is conceivable that governments could agree, for example, to limit the use of autonomous weapons systems, such as permitting their use against materiel only.

The third would be a declaratory, nonbinding agreement on best practices. Such a code of conduct would likely emphasize compliance with existing international humanitarian law and rigorous weapons review processes, in accordance with Article 36 of Additional Protocol I to the Geneva Conventions.

Finally, there may be no tangible result, perhaps with one of the technologically leading countries setting a precedent by fielding autonomous weapons systems. That would certainly prompt others to follow, fueling an arms race. In light of some of the most advanced standoff weapons, such as the U.S. Long Range Anti-Ship Missile or the UK Brimstone, each capable of autonomous targeting during terminal flight phase, one might argue that the world is already headed for such an autonomy arms race.

Implementing autonomy, which mainly comes down to software, in systems drawn from a vibrant global ecosystem of unmanned vehicles in various shapes and sizes is a technical challenge, but doable for state and nonstate actors, particularly because so much of the hardware and software is dual use. In short, autonomous weapons systems are extremely prone to proliferation. An unchecked autonomous weapons arms race and the diffusion of autonomous killing capabilities to extremist groups would clearly be detrimental to international peace, stability, and security.

The nascent social taboo against machines autonomously making kill decisions – Frank Sauer in @ArmsControlNow https://t.co/nBTGtXLT5R #CCWUN

— Mary Wareham (@marywareham) October 19, 2016

This underlines the importance of the current opportunity for putting a comprehensive, verifiable ban in place. The hurdles are high, but at this point, a ban is clearly the most prudent and thus desirable outcome. After all, as long as no one possesses them, a verifiable ban is the optimal solution. It stops the currently commencing arms race in its tracks, and everyone reaps the benefits. A prime goal of arms control would be fulfilled by facilitating the diversion of resources from military applications toward research and development for peaceful purposes—in the fields of AI and robotics no less, two key future technologies.

Author information

Frank Sauer

by Juergen Altmann and Frank Sauer

This analysis originally appeared as a guest post on duckofminerva.com

Autonomous weapon systems: rarely has an issue gained the attention of the international arms control community as quickly as these so-called killer robots. “Once activated, they can select and engage targets without further intervention by a human operator“, according to the Pentagon. They are, judging from the skepticism prevalent in epistemic communities and public opinion alike, a controversial development.

Come next Monday, the United Nations in Geneva will begin its third informal experts meeting on this emerging arms technology. For the third year in a row, various technical, legal and ethical questions surrounding autonomous weapons will be discussed at the UN’s Convention on Certain Conventional Weapons (CCW): Where does autonomy begin, where does meaningful human control end? Can these systems function in compliance with international humanitarian law? Who is accountable if things go awry? Can “outsourcing” kill-decisions to machines be morally acceptable in the first place?

Depending on how CCW States Parties answer these questions, the still nascent social taboo that forbids the use of machines autonomously making kill-decisions might spawn a human security regime and be codified in a CCW protocol. In short, a ban might be in the cards for killer robots.

And in fact, there is an additional set of compelling reasons for preventive arms control that received comparably less attention so far (with notable exceptions, of course): the impact of killer robots on peace and stability.

Stability: not a Cold War relic

Stability became a key notion in Cold War international thought for two reasons. First, the arms race. Arms competition instability exists if the classic dynamic of one side deploying systems which lead adversaries to respond in kind and vice versa goes unchecked, with horizontal and vertical proliferation in tow. Crises were the second reason. Crisis instability exists if there are significant incentives to initiate an attack quickly. These can also arise when (conventional) war is already underway; hastening the escalation to higher levels of conflict, potentially even across the nuclear threshold due to a “use them or lose them”-situation.

The vicious cycle of an uncurbed arms race as well as the dangers of overboiling crises and deterrence failure – backed up by the accidental nuclear war scares caused by early-warning slipups and human error – provided cautionary tales and fueled the strive for stability via arms control during the Cold War, not only in the nuclear but also in the conventional realm with the Conventional Armed Forces in Europe (CFE) Treaty. IR and arms control literature documents these lessons. They carry over to the dawning age of autonomous weapons.

Proliferation and arms race instability

Strictly speaking, autonomous weapons do not exist yet. They are not to be confused with automatic defense systems capable of “firing without a human in the loop”. These are stationary or fixed on ships or trailers and mostly fire at inanimate targets such as incoming munitions. More importantly, they just repeatedly perform pre-programmed actions and operate in a comparably structured and controlled environment.

Autonomous weapon systems, in contrast, would have their own means of propulsion and be able to operate without human control or supervision in dynamic, unstructured, open environments over an extended period of time, potentially learning and adapting their behavior on the go. The military advantages – compared to today’s remotely piloted systems – are obvious. Think future autonomous combat drone sent off to seek, identify, track and attack targets on its own, and you’re spot on. They are called killer robots for a reason.

The drone sector gives an indication of what to expect. Between 2001 and 2015, the number of countries with armed drones has increased from two to ten (add Hamas and Hezbollah to that), and at least 11 countries are currently developing them.

Meanwhile, everything points toward weapon autonomy as the next logical step. The US, with its newly stated third offset strategy explicitly embraces autonomy to achieve military-technological superiority and is consequently leading the way in the air, on the ground, on the sea and below it. And while the US is the only country to have introduced a doctrine for the deployment and use of autonomous weapon systems, claiming restraint, Deputy Secretary of Defense Bob Work just recently stated that the delegation of lethal authority will inexorably happen.

Absent an international ban, one would expect others to follow that lead. After all, who would allow a “killer robot gap”? Especially considering that implementing autonomy in already existing systems in a vibrant ecosystem of unmanned vehicles in various shapes and sizes is not the equivalent of starting a nuclear program from scratch – it’s a technical challenge, yes, but doable, particularly with significant portions of the hard- and software being dual-use. And we are not even considering technology export yet. In short, an unchecked robotics arms race is in the making – with weapons potentially proliferating to everyone, including oppressive regimes and non-state actors.

Crisis escalation and instability

Autonomous weapons are commonly projected as systems of systems operating in swarms. With that in mind, imagine a severe crisis, the swarms of adversaries operating in close proximity of each other. A coordinated attack of one could wipe out the other within missile flight time – that is seconds. The control software would have to react fast in order to use its weapons before they are lost. Sun glint in visual data misinterpreted as a rocket flame, sudden, unforeseen moves of the enemy swarm, a simple software bug could trigger an erroneous “counter”-attack. And while this could happen on a small scale at first, the sequence of events developing from two autonomous systems of systems interacting at rapid speed could never be trained nor tested nor, really, foreseen. The stock market provides cautionary tales of such unforeseeable algorithm interactions. Introducing algorithms in conflict bears an enormous risk of uncontrolled escalation from crisis to war.

In addition, swarms of autonomous weapons would generate new possibilities for disarming surprise attacks. Small, stealthy or extremely low-flying systems are difficult to detect, the absence of a remote-control radio link makes detection even harder. Russia already was not very amused when the idea of using stealthy drones for missile defense was floated in the US. It’s easy to see why. When nuclear weapons or strategic command-and-control systems are, or are perceived to be, put at risk by undetectable swarms that are hard to defend against, autonomous conventional capabilities end up causing instability at the strategic level.

Hitting the brakes

The case of autonomous weapon systems is not one of “we need them because they have them”. After all, no one has them – yet. We would be well-advised to keep it this way. Preventive arms control is prudent. Not only would it curb the looming arms race, a ban would prevent the excessive acceleration of battle that threatens to escape human understanding and the possibility of staying in control during crises. Sometimes humans make mistakes, and humans are slower than machines. But when things threaten to get out of hand, slow is good. That is why we need to hit the brakes now.

ICRAC’s Juergen Altmann, PhD, Researcher and Lecturer, Technische Universität Dortmund, is a physicist and peace researcher specialized in the assessment of military technology and preventive arms control. He was among the first scholars to study the military uses of nanotechnology.

ICRAC’s Frank Sauer, PhD, Senior Research Fellow and Lecturer, Bundeswehr University Munich, is an International Relations scholar focusing on issues of international security. He is the author of Atomic Anxiety: Deterrence, Taboo and the Non-Use of U.S. Nuclear Weapons.

Author information

Frank Sauer

You can read more on the importance of this letter to the current global effort of banning lethal autonomous weapon systems (LAWS) on the Campaign to Stop Killer Robots website.

ICRAC is committed to the peaceful uses of robotics and the regulation of robot weapons. We welcome this open letter, and numerous members of ICRAC have not only signed but also commented on this youngest development over the last couple of days in various media. Below is a selection of audio and video content in English, Italian and German language.

CNET – Ban autonomous weapons, urge AI experts including Hawking, Musk and Wozniak
(with ICRAC’s Noel Sharkey and Thomas Nash)
Release date: 27 Jul 2015

Huffpost Live – A.I. Experts Push For Military Robot Ban
(with ICRAC’s Heather Roff and Ian Kerr)
Hundreds of artificial intelligence experts and revered thinkers, including Stephen Hawking and Elon Musk, are calling for a global ban on military robots. We explore the issue and whether these autonomous weapons could lower the threshold for war.
Release date: 28 Jul 2015

CTV – Will the growth of killer robots set off a global arms race?
(with ICRAC’s Ian Kerr)
In this 5 minute interview with Canada AM host, Beverly Thompson, Ian Kerr discusses the the difference between semi-autonomous and autonomous weapons, the call to ban killer robots, why he is a signatory to the open letter, and why efficacy is not the only consideration in deciding whether to ban a dangerous use of technology.
Release date: 04 Aug 2015

BBC World Service – Fighting off ‘Killer Robots’
(with ICRAC’s Heather Roff)
More than 1,000 tech experts, scientists and researchers have written a letter warning about the dangers of autonomous weapons, warning that a ‘military AI race is a bad idea’. One signatory, Heather Roff Perkins from the University of Denver spoke to the BBC’s Dominic Laurie.
Release date: 28 Jul 2015

Killer robots: the coming arms race?
(with ICRAC’s Peter Asaro)

Are you worried about killer robots? Last week, some of the most prominent thinkers in science and technology signed an open letter that warned of the coming arms race should militaries pursue the development and deployment of artificially intelligent weaponry. The letter was written by The Campaign to Stop Killer Robots, an international coalition of NGOs, and was signed by almost 14,000 people, including Stephen Hawking, Elon Musk, and Steve Wozniak. Today, we discuss the threat of fully autonomous weapons and artificially-intelligent warfare with PETER ASARO, professor of media studies at The New School and spokesperson for The Campaign to Stop Killer Robots, roboticist and Georgia Tech professor RONALD ARKIN, as well as GEORGE ZARKADAKIS, writer and AI architect.

Release date: 6 Aug 2015

Ban Killer Robots Interview – Calgary Newstalk 770
(with ICRAC’s Ian Kerr)
In this 25 minute interview with Roger Kinkade and Rob Breakenridge of Newstalk770, CHQR Radio, Ian Kerr talks at length about autonomous weapons and the AI community’s call to ban “killer robots”. The conversation is about what a killer robot is, why they are likely to be developed, what the dangers are if we don’t ban them and a number of broader issues regarding the future of artificial intelligence.
Release date: 8 Aug 2015

Author information

Frank Sauer

But we are only at the beginning. There is lots of work left to do in regard to raising awareness, clarifying the issue, fleshing out the details of the concepts involved, and moving the debate forward at both the CCW and other international fora.

Some background: LARS or LAWS or Killer Robots?

PHALANX – Automatic defensive systems to protect human life are not the problem. Source: Wikimedia Commons

Obviously, militaries all over the world already deploy systems which operate “on their own“, but these systems are currently confined to defensive functions such as the interception of rockets, artillery fire and mortars, either ship-based or stationary on land. The most prominent ones are PHALANX, PATRIOT, IRON DOME or MANTIS (see also Human Rights Watch 2012) designed for use against such inanimate targets, if necessary, without human intervention (the rationale being that there may not be enough time for human intervention). However, these defensive systems operate automatically rather than autonomously, simply performing repeated pre-programmed actions.

To distinguish them from these precursors, weapons systems are described as autonomous if they operate without human control or supervision, perhaps over a longer period of time, in dynamic, unstructured, open environments. In other words, these are mobile (assault) weapons platforms which are equipped with on-board sensors and decision-making algorithms, enabling them to guide themselves. As they could potentially have the autonomous capability to identify, track and attack humans or living targets, they are known as “lethal autonomous robots“ (LARS) or, to use CCW’s current terminology, “lethal autonomous weapons systems“ (LAWS). Of course, one might just call them Killer Robots for short instead – because that is essentially what they are.

Why “autonomy” in mobile weapon systems – and what are the problems with LAWS?

As of today, it is mainly for applications underwater or in the air – in other words, in less complex but more inaccessible environments – where the drive towards more autonomy is becoming most apparent.
In a nutshell, three driving factors can be identified and are put forward by proponents of LAWS.

1. Transferring all the decision-making to the weapons system offers various benefits from a military perspective. After all, there is no longer any need for a control and communication link, which is vulnerable to disruption or capture and may well reveal the system’s location, and in which there is invariably some delay between the issuing of the command by the responsible person and the execution of the command. The time benefits already afforded by defensive systems are also valuable from a tactical perspective during military assaults. In the drone sector, a number of research and technology demonstrator programs have therefore been launched to develop (more) autonomous systems; examples are the X-47B in the US, Taranis in the UK, and the French nEUROn project.
2. As autonomous systems are immune to fear, stress and overreactions, some observers believe that they offer the prospect of more humane warfare (see Ron Arkin’s arguments in his debate with ICRAC’s co-founder Rob Sparrow). Proponents further argue that not only are machines devoid of negative human emotions; they also lack a self-preservation instinct, so could well delay returning fire in extreme cases. All in all, this, it is argued, could prevent some of the atrocities of war.
3. Some observers draw attention to the superior efficiency of LAWS and their cost-cutting potential, especially due to the reduced need for personnel.

But the problems raised by LAWS are manifold.

The US X-47B technology demonstrator. Source: Wikimedia Commons

From a military perspective, there is a certain amount of tension between autonomous systems and military leadership structures. In fact, it was noticeable how support from the military at the Expert Meeting in Geneva was somewhat muted. For them, it was mostly about toying with ideas and looking at ways of deploying these systems in strictly controlled scenarios – for example as anti-materiel weapons. But these scenarios remain highly artificial. It is entirely unclear how the use of force could be restricted against only other military hardware in the chaos of battle. Also, with much of the robotics revolution in the military driven by commercial off-the-shelf-technology (see below) the risks of proliferation and potentially destabilizing effects to peace and security quickly come to mind regarding the wider politico-military context (Altmann 2013). And here another downside of unmanned systems in general, the lowering of the threshold to the use of military force (Sauer/Schoernig 2012), might become an even more vexing problem with LAWS in the future.

From an international law perspective, there is considerable doubt that LAWS could potentially be capable of distinguishing between civilians and combatants and ensure that the military use of force is proportionate. Numerous international law and robotics experts doubt that it is possible, in the foreseeable future, to pre-programme machines to abide by international law in the notoriously grey area of decision-making in times of war (Sharkey 2012; Wagner 2012). A further objection against LAWS is that the body of international law is based on the premise of human agency; it is therefore unclear who would be legally responsible and accountable if people – particularly civilians – were unlawfully injured or killed by LAWS (Sparrow 2007). Lastly, the Martens Clause, which forms part of customary international law, holds that in cases not (yet) covered in the regulations adopted in international law, the principles of the laws of humanity and the dictates of the public conscience apply. And in fact, the general public has serious concerns about LAWS. The findings of a representative survey, unfortunately available only for the US at the moment, show that a majority (55%) of Americans are opposed to autonomous weapons on humanitarian grounds, with 40% “strongly opposed” (Carpenter 2014).

What follows from this is that the ethical dimension may well pose the greatest problem regarding LAWS. In short, it stands to reason that giving machines the power to decide on the use of force against people violates basic principles of humanity and is, per se, unacceptable (Gubrud 2012; Asaro 2012). In fact, the report of the CCW Expert Meeting emphasised this very point, stressing that LAWS could end up undermining human dignity, as these systems cannot understand or respect the value of human life, yet would have the power to determine when to take it away.

LAWS on the international agenda: The challenges involved

Noone likes a Killer Robot! Source: Charli Carpenter’s survey on “How Do Americans Feel About Fully Autonomous Weapons?”

With LAWS, the CCW has identified a new topic, which – according to seasoned CCW participants – has been placed on the agenda with unprecedented speed and is attracting lively interest from the international community. Exactly what is behind this is not one hundred percent clear though. On the one hand, it seems plausible that countries have discovered their genuine interest in a development which is deemed to require regulation, and, after the process on cluster munitions failed to conclude a new protocol in 2011, are keen to demonstrate the CCW’s capacity to act. However, the CCW has a fearsome reputation as a place where good ideas go to die a slow and silent death. So it is also possible that some countries which might have an interest in developing and deploying LAWS (from purely a military technology perspective, this applies primarily to the US, Israel, China, Russia and the United Kingdom) will use the CCW process to stall for time and smother the anti-LAWS campaign over the coming years.

But the Campaign to Stop Killer Robots not only brought the issue of LAWS to the CCW’s attention in record time. The Campaign – a coalition of 52 NGOs from 24 countries which provides a platform for coordinated civil society and academic activities – is, of course, also aware of the time sensitivity of the issue and other hurdles and intricacies involved.

Hence the first goal must be to work towards a CCW protocol banning LAWS as swiftly as possible – a preemptive ban, that is, which would come into effect before countries and the arms industry invest so much in LAWS that the window of opportunity for a preemptive solution closes.

The dual use issue makes this especially pressing. Research on autonomous robots is already underway in countless university laboratories and companies, and there is massive commercial interest in robotics. And the problem lies in the fact that the integration of commercial off-the-shelf-technology has long been a driver of developments in the field of military technology.

So, to be clear: The Campaign is all for research and innovation in all fields of autonomous systems and robotics. Especially we at ICRAC like to say: We love robots! However, we want to see them used for peaceful purposes, i.e. we love autonomous robots that do not have a “kill function” or are deployed to coerce or terrorize human beings. The speed of technological progress makes drawing this line a challenging endeavor. Especially since the potential military relevance of LAWS is, obviously, much greater than that of, say, blinding lasers (CCW Protocol IV) to which a comparison is sometimes drawn.

Where automation can serve to protect human life – as in the aforementioned defensive systems – it is not necessarily a problem, but when machines make life and death decisions without human intervention and responsibility, a line is arguably being crossed. But where and how to draw that line to preserve basic human dignity in the practice of warfighting will be the subject of upcoming debates. At this early stage in the CCW process it is still unclear if the CCW is the right place to sort these things out and ensure a preemptive ban of lethal robots. It will thus have to be discussed in other fora as well, such as the Human Rights Council and the UN General Assembly First Committee.

Nevertheless, there are two graspable results of the CCW process already. First, five CCW parties (Cuba, Ecuador, Egypt, Pakistan and the Holy See) were already calling for a ban on LAWS at the informal CCW Expert Meeting. And no country vigorously defended or argued for the development and deployment of LAWS, although the Czech Republic and Israel underlined, in their statements in Geneva, that autonomous weapons systems may offer certain benefits. The US pursued a similar line of argument. Second, many countries (including Germany, Austria, France, Norway, Netherlands, Switzerland and the United Kingdom), made one thing very clear: they want to see guarantees of what is now called “meaningful human control“ over the use of armed force.

“(Meaningful?) human control” as the way forward?

The concept of “meaningful human control”, which was introduced into the CCW debate by Campaign NGOs (Article36) and has now been taken up by governments, is the counter-concept to “appropriate human involvement“ in the operation of (semi-)autonomous weapons systems, as specified by the US in its Directive on Autonomy in Weapon Systems, issued in November 2012. It essentially argues that “appropriate human involvement“ does not go far enough – for there may be certain circumstances in which zero human involvement may be deemed “appropriate“.

The idea is that human control over life and death decisions must always be significant – in other words, it must be considerably more than none at all and, putting it bluntly, it must also involve more than the mindless pressing of a button in response to machine-processed information. According to current practice, a human operator of weapons must have sufficient information about the target and sufficient control of the weapon, and must be able to assess its effects, in order to be able to make decisions in accordance with international law. But how much human judgment can be transferred into a technical system and exercised by algorithms before human control ceases to be “meaningful“ – in other words, before warfare is quite literally “dehumanized“?

One thing seems clear: in the future, certain time limits would have to apply if LAWS are not to become a reality across a broad front. The fact is that the human brain needs time for complex evaluation and decision-making processes – time which must not be denied to it in the interaction between human and machine, if the human role is to remain relevant; in other words, if the decision-making process is merely to be supported, not dominated, by the machine.

Some of ICRAC’s members in discussion at the UN in Geneva

The concept of “meaningful human control“ is, at present, not fully fleshed out yet, and in the further course of the CCW process there will undoubtedly be considerable wrangling over precisely how it should be filled with meaning. In that process, the Campaign will be pressing for the greatest possible role for the exercise of human judgment – not only in relation to killing but also in other decisions on the use of violence or non-lethal force.

Against this background, members of ICRAC are currently (and have been for some time) thinking more in-depth about what (meaningful) human control can and should be all about, e.g. both in terms of differentiating discrete levels of supervisory control from an analytical perspective (Sharkey 2014) and in terms of a normative reminder to seek a definition that is as clear-cut, simple and with as little degrees of meaning as possible (Gubrud 2014). In its working paper series, ICRAC has already been thinking even further ahead, pondering the design of legally binding instruments and suggesting verification and compliance measures for a possible future convention on autonomous weapons (Gubrud and Altmann 2013).

There is lots of work to do, but considerable progress has been made. As a founding member of the Campaign to Stop Killer Robots, ICRAC will keep working towards a pre-emptive ban on lethal autonomous weapon systems to ensure that the future of robotics is a peaceful one.

This post is based on a policy paper titled Autonomous Weapons Systems – Humanising or Dehumanising Warfare?, published by the German “Development and Peace Foundation” as “Global Governance Spotlight 4|2014″.

Author information

Frank Sauer

The short answer is no. But Russia is testing and may deploy at its ICBM bases a lethal mobile system which has “automatic and semi-automatic control modes.” Additionally, Deputy Prime Minister Dmitry Rogozin has recently called for “robotic systems that are fully integrated in the command and control system, capable not only to gather intelligence and to receive from the other components of the combat system, but also on their own strike.”

However, Russia’s statements at the UN have expressed concern about autonomous weapons as a threat to human life and international law, and Russia will be a participant in the Geneva CCW meeting. Moreover, a critical examination of claims that Russia is notably more aggressive in its early deployments of autonomous weapon systems than other nations, let alone that Russia is “leading a new robotic arms race,” shows these claims to be inflated and unwarranted. I have posted a detailed report at 1.0 Human.

Author information

Mark Gubrud

1. Unequivocally assert that the Arms Trade Treaty Scope includes both manned and unmanned conventional arms,
2. Build on the recent clarifications by the Group of Governmental Experts of the UN Register of Conventional Arms of the categories of weapons borrowed by the Arms Trade Treaty in its Scope. The Group authoritatively defined the categories as including armed aerial drones,
3. Develop and promote comprehensive National Control Lists of the weapons to be controlled by states party to the Arms Trade Treaty,
4. Influence the interpretation of the Arms Trade Treaty through careful monitoring and calling out states acting in bad faith, and
5. Build connections between the community working on the Arms Trade Treaty (such as Control Arms) and those working on related campaigns (such as the Campaign to Stop Killer Robots) and control regimes (such as the UN Register, Missile Technology Control Regime, the Wassenaar Arrangement and dual-use equipment control programs).

Click here to read the full paper.

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

Briefly, Foust argues that teleoperated drones today are too vulnerable to hacking through their communications links, and that the solution is to make them fully autonomous. And just as briefly, I show that this is wrong in a number of ways.

Author information

Mark Gubrud