International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Prof. Noel Sharkey, on 11 April 2018

Mr Chairperson,

We have been very pleased with this morning’s session as states begin to contemplate a move towards policies on the human control of weapons systems. On a pedantic note: we cannot talk about the meaningful human control of LAWS as that would make them no longer an autonomous weapon.

In the view of ICRAC, the control of weapons systems is more nuanced than can be captured by terms such as in-the-loop, on-the-loop, the broader loop, looping-the loop, human oversight, and appropriate human judgement. In this way we agree strongly with the statement made by Brazil and several others in this session who believe that the devil is in the detail.

For human control to be meaningful we need to examine how humans interact with machines and understand the types of human-machine biases that can occur in the selection of legitimate targets. Lessons should be learned from 30 years of research on human supervisory control of machinery xn--yxadbbg milf and more than 100 years of research on the psychology of human reasoning. This combination of this work can help us to design human-machine interfaces that allow weapons to be controlled in a manner that is fully compliant with international law and the principle of humanity.

First, there should be a focus on what the human operator MUST do in the targeting cycle. This is control by use which is governed by targeting rules under International Humanitarian Law and International Human Rights Law. Further, international law rules that apply after the use of weapons – such as those that relate to human responsibility – must be satisfied.

Second, the design of weapon systems must render them INCAPABLE of operating without meaningful human control.  This is control by design, which is governed by international weapons law. In terms of international weapons law, if the weapon system, by its design, is incapable of being sufficiently controlled in terms of the law, then such a weapon is illegal per se.

Ideally the following three conditions should be followed for the control of weapons systems:

1. a human commander (or operator) will have full contextual and situational awareness of the target area for each and every attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack.
2. there will be active cognitive participation in every attack with sufficient time for deliberation on the nature of any target, its significance in terms of the necessity and appropriateness of attack, and likely incidental and possible accidental effects of the attack and
3. there will be a means for the rapid suspension or abortion of every attack.

Thank you Mr Chairperson

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

Guidelines for the human control of weapons systems [PDF]

Authored by Noel Sharkey, chair of ICRAC[1]

Since 2014, high contracting parties to the CCW have expressed interest and concern about the meaningful human control of weapons systems. There is an extensive scientific and engineering literature on the dynamics of human-machine interaction and human supervisory control of machinery. A short guide is presented here consisting of two parts. Part 1 is a simple primer on the psychology of human reasoning. Part 2 outlines different levels for the control of weapons systems, adapted from human-machine interaction research, and discusses them in terms of the properties of human reasoning. This outlines which of the levels can ensure the legality of human control of weapons systems and guarantee that precautionary measures are taken to assess the significance of potential targets, their necessity and appropriateness, as well as the likely incidental and possible accidental effects of the attack.

1. A short primer on human reasoning for the control of weapons

A well-established distinction in human psychology, backed by over 100 years of substantial research, divides human reasoning into two types: (i) fast automatic processes needed for routine and/or well tasks like riding a bicycle or playing tennis and (ii) slower deliberative processes needed for thoughtful reasoning such as making a diplomatic decision.

The drawback of deliberative reasoning is that it requires attention and memory resources and so it can easily be disrupted by anything like stress, or being pressured into making a quick decision.

Automatic processes kick in first, but we can override them if we are operating in novel circumstances or performing tasks that require active control or attention. Automatic processes are essential to our normal functioning, but they have a number of liabilities when it comes to making important decisions such as those required to determine the legitimacy of a target.

Four of the known properties of automatic reasoning[2] illustrate why it is it problematic for the supervisory control of weapons.

• neglects ambiguity and suppresses doubt. Automatic processes jump to conclusions. An unambiguous answer pops up instantly without question. There is no search for alternative interpretations or uncertainty. If something looks like it might be a legitimate target, in ambiguous circumstances, automatic reasoning will be certain that it is legitimate.
• infers and invents causes and intentions. Automatic reasoning rapidly invents coherent causal stories by linking fragments of available information. Events that include people are automatically attributed with intentions that fit a causal story. For example, people loading muckrakes onto a truck could initiate a causal story that they were loading rifles. This is called assimilation bias in the human supervisory control literature.[3]
• is biased to believe and confirm. Automatic reasoning favours uncritical acceptance of suggestions and maintains a strong bias. If a computer suggests a target to an operator, automatic reasoning alone would make it highly likely to be accepted. This is automation bias.[4] Confirmation bias[5] selects information that confirms a prior belief.
• focuses on existing evidence and ignores absent evidence. Automatic reasoning builds coherent explanatory stories without consideration of evidence or contextual information that might be missing. What You See Is All There Is (WYSIATI)[6]. It facilitates the feeling of coherence that makes us confident to accept information as true. For example, a man firing a rifle may be deemed to be a hostile target with WYSIATI when a quick look around might reveal that he is shooting a wolf hunting his goats.

2. Levels of human control and how they impact on human decision-making

We can look at levels of human control for weapons systems by adapting research from the human supervisory control literature as shown in Table 1.[7]

Level 1 control is the ideal. A human commander (or operator) has full contextual and situational awareness of the target area at the time of a specific attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack. There is active cognitive participation in the attack and sufficient time for deliberation on the nature of the target, its significance in terms of the necessity and appropriateness, and likely incidental and possible accidental effects. There must also be a means for the rapid suspension or abortion of the attack.

Level 2 control could be acceptable if it is shown to meet the requirement of deliberating on potential targets. The human operator or commander should deliberatively assess necessity and appropriateness and whether any of the suggested alternatives are permissible objects of attack. Without sufficient time or in a distracting environment the illegitimacy of a target could be overlooked.

A rank ordered list of targets is particularly problematic as automation bias could create a tendency to accept the top ranked target unless sufficient time and attentional space is given for deliberative reasoning.

Level 3 is unacceptable. This type of control has been experimentally shown to create what is known as automation bias in which human operators come to trust computer generated solutions as correct and disregard or don’t search for contradictory information. Cummings experimented with automation bias in a study on an interface designed for supervision and resource allocation of in-flight GPS guided Tomahawk missiles.[8] She found that when the computer recommendations were wrong, operators using Level 3 control had a significantly decreased accuracy.

Level 4 is unacceptable because it does not promote target validation and a short time to veto would reinforce automation bias and leave no room for doubt or deliberation. As the attack will take place unless a human intervenes, this undermines well-established presumptions under international humanitarian law that promote civilian protection.

The time pressure will result in operators neglecting ambiguity and suppressing doubt, inferring and inventing causes and intentions, being biased to believe and confirm, focusing on existing evidence and ignoring absent but needed evidence. An example of the errors caused by demands of fast veto was in the 2004 Iraq war when the U.S. Army’s Patriot missile system shot down a British Tornado and an American F/A-18, killing three pilots.

Level 5 control is unacceptable as it describes weapons that are autonomous in the critical functions of target selection and the application of violent force.

It should be clear from the above that there are lessons to be drawn both from the psychology of human reasoning and from the literature on human-machine interaction. An understanding of this research is urgently needed to ensure that human-machine interaction is designed to get the best level of human control needed to comply with the international law in all circumstances.

Conclusion: Necessary conditions for meaningful human control of weapons.

A commander or operator should

• have full contextual and situational awareness of the target area at the time of initiating a specific attack;
• be able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack, such as changes in the legitimacy of the targets;
• have active cognitive participation in the attack;
• have sufficient time for deliberation on the nature of targets, their significance in terms of the necessity and appropriateness of an attack and the likely incidental and possible accidental effects of the attack and…
• …have a means for the rapid suspension or abortion of the attack.

—

[1] Special thanks to Lucy Suchman, Frank Sauer and Amanda Sharkey and members of ICRAC for helpful comments.

[2] D. Kahneman 2011:, Thinking, Fast and Slow, Penguin Books. He refers to the two processes as System 1 and System 2, These are exactly the same as the terms automatic and deliberative used here for clarity and consistency.

[3] J.M. Carroll and M.B. Rosson, ‘Paradox of the active user’, in J.M. Carroll (eds.), Interfacing Thought: Cognitive Aspects of Human-Computer Interaction (MIT Press, 1987), 80–111.

[4] K.L. Mosier and L.J. Skitka 1996: Human decision makers and automated decision aids: made for each other?, in: Mouloua, M. (Eds.): Automation and Human Performance: Theory and Applications, Lawrence Erlbaum Associates, 201–220.

[5] C.G. Lord, L. Ross and M. Lepper 1979: ‘Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence’, Journal of Personality and Social Psychology, 47, 1231–1243.

[6] Kaheneman ibid.

[7] For a more in-depth understanding of these analyses and references see N. Sharkey 2016: Staying in the Loop. Human Supervisory Control of Weapons, in: Bhuta, Nehal et al. (Eds.): Autonomous Weapons Systems. Law, Ethics, Policy. Cambridge University Press, 23-38.

[8] M.L. Cummings 2006: Automation and Accountability in Decision Support System Interface Design, in: Journal of Technology Studies 32: 1, 23–31.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Prof. Noel Sharkey, on 10 April 2018

Mr. Chairperson,

There have been very useful and interesting discussions this morning.

I speak here as chair of an academic NGO: the International Committee for Robot Arms Control (ICRAC) and as a member of the scientific community in the field of Artificial Intelligence and Robotics with specialty in Machine Learning.

We stress again that it would be confusing to broaden the discussion of LAWS into issues about Artificial Intelligence or weapons with emerging intelligence. By chasing definitions of LAWS down the rabbit hole of AI, we remove ourselves from the key issues that need to be urgently discussed here. The definition extracted from ICRC, and echoed by a number of states this morning, is concerned with weapons that have autonomy in the critical functions of target selection and the application of force. This is sufficient for our definitional purposes here: decisions about target selection and the application of force are delegated to a machine. Let me highlight that it does not matter what techniques or computing methods are used to create autonomy in these critical functions.

What is important here are questions about the nature of human control required and acceptable to ensure compliance with international law. It is key that we get this right. You can read more about this in ICRAC’s new working paper Guidelines for the Human Control of Weapons Systems that will be delivered at Wednesday’s side event. We support those states who have stated that the focus of this meeting should be on human control of weapons systems and human-machine interaction. In this way we can make real progress this week and protect our future no matter what the technological developments.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

icelandic institute for intelligent machines.

The Icelandic Institute for Intelligent Machines (IIIM) has issued an ethical policy that makes them the first Artificial Intelligence research and development group to reject the development of technologies intended for military operations. IIIM is an independent research institute affiliated with Reykjavik University in Reykjavik, Iceland

“It is only fitting that a research center in Iceland should field such a policy” says Kristinn R. Thórisson, Managing Director of IIIM. “A nation without a standing army and virtually no history of war in its 1100 years”.

Thórisson believes that Artificial Intelligence has great potential for the immediate benefit of society. He asks, “Why should the taxpayers’ money fund autonomous weapons meant for killing humans when they could be funding applications for civilian uses, with enormous immediate benefits to society? Why not spend the large sums of money poured into weapons development instead for a peaceful, civilian agenda?”

He adds a strong note of caution, “When people think of a war waged with machines — autonomous killing machines — they may imagine armies or robots fighting other armies of robots. And no one gets hurt, right? But evidence so far tells us that the reality may in fact be very different —much more likely is the scenario of armies of killer robots set against individuals, groups and even nation states who are not in a good position to defend themselves, or even worse, largely at at a loss to do so.”

It is certainly heartwarming to see an AI R&D institute showing such a great sense of social responsibility and it would be great to see others following their lead.

This comes after Canada’s Clearpath robotics became the first robotics company to issue a policy against autonomous weapons systems. The cards are  stacking up against autonomous weapons with support for a ban this year from Stephen Hawking, Elon Musk, Steve Wozniak, Bill Gates and Noam Chomsky. The Future of Life Institute issued an open letter against AI weapons last month with thousands of signatures from prominent figures in the field.

The full ethics policy of IIIM is given below and can also be found at http://www.iiim.is/about-iiim/ethics-policy/ For more information about the IIIM visit http://iiim.is or email (info@iiim.is).

The Board of Directors of IIIM believes that the freedom of researchers to explore and uncover the principles of intelligence, automation, and autonomy, and to recast these as the mechanized runtime principles of man-made computing machinery, is a promising approach for producing advanced software with commercial and public applications, for solving numerous difficult challenges facing humanity, and for answering important questions about the nature of human thought.

A significant part of all past artificial intelligence (AI) research in the world is and has been funded by military authorities, or by funds assigned various militaristic purposes, indicating its importance and application to military operations. A large portion of the world’s most advanced AI research is still supported by such funding, as opposed to projects directly and exclusively targeting peaceful civilian purposes. As a result, a large and disconcerting imbalance exists between AI research with a focus on hostile applications and AI research with an explicitly peaceful agenda. Increased funding for military research has a built-in potential to fuel a continual arms race; reducing this imbalance may lessen chances of conflict due to international tension, distrust, unfriendly espionage, terrorism, undue use of military force, and unjust use of power.

Just as AI has the potential to enhance military operations, the utility of AI technology for enabling perpetration of unlawful or generally undemocratic acts is unquestioned. While less obvious at present than the military use of AI and other advanced technologies, the falling cost of computers is likely to make highly advanced automation technology increasingly accessible to anyone who wants it. The potential for all technology of this kind to do harm is therefore increasing.

 For these reasons, and as a result of IIIM’s sincere goal to focus its research towards topics and challenges of obvious benefit to the general public, and for the betterment of society, human livelihood and life on Earth, IIIM’s Board of Directors hereby states the Institute’s stance on such matters clearly and concisely, by establishing the following Ethical Policy for all current and future activities of IIIM:

1 – IIIM’s aim is to advance scientific understanding of the world, and to enable the application of this knowledge for the benefit and betterment of humankind.

2 – IIIM will not undertake any project or activity intended to (2a) cause bodily injury or severe emotional distress to any person, (2b) invade the personal privacy or violate the human rights of any person, as defined by the United Nations Declaration of Human Rights, (2c) be applied to unlawful activities, or (2d) commit or prepare for any act of violence or war.

2.1 – IIIM will not participate in projects for which there exists any reasonable evidence of activities 2a, 2b, 2c, or 2d listed above, whether alone or in collaboration with governments, institutions, companies, organizations, individuals, or groups.

2.2 – IIIM will not accept military funding for its activities. ‘Military funding’ is defined as any and all funds designated to support the activities of governments, institutions, companies, organizations, and groups, explicitly intended for furthering a military agenda, or to prepare for or commit to any act of war.

2.3 – IIIM will not collaborate with any institution, company, group, or organization whose existence or operation is explicitly, whether in part or in whole, sponsored by military funding as described in 2.2 or controlled by military authorities. For civilian institutions with a history of undertaking military-funded projects a 5-15 rule will be applied: if for the past 5 years 15% or more of their projects were sponsored by such funds, they will not be considered as IIIM collaborators.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

Interviews with:
Rasha Abdul Rahim, Amnesty International, Advocate/Advisor in Arms Control, Security Trade, and Human Rights
Steve Wright, Reader in the School of Social Science at Leeds Beckett University, and ICRAC Member

This is episode 4 of 6 in the CCW Special that kicks off the new official ICRAC Podcast series.

Rasha Abdul Rahim is part of the Arms Control, Security Trade and Human Rights Team at Amnesty International and an expert in the human rights implications of autonomous weapons in military and policing applications.

Dr. Steve Wright is a professor at Leeds Beckett University who has more than 30 years under his belt working on the proliferation of weapon technologies. Steve is also ICRAC’s specialist on Less Lethal Autonomous Weapons (LLAWS).

We talk about how LLAWS could come back from conflict zones to haunt civil society and we discuss the inherent dangers for policing, border control and the suppression of populations.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

This is episode 4 of 6 in the CCW Special that kicks off the new official ICRAC Podcast series.

Nehal Bhuta is a highly respected Professor of International Law at the European University Institute in Florence. He was one of the experts chosen to speak at the 2015 CCW Informal Meeting of Experts on LAWS.

In this interview we probe some of the legal complexities of the relationship between the law and the weapons review process. We also discuss the concept of “meaningful human control”.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

This is episode 3 of 6 in the CCW Special that kicks off the new official ICRAC Podcast series.

Jody Williams has been a humanitarian campaigner for most of her adult life. In 1991 she began coordinating the successful Campaign to Ban Landmines and received the Nobel Prize for Peace in 1997. But she didn’t stop there. In 2004 Jody took the lead in establishing the Nobel Women’s Initiative with other women Nobel peace laureates and has been the chairwoman ever since. Their aim is to support and promote the work of women around the world working for peace with justice and equality.

In this interview Jody talks about the importance of the Campaign to Stop Killer Robots and tells us what is most annoying about its opponents.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

This is episode 2 of 6 in the CCW Special that kicks off the new official ICRAC Podcast series.

Steve Goose has been heavily involved in the prohibition of three nasty weapons systems. In this podcast he tells us about the discussions at the CCW are panning out and compares the pace with previous campaigns. He then explains clearly the steps at the Convention on Certain Conventional Weapons (CCW) from where we are now right through to a new protocol to prohibit the development, production and use of fully autonomous weapons systems. Steve Goose CCW intervention.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).