Author information

Peter Asaro

Dr. Peter Asaro is a philosopher of science, technology and media. His work examines the interfaces between social relations, human minds and bodies, artificial intelligence and robotics, and digital media. His current research focuses on the social, cultural, political, legal and ethical dimensions of military robotics and UAV drones, from a perspective that combines media theory with science and technology studies. He has written widely-cited papers on lethal robotics from the perspective of just war theory and human rights. Dr. Asaro's research also examines agency and autonomy, liability and punishment, and privacy and surveillance as it applies to consumer robots, industrial automation, smart buildings, and autonomous vehicles. His research has been published in international peer reviewed journals and edited volumes, and he is currently writing a book that interrogates the intersections between military robotics, interface design practices, and social and ethical issues. Dr. Asaro has held research positions at the Center for Cultural Analysis at Rutgers University, the HUMlab of Umeå University in Sweden, and the Austrian Academy of Sciences in Vienna. He has also developed technologies in the areas of virtual reality, data visualization and sonification, human-computer interaction, computer-supported cooperative work, artificial intelligence, machine learning, robot vision, and neuromorphic robotics at the National Center for Supercomputer Applications (NCSA), the Beckman Institute for Advanced Science and Technology, and Iguana Robotics, Inc., and was involved in the design of the natural language interface for the Wolfram|Alpha computational knowledge engine (winner of the 2010 SXSW Web Interactive Award for Technical Achievement), for Wolfram Research. He is currently working on an Oral History of Robotics project that is funded by the IEEE Robotics and Automation Society and the National Endowment for the Humanities Office of Digital Humanities. Dr. Asaro received his PhD in the History, Philosophy and Sociology of Science from the University of Illinois at Urbana-Champaign, where he also earned a Master of Arts from the Department of Philosophy, and a Master of Computer Science from the Department of Computer Science.

Guidelines for the human control of weapons systems [PDF]

Authored by Noel Sharkey, chair of ICRAC[1]

Since 2014, high contracting parties to the CCW have expressed interest and concern about the meaningful human control of weapons systems. There is an extensive scientific and engineering literature on the dynamics of human-machine interaction and human supervisory control of machinery. A short guide is presented here consisting of two parts. Part 1 is a simple primer on the psychology of human reasoning. Part 2 outlines different levels for the control of weapons systems, adapted from human-machine interaction research, and discusses them in terms of the properties of human reasoning. This outlines which of the levels can ensure the legality of human control of weapons systems and guarantee that precautionary measures are taken to assess the significance of potential targets, their necessity and appropriateness, as well as the likely incidental and possible accidental effects of the attack.

1. A short primer on human reasoning for the control of weapons

A well-established distinction in human psychology, backed by over 100 years of substantial research, divides human reasoning into two types: (i) fast automatic processes needed for routine and/or well tasks like riding a bicycle or playing tennis and (ii) slower deliberative processes needed for thoughtful reasoning such as making a diplomatic decision.

The drawback of deliberative reasoning is that it requires attention and memory resources and so it can easily be disrupted by anything like stress, or being pressured into making a quick decision.

Automatic processes kick in first, but we can override them if we are operating in novel circumstances or performing tasks that require active control or attention. Automatic processes are essential to our normal functioning, but they have a number of liabilities when it comes to making important decisions such as those required to determine the legitimacy of a target.

Four of the known properties of automatic reasoning[2] illustrate why it is it problematic for the supervisory control of weapons.

• neglects ambiguity and suppresses doubt. Automatic processes jump to conclusions. An unambiguous answer pops up instantly without question. There is no search for alternative interpretations or uncertainty. If something looks like it might be a legitimate target, in ambiguous circumstances, automatic reasoning will be certain that it is legitimate.
• infers and invents causes and intentions. Automatic reasoning rapidly invents coherent causal stories by linking fragments of available information. Events that include people are automatically attributed with intentions that fit a causal story. For example, people loading muckrakes onto a truck could initiate a causal story that they were loading rifles. This is called assimilation bias in the human supervisory control literature.[3]
• is biased to believe and confirm. Automatic reasoning favours uncritical acceptance of suggestions and maintains a strong bias. If a computer suggests a target to an operator, automatic reasoning alone would make it highly likely to be accepted. This is automation bias.[4] Confirmation bias[5] selects information that confirms a prior belief.
• focuses on existing evidence and ignores absent evidence. Automatic reasoning builds coherent explanatory stories without consideration of evidence or contextual information that might be missing. What You See Is All There Is (WYSIATI)[6]. It facilitates the feeling of coherence that makes us confident to accept information as true. For example, a man firing a rifle may be deemed to be a hostile target with WYSIATI when a quick look around might reveal that he is shooting a wolf hunting his goats.

2. Levels of human control and how they impact on human decision-making

We can look at levels of human control for weapons systems by adapting research from the human supervisory control literature as shown in Table 1.[7]

Level 1 control is the ideal. A human commander (or operator) has full contextual and situational awareness of the target area at the time of a specific attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack. There is active cognitive participation in the attack and sufficient time for deliberation on the nature of the target, its significance in terms of the necessity and appropriateness, and likely incidental and possible accidental effects. There must also be a means for the rapid suspension or abortion of the attack.

Level 2 control could be acceptable if it is shown to meet the requirement of deliberating on potential targets. The human operator or commander should deliberatively assess necessity and appropriateness and whether any of the suggested alternatives are permissible objects of attack. Without sufficient time or in a distracting environment the illegitimacy of a target could be overlooked.

A rank ordered list of targets is particularly problematic as automation bias could create a tendency to accept the top ranked target unless sufficient time and attentional space is given for deliberative reasoning.

Level 3 is unacceptable. This type of control has been experimentally shown to create what is known as automation bias in which human operators come to trust computer generated solutions as correct and disregard or don’t search for contradictory information. Cummings experimented with automation bias in a study on an interface designed for supervision and resource allocation of in-flight GPS guided Tomahawk missiles.[8] She found that when the computer recommendations were wrong, operators using Level 3 control had a significantly decreased accuracy.

Level 4 is unacceptable because it does not promote target validation and a short time to veto would reinforce automation bias and leave no room for doubt or deliberation. As the attack will take place unless a human intervenes, this undermines well-established presumptions under international humanitarian law that promote civilian protection.

The time pressure will result in operators neglecting ambiguity and suppressing doubt, inferring and inventing causes and intentions, being biased to believe and confirm, focusing on existing evidence and ignoring absent but needed evidence. An example of the errors caused by demands of fast veto was in the 2004 Iraq war when the U.S. Army’s Patriot missile system shot down a British Tornado and an American F/A-18, killing three pilots.

Level 5 control is unacceptable as it describes weapons that are autonomous in the critical functions of target selection and the application of violent force.

It should be clear from the above that there are lessons to be drawn both from the psychology of human reasoning and from the literature on human-machine interaction. An understanding of this research is urgently needed to ensure that human-machine interaction is designed to get the best level of human control needed to comply with the international law in all circumstances.

Conclusion: Necessary conditions for meaningful human control of weapons.

A commander or operator should

• have full contextual and situational awareness of the target area at the time of initiating a specific attack;
• be able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack, such as changes in the legitimacy of the targets;
• have active cognitive participation in the attack;
• have sufficient time for deliberation on the nature of targets, their significance in terms of the necessity and appropriateness of an attack and the likely incidental and possible accidental effects of the attack and…
• …have a means for the rapid suspension or abortion of the attack.

—

[1] Special thanks to Lucy Suchman, Frank Sauer and Amanda Sharkey and members of ICRAC for helpful comments.

[2] D. Kahneman 2011:, Thinking, Fast and Slow, Penguin Books. He refers to the two processes as System 1 and System 2, These are exactly the same as the terms automatic and deliberative used here for clarity and consistency.

[3] J.M. Carroll and M.B. Rosson, ‘Paradox of the active user’, in J.M. Carroll (eds.), Interfacing Thought: Cognitive Aspects of Human-Computer Interaction (MIT Press, 1987), 80–111.

[4] K.L. Mosier and L.J. Skitka 1996: Human decision makers and automated decision aids: made for each other?, in: Mouloua, M. (Eds.): Automation and Human Performance: Theory and Applications, Lawrence Erlbaum Associates, 201–220.

[5] C.G. Lord, L. Ross and M. Lepper 1979: ‘Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence’, Journal of Personality and Social Psychology, 47, 1231–1243.

[6] Kaheneman ibid.

[7] For a more in-depth understanding of these analyses and references see N. Sharkey 2016: Staying in the Loop. Human Supervisory Control of Weapons, in: Bhuta, Nehal et al. (Eds.): Autonomous Weapons Systems. Law, Ethics, Policy. Cambridge University Press, 23-38.

[8] M.L. Cummings 2006: Automation and Accountability in Decision Support System Interface Design, in: Journal of Technology Studies 32: 1, 23–31.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

A common remark is that verification would require inspection of software as well as hardware, and that nations will never permit such intrusive inspections—as well as the fact that even clearly-written and well-documented software can be very difficult to read and interpret, let alone the case in which software may be deliberately obscured or encrypted. The physical form of systems that are capable of being autonomous may also not always be readily definable or discernible. Where both a cockpit and communications link to a remote operator are lacking, one may reasonably infer that a system is intended to operate autonomously, but their presence does not necessarily ensure that the system is incapable of operating autonomously.

The solution to this conundrum is already at hand, however, in the increasing emphasis in these discussions on the need for meaningful human control. This approach is increasingly recognized as a conceptual reframing of the problem of banning autonomous weapons, as already proposed in 2010 with the Berlin Statement (originally titled “The Principle of Human Control of Weapons and All Technology”). That statement asserts positively “That it is unacceptable for machines to control, determine, or decide upon the application of force or violence in conflict or war. In all cases where such a decision must be made, at least one human being must be held personally responsible and legally accountable for the decision and its foreseeable consequences.” The emphasis on personal responsibility and legal accountability for the decision to use violent force has become recognized as one of the elements of the concept of meaningful human control, which also emphasizes the role of adequate information and deliberation by the decision maker.

Thus, while it may indeed be impractical to verify compliance with a ban on “autonomous weapons” as such, it may very well be possible to verify compliance with a requirement for accountable and meaningful human control and decision in each use of violent force.

This is not to say that we should not also declaratively ban autonomous weapons–minus a list of exceptions for systems that operate autonomously but do not make significant lethal decisions autonomously, or that are purely defensive in nature and defend human life against immediate threats from incoming munitions, or are to be allowed for other, pragmatic reasons. Certainly, we should ban fully autonomous weapons. But the way to implement and verify such a ban may be better framed in terms of human control.

Two years ago, ICRAC members took part in an effort to consider measures for promoting compliance with an autonomous weapons ban. The result was a working paper, “Compliance Measures for an Autonomous Weapons Convention,” which is posted here. The work has not received wide recognition, but given that the question has begun to arise, it seems appropriate to highlight it now, rather than witness the emergence of  “a ban on killer robots would be nice, but it’s unverifiable” as a persistent canard.

The paper highlights many aspects of ensuring compliance with an autonomous weapons convention, including the enunciation of strong, simple, intuitive principles as the moral foundation for such an agreement, framing in terms of clear definitions, articulation of allowed exceptions, declaration of previously existing autonomous weapon systems, national implementing legislation, and the creation of an international treaty implementing organization (TIO). The role of the TIO in verification is detailed, particularly its support for cryptographic validation of records to tie them to particular weapon systems and the use of force at particular times (and potentially, places). These records, it is proposed, would be held by the compliant States Parties themselves and not released to the TIO or subject to any other possible compromise of military secrets, except in the case of an orderly inquiry into particular suspicious events, and possibly also some quota of routine, random inspections to verify continuous compliance. The cryptographic principles on which the tamper-proofing and time-stamping of such records can be carried out are simple and well-understood, and full encrypted records need not be exposed to the possibility of decryption if only “digital signatures” of the records are supplied to the TIO for archiving.

We believe that a scheme of this type can support rigorous verification of compliance in cases where it is suspected that a fully autonomous weapon system has been used, which should be a sufficient deterrent to their use. If coupled with other transparency and confidence-building measures, including routine on-site inspections of facilities in which remotely operated or nearly autonomous systems are developed, tested, manufactured, stockpiled, deployed or used, and with national means of intelligence which should suffice to reveal any prohibited activities large enough in scale and scope to pose a significant strategic security threat. These measures should suffice to ensure that no State Party will find the risks of non-compliance to be outweighed by uncertain and hypothetical military benefits.

– by Mark Gubrud (@mgubrud) and ICRAC’s Juergen Altmann

Author information

Mark Gubrud

1. Unequivocally assert that the Arms Trade Treaty Scope includes both manned and unmanned conventional arms,
2. Build on the recent clarifications by the Group of Governmental Experts of the UN Register of Conventional Arms of the categories of weapons borrowed by the Arms Trade Treaty in its Scope. The Group authoritatively defined the categories as including armed aerial drones,
3. Develop and promote comprehensive National Control Lists of the weapons to be controlled by states party to the Arms Trade Treaty,
4. Influence the interpretation of the Arms Trade Treaty through careful monitoring and calling out states acting in bad faith, and
5. Build connections between the community working on the Arms Trade Treaty (such as Control Arms) and those working on related campaigns (such as the Campaign to Stop Killer Robots) and control regimes (such as the UN Register, Missile Technology Control Regime, the Wassenaar Arrangement and dual-use equipment control programs).

Click here to read the full paper.

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

The full text of the paper is available here: Gubrud-Altmann_Compliance Measures AWC_ICRAC-WP2

Author information

Frank Sauer