As has been discussed in detail at the CCW GGE over the past decade, autonomous weapon systems (AWS) raise serious concerns for international humanitarian law in regard to complying with the principles of distinction and proportionality. The risk of triggering the proliferation of arms is another stark reality posed by AWS, as is the accessibility of these types of weapon systems to non-state armed groups, among other actors. The use of AWS may further spill into the arena of national and transnational organized crime in addition to policing at the domestic level. All the while, several operational concerns remain as to the use of AWS from the perspective of accountability, bias and the use of machine-learning algorithms which may develop beyond the capacity of “the human in the loop.” There are also serious risks to regional and global stability posed by replacing human decision making with machine decision making, as it becomes more difficult for political and military leaders to anticipate and interpret the intentions, decision and actions of their adversaries, and thus find ways to avoid or de-escalate conflicts.

We also note the threat that AWS pose to compliance with international human rights, particularly the right to life, the prohibition against torture, cruel and inhumane treatment, and above all the human right to dignity. We fear that an additional protocol to the CCW would fail to address these human rights concerns. We are concerned that the automated targeting and release of non-conventional weapons, including nuclear weapons, may also fall outside the scope of any legally binding CCW protocol. We thus advocate and support all calls for a legally binding instrument to prohibit and restrict the use of AWS, and urge the Secretary-General to encourage the initiation of a forum within the United Nations General Assembly that can include all States, cover autonomy and automation in the use of all weapons, and address international humanitarian law as well as human rights concerns.

This submission is informed by our comprehensive interdisciplinary expertise. We have published extensively on the ethical, legal, technical and security challenges of autonomous weapon systems, on the question of meaningful human control, and on the challenges of escalation at speed.

Scope

In accordance with the International Committee of the Red Cross we understand an autonomous weapon system as one that, potentially after initial activation or launch by a human, selects targets based on sensor data and engages the targets without human intervention. We endorse the recommendations of the International Committee of the Red Cross for a two-tiered approach that prohibits unpredictable systems and systems that explicitly target humans, while strictly regulating the use of autonomy in all other systems for the command, control and engagement of lethal force. This includes restrictions on the time, space, scope and scale of operations of such systems, as well as the types of targets and situations in which they may be used. In particular, we strongly agree that the only permissible targets of
such systems should be military objects by nature, and never civilian or dual-use targets, which should always require human judgment. More discussion is needed on the appropriate forms and regulation of the human-machine interaction in complex command and control systems. In particular, as computers and artificial intelligence collect and automatically analyze more and more data, greater clarity is needed in what constitutes meaningful human control in the context of automated target generation and identification, and how to ensure respect and responsibility for international law when such systems are used.

Key Challenges to Global Peace and Security

• Uncontrolled Escalation and Missed Opportunities for De-escalation and Diplomacy

The technical characteristics of AWS pose a considerable risk in enabling uncontrolled escalation at speed. As the thresholds for applying military force will be lowered, the likelihood of conflicts will go up. Actions and reactions to the adversary will have to be programmed in advance. Two AWS swarms moving at relatively close distance from each other, in international air space, for example, might interact in ways that could not be mitigated or controlled by a human in an appropriate time window. In case of an enemy attack, even a few seconds delay could mean loss of one’s systems, thus there will be strong pressure for fast counterattacks that preclude human consideration.

Escalation from crisis to war, or escalating a conflict to a higher level of violence, could come about by erroneous indications of attack or a simple sensor or computer error. Mutual interaction between the control programs could not be tested in advance. The outcome of the interaction of such complex systems would be intrinsically unpredictable, but fast escalation is possible and likely. In a severe crisis with fear of preemption this could greatly destabilize the military situation between potential enemies.

As political and military leaders become increasingly dependent on systems they cannot explain or predict, it will make the traditional means of conflict resolution and de-escalation more difficult or impossible. Unpredictable systems will give leaders false impressions of their capabilities, leading to overconfidence or encouraging preemptive attacks. Moreover, automated attacks, responses, and escalations will make it more difficult for leaders to interpret the intentions, decisions and actions of their adversaries, and will also limit their options for response. Systems that automatically react or attack may miss opportunities to find other, less violent, ways to achieve military objectives, or preclude opportunities for diplomatic or political resolutions to a conflict. The overall effect of these systems will be to close off avenues and opportunities to avoid conflicts, to de-escalate conflicts, and to find means to end hostilities.

• Moral responsibility

No machine, computer or algorithm is capable of recognizing a human as a human being, nor can it respect humans as inherent bearers of rights and dignity. A machine cannot even understand what it means to be in a state of war, much less what it means to have, or to end, a human life. Decisions to end human life must be made by humans in order to be morally justifiable. These are responsibilities of unavoidable moral weight that cannot be delegated to machines or satisfied by the mere inclusion of humans in the writing of computer programs. While accountability for the deployment of lethal force is a necessary condition for moral responsibility in war, accountability alone is not sufficient for moral responsibility. This also requires the recognition of the human, respect for the human right to life and dignity, and reflection upon the value of life and the justification for the use of violent force.

• Meaningful Human Control

Much hinges on the degree to which AWS can be meaningfully controlled by humans. Robust scientific scholarship on human psychology suggests that humans experience cognitive limitations when it comes to technological/computational systems. This condition is known as automation bias by which the human is cognitively hindered from having sufficient contextual understanding to be able to intervene with systems that are fully autonomous and function at speeds beyond human capabilities. In order to safeguard meaningful human control (not merely functional control) over AI-enabled AWS, those
involved in operating or deciding to deploy AWS should have full contextual and situational awareness of the target area at the time of a specific attack. They must also be able to perceive and react to changes or unanticipated situations that arise; ensure active and deliberate participation in the action; have sufficient training and understanding of the system and its likely actions; have adequate time for meaningful control and have the means and knowledge for a rapid suspension of an action. For many AWS this is not possible. Meaningful human control is fundamental to the edifice of the laws of war and the ethics of war.

Moving Forward: A Treaty to Prohibit and Regulate the Use of AWS

We support calls from States, as well as the UN Secretary-General and the President of the ICRC, for an international legally binding treaty prohibiting and regulating the use of AWS.

What is needed is a legally binding instrument that obligates States to adhere to prohibitions and regulatory limitations for AWS. Codes of conduct and political declarations are not enough for systems that pose such grave risks to global peace and security. This legally binding instrument must apply to the automated control of all weapons, and require meaningful human control in compliance with substantive regulations for the use of force in all cases. Such a treaty should apply to all military uses of AWS and systems that generate or select targets, as well as to all police, border security and other civilian applications that automate the use of force.

The treaty should prohibit autonomous weapons systems that are ethically or legally unacceptable. This includes autonomous weapons systems for which the operation or effects cannot be sufficiently understood, predicted and explained; autonomous weapons systems that cannot be used with meaningful human control; and autonomous weapons systems designed to target human beings.

The treaty should include positive obligations for States to use AWS systems that are permitted only within the bounds of clearly stipulated regulations that ensure adherence to international human rights and the key principles of international humanitarian law. We believe that an emerging norm around meaningful human control can be articulated and codified through a treaty negotiation in a process that includes all States, civil society, and industry and technical experts. We urge the Secretary-General to advance the creation of such a forum within the General Assembly, and look forward to offering our expertise to those discussions.

Author information

Laura Nolan

As delivered by Prof. Peter Asaro, March 26, 2019.

ICRAC has been pleased to hear states shift their focus away from definitions of the technologies of autonomous weapons systems and move towards discussing restriction of their use with regards to how they should be controlled. Of course, by definition, if states wanted genuine meaningful human control of weapons systems, they would not be using autonomous weapons systems. And (as an aside) we should not forget the scientifically recognized limitations of the technology or the foreseeable threats to global security such weapons pose.

We are also pleased with the statements and working papers beginning to examine the requirements for human control and planning in military systems. While this can be multifaceted, we must not let the complexity of military planning throw a smoke screen over the core issues of the meaningful human assessment of all targets, their legitimacy and the proportionate use of force.

We are glad that we see the beginnings of a more nuanced approach to the control of weapons systems that cannot be captured by gross terms such as in-the-loop, on-the-loop, the broader loop, human oversight, and appropriate levels of human judgement. However, these terms continue to insinuate themselves in military, political and defence contractor’s narratives outside of the CCW. We welcome the suggestion of the IPRAW report to distinguish control-by-design and control-in-use—acknowledging that ultimate responsibility for the use of force lies in the specific context of its use.

As a scientific and scholarly group, our focus is on how we can make control effective and ensure that operators, commanders and planners are making clear judgements about the validity of every attack at the time of that attack.

To do this we need to move away from blanket terms and examine in detail how humans interact with automated machinery. As we have pointed out before, there has been more than 30 years of scientific research on human supervisory control of machinery and more than 100 years of research on the psychology of human reasoning. Ignoring the science for sake of expediency could lead us down a path to a humanitarian disaster.

The scientific approach is not mutually exclusive to an examination of the military control of weapons and the many lessons to be learned for current methods. Indeed, we applaud the UK’s paper on human control in 2018 and that of the Netherlands and others this year.  We may not agree with all of the detail, but it is what we have urged all of the high contracting parties to bring to the table.

This combination of work can help us to design human-machine interfaces that allow weapons to be controlled in a manner that is fully compliant with international law and the principle of humanity.

First, there should be a focus on what the human operator MUST do in the targeting cycle. This is control by use which is governed by targeting rules under International Humanitarian Law and International Human Rights Law, which were well articulated by the ICRC in their statement this morning. Further, international law rules that apply after the use of weapons – such as those that relate to human responsibility – must be satisfied.

Second, the design of weapon systems must render them INCAPABLE of operating without meaningful human control.  This is control by design, which is governed by international weapons law. In terms of international weapons law, if the weapon system, by its design, is incapable of being sufficiently controlled in terms of the law, then such a weapon should be prohibited.

We need further discussion of the details of human-machine interfaces, the distribution of responsibility in the targeting cycle, and how their design can ensure IHL and IHRL compliance. Such details need not be the substance of a treaty, and we must resist being caught up in the weeds of process. We support German’s goal of finding a shared understanding of the principles of human control that apply to all weapons systems now and in the future, regardless of context, planning or process. This is not different from the normal processes that operate in science. One of the goals of science is to reduce the complexity of the world to simple theories or principles that capture all of the experimental data. In other words, we create abstractions of the details that are firmly coupled with and informed by the details. As Einstein once said, explanations should be a simple as possible but no simpler. “Human in the loop” and its variants fall under the too simple category. Detailed accounts of every weapon type and how it is controlled in every context is far too complex.

Let me give you an example of an abstraction with three conditions that could make a good starting point for discussions on the control of weapons systems. I have said this before but clearly there is no prohibition on repeating yourself in this room.

1. a human commander (or operator) will have full contextual and
   situational awareness of the target area for each and every attack and be able
   to perceive and react to any change or unanticipated situations that may have
   arisen since planning the attack.
2. there will be active cognitive participation in every attack with
   sufficient time for deliberation on the nature of any target, its significance
   in terms of the necessity and appropriateness of attack, and likely incidental
   and possible accidental effects of the attack and
3. there will be a means for the rapid suspension or abortion of
   every attack.

These are general principles that could provide a starting point for discussion by states in the context of negotiating a legally binding treaty that clearly articulates the legal obligations of human control.

Author information

Peter Asaro

Dr. Peter Asaro is a philosopher of science, technology and media. His work examines the interfaces between social relations, human minds and bodies, artificial intelligence and robotics, and digital media. His current research focuses on the social, cultural, political, legal and ethical dimensions of military robotics and UAV drones, from a perspective that combines media theory with science and technology studies. He has written widely-cited papers on lethal robotics from the perspective of just war theory and human rights. Dr. Asaro's research also examines agency and autonomy, liability and punishment, and privacy and surveillance as it applies to consumer robots, industrial automation, smart buildings, and autonomous vehicles. His research has been published in international peer reviewed journals and edited volumes, and he is currently writing a book that interrogates the intersections between military robotics, interface design practices, and social and ethical issues. Dr. Asaro has held research positions at the Center for Cultural Analysis at Rutgers University, the HUMlab of Umeå University in Sweden, and the Austrian Academy of Sciences in Vienna. He has also developed technologies in the areas of virtual reality, data visualization and sonification, human-computer interaction, computer-supported cooperative work, artificial intelligence, machine learning, robot vision, and neuromorphic robotics at the National Center for Supercomputer Applications (NCSA), the Beckman Institute for Advanced Science and Technology, and Iguana Robotics, Inc., and was involved in the design of the natural language interface for the Wolfram|Alpha computational knowledge engine (winner of the 2010 SXSW Web Interactive Award for Technical Achievement), for Wolfram Research. He is currently working on an Oral History of Robotics project that is funded by the IEEE Robotics and Automation Society and the National Endowment for the Humanities Office of Digital Humanities. Dr. Asaro received his PhD in the History, Philosophy and Sociology of Science from the University of Illinois at Urbana-Champaign, where he also earned a Master of Arts from the Department of Philosophy, and a Master of Computer Science from the Department of Computer Science.

Mr. President, representatives of nations, members of civil society,

During the past 5 years, at the Convention on Conventional Weapons, we have seen a greater understanding of the problems and challenges posed by autonomous weapon systems. The ICRAC is satisfied with the general consensus on the need to retain human control over these systems, in particular, on the critical functions of selection and elimination of objectives. Therefore, we believe that the time has come to establish binding legal mechanisms that restrict the use of autonomous weaponry, underlining the importance of human judgment in critical decisions.

During our participation in this convention, we have generated a large number of scientific articles, books and reports that emphasize three main classes of risk.

First, this type of weapons can not guarantee compliance with international humanitarian law. We should not give a blank check to future technology. With the large-scale commercialization of AI it is true that we are observing a great innovation in areas that are beneficial for humanity, but at the same time we are witnessing the appearance of many problems with biases in decision and facial recognition algorithms that can be dramatic if they are applied in a warlike context.

If nations invest based on techno-scientific speculations, we believe that it will be practically impossible to return to the starting position when the new typologies of conflict that announce these weapons materialize. We urge states to consider the veracity of current technology and its limitations in the critical selection of objectives.

Second, there are considerable moral values ​​at risk. No machine, computer or algorithm is capable of recognizing a human being as such, nor can he respect it as a human being with rights and dignity. He only observes it as a bit of information. A machine, without intuition, without ethics or morals, can not even understand what it means to be in a state of war, much less what it means to end a human life.

Decisions to end human life must be made by humans and in a non-arbitrary way to be justified. In addition, we must not confuse the fact that humans develop computer programs with the objective that the calculated results of these programs constitute human decisions. While responsibility for the deployment of lethal force is a necessary condition for compliance with minimum ethical standards in armed conflict, that responsibility alone is not enough, also requiring recognition of the human, of its dignity, and the reflection on the value of life and the justification of the use of violent force.

Third, autonomous weapons systems represent a great danger to global security. The threshold for the application of military force will be reduced and the likelihood of conflict will increase. We are concerned that the human control mechanisms established and controlled for double verification and reconsideration, function as security boxes or switches and can be easily disconnected. This, in combination with unpredictable algorithmic interactions and unpredictable results, will increase the instability of the conflict. In addition, the development and use of autonomous weapons by some States unilaterally will provide strong incentives for their proliferation, including their use by actors who are not responsible to the legal frameworks governing the use of force. Do we really need this new competitive arms race?

From the ICRAC as well as from other organizations involved in the Stop Killer Robots Campaign, representing a large part of international civil society, we urge the Convention to lay the foundations for the elaboration of an international treaty whose main objective is to prohibit preventive way autonomous weapons in clear application of the precautionary principle.

***

Señor presidente, representantes de las naciones, miembros de la sociedad civil,

Durante los últimos 5 años en la Convención de armas Convencionales, hemos visto una mayor comprensión de los problemas y desafíos planteados por los sistemas de armamento autónomo. El ICRAC está satisfecho con el consenso general sobre la necesidad de retener el control humano sobre estos sistemas de armamento, en particular, sobre las funciones críticas de selección y eliminación de objetivos. Por ello consideramos que ha llegado el momento de establecer unos mecanismos legales vinculantes que restrinjan el uso de armamento autónomo subrayando la importancia del juicio humano en decisiones críticas.

Durante nuestra participación en esta convención, hemos generado un gran número artículos científicos, libros e informes que enfatizan tres clases principales de riesgo.

Primero, esta tipología de armas no puede garantizar el cumplimiento del derecho internacional humanitario. No debemos dar un cheque en blanco a la tecnología futura. Con la comercialización a gran escala de la IA es cierto que estamos observando una gran innovación en ámbitos benéficos para la humanidad, pero al mismo tiempo estamos comprobando la aparición de muchos problemas con sesgos en los algoritmos de decisión y de reconocimiento facial que pueden ser dramáticos si se aplican en un contexto bélico.

Si las naciones invierten en base a especulaciones tecnocientíficas, creemos que será prácticamente imposible volver a la posición de partida cuando las nuevas tipologías de conflicto que anuncian estas armas, se materialicen. Instamos a los estados a considerar la veracidad de la tecnología actual y sus limitaciones en la selección crítica de objetivos legítimos.

Segundo, hay considerables valores morales en riesgo. Ninguna máquina, computadora o algoritmo es capaz de reconocer a un ser humano como tal, ni puede respetarlo como un ser con derechos y dignidad humana. Solo lo observa como un bit de información. Una máquina, sin intuición, sin ética ni moral, ni siquiera puede entender lo que significa estar en estado de guerra, y mucho menos lo que significa terminar con una vida humana.

Las decisiones para acabar con la vida humana deben ser tomadas por los humanos y de forma no arbitraria para ser justificadas. Además, no debemos confundir el hecho de que los humanos desarrollan programas informáticos con el objetivo que los resultados calculados de esos programas constituyan decisiones humanas. Si bien la responsabilidad por el despliegue de la fuerza letal es una condición necesaria para el cumplimiento de los estándares mínimos éticos en el conflicto armado, esa responsabilidad por sí sola no es suficiente, requiriendo también el reconocimiento de lo humano, de su dignidad, y la reflexión sobre el valor de la vida y la justificación del uso de la fuerza violenta.

En tercer lugar, los sistemas de armas autónomos representan un gran peligro para la seguridad global. El umbral para la aplicación de la fuerza militar se reducirá y la probabilidad de conflicto aumentará. Nos preocupa que los mecanismos de control humano establecidos y controlados encaminados a una doble verificación y reconsideración, funcionen como cajas de seguridad o interruptores y puedan ser fácilmente desconectados. Esto, en combinación con interacciones de algoritmos imprevisibles y resultados impredecibles, aumentará la inestabilidad del conflicto. Además, el desarrollo y uso de armas autónomas por parte de algunos Estados proporcionará fuertes incentivos para su proliferación, incluido su uso por parte de actores que no son responsables ante los marcos legales que rigen el uso de la fuerza. ¿Realmente necesitamos esta nueva carrera competitiva de armamentos?

Desde el ICRAC así como desde otras organizaciones involucradas en la Campaña Stop Killer Robots, en representación de una gran parte de la sociedad civil internacional, instamos a la Convención a sentar las bases para la elaboración de un tratado internacional que tenga como objetivo principal prohibir de manera preventiva las armas autónomas en aplicación clara del principio de precaución.

Author information

Frank Sauer

Mr Chairman,

Over the last 5 years at the CCW we have seen an increased understanding of the issues and challenges posed by autonomous weapons systems. ICRAC is pleased with the general consensus that we must retain human control over weapons systems, in particular, over the critical functions of selecting and killing targets. During our time at the CCW, we have produced many scientific papers and reports emphasising three major classes of risk.

First, we do not believe that IHL compliance can be guaranteed with autonomous weapons systems. Some argue that the technology will be able to comply with IHL in the future. But there is absolutely no evidence for that. We must not rely on hopeware and speculations about future technology. With the mass scale commercialisation of AI we are seeing great innovation but we are also seeing the emergence of many problems with bias in decision algorithms and face recognition (see my new ICRC blog post for more on this). If nations invest heavily on the basis of technical speculations, we believe that it will be difficult to put the toothpaste back in the tube when the humanitarian crises begin to emerge. We urge states to look at the plausibility of the current technology and how it falls short in the critical function of selecting legitimate targets.

Second, there are considerable moral values at risk. No machine, computer or algorithm is capable of recognizing a human as a human being, nor can it respect the human as a being with human rights and human dignity.  A machine cannot even understand what it means to be in a state of war, much less what it means to have, or to end a human life. Decisions to end human life must be made by humans in order to be justified.  Further, we should not mistake the fact that humans write computer programs to imply that the calculated results of those programs constitute human decisions.  While accountability for the deployment of lethal force is a necessary condition for moral responsibility in war, accountability alone is not sufficient for moral responsibility.  This also requires the recognition of the human, respect for the human right to life and dignity, and reflection upon the value of life and the justification for the use of violent force.

Third, Autonomous Weapons Systems pose great dangers to global security. The threshold for applying military force will be lowered and the likelihood of conflict will go up. We are concerned that tried and tested human control mechanisms for double checking and reconsidering, with humans functioning as fail-safes or circuit-breakers, would be discontinued. This, in combination with unforeseeable algorithm interactions and their unpredictable outcomes, increases crisis instability. In addition, the development and use of Autonomous weapons by some States will provide strong incentives for their proliferation, including their use by actors not accountable to legal frameworks governing the use of force. Do we really need a new arms race?

Finally, we urge that nations urgently move towards negotiations for a legally binding instrument in further deliberations next year. I am going off script here but look – come on – and no offence intended – but I am a scientist and not a diplomat so in plain speech there are those here who have an interest in slowing down the move towards a ban while they quickly continue to develop the weapons. Don’t be fooled or bullied by these tactics or the mudslide of refining definitions. We ask you – please – get on with ridding us of these morally reprehensible weapons before it is too late.

Thank you, Mr Chairman

Author information

Frank Sauer

Thank you, Mr Chairperson,

The International Committee for Robot Arms Control is pleased to see states move away from the use of broad, brush-stroke terms such as in-the-loop, on-the-loop, the wider loop, human oversight, and appropriate human judgement. We agree with the working paper from Estonia and Finland that complex definitions of autonomy and autonomous weapons systems is moving us in the wrong direction. As scientists we believe, following Einstein, that definitions should be as simple as possible but no simpler. In that, we applaud the approach of the ICRC, that focus should be on the critical functions of target selection and the application of violent force. This counters concerns that a prohibition of autonomous weapons systems (AWS) would impact on innovation in other civilian and non-lethal military applications.

ICRAC holds that the way forward is to focus on the meaningful human control of weapons systems. For human control to be meaningful we need to examine how humans interact with machines and understand the types of human-machine biases that can occur in the selection of legitimate targets. Lessons should be learned from 30 years of research on human supervisory control of machinery and more than 100 years of research on the psychology of human reasoning. A combination of this work can help us to design human-machine interfaces that allow weapons to be controlled in a manner that is fully compliant with international law and the principles of humanity.

First, there should be a focus on what the human operator MUST do in the targeting cycle. This is control in use which is governed by targeting rules under International Humanitarian Law and International Human Rights Law. Further, international law rules that apply after the use of weapons – such as those that relate to human responsibility – must be satisfied.

Second, the design of weapon systems must render them INCAPABLE of operating without meaningful human control.  This is control by design, which is governed by international weapons law. In terms of international weapons law, if the weapon system, by its design, is incapable of being sufficiently controlled, then such a weapon is illegal per se. Systems MUST be designed to ensure human responsibility and accountability.

Ideally the following three conditions should be followed for the control of weapons systems:

1. a human commander (or operator) will have full contextual and situational awareness of the target area for each and every attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack.
2. there will be active cognitive participation in every attack with sufficient time for deliberation on the nature of any target, its significance in terms of the necessity and appropriateness of attack, and likely incidental and possible accidental effects of the attack and
3. there will be a means for the rapid suspension or abortion of every attack.

For further details please see our guidelines for the human control of weapons systems from the April meeting this year.

While systems must be designed to ensure safety and responsibility, we should not mistake the review of weapons and good design as itself a form of human control. The responsibility to make decisions of life and death cannot be delegated to machines, nor to the review- or design process of those machines.

Thank you Mr Chairperson

Author information

Frank Sauer

International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Prof. Noel Sharkey, on 11 April 2018

Mr Chairperson,

We have been very pleased with this morning’s session as states begin to contemplate a move towards policies on the human control of weapons systems. On a pedantic note: we cannot talk about the meaningful human control of LAWS as that would make them no longer an autonomous weapon.

In the view of ICRAC, the control of weapons systems is more nuanced than can be captured by terms such as in-the-loop, on-the-loop, the broader loop, looping-the loop, human oversight, and appropriate human judgement. In this way we agree strongly with the statement made by Brazil and several others in this session who believe that the devil is in the detail.

For human control to be meaningful we need to examine how humans interact with machines and understand the types of human-machine biases that can occur in the selection of legitimate targets. Lessons should be learned from 30 years of research on human supervisory control of machinery xn--yxadbbg milf and more than 100 years of research on the psychology of human reasoning. This combination of this work can help us to design human-machine interfaces that allow weapons to be controlled in a manner that is fully compliant with international law and the principle of humanity.

First, there should be a focus on what the human operator MUST do in the targeting cycle. This is control by use which is governed by targeting rules under International Humanitarian Law and International Human Rights Law. Further, international law rules that apply after the use of weapons – such as those that relate to human responsibility – must be satisfied.

Second, the design of weapon systems must render them INCAPABLE of operating without meaningful human control.  This is control by design, which is governed by international weapons law. In terms of international weapons law, if the weapon system, by its design, is incapable of being sufficiently controlled in terms of the law, then such a weapon is illegal per se.

Ideally the following three conditions should be followed for the control of weapons systems:

1. a human commander (or operator) will have full contextual and situational awareness of the target area for each and every attack and is able to perceive and react to any change or unanticipated situations that may have arisen since planning the attack.
2. there will be active cognitive participation in every attack with sufficient time for deliberation on the nature of any target, its significance in terms of the necessity and appropriateness of attack, and likely incidental and possible accidental effects of the attack and
3. there will be a means for the rapid suspension or abortion of every attack.

Thank you Mr Chairperson

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Prof. Noel Sharkey, on 10 April 2018

Mr. Chairperson,

There have been very useful and interesting discussions this morning.

I speak here as chair of an academic NGO: the International Committee for Robot Arms Control (ICRAC) and as a member of the scientific community in the field of Artificial Intelligence and Robotics with specialty in Machine Learning.

We stress again that it would be confusing to broaden the discussion of LAWS into issues about Artificial Intelligence or weapons with emerging intelligence. By chasing definitions of LAWS down the rabbit hole of AI, we remove ourselves from the key issues that need to be urgently discussed here. The definition extracted from ICRC, and echoed by a number of states this morning, is concerned with weapons that have autonomy in the critical functions of target selection and the application of force. This is sufficient for our definitional purposes here: decisions about target selection and the application of force are delegated to a machine. Let me highlight that it does not matter what techniques or computing methods are used to create autonomy in these critical functions.

What is important here are questions about the nature of human control required and acceptable to ensure compliance with international law. It is key that we get this right. You can read more about this in ICRAC’s new working paper Guidelines for the Human Control of Weapons Systems that will be delivered at Wednesday’s side event. We support those states who have stated that the focus of this meeting should be on human control of weapons systems and human-machine interaction. In this way we can make real progress this week and protect our future no matter what the technological developments.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).

International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Dr Thompson Chengeta, on 9 April 2018

Mr. Chairperson,

I speak on behalf of the International Committee for Robot Arms Control [ICRAC], a founding member of the Campaign to Stop Killer Robots. Ambassador Gill, we thank you for your important work. Mr Chairperson, we are going to focus here on four points:

FIRST, a ban on LAWS will have no negative impact on the development of socially beneficial uses of autonomy, robotics or artificial intelligence. In fact, such a ban will direct more resources and specialists to work on humanitarian and beneficial applications.

SECOND, human control of weapon systems is a critical key component of the present discussions. It does not matter what name or term is used to describe human control, what is imperative is that we make sure that human control is consistent with applicable legal, ethical and moral standards.

THIRD, human input in the making of judgements to use violent force is at the centre of legal, ethics and moral standards pertaining to human responsibility for use of such force. No matter how attractive, if a proposed definition of human control does not resolve the accountability gap challenge, then such a proposal is legally inadequate. To that end, States should ask the question: What is the Legally Required Level of Human Control at each “touch point” in the human-machine interaction chain? At every step in the development, deployment, targeting and use of a weapon system, there is an obligation to ensure that the system is both capable of being used in compliance with applicable legal norms.

FOURTH, Poland and ICRC Working Papers’ emphasis on ethics and reassertion of the Principle of Non-Delegation of the Authority to Kill to non-human mechanisms is worth noting. Dictates of public conscience must always take precedence over any short-term advantage that might be gained from autonomous technologies. Furthermore, respect for human rights and human dignity, even within armed conflict, is a moral imperative recognized by the UN and the CCW. ICRAC reiterates the spirit of the Martens Clause—that morality can provide a strong basis for new law.

Finally, human control over critical functions of weapon systems and a ban on fully autonomous weapon systems are two sides of the same coin. States are urged to focus on the requirement of human control rather than technical definitions of autonomy. Further, States must move towards negotiation of a legally binding instrument on this issue.

Mr Chairperson, I thank you.

Author information

Peter Asaro

Dr. Peter Asaro is a philosopher of science, technology and media. His work examines the interfaces between social relations, human minds and bodies, artificial intelligence and robotics, and digital media. His current research focuses on the social, cultural, political, legal and ethical dimensions of military robotics and UAV drones, from a perspective that combines media theory with science and technology studies. He has written widely-cited papers on lethal robotics from the perspective of just war theory and human rights. Dr. Asaro's research also examines agency and autonomy, liability and punishment, and privacy and surveillance as it applies to consumer robots, industrial automation, smart buildings, and autonomous vehicles. His research has been published in international peer reviewed journals and edited volumes, and he is currently writing a book that interrogates the intersections between military robotics, interface design practices, and social and ethical issues. Dr. Asaro has held research positions at the Center for Cultural Analysis at Rutgers University, the HUMlab of Umeå University in Sweden, and the Austrian Academy of Sciences in Vienna. He has also developed technologies in the areas of virtual reality, data visualization and sonification, human-computer interaction, computer-supported cooperative work, artificial intelligence, machine learning, robot vision, and neuromorphic robotics at the National Center for Supercomputer Applications (NCSA), the Beckman Institute for Advanced Science and Technology, and Iguana Robotics, Inc., and was involved in the design of the natural language interface for the Wolfram|Alpha computational knowledge engine (winner of the 2010 SXSW Web Interactive Award for Technical Achievement), for Wolfram Research. He is currently working on an Oral History of Robotics project that is funded by the IEEE Robotics and Automation Society and the National Endowment for the Humanities Office of Digital Humanities. Dr. Asaro received his PhD in the History, Philosophy and Sociology of Science from the University of Illinois at Urbana-Champaign, where he also earned a Master of Arts from the Department of Philosophy, and a Master of Computer Science from the Department of Computer Science.

In the lead up to the next meeting of the CCW’s Group of Governmental Experts at the United Nations April 9-13th in Geneva, the UN’s Institute for Disarmament Research has issued a briefing paper titled The Weaponization of Increasingly Autonomous Technologies: Artificial Intelligence. Designated a primer for CCW delegates, the paper lists no authors, but a special acknowledgement to Paul Scharre, Senior Fellow and Director of the Technology and National Security Program at the Center for a New American Security, suggests that the viewpoints of the Washington, D.C.-based CNAS are well represented.

Surprisingly for a document positioning itself as “an introductory primer for non-technical audiences on the current state of AI and machine learning, designed to support the international discussions on the weaponization of increasingly autonomous technologies” (pp. 1-2), the paper opens with a series of assertions regarding “rapid advances” in the field of AI. The evidence offered is the case of Google/Alphabet affiliate Deep Mind’s AlphaGo Zero, announced in December 2017 (“only a few weeks after the November 2017 GGE”) as having achieved better-than-human competency at (simulations of) the game of Go:

Although AlphaGo Zero does not have direct military applications, it suggests that current AI technology can be used to solve narrowly defined problems provided that there is a clear goal, the environment is sufficiently constrained, and interactions can be simulated so that computers can learn over time (p.1).

The requirements listed – a clear (read computationally specifiable) goal, within a constrained environment that can be effectively simulated – might be underscored as cautionary qualifications on claims for AI’s applicability to military operations. The tone of these opening paragraphs suggests, however, that these developments are game-changers for the GGE debate.

The paper’s first section, titled ‘What is artificial intelligence,’ opens with the tautological statement that “Artificial intelligence is the field of study devoted to making machines intelligent” (p. 2). A more demystifying description might say, for example, that AI is the field of study devoted to developing computational technologies that automate aspects of human activity conventionally understood to require intelligence. While the authors observe that as systems become more established they shift from characterizations of “intelligence” to more mundane designations like “automation” or “computation,” they suggest that rather than the result of demystification this is itself somehow an effect of the field’s advancement. One implication of this logic is that the ever-receding horizon of machine intelligence should be understood not as a marker of the technology’s limits, but of its success.

We begin to get a more concrete sense of the field in the section titled ‘Machine learning,’ which outlines the latter’s various forms. Even here, however, issues central to the deliberations of the GGE are passed over. For example, in the statement that “[r]ather than follow a proscribed [sic] set of if–then rules for how to behave in a given situation, learning machines are given a goal to optimize – for example, winning at the game of chess” (p. 2) the example is not chosen at random, but rather is illustrative of the unstated requirement that the ‘goal’ be computationally specifiable. The authors do helpfully explain that “[s]upervised learning is a machine learning technique that makes use of labelled training data” (my emphasis, p. 3), but the contrast with “unsupervised learning,” or “learning from unlabelled data based on the identification of patterns” fails to emphasize the role of the human in assessing the relevance and significance of patterns identified. In the case of reinforcement learning “in which an agent learns by interacting with its environment,” the (unmarked) examples are again from strategy games in which, implicitly, the range of agent/environment interactions are sufficiently constrained. And finally, the section on ‘Deep learning’ helpfully emphasizes that so called neural networks rely either on very large data sets and extensive labours of human classification (for example, the labeling of images to enable their ‘recognition’), or on domains amenable to the generation of synthetic ‘data’ through simulation (for example, in the case of strategy games like Go). Progress in AI, in sum, has been tied to growth in the availability of large data sets and associated computational power, along with increasingly sophisticated algorithms within highly constrained domains of application.

Yet in spite of these qualifications, the concluding sections of the paper return to the prospects for increasing machine autonomy:

Intelligence is a system’s ability to determine the best course of action to achieve its goals. Autonomy is the freedom a system has in accomplishing its goals. Greater autonomy means more freedom, either in the form of undertaking more tasks, with less supervision, for longer periods in space and time, or in more complex environments … Intelligence is related to autonomy in that more intelligent systems are capable of deciding the best course of action for more difficult tasks in more complex environments. This means that more intelligent systems could be granted more autonomy and would be capable of successfully accomplishing their goals (p. 5, original emphasis).

The logical leap exemplified in this passage’s closing sentence is at the crux of the debate regarding lethal autonomous weapon systems. The authors of the primer concede that “all AI systems in existence today fall under the broad category of “narrow AI”. This means that their intelligence is limited to a single task or domain of knowledge” (p. 5). They acknowledge as well that “many advance [sic] AI and machine learning methods suffer from problems of predictability, explainability, verifiability, and reliability” (p. 8). These are precisely the concerns that have been consistently voiced, over the past five meetings of the CCW, by those states and civil society organizations calling for a ban on autonomous weapon systems. And yet the primer takes us back, once again, to a starting point premised on general claims for the field of AI’s “rapid advance,” rather than careful articulation of its limits. Is it not the latter that are most relevant to the questions that the GGE is convened to consider?

The UNIDIR primer comes at the same time that the United States has issued a new position paper in advance of the CCW titled ‘Humanitarian benefits of emerging technologies in the area of lethal autonomous weapon systems’ (CCW/GGE.1/2018/WP.4). While the US has taken a cautionary position in relation to lethal autonomous weapon systems in past meetings, asserting the efficacy of already-existing weapons reviews to address the concerns raised by other member states and civil society groups, it now appears to be moving in the direction of active promotion of LAWS on the grounds of promised increases in precision and greater accuracy of targeting, with associated limits on unintended civilian casualties – promises that have been extensively critiqued at previous CCW meetings. Taken together, the UNIDIR primer and the US working paper suggest that, rather than moving forward from the debates of the past five years, the 2018 meetings of the CCW will require renewed efforts to articulate the limits of AI, and their relevance to the CCW’s charter to enact Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects.

Author information

Lucy Suchman

Lucy Suchman is a Professor of the Anthropology of Science and Technology at Lancaster University in the UK. Before taking up her present post she was a Principal Scientist at Xerox's Palo Alto Research Center (PARC), where she spent twenty years as a researcher. During this period she became widely recognized for her critical engagement with artificial intelligence (AI), as well as her contributions to a deeper understanding of both the essential connections and the profound differences between humans and machines.

Actually, Markoff does clearly say that the Pentagon is falling behind Google. In fact, the entire article seems to suggest that the military has neglected land robots, which is simply untrue, as I explain in my response, posted on my own blog.

I felt compelled to respond because, just days before Markoff’s article was posted (it was also included in the Sept. 24 New York print edition), I had published in the Bulletin of the Atomic Scientists an analysis of US policy for autonomous weapons which shows that it is actually an aggressive, “full speed ahead” policy. I don’t know if Markoff read my piece before he wrote his, but if you put them side by side and step back until you can only read the headlines, his looks like a rebuttal of mine, even though he is far too good a writer to say something that is actually wrong, and therefore he doesn’t actually contradict anything I wrote.

Author information

Mark Gubrud