ICRAC Closing Statement to the Convention on Certain Conventional Weapons Informal Meeting of Experts at the United Nations in Geneva

Thank you Mr. Chairperson,

Distinguished Delegates,

The International Committee for Robot Arms Control – or ICRAC as we are known – would like to commend the German Ambassador for convening this meeting with such distinguished experts on autonomous weapons systems. ICRAC would like to express appreciation to the States Parties and distinguished experts for engaging in a fruitful exchange of views. ICRAC has called for an international process on autonomous weapons systems since 2009 and is delighted to now be a part of it.

ICRAC is particularly grateful to the Chair for assembling a group of experts representing greater gender diversity. We hope that this is indicative of a new trend of allowing previously excluded voices – of women and experts from the Global South – to be heard in this forum and others dealing with international peace and security.

ICRAC urges States Parties to continue their work in this forum and others, starting with an open-ended Group of Governmental Experts and moving to substantive negotiations on a preemptive prohibition of all weapons systems that lack meaningful human control over all individual attacks. Increased transparency and better weapons reviews, while crucial, are not enough. We would welcome a more transparent discussion of how current semi-autonomous weapons systems, automated defense systems, and remotely-operated weapons systems are kept under meaningful human control. While transparency is good as a general principle, it is not on its own a sufficient means to regulate autonomous weapons, especially given the challenges to its effective realization.

We welcome the views of Professor van den Hoven on value sensitive design, and the need to include explicit consideration of moral reasoning and human values in the design process of all technologies, including weapons systems.  However, the responsible design of technologies is not in itself a sufficient means to mitigate the multiple and grave risks posed by autonomous weapons systems.  Engineers operate in a policy and legal environment that is defined by states. As such, it is incumbent upon the States Parties to clearly communicate the requirement that all weapons must be kept under meaningful human control though a binding instrument.

Autonomous weapons systems would threaten international peace and security. As outlined in ICRAC’s leaflet, distributed to States Parties this week, ICRAC experts fear the threats of proliferation, arms races and lowered thresholds for armed violence.  ICRAC fears that the unpredictable interaction between opposing complex autonomous weapons systems, coupled with increasing speed, could spiral out of control and trigger accidental conflict and/or indiscriminate civilian harm.

Autonomous weapons systems raise legal and ethical concerns. Peer-reviewed research by ICRAC experts has shown for the foreseeable future, autonomous weapon systems could not comply with international humanitarian law and international human rights law. A precautionary approach is needed.

A preemptive ban would honor the Marten’s Clause, following the guidance of the principle of humanity and the dictates of public conscience in developing new law.  The delegation of violence to a machine – whether lethal or less lethal – is a violation of human dignity. Last month, a Model UN conference of 2,500 undergraduate students from around the world – meeting in the UN General Assembly Room in New York – were commended by UN Secretary General Ban Ki-moon for passing resolutions calling for a ban on autonomous weapons systems. ICRAC urges states to follow this ethical leadership shown by the world’s youth, as well as Nobel Laureates, clergy and faith-based organizations, concerned scientists, ethicists and the civil society Campaign to Stop Killer Robots.

As a result, ICRAC calls on States Parties to:

• Develop national policies in favor of a ban on autonomous weapons systems,
• Support an international mandate leading to substantive negotiations on a preemptive prohibition, either by a CCW Sixth Protocol or other means, and
• Ensure that discussions on autonomous weapons systems are open and inclusive especially of women, experts from the Global South and civil society.

ICRAC looks forward to the ongoing process on autonomous weapons systems and offers the expertise of its members – scientists, technologists, academics, lawyers and policy experts – to the CCW’s States Parties.

Thank you, Mr. Chairperson.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

Thank you Mr. Chair.

In response to China’s question, a number of security experts have pointed to the danger of mass attack by conventional weapons on strategically important installations.  Autonomous weapons would open new modes for such attacks, and would heighten the risk of preemptive attacks up to and including nuclear weapons.

I would like comment on the survey results presented by Professor Horowitz, and the conclusions he asks us to draw from them.

You are correct that public opinion is not equivalent to public conscience. Unfortunately, we know all too well that the desires of the majority can often result in the injustices towards a minority. And while opinions may shift, the underlying moral principles upon which they are built do not, namely the moral foundations of human rights and human dignity, and respect for the law.

It is difficult to determine the exact causes for difference in the results between your survey and previous ones without knowing the methodological details of your survey. You claim that the differences you found in your results are due to the presentation of additional contextual information. Yet, the survey you are comparing it to, which was presented at a side event on Tuesday by the OpenRoboethics Initative, actually provided more situational and contextual information than your survey, and compared responses to more alternatives than your survey, including remote operated weapons such as drones, and questions of offensive and defensive uses.

More importantly, for public opinion to serve as an indicator of the public conscience, it is necessary to clearly demonstrate its relationship to underlying moral values.

In the survey conducted by the Open Roboethics Initiative there were separate questions investigating the reasons underlying the general opinions elicted from respondents.  Concurring somewhat with your findings, the strongest reason indicated for supporting the use of autonomous weapons was to save the lives of the soldiers of your own nation. Furthermore, the primary reason given by the overwhelming majorities of multiple nations for rejecting the use of autonomous weapons is that it was simply wrong for machines to make life and death decisions, followed by reasons such as the  fears of the risks of errors and failures, and the risks from the proliferation of these weapons.

It would thus seem that a reasonable explanation of your survey results is that you have conflated the desire to protect troops, and desire for self-defense with participants opinions on autonomous weapons.  Rather than offering informational context, the structure of the question has combined multiple factors and skewed the responses.

I would thus respectfully submit that in addition to public opinion polls, we should confer with the legal and ethical experts we heard from this morning and well as our own individual moral conscience.

Author information

Peter Asaro

Dr. Peter Asaro is a philosopher of science, technology and media. His work examines the interfaces between social relations, human minds and bodies, artificial intelligence and robotics, and digital media. His current research focuses on the social, cultural, political, legal and ethical dimensions of military robotics and UAV drones, from a perspective that combines media theory with science and technology studies. He has written widely-cited papers on lethal robotics from the perspective of just war theory and human rights. Dr. Asaro's research also examines agency and autonomy, liability and punishment, and privacy and surveillance as it applies to consumer robots, industrial automation, smart buildings, and autonomous vehicles. His research has been published in international peer reviewed journals and edited volumes, and he is currently writing a book that interrogates the intersections between military robotics, interface design practices, and social and ethical issues. Dr. Asaro has held research positions at the Center for Cultural Analysis at Rutgers University, the HUMlab of Umeå University in Sweden, and the Austrian Academy of Sciences in Vienna. He has also developed technologies in the areas of virtual reality, data visualization and sonification, human-computer interaction, computer-supported cooperative work, artificial intelligence, machine learning, robot vision, and neuromorphic robotics at the National Center for Supercomputer Applications (NCSA), the Beckman Institute for Advanced Science and Technology, and Iguana Robotics, Inc., and was involved in the design of the natural language interface for the Wolfram|Alpha computational knowledge engine (winner of the 2010 SXSW Web Interactive Award for Technical Achievement), for Wolfram Research. He is currently working on an Oral History of Robotics project that is funded by the IEEE Robotics and Automation Society and the National Endowment for the Humanities Office of Digital Humanities. Dr. Asaro received his PhD in the History, Philosophy and Sociology of Science from the University of Illinois at Urbana-Champaign, where he also earned a Master of Arts from the Department of Philosophy, and a Master of Computer Science from the Department of Computer Science.

Statement from the International Committee for Robot Arms Control: On Panel Challenges to IHL due to increasing degrees of autonomy.

This discussion has been directed almost entirely to considerations of law derived from the principle of jus in bello. We appear to be overlooking, or excluding, considerations of jus ad bellum that arise from the use of autonomous weapons systems. It is in this context that those considerations also typically discussed as matters of international peace and security may be considered to have implications under the law of armed conflict.

We are concerned about the destabilization and chaos that may be introduced into the international system by arms races and the appearance of new, unfamiliar threats. In addition, we are concerned as scientists, about what may happen when nations with an uneasy relationship field increasingly complex, autonomous systems in confrontation with one another.  We know that the interactions of such systems are unpredictable for two reasons.

The first is the inherent error-proneness of complex software even when it is engineered by a single co-operative team. The second is that, in reality, these interacting systems will have been developed by non-cooperating teams, who will do their utmost to maintain secrecy and to ensure that their systems will exploit every opportunity to prevail once hostilities are understood to have commenced or, perhaps, are believed to be imminent. Once hostilities have begun, it may become very difficult for humans to intervene and to reestablish peace, due to the high speed and complexity of events. Niether side would want to risk losing the battle once it had begun

Do these considerations have no implications for the legality of autonomous weapons? Can we consider a war that has been initiated as a result of needless political or military instability, or due to the unpredictable interactions of machines, or escalated out of human control due to the high speed and complexity of events, and not for any human moral or political cause, to be a just war?

Author information

altmann

Jürgen Altmann (PhD) is a physicist and peace researcher (retired) at TU Dortmund University, Germany. Since 1985 he has studied scientific-technical problems of disarmament. An experimental focus is automatic sensor systems for co-operative verification of disarmament and peace agreements and for IAEA safeguards for an underground final repository. The second focus is assessment of new military technologies and preventive arms control, including verification. Studies have dealt with “non-lethal” weapons, civilian and military technologies in aviation, military uses of microsystems technology and of nanotechnology, confidence and security building measures for cyber forces, armed uncrewed vehicles and autonomous weapon systems. He co-founded and chairs the German Research Association for Science, Disarmament and International Security (FONAS) and has authored book chapters on the relationship of natural science, armament and disarmament. He teaches a lecture "Physics and Technology of the Verification of Arms-Limitation Agreements".

International Committee for Robot Arms Control opening statement to the Convention on Conventional Weapons Meeting of Experts on lethal autonomous weapons systems, United Nations Geneva 13 April 2015

I am speaking on behalf of the International Committee for Robot Arms Control (or ICRAC as we are known), a founding NGO of the Campaign to Stop Killer Robots. We would very much like to thank Ambassador Biontino for preparations in chairing this second meeting of experts and for inviting our members to share their expertise. And we thank all of the States Parties for their participation.

ICRAC is an international association of scientists, technologists, lawyers, and policy experts committed to the peaceful use of robotics in the service of humanity and the regulation of robotic weapons.

ICRAC members have carried out research on various aspects of autonomous weapons systems and published their results in scientific journals as well as at conferences and in mass media.

We would like to stress that ICRAC experts are available and willing to provide technical expertise to the High Contracting Parties as they engage in discussions about autonomous weapons systems.

ICRAC urges the international community to seriously consider the prohibition of autonomous weapons systems in light of the pressing dangers they pose to global peace and security. We have produced a new leaflet on problems for global security posed by LAWS that is now available.

We fear that once they are developed, they will proliferate rapidly, and if deployed they may interact unpredictably and contribute to regional and global destabilization and arms races.

ICRAC urges nations to be guided by the principles of humanity in its deliberations and take into account considerations of human security, human rights, human dignity, humanitarian law and the public conscience. The Martens Clause reminds us that such moral principles are the basis for international law. Human judgment and meaningful human control over the use of violence must be made an explicit requirement in international policymaking on autonomous weapons.

ICRAC urges a broader discussion about the arming of autonomous systems beyond just lethal weapons, to include so-called “sub-lethal” or “less-than-lethal” weapons. These could still cause unnecessary suffering to humans. We urge nations to consider the human rights implications of the development and potential use of these weapons in any situation, including domestic policing, border control and internal law enforcement.

In conclusion, ICRAC encourages the CCW to move towards a preemptive ban on the development, production and use of autonomous weapons systems. ICRAC urges delegates to build consensus for negotiating a legally binding instrument to ban autonomous weapons systems.

Author information

Mark Gubrud

UN Secretary General Ban Ki-moon addresses National Model UN conference in the General Assembly Room, 26 March 2015. Photo: NMUN.

In less than two weeks, diplomats from around the world will gather at the United Nations in Geneva to discuss potential global regulations on “lethal autonomous weapons systems” that would be able to select and attack targets without direct human control.

But last week, at the National Model UN conference in New York, attended by some 2,500 undergraduate students from all over the world, a simulation of the UN General Assembly passed three resolutions calling for states to take action to prevent the threat of these “killer robots” to security, human rights and humanitarian law.

Addressing the closing ceremony of the conference, UN Secretary General Ban Ki-moon told students he was “energized by this dynamic gathering” and its “serious discussions” on “cutting-edge issues on the international agenda”, such as “lethal autonomous weapons systems.”

“You are not just leaders of the future – you can start to lead right now,” he told them, “now is the time for your generation to build human solidarity around the world.”

The NMUN NY resolutions defined lethal autonomous robots as “weapons that can select and attack targets independently – without meaningful human input or control”, suggested all countries immediate adopt a national moratorium on such weapons, and urged the negotiation of an international ban through an additional Protocol VI at the Convention on Certain Conventional Weapons (NMUN NY 2015A/GA1-1-1).

Model UN is a simulation of diplomacy, negotiation and decisionmaking by international organizations. Students play the role of diplomats from Member States of the UN and discuss issues at the top of the global policymaking agenda. NMUN NY is one of the biggest undergraduate Model UN conferences in the world.

The students assigned to simulate the General Assembly First Committee – which deals with issues of disarmament and international security – spent several months learning about their countries’ policy positions, the General Assembly and the politics of killer robots. (See for example, their background guide). After debate and drafting in the First Committee, the resolutions were passed by students representing the full plenary body in the actual General Assembly Room at the UN in New York.

The resolutions also called attention to the “work and expertise” of civil society, particularly the Campaign to Stop Killer Robots, Human Rights Watch and the International Committee for Robot Arms Control (ICRAC) (NMUN NY 2015A/GA1-1 -1, GA1-1-2 and GA1-1-3).

In a briefing, Dr. Matthew Bolton, Model UN advisor for Pace University New York City and member of ICRAC, told students at the conference that when new weapons technologies are not adequately addressed by existing regulations, the Marten’s Clause in international law requires states to be guided by “the principle of humanity and the dictates of public conscience.”

“Avoid the temptation to think this simulation is a meaningless game,” said Bolton, “A statement of strong concern from you could be considered an expression of public conscience – a challenge to policymakers in the real world to take action against killer robots.”

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

ICRAC welcomes the historic decision taken by nations to begin international discussions on how to address the challenges posed by fully autonomous weapons. The agreement marks the beginning of a process that the campaign believes should lead to an international ban on these weapons to ensure there will always be meaningful human control over targeting decisions and the use of violent force.

At 16:48 on Friday, 15 November 2013, at the United Nations in Geneva, states parties to the Convention on Conventional Weapons adopted a report containing a decision to convene on May 13-16, 2014 for their first meeting to discuss questions related to “lethal autonomous weapons systems” also known as fully autonomous weapons or “killer robots.” These weapons are at the beginning of their development, but technology is moving rapidly toward increasing autonomy.

“This is a very significant step forward for the International Committee for Robot Arms Control (ICRAC ),” said Professor Noel Sharkey, Chairman of ICRAC. “We are now on the first rung of the international ladder to fulfill our goal of stopping these morally obnoxious weapons from ever being deployed.”

ICRAC was formed in 2009 to initiate international discussion on autonomous weapons systems. It is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. It is a co-founder of the Campaign to Stop Killer Robots.

The Campaign to Stop Killer Robots believes that robotic weapons systems should not be making life and death decisions on the battlefield. That would be inherently wrong, morally and ethically. Fully autonomous weapons are likely to run afoul of international humanitarian law, and that there are serious technical, proliferation, societal, and other concerns that make a preemptive ban necessary.

“Law follows technology.  With robotic weapons, we have an rare opportunity to regulate a category of dangerous weapons before they are fully realized and the CCW is our best opportunity for regulation,” said Dave Akerson an ICRAC legal expert.

A total of 117 states are party to the Convention on Conventional Weapons, including nations known to be advanced in developing autonomous weapons systems: United States, China, Israel, Russia, South Korea, and United Kingdom. Adopted in 1980, this framework convention contains five protocols, including Protocol I prohibiting non-detectable fragments, Protocol III prohibiting the use of air-dropped incendiary weapons in populated areas, and Protocol IV, which preemptively banned blinding lasers.

“This is a momentous opportunity to get states on the record and behind a ban on fully autonomous offensive weapons,” said Heather Roff, an ICRAC philosopher. “If we can gain enough support, we might succeed in banning a technology before it actually harms innocent civilians.”

The agreement to begin work in the Convention on Conventional Weapons could lead to a future CCW Protocol VI prohibiting fully autonomous weapons.

ICRAC with the Campaign to Stop Killer Robots supports any action to urgently address fully autonomous weapons in any forum. The decision to begin work in the Convention on Conventional Weapons does not prevent work elsewhere, such as the Human Rights Council.

Since the topic was first discussed at the Human Rights Council on 30 May 2013, a total of 44 nations have spoken publicly on fully autonomous weapons since May: Algeria, Argentina, Australia, Austria, Belarus, Belgium, Brazil, Canada, China, Costa Rica, Cuba, Ecuador, Egypt, France, Germany, Ghana, Greece, Holy See, India, Indonesia, Iran, Ireland, Israel, Italy, Japan, Lithuania, Madagascar, Mexico, Morocco, Netherlands, New Zealand, Pakistan, Russia, Sierra Leone, Spain, South Africa, South Korea, Sweden, Switzerland, Turkey, Ukraine, United Kingdom, and United States. All nations that have spoken out have expressed interest and concern at the challenges and dangers posed by fully autonomous weapons.

Together with the Campaign to Stop Killer Robots, ICRAC urges nations to prepare for extensive and intensive work next year, both within the CCW and outside the CCW context.  We urge states to develop national policies, and to respond to the UN Special Rapporteur on Extrajudicial Executions’ call for national moratoria on fully autonomous weapons. We urge states to come back one year from now and agree to a new mandate to begin negotiations. The new process must be underscored by  a sense of urgency.

Peter Asaro, vice-chairman of ICRAC said “The actions of the CCW this week are a hopeful first step towards an international ban on autonomous weapons systems.’

Mathew Bolton delivered a statement on behalf ICRAC at the UN CCW meeting yesterday. As a group of experts we are prepared to help any nations with expert discussions of autonomous weapons systems and to help develop clear definitions for the language to be used in a treaty to ban them. Video footage of the statement, ICRAC’s first ever statement in an official diplomatic forum, is available here.

ICRAC recently coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

International Community Must Act Now to Ensure Combat Will Never Be Outsourced to “Killer Robots”! —

I am speaking on behalf of the International Committee for Robot Arms Control (ICRAC), a founding NGO member of the Campaign to Stop Killer Robots. Since its establishment in 2009, ICRAC has urged the international community to discuss the prohibition of fully autonomous weapons systems – “Killer Robots” – in light of the pressing dangers they pose to global peace and security, in addition to their alarming humanitarian implications for civilians threatened with armed violence.

ICRAC is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. As an indication of our concentration of expertise, over 80% of our members have doctoral or Juris Doctor degrees. As such, ICRAC is available and willing to provide technical expertise to the High Contracting Parties as they engage in further discussions about fully autonomous weapons systems.

ICRAC has coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

ICRAC urges High Contracting Parties to be guided by principles of humanity in its deliberations on existing and emerging weapons technologies –  taking into account considerations of human security, human rights, human dignity, humanitarian law and the public conscience, as well as the justified worries about robotic arms races and proliferation. This means meaningful human deliberation and control over the use of violence must remain the cornerstone of any eventual global policymaking on robotic weapons.

ICRAC welcomes the presentation of the Draft Mandate text enabling the Convention on Certain Conventional Weapons to convene expert meetings in 2014 to look at the challenges posed by fully autonomous weapons systems. ICRAC also welcomes the statements by more than 40 States which have expressed concerns about autonomous weapons systems. We believe the Convention on Certain Conventional Weapons represents a useful forum to begin discussions, though dialogue about autonomous weapons systems need not be confined to this body alone. [END OF ORAL VERSION OF STATEMENT]

In light of our concerns, ICRAC respectfully submits the following recommendations to improve and strengthen the proposed Mandate on autonomous weapons systems before the body:

1)      It is not enough to consider only “lethal” autonomous weapons. We urge the body to broaden the Mandate to include consideration of all autonomous weapons systems, given that:

a) Arming autonomous systems with so-called “sub-lethal” or “less-than-lethal” weapons could still cause unnecessary suffering, and

b) Autonomous weapons systems could be potentially destabilizing even if they are only programmed to attack materiel, particularly if such systems interacted with each other in unexpected ways.

2)      By covering only “emerging technologies”, the Mandate may discourage High Contracting Parties from considering the implications of existing precursor autonomous systems that nonetheless raise unsettling humanitarian questions.

3)      While we welcome an Informal Meeting, a Mandate for a Group of Governmental Experts would more clearly send a message of the seriousness of this matter to the High Contracting Parties.

4)      As experts in this field, we believe the legal and ethical complexities of autonomous weapons systems will be difficult to cover adequately in three days. Thus we suggest amending the Draft Mandate to allow for a five (5) day meeting.

Delivered on behalf of the International Committee for Robot Arms Control, by:

Matthew Bolton, MSc. (LSE), PhD (LSE).
Membership Secretary, International Committee for Robot Arms Control
Advisor, Article 36
Assistant Professor of Political Science, Dyson College of Arts and Sciences, Pace University New York City.

For a PDF version of this statement, click here.

Video footage of the oral statement at the UN in Geneva is available for download here.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).