Despite the challenges of COVID-19, 21-25 September the world’s governments will discuss at the UN in Geneva the humanitarian and security threat posed by “lethal autonomous weapons systems” – high-tech killer robots that could target people without meaningful human control over the use of violence.

But despite the media clichés that often accompany these discussions – journalists seem unable to resist references to Terminator and Skynet – killer robots are not some sci-fi fantasy Splitgate Arena. They are the result of a long trend in weapons development of using technology to disembody killing.

In my new book, Political Minefields (I.B. Tauris), I trace the history of efforts to avoid responsibility for violence by using remote and automated weapons like landmines, cluster munitions, armed drones and, now killer robots.

“Weapons developers are seeking to pervert the power of information and communications technology for deadly ends, taking humans entirely out of the decision to kill,” writes Jody Williams, who was awarded the 1997 Nobel Peace Prize along with the International Campaign to Ban Landmines (ICBL), in her foreword to my book. “In the autonomous, weaponized robot, they are essentially designing mines that actively seek out their targets, that can follow you, that can fly.”

From the WWII minefields of North Africa and US automated bombing of Laos to armed drones targeting makers of improvised explosive devices, military planners have tried, as US Vietnam War General William Westmoreland put it, to “replace wherever possible the man with the machine.”

But in my research, I’ve learned that the fever dream of algorithmic warfare never delivers on its promise of victory by remote control. People are too messy, unpredictable, clever, and tricky to be meet the assumptions programmed into military technology.

Allied troops just marched through the Nazi minefields at El Alamein, taking the casualties and repurposing the mines they found for their own uses. Vietnamese communist soldiers spoofed the various electronic detectors dropped from US warplanes onto their pathways through the jungle. They sent animals down the trail, placed bags of urine next to so-called “people sniffers”, and played tapes of vehicle noises next to microphones – prompting computerized bombers to unload explosives onto phantom guerillas.

As I have travelled in and around the world’s minefields and cluster munition strike zones, I have heard the story over and over again, in Afghanistan, Bosnia, Cambodia, Iraq, Laos and South Sudan. In each place, it is civilians who have borne the consequences of turning warfare over to automated devices. Decades after the soldier who placed and armed them, landmines in Afghanistan continue to maim children. Lao farmers still risk setting off unexploded cluster bomblets when they plough their rice fields.

The final chapter of my book highlights the far-sighted work of the International Committee for Robot Arms Control (ICRAC) and Campaign to Stop Killer Robots, which have sounded the alarm on the emerging militarization of artificial intelligence and robotics. They are not anti-technology Luddites. “It’s OK for a plane to fly itself,” Dr. Peter Asaro, co-founder of ICRAC and New School professor told me. “It’s not OK for a plane to decide who to shoot at.” We have, he says, “the right not to be killed by a machine.”

For the last six years, governments have gathered in Geneva to consider how to address lethal autonomous weapons systems, under the auspices of the Convention on Certain Conventional Weapons (CCW). Despite pressure from a wide range of countries and civil society, the big military powers have dragged their feet, abusing the rule of consensus decisionmaking, running down the clock to avoid constraining their high-tech weapons R&D.

But diplomatic patience of the majority of nations in the CCW is running out. A global poll found that 61% of people in 26 countries opposed killer robots. And UN Secretary-General António Guterres has directly called on governments “to ban these weapons, which are politically unacceptable and morally repugnant.”

Just as human reality is more complex than software programs, society is not the passive recipient of technological change. In writing my book, I have been inspired to meet people who have mobilized to clear up the minefields, provide support to survivors of cluster munitions and pressured diplomats to negotiate treaties banning inhumane weapons. We must following their example to demand new international law ensuring that the use of force remains under meaningful human control.

Matthew Bolton, ICRAC member and associate professor of political science at Pace University, is author of Political Minefields: The Struggle against Automated Killing. If you can’t find a copy in your local bookstore, it is available with a 35% discount from Bloomsbury with the code GLR TW5

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

ICRAC Closing Statement to the Convention on Certain Conventional Weapons Informal Meeting of Experts at the United Nations in Geneva

Thank you Mr. Chairperson,

Distinguished Delegates,

The International Committee for Robot Arms Control – or ICRAC as we are known – would like to commend the German Ambassador for convening this meeting with such distinguished experts on autonomous weapons systems. ICRAC would like to express appreciation to the States Parties and distinguished experts for engaging in a fruitful exchange of views. ICRAC has called for an international process on autonomous weapons systems since 2009 and is delighted to now be a part of it.

ICRAC is particularly grateful to the Chair for assembling a group of experts representing greater gender diversity. We hope that this is indicative of a new trend of allowing previously excluded voices – of women and experts from the Global South – to be heard in this forum and others dealing with international peace and security.

ICRAC urges States Parties to continue their work in this forum and others, starting with an open-ended Group of Governmental Experts and moving to substantive negotiations on a preemptive prohibition of all weapons systems that lack meaningful human control over all individual attacks. Increased transparency and better weapons reviews, while crucial, are not enough. We would welcome a more transparent discussion of how current semi-autonomous weapons systems, automated defense systems, and remotely-operated weapons systems are kept under meaningful human control. While transparency is good as a general principle, it is not on its own a sufficient means to regulate autonomous weapons, especially given the challenges to its effective realization.

We welcome the views of Professor van den Hoven on value sensitive design, and the need to include explicit consideration of moral reasoning and human values in the design process of all technologies, including weapons systems.  However, the responsible design of technologies is not in itself a sufficient means to mitigate the multiple and grave risks posed by autonomous weapons systems.  Engineers operate in a policy and legal environment that is defined by states. As such, it is incumbent upon the States Parties to clearly communicate the requirement that all weapons must be kept under meaningful human control though a binding instrument.

Autonomous weapons systems would threaten international peace and security. As outlined in ICRAC’s leaflet, distributed to States Parties this week, ICRAC experts fear the threats of proliferation, arms races and lowered thresholds for armed violence.  ICRAC fears that the unpredictable interaction between opposing complex autonomous weapons systems, coupled with increasing speed, could spiral out of control and trigger accidental conflict and/or indiscriminate civilian harm.

Autonomous weapons systems raise legal and ethical concerns. Peer-reviewed research by ICRAC experts has shown for the foreseeable future, autonomous weapon systems could not comply with international humanitarian law and international human rights law. A precautionary approach is needed.

A preemptive ban would honor the Marten’s Clause, following the guidance of the principle of humanity and the dictates of public conscience in developing new law.  The delegation of violence to a machine – whether lethal or less lethal – is a violation of human dignity. Last month, a Model UN conference of 2,500 undergraduate students from around the world – meeting in the UN General Assembly Room in New York – were commended by UN Secretary General Ban Ki-moon for passing resolutions calling for a ban on autonomous weapons systems. ICRAC urges states to follow this ethical leadership shown by the world’s youth, as well as Nobel Laureates, clergy and faith-based organizations, concerned scientists, ethicists and the civil society Campaign to Stop Killer Robots.

As a result, ICRAC calls on States Parties to:

• Develop national policies in favor of a ban on autonomous weapons systems,
• Support an international mandate leading to substantive negotiations on a preemptive prohibition, either by a CCW Sixth Protocol or other means, and
• Ensure that discussions on autonomous weapons systems are open and inclusive especially of women, experts from the Global South and civil society.

ICRAC looks forward to the ongoing process on autonomous weapons systems and offers the expertise of its members – scientists, technologists, academics, lawyers and policy experts – to the CCW’s States Parties.

Thank you, Mr. Chairperson.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

UN Secretary General Ban Ki-moon addresses National Model UN conference in the General Assembly Room, 26 March 2015. Photo: NMUN.

In less than two weeks, diplomats from around the world will gather at the United Nations in Geneva to discuss potential global regulations on “lethal autonomous weapons systems” that would be able to select and attack targets without direct human control.

But last week, at the National Model UN conference in New York, attended by some 2,500 undergraduate students from all over the world, a simulation of the UN General Assembly passed three resolutions calling for states to take action to prevent the threat of these “killer robots” to security, human rights and humanitarian law.

Addressing the closing ceremony of the conference, UN Secretary General Ban Ki-moon told students he was “energized by this dynamic gathering” and its “serious discussions” on “cutting-edge issues on the international agenda”, such as “lethal autonomous weapons systems.”

“You are not just leaders of the future – you can start to lead right now,” he told them, “now is the time for your generation to build human solidarity around the world.”

The NMUN NY resolutions defined lethal autonomous robots as “weapons that can select and attack targets independently – without meaningful human input or control”, suggested all countries immediate adopt a national moratorium on such weapons, and urged the negotiation of an international ban through an additional Protocol VI at the Convention on Certain Conventional Weapons (NMUN NY 2015A/GA1-1-1).

Model UN is a simulation of diplomacy, negotiation and decisionmaking by international organizations. Students play the role of diplomats from Member States of the UN and discuss issues at the top of the global policymaking agenda. NMUN NY is one of the biggest undergraduate Model UN conferences in the world.

The students assigned to simulate the General Assembly First Committee – which deals with issues of disarmament and international security – spent several months learning about their countries’ policy positions, the General Assembly and the politics of killer robots. (See for example, their background guide). After debate and drafting in the First Committee, the resolutions were passed by students representing the full plenary body in the actual General Assembly Room at the UN in New York.

The resolutions also called attention to the “work and expertise” of civil society, particularly the Campaign to Stop Killer Robots, Human Rights Watch and the International Committee for Robot Arms Control (ICRAC) (NMUN NY 2015A/GA1-1 -1, GA1-1-2 and GA1-1-3).

In a briefing, Dr. Matthew Bolton, Model UN advisor for Pace University New York City and member of ICRAC, told students at the conference that when new weapons technologies are not adequately addressed by existing regulations, the Marten’s Clause in international law requires states to be guided by “the principle of humanity and the dictates of public conscience.”

“Avoid the temptation to think this simulation is a meaningless game,” said Bolton, “A statement of strong concern from you could be considered an expression of public conscience – a challenge to policymakers in the real world to take action against killer robots.”

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

ICRAC welcomes the historic decision taken by nations to begin international discussions on how to address the challenges posed by fully autonomous weapons. The agreement marks the beginning of a process that the campaign believes should lead to an international ban on these weapons to ensure there will always be meaningful human control over targeting decisions and the use of violent force.

At 16:48 on Friday, 15 November 2013, at the United Nations in Geneva, states parties to the Convention on Conventional Weapons adopted a report containing a decision to convene on May 13-16, 2014 for their first meeting to discuss questions related to “lethal autonomous weapons systems” also known as fully autonomous weapons or “killer robots.” These weapons are at the beginning of their development, but technology is moving rapidly toward increasing autonomy.

“This is a very significant step forward for the International Committee for Robot Arms Control (ICRAC ),” said Professor Noel Sharkey, Chairman of ICRAC. “We are now on the first rung of the international ladder to fulfill our goal of stopping these morally obnoxious weapons from ever being deployed.”

ICRAC was formed in 2009 to initiate international discussion on autonomous weapons systems. It is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. It is a co-founder of the Campaign to Stop Killer Robots.

The Campaign to Stop Killer Robots believes that robotic weapons systems should not be making life and death decisions on the battlefield. That would be inherently wrong, morally and ethically. Fully autonomous weapons are likely to run afoul of international humanitarian law, and that there are serious technical, proliferation, societal, and other concerns that make a preemptive ban necessary.

“Law follows technology.  With robotic weapons, we have an rare opportunity to regulate a category of dangerous weapons before they are fully realized and the CCW is our best opportunity for regulation,” said Dave Akerson an ICRAC legal expert.

A total of 117 states are party to the Convention on Conventional Weapons, including nations known to be advanced in developing autonomous weapons systems: United States, China, Israel, Russia, South Korea, and United Kingdom. Adopted in 1980, this framework convention contains five protocols, including Protocol I prohibiting non-detectable fragments, Protocol III prohibiting the use of air-dropped incendiary weapons in populated areas, and Protocol IV, which preemptively banned blinding lasers.

“This is a momentous opportunity to get states on the record and behind a ban on fully autonomous offensive weapons,” said Heather Roff, an ICRAC philosopher. “If we can gain enough support, we might succeed in banning a technology before it actually harms innocent civilians.”

The agreement to begin work in the Convention on Conventional Weapons could lead to a future CCW Protocol VI prohibiting fully autonomous weapons.

ICRAC with the Campaign to Stop Killer Robots supports any action to urgently address fully autonomous weapons in any forum. The decision to begin work in the Convention on Conventional Weapons does not prevent work elsewhere, such as the Human Rights Council.

Since the topic was first discussed at the Human Rights Council on 30 May 2013, a total of 44 nations have spoken publicly on fully autonomous weapons since May: Algeria, Argentina, Australia, Austria, Belarus, Belgium, Brazil, Canada, China, Costa Rica, Cuba, Ecuador, Egypt, France, Germany, Ghana, Greece, Holy See, India, Indonesia, Iran, Ireland, Israel, Italy, Japan, Lithuania, Madagascar, Mexico, Morocco, Netherlands, New Zealand, Pakistan, Russia, Sierra Leone, Spain, South Africa, South Korea, Sweden, Switzerland, Turkey, Ukraine, United Kingdom, and United States. All nations that have spoken out have expressed interest and concern at the challenges and dangers posed by fully autonomous weapons.

Together with the Campaign to Stop Killer Robots, ICRAC urges nations to prepare for extensive and intensive work next year, both within the CCW and outside the CCW context.  We urge states to develop national policies, and to respond to the UN Special Rapporteur on Extrajudicial Executions’ call for national moratoria on fully autonomous weapons. We urge states to come back one year from now and agree to a new mandate to begin negotiations. The new process must be underscored by  a sense of urgency.

Peter Asaro, vice-chairman of ICRAC said “The actions of the CCW this week are a hopeful first step towards an international ban on autonomous weapons systems.’

Mathew Bolton delivered a statement on behalf ICRAC at the UN CCW meeting yesterday. As a group of experts we are prepared to help any nations with expert discussions of autonomous weapons systems and to help develop clear definitions for the language to be used in a treaty to ban them. Video footage of the statement, ICRAC’s first ever statement in an official diplomatic forum, is available here.

ICRAC recently coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

International Community Must Act Now to Ensure Combat Will Never Be Outsourced to “Killer Robots”! —

I am speaking on behalf of the International Committee for Robot Arms Control (ICRAC), a founding NGO member of the Campaign to Stop Killer Robots. Since its establishment in 2009, ICRAC has urged the international community to discuss the prohibition of fully autonomous weapons systems – “Killer Robots” – in light of the pressing dangers they pose to global peace and security, in addition to their alarming humanitarian implications for civilians threatened with armed violence.

ICRAC is made up of experts in robotic technology, artificial intelligence, computer science, international security and arms control, ethics and international law. As an indication of our concentration of expertise, over 80% of our members have doctoral or Juris Doctor degrees. As such, ICRAC is available and willing to provide technical expertise to the High Contracting Parties as they engage in further discussions about fully autonomous weapons systems.

ICRAC has coordinated the circulation of a “Scientists Call” to ban fully autonomous weapons systems, signed by more than 270 Computer Scientists, Engineers, Artificial Intelligence experts, Roboticists and professionals from related disciplines in 37 countries, saying: “given the limitations and unknown future risks of autonomous robot weapons technology, we call for a prohibition on their development and deployment. Decisions about the application of violent force must not be delegated to machines.”

ICRAC urges High Contracting Parties to be guided by principles of humanity in its deliberations on existing and emerging weapons technologies –  taking into account considerations of human security, human rights, human dignity, humanitarian law and the public conscience, as well as the justified worries about robotic arms races and proliferation. This means meaningful human deliberation and control over the use of violence must remain the cornerstone of any eventual global policymaking on robotic weapons.

ICRAC welcomes the presentation of the Draft Mandate text enabling the Convention on Certain Conventional Weapons to convene expert meetings in 2014 to look at the challenges posed by fully autonomous weapons systems. ICRAC also welcomes the statements by more than 40 States which have expressed concerns about autonomous weapons systems. We believe the Convention on Certain Conventional Weapons represents a useful forum to begin discussions, though dialogue about autonomous weapons systems need not be confined to this body alone. [END OF ORAL VERSION OF STATEMENT]

In light of our concerns, ICRAC respectfully submits the following recommendations to improve and strengthen the proposed Mandate on autonomous weapons systems before the body:

1)      It is not enough to consider only “lethal” autonomous weapons. We urge the body to broaden the Mandate to include consideration of all autonomous weapons systems, given that:

a) Arming autonomous systems with so-called “sub-lethal” or “less-than-lethal” weapons could still cause unnecessary suffering, and

b) Autonomous weapons systems could be potentially destabilizing even if they are only programmed to attack materiel, particularly if such systems interacted with each other in unexpected ways.

2)      By covering only “emerging technologies”, the Mandate may discourage High Contracting Parties from considering the implications of existing precursor autonomous systems that nonetheless raise unsettling humanitarian questions.

3)      While we welcome an Informal Meeting, a Mandate for a Group of Governmental Experts would more clearly send a message of the seriousness of this matter to the High Contracting Parties.

4)      As experts in this field, we believe the legal and ethical complexities of autonomous weapons systems will be difficult to cover adequately in three days. Thus we suggest amending the Draft Mandate to allow for a five (5) day meeting.

Delivered on behalf of the International Committee for Robot Arms Control, by:

Matthew Bolton, MSc. (LSE), PhD (LSE).
Membership Secretary, International Committee for Robot Arms Control
Advisor, Article 36
Assistant Professor of Political Science, Dyson College of Arts and Sciences, Pace University New York City.

For a PDF version of this statement, click here.

Video footage of the oral statement at the UN in Geneva is available for download here.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

ICRAC member Dr. Matthew Bolton, presenting a statement on disarmament at the UN General Assembly’s First Committee on Tuesday. Photo by Shant Alexander for Control Arms.

On behalf of global civil society organizations, International Committee for Robot Arms Control member Matthew Bolton calls for disarmament and arms control “driven by the needs and rights of people most affected by armed violence.” The Campaign to Stop Killer Robots also spoke, calling for fully autonomous weapons to “be prohibited through an international treaty, as well as through national laws and other measures.” To watch video footage of the NGO speeches, click here.

Dr. Matthew Bolton, a member of the International Committee for Robot Arms Control (ICRAC),  addressed the United Nations General Assembly First Committee Tuesday afternoon, on behalf of Article 36 and other international non-governmental organizations (NGOs) working on disarmament, peacebuilding and humanitarian issues.

“We call for an approach to disarmament that is driven by the needs and rights of people most affected by armed violence, not by the discretion of states and organizations most responsible for it,” said Dr. Bolton to representatives of the 193 UN member states, as well as UN agencies and NGOs. The First Committee has responsibility for disarmament and international security.

The NGO statement, read by Dr. Bolton and endorsed by 11 organizations, congratulated states on “some noteworthy progress” in recent international discussions on the elimination of nuclear weapons, the recent Security Council resolution on small arms and light weapons as well as the Arms Trade Treaty, signed by over 100 states since June.

Despite these developments in global policy making on controlling weapons, however, Dr. Bolton asserted that “now is not the time for resting on laurels.” The NGO statement identified numerous concerns, including the abuse of the consensus rule in disarmament forums, exclusion of meaningful civil society participation, lack of equal opportunities for women in decisionmaking and the marginalization of the voices of victims and survivors of armed violence.

“Creativity and new human-centered approaches must be a requirement for all states advocating nuclear disarmament, conventional arms control and reduced military expenditure,” said Dr. Bolton, reading the NGO statement. “We can and must replace stalemate and watered-down outcomes with alternatives that advance human security and social and economic justice.”

The Campaign to Stop Killer Robots also delivered a statement in the same session, calling for a prohibition on fully autonomous weapons.

“Our campaign believes that human control is essential to ensure the protection of civilians and to ensure compliance with international law,” said Mary Wareham of Human Rights Watch, delivering the statement on behalf of the campaign. “We seek a comprehensive and preemptive ban on weapons systems that would be able to select and attack targets without meaningful human intervention. These fully autonomous weapons or ‘lethal autonomous robots’ must be prohibited through an international treaty, as well as through national laws and other measures.”

Dr. Bolton is an expert on global disarmament policy and assistant professor of political science at Pace University. He is author of Foreign Aid and Landmine Clearance: Governance, Politics and Security in Afghanistan, Bosnia and Sudan (I.B. Tauris, 2010) and a forthcoming travelogue Political Minefields (I.B. Tauris, 2014). He has written widely on the politics of landmines, cluster munitions, the Arms Trade Treaty and fully autonomous military robotics (“killer robots”). He recently co-authored an ICRAC Working Paper on regulating robotic weapons with the Arms Trade Treaty.

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

1. Unequivocally assert that the Arms Trade Treaty Scope includes both manned and unmanned conventional arms,
2. Build on the recent clarifications by the Group of Governmental Experts of the UN Register of Conventional Arms of the categories of weapons borrowed by the Arms Trade Treaty in its Scope. The Group authoritatively defined the categories as including armed aerial drones,
3. Develop and promote comprehensive National Control Lists of the weapons to be controlled by states party to the Arms Trade Treaty,
4. Influence the interpretation of the Arms Trade Treaty through careful monitoring and calling out states acting in bad faith, and
5. Build connections between the community working on the Arms Trade Treaty (such as Control Arms) and those working on related campaigns (such as the Campaign to Stop Killer Robots) and control regimes (such as the UN Register, Missile Technology Control Regime, the Wassenaar Arrangement and dual-use equipment control programs).

Click here to read the full paper.

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

Today, there is vigorous debate among scholars, activists, soldiers and journalists about how to govern the use of robotic weapons systems like aerial drones, in order to limit civilian casualties and avoid undermining global regulations on the use of violence.

On 5 June, scholars from Pace University and beyond joined this debate with a ‘Robotic Weapons Control Symposium’ at the Downtown Campus, organized by the Political Science department in theDyson College of Arts and Sciences, funded by a $7,500 Thinkfinity Grant from Verizon Foundation, in partnership with Pace University’s Center for Teaching Learning and Technology.

“We are on the cusp of a technological revolution in the way wars are waged,” said Dr. Matthew Bolton, Assistant Professor of Political Science at Pace University and organizer of the symposium. “Intellectuals have a responsibility to contribute to conversations about how we will reinterpret and renew traditional constraints on killing for a digital age.”

The conference drew together around 20 thinkers from a variety of disciplines, including robotics, computer science, political science, philosophy, physics and the law. Presenters included a mix of Pace University professors and students from various schools and departments, as well as participants from the New School, NYU, Princeton, Carnegie Mellon, University of Denver, Sheffield University in the UKand Berlin School of Mind and Brain, Humboldt-Universität zu Berlin in Germany.

Participants considered three key questions:

• What are the ways that robotics is changing weapons technology?
• What will be the military, political, ethical and humanitarian impacts of the growing roboticization of warfare?
• What, if any, legal and normative restrictions should be placed on armed robots?

To view videos of several of the lectures from this event, click here.

The conference started with an introduction to robotic ethics by Dr. Illah Nourbakhsh,  Professor of Robotics at Carnegie Mellon University. Dr. Nourbakhsh argued that the disciplines of computer science, robotics and artificial intelligence needed to think carefully about social responsibility and the implications of their research, particularly in the context of weapons development.

“We, the practitioners of robotics need to embrace ethical analysis if we are to understand the consequences of our research decisions,” said Dr. Nourbakhsh, author of the critically-acclaimed bookRobot Futures. “The technology of robotics has progress so rapidly in the last decade that researchers’ innovations are emerging in the battlefield, hospital and home without proper strategic thinking about the ethical consequences of robotic design and downstream impact.”

He was followed by a several presenters looking at different ethical questions in the context of using robotics weapons in war. Dr. Noel Sharkey, professor of artificial intelligence and robots, and professor of public engagement at the University of Sheffield, argued for a ban on the emerging class of fully-autonomous armed robots – ‘killer robots’ – which select and fire upon targets without any direct human control.

“The continued automation of killing will have disastrous consequences for humanity and we should stop it now,” said Dr. Sharkey, chair of the International Committee for Robot Arms Control (ICRAC), one of the organizations leading the new Campaign to Stop Killer Robots.

“Robots cannot comply with the Laws of War in discriminating civilians from combatants and they lack the reasoning necessary to make judgments about the proportional use of force. Unlike a human, they cannot be held accountable for their errors. We must not cross the line where machines are delegated the decision to kill humans. To do otherwise is both morally repugnant and unthinkable.”

The implications of fully autonomous weapons were discussed in depth with presentations by Dr. Peter Asaro of the New School, Dr. Micha? Klincewicz of Humboldt University, Dr. D. Paul Benjamin of Pace University’s Seidenberg School of Computer Science and Cayman Mitchell ‘14, a Pace University undergraduate.

“Preparing for and participating in this symposium was a highlight of my undergraduate career at Pace,” said Mitchell, an honors student who studies computer science and peace and justice studies on the New York City campus. “The interdisciplinary nature of the conference was exciting because it allowed me to listen to and converse with scholars from many different fields, almost all of whom overlapped to some extent with my diverse academic interests.”

In the afternoon, participants began to explore the legal and political ramifications of the roboticization of war. Professor Thomas McDonnell of the Pace University Law School considered the legalities of the US use of armed drones to attack targets in Pakistan, Yemen and Somalia, whileProfessor Robert Wiener of the Lubin School of Business offered reflections on how Jewish law and the Golem myth might offer insights into the regulation of ‘unmanned’ weapons.

“Today’s conference was a unique and exciting opportunity to learn from a breadth of scholars engaging in this important and timely issue.  It is reassuring to see so many from such diverse fields coming together on this” said Dr. Heather M. Roff, Visiting Associate Professor at the University of Denver and Research Associate at the United States Air Force Academy. “The moral, political and legal questions pertaining to the use of autonomous machines in war are extremely pressing. This symposium was extremely interesting in this respect, as it engaged each type of question from such bright scholars.”

In her presentation, Dr. Roff offered insights into how robotic weapons are reshaping the political structure of decisionmaking about military strategy. She was followed by recent Pace graduateCassandra Stimpson ‘13, who used scapegoating theory to explain the socio-political functions of extrajudicial killings by armed drones.

“The symposium was a rapid exchange of ideas from some of the most educated and passionate on the subject of autonomous and drone warfare. I was able to contribute overarching themes in my presentation about the general inefficiency and danger of targeted killing in a technological age where this act can be done with such ease.” said Stimpson, an honors political science major and peace and justice studies minor. “To be able to interact and present ideas on a new scholarly plane was vital to my growth, and the type of event I want to continue to contribute to in the future.”

The conference concluded with an open discussion about how to take the conversation forward and communicate with the broader public, governments and armed groups about the urgency of reinvigorating humanitarian and human rights norms in digitized warfare.

“We are very grateful to Verizon Foundation and the Center for Teaching, Learning and Technology for supporting this crucial conversation, all the participants for their thought-provoking contributions and, in particular, the work of Cayman and Cassandra in arranging logistics for the conference,” said Dr. Bolton.

The Pace University Political Science Department is home to one of the most popular and fastest growing majors on the New York City campus. Capitalizing on its location in the heart of the Financial District, opposite City Hall and two express subway stops from the United Nations, the department encourages students to reflect on how power works at various levels of government and society in addressing to major local, national and global challenges. Political Science faculty and students apply their lessons outside the classroom in international policymaking processes, including on peace, security, disarmament, human rights and humanitarian issues.

[Reposted from http://politicalminefields.com/2013/08/23/the-politics-of-killer-robots/]

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

Several members of the International Committee for Robot Arms Control (ICRAC) – Wim Zwijnenburg of IKV Pax Christi, Thomas Nash and Richard Moyes of Article 36 and Matthew Bolton of Pace University New York City – were engaged in supporting the advocacy work of Control Arms, the global civil society coalition campaigning for a ‘bulletproof’ treaty.

Pushing states to develop text that would cover emerging weapons technologies was a particular emphasis of ICRAC members’ lobbying at the July 2012 and March 2013 Diplomatic Conferences. Many campaigners and diplomats were concerned that the draft treaty did not include specific provisions for ‘unmanned’ weapons, such as aerial drones, or robotic systems that have ‘dual uses.’ A recent report from the Stockholm International Peace Research Institute (SIPRI) raised concerns that the text “looks dangerously likely to be a relic before it ever comes into force.”

Futureproofing

Drawing on technical advice from other ICRAC members, Zwijnenburg and Bolton wrote a policy brief titled “Futureproofing the Draft Arms Trade Treaty” that called on states to make five “critical changes” to the text “in order to cover the emerging class of robotic, ‘unmanned’ and autonomous weapons.” The paper was distributed widely in the conference and online to governments and civil society organizations and was reprinted in Reaching Critical Will’s widely read ATT Monitor newsletter (pp. 3-4). The phrase “futureproofing” caught on and was soon being used widely by Control Arms campaigners, Amnesty International and even the representative of the Holy See.

Not all of the changes suggested in Zwijnenburg and Bolton’s “Futureproofing” paper were made in the final treaty text and it would be disingenuous to overstate ICRAC’s impact. However, by helping to shape and frame the conversation, the policy brief, amplified by Control Arms lobbying, contributed to efforts that changed the treaty text to allow for the future conferences of States Parties to the treaty to review “developments in the field of conventional arms” (Article 17) and adopt amendments by three-quarters vote instead of consensus (Article 20). This means that activists and advocacy organizations will be able to push states to amend the treaty to address developing new weapons technologies. This new text has essentially created a forum in which ICRAC and other stakeholders concerned about emerging weapons technologies can press their case in the future.

What Next?

The next push for campaigners will be to make sure states sign and ratify the ATT, to make it enter into force as quickly as possible. Another important area for advocacy will be to push for a broadening of the categories used by the UN Register of Conventional Weapons. The ATT relies on these categories, which at the moment do not explicitly cover many types of robotic weapons. If civil society can push for states to include unmanned armed systems in this register before the treaty enters into force, the treaty will actually cover a broader scope of weapons.

While the ATT and broadening the UN Register have not been the primary focus of ICRAC’s advocacy, they are establishing important precedents and norms that provide important foundations for the regulation of robotic weapons. Indeed, passing the treaty in a majority vote in the UN General Assembly, instead of consensus, has opened the possibility of developing arms control instruments with high standards, instead of the lowest common denominator.

The ATT is not really a disarmament treaty – it is more of an amalgamation of humanitarian and trade law. Even if it works well, it will only regulate the flows of weapons, not the kind of weapons in circulation. As a result, those who are concerned about the trends toward ‘autonomy’ in robotic weapons, threatening to reduce direct human control over killing, cannot rely on the ATT to prevent this dangerous possibility. This is one of many reasons why ICRAC is part of a growing number of NGOs and faith groups calling for a specific on ban fully autonomous armed robots – “killer robots.”

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

Enough concern and information exists now to consider appropriate governance models in a timely and proactive manner; yet, the time to take action is short before the current window of opportunity to design a relevant governance or oversight system for LARs [Lethal Autonomous Robotics] closes.

The 2009 statement by International Committee for Robot Arms Control (ICRAC), which called for bans on autonomous weapons, the arming of robotic weapons with nuclear arms and the deployment of armed robots in space is probably one of the most developed proposals I have seen. As a member of ICRAC, I have endorsed this statement. However, I have been considering ways to flesh out the ICRAC proposal with further provisions building on the precedents of the landmine and cluster munition treaties, as well as the draft Arms Trade Treaty (currently still being negotiated) and the broader corpus of international humanitarian law governing conduct in war.

Yesterday, I published an article (for Global Policy’s online Comment & Opinion section) and a post on my blog outlining possible provisions for an International Convention on Robotic Weapons. I am a political scientist, not a lawyer or roboticist, so my purpose in doing this is to provoke discussion and dialogue, not to pin down precise language. Moreover, while I have used the ICRAC statement as a foundation, this is my own personal adaptation and expansion of it and so should not be interpreted as an institutional position of ICRAC. Following discussions since yesterday, I have consolidated my proposed provisions into a revised list:

A. Outright Bans on Robotic Weapons Incapable of Discriminating Between Combatants and Non-Combatants and/or that Cause Disproportionate Harm

1. Prohibition of the development, deployment and use of armed autonomous unmanned systems that make decisions to kill without a ‘human-in-the-loop’;
2. Prohibition of arming robotic, particularly ‘unmanned’, systems with nuclear, biological or chemical weapons, blinding lasers, landmines, cluster munitions or other weapons prohibited by international humanitarian norms.

B. Restrictions on Deployment and Targeting to Ensure Discrimination and Proportionality and Limit Aggressive Deployment

1. Limitations on the development, deployment and use of robotic systems to ensure discrimination and proportionality in targeting, prevent civilian casualties and limit psycho-social harm;
2. Prohibition of robotic deployment of explosive violence in populated areas;
3. Limitations on the range and weapons carried by remotely-operated ‘unmanned’ systems and on their deployment in postures threatening to other states;
4. Regulation of the use of so-called ‘less-lethal’, ‘non-lethal’ or ‘pain’ weapons by robotic systems;
5. Prohibition of the development, deployment and use of robot space weapons.

C. Restrictions on the Import, Export, Trade, Transshipment, Brokering and Transfer of Robotic Weapons

1. Prohibition of the trade or transfer of robotic weapons, parts, components, related technologies and munitions, to countries and armed groups that commit serious violations of international human rights and humanitarian law;
2. Transparent reporting on the trade and transfer of robotic weapons.

D. International Cooperation in the Mitigation of the Harm Caused by Robotic Weapons

1. International cooperation in the reconstruction of communities damaged by robotic weapons;
2. Recognition of the rights of survivors of robotic weapon violence and international cooperation in the provision of victim assistance;
3. National and international mechanisms for recording the casualties of robotic weapon violence;
4. National reporting on treaty implementation;
5. International cooperation on treaty implementation, including technical assistance.

Do you think a treaty is needed to govern drones and other robotic weapons? What would such a treaty cover? How would it be monitored and enforced? On reading my proposal, a friend asked me whether it makes more sense to focus on the banning and circumscribing the effects of weapons (indiscriminate and disproportionate harm) rather than types of weapons (like robotic weapons, landmines, cluster munitions, etc.). What do you think? 

– Dr. Matthew Bolton, Department of Political Science, Pace University (Note that the views expressed here are the author’s alone and have not been endorsed by the International Committee for Robot Arms Control).

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).