In the lead up to the next meeting of the CCW’s Group of Governmental Experts at the United Nations April 9-13th in Geneva, the UN’s Institute for Disarmament Research has issued a briefing paper titled The Weaponization of Increasingly Autonomous Technologies: Artificial Intelligence. Designated a primer for CCW delegates, the paper lists no authors, but a special acknowledgement to Paul Scharre, Senior Fellow and Director of the Technology and National Security Program at the Center for a New American Security, suggests that the viewpoints of the Washington, D.C.-based CNAS are well represented.

Surprisingly for a document positioning itself as “an introductory primer for non-technical audiences on the current state of AI and machine learning, designed to support the international discussions on the weaponization of increasingly autonomous technologies” (pp. 1-2), the paper opens with a series of assertions regarding “rapid advances” in the field of AI. The evidence offered is the case of Google/Alphabet affiliate Deep Mind’s AlphaGo Zero, announced in December 2017 (“only a few weeks after the November 2017 GGE”) as having achieved better-than-human competency at (simulations of) the game of Go:

Although AlphaGo Zero does not have direct military applications, it suggests that current AI technology can be used to solve narrowly defined problems provided that there is a clear goal, the environment is sufficiently constrained, and interactions can be simulated so that computers can learn over time (p.1).

The requirements listed – a clear (read computationally specifiable) goal, within a constrained environment that can be effectively simulated – might be underscored as cautionary qualifications on claims for AI’s applicability to military operations. The tone of these opening paragraphs suggests, however, that these developments are game-changers for the GGE debate.

The paper’s first section, titled ‘What is artificial intelligence,’ opens with the tautological statement that “Artificial intelligence is the field of study devoted to making machines intelligent” (p. 2). A more demystifying description might say, for example, that AI is the field of study devoted to developing computational technologies that automate aspects of human activity conventionally understood to require intelligence. While the authors observe that as systems become more established they shift from characterizations of “intelligence” to more mundane designations like “automation” or “computation,” they suggest that rather than the result of demystification this is itself somehow an effect of the field’s advancement. One implication of this logic is that the ever-receding horizon of machine intelligence should be understood not as a marker of the technology’s limits, but of its success.

We begin to get a more concrete sense of the field in the section titled ‘Machine learning,’ which outlines the latter’s various forms. Even here, however, issues central to the deliberations of the GGE are passed over. For example, in the statement that “[r]ather than follow a proscribed [sic] set of if–then rules for how to behave in a given situation, learning machines are given a goal to optimize – for example, winning at the game of chess” (p. 2) the example is not chosen at random, but rather is illustrative of the unstated requirement that the ‘goal’ be computationally specifiable. The authors do helpfully explain that “[s]upervised learning is a machine learning technique that makes use of labelled training data” (my emphasis, p. 3), but the contrast with “unsupervised learning,” or “learning from unlabelled data based on the identification of patterns” fails to emphasize the role of the human in assessing the relevance and significance of patterns identified. In the case of reinforcement learning “in which an agent learns by interacting with its environment,” the (unmarked) examples are again from strategy games in which, implicitly, the range of agent/environment interactions are sufficiently constrained. And finally, the section on ‘Deep learning’ helpfully emphasizes that so called neural networks rely either on very large data sets and extensive labours of human classification (for example, the labeling of images to enable their ‘recognition’), or on domains amenable to the generation of synthetic ‘data’ through simulation (for example, in the case of strategy games like Go). Progress in AI, in sum, has been tied to growth in the availability of large data sets and associated computational power, along with increasingly sophisticated algorithms within highly constrained domains of application.

Yet in spite of these qualifications, the concluding sections of the paper return to the prospects for increasing machine autonomy:

Intelligence is a system’s ability to determine the best course of action to achieve its goals. Autonomy is the freedom a system has in accomplishing its goals. Greater autonomy means more freedom, either in the form of undertaking more tasks, with less supervision, for longer periods in space and time, or in more complex environments … Intelligence is related to autonomy in that more intelligent systems are capable of deciding the best course of action for more difficult tasks in more complex environments. This means that more intelligent systems could be granted more autonomy and would be capable of successfully accomplishing their goals (p. 5, original emphasis).

The logical leap exemplified in this passage’s closing sentence is at the crux of the debate regarding lethal autonomous weapon systems. The authors of the primer concede that “all AI systems in existence today fall under the broad category of “narrow AI”. This means that their intelligence is limited to a single task or domain of knowledge” (p. 5). They acknowledge as well that “many advance [sic] AI and machine learning methods suffer from problems of predictability, explainability, verifiability, and reliability” (p. 8). These are precisely the concerns that have been consistently voiced, over the past five meetings of the CCW, by those states and civil society organizations calling for a ban on autonomous weapon systems. And yet the primer takes us back, once again, to a starting point premised on general claims for the field of AI’s “rapid advance,” rather than careful articulation of its limits. Is it not the latter that are most relevant to the questions that the GGE is convened to consider?

The UNIDIR primer comes at the same time that the United States has issued a new position paper in advance of the CCW titled ‘Humanitarian benefits of emerging technologies in the area of lethal autonomous weapon systems’ (CCW/GGE.1/2018/WP.4). While the US has taken a cautionary position in relation to lethal autonomous weapon systems in past meetings, asserting the efficacy of already-existing weapons reviews to address the concerns raised by other member states and civil society groups, it now appears to be moving in the direction of active promotion of LAWS on the grounds of promised increases in precision and greater accuracy of targeting, with associated limits on unintended civilian casualties – promises that have been extensively critiqued at previous CCW meetings. Taken together, the UNIDIR primer and the US working paper suggest that, rather than moving forward from the debates of the past five years, the 2018 meetings of the CCW will require renewed efforts to articulate the limits of AI, and their relevance to the CCW’s charter to enact Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects.

Author information

Lucy Suchman

Lucy Suchman is a Professor of the Anthropology of Science and Technology at Lancaster University in the UK. Before taking up her present post she was a Principal Scientist at Xerox's Palo Alto Research Center (PARC), where she spent twenty years as a researcher. During this period she became widely recognized for her critical engagement with artificial intelligence (AI), as well as her contributions to a deeper understanding of both the essential connections and the profound differences between humans and machines.

Sauer, Frank 2016: Stopping ‘Killer Robots’: Why Now Is the Time to Ban Autonomous Weapons Systems, in: Arms Control Today 46 (8): 8-13.

Click here to read the full article.

NEW: Click here for the BRF file of the full article

[F]our possible outcomes can be predicted for the CCW process. The first would be a legally binding and preventive multilateral arms control agreement derived by consensus in the CCW and thus involving the major stakeholders, the outcome referenced as “a ban.” Considering the growing number of states-parties calling for a ban and the large number of governments calling for meaningful human control and expressing considerable unease with the idea of autonomous weapons systems, combined with the fact that no government is openly promoting their development, this seems possible. It would require mustering considerable political will. Verification and compliance for a ban, as well as for weaker restrictions, would then require creative arms control solutions. After all, with full autonomy in a weapons system eventually coming down to merely flipping a software switch, how can one tell if a specific system at a specific time is not operating autonomously? A few arms control experts are already wrapping their heads around these questions.

 

Can #KillerRobots (autonomous weapons systems) work as preventive arms control? More in October’s #ArmsControlToday https://t.co/E7sDVzdmbn pic.twitter.com/LwPSojH9Gr

— Arms Control Assoc (@ArmsControlNow) October 13, 2016

The second outcome would be restrictions short of a ban. The details of such an agreement are impossible to predict, but it is conceivable that governments could agree, for example, to limit the use of autonomous weapons systems, such as permitting their use against materiel only.

The third would be a declaratory, nonbinding agreement on best practices. Such a code of conduct would likely emphasize compliance with existing international humanitarian law and rigorous weapons review processes, in accordance with Article 36 of Additional Protocol I to the Geneva Conventions.

Finally, there may be no tangible result, perhaps with one of the technologically leading countries setting a precedent by fielding autonomous weapons systems. That would certainly prompt others to follow, fueling an arms race. In light of some of the most advanced standoff weapons, such as the U.S. Long Range Anti-Ship Missile or the UK Brimstone, each capable of autonomous targeting during terminal flight phase, one might argue that the world is already headed for such an autonomy arms race.

Implementing autonomy, which mainly comes down to software, in systems drawn from a vibrant global ecosystem of unmanned vehicles in various shapes and sizes is a technical challenge, but doable for state and nonstate actors, particularly because so much of the hardware and software is dual use. In short, autonomous weapons systems are extremely prone to proliferation. An unchecked autonomous weapons arms race and the diffusion of autonomous killing capabilities to extremist groups would clearly be detrimental to international peace, stability, and security.

The nascent social taboo against machines autonomously making kill decisions – Frank Sauer in @ArmsControlNow https://t.co/nBTGtXLT5R #CCWUN

— Mary Wareham (@marywareham) October 19, 2016

This underlines the importance of the current opportunity for putting a comprehensive, verifiable ban in place. The hurdles are high, but at this point, a ban is clearly the most prudent and thus desirable outcome. After all, as long as no one possesses them, a verifiable ban is the optimal solution. It stops the currently commencing arms race in its tracks, and everyone reaps the benefits. A prime goal of arms control would be fulfilled by facilitating the diversion of resources from military applications toward research and development for peaceful purposes—in the fields of AI and robotics no less, two key future technologies.

Author information

Frank Sauer

Doomsday Clock

In November 2012, United States Deputy Defense Secretary Ashton Carter signed directive 3000.09, establishing policy for the “design, development, acquisition, testing, fielding, and … application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.”  Without fanfare, the world had its first openly declared national policy for killer robots.

The policy has been widely misperceived as one of caution. According to one account, the directive promises that a human will always decide when a robot kills another human. Others even read it as imposing a 10-year moratorium to allow for discussion of ethics and safeguards. However, as a Defense Department spokesman confirmed for me, the 10-year expiration date is routine for such directives, and the policy itself is “not a moratorium on anything.”

A careful reading of the directive finds that it lists some broad and imprecise criteria and requires senior officials to certify that these criteria have been met if systems are intended to target and kill people by machine decision alone. But it fully supports developing, testing, and using the technology, without delay. Far from applying the brakes, the policy in effect overrides longstanding resistance within the military, establishes a framework for managing legal, ethical, and technical concerns, and signals to developers and vendors that the Pentagon is serious about autonomous weapons.

Did soldiers ask for killer robots? In the years before this new policy was announced, spokesmen routinely denied that the US military would even consider lethal autonomy for machines.  Over the past year, speaking for themselves, someretired and even active duty officers have written passionately against both autonomous weapons and the overuse of remotely operated drones. In May 2013, the first nationwide poll ever taken on this topic found that Americans opposed to autonomous weapons outnumbered supporters by two to one. Strikingly, the closer people were to the military—family, former military, or active duty—the more likely they were to strongly oppose autonomous weapons and support efforts to ban them.

Since the 1990s, the military has exhibited what autonomy proponent Barry Watts has called “a cultural disinclination to turn attack decisions over to software algorithms.” Legacy weapons such as land and sea mines have been deemphasized and some futuristic programs canceled—or altered to provide greater capabilities for human control. Most notably, the Army’s Future Combat Systems program, which was to include a variety of networked drones and robots at an eventual cost estimated as high as $300 billion, was cancelled in 2009, with $16 billion already spent.

At the same time, calls for autonomous weapons have been rising both outside and from some inside  the military. In 2001, retired Army lieutenant colonel T. K. Adams argued that humans were becoming the most vulnerable, burdensome, and performance-limiting components of manned systems. Communications links for remote operation would be vulnerable to disruption, and full autonomy would be needed as a fallback. Furthermore, warfare would become too fast and too complex for humans to direct. Realistic or not, such thinking, together with budget pressures and the perception that robots are cheaper than people, has supported a steady growth of autonomy research and development in military and contractor-supported labs. In March 2012, the Naval Research Lab opened a new facility dedicated to development and testing of autonomous systems, complete with simulated rainforest, desert, littoral, and shipboard or urban combat environments. But the killer roboticists’ brainchildren have continued to face what a 2012 Defense Science Board report, commissioned by then-Undersecretary Carter, called “material obstacles within the Department that are inhibiting the broad acceptance of autonomy.”

The discrimination problem. Navy scientist John Canning recounts a 2003 meeting at which high-level lawyers from the Navy and Pentagon objected to autonomous weapons. They assumed that robots could not comply with international humanitarian law, core principles of which include a responsibility to distinguish civilians from combatants and to refrain from attacks that would cause excessive harm to civilians. These principles, and the military rules of engagement intended to implement them, assume a level of awareness, understanding, and judgment that computers simply don’t have. Weapons are also subject to mandated legal review, and indiscriminate weapons—that is, weapons that cannot be selectively directed to attack lawful targets and avoid civilians—are forbidden. The lawyers did not think they would ever be able to sign off on autonomous weapons.

Georgia Tech roboticist Ron Arkin has argued that unemotional robots, following rigid programs, could actually be more ethical than human soldiers. But his proposals fail to solve the hard problems of distinguishing civilians, understanding and predicting social and tactical situations, or judging the proportionality of force. Others argue, philosophically, that only humans can make such targeting judgments legitimately. In a world getting used to talking about virtual assistants and self-driving cars, it may not be obvious what the limits of artificial intelligence will be, or what people will accept, in 10, 20, or 40 years. But for now, and for the immediate future, the robot discrimination problem is hard to dispute.

To break the legal deadlock, Canning suggested that robots might normally be granted autonomy to attack materiel, including other robots, but not humans. Yet in many situations it might be impossible to avoid the risk—or the intent—of killing or injuring people. For such cases, Canning proposed what he called “dial-a-level” autonomy; that is, the robot might ordinarily be required to ask a human what to do, but in some circumstances it could be authorized to take action on its own.

In recent years, autonomy visionaries have stressed human-machine partnerships and flexibility to decide the level of autonomy a weapon may be allowed, based on tactical needs.  In a 2011 roadmap, for example, the Defense Department envisions unmanned systems that seamlessly operate with manned systems while gradually reducing the degree of human control and decision making required. A 2011 Navy presentationdepicts decisions about autonomy and control as a continuous tradeoff, explaining that while human control minimizes the risk of attacking unintended targets, machine autonomy maximizes the chance of defeating the intended ones. It seems likely that in desperate combat, autonomy would be dialed up to the highest level.

Appropriate levels of human judgment. In the spring of 2011, the Defense Department convened a group of uniformed and civilian personnel to begin developing a policy for autonomous weapons. The directive that emerged 18 months later lists a number of requirements for autonomous systems and draws a line at systems intended to autonomously target and engage humans–or to apply kinetic force (e.g., bullets and bombs) against any targets. But the directive neither states nor implies that this line should not be crossed.

Rather, the line may be crossed if two undersecretaries and the Chairman of the Joint Chiefs of Staff affirm that the listed requirements have been met. In the event of an urgent military need, any of the requirements can be waived—with the exception of a legal review. Furthermore, the line is not as clearly drawn as it may seem to be.

The requirements listed in the directive are not much more stringent than those that apply to any weapon system. Tactics, techniques, and procedures must be developed to specify how an autonomous weapon system should be used. Hardware and software must undergo rigorous verification and validation. Human-machine interfaces must be understandable to trained operators and must provide clear activation and deactivation procedures and have “safeties, anti-tamper mechanisms, and information assurance” that minimize the probability of unintended engagements.

These requirements sound reassuring; they promise to address many of the concerns people have about autonomous weapons. According to the directive, it is Defense Department policy that the measures listed will ensure that the systems will work in realistic environments against adaptive adversaries. But saying it doesn’t necessarily make it so.

In reality, neither mathematical analysis nor field testing can possibly locate every software bug or situation in which such complex systems may fail or behave inappropriately. Adversaries will strive to locate points of vulnerability, and it is terribly hard to anticipate everything that adversaries may do, let alone know how their actions may affect system performance. The notion of information assurance implies a promise to solve problems of software reliability and computer security that bedevil contemporary technology.

The centerpiece of the entire directive is this statement: “Autonomous and semi-autonomous weapon systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.” Although the phrase is never defined, it does not appear that appropriate levels always require at least one human being to make the decision to kill another. Rather, the appropriate level might well be the decision to dispatch a robot on a mission and let it select the targets to engage. In making such decisions, it appears that the burden of ensuring compliance with rules of engagement and laws of war falls on commanders and operators when the robots themselves are incapable of ensuring this. But in practice, it seems likely that unintended atrocities committed by autonomous weapons will be blamed on technical failures.

Semi-autonomy: Smudging the line. In theory, as long as three senior officials withhold their signatures, autonomous weapon systems that are intended to target humans or use kinetic or lethal force would be blocked. But the policy green lights—no extra signatures needed—semi-autonomous weapon systems that may apply any kind of force against any targets, including people. The crucial line that the policy draws between semi- and fully autonomous systems is fuzzy and broken. As technology advances, it is likely to be crossed as a matter of course.

The directive defines a semi-autonomous weapon system as one intended to engage only those targets that have been selected by a human operator. But the system itself is allowed to use autonomy to acquire, track and identify potential targets. It can cue the operator, prioritize targets, and decide when to fire. What the operator must do to select targets is left unspecified. Would a verbal OK, gesture, or even neurological interface be acceptable?

A system with such capabilities may not be intended to function without a human operator, but at most it would require a trivial modification to do so—perhaps a hack.At least three companies already market such systems. The policy clears them for immediate use after acquisition, via standard procedures.

The policy also addresses fire-and-forget or lock-on-after-launch homing munitions, which would include many systems in use today. Such munitions have seekers that autonomously find and home on targets. The directive classifies them as semi-autonomous weapon systems, on the theory that the operator selects targets by using tactics, techniques and procedures that “maximize the probability that the only targets within the seeker’s acquisition basket” will be the intended targets. Yet, upon launch, such munitions become, de facto, fully autonomous.

No restrictions are placed on the technology that a seeker may use to find a target and decide whether that is what it was looking for. This opens a clear path for weapons that can be sent on hunt-and-kill missions, limited only by the ability of their onboard sensors and computers to narrow their acquisition baskets to selected targets.

The way forward—to what? In the mid-2000s, Lockheed Martin was developing a small autonomous drone missile for the Air Force, and a similar system for the Army. Equipped with several types of onboard sensors, the missiles would fly out to designated areas and wander in search of generic targets, such as tanks, rocket launchers, radars, or personnel, which they would autonomously recognize and attack. Both programs were canceled, amid legal, ethical, and technical questions, to be superseded by systems that combine autonomous capabilities with radio links to human operators. Under the new policy, would such wide-area search munitions be classified as autonomous or semi-autonomous? Either way, the policy establishes that weapons like these may be developed, acquired, and used.

Given the long internal debate and general public opposition to killer robots, this is a highly aggressive policy. The US military never intended to replace foot soldiers with autonomous lethal robots during this decade, particularly not where civilians might be at risk. But funding the development and acquisition of systems that have autonomous targeting and fire-control capabilities—even if they are not intended for fully autonomous killing—will spur the weapons industry, in the United States and elsewhere, to accelerate exploration and investment in the technology of autonomous warfare.

The real issue is whether the world needs to go this way at all. The message of this policy is: full speed ahead.

Author information

Mark Gubrud

Several members of the International Committee for Robot Arms Control (ICRAC) – Wim Zwijnenburg of IKV Pax Christi, Thomas Nash and Richard Moyes of Article 36 and Matthew Bolton of Pace University New York City – were engaged in supporting the advocacy work of Control Arms, the global civil society coalition campaigning for a ‘bulletproof’ treaty.

Pushing states to develop text that would cover emerging weapons technologies was a particular emphasis of ICRAC members’ lobbying at the July 2012 and March 2013 Diplomatic Conferences. Many campaigners and diplomats were concerned that the draft treaty did not include specific provisions for ‘unmanned’ weapons, such as aerial drones, or robotic systems that have ‘dual uses.’ A recent report from the Stockholm International Peace Research Institute (SIPRI) raised concerns that the text “looks dangerously likely to be a relic before it ever comes into force.”

Futureproofing

Drawing on technical advice from other ICRAC members, Zwijnenburg and Bolton wrote a policy brief titled “Futureproofing the Draft Arms Trade Treaty” that called on states to make five “critical changes” to the text “in order to cover the emerging class of robotic, ‘unmanned’ and autonomous weapons.” The paper was distributed widely in the conference and online to governments and civil society organizations and was reprinted in Reaching Critical Will’s widely read ATT Monitor newsletter (pp. 3-4). The phrase “futureproofing” caught on and was soon being used widely by Control Arms campaigners, Amnesty International and even the representative of the Holy See.

Not all of the changes suggested in Zwijnenburg and Bolton’s “Futureproofing” paper were made in the final treaty text and it would be disingenuous to overstate ICRAC’s impact. However, by helping to shape and frame the conversation, the policy brief, amplified by Control Arms lobbying, contributed to efforts that changed the treaty text to allow for the future conferences of States Parties to the treaty to review “developments in the field of conventional arms” (Article 17) and adopt amendments by three-quarters vote instead of consensus (Article 20). This means that activists and advocacy organizations will be able to push states to amend the treaty to address developing new weapons technologies. This new text has essentially created a forum in which ICRAC and other stakeholders concerned about emerging weapons technologies can press their case in the future.

What Next?

The next push for campaigners will be to make sure states sign and ratify the ATT, to make it enter into force as quickly as possible. Another important area for advocacy will be to push for a broadening of the categories used by the UN Register of Conventional Weapons. The ATT relies on these categories, which at the moment do not explicitly cover many types of robotic weapons. If civil society can push for states to include unmanned armed systems in this register before the treaty enters into force, the treaty will actually cover a broader scope of weapons.

While the ATT and broadening the UN Register have not been the primary focus of ICRAC’s advocacy, they are establishing important precedents and norms that provide important foundations for the regulation of robotic weapons. Indeed, passing the treaty in a majority vote in the UN General Assembly, instead of consensus, has opened the possibility of developing arms control instruments with high standards, instead of the lowest common denominator.

The ATT is not really a disarmament treaty – it is more of an amalgamation of humanitarian and trade law. Even if it works well, it will only regulate the flows of weapons, not the kind of weapons in circulation. As a result, those who are concerned about the trends toward ‘autonomy’ in robotic weapons, threatening to reduce direct human control over killing, cannot rely on the ATT to prevent this dangerous possibility. This is one of many reasons why ICRAC is part of a growing number of NGOs and faith groups calling for a specific on ban fully autonomous armed robots – “killer robots.”

ICRAC is an international committee of experts in robotics technology, robot ethics, international relations, international security, arms control, international humanitarian law, human rights law, and public campaigns, concerned about the pressing dangers that military robots pose to peace and international security and to civilians in war.

Author information

mbolton

Matthew Bolton is professor of political science at Pace University in New York City. He is an expert on global peace and security policy, focusing on multilateral disarmament and arms control policymaking processes. He has a PhD in Government and Master's in Development Studies from the London School of Economics and a Master's from SUNY Environmental Science and Forestry. Since 2014, Bolton has worked on the UN and New York City advocacy of the International Campaign to Abolish Nuclear Weapons (ICAN), recipient of the 2017 Nobel Peace Prize. Bolton has published six books, including Political Minefields (I.B. Tauris) and Imagining Disarmament, Enchanting International Relations (Palgrave Pivot).

For example, they must not be equipped with biological or chemical weapons. In Europe most land and air vehicles are covered by the definitions of the Treaty on Conventional Armed Forces (CFE Treaty), thus they are limited by numbers and subject to verification. If armed uninhabited vehicles cannot be prohibited outright, then limitations similar to the CFE Treaty are needed in other regions of the world. To avoid dangers for international humanitarian law and military stability, autonomous attack, that is attacks without a human decision in each single case, should be prohibited. Additional prohibitions are needed, among others, for small and very small armed vehicles.

Jürgen Altmann, Arms control for armed uninhabited vehicles: an ethical issue, Ethics and Information Technology, 2013, DOI 10.1007/s10676-013-9314-5. Read the full article  (open access, 17 pages) here.

Author information

altmann

Jürgen Altmann (PhD) is a physicist and peace researcher (retired) at TU Dortmund University, Germany. Since 1985 he has studied scientific-technical problems of disarmament. An experimental focus is automatic sensor systems for co-operative verification of disarmament and peace agreements and for IAEA safeguards for an underground final repository. The second focus is assessment of new military technologies and preventive arms control, including verification. Studies have dealt with “non-lethal” weapons, civilian and military technologies in aviation, military uses of microsystems technology and of nanotechnology, confidence and security building measures for cyber forces, armed uncrewed vehicles and autonomous weapon systems. He co-founded and chairs the German Research Association for Science, Disarmament and International Security (FONAS) and has authored book chapters on the relationship of natural science, armament and disarmament. He teaches a lecture "Physics and Technology of the Verification of Arms-Limitation Agreements".

For the ban, he notes that

The arguments against developing fully autonomous weapons, as they are called, range from moral (“they are evil”) to technical (“they will never be that smart”) to visceral (“they are creepy”).

“This is something people seem to feel at a very gut level is wrong,” says Stephen Goose, director of the arms division of Human Rights Watch, which has assumed a leading role in challenging the dehumanizing of warfare. “The ugh factor comes through really strong.”

He then discusses the three International Humanitarian issues with autonomous robot weapons (i) inability to conform to the principle of distinction; (ii) inability to conform to the principle of Proportionality and (iii) difficulties with accountability with mishaps or war crimes.

He brings out the usual suspect, Ron Arkin, to argue against a ban. Arkin still believes that robots could do better than human because they don’t have emotional responses. Others argue that that is one of the main problems. The funniest comment to Keller’s article was a response to Ron Arkin:

Professor Arkin argues that automation can also make war more humane.” This guy has obviously been a civilian all his life. Only a civilian would believe there is a humane way to kill another human being. Does he get out of the house on a regular basis?

But Arkin’s position in other respects does not now seem that removed from those calling for a ban. “He advocates a moratorium on deployment and a full-blown discussion of ways to keep humans in charge.” The human’s in charge is a subtle change in Arkin’s position that is greatly appreciated. It moves us some way toward the discussions that should be had.

However, without a ban on the development and research on these weapons systems, they are going to end up in the US arsenal. Other countries have not said that they will have a moratorium and so we can expect and arms race that the US will not be able to resist.

In fact in terms of a moratorium, Keller appears to have made an error of interpretation with regards to the recent Department of Defence directive (November 21 2012) ” Last November the Defense Department issued what amounts to a 10-year moratorium on developing them while it discusses the ethical implications and possible safeguards.”

ICRAC member Mark Gubrud picks up on this error in a comment after Keller’s piece:

The DoD Directive (3000.09) does not impose any moratorium. It says that the United States will develop and use autonomous weapons.

Although it draws a line at AW that kill humans autonomously, it does not forbid crossing the line; rather, it sets forth the procedure for doing so. Four sub-cabinet level signatures are required. Other than that, the rules for AW that kill humans are essentially the same as for AW that target materiel, which the Directive approves already.

The directive also approves for immediate development and use “semi-autonomous weapons” which may automatically acquire, track, identify and prioritize potential targets, cue a human operator to their presence, and upon approval, engage them, automatically determining the timing of when to fire.

So, a semi-autonomous weapon system might detect a group of persons, highlight their dim outlines on a screen, and say to the operator “target group identified.” The operator says “engage” and the machine kills them.

Such a system already has every capability needed for full lethal autonomy. It has only been programmed to request approval. One trivial software modification will fix that, if the system doesn’t already have a switch to throw it into full autonomous mode.

DoDD 3000.09 approves such systems for immediate development, acquistion and use.

There is no moratorium; it is a full-speed charge into the unknown.

Nonetheless, Keller is clearly on the right side of the issues and shows a clear understanding: ” It’s a squishy directive, likely to be cast aside in a minute if we learn that China has sold autonomous weapons to Iran”

Although Keller is not optimistic about the chance of us getting a ban on killer robots, he supports it and ICRAC appreciates him for that:

I don’t hold out a lot of hope for an enforceable ban on death-dealing robots, but I’d love to be proved wrong. If war is made to seem impersonal and safe, about as morally consequential as a video game, I worry that autonomous weapons deplete our humanity. As unsettling as the idea of robots’ becoming more like humans is the prospect that, in the process, we become more like robots.

It is well worth reading Bill Keller’s full story and the comments that come afterwards – Smart Robots.

Author information

nsharkey

Noel SharkeyPhD, DSc FIET, FBCS CITP FRIN FRSA is Professor of AI and Robotics and Professor of Public Engagement at the University of Sheffield and was an EPSRC Senior Media Fellow (2004-2010).