Peter Asaro – ICRAC

Statement on Ethical Considerations in Open Informal Meeting at UNGA 1st Committee

Peter Asaro — Tue, 13 May 2025 20:45:00 +0000

UNGA Informals on LAWS
ICRAC Statement on Ethical Considerations
Delivered by Prof. Peter Asaro on May 13, 2025

Thank you, Chair. I speak on behalf of the International Committee for Robot Arms Control, or ICRAC, a group of academics, experts, scholars and researchers in computer science, artificial intelligence, robotics, international law, political science, philosophy and ethics. ICRAC is a co-founding member of the Stop Killer Robots Campaign.

We appreciate the organizers of this Informal Meeting including a Session on Ethical Considerations. It has been many years since Ethics has been the primary focus of substantive discussion within the CCW GGE meetings. Yet ethics and morality has provided a valuable basis for international law in the past, and is precisely where we must ground new laws to prohibit and regulate AWS in the near future. That is, in our common shared humanity, and principles which transcend human laws, particularly human dignity in a deep sense as discussed by Prof. Chengeta, and ethical decisions as discussed by the Representative of the Holy See.

Whenever violent force is used, there are risks involved. But merely managing those risks is not sufficient to meet the requirements for morally justifiable killing. Understanding the reasons and the potential consequences for the use of force is required for its justification. It has been argued that AWS may be highly accurate and precise in their use of force, but these are not sufficient to meet the requirements for the ethically discriminant use of force, and do not begin to address the requirements of the proportionate use of force.

Following the outlines of the two-tiered approach advanced by the ICRC, regulated AWS would be permitted to target autonomously. In these limited cases, more specifically cases where the target is a military object by nature, such as military vehicles and installations, automated targeting must still be carefully regulated to ensure that humans can safely supervise those systems.

But as soon as we start considering civilian objects, even those which might be used for military purposes and might be lawfully targeted under IHL, we must not permit their targeting by automated processes. The moral argument that leads to this conclusion is clear. It may be tempting to think that we can automate proportionality decisions–how much force is needed, or how much risk is acceptable, or how much collateral harm to civilians might be acceptable relative to a military objective. But the nature of proportionality judgments is fundamentally moral.

These decisions are inherently about values–the value of a target to a military objective, the value of a military objective to an operation and an overall strategy; the value of civilian infrastructure to a family, a community, a country; the value of a natural environment; and above all the value of human lives and the cost of taking those lives. They are also about duties, our duties to protect, our duties to each other.

These values are not intrinsically numerical or quantitative in nature, and assigning them such values in a computer program is arbitrary at best. Computers do not “understand” in any meaningful sense. They represent the world through mathematical abstractions that we design and understand, and from which we assign and seek meaning. Worse, training an algorithm to “learn” these values from a dataset is to abdicate any human responsibility in establishing the values represented in the systems, including the value of human life and the necessary conditions of human flourishing.

These are moral values, only understood through the lived experience of human life, moral reflection, and ethical development. In those limited cases where the decision to end a human life can be morally justified, it must be made by a moral agent who truly understands these values. Any life lost by the decision of an algorithm is, by definition, taken arbitrarily. ICRAC appreciates the work of the CCW GGE and this section of latest draft of the Chair’s Rolling Text:

States should ensure context-appropriate human judgement and control in the use of
LAWS, through the following measures … [which] … includes ensuring assessment of legal
obligations and ethical considerations by a human, in particular, with regard to the effects
of the selection and engagement functions.

The ethical considerations of the use of force must remain a matter of human judgement. We must not eliminate ethical considerations altogether by delegating them to machines wholly incapable of grasping such considerations. Human dignity requires that we consider a human as human–no machine can do this for us.

Similarly for anti-personnel AWS, in order to design systems to autonomously target people, it would be necessary to create digital representations of people, or target profiles. The same moral logic applies here.

While from a legal perspective, it could be argued that unmounted infantry are military objects by nature, and can pose a threat just as a tank does. But there is an important moral difference between targeting people directly, versus targeting a tank, and accepting that people inside it may be killed. People are not to be treated as objects, but always as moral subjects.

The aim of war, and the moral justification of killing in war, depends critically on using force to diminish the ability of your adversary to use force against you. The ultimate aim is not to harm or kill the enemy directly, this is only a means to an end, namely the end of hostilities. Targeting a human directly is to make the destruction of a human a goal in itself, rather than the true goal of eliminating the threat they pose. This might sound like a minor distinction, but by making the targeting and killing of humans the goal of a machine, rather than the elimination of military threats, we stand to vastly undermine human dignity.

By designing systems to target people directly, we essentially and effectively “pre-authorize” the moral judgement to take their lives. By pre-authorizing the killing of humans, and making personnel the targets of autonomous weapons, we would fundamentally violate and diminish human dignity. If we accept that a soldier on the battlefield can be directly targeted, without a human moral judgement or moral justification, then we make it more acceptable to do so in other contexts as well.

When we violate human dignity, it is not just the immediate victim who loses their dignity. All of humanity suffers from this loss. This is why we feel such moral disgust at the injustices of slavery, and torture, and the dropping of bombs on children–these atrocities undermine our collective dignity as human beings and offend our moral sensibility.

While the use of violent force against unjust aggression is sometimes necessary, it is our moral responsibility to ensure that force is used justly. The only way to ensure that force is used justly is through moral judgement, and this requires a moral agent. Machines and automated algorithms, however sophisticated they may appear, are not moral agents, and are not capable of moral judgements–only thin and arbitrary approximations. We must not delegate our morality to machines, as doing so threatens the very essence of our human dignity.

To quote the wise words of Christof Heyns, “War without reflection is mechanical slaughter.”

Author information

Peter Asaro

Dr. Peter Asaro is a philosopher of science, technology and media. His work examines the interfaces between social relations, human minds and bodies, artificial intelligence and robotics, and digital media. His current research focuses on the social, cultural, political, legal and ethical dimensions of military robotics and UAV drones, from a perspective that combines media theory with science and technology studies. He has written widely-cited papers on lethal robotics from the perspective of just war theory and human rights. Dr. Asaro's research also examines agency and autonomy, liability and punishment, and privacy and surveillance as it applies to consumer robots, industrial automation, smart buildings, and autonomous vehicles. His research has been published in international peer reviewed journals and edited volumes, and he is currently writing a book that interrogates the intersections between military robotics, interface design practices, and social and ethical issues. Dr. Asaro has held research positions at the Center for Cultural Analysis at Rutgers University, the HUMlab of Umeå University in Sweden, and the Austrian Academy of Sciences in Vienna. He has also developed technologies in the areas of virtual reality, data visualization and sonification, human-computer interaction, computer-supported cooperative work, artificial intelligence, machine learning, robot vision, and neuromorphic robotics at the National Center for Supercomputer Applications (NCSA), the Beckman Institute for Advanced Science and Technology, and Iguana Robotics, Inc., and was involved in the design of the natural language interface for the Wolfram|Alpha computational knowledge engine (winner of the 2010 SXSW Web Interactive Award for Technical Achievement), for Wolfram Research. He is currently working on an Oral History of Robotics project that is funded by the IEEE Robotics and Automation Society and the National Endowment for the Humanities Office of Digital Humanities. Dr. Asaro received his PhD in the History, Philosophy and Sociology of Science from the University of Illinois at Urbana-Champaign, where he also earned a Master of Arts from the Department of Philosophy, and a Master of Computer Science from the Department of Computer Science.

Statement on Technical Considerations in Open Informal Meeting at UNGA 1st Committee

Peter Asaro — Tue, 13 May 2025 20:40:00 +0000

UNGA LAWS Informals
ICRAC Statement on Technical Considerations
Delivered by Prof. Peter Asaro, 13 May 2025

Thank you Chair. I speak on behalf of the International Committee for Robot Arms Control, or ICRAC, a co-founding member of the Stop Killer Robots Campaign.

ICRAC has many concerns about the development and use of autonomous weapons and the accelerated production and promotion of these systems by private technology companies. Far from being a technically inevitable and practically necessary, autonomous weapons pose a considerable risk to global stability and security, and are likely to cause more civilian harm, rather than less. As a group of scholars with expertise in relevant domains, including robotics, AI, and digital information systems, we strongly urge caution.

We are concerned that the technology that underpins the functionalities of AWS is dangerously unsuitable for the complex and dynamic contexts of conflict. Specifically, the AI element in AWS poses considerable risks. Testing such systems is difficult and time-consuming, and the tools and methods for the verification and validation of AI systems do not yet exist, if they are possible at all. The questionable reliability of prediction based on historical data when applied to dynamically unfolding situations in conflict raises further questions regarding the validity and legality of using AI supported AWS.

At best, AI supported systems are only as good as the data on which they are trained on and appropriate, comprehensive and up-to-date data is hard to come by in contested conflict spaces. AI systems need frequent updates to remain relevant and functional, but with each substantial update, vital systems-aspects may become compromised requiring further verification and validation.

As we heard from these presenters, it is a well-known fact in technology and industry circles that AI systems remain unproven in terms of reliability for safety-critical situations and complex situations such as armed conflict. They are known to give inaccurate outputs, and newer generative AI systems, which are likely to find their way into the wider AWS environment, are known to hallucinate – that is they give false or misleading output which is difficult to distinguish from accurate results. In the case of generative AI, this behavior is guaranteed by its technical architecture and these types of errors can only be managed not eliminated. When AI experts and those that make the technologies used in AWS raise alarms about the inadequacies of AWS for conflict, we should listen.

We are concerned that the technical characteristics of AWS pose a considerable risk in enabling uncontrolled escalation and conflict at speed. Escalation from crisis to war, or escalating a conflict to a higher level of violence, could come about due to erroneous indications of attack or a simple sensor or computer error. Unpredictable systems, and systems which operators cannot understand or explain, will give leaders false impressions of their capabilities, leading to overconfidence or encouraging pre-emptive
attacks. This will lead to greater global instability and insecurity.

Finally, there are operational risks posed by AWS in that they give the illusion that such weapons are more precise and accurate, and will therefore inflict less harm. The extensive use of AI in current conflicts has given us an indication that the contrary might be the case. This is particularly so for database-driven systems that generate targeting lists faster than humans can evaluate and verify the lawfulness of targets. The technical capacity for precision or accuracy is not a warrant for discrimination or proportionality in use. Unless we establish clear legally binding limitations on AWS, there is no safeguard that systems that prioritize speed and scale are not used in an indiscriminate and disproportional manner, either intentionally or because humans have abdicated their judgement to a machine.

Thank you.

Author information

Peter Asaro

ICRAC Releases New Report on Meaningful Human Control

Peter Asaro — Tue, 20 Aug 2019 08:38:40 +0000

ICRAC Members Daniele Amoroso and Guglielmo Taburrini have completed a new ICRAC Working paper #4 on “What makes human control over weapons “Meaningful”? The paper was prepared for distribution at the August 2019 meeting of the United Nations CCW GGE on Lethal Autonomous Weapons. The paper can be downloaded from our Resources Page, along with ICRAC’s other working papers.

Author information

Peter Asaro

ICRAC Statement at Informal Consultations of the August 2019 CCW GGE on LAWS

Peter Asaro — Tue, 20 Aug 2019 08:29:57 +0000

Statement delivered by ICRAC Vice-chair Peter Asaro to the CCW GGE Informal Session on the Chair’s Non-Paper, August 19, 2019.

“The International Committee for Robot Arms Control, which is a member of the Campaign to Stop Killer Robots, would like to thank the Chair for this Draft, and make the following comments and requests.

First and most importantly, we would urge the Chair to set a higher bar for the goals of this GGE and the discussions of the next two years. In particular, we would like to see the set goal to be a legally binding instrument, and not merely a “Normative Framework” of an unknown or unstated legal status. This GGE can and should begin discussing what a legally binding instrument that could effectively regulate autonomy in weapons systems might look like. Normativity could also imply ethical and moral norms, and we would welcome a broader discussion of the ethical and moral issues raised by autonomous weapons, particularly with respect to human dignity.

Further, we would like to remind the Chair that the “Guiding Principles” were developed to guide discussions of this body over the past few years, and were never meant to be a goal or outcome of those discussions. We would like to see a more substantive outcome of the current GGE.

Finally, we are concerned that the current draft does not mention “human control” much less “meaningful human control” or its other variants. This is despite the fact that many States, as well as civil society, have repeatedly expressed the view that human control is central to both understanding and regulating autonomy in weapons systems. Towards this end, ICRAC has produced a new white paper entitled “What makes human control over weapons “Meaningful”? You will find copies of this new report in the back of the room tomorrow. In it you will find a rigorous analysis of the requirements for human control in weapons, which could provide useful concepts for the elements of a treaty, including the positive obligation on states to ensure that weapons have the necessary elements of control to ensure accountable and responsible use of weapons under international law. And we hope the Chair will stand by his recent tweet, and allow this document to inform discussions of the Legal, Technical and Military work streams, as well as a much needed ethical discussion that cuts across all three.

We hope that tomorrow’s formal discussions are productive, and will continue to urge this body to work on the substantive concepts necessary to build a legally binding instrument.”

Author information

Peter Asaro

ICRAC statement at the March 2019 CCW GGE

Peter Asaro — Tue, 26 Mar 2019 15:50:46 +0000

As delivered by Prof. Peter Asaro, March 26, 2019.

ICRAC has been pleased to hear states shift their focus away from definitions of the technologies of autonomous weapons systems and move towards discussing restriction of their use with regards to how they should be controlled. Of course, by definition, if states wanted genuine meaningful human control of weapons systems, they would not be using autonomous weapons systems. And (as an aside) we should not forget the scientifically recognized limitations of the technology or the foreseeable threats to global security such weapons pose.

We are also pleased with the statements and working papers beginning to examine the requirements for human control and planning in military systems. While this can be multifaceted, we must not let the complexity of military planning throw a smoke screen over the core issues of the meaningful human assessment of all targets, their legitimacy and the proportionate use of force.

We are glad that we see the beginnings of a more nuanced approach to the control of weapons systems that cannot be captured by gross terms such as in-the-loop, on-the-loop, the broader loop, human oversight, and appropriate levels of human judgement. However, these terms continue to insinuate themselves in military, political and defence contractor’s narratives outside of the CCW. We welcome the suggestion of the IPRAW report to distinguish control-by-design and control-in-use—acknowledging that ultimate responsibility for the use of force lies in the specific context of its use.

As a scientific and scholarly group, our focus is on how we can make control effective and ensure that operators, commanders and planners are making clear judgements about the validity of every attack at the time of that attack.

To do this we need to move away from blanket terms and examine in detail how humans interact with automated machinery. As we have pointed out before, there has been more than 30 years of scientific research on human supervisory control of machinery and more than 100 years of research on the psychology of human reasoning. Ignoring the science for sake of expediency could lead us down a path to a humanitarian disaster.

The scientific approach is not mutually exclusive to an examination of the military control of weapons and the many lessons to be learned for current methods. Indeed, we applaud the UK’s paper on human control in 2018 and that of the Netherlands and others this year. We may not agree with all of the detail, but it is what we have urged all of the high contracting parties to bring to the table.

This combination of work can help us to design human-machine interfaces that allow weapons to be controlled in a manner that is fully compliant with international law and the principle of humanity.

First, there should be a focus on what the human operator MUST do in the targeting cycle. This is control by use which is governed by targeting rules under International Humanitarian Law and International Human Rights Law, which were well articulated by the ICRC in their statement this morning. Further, international law rules that apply after the use of weapons – such as those that relate to human responsibility – must be satisfied.

Second, the design of weapon systems must render them INCAPABLE of operating without meaningful human control. This is control by design, which is governed by international weapons law. In terms of international weapons law, if the weapon system, by its design, is incapable of being sufficiently controlled in terms of the law, then such a weapon should be prohibited.

We need further discussion of the details of human-machine interfaces, the distribution of responsibility in the targeting cycle, and how their design can ensure IHL and IHRL compliance. Such details need not be the substance of a treaty, and we must resist being caught up in the weeds of process. We support German’s goal of finding a shared understanding of the principles of human control that apply to all weapons systems now and in the future, regardless of context, planning or process. This is not different from the normal processes that operate in science. One of the goals of science is to reduce the complexity of the world to simple theories or principles that capture all of the experimental data. In other words, we create abstractions of the details that are firmly coupled with and informed by the details. As Einstein once said, explanations should be a simple as possible but no simpler. “Human in the loop” and its variants fall under the too simple category. Detailed accounts of every weapon type and how it is controlled in every context is far too complex.

Let me give you an example of an abstraction with three conditions that could make a good starting point for discussions on the control of weapons systems. I have said this before but clearly there is no prohibition on repeating yourself in this room.

a human commander (or operator) will have full contextual and
situational awareness of the target area for each and every attack and be able
to perceive and react to any change or unanticipated situations that may have
arisen since planning the attack.
there will be active cognitive participation in every attack with
sufficient time for deliberation on the nature of any target, its significance
in terms of the necessity and appropriateness of attack, and likely incidental
and possible accidental effects of the attack and
there will be a means for the rapid suspension or abortion of
every attack.

These are general principles that could provide a starting point for discussion by states in the context of negotiating a legally binding treaty that clearly articulates the legal obligations of human control.

Author information

Peter Asaro

ICRAC Statement at the April 2018 CCW GGE

Peter Asaro — Mon, 09 Apr 2018 13:56:04 +0000

International Committee for Robot Arms Control
Statement to the UN GGE Meeting 2018
Delivered by Dr Thompson Chengeta, on 9 April 2018

Mr. Chairperson,

I speak on behalf of the International Committee for Robot Arms Control [ICRAC], a founding member of the Campaign to Stop Killer Robots. Ambassador Gill, we thank you for your important work. Mr Chairperson, we are going to focus here on four points:

FIRST, a ban on LAWS will have no negative impact on the development of socially beneficial uses of autonomy, robotics or artificial intelligence. In fact, such a ban will direct more resources and specialists to work on humanitarian and beneficial applications.

SECOND, human control of weapon systems is a critical key component of the present discussions. It does not matter what name or term is used to describe human control, what is imperative is that we make sure that human control is consistent with applicable legal, ethical and moral standards.

THIRD, human input in the making of judgements to use violent force is at the centre of legal, ethics and moral standards pertaining to human responsibility for use of such force. No matter how attractive, if a proposed definition of human control does not resolve the accountability gap challenge, then such a proposal is legally inadequate. To that end, States should ask the question: What is the Legally Required Level of Human Control at each “touch point” in the human-machine interaction chain? At every step in the development, deployment, targeting and use of a weapon system, there is an obligation to ensure that the system is both capable of being used in compliance with applicable legal norms.

FOURTH, Poland and ICRC Working Papers’ emphasis on ethics and reassertion of the Principle of Non-Delegation of the Authority to Kill to non-human mechanisms is worth noting. Dictates of public conscience must always take precedence over any short-term advantage that might be gained from autonomous technologies. Furthermore, respect for human rights and human dignity, even within armed conflict, is a moral imperative recognized by the UN and the CCW. ICRAC reiterates the spirit of the Martens Clause—that morality can provide a strong basis for new law.

Finally, human control over critical functions of weapon systems and a ban on fully autonomous weapon systems are two sides of the same coin. States are urged to focus on the requirement of human control rather than technical definitions of autonomy. Further, States must move towards negotiation of a legally binding instrument on this issue.

Mr Chairperson, I thank you.

Author information

Peter Asaro

ICRAC Video: Peaceful Uses of Robotics and Banning LAWS

Peter Asaro — Thu, 12 Nov 2015 17:36:45 +0000

Stop the Killer Robots from Kamille Rodriguez on Vimeo.

The video explains that a ban on killer robots would not have negative impacts on the development of other robotics applications and research.

It was created for ICRAC by digital animation artist Kamille Rodriguez:
http://www.kamillerodriguez.com/

Author information

Peter Asaro

ICRAC second statement on Overarching Issues, 2015 CCW Expert Meeting

Peter Asaro — Fri, 17 Apr 2015 22:35:01 +0000

On Thursday April 16, ICRAC’s Dr. Peter Asaro delivered the following statement to the informal meeting of experts at the United Nations in Geneva:

Thank you Mr. Chair.

In response to China’s question, a number of security experts have pointed to the danger of mass attack by conventional weapons on strategically important installations. Autonomous weapons would open new modes for such attacks, and would heighten the risk of preemptive attacks up to and including nuclear weapons.

I would like comment on the survey results presented by Professor Horowitz, and the conclusions he asks us to draw from them.

You are correct that public opinion is not equivalent to public conscience. Unfortunately, we know all too well that the desires of the majority can often result in the injustices towards a minority. And while opinions may shift, the underlying moral principles upon which they are built do not, namely the moral foundations of human rights and human dignity, and respect for the law.

It is difficult to determine the exact causes for difference in the results between your survey and previous ones without knowing the methodological details of your survey. You claim that the differences you found in your results are due to the presentation of additional contextual information. Yet, the survey you are comparing it to, which was presented at a side event on Tuesday by the OpenRoboethics Initative, actually provided more situational and contextual information than your survey, and compared responses to more alternatives than your survey, including remote operated weapons such as drones, and questions of offensive and defensive uses.

More importantly, for public opinion to serve as an indicator of the public conscience, it is necessary to clearly demonstrate its relationship to underlying moral values.

In the survey conducted by the Open Roboethics Initiative there were separate questions investigating the reasons underlying the general opinions elicted from respondents. Concurring somewhat with your findings, the strongest reason indicated for supporting the use of autonomous weapons was to save the lives of the soldiers of your own nation. Furthermore, the primary reason given by the overwhelming majorities of multiple nations for rejecting the use of autonomous weapons is that it was simply wrong for machines to make life and death decisions, followed by reasons such as the fears of the risks of errors and failures, and the risks from the proliferation of these weapons.

It would thus seem that a reasonable explanation of your survey results is that you have conflated the desire to protect troops, and desire for self-defense with participants opinions on autonomous weapons. Rather than offering informational context, the structure of the question has combined multiple factors and skewed the responses.

I would thus respectfully submit that in addition to public opinion polls, we should confer with the legal and ethical experts we heard from this morning and well as our own individual moral conscience.

Author information

Peter Asaro

Everyone is a target

Peter Asaro — Mon, 12 Aug 2013 23:11:47 +0000

A new short documentary (8 minutes) by Amy Kohn – Autonomous Weapons: everyone is a target – features members of ICRAC giving the reason why we need to move forward with an international legally binding treaty to prohibit research, use and development of autonomous weapons – weapons that once activated can select targets and kill them without further human supervisory control.

ICRAC members discuss the difficulties of compliance with International Humanitarian law using autonomous weapons as well as international security issues and the moral red line.

final video – robots from Amy Kohn on Vimeo.

Author information

Peter Asaro