CENTER FOR ETHICS AND THE RULE OF LAW​

The Ethics of Autonomous Weapons Systems

November 21 -
 21, 2014

The Conference

Autonomous Weapons Systems (AWS) are defined by the U.S. Department of Defense as “a weapon system(s) that, once activated, can select and engage targets without further intervention by a human operator.” Since the crucial distinguishing mark of human reasoning is the capacity to set ends and goals, the AWS suggests for the first time the possibility of eliminating the human operator from the battlefield.  The development of AWS technology on a broad scale, therefore, represents the potential for a transformation in the structure of war that is qualitatively different from previous military technological innovations. 

In May, the first Meeting of Experts on Lethal Autonomous Weapons Systems was held at the United Nations in Geneva.  The participants recognized the potential of AWS to alter radically the nature of war, as well as a variety of ethical dilemmas such weapons systems raise.  Worldwide concern has been growing about the idea of developing weapons systems that take human beings “out of the loop,” though the precise nature of the ethical challenges to developing such systems, and even possible ethical benefits, have not yet been clearly identified. 

The idea of fully autonomous weapons systems raises a host of intersecting philosophical, psychological, and legal issues. For example, it sharply raises the question of whether moral decision-making by human beings involves an intuitive, non-algorithmic capacity that is not likely to be captured by even the most sophisticated of computers?  Is this intuitive moral perceptiveness on the part of human beings ethically desirable?  Does the automaticity of a series of actions make individual actions in the series easier to justify, as arguably is the case with the execution of threats in a mutually assured destruction scenario?  Or does the legitimate exercise of deadly force should always require a “meaningful human control?”  If the latter is correct, what should be the nature and extent of a human oversight over an AWS? 

Additional questions arise with regard to the very definition of an AWS.  Should the definition focus on the system’s capabilities for autonomous target selection and engagement, or on the human operator’s use of such capabilities?  Should the human operator’s pre-engagement intention have a decisive bearing on the system’s definition as an AWS?  Furthermore, AWS present a unique challenge to the way legal responsibility in combat should be assessed.  If a given AWS is merely applying a set of preprogrammed instructions, then, presumably its designers and operators are the ones morally responsible for its behavior.  But if the AWS in question is a genuine moral discerner in its own right, that appears to shift the locus of responsibility to the automated system itself.  And if this is the case, what are the implications for legal liability?  Who, if anyone, should bear the legal liability for decisions the AWS makes? 

The purpose of this conference is to address such questions by bringing together distinguished scholars and practitioners from the academy, civil society, government service and the military, to engage in two days of constructive discussion and exploration of the moral and legal challenges posed by Autonomous Weapons Systems

Schedule

Friday, Nov. 21, 2014 

8:30 a.m. – 9:15 a.m.Registration & Breakfast – Silverman 147
9:15 a.m. – 9:30 a.m.Welcome Remarks: Dean Wendell Pritchett, Interim Dean, Penn Law and Bill Burke-White, Director of the Perry World House
9:30 a.m. – 10:45 a.m.Session 1 – Human Morality and the Problem of Intelligent MachinesSilverman 147 
Moderator: Claire Finkelstein
10:45 a.m. – 11:15 a.m.Break
11:15 a.m. – 12:30 p.m.Session 2 – The Law and Ethics of Autonomous Weapons SystemsSilverman 147
Moderator: Jens Ohlin
12:30 p.m. – 1:45 p.m. Lunch – Silverman 245, Levy Conference Room
Lunch Keynote – Keeping Weapons Control in Human Hands
Professor Noel Sharkey, University of Sheffield
1:45 p.m.Walk to Van Pelt Library, 3420 Walnut Street, for final session, keynote address, cocktail hour and dinner
2:15 p.m. – 3:30 p.m.Session 3 – The Rationality and Morality of Automaticity
Meyerson Conference Room, Van Pelt Library, 2nd Floor 
Moderator: Duncan MacIntosh
3:30 p.m. – 4:00 p.m.Break
4:00 p.m. – 5:30 p.m.Keynote Panel – Should “Killer Robots” be Banned?

The Class of 78 Pavilion, Van Pelt Library, 6th Floor
Moderator: Professor Bill Burke-White, Director, Perry World House
Participants: Mr. Charles A. Blanchard, Arnold & Porter, LLP; former General Counsel and Chief Ethics Officer for the U.S. Air Force, Ms. Bonnie Docherty Human Rights Watch; Harvard Law School, Major General (ret.) Robert LatiffUniversity of Notre Dame, Professor Wendell WallachYale University
5:30 p.m. – 6:30 p.m.Cocktail Reception – Moelis Terrace, Van Pelt Library, 6th Floor
6:30 p.m. – 8:00 p.m.Dinner –   Seminar Rooms, Van Pelt Library , 6th Floor 

Saturday, Nov. 22, 2014

8:45 a.m. – 9:30 a.m.Breakfast – Silverman 147
9:30 a.m. – 10:45 a.m.Session 4 – Uniquely Human? On Intuition, Mercy and Moral Decision MakingSilverman 147
Moderator: Sharon Lloyd
10:45 a.m. – 11:15 a.m.Break
11:15 a.m. – 12:30 p.m.Session 5 – On the Concept of Meaningful Human ControlSilverman 147
Moderator: Michael Horowitz
12:30 p.m. – 2:15 p.m.Lunch – Silverman 245, Levy Conference Room 
2:15 a.m. – 3:30 p.m.Session 6 – Responsibility for Acts of Intelligent MachinesSilverman 147
Moderator: Kenneth Anderson

Participants

Background Readings


Recent Articles on Autonomous Weapons Systems

Michael C. Horowitz and Paul Scharre
The Morality of Robotic War, NYT, May 26, 2015

Nicholas Carr
Automation Makes Us Dumb, WSJ, Nov 22, 2014

Jane Wakefield,
Robots Face New Creativity Test, BBC News, Nov, 21, 2014

Michael C. Horowitz and Paul Scharre
Killer Robots Save Lives. At Least Some of Them Do, Politico, Nov 19, 2014

John Markoff
Fearing Bombs That Can Pick Whom to Kill, NYT, Nov 11, 2014

On the Concept of Meaningful Human Control

Peter Asaro
On Banning Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal Decision-Making, in Ethics of 21st Century Military Conflict 257-269 (Erica L. Gaston & Patti Tamara Lenard eds. 2012).

Shane Harris
Out of the Loop: The Human-free Future of Unmanned Aerial Vehicles

Ezzio Di Nucci & Fillipo Santoni de Dio
Who’s Afraid of Robots? Fear of Automation and the Ideal of Direct Control, in Roboethics in Film (Fiorella Battaglia & Nathalie Weidenfield eds. forthcoming).

Geoffrey S. Corn
Autonomous Weapon Systems: Managing the Inevitability of ‘Taking the Man out of the Loop’. Social Science Research Network, June 14, 2014.

William Marra and Sonia McNeil
Understanding ‘The Loop’: Regulating the Next Generation of War Machines. Harvard Journal of Law and Public Policy, 36 no. 3 (2013).

Mary Ellen O’Connell
Banning Autonomous Killing: The Legal and Ethical Requirement That Humans Make Near-Time Lethal Decisions

Responsibility for Acts of Intelligent Machines

Daniel C. Dennett
When Hal Kills, Who’s to Blame? Computer Ethics, in Hal’s Legacy: 2001’s Computer as Dream and Reality 351-365 (David Stork ed. 1997).

Peter M. Asaro
A Body to Kick But Still No Soul to Damn: Legal Perspectives on Robotics, in Robot Ethics: The Ethical and Social Implications of Robotics 169-186 (Keith Abney, George Bekey, & Patrick Lin eds., 2011)

Deborah G. Johnson
Technology with No Human Responsibility?

Deborah G. Johnson and Merel Noorman
Responsibility Practices in Robotic Warfare

Deborah G. Johnson and Merel Noorman
Negotiating Autonomy and Responsibility in Military Robots

Merel Noorman
Computing and Moral Responsibility, The Stanford Encyclopedia of Philosophy (June 2014)

Other

Gabriella Blum
An Emerging Threats Essay: Invisible Threats

Bonnie Docherty
Shaking the Foundations: the Human Rights Implications of Killer Robots, Human Rights Watch (2014)

Mark Hagerott
Robots on the Battlefield: Contemporary Issues and Implications for the future, Chapter 3: Robots, Cyber, History, and War

Mark Hagerott
Limiting automation in a cyber-insecure world

Michael Hastings
The Rise of the Killer Drones: How America Goes to War in Secret

Christof Heyns
Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions

Michael C. Horowitz
Coming Next in Military Tech, 70.1 Bulletin of the Atomic Scientists 54 (2014)

Michael C. Horowitz
The Looming Robotics Gap: Why America’s Global Dominance in Military Technology is Starting to Crumble, Foreign Policy, May 5, 2014

Aaron M. Johnson
The Morality of Autonomous Robots, Journal of Military Ethics, Vol. 12, No. 2, 2013.

Deborah G. Johnson and Merel Noorman
Recommendations for Future Development of Artificial Agents

Deborah G. Johnson and Thomas M. Powers
Computers as Surrogate Agents, in Moral Philosophy and Information Technology 251-269 (Jeroen van de Hoen & John Weckert eds. 2008).

John F. Keane and Stephen S. Carr
A Brief History of Early Unmanned Aircraft

Gary E. Marchant et al.
International Governance of Autonomous Military Robots, 12 U. PA L. Rev. 2010 (2011)

John Markoff
Scientists Consider Repurposing Robots for Ebola

John Markoff
Fearing Bombs That Can Pick Whom to Kill

Nicholas Marsh
Defining the Scope of Autonomy: Issues for the Campaign to Stop Killer Robots, Peace Research Institute Oslo (

Markus Wagner
The Dehumanization of International Humanitarian Law: Legal, Ethical, and Political Implications of Autonomous Weapons Systems

Wendell Wallach
Terminating the Terminator: What to do About Autonomous Weapons

Required Readings

Human Morality and the Problem of Intelligent Machines

  1. Thomas Hobbes, Leviathan, Introduction, ed. Curley pp. 3 – 5
  2. John Searle, Minds, Brains, and Programs, 3 The Behavioral and Brain Sciences 417  (1980).
  3. Ronald Arkin, Lethal Autonomous Systems and the Plight of the Non-Combatant,1 AISB Quarterly 137 (2013)
  4. Wendell Wallach, Colin Allen Framing Robot Arms Control, 15 Ethics and Information Technology 125      (2013)

The Law and Ethics of Autonomous Weapons Systems

  1. Kenneth Anderson & Matthew Waxman, Law and Ethics for Autonomous Weapons Systems: Why a Ban Won’t Work and How the Laws of War Can,Policy  Review (Excerpt, pp 8-18)
  2. Mary Ellen O’Connell, Banning Autonomous Killing-The Legal and Ethical Requirement That Humans Make Near-Time Lethal Decisions, in The American Way of Bombing Changing Ethical and Legal Norms from Flying Fortresses to Drones(Matthew Evangelista and Henry Shue editors, 2014) (Excerpted)
  3. Keith Abney, George Bekey, and Patrick Lin, Autonomous Military Robotics: Risk, Ethics, and Design, Cal Poly (2008) (Excerpt, pp 77-90)
  4. Michael W. Lewis, Legal Issues in the Development of Autonomous WeaponsAbstract | Paper
  5. U.S. Department of Defense Directive 3000.09 Autonomy in Weapon Systems (2012)
  6. Additional Protocol I to the Geneva Conventions (Article 1 par. 2, Article 36)
  7. Kevin H. Govern, Discriminant Actions Via (Semi-) Autonomous Weapons Systems: Matching Emergent Capabilities With Legal Prescriptions Abstract

The Rationality and Morality of Automaticity

  1. Duncan MacIntosh, Firing, Forgetting and How  Rule-of-Law Values Require Automation of the Rule of Law; A Defense of the  Use of Autonomous Weapons Systems in War and Peace Abstract | Paper
  2. Gregory Kavka, The Toxin  Puzzle, Analysis, Vol. 43, No.1 (1983), pp.33-36
  3. David Gauthier, Assure and Threaten, Ethics, Vol. 104, No.4 (1994), pp.690-721
  4. Claire Finkelstein,  Acting on an Intentionin REASON, INTENTION AND MORALITY (Gijs Van Donselaar & Bruno Verbeek eds., Ashgate Publishing, 2008) (Excerpt 67-77)
  5. Larry Alexander, The Doomsday Machine:Proportionality, Punishment and Prevention, 63.2 The Monist 199  (1980)

Should “Killer Robots” be Banned?

Uniquely Human? On Intuition, Mercy and Moral Decision Making

  1. M. L. Cummings, The Human Role in Autonomous Weapon Design and Deployment Abstract | Paper
  2. Martha Nussbaum, Equity and Mercy, 22.2 Philosophy and Public Affairs 83 (1993) (Excerpt, pp 83-92)
  3. Human Rights Watch, Shaking The Foundations: The Human Rights Implications of Killer Robots      (Excerpt pp 5-16, 23-24)
  4. Matthias Scheutz & Bertram Malle, May Machines Take Lives to Save Lives? Human Perceptions of Autonomous Robots (with the Capacity to Kill)  Abstract | Paper

On the Concept of Meaningful Human Control

  1. Peter M. Asaro, Jus nascendi, Robotic Weapons & the Martens Clause  Abstract |  Paper
  2. Noel Sharkey, The Human Control of Weapons: a  humanitarian perspective (Draft of chapter to appear in “Autonomous Weapons Systems: Law, Ethics, Policy” edited by Nehal Bhuta, Susanne Beck, Robin Geiss, Claus Kress and Hin Yan Liu.) Abstract | Paper
  3. Peter Asaro, On Banning Autonomous Weapon Systems: Human Rights, Automation, and the      Dehumanization of Lethal Decision-Making, in Ethics of 21st Century Military Conflict Special Issue on New Technologies and Warfare, International Review of the Red Cross94 (886), Summer 2012, pp. 687-709.
  4. Shane Harris, Out of the Loop: The Human-free Future of Unmanned Aerial Vehicles, in Harris, edited by Peter Berkowitz, Hoover Institution, 2012. (Excerpt, pp 1-5, 9-12)

Responsibility for Acts of Intelligent Machines

  1. Jens David Ohlin, Machine Liability & the Combatant’s Stance Abstract |  Paper
  2. Daniel Dennet, When HAL Kills, Who’s to Blame?, in Hal’s Legacy: 2001’s Computer as Dream and      Reality 351-365 (David Stork ed. 1997).
  3. Deborah G. Johnson, “Technology with No Human Responsibility?” J Bus Ethics (2014)

Contact us

For any questions regarding the conference or registration, please contact: Jennifer Cohen at jenn.cohen@appc.upenn.edu

Share The Ethics of Autonomous Weapons Systems on:

LinkedIn
Twitter
Facebook
Reddit
Email
Print
The Ethics of Autonomous Weapons Systems