The Conference
Autonomous Weapons Systems (AWS) are defined by the U.S. Department of Defense as “a weapon system(s) that, once activated, can select and engage targets without further intervention by a human operator.” Since the crucial distinguishing mark of human reasoning is the capacity to set ends and goals, the AWS suggests for the first time the possibility of eliminating the human operator from the battlefield. The development of AWS technology on a broad scale, therefore, represents the potential for a transformation in the structure of war that is qualitatively different from previous military technological innovations.
In May, the first Meeting of Experts on Lethal Autonomous Weapons Systems was held at the United Nations in Geneva. The participants recognized the potential of AWS to alter radically the nature of war, as well as a variety of ethical dilemmas such weapons systems raise. Worldwide concern has been growing about the idea of developing weapons systems that take human beings “out of the loop,” though the precise nature of the ethical challenges to developing such systems, and even possible ethical benefits, have not yet been clearly identified.
The idea of fully autonomous weapons systems raises a host of intersecting philosophical, psychological, and legal issues. For example, it sharply raises the question of whether moral decision-making by human beings involves an intuitive, non-algorithmic capacity that is not likely to be captured by even the most sophisticated of computers? Is this intuitive moral perceptiveness on the part of human beings ethically desirable? Does the automaticity of a series of actions make individual actions in the series easier to justify, as arguably is the case with the execution of threats in a mutually assured destruction scenario? Or does the legitimate exercise of deadly force should always require a “meaningful human control?” If the latter is correct, what should be the nature and extent of a human oversight over an AWS?
Additional questions arise with regard to the very definition of an AWS. Should the definition focus on the system’s capabilities for autonomous target selection and engagement, or on the human operator’s use of such capabilities? Should the human operator’s pre-engagement intention have a decisive bearing on the system’s definition as an AWS? Furthermore, AWS present a unique challenge to the way legal responsibility in combat should be assessed. If a given AWS is merely applying a set of preprogrammed instructions, then, presumably its designers and operators are the ones morally responsible for its behavior. But if the AWS in question is a genuine moral discerner in its own right, that appears to shift the locus of responsibility to the automated system itself. And if this is the case, what are the implications for legal liability? Who, if anyone, should bear the legal liability for decisions the AWS makes?
The purpose of this conference is to address such questions by bringing together distinguished scholars and practitioners from the academy, civil society, government service and the military, to engage in two days of constructive discussion and exploration of the moral and legal challenges posed by Autonomous Weapons Systems
Schedule
Friday, Nov. 21, 2014
| 8:30 a.m. – 9:15 a.m. | Registration & Breakfast – Silverman 147 |
| 9:15 a.m. – 9:30 a.m. | Welcome Remarks: Dean Wendell Pritchett, Interim Dean, Penn Law and Bill Burke-White, Director of the Perry World House |
| 9:30 a.m. – 10:45 a.m. | Session 1 – Human Morality and the Problem of Intelligent MachinesSilverman 147 Moderator: Claire Finkelstein |
| 10:45 a.m. – 11:15 a.m. | Break |
| 11:15 a.m. – 12:30 p.m. | Session 2 – The Law and Ethics of Autonomous Weapons SystemsSilverman 147 Moderator: Jens Ohlin |
| 12:30 p.m. – 1:45 p.m. | Lunch – Silverman 245, Levy Conference Room Lunch Keynote – Keeping Weapons Control in Human Hands Professor Noel Sharkey, University of Sheffield |
| 1:45 p.m. | Walk to Van Pelt Library, 3420 Walnut Street, for final session, keynote address, cocktail hour and dinner |
| 2:15 p.m. – 3:30 p.m. | Session 3 – The Rationality and Morality of Automaticity Meyerson Conference Room, Van Pelt Library, 2nd Floor Moderator: Duncan MacIntosh |
| 3:30 p.m. – 4:00 p.m. | Break |
| 4:00 p.m. – 5:30 p.m. | Keynote Panel – Should “Killer Robots” be Banned? The Class of 78 Pavilion, Van Pelt Library, 6th Floor Moderator: Professor Bill Burke-White, Director, Perry World House Participants: Mr. Charles A. Blanchard, Arnold & Porter, LLP; former General Counsel and Chief Ethics Officer for the U.S. Air Force, Ms. Bonnie Docherty Human Rights Watch; Harvard Law School, Major General (ret.) Robert Latiff, University of Notre Dame, Professor Wendell Wallach, Yale University |
| 5:30 p.m. – 6:30 p.m. | Cocktail Reception – Moelis Terrace, Van Pelt Library, 6th Floor |
| 6:30 p.m. – 8:00 p.m. | Dinner – Seminar Rooms, Van Pelt Library , 6th Floor |
Saturday, Nov. 22, 2014
| 8:45 a.m. – 9:30 a.m. | Breakfast – Silverman 147 |
| 9:30 a.m. – 10:45 a.m. | Session 4 – Uniquely Human? On Intuition, Mercy and Moral Decision MakingSilverman 147 Moderator: Sharon Lloyd |
| 10:45 a.m. – 11:15 a.m. | Break |
| 11:15 a.m. – 12:30 p.m. | Session 5 – On the Concept of Meaningful Human ControlSilverman 147 Moderator: Michael Horowitz |
| 12:30 p.m. – 2:15 p.m. | Lunch – Silverman 245, Levy Conference Room |
| 2:15 a.m. – 3:30 p.m. | Session 6 – Responsibility for Acts of Intelligent MachinesSilverman 147 Moderator: Kenneth Anderson |
Participants
American University
Georgia Institute of Technology
The New School for Public Engagement; Stanford Law School
Department of State
Arnold & Porter, LLP; former General Counsel and Chief Ethics Officer for the U.S. Air Force
Professor William W. Burke-White
University of Pennsylvania Law School
Federal Systems Inc
Duke University
Human Rights Watch; Harvard Law School
University of Pennsylvania Law School
CAPT (Ret) Robert G. Fuller, Jr.
Former Navy Jag and Penn Law Alum
University of Pennsylvania
Ave Maria School of Law
University of Minnesota Law School
CEO of NPR
CAPT USN (Ret) Professor Mark Hagerott
US Naval Academy Center for Cyber Studies
International Committee of the Red Cross
University of Pennsylvania
Department of Defense
University of Virginia
University of Pennsylvania
University of Notre Dame
University of Southern California
Dalhousie University
Professor Christopher W. Morris
University of Maryland
Cornell University School of Law
Professor Daniel D. Perlmutter
University of Pennsylvania
University of Arizona
University of Pennsylvania
University of Pennsylvania
Center for a New American Security
Tufts School of Engineering
University of Sheffield
University of Miami
Yale University
University of Pennsylvania
Center for Translational Medicine; Physicians for Human Rights
Council for a Livable World
Background Readings
Recent Articles on Autonomous Weapons Systems
Michael C. Horowitz and Paul Scharre
The Morality of Robotic War, NYT, May 26, 2015
Nicholas Carr
Automation Makes Us Dumb, WSJ, Nov 22, 2014
Jane Wakefield,
Robots Face New Creativity Test, BBC News, Nov, 21, 2014
Michael C. Horowitz and Paul Scharre
Killer Robots Save Lives. At Least Some of Them Do, Politico, Nov 19, 2014
John Markoff
Fearing Bombs That Can Pick Whom to Kill, NYT, Nov 11, 2014
Human Morality and the Problem of Intelligent Machines
Ronald Arkin
Lethal Autonomous Systems and the Plight of the Non-combatant
John Searle
Minds, Brains, and Programs, 3 The Behavioral and Brain Sciences 417 (1980).
Robert Sheckley
Watchbird, Galaxy, Feb. 1953, at 74
David Cole
The Chinese Room Argument, The Stanford Encyclopedia of Philosophy (June 2014)
The Law and Ethics of Autonomous Weapons Systems
Keith Abney, Patrick Lin and George Bekey.
Autonomous Military Robotics: Risk, Ethics, and Design, Cal Poly (2008)
Kenneth Anderson, Matthew C. Waxman
Law and Ethics for Autonomous Weapons Systems: Why a Ban Won’t Work and How the Laws of War Can, Policy Review 2013, at 1.
U.S. Department of Defense (DoD) policy on autonomy in weapons: DoD Directive 3000.09
Autonomy in Weapon Systems, (2012)
Additional Protocol I to the Geneva Conventions (Article 1 par. 2, Article 36)
Mary Ellen O’Connell
The Legal and Ethical Requirement That Humans Make Near-Time Lethal Decisions, in The American Way of Bombing Changing Ethical and Legal Norms from Flying Fortresses to Drones (Matthew Evangelista and Henry Shue editors, 2014)
Colin Allen, Wendell Wallach
Framing Robot Arms Control, 15 Ethics and Information Technology 125 (2013)
Chantal Grut
The Challenge of Autonomous Lethal Weapon Systems to International Law, 18.1 Journal of Conflict and Security Law 5 (2013)
Michael N. Schmitt
Autonomous Weapon Systems and International Humanitarian Law: Reply to the Critics, Harvard National Security Journal Features (Feb. 5, 2013, 2:07 PM)
UK Ministry of Defense, Joint Doctrine Note 2/11
The UK Approach to Unmanned Aircraft Systems
The Rationality and Morality of Automaticity
Larry Alexander
The Doomsday Machine: Proportionality, Punishment and Prevention, 63.2 The Monist 199 (1980)
David Gauthier
Assure and Threaten, 690-721 Ethics, 104,4 (1994)
Gregory Kavka
The Toxin Puzzle, 33-36 Analysis 43,1 (1983)
Should “Killer Robots” be Banned?
Bonnie Docherty
The Trouble with Killer Robots: Why we need to ban fully autonomous weapons systems, before it’s too late
Uniquely Human? On Intuition, Mercy and Moral Decision Making
Martha Nussbaum
Equity and Mercy, 22.2 Philosophy and Public Affairs 83 (1993)
Deborah G. Johnson and Merel Noorman
Artefactual Agency and Artefactual Moral Agency
Noel E. Sharkey
The Evitability of Autonomous Robot Warfare, 94 International Review of the Red Cross 787 (2012)
On the Concept of Meaningful Human Control
Peter Asaro
On Banning Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal Decision-Making, in Ethics of 21st Century Military Conflict 257-269 (Erica L. Gaston & Patti Tamara Lenard eds. 2012).
Shane Harris
Out of the Loop: The Human-free Future of Unmanned Aerial Vehicles
Ezzio Di Nucci & Fillipo Santoni de Dio
Who’s Afraid of Robots? Fear of Automation and the Ideal of Direct Control, in Roboethics in Film (Fiorella Battaglia & Nathalie Weidenfield eds. forthcoming).
Geoffrey S. Corn
Autonomous Weapon Systems: Managing the Inevitability of ‘Taking the Man out of the Loop’. Social Science Research Network, June 14, 2014.
William Marra and Sonia McNeil
Understanding ‘The Loop’: Regulating the Next Generation of War Machines. Harvard Journal of Law and Public Policy, 36 no. 3 (2013).
Mary Ellen O’Connell
Banning Autonomous Killing: The Legal and Ethical Requirement That Humans Make Near-Time Lethal Decisions
Responsibility for Acts of Intelligent Machines
Daniel C. Dennett
When Hal Kills, Who’s to Blame? Computer Ethics, in Hal’s Legacy: 2001’s Computer as Dream and Reality 351-365 (David Stork ed. 1997).
Peter M. Asaro
A Body to Kick But Still No Soul to Damn: Legal Perspectives on Robotics, in Robot Ethics: The Ethical and Social Implications of Robotics 169-186 (Keith Abney, George Bekey, & Patrick Lin eds., 2011)
Deborah G. Johnson
Technology with No Human Responsibility?
Deborah G. Johnson and Merel Noorman
Responsibility Practices in Robotic Warfare
Deborah G. Johnson and Merel Noorman
Negotiating Autonomy and Responsibility in Military Robots
Merel Noorman
Computing and Moral Responsibility, The Stanford Encyclopedia of Philosophy (June 2014)
Reports and Memoranda
International Committee of the Red Cross
Autonomous Weapons Systems Technical, Military, Legal and Humanitarian Aspects (November 1, 2014)
International Committee of the Red Cross
Expert Meeting Report: Autonomous Weapon Systems, Technical, Military, Legal and Humanitarian Aspects (Geneva, Switzerland 26 to 28 March 2014)
Other
Gabriella Blum
An Emerging Threats Essay: Invisible Threats
Bonnie Docherty
Shaking the Foundations: the Human Rights Implications of Killer Robots, Human Rights Watch (2014)
Mark Hagerott
Robots on the Battlefield: Contemporary Issues and Implications for the future, Chapter 3: Robots, Cyber, History, and War
Mark Hagerott
Limiting automation in a cyber-insecure world
Michael Hastings
The Rise of the Killer Drones: How America Goes to War in Secret
Christof Heyns
Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions
Michael C. Horowitz
Coming Next in Military Tech, 70.1 Bulletin of the Atomic Scientists 54 (2014)
Michael C. Horowitz
The Looming Robotics Gap: Why America’s Global Dominance in Military Technology is Starting to Crumble, Foreign Policy, May 5, 2014
Aaron M. Johnson
The Morality of Autonomous Robots, Journal of Military Ethics, Vol. 12, No. 2, 2013.
Deborah G. Johnson and Merel Noorman
Recommendations for Future Development of Artificial Agents
Deborah G. Johnson and Thomas M. Powers
Computers as Surrogate Agents, in Moral Philosophy and Information Technology 251-269 (Jeroen van de Hoen & John Weckert eds. 2008).
John F. Keane and Stephen S. Carr
A Brief History of Early Unmanned Aircraft
Gary E. Marchant et al.
International Governance of Autonomous Military Robots, 12 U. PA L. Rev. 2010 (2011)
John Markoff
Scientists Consider Repurposing Robots for Ebola
John Markoff
Fearing Bombs That Can Pick Whom to Kill
Nicholas Marsh
Defining the Scope of Autonomy: Issues for the Campaign to Stop Killer Robots, Peace Research Institute Oslo (
Markus Wagner
The Dehumanization of International Humanitarian Law: Legal, Ethical, and Political Implications of Autonomous Weapons Systems
Wendell Wallach
Terminating the Terminator: What to do About Autonomous Weapons
Required Readings
Human Morality and the Problem of Intelligent Machines
- Thomas Hobbes, Leviathan, Introduction, ed. Curley pp. 3 – 5
- John Searle, Minds, Brains, and Programs, 3 The Behavioral and Brain Sciences 417 (1980).
- Ronald Arkin, Lethal Autonomous Systems and the Plight of the Non-Combatant,1 AISB Quarterly 137 (2013)
- Wendell Wallach, Colin Allen Framing Robot Arms Control, 15 Ethics and Information Technology 125 (2013)
The Law and Ethics of Autonomous Weapons Systems
- Kenneth Anderson & Matthew Waxman, Law and Ethics for Autonomous Weapons Systems: Why a Ban Won’t Work and How the Laws of War Can,Policy Review (Excerpt, pp 8-18)
- Mary Ellen O’Connell, Banning Autonomous Killing-The Legal and Ethical Requirement That Humans Make Near-Time Lethal Decisions, in The American Way of Bombing Changing Ethical and Legal Norms from Flying Fortresses to Drones(Matthew Evangelista and Henry Shue editors, 2014) (Excerpted)
- Keith Abney, George Bekey, and Patrick Lin, Autonomous Military Robotics: Risk, Ethics, and Design, Cal Poly (2008) (Excerpt, pp 77-90)
- Michael W. Lewis, Legal Issues in the Development of Autonomous WeaponsAbstract | Paper
- U.S. Department of Defense Directive 3000.09 Autonomy in Weapon Systems (2012)
- Additional Protocol I to the Geneva Conventions (Article 1 par. 2, Article 36)
- Kevin H. Govern, Discriminant Actions Via (Semi-) Autonomous Weapons Systems: Matching Emergent Capabilities With Legal Prescriptions Abstract
The Rationality and Morality of Automaticity
- Duncan MacIntosh, Firing, Forgetting and How Rule-of-Law Values Require Automation of the Rule of Law; A Defense of the Use of Autonomous Weapons Systems in War and Peace Abstract | Paper
- Gregory Kavka, The Toxin Puzzle, Analysis, Vol. 43, No.1 (1983), pp.33-36
- David Gauthier, Assure and Threaten, Ethics, Vol. 104, No.4 (1994), pp.690-721
- Claire Finkelstein, Acting on an Intention, in REASON, INTENTION AND MORALITY (Gijs Van Donselaar & Bruno Verbeek eds., Ashgate Publishing, 2008) (Excerpt 67-77)
- Larry Alexander, The Doomsday Machine:Proportionality, Punishment and Prevention, 63.2 The Monist 199 (1980)
Should “Killer Robots” be Banned?
Uniquely Human? On Intuition, Mercy and Moral Decision Making
- M. L. Cummings, The Human Role in Autonomous Weapon Design and Deployment Abstract | Paper
- Martha Nussbaum, Equity and Mercy, 22.2 Philosophy and Public Affairs 83 (1993) (Excerpt, pp 83-92)
- Human Rights Watch, Shaking The Foundations: The Human Rights Implications of Killer Robots (Excerpt pp 5-16, 23-24)
- Matthias Scheutz & Bertram Malle, May Machines Take Lives to Save Lives? Human Perceptions of Autonomous Robots (with the Capacity to Kill) Abstract | Paper
On the Concept of Meaningful Human Control
- Peter M. Asaro, Jus nascendi, Robotic Weapons & the Martens Clause Abstract | Paper
- Noel Sharkey, The Human Control of Weapons: a humanitarian perspective (Draft of chapter to appear in “Autonomous Weapons Systems: Law, Ethics, Policy” edited by Nehal Bhuta, Susanne Beck, Robin Geiss, Claus Kress and Hin Yan Liu.) Abstract | Paper
- Peter Asaro, On Banning Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal Decision-Making, in Ethics of 21st Century Military Conflict Special Issue on New Technologies and Warfare, International Review of the Red Cross, 94 (886), Summer 2012, pp. 687-709.
- Shane Harris, Out of the Loop: The Human-free Future of Unmanned Aerial Vehicles, in Harris, edited by Peter Berkowitz, Hoover Institution, 2012. (Excerpt, pp 1-5, 9-12)
Responsibility for Acts of Intelligent Machines
- Jens David Ohlin, Machine Liability & the Combatant’s Stance Abstract | Paper
- Daniel Dennet, When HAL Kills, Who’s to Blame?, in Hal’s Legacy: 2001’s Computer as Dream and Reality 351-365 (David Stork ed. 1997).
- Deborah G. Johnson, “Technology with No Human Responsibility?” J Bus Ethics (2014)
Contact us
For any questions regarding the conference or registration, please contact: Jennifer Cohen at jenn.cohen@appc.upenn.edu
