The Machine Question: Teaching Silicon Right from Wrong

James Moor, writing in IEEE Intelligent Systems in July 2006, set out four grades of what he calls ethical agents, and the ladder is still the cleanest way into the subject. Ethical impact agents are systems whose use has ethical consequences, which is nearly all of them. Implicit ethical agents are built so that their design avoids unethical outcomes, in the way a cash machine is built not to dispense money it has not debited. Explicit ethical agents represent ethics and reason about it. Full ethical agents are the ones with whatever it is that makes a person answerable. Almost everything in production today sits in the first two rungs, and the interesting arguments are about whether the gap between the second and the third is a matter of engineering or of kind.

Wendell Wallach and Colin Allen's Moral Machines, published by Oxford in 2009 and subtitled Teaching Robots Right from Wrong, distinguishes three approaches to building an ethical machine. Top-down means encoding an explicit ethical theory or rule set into the system. Bottom-up means letting the system develop something like moral competence through learning and experience. Hybrid approaches combine the two. The subtitle of this page is a deliberate echo of theirs, and the echo is the point: the framing has been on the table for the better part of two decades, and no approach has an accepted test for whether it worked.

Andreas Matthias, in Ethics and Information Technology in 2004, named the problem that runs underneath every section below. Traditionally the manufacturer or operator of a machine is held morally and legally responsible for its behaviour. Systems that learn from their environment produce behaviour their makers can neither predict nor control, and so the traditional ascription of responsibility fails: there is a class of action for which nobody can properly be held answerable, not because anybody has escaped, but because the conditions under which we normally assign responsibility no longer obtain. He called it the responsibility gap.

COMPAS stands for Correctional Offender Management Profiling for Alternative Sanctions. It is a commercial risk and needs assessment instrument. ProPublica reports that its core product is a set of scores derived from 137 questions either answered by defendants or pulled from criminal records, that race is not among those questions, and that the company does not publicly disclose the calculations that produce a defendant's score. Its predictive validity had been published: Brennan, Dieterich and Ehret, in Criminal Justice and Behavior in 2009, reported area under the curve values ranging from .66 to .80 across offender subpopulations and outcome criteria.

In May 2016 ProPublica published an analysis of COMPAS scores under the headline Machine Bias. Their published table, headed Prediction Fails Differently for Black Defendants, gives two rows. Labeled higher risk but did not re-offend: 23.5 percent for white defendants, 44.9 percent for black defendants. Labeled lower risk yet did re-offend: 47.7 percent for white defendants, 28.0 percent for black defendants. They also report running a statistical test isolating the effect of race from criminal history, recidivism, age and gender, and finding black defendants still 77 percent more likely to be pegged as at higher risk of a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind.

The same analysis reported something that gets quoted far less often than the disparity, and which bears on whether the instrument should be in a courtroom at all regardless of how its errors distribute. In ProPublica's own words: the score proved remarkably unreliable in forecasting violent crime, and only 20 percent of the people predicted to commit violent crimes actually went on to do so. Across a full range of crimes the algorithm was, again in their words, somewhat more accurate than a coin flip, with 61 percent of those deemed likely to re-offend arrested for a subsequent offence within two years.

Northpointe disputed the analysis. ProPublica quotes the company's letter: Northpointe does not agree that the results of your analysis, or the claims being made based upon that analysis, are correct or that they accurately reflect the outcomes from the application of the model. The company's fuller technical reply, by Dieterich, Mendoza and Brennan, asserted that the instrument satisfied accuracy equity and predictive parity, meaning that a given score carried the same meaning regardless of the defendant's race.

This is the part that turns a shouting match into a result. Alexandra Chouldechova, in Big Data in June 2017, proved that when a risk instrument satisfies predictive parity, meaning the score means the same thing for every group, and when the underlying base rates of the outcome differ between groups, then the false positive and false negative rates cannot also be equal across those groups. Independently and at the same time, Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan proved that three natural conditions on a risk score cannot in general all hold at once except in degenerate cases. ProPublica measured unequal error rates. Northpointe measured equal predictive accuracy. Both measurements can be correct simultaneously, and the mathematics says they must be, whenever the base rates differ.

Cynthia Rudin, with Caroline Wang and Beau Coker in the Harvard Data Science Review in January 2020, argues a position she has held consistently across this literature: where a decision is consequential and public, an interpretable model whose reasoning can be inspected should be preferred to a proprietary black box, and the secrecy is a policy choice rather than a technical necessity. Andrew Selbst and colleagues, at the FAT* conference in 2019, argue a complementary point from the other direction: treating algorithmic fairness as a property of a model, abstracted away from the institution the model sits inside, is itself a failure mode, because the model is never the whole system that produces the outcome.

Called as a witness in a Wisconsin appeal, COMPAS's co-creator Tim Brennan testified that he had not designed the software to be used in sentencing. ProPublica quotes him: I wanted to stay away from the courts. And then: but as time went on I started realizing that so many decisions are made, you know, in the courts, so I gradually softened on whether this could be used in the courts. That is the responsibility gap arriving from an unexpected direction. The instrument drifted into a role its designer had deliberately avoided, and by the time it was there, no single decision had been made by anyone to put it there.

The term of art is lethal autonomous weapons systems, abbreviated LAWS, and the forum in which states have discussed them is the Group of Governmental Experts convened under the Convention on Certain Conventional Weapons. The International Committee of the Red Cross defines the class in its own words: autonomous weapon systems select and apply force to targets without human intervention; after initial activation or launch by a person, the system self-initiates or triggers a strike in response to information from the environment. You will more often see a two-word phrase from the campaign that opposes them. That phrase is advocacy framing, chosen deliberately and effectively by Stop Killer Robots, and this page uses the technical term instead, not because the campaign is wrong but because the framing does argumentative work that a reader should be able to see happening.

In a position dated Geneva, 12 May 2021, the ICRC recommended that states adopt new legally binding rules, in three parts. Unpredictable autonomous weapon systems should be expressly ruled out, notably because of their indiscriminate effects. The use of autonomous weapon systems to target human beings should be ruled out. And the design and use of any other autonomous weapon systems should be regulated by constraints. Note what kind of document that is: a recommendation that binding rules be created, issued by the guardian of international humanitarian law, which is itself the clearest available evidence that no such rules exist.

Robert Sparrow, in the Journal of Applied Philosophy in February 2007, put the responsibility-gap case in its sharpest military form: he asks who we should hold responsible when an autonomous weapon system is involved in an atrocity of the sort that would normally be described as a war crime. Not the programmer, who could not foresee the specific act. Not the commander, who did not select the target. Not the machine, which cannot be punished in any sense that means anything. If nobody can be held responsible, Sparrow argues, then the deployment is illegitimate, because being able to hold someone responsible is a condition of fighting justly and not an optional extra.

Michael Horowitz, in Daedalus in September 2016, describes and assesses the debate rather than joining one side of it, and his framing carries the strongest counter-case: these weapons, though they do not generally exist today, have already been the subject of multiple discussions at the United Nations. Our own research files record the substantive counter-argument that the ICRC position is answering, which is that autonomous systems could in principle reduce civilian casualties through more precise targeting and through the removal of fear, fatigue and revenge from the decision to fire. That argument is not frivolous, and an article that presents the ban case without it has not presented the debate.

Our research files report that in July 2015 an open letter from the Future of Life Institute, signed by over 3,000 AI and robotics researchers including Stephen Hawking, Stuart Russell, Yoshua Bengio and Demis Hassabis, called for a ban on offensive autonomous weapons beyond meaningful human control, and argued that such weapons represent a third revolution in warfare after gunpowder and nuclear weapons. That is carried here as our own files report it, at their strength and not above it.

You will encounter the claim that an autonomous weapon has already killed a human being without human intervention. This page does not assert it. The claim circulates widely and is usually traced to a single United Nations panel report concerning events in Libya in 2020, whose own language is conditional and which has been publicly contested by specialists. The research behind this article did not verify that report, its wording, its document number or the state of the dispute about it, and an unverified first-of-its-kind claim is exactly the kind of thing that becomes true by repetition. If it happened, it matters enormously. This page cannot tell you that it happened.

Regulation (EU) 2024/1689, the Artificial Intelligence Act, prohibits outright, at Article 5(1)(d), the placing on the market, the putting into service for this specific purpose, or the use of an AI system for making risk assessments of natural persons in order to assess or predict the risk of a natural person committing a criminal offence, based solely on the profiling of a natural person. Read that against section 03 of this page and note what it does: the European Union has banned, prospectively, an application closely resembling the one the COMPAS dispute was about.

The Regulation is dated 13 June 2024 and was published in the Official Journal on 12 July 2024. It applies from 2 August 2026, and its application is deliberately staggered: the chapters containing the Article 5 prohibitions applied from 2 February 2025, the chapters covering notified bodies, general-purpose AI models, governance and penalties from 2 August 2025. Three of our own research documents print a wrong date for this instrument, which is why the dates here are taken from the Official Journal record rather than from our files. As read on 8 August 2026, the EUR-Lex entry carries the status line In force: this act has been changed, and names a consolidated version dated 27 July 2026. Legislation this new moves, and a page that quotes it should say when it looked.

The AI Act does not apply to military AI. Article 2(3) states that the Regulation does not apply to AI systems where and in so far as they are placed on the market, put into service, or used with or without modification exclusively for military, defence or national security purposes, regardless of the type of entity carrying out those activities. So the most developed binding instrument in the world on this subject exempts precisely the application that section 04 could find no binding rules for. The two absences are the same absence, and they are not an oversight; they are a jurisdictional boundary that states have kept.

Anna Jobin, Marcello Ienca and Effy Vayena, in Nature Machine Intelligence in September 2019, identified 84 AI ethics documents issued globally and counted their principles. The five most common were transparency, in 73 of 84; justice and fairness, in 68; non-maleficence, in 60; responsibility, in 60; and privacy, in 47. That is a count of documents, not of people, and it is the only distribution this article quotes for anything, because it is the only one it can reach. Every other position on this page belongs to a named person who argued for it.

Two published critiques bear directly on what those 84 documents accomplish. Thilo Hagendorff, in Minds and Machines in 2020, reports that major corporate AI ethics guidelines systematically neglect topics including sustainability, labour impacts and political misuse, while high-profile principles function as what critics call ethics washing. Brent Mittelstadt, in Nature Machine Intelligence in November 2019, makes a comparative argument: principle-based AI ethics borrows its form from medical ethics, but medicine has common aims, a professional history, proven methods for translating principle into practice, and accountability mechanisms with teeth. AI development has none of those four, so the borrowed form arrives without the machinery that made it work.

Joanna Bryson holds the clearest contrary position, and it is not the dismissal it is often taken for. Her argument, in Ethics and Information Technology in 2018, is that moral patiency is a design decision. We are not obliged to build systems that have interests, and if we do build them we have manufactured an obligation that did not previously exist and need not have. The earlier and more provocatively titled statement of the position is her chapter Robots Should Be Slaves. With Mihailis Diamantis and Thomas Grant she made the separate legal case: legal personhood is a construct that legal systems already extend to non-humans such as corporations, and extending it to synthetic agents would create a lacuna in which liability could be parked. Note that this is an argument with premises. Reaching its conclusion without them, by simply asserting that machines are machines, is not the same position and is refused below.

Two positions try to answer the moral-status question without first settling what is going on inside. Mark Coeckelbergh, in 2010, proposes a social-relational justification: instead of asking what properties an entity must possess to deserve moral consideration, ask how moral consideration actually arises in practice, which is out of the relations between entities rather than out of an inventory of their insides. John Danaher, in Science and Engineering Ethics in 2019, argues for ethical behaviourism: if an entity is consistently indistinguishable in its behaviour from an entity we already grant moral status to, that behavioural equivalence is sufficient grounds for granting it too. Both are serious. Both also relocate rather than remove the difficulty, because somebody still has to decide which relations count, or how consistently indistinguishable is consistent enough.

Eric Schwitzgebel and Mara Garza, in Midwest Studies in Philosophy in 2015, frame the problem as a dilemma of moral risk under deep uncertainty. Two errors are possible. You can wrongly deny moral status to something that has it, and thereby cause suffering nobody recognises as suffering. Or you can wrongly grant moral status to something that lacks it, and thereby accept real costs for nothing. Under uncertainty about which error you are making, the question becomes which mistake you would rather risk, which is a different question from what is true, and is answerable when the first one is not.

Jeff Sebo and Robert Long, in AI and Ethics, make what they call a simple case for extending moral consideration to some AI systems by 2030, resting on a normative premise and a descriptive premise. The paper is recent, peer-reviewed and explicitly conditional, and it is carried here at Tier 3 for a reason worth stating: its descriptive premise is a claim about what AI systems will plausibly be like, and claims about what systems will be like in four years have a poor record. What makes it worth reading is not its confidence but its structure, which separates the value question from the empirical one so that a reader can disagree with one without disturbing the other.

Several of the arguments above depend on a premise this article does not settle: that an artificial system could be conscious, or could have experiences that go well or badly for it. That question belongs to The Measure of Mind, which treats it separately and at length, and nothing here should be read as answering it. Our own research files record two refusals on the subject, one in each direction, and both are worth carrying because together they mark out the honest position: claims that current large language models are conscious or sentient are not supported under any mainstream theory of consciousness, AND claims that consciousness is inherently biological and can never arise in an artificial substrate are metaphysical assertions that no current theory establishes either. Not proven, and not ruled out.

Three items make up the legal-personhood record, and our own file is right to tier them low. Saudi Arabia granted citizenship to the robot Sophia in 2017, which our file itself describes as widely criticised as a publicity stunt rather than genuine legal recognition. The European Parliament considered and rejected a proposal for electronic personhood for sophisticated AI systems in 2017. And the question of whether artificial entities should hold legal standing remains open in the sense that nobody has settled it, not in the sense that anybody is close. That is the entire record, and the gap between it and the volume of commentary about it is instructive.

The claim that AI is a neutral tool, and that any bias in it is attributable entirely to users or to data, is refused. Design choices about what to optimise, what data to collect, which metrics to evaluate against and which populations to test on all embed values, so there is no value-free architecture. Buolamwini and Gebru's Gender Shades audit is the standing demonstration: commercial gender classification systems showing error rates of 0.8 percent for lighter-skinned males against 34.7 percent for darker-skinned females, a disparity that was in the systems and not in the users.

Two refusals, one in each direction, and they are the reason section 03 is written the way it is. This page does not state that COMPAS was shown to be racially biased, full stop, as a settled finding: ProPublica measured unequal error rates and that measurement stands, but whether that constitutes bias depends on which definition of fairness is applied, and the formal results establish that the competing definitions cannot be jointly satisfied. And this page does not state that Northpointe refuted ProPublica or that the analysis was debunked: the company asserted properties its instrument had, those are different properties from the one ProPublica measured, and a system can have Northpointe's and still produce ProPublica's.

Two more refusals, again one in each direction. This page does not conclude that machines are owed nothing because they are just machines. That conclusion is available in the literature but not in that form; Bryson's version is an argument with premises about design decisions and liability, and an article that arrives at the same destination without the argument has not made the case, it has skipped it. And this page does not conclude, or imply through tone, that machines are or will be owed something because they seem to understand us, because they converse fluently, or because people become attached to them. Fluency is what these systems are optimised to produce. The serious arguments for extending moral consideration do not rest on the impression a system makes, and the ones that seem to are usually resting on the impression instead of on an argument.

This page does not attribute any position above to most ethicists, to the consensus, to philosophers generally, to the field, or to what few would now defend. Exactly one distribution was reachable for this article and it is a count of 84 documents, not of people. Every other position here belongs to a named person, with a dated publication, and is presented as theirs. On a subject this contested, a sentence beginning most philosophers agree is almost always doing the work of an argument that was not made.