Work in progress. Here we will give some examples of cases where AI seems to have ‘gone bad’, with commentary.

Reasons for doing this: many codes of ethics, and much regulation and law, develop in response to past failings. The pattern of which events get classed as failings, and how they are seen to have failed, can therefore have a big impact upon subsequent development of codes and regulations and also of laws.

Policies and codes are drawn up in concrete settings, and what issues are highlighted, and how they are highlighted, is crucial. The examples used in the preambles to policy make a large difference. For some more reading on this, see, for example, Tod Chambers’ book The Fiction of Bioethics; or this joint work by a project member on the effects of example choice in developing policy for genetic testing.

Nonetheless, we need case studies. We can’t think in terms of mere abstract principles, and this is nowhere more the case than in ethics and in ethics education. We need to be able to relate any abstract principles or general values to particular concrete circumstances, to be able fully to understand them, to apply them to new cases, and to see how they actually impact upon the world, what they have to do with others, what they have to do with us.

Some case studies may be imagined, some are real. The topic of real versus imagined scenarios is one that merits consideration in its own right, not just for ethics in general, but for AI in particular. No account of AI would be complete without an acknowledgement of the role of Sci Fi in exploring our hopes and fears about AI. After all, how could we have any notion of how to respond to far distant, whizzy technology without fictional accounts? This page here however, will look at some actual cases of problems, or possible problems, that AI has already encountered.

Having thus stated how important example choice is, remember that this page itself is subject to such bias! This is work in progress and we do not intend to suggest that it represents a full and ‘unbiased’ accounts of examples of AI going awry.

Tay, the foul-mouthed Hitler-loving racist millennial chatbot: coming soon

Google’s Photo app labels black person as ‘gorilla’: coming soon

The use of algorithms to make decisions has provided a host of problematic examples, and other examples where the jury is still out, an apt metaphor, given that many of these concern judicial use for instance in sentencing reports, and in policing, where it is alleged that it might increase injustice.  In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, judges can use such algorithms in determining criminal sentencing. The Supreme Court in Wisconsin ruled recently in the 2016 case of State of Wisconsin v Eric Loomis that the use of the COMPAS algorithm used to help determine sentence length was constitutional.

Watch Cathy O’Neil talk about Weapons of Math Destruction.

Recommendation for those developing codes of ethics: be very careful what examples are being drawn upon in the discussion around these codes, and how these examples are described, and what lessons are learned from these examples. Be deliberately self-conscious about what examples have influenced you. Canvass further examples, from diverse sources and diverse people, and invite alternative viewpoints of the examples. Think about what the examples are missing out, and what they are highlighting, and what is played down.

We would like to thank the Future of Life Institute for generously sponsoring our research.