THIS INTERVIEW WAS PUBLISHED BY THE GUARDIAN

Zoë Corbyn

Sun 6 Jun 2021 09.00 BST

‘AI systems are empowering already powerful institutions – corporations, militaries and police’: Kate Crawford. Photograph: Stephen Oxenbury

The AI researcher on how natural resources and human labour drive machine learning and the regressive stereotypes that are baked into its algorithms

Kate Crawford studies the social and political implications of artificial intelligence. She is a research professor of communication and science and technology studies at the University of Southern California and a senior principal researcher at Microsoft Research. Her new book, Atlas of AI, looks at what it takes to make AI and what’s at stake as it reshapes our world.

You’ve written a book critical of AI but you work for a company that is among the leaders in its deployment. How do you square that circle?
I work in the research wing of Microsoft, which is a distinct organisation, separate from product development. Unusually, over its 30-year history, it has hired social scientists to look critically at how technologies are being built. Being on the inside, we are often able to see downsides early before systems are widely deployed. My book did not go through any pre-publication review – Microsoft Research does not require that – and my lab leaders support asking hard questions, even if the answers involve a critical assessment of current technological practices.

What’s the aim of the book?
We are commonly presented with this vision of AI that is abstract and immaterial. I wanted to show how AI is made in a wider sense – its natural resource costs, its labour processes, and its classificatory logics. To observe that in action I went to locations including mines to see the extraction necessary from the Earth’s crust and an Amazon fulfilment centre to see the physical and psychological toll on workers of being under an algorithmic management system. My hope is that, by showing how AI systems work – by laying bare the structures of production and the material realities – we will have a more accurate account of the impacts, and it will invite more people into the conversation. These systems are being rolled out across a multitude of sectors without strong regulation, consent or democratic debate.

What should people know about how AI products are made?
We aren’t used to thinking about these systems in terms of the environmental costs. But saying, “Hey, Alexa, order me some toilet rolls,” invokes into being this chain of extraction, which goes all around the planet… We’ve got a long way to go before this is green technology. Also, systems might seem automated but when we pull away the curtain we see large amounts of low paid labour, everything from crowd work categorising data to the never-ending toil of shuffling Amazon boxes. AI is neither artificial nor intelligent. It is made from natural resources and it is people who are performing the tasks to make the systems appear autonomous.

Unfortunately the politics of classification has become baked into the substrates of AI

Problems of bias have been well documented in AI technology. Can more data solve that?
Bias is too narrow a term for the sorts of problems we’re talking about. Time and again, we see these systems producing errors – women offered less credit by credit-worthiness algorithms, black faces mislabelled – and the response has been: “We just need more data.” But I’ve tried to look at these deeper logics of classification and you start to see forms of discrimination, not just when systems are applied, but in how they are built and trained to see the world. Training datasets used for machine learning software that casually categorise people into just one of two genders; that label people according to their skin colour into one of five racial categories, and which attempt, based on how people look, to assign moral or ethical character. The idea that you can make these determinations based on appearance has a dark past and unfortunately the politics of classification has become baked into the substrates of AI.

You single out ImageNet, a large, publicly available training dataset for object recognition…
Consisting of around 14m images in more than 20,000 categories, ImageNet is one of the most significant training datasets in the history of machine learning. It is used to test the efficiency of object recognition algorithms. It was launched in 2009 by a set of Stanford researchers who scraped enormous amounts of images from the web and had crowd workers label them according to the nouns from WordNet, a lexical database that was created in the 1980s.

Beginning in 2017, I did a project with artist Trevor Paglen to look at how people were being labelled. We found horrifying classificatory terms that were misogynist, racist, ableist, and judgmental in the extreme. Pictures of people were being matched to words like kleptomaniac, alcoholic, bad person, closet queen, call girl, slut, drug addict and far more I cannot say here. ImageNet has now removed many of the obviously problematic people categories – certainly an improvement – however, the problem persists because these training sets still circulate on torrent sites .

And we could only study ImageNet because it is public. There are huge training datasets held by tech companies that are completely secret. They have pillaged images we have uploaded to photo-sharing services and social media platforms and turned them into private systems.

You debunk the use of AI for emotion recognition but you work for a company that sells AI emotion recognition technology. Should AI be used for emotion detection?
The idea that you can see from somebody’s face what they are feeling is deeply flawed. I don’t think that’s possible. I have argued that it is one of the most urgently needed domains for regulation. Most emotion recognition systems today are based on a line of thinking in psychology developed in the 1970s – most notably by Paul Ekman – that says there are six universal emotions that we all show in our faces that can be read using the right techniques. But from the beginning there was pushback and more recent work shows there is no reliable correlation between expressions on the face and what we are actually feeling. And yet we have tech companies saying emotions can be extracted simply by looking at video of people’s faces. We’re even seeing it built into car software systems.

What do you mean when you say we need to focus less on the ethics of AI and more on power?
Ethics are necessary, but not sufficient. More helpful are questions such as, who benefits and who is harmed by this AI system? And does it put power in the hands of the already powerful? What we see time and again, from facial recognition to tracking and surveillance in workplaces, is these systems are empowering already powerful institutions – corporations, militaries and police.

What’s needed to make things better?
Much stronger regulatory regimes and greater rigour and responsibility around how training datasets are constructed. We also need different voices in these debates – including people who are seeing and living with the downsides of these systems. And we need a renewed politics of refusal that challenges the narrative that just because a technology can be built it should be deployed.

Any optimism?
Things are afoot that give me hope. This April, the EU produced the first draft omnibus regulations for AI. Australia has also just released new guidelines for regulating AI. There are holes that need to be patched – but we are now starting to realise that these tools need much stronger guardrails. And giving me as much optimism as the progress on regulation is the work of activists agitating for change.

The AI ethics researcher Timnit Gebru was forced out of Google late last year after executives criticised her research. What’s the future for industry-led critique?
Google’s treatment of Timnit has sent shockwaves through both industry and academic circles. The good news is that we haven’t seen silence; instead, Timnit and other powerful voices have continued to speak out and push for a more just approach to designing and deploying technical systems. One key element is to ensure researchers within industry can publish without corporate interference, and to foster the same academic freedom that universities seek to provide.

Atlas of AI by Kate Crawford is published by Yale University Press (£20). To support the Guardian order your copy at guardianbookshop.com. Delivery charges may apply.