We Humans and the Intelligent Machines. Jörg Dräger
Читать онлайн книгу.
in 1996 and are German citizens. Both attend a secondary school in a medium-sized town. Their biographies are almost identical – apart from their names. Tim and Hakan do not really exist. The two were invented by researchers for a study on discrimination in Germany’s vocational training system, commissioned by the Expert Council of German Foundations on Integration and Migration.4
The researchers sent applications from Tim and Hakan to 1,794 companies. Result: Tim was invited to interview considerably more often than Hakan, whose success rate was 50 percent lower. Such discrimination is higher for small companies than for large ones, and higher for automotive technicians than for office clerks. Despite any differences in the details, however, the overall findings are clear: Even when they have the same qualifications as other candidates, applicants with a Turkish name are clearly at a disadvantage when they go looking for a position as an apprentice in Germany.
Other studies show discrimination based on heritage in many countries: in Ireland against German names, in the Netherlands against Surinamese, in Norway against Pakistani, in Switzerland against Portuguese, in Spain against Moroccan, in the US against typically African-American names such as Lakisha and Jamal, as opposed to names such as Emily and Greg.5 This discrimination, whether intentional or not, is not limited to the labor market. It shapes our everyday lives. At universities in the US, a field trial has shown that professors working with PhD candidates are more likely to interview white men than women; Hispanics, blacks, Chinese and Indians have also been disadvantaged.
We are guided in important decisions by mental associations and unconsciously activated stereotypes. Our judgments and also our perception and our thinking are subject to systematic errors. These so-called cognitive biases have been documented by psychologists for decades. They seem to be almost universally human and unconscious despite the various ways they are expressed.6
Well over 100 such malfunctions in our thinking are known.7 Many of these have been researched by psychologist Daniel Kahneman, who was awarded the Nobel Prize for his work on these shortcomings. This is how Kahneman has summed up his many decades of empirical research on the way humans process information: “The confidence we experience as we make a judgment is not a reasoned evaluation of the probability that it is right. Confidence is a feeling, one determined mostly by the coherence of the story and by the ease with which it comes to mind, even when the evidence for the story is sparse and unreliable.”8
Cognitive biases lead to a wide variety of misjudgments. When a plane has just crashed and the media report about it, people temporarily estimate the frequency of plane accidents to be higher than usual, and higher than it actually is. We overestimate the risk of being murdered and underestimate the likelihood of having a stroke. People react more favorably towards others if they find them outwardly attractive. For example, they judge the same offense more mildly when it has been perpetrated by a good-looking defendant and consider attractive candidates for political office to be more knowledgeable.9
In other words, in many areas of life people act neither rationally nor fairly. We use inappropriate criteria without necessarily being aware of it. In our memories, we are often firmly convinced that we have solely assessed a person’s expertise or competence, even if it was their appearance, skin color or gender that ultimately tipped the scales. Some misjudgments in everyday life are harmless or even amusing. However, systematic biases can also have significant negative social consequences. Human rights are based on the principle of equality and protection against discrimination. We should therefore use the “intelligence” offered by machines to create equitable opportunities for all demographic groups.
Inconsistency: Rating the same things differently
Two lawyers, three opinions – Germans use this aphorism to bemoan the inconsistency of legal judgments. Yet lawyers are not the only ones known for judging the same facts differently. Contradictory viewpoints are human and part of everyday life among physicians as well. This is why many patients with serious illnesses often seek a second opinion before agreeing to treatment. This does not necessarily reflect a lack of confidence in the medical profession. It more likely attests to a healthy gut instinct which senses that even experts can disagree with a colleague’s opinion and make a mistake.
A test by the Boston radiologist Hani Abujudeh shows how justified this gut feeling is. In 2010, he invited three of his colleagues at Massachusetts General Hospital, specialists in abdominal and pelvic conditions, to participate in an experiment. Abujudeh gave each of them CT scans of 60 old medical cases, which he had randomly selected from the hospital’s database. What the three doctors did not know was, first, what diagnosis had originally been made and, second, that in half of the cases they had made it themselves. Abujudeh asked his colleagues to examine the scans again, before he compared the new with the original findings. The result: In one instance in four, the radiologists deviated considerably from their own past diagnosis. And when looking at scans that had previously been evaluated by someone else, the doctors came to a different conclusion one-third of the time.10
Just so no wrong conclusions are reached: These findings do not mean that incorrect diagnoses or improper treatment are widespread. After all, hospitals have put detailed procedures in place to make up for humans’ limited abilities. In difficult cases, for example, a radiologist never decides alone. Second opinions are often required, the progression of the disease is interpreted in light of laboratory results, in cases of doubt interdisciplinary councils are asked to provide an opinion. The variance in clinical findings in everyday settings is therefore considerably lower than in Abujudeh’s experiment; it is more likely to be between 3 and 4 percent than between 25 and 30 percent.11
There is no doubt, however, that people evaluate the same things differently. Even recognized experts have to include complex feedback loops in their decision-making processes if they want to minimize outlying assessments. This is not only true when the same thing is examined by different experts; even the same person often assesses the same facts differently at different times.
This is hardly a phenomenon unique to the world of medicine. In another study, seven experienced software developers assessed the workload for new projects completely differently. They had to supply estimates for the same 60 projects, and their forecasts of the required working days differed by 71 percent on average.12 Supervisors assess the performance of employees in similarly inconsistent fashion, as do financial experts the value of stocks and real estate.13 Inconsistency makes decisions unpredictable and erroneous. It can result from very different causes, such as fatigue, stress or a person’s recent experiences. Even though it is rarely possible to determine the specific reason in individual cases, it is clear that people tend to behave inconsistently. New software-supported tools, however, could help reduce the impact such inconsistency has in important areas of life.
Complexity: Overwhelmed by too many options
German cities are not often compared with New York City. However, those public administrators in Berlin who are involved in school planning will probably have recognized the situation described in Chapter 1. Assigning children to a school efficiently and fairly while taking many relevant factors into account is not a task carried out only in the Big Apple.
School catchment areas have to be recalculated again and again. Birth rates develop differently from neighborhood to neighborhood, new residential areas emerge and socio-economic structures change, as do the traffic situation and the wishes of parents. All this has to be taken into account. Moreover, schools’ capacity utilization must be maximized, the routes children travel every day should be as short and safe as possible, and there should be a diverse mix of students at each school.
Another reason why this task is so difficult for administrators is that its complexity can hardly be reduced by excluding or prioritizing certain factors. All criteria are relevant. And each in itself is difficult to assess and address. The length of the route to school, for example. This is of particular importance in the German state of Brandenburg which mandates that as soon as registrations at a primary