↑ Bias and discrimination

Human agency and oversight: AI systems are to enable equitable societies by supporting human agency and fundamental rights, and not decrease, limit or misguide human autonomy.

The models used are opaque, using them with real data is officially regulated in Europe (but public authorities and civil society struggle to apply its rules in concrete ways), and the results of the models are mostly incontestable, even when they introduce and reinforce bias and discrimination at every level (selection, confirmation, etc):

• A chatbot that became racist. Tay made it to the MIT Technology Review’s list of 2016’s biggest technology failures.

• HR resume/candidate screening applications that discriminate based on some characteristic (age, gender, religion, etc). If a marginalised person does not get a job because an employer receives a report that contains by the employer unwanted characteristics, he/she will not get the opportunity to gain respectability and independence. Instead, the vicious spiral continues.

• If a poor student cannot get a loan because a lending model deems him or her too risky (by virtue of where he/she lives), he/she is then cut off from the kind of education that could pull him/her out of poverty, and another vicious spiral continues.

• If DNA analysis companies start selling anonymised data to insurance companies they will be able to include it into their risk analysis. There will be a lot of biases, a lot of mistakes. On average nothing will change but there will be biased shifts in premiums in subgroups.

In short, if your data belongs to a subgroup even if you do not share every characteristic of that subgroup you might be negatively impacted. Models favour the lucky and punish the underdogs. The dark side of Big Data. Yet recommending that humans should be involved in the decision-making processes also has to take into account that it is people that have the most bias. They create algorithms that are biased.

Avoiding bias is hard. Techniques and checks to identify it is not so hard to make and critical thinking to rectify it is not that hard to learn either. But as with security in the past decades, it seems not a main concern of digital business-as-usual because it doesn’t make money.