Is AI Technology Equitable and Unbiased?


Table of Contents

In response to students in the UK being unable to sit their end-of-year exams because of the pandemic, the government and Ofqual introduced an algorithm to determine their grades. 

Algorithms are increasingly used to decide outcomes for things like criminal sentencingrecruitment, and banking. While we may assume that an algorithm will produce objective answers – predicated as it is on ‘data’ without ‘human perception’ – which data we use, how it is deployed, and what questions we ask are informed by historical realities shaped by oppression. This Ofqual algorithm considered a teacher-assessed grade, the school’s past performance, and class size to determine whether the teacher’s predicted grade for that student was accurate or should be adjusted. The algorithm’s outcome was that 39% of students had their grades downgraded, while independent schools had double the improvement of A – A* grades, only for the government to ‘u-turn’ and rely on teacher predictions alone. 

So in what ways might oppression have played a role in this grading algorithm? 

  • A school’s performance – historic or present – is not neutral. It’s informed by histories where resources are allocated away from and denied to working class communities and communities of colour through mechanics such as catchment areas and house prices that are connected to enrollment numbers and in turn central funding. What’s more the negative outcomes designed by classism and racism for these young people are used as justifications – by those whom the system uplifts, such as middle class and white parents, to perpetuate these decisions.
  • A teacher’s predicted grade is also not neutral. While this may have been the preferred outcome in this situation, and is ultimately what the government decided to use, when black students are excluded at three times the rate of white students across the country we can safely conclude that teachers are not embodiments of fairness. The very decisions that disproportionately criminalize black students in school, certainly inform their decisions around their intelligence and capability when it comes to their grades too. 

Which leads us to conclude what we also see continuing in our workplaces. White and middle class people are afforded the benefit of the doubt. Their default position is to be granted access, as all those who benefit from systems of oppression do, to what is valuable – whether it’s places at university as this situation shows us, or promotions and new roles in the workplace – based on their potential. To move through the world with one’s ‘potential’ as the lens through which you are seen is to have endless possibilities, reinforced by structures, producing positive outcomes, whether acknowledged or not by the individual. 

So this week, we invite you to interrogate to whom you are giving the benefit of the doubt at work. Why? And what outcomes does that produce. And perhaps also extend that possibility and potential elsewhere. 

and invisibilizing and minimizing the horror of white violence against black people, people of color, indigenous people and Jewish people. The real question is: when the KKK can exist freely – why would anyone ever be ‘shocked’ by other state sanctioned acts of violence against people of color? 

Share this article with a friend

Create an account to access this functionality.
Discover the advantages