🧪 This platform is in early beta. Features may change and you might encounter bugs. We appreciate your patience!
algorithmic_bias
Algorithmic bias is a technical statistical problem that arises when machine learning models encode and perpetuate historical biases present in training data. This occurs through biased training data, proxy variables that correlate with protected characteristics, optimization criteria that weight majority performance, and feedback loops that amplify initial biases over time.
A recidivism prediction algorithm (COMPAS) was trained on historical arrest data. Because minority defendants were historically arrested at higher rates due to discriminatory policing, the algorithm assigned higher risk scores to minority defendants with similar actual recidivism rates as white defendants.
A resume-screening algorithm trained on a decade of successful hires at a tech company learns to downrank candidates who attended all-women's colleges, because historical hiring patterns reflect past gender discrimination rather than candidate quality. The algorithm encodes and automates that discrimination at scale.
A bank's loan approval model trained on historical lending data systematically assigns lower credit scores to applicants from certain zip codes. Because those zip codes were historically redlined and underserved, the model treats the resulting lack of credit history as a risk signal — perpetuating exclusion under the appearance of objective data-driven decision-making.
Binary (yes/no) questions an LLM must answer to identify this aspect:
Was the model trained on historical data that encodes past discriminatory practices?
Type: binaryAre model outputs systematically less accurate or more biased for protected subgroups?
Type: binaryWere fairness metrics (e.g., demographic parity, equalized odds) evaluated alongside accuracy metrics?
Type: binaryIs the claimed neutrality of the algorithm being used to obscure its differential impact?
Type: binaryAlgorithmic bias is a technical statistical problem that arises when machine learning models encode and perpetuate historical biases present in training data. This occurs through biased training data, proxy variables that correlate with protected characteristics, optimization criteria that weight majority performance, and feedback loops that amplify initial biases over time.
Data reflects historical social structures. When those structures encode discrimination, models optimized to fit historical data will replicate the discrimination. The mathematical appearance of objectivity masks the normative choices embedded in training data.
Audit training data for representativeness. Measure model performance separately for demographic subgroups. Apply fairness constraints during optimization. Use causal models that distinguish legitimate from illegitimate sources of variation.
Facial recognition systems trained predominantly on light-skinned faces show dramatically higher error rates for darker-skinned women (MIT Media Lab study, Buolamwini & Gebru, 2018).
Use these tools to detect, analyze, or train this aspect.