“Now that generative AI systems are widely available, almost anyone can use these models for critical tasks that affect their own and other people’s lives, such as hiring,” said senior author Aylin Caliskan, a UW assistant professor in the iSchool. “Small companies could attempt to use these systems to make their hiring processes more efficient, for example, but it comes with great risks. The public needs to understand that these systems are biased. And beyond allocative harms, such as hiring discrimination and disparities, this bias significantly shapes our perceptions of race and gender and society.”
This research was funded by the U.S. National Institute of Standards and Technology.
https://www.washington.edu/news/2024/10/31/ai-bias-resume-screening-race-gender/
Draft Bill for Regulation of Artificial Intelligence and Machine Learning in Employment Services
Section 1. Short Title
This Act may be cited as the
"Fair and Transparent AI Employment Act of 2024".
Section 2. Findings and Purpose
(a) Findings
The Congress finds the following:
The use of artificial intelligence (AI) and machine learning (ML) in employment services is expanding rapidly, yet such systems often lack transparency and can unintentionally perpetuate discrimination.
Algorithms that screen job applicants may lead to adverse impacts on protected classes under Title VII of the Civil Rights Act of 1964.
Biases in AI and ML systems can result in discrimination based on age, race, ethnicity, gender, or other protected characteristics.
Greater transparency, accountability, and fairness are required to ensure the equitable use of AI and ML in employment services.
(b) Purpose
The purpose of this Act is to establish clear guidelines and restrictions on the use of AI and ML in employment services to prevent discrimination, ensure transparency, and protect applicants' rights.
Section 3. Regulation of Machine Learning in Recruitment Software
(a) Machine learning systems used in recruitment software shall only be used to:
1. Train models based strictly on job requisition criteria and applicant employment experience relevant to the role applied for.
2. Match applicants to roles within the specific company and job title for which they have applied.
3. Exclude from training data any variables or features that are unrelated to job performance, such as names, addresses, or other personal identifiers that may infer demographic characteristics, likelihood of success, or cultural fit based on non-job-related factors.
(b) Prohibitions and Limitations:
1. Use of applicant data pools or lakes in mass by recruitment systems is hereby prohibited unless explicitly authorized by applicants for data portability purposes.
2. Algorithms shall not incorporate data proxies that indirectly infer demographic characteristics, such as postal codes, alma maters, or hobbies, unless explicitly shown to be job-relevant.
3. Recruitment software must maintain a comprehensive audit trail of data sources and decision-making processes to allow for third-party evaluations of adherence to job-related criteria.
(c) Transparency and Accountability Measures:
1. All recruitment software vendors must disclose the data attributes, weights, and training methods used in model development to clients and regulatory bodies.
2. Employers are responsible for ensuring that ML systems comply with anti-discrimination laws and must conduct periodic reviews to identify and mitigate any predictive patterns unrelated to job qualifications.
3. Applicants must be informed about the types of data analyzed and their rights to contest decisions influenced by recruitment algorithms.
Section 4. Restrictions on Demographic Data Collection
(a) Employers and recruitment platforms are prohibited from requiring or soliciting demographic information on initial job applications.
(b) Demographic data may only be collected when:
Applicants are informed of the specific use of such data.
Transparency measures are in place affirming that such data will not influence employment decisions.
(c) Employers shall publicly affirm and document compliance with the above conditions.
3. Restricted Use of Machine Learning in Recruitment Software:
• ML models shall only be trained on job requisition criteria and applicant employment experience relevant to the role.
4. Prohibition on Mass Data Pools:
• Applicant data pools or lakes shall not be used to analyze applicants for roles they did not directly apply to.
5. Ban on Demographic Data Collection:
• Initial applications must not request demographic information unless accompanied by explicit, transparent statements affirming that such information will not influence hiring decisions.
6. Elimination of Algorithms Screening Personal Characteristics:
• Algorithms must not use names, addresses, or other identifiers that may infer race, ethnicity, religion, gender, or other protected attributes.
7. Age Discrimination Protections:
• Algorithms may only analyze prior employment data relevant to job descriptions, excluding considerations like unemployment gaps or tenure that could infer age bias.
Section 5. Transparency of AI and ML Algorithms and Accountability
(a) Employers and third-party vendors using AI and ML systems in recruitment or employment selection shall provide:
Detailed documentation of the algorithms, including their training data sources, logic, and decision-making criteria.
Open and accessible data explaining how the algorithms mitigate biases and ensure compliance with Title VII of the Civil Rights Act of 1964.
3. Employers must publicly disclose the data sources, training methods, and logic behind AI/ML systems.
4. Data and algorithms must be auditable, with documentation accessible to third-party reviewers and regulatory bodies.
5. Employers are required to maintain an open feedback system for applicants to report perceived biases or errors.
(b) These documents shall be made available to regulatory authorities and applicants upon request.
Section 6. Statistical Methods for Bias Detection and Audits
1. Adverse Impact Analysis:
• Four-Fifths Rule: Selection rates for any protected group must not fall below 80% of the most favored group.
• Chi-Square Testing: Evaluates whether differences in outcomes for protected groups are statistically significant.
2. Feature Importance Analysis:
• Employs Shapley Additive Explanations (SHAP) to identify features disproportionately influencing decisions.
• Irrelevant data, such as names or addresses, must be flagged and removed.
3. Randomized Testing:
• Simulated applicant profiles with identical qualifications but differing demographic markers are used to test for disparate outcomes.
4. Intersectional Bias Testing:
• Evaluates combined effects of multiple characteristics (e.g., age and gender) to ensure fairness across intersections of identity.
Section 7. Restriction of Algorithmic Review
(a) Algorithms used for applicant screening or selection shall be restricted to evaluating:
Prior employment data strictly aligned with job descriptions.
Professional qualifications and job-related experience.
(b) Algorithms shall exclude factors such as:
Employment dates, periods of unemployment, or tenure length.
Names, or any data points that could indirectly infer race, ethnicity, gender, or other protected characteristics.
Section 8. Ban on Discriminatory Algorithmic Features
(a) The use of programming features or criteria that analyze:
Names, voice samples, or any proxies for race, ethnicity, or gender, is prohibited.
Physical attributes or appearance unless expressly job-related and nondiscriminatory.
(b) Employers must conduct annual audits of AI/ML systems to identify and mitigate potential adverse impacts on protected classes.
Section 9. Compliance with EEOC and Title VII Requirements
(a) All AI and ML systems must comply with guidelines set by the Equal Employment Opportunity Commission (EEOC).
(b) Employment selection procedures using AI/ML shall be assessed for adverse impact, including compliance with studies and documentation identified by the EEOC.
Section 10. Case Studies, Research Requirements, and Legal Precedents
(a) Prominent Studies on AI Bias and Discrimination
"Disparate Impact in Algorithmic Decision-Making" (Barocas & Selbst, 2016)
Demonstrates how AI systems can unintentionally perpetuate systemic biases due to biased training data, particularly in hiring practices.
"Gender and Racial Bias in Hiring Algorithms" (Raghavan et al., 2020)
Highlights how facial recognition and natural language processing models have been shown to disproportionately disadvantage women and minority groups during the hiring process.
"The Impact of Automated Resume Screening" (Upturn, 2018)
Examines the ways automated systems reinforce workplace discrimination through over-reliance on historical hiring data that reflects biased human decisions.
(b) Notable Legal Cases Involving AI Discrimination in Employment
Amazon AI Hiring Bias Case (2018)
Amazon abandoned an AI recruitment tool after it was discovered that the system systematically downgraded resumes with terms associated with women (e.g., “women’s chess club”). The bias arose from training data reflecting historical male-dominated hiring patterns.
EEOC v. Griggs Power Company (1971)
While predating AI, this landmark Supreme Court case established the principle of “disparate impact,” which remains relevant for assessing algorithmic biases that disproportionately affect protected groups.
Li et al. v. HireVue Inc. (2021)
A legal challenge over HireVue’s use of facial analysis and video interviews, which plaintiffs alleged discriminated against candidates with disabilities and people of color due to biases embedded in its AI.
(c) Reporting on Legal and Ethical Concerns
Employers must reference these and similar cases in their annual reports, demonstrating an understanding of legal precedents and providing actionable steps taken to mitigate risks of discriminatory practices.
Section 10. Annual Reporting and Independent Audits
(a) Annual Reporting Requirements
Employers must submit an annual report to the Equal Employment Opportunity Commission (EEOC), including:
A detailed overview of AI/ML tools utilized in employment processes.
Disaggregated employment outcome data by protected classes (e.g., race, gender, age, disability, veteran status).
Documentation of steps taken to mitigate algorithmic bias, referencing ongoing challenges and solutions informed by case studies.
Demonstrations of compliance with the EEOC’s guidance on the use of software, algorithms, and artificial intelligence in employment selection procedures.
(b) Independent Audits
Purpose of Audits
To assess whether AI/ML systems result in disparate impacts on protected classes.
To verify that AI/ML systems align with job-relevant criteria and comply with this Act and Title VII of the Civil Rights Act of 1964.
Required Audit Methods
Bias Testing: Evaluate AI models against diverse applicant datasets to identify and correct disparities.
Algorithm Transparency Analysis: Review the decision-making logic, training data, and implementation of algorithms.
Adverse Impact Analysis: Conduct statistical tests (e.g., the Four-Fifths Rule) to determine whether AI/ML systems disproportionately disadvantage any protected class.
Feature Importance Assessment: Examine and document how features such as work history, education, or skills influence decisions. Features deemed irrelevant (e.g., names or addresses) must be flagged and removed.
Ongoing Monitoring: Implement periodic re-audits to ensure compliance and adaptation to new fairness benchmarks or guidelines.
Demographic Considerations in Audits
Intersectionality: Assess combined impacts on individuals belonging to multiple protected classes (e.g., women of color, older LGBTQ+ applicants).
Geographical Representation: Ensure systems function equitably across different regions, including urban and rural settings.
Disability Inclusion: Evaluate accessibility and fairness for individuals with disabilities, ensuring compliance with the Americans with Disabilities Act (ADA).
Veteran Status: Analyze hiring patterns for veterans to prevent biases against individuals with military service records.
(c) Reporting and Transparency Requirements
Audit findings must include:
Data sources and methodologies used.
Detailed results of bias detection and mitigation efforts.
Recommendations for compliance improvement.
These findings shall be submitted to the EEOC, with publicly accessible versions published online to promote transparency.
Section 11. Compliance and Enforcement Mechanisms
(a) Regulatory Oversight:
1. The Equal Employment Opportunity Commission (EEOC) shall oversee enforcement in partnership with:
• The Department of Labor (DOL) for technical standards.
• The Federal Trade Commission (FTC) for consumer protection.
2. The EEOC shall have the authority to enforce this Act and investigate claims of algorithmic discrimination.
(b) Penalties for Non-Compliance:
1. Civil penalties of up to $250,000 per violation.
2. Mandatory corrective action plans for non-compliant employers.
3. Repeat violations may result in revocation of AI/ML system usage licenses.
(c) Incentives for Transparency:
1. Employers exceeding compliance standards may qualify for tax incentives and public recognition.
Section 12. Research and Development Grant Program
(a) The Department of Labor shall establish a grant program to fund research into bias mitigation and fairness enhancement in AI and ML systems for employment.
(b) Grants shall be prioritized for projects that:
Focus on creating transparent, fair algorithms.
Advance methods to reduce adverse impacts on protected classes.
Section 13. Expanded Research Mandate
(a) Department of Labor Research Grants
The Department of Labor, in collaboration with academic institutions, shall prioritize funding for research projects such as:
Bias detection and mitigation in AI/ML systems.
Development of alternative algorithmic approaches to reduce reliance on biased historical data.
Longitudinal studies on the impacts of AI/ML hiring tools on workplace diversity.
(b) Contributions to Legal and Ethical Understanding
Grant recipients shall publish findings that directly inform future legislation and public discourse on AI in employment practices.
Section 14. Bibliography
Key Academic Studies and Reports
Barocas, S., & Selbst, A. D. (2016). Big Data's Disparate Impact. California Law Review.
Explores how algorithmic decision-making can lead to systematic discrimination.
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020). Mitigating Bias in Algorithmic Hiring Systems: A Framework for Evaluating Technical and Legal Remedies. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAT*).
Upturn (2018). Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias.
Examines how automated systems perpetuate biases and suggests best practices for ethical deployment.
EEOC (2023). Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures.
Provides actionable guidance on identifying and mitigating discriminatory effects under Title VII of the Civil Rights Act.
Binns, R. (2020). On the Apparent Conflict Between Individual and Group Fairness. FAT*.
Discusses trade-offs in achieving fairness across individual and demographic group levels in AI decision-making.
O’Neil, C. (2016). Weapons of Math Destruction.
Highlights real-world examples of biased algorithms, particularly in hiring and employment.
Relevant Legal Cases and Precedents
Griggs v. Duke Power Co. (1971): Landmark case on disparate impact, emphasizing the importance of job-related selection criteria.
Amazon AI Recruitment Case (2018): Illustrates the risks of biased historical data in training algorithms.
HireVue Discrimination Allegations (2021): Challenges to facial recognition and algorithmic hiring tools.
Consolidation of Proposed Legislation
This Act integrates best practices, rigorous statistical methods, transparency requirements, and accountability mechanisms to create a framework ensuring fairness in AI-driven hiring practices. By combining case studies, legal precedents, and ongoing research, the Act sets a high standard for ethical AI deployment in employment while providing employers with actionable compliance pathways.