End the use of biased student evaluations of teaching in employment decisions
End the use of biased student evaluations of teaching in employment decisions
A large body of research shows that using SETs in employment decisions is discriminatory against women and racial and sexual minorities, sometimes costing them their livelihoods; curtails free speech and academic freedom; and does not help students’ education. We know this too well now and it is time that we put an end to this problem.
We know that it is difficult to individually file a lawsuit to combat the disparate impact and curtailing of free speech that results from using SETs—damaging to reputations, expensive via private attorneys, and generally the EEOC, which is largely underfunded is unhelpful in these types of cases. Faculty experiencing the disparate impact but not being discharged are even less likely to legally challenge the use of SETs. More difficult to defend individually is the situation of applicants denied employment because search committees consider SETs regardless of whether SETs are explicitly requested. The best way to defend these faculty and applicants is having an organization file a class action lawsuit to protect its members. Given the sociologists’ concern with unmasking seemingly neutral forms of institutional discrimination, the ASA should take a leading role here.
In the letter below, we explain how research shows that using SETs in employment decisions is unlawful and ask that the ASA take action by filing as a plaintiff against employers and administrators using student evaluations in employment decisions, if possible, joining forces with other professional associations. We want to send the letter before the Council meeting on August 14th. Please help us make this change by signing and sharing! And because it's the ASA's annual meeting starts now, please use the hashtag #ASA2019 and #sociology to help spread the word.
American Sociological Association Council
1430 K Street, NW, Suite 600
Washington, DC 20005
Dear Members of the Council,
As sociologists, we are writing to request that the American Sociological Association defend its membership by filing a class-action lawsuit against those colleges, universities, and administrators who use student evaluations of teaching (SETs) in employment decisions affecting sociologists. Using SETs in employment decisions is discriminatory against women and racial and sexual minorities, sometimes costing them their livelihoods; curtails free speech and academic freedom; and does not help students’ education. We know this too well now and it is time that we put an end to this problem.
Multiple studies have shown that SETs are biased against women, racial minorities, and sexual minorities (on gender see Abel and Meltzer 2007, Boring, Ottoboni and Stark 2016, MacNell, Driscoll and Hunt 2015, Mengel, Sauermann and Zölitz 2018, Mitchell and Martin 2018, Rivera and Tilcsik 2019; on race see Aruguete, Slater and Mwaikinda 2017; Chisadza, Nicholls and Yitbarek 2019; Chowdhary 1988; Fan et al. 2019; Reid 2010; Smith 2009; on sexual minorities see Anderson and Kanner 2011, Russ, Simonds and Hunt 2002; Subtirelu 2015). This makes SETs problematic and a good example of seemingly neutral, institutionalized forms of discrimination that many sociologists seek uncover.
Title VII of the Civil Rights Act of 1964 prohibits the use of employment evaluation instruments that have a disparate impact on a group protected from discrimination—e.g. by race, sex, etc., unless such instruments are shown to be valid and reliable indicators of job performance or a business necessity. Note that, in order to be unlawful, the adverse impact does not have to result from the entire hiring or employment policy, but could exist in one or more discrete part of these policies—i.e. using student evaluations as one component of a tenure or promotion decisions. There is a disparate impact of women or minorities having to work longer hours to make students happy and obtain the same ratings as men—women, unlike men, are judged by how nurturing they are (Sprague and Massoni, 2005). Universities and colleges often attempt to justify the use of SETs by suggesting that they measure “teaching effectiveness.”
However, major recent experimental and quasi-experimental studies and meta-analyses have shown that student evaluations are not associated with teaching effectiveness—measured as learning in common finals of multi-section courses, either in random assignments or controlling for things like prior learning ability. Uttl, White, and Gonzalez’s (2017) meta-analysis found that the belief that SETs are associated with learning is an artifact of small sample size and publication bias of studies, and of low standards in meta-analyses, almost all from the 1980s or earlier. Furthermore Uttl, Cnudde, and White (2019) show that studies reporting an association between SETs and learning were, on average, more likely to be written by researchers with a vested interest validating SETs—owners or employers of corporations selling SET products, funded by those corporations, or working in administrative positions managing SETs. Findings from random assignment of students to sections of the same course to evaluate both short-term learning in the course and long-term learning in follow-up courses are even more damning of SETs. The only two studies of this kind found that receiving low ratings had a small positive association with learning in the course, but a negative association with learning in subsequent courses (Braga, Paccagnella and Pellizzari 2014; Carrell and West 2010; see Kornell and Hausman 2016 for a meta-analysis). A likely reason for this is that making learning more difficult “decreases short-term performance, decreases students’ subjective ratings of their own learning, and it simultaneously increases long-term learning and transfer” (Kornell and Hausman 2016). Braga et al. (2014) find support for the hypothesis that low- and average-ability students do not like exerting the effort needed to produce long-term learning. Interestingly, even if we only cared about contemporaneous learning, association is small, with too much noise to be useful even for extreme cases, as SET defendants sometimes believe: instructors with the lowest SET scores produce average short-term (and long-term) learning; instructors producing low short-term (and long-term) learning had about average SET scores (Braga et al. 2014). Consequently, employers cannot claim that it is a business necessity to use SETs to evaluate teaching effectiveness.
Contrary to what employers would need to claim to defend their use of SETs, studies overwhelmingly show that SETs are associated with many other things rather than teaching effectiveness, often beyond the instructor’s control. Beyond the biases, they depend on things like enjoyment of the course, false perceptions of learning, student grade expectations, instructor’s beauty, class size, students’ seemingly deliberate false answers, and handing out cookies in class (Boring et al. 2016; Braga et al. 2014; Carpenter et al. 2016; Carrell and West 2010; Cho, Baek and Cho 2015; Hamermesh and Parker 2005; Hessler et al. 2018; Stanfel 1995; Stark and Freishtat 2014). Even putatively objective “good teaching practices,” such as returning grades in a timely manner, are biased against women (Boring et al 2016).
This overwhelming evidence shows that student evaluations should be unlawful under Title VII of the Civil Rights Act of 1964. However, we know that individually taking over a lawsuit to combat the disparate impact is difficult—damaging to reputations, expensive via private attorneys, and generally the EEOC, which is largely underfunded is unhelpful in these types of cases (Jameel 2019). Faculty experiencing the disparate impact but not being discharged are even less likely to legally challenge the use of SETs. Perhaps more difficult to defend individually is the situation of applicants denied employment because search committees consider SETs regardless of whether SETs are explicitly requested.
Research also shows that teaching about White privilege and other key topics in sociology courses that discomfort some students, also lowers student evaluation scores and stimulates angry comments, especially for faculty of underprivileged groups (Abel and Meltzer 2007; Anderson and Smith 2005; Boatright-Horowitz and Soeung 2009; Mowatt 2019; Russ et al. 2002). Allowing student evaluations to be used in personnel decisions can therefore curtail free speech by not hiring or discharging faculty who teach about those topics, and by pressing faculty to reduce their teaching of topics that discomfort students. When public universities do this, states are effectively abridging faculty’s free speech and academic freedom.
The best way to defend these faculty and applicants is by filing a class action lawsuit. Because it is individually damaging to file a class action complaint with the names of leading plaintiffs, the best way to do it is for an organization to be the plaintiff defending its members. That is where the ASA—and other professional associations—can play a major role. Of course, unions could do so to, but so far they seem to want to avoid this, and they cannot defend job applicants who have been denied employment because of SETs. Given the sociologists’ concern with unmasking seemingly neutral forms of institutional discrimination, the ASA should take a leading role here. We ask that you take action by filing as a plaintiff against employers using student evaluations in employment decisions, if possible, joining forces with other professional associations.
Abel, Millicent H. and Andrea L. Meltzer. 2007. "Student Ratings of a Male and Female Professors’ Lecture on Sex Discrimination in the Workforce." Sex Roles 57(3):173-80.
Anderson, Kristin J and Melinda Kanner. 2011. "Inventing a Gay Agenda: Students' Perceptions of Lesbian and Gay Professors." Journal of Applied Social Psychology 41(6):1538-64.
Anderson, Kristin J. and Gabriel Smith. 2005. "Students’ Preconceptions of Professors: Benefits and Barriers According to Ethnicity and Gender." Hispanic Journal of Behavioral Sciences.
Aruguete, Mara S, Joshua Slater and Sekela R Mwaikinda. 2017. "The Effects of Professors’ Race and Clothing Style on Student Evaluations." The Journal of Negro Education 86(4):494-502.
Boatright-Horowitz, Su L. and Sojattra Soeung. 2009. "Teaching White Privilege to White Students Can Mean Saying Good-Bye to Positive Student Evaluations." American Psychologist 64(6):574-75.
Boring, Anne, Kellie Ottoboni and Philip B. Stark. 2016. "Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness." ScienceOpen Research. doi: 10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1.
Braga, Michela, Marco Paccagnella and Michele Pellizzari. 2014. "Evaluating Students’ Evaluations of Professors." Economics of Education Review 41:71-88.
Carpenter, Shana K, Laura Mickes, Shuhebur Rahman and Chad Fernandez. 2016. "The Effect of Instructor Fluency on Students' Perceptions of Instructors, Confidence in Learning, and Actual Learning." Journal of Experimental Psychology Applied 22(2):161.
Carrell, Scott E and James E West. 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors." Journal of Political Economy 118(3):409-32.
Chisadza, Carolyn, Nicky Nicholls and Eleni Yitbarek. 2019. "Race and Gender Biases in Student Evaluations of Teachers." Economics Letters 179:66-71.
Cho, Donghun, Wonyoung Baek and Joonmo Cho. 2015. "Why Do Good Performing Students Highly Rate Their Instructors? Evidence from a Natural Experiment." Economics of Education Review 49:172-79.
Chowdhary, Usha. 1988. "Instructor's Attire as a Biasing Factor in Students' Ratings of an Instructor." Clothing and Textiles Research Journal 6(2):17-22.
Fan, Y., L. J. Shepherd, E. Slavich, D. Waters, M. Stone, R. Abel and E. L. Johnston. 2019. "Gender and Cultural Bias in Student Evaluations: Why Representation Matters." PLOS ONE 14(2):e0209749.
Hamermesh, Daniel S. and Amy Parker. 2005. "Beauty in the Classroom: Instructors’ Pulchritude and Putative Pedagogical Productivity." Economics of Education Review 24(4):369-76.
Hessler, Michael, Daniel M Pöpping, Hanna Hollstein, Hendrik Ohlenburg, Philip H Arnemann, Christina Massoth, Laura M Seidel, Alexander Zarbock and Manuel Wenk. 2018. "Availability of Cookies During an Academic Course Session Affects Evaluation of Teaching." Medical education 52(10):1064-72.
Jameel, Maryam. 2019, "More and More Workplace Discrimination Cases Are Being Closed before They’re Even Investigated" Vox. Retrieved August 8, 2019 (<a href="https://www.vox.com/identities/2019/6/14/18663296/congress-eeoc-workplace-discrimination" rel="nofollow">https://www.vox.com/identities/2019/6/14/18663296/congress-eeoc-workplace-discrimination</a>
Kornell, Nate and Hannah Hausman. 2016. "Do the Best Teachers Get the Best Ratings?" Frontiers in Psychology 7. doi: 10.3389/fpsyg.2016.00570.
MacNell, Lillian, Adam Driscoll and Andrea N. Hunt. 2015. "What’s in a Name: Exposing Gender Bias in Student Ratings of Teaching." Innovative Higher Education 40:291-303.
Mengel, Friederike, Jan Sauermann and Ulf Zölitz. 2018. "Gender Bias in Teaching Evaluations." Journal of the European Economic Association 17(2):535-66.
Mitchell, Kristina M. W. and Jonathan Martin. 2018. "Gender Bias in Student Evaluations." PS: Political Science & Politics 51(3):648-52.
Mowatt, Rasul A. 2019. "Twelve Years a Servant: Race and the Student Evaluation of Teaching." SCHOLE: A Journal of Leisure Studies and Recreation Education 34(2):109-19.
Reid, Landon D. 2010. "The Role of Perceived Race and Gender in the Evaluation of College Teaching on Ratemyprofessors.Com." Journal of Diversity in Higher Education 3(3):137-52.
Rivera, Lauren A. and András Tilcsik. 2019. "Scaling Down Inequality: Rating Scales, Gender Bias, and the Architecture of Evaluation." American Sociological Review 84(2):248-74.
Russ, Travis, Cheri Simonds and Stephen Hunt. 2002. "Coming out in the Classroom... An Occupational Hazard?: The Influence of Sexual Orientation on Teacher Credibility and Perceived Student Learning." Communication Education 51(3):311-24.
Smith, Bettye P. 2009. "Student Ratings of Teaching Effectiveness for Faculty Groups Based on Race and Gender." Education 129(4).
Stanfel, Larry E. 1995. "Measuring the Accuracy of Student Evaluation of Teaching." Journal of Instructional Psychology 22(2):117.
Stark, Philip B. and Richard Freishtat. 2014. "An Evaluation of Course Evaluations." ScienceOpen Research.
Subtirelu, Nicholas Close. 2015. "“She Does Have an Accent but …”: Race and Language Ideology in Students' Evaluations of Mathematics Instructors on Ratemyprofessors.Com." Language in Society 44(1):35-62.
Uttl, Bob, Kelsey Cnudde and Carmela A White. 2019. "Conflict of Interest Explains the Size of Student Evaluation of Teaching and Learning Correlations in Multisection Studies: A Meta-Analysis." PeerJ 7:e7225. doi: 10.7717/peerj.7225.
Uttl, Bob, Carmela A. White and Daniela Wong Gonzalez. 2017. "Meta-Analysis of Faculty's Teaching Effectiveness: Student Evaluation of Teaching Ratings and Student Learning Are Not Related." Studies in Educational Evaluation 54:22-42.