Test scores are not irrelevant(dynomight.net)

281 pointsby colinprince4 tahun yang lalu476 comments

https://dynomight.net/are-tests-irrelevant/

476 comments

ajwinn4 tahun yang lalu

I worked in Admissions at a large state school from 2005-2007. Our method for determining admissibility reminded me of the heat map chart in this article - ie GPA on one axis, SAT/ACT score on the another axis, trace your finger along both paths to see if the applicant was in or out.

I probably reviewed 20-30 applications a day. If high school transcripts were universally formatted, decisions could have been instantaneous but alas, we live in the real world and some human-in-the-loop normalizing had to be done over-and-over.

With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. Now, I’m the kind of person that roots for the genius to overcome his grades and emerge a genius on the SAT/ACT. But in two years, it probably happened only twice. Before calculating a normalized GPA, I could look at the test scores and predict “admit” or “deny”.

While the author is correct to say “the irrelevance of test scores is greatly exaggerated”, in my experience, whether or not something is irrelevant has very little to do with what universities do.

I’d recommend only using test scores. Or, only go with GPAs. Only test scores is more efficient. Only GPAs looks better on press releases.

ramraj074 tahun yang lalu

So one of the best standardized tests I’ve ever taken is the Subject GRE. When I was applying for PhD programmes I and a few of my colleagues took it. It was perfect - it tested both knowledge and critical thinking skills and experimental design. It was obscure so no material was available for specific training.

In the end the scores were so accurate in predicting eventual success - a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, while the rest just took industry jobs (and in my opinion because they realized that’s best for them).

Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process. None of the people who got in the top percentiles (4 I know) made it to a top 10 institution in the US, while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler. It’s sad that such an awesome test is deliberately ignored by these institutions.

The irony was that these tests were also not elitist. All you needed to do was thoroughly read Lodish and Lehninger and you’re good. We did study in a semi-premium institution in India, but by no means were we privileged by any special facilities or help (at least in the context of this test). The only barrier might be the test fee itself if anything.

mradek4 tahun yang lalu

I feel like standardized tests are way more fair than GPA.

I personally struggled a lot in school due to health and other issues. I also hated doing homework which is BS work, and I always had high scores on my midterms and finals which I think matter way more.

Why does busywork play such a big role? It should be exempted. If I can get a 95+ on midterm and final I obviously understand the material. Why did I need to spend hours of my day doing BS exercises when I already understood the material? Nonsense.

So I get a B at best overall because I get a F on homework? What a joke. I really think school is a waste of time when it’s done that way. Learn stuff, move on. I wish that when I have kids I can afford to send them somewhere they can actually grow and thrive, not just sit down and do a bunch of homework problems.

(Just my $0.02)

rayiner4 tahun yang lalu

The implication of the “test scores don’t matter” argument is that we have the wrong people staffing virtually every credential-based position. There’s lots of kids with 4.0 GPAs from high school at directional state university getting education degrees. Test scores is what distinguishes those kids from the ones at Stanford. And Stanford is where everyone from Big Tech to Big Banks to Big Law to Big Consulting goes to hire. Who is working at Pfizer or Astra Zeneca doing drug development? It’s the people with high test scores. Every single Supreme Court justice in the last half century got there based on test scores. (The nominee whose test scores led her to a law school ranked outside the top 50, Harriet Miers, has to withdraw because of that.) Warren Buffett got into Wharton based on test scores. Charlie Munger got into Caltech based on his scores on the army’s intelligence test. Bill Gates got an 1590 on the SAT. Even the professors saying test scores don’t matter got where they are because of test scores.

Maybe that’s all true. But it’s quite a remarkable claim that would turn the world as we know it upside down.

faeriechangling4 tahun yang lalu

Test scores are accurate at predicting your ability to succeed at further testing, I don't know if this is all that controversial.

What people are getting up in arms about is the validity of testing as a whole regarding real life outcomes, and this is an effective argument as it is damn hard to grade people by any metric that doesn't get gamified to death and resulting in people wasting a ton of time on playing the game.

PaulHoule4 tahun yang lalu

My take is that standardized tests are a road to social mobility. People can bullshit their way to higher grades but there are some rich kids who can't test their way out of a paper bag and the one way we can stop them from laundering their parents accomplishments is standardized testing.

WBrentWilliams4 tahun yang lalu

Cute. Disclosure, I make the data (transcripts, test scores) flow to decision makers as part of my day job at a Big 10 University. We just went "test optional".

I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? The key is "test optional". If you sit for the test, you get more than just admissions, as merit-based financial awards are unlocked. The University Admissions does not require a test, but many of the donor contracts that fund scholarships do require those tests.

pwpw4 tahun yang lalu

I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented and often have to put up with a lot more barriers (such as frequently messed up accommodations on test day leading to anxiety). Usually these accommodations help normalize the specific testing scenario and people with disabilities do not need as drastic accommodations in real job setups. The tests are designed for the average test taker but have lots of hiccups for those that deviate from the average.

It’s a difficult problem to solve, but it’s unfair if a student gets easily distracted or anxious when taking a test around others in a time constrained setting. Those types of external factors generally shouldn’t be represented in a student’s test results yet often are. If tests are to be a good indicator for all students, I think accommodations have a ways to go.

If tests are indeed a generally good indicator of a student’s success and are continued to be used in such a manner, I feel it’s important that the experience is fair for all types of test takers. Many US tests such as the CPA exam do a great job of handling accommodations. However, many others (like the PSAT) have a lot of room for improvement.

vivegi4 tahun yang lalu

This whole concept of measuring SAT/ACT scores or GPAs and using them to "predict" graduation rates is bonkers.

Sounds a lot like the precogs in Philip K. Dick's Minority Report.

Several decades ago, I had a high-school History teacher. We had an elite private high-school in the same city as I lived (India has both private and public schools). They had an extremely rigorous selection process, including infamous interviews with parents for admission to kindergarten. Our history teacher once remarked, "They take horses and send them out as horses. We take horses, asses and everything else and send them out as horses".

The SAT/ACT/GPA filter seems to be focused on selecting horses and proudly declaring all horses graduated.

jklm4 tahun yang lalu

Test scores have already been irrelevant for certain segments for over a decade, speaking as someone of East Asian ethnicity. Personal anecdote: 3.7 GPA in average high school with 8 APs (some of which were self-study), 2400 SAT, 36 ACT, lots of state-level top 3 finishes in multiple academic competitions, varsity track, musical instruments, first generation immigrant, etc.

Ended up going to what I thought was my safety school and rejected/waitlisted at every reach school I applied to. I sometimes wish colleges mandated name/ethnicity-blind application reviews - not to sound ungrateful, but I’m still incredulous to this day that the best I could do was a public state school (still top 50 admittedly).

It ended up being a great experience in that it was a forcing function and made me realize most of these rules and expectations around admissions were meaningless. But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.

scarface744 tahun yang lalu

Let me tell two stories:

I graduated with the highest SAT scores in my school and the second highest in my city back in the early 90s. I’m not bragging, it was a relatively small town in the south with not great school systems and a total graduating class of 1500 across all of the schools.

What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?

Second story: my step son didn’t have the ACT scores to get into the college he wanted to go to. Four months later and with the help of a private tutor who was a teacher (not in his school) that we paid $100/hour for 12 hours, he suddenly raised his scores more than enough. Did he gain aptitude in 12 weeks or did he learn test taking techniques?

cr4nberry4 tahun yang lalu

Why is it that the anti test score argument almost always tries to chalk it up to hiring tutors? They consistently try to make it sound like people "buy" higher test scores with tutoring in order to discredit tests like the SAT

KevinBenSmith4 tahun yang lalu

Great & useful article. We should have more of these for “popular papers”.

xbar4 tahun yang lalu

My favorite point:

"2. It might be that test scores don’t predict college graduation rates[...] is provably false."

MarcoSanto4 tahun yang lalu

It'a sunken cost effect at paly, little more than that. Oh, and parents pressure, least we forget the role of family. You can test my claim empirically by asking yourself: "are all college graduates I met more intelligent than me?". Admission test measure how well you can prepare for admission tests, nothing more. There are too many confounding factors and co-factors that play in the career of a relatively fresh semi-adult aged 17-18, that reducing it to a single number is just moronic or bad-faith utilitaristic escape route. Universities only care that you graduate so they can make money, therefore they use the simplest KPI to measure that. Let's not forget Universities are staffed with people and are a business.

awkward4 tahun yang lalu

> The full dataset isn’t available, but since we have the number of students in each ACT / GPA bin above, we can create a “pseudo” dataset, with just a small loss of precision.

Big oof. This is an example of terminal data science poisoning. The data in the paper already proved the point of the article! Using it to create some chart gore where you reproduce the same plot in uncanny valley levels of resolution doesn't add to the argument. This is using very sophisticated techniques to achieve what you could by simply blurring an image of the first chart.

forgotmypw174 tahun yang lalu

It is not that they are irrelevant, it is that they are ableist, selecting for a narrow skill and ability combination, and creating a barrier for anyone else who lacks it, for whatever reason.

tagami4 tahun yang lalu

“C’s get degrees”. Measuring college graduation rate is a vanity metric

dirtyid4 tahun yang lalu

If anything over indexing on solely on test scores will filter out a large % of psycho/socialpaths who gets by on personality scores.

readthenotes14 tahun yang lalu

Did I miss the correlation across school systems? Everything in the study was from Chicago public schools.

My older brother switched from a B student in a private high school to the public high school and slept his ways to A's

bigbacaloa4 tahun yang lalu

Every metric has problem, but experience in a university suggests that, if one has to use a single metric to rank students, by far the best choice (the only choice really) is average grade.

sakex4 tahun yang lalu

I think that when people say test scores are irrelevant, they don't mean irrelevant for studies but irrelevant as an indicator of job performance.

3vidence4 tahun yang lalu

Maybe a hot take but I think school pedigree is irrelevant.

I went to a top university in my country after transferring in after not getting in from highschool. Turns out myself and many others would have got a lot of value from going to a less prestigious university. Ultimately most of us end up working at the same select companies. But idk I am from Canada so maybe the intensity of selection is not the same.

We put far too much value in prediction of success and not enough value in actual success. Seems like the academic version of keeping up with the Jones's

Ekaros4 tahun yang lalu

Test scores are the most objective metric available. And thus I really consider them fairest. Or we could just have entirely random lottery.

godelski4 tahun yang lalu

I think the author is missing some really key points here. Namely that GPA and test scores are highly correlated. I'd go so far as to say that there is a confounding variable that predicts both. I'd even say there are several upstream predictors, but I think we all know that (or I thought we did). Every single graph they show demonstrates this yet they keep missing it. The top comment right now (by ajwinn[0]) notes that they used a heatmap like the one in the article for determining admission. This is great because it builds multiple pathways to gain admission. That's honestly how we should treat variables like these: rely on either.

As to explain the confusion in the models, well when you have confounding variables like this it is easy to reduce one variable's weight nearly to zero and place the rest in the other variable. So if I'm right (might not be) I think you could also show that the inverse is true, that grades aren't a good predictor and test scores are. The breakdown suggests that this is true too tbh.

But I also disagree with some of the analysis and techniques here. It's not always easy to use models that have continuous(ish) variables (GPA/ACT/income) and discrete ones (race/gender). I bring this up because there's some hints that they aren't a stats expert. For example there's the line

> The “likely-to-graduate” score becomes a probability after a sigmoid transformation.

Which just isn't true. A Softmax function[1] creates a probability distribution but a Sigmoid[2] doesn't. A Sigmoid bounds the output to [0, 1] (or [-1,1]) but being bound on that range isn't the only requirement to a probability distribution. A sigmoid does not guarantee that the sum of the elements is 1 (cumulative density), but a softmax does. In fact, it almost never will. (I mention this in my ML class a lot so make sure you all don't make this mistake too! I'll be really annoying if you do)

There's some more errors and novice approaches here and so I just want to note this because we shouldn't just take the results and conclusions as is when the modeling is bad. But I still think he makes some good (and bad) points and those are definitely up for discussion.

[0] https://news.ycombinator.com/item?id=33338809

[1] https://en.wikipedia.org/wiki/Softmax_function

[2] https://en.wikipedia.org/wiki/Sigmoid_function

qwerty4561274 tahun yang lalu

In my opinion discriminating against people given their past test scores is unfair. People can improve and having scored bad (which might well be a consequence of hard life circumstances at a specific moment of their life) in the past should not be allowed to ruin their future. So I believe nobody should be allowed to see one's previous test scores and judge them this way. Whenever it is legitimately necessary to assess a person's proficiency a new test should be taken.

RobLach4 tahun yang lalu

Who's saying that test scores are uncorrelated with college graduation, which is largely dependent on passing further tests?

rajin4444 tahun yang lalu

Does the same hold true of medical professionals? ie MCAT scores correlate with med school and/or practice performance?

faangiq4 tahun yang lalu

I’d gladly hire on test scores alone.

olliej4 tahun yang lalu

I think there’s an issue on how these data are examined.

There is a huge amount of data backing up the claim that the school attended influences the SAT, etc.

There is a huge amount of evidence backing up the claim that SAT/ACT aren’t 100% predictive of later academic success.

But “not being 100% predictive” does not mean “is not predictive”.

The core problem the removal of SAT, etc is attempting (imo hamfistedly) to mitigate is the following chain:

(1) Student A comes from affluent family, in an affluent region.

(2) Student A goes to a private school with better teachers, etc, or because it’s an affluent region the public schools have better teachers, etc

(3) Student A’s family can afford additional SAT, etc tutors

Now imagine a somehow identical Student B, but from a poor family in a poor region. Before any other factors become involved, they don’t get the benefits of 2 or 3.

We would expect that on average 2&3 result in higher scores, and so a higher acceptance rate, and a higher rate of scholarships for Student A vs Student B. Again this is on average.

The TLDR is given two kids who have identical mental ability, the one from the richer family will, on average, have better scores. So schools say let’s drop the scores entirely, but that is also clearly nonsense: Even in this exact scenario Student A will do better in uni or what have you, because part of why they got a better score is that they have been taught and learned more - IE they absolutely do know more and are more academically proficient than Student B by this point. Moreover we know (because of yet more studies), Student A will tend to continue to do better than Student B once they’re at school, even once B “catches up”, because we know that people with fewer external stressors perform better at everything.

So even if we discard the score based entry to unis, we’ll see a strong correlation between original score and subsequent academic success.

So while I understand the desire to remove the “accident of birth” factor that benefits the lucky and penalizes the unlucky, simply removing the test requirements is fairly questionable as an approach.

This is also before asking about Students C and D who are the peers of students A and B - surely the better scoring of A and C (or B and D) is who you should prefer, but now you’re discarding theoretically ideal and unbiased data in favour of…?