University Rankings are Awful. Can We Stop Doing the Same Thing With Countries?(rottenindenmark.wordpress.com)
rottenindenmark.wordpress.com
University Rankings are Awful. Can We Stop Doing the Same Thing With Countries?
http://rottenindenmark.wordpress.com/2013/09/10/university-rankings-are-terrible-now-can-we-stop-doing-the-same-thing-with-countries/
13 comments
Right. The rankings aren't the last word on an institution. They aren't even close to it. But that doesn't mean they're meaningless. Similarly with countries. These metrics refer to real data corresponding to the actual, real-life experience of people living in these places. If the number of emergency rooms per capita in a country is low, for example, that corresponds to actually higher likelihood that you'll be unable to get to one if you live in that country and are in an emergency situation. The data is real - the author might not like that it seems like a simplistic representation and glosses over a lot of nuances, but then why else do we measure and quantify these trends than to tease out meaningful information from an otherwise incomprehensible, highly complex system?
Even though a single year snapshot of college rankings isn't completely meaningful by itself, you can glean a lot from the movements in the rankings, even with small movements. If a college goes from 200 to 190 in a single year, you can probably determine precisely what changed how by looking at the changes in the internals that they used to derive the number. And sometimes even a small change in ranking does reflect a significant, concrete change in the experience of students attending these institutions.
Similarly, with countries. The dynamics over time of these metrics is what gives this data meaning - not any single ranking.
Even though a single year snapshot of college rankings isn't completely meaningful by itself, you can glean a lot from the movements in the rankings, even with small movements. If a college goes from 200 to 190 in a single year, you can probably determine precisely what changed how by looking at the changes in the internals that they used to derive the number. And sometimes even a small change in ranking does reflect a significant, concrete change in the experience of students attending these institutions.
Similarly, with countries. The dynamics over time of these metrics is what gives this data meaning - not any single ranking.
Author is getting paid to organize information on a multitude of nations and their employment and economic prospects and human rights conditions. Businesses need guidance on where they can get cheap foreign labor to assemble products without long term problems. Author is the one who is charged with solving the problem for them.
Then the author decides that businessmen are shallow jerks because they just want the answer to their problems. They want help to pick a country and guidance about how to exploit the low wage workers without abusing anyone. Author thinks they should instead deeply explore the relative conditions and balance the ineffable cultural and economic differences in the deep way that only a full-time professional country analyst can. They should be like him.
But the businessmen are paying him to do that so they don't have to.
Perhaps the author would be happier with a nice UN or civil service job where he's not accountable to anyone for any kind of useful information. Ideally it would be a job where he can't be fired for self-indulgently coddling his broad-minded curiosity and refusing to give useful answers. Don't worry though; he's not the kind of high-minded idealist who refuses to make his money sending work to the lowest cost country. He's probably an expert on the authentic third-world experience of getting ladies to do his laundry for him cheap.
Then the author decides that businessmen are shallow jerks because they just want the answer to their problems. They want help to pick a country and guidance about how to exploit the low wage workers without abusing anyone. Author thinks they should instead deeply explore the relative conditions and balance the ineffable cultural and economic differences in the deep way that only a full-time professional country analyst can. They should be like him.
But the businessmen are paying him to do that so they don't have to.
Perhaps the author would be happier with a nice UN or civil service job where he's not accountable to anyone for any kind of useful information. Ideally it would be a job where he can't be fired for self-indulgently coddling his broad-minded curiosity and refusing to give useful answers. Don't worry though; he's not the kind of high-minded idealist who refuses to make his money sending work to the lowest cost country. He's probably an expert on the authentic third-world experience of getting ladies to do his laundry for him cheap.
> I don’t know if high schoolers use college rankings to decide where they should get educated.
I've found that some colleges have a reputation centered around specializations, for example biotech, compsci, civil engineering, and so forth. Prospective students make decisions based on this reputation, and it's not difficult to assign a number to that. Whether reputation means anything qualitative is something else entirely, though.
More relevant, though, is that students also use a variety of other factors in their decision making. Cost and relative location (ie: distance from home of closest family, distance to place of work after graduating) are high-priority factors, meanwhile student body composition (ie: asian vs white) is another.
My point is that companies and people don't make decisions based on a single and opaque number, it's just one of many variables in determining where to go and what to do.
The factors I listed above can have numeric values assigned to them, and a single metric can be derived from a combination of these numbers, however that's still somewhat opaque.
It may be unfair to assign a number to an entire country, but that doesn't diminish its value. It would be unwise to use a single metric as input for decision making, and again that doesn't diminish the value of that metric. It's just another metric.
I've found that some colleges have a reputation centered around specializations, for example biotech, compsci, civil engineering, and so forth. Prospective students make decisions based on this reputation, and it's not difficult to assign a number to that. Whether reputation means anything qualitative is something else entirely, though.
More relevant, though, is that students also use a variety of other factors in their decision making. Cost and relative location (ie: distance from home of closest family, distance to place of work after graduating) are high-priority factors, meanwhile student body composition (ie: asian vs white) is another.
My point is that companies and people don't make decisions based on a single and opaque number, it's just one of many variables in determining where to go and what to do.
The factors I listed above can have numeric values assigned to them, and a single metric can be derived from a combination of these numbers, however that's still somewhat opaque.
It may be unfair to assign a number to an entire country, but that doesn't diminish its value. It would be unwise to use a single metric as input for decision making, and again that doesn't diminish the value of that metric. It's just another metric.
You have missed the entire point of the article. Arbitrarily quantifying essentially qualitative measures is misleading (bordering on dishonest) and potentially harmful, when these made-up numbers are then used to influence major decisions.
E.g. these bogus college rankings are used to set immigration and other government policy in many countries: http://www.theguardian.com/higher-education-network/blog/201...
The more operations you perform on these made-up numbers, like the "combination" you mention, the more authentic they might seem. But the result of averaging a bunch of made-up numbers is just as meaningless, if not more so.
E.g. these bogus college rankings are used to set immigration and other government policy in many countries: http://www.theguardian.com/higher-education-network/blog/201...
The more operations you perform on these made-up numbers, like the "combination" you mention, the more authentic they might seem. But the result of averaging a bunch of made-up numbers is just as meaningless, if not more so.
A useful way to compare countries is by revealed preference: where people choose to immigrate, given a chance. So far the United States fares well in that kind of comparison, because as much as people complain about the United States--including people like me who live here--there are many people who think the overall trade-off of living in the United States is better for them than the trade-off involved in living somewhere else, even a place where they grew up and where all their friends and relatives live. Net immigration flows are pretty strongly suggestive of where people find a good life. International surveys show a strong preference (not always realized) to live in the United States.[1] I see a lot more Hacker News threads complaining that immigration to the United States is too difficult than I see about leaving the United States being difficult. (Leaving the United States is ridiculously easy, and I have done that more than once. But even many people who leave to live abroad eventually come back.)
[1] "More Than 100 Million Worldwide Dream of a Life in the U.S." http://www.gallup.com/poll/161435/100-million-worldwide-drea...
[1] "More Than 100 Million Worldwide Dream of a Life in the U.S." http://www.gallup.com/poll/161435/100-million-worldwide-drea...
Simply looking at net migration ignores all sorts of issues. The first is that most peoples impressions of what a country would be like to live in are quite different from the reality. The US is the figurehead of the developed world, and so of course those in less developed countries would say they dreamed of a life there. The reality is that almost none of those people actually know what their life - the life of an immigrant - would be. It's really not hard to find immigrants all over the developed world who are completely disillusioned with their life and would jump at a chance to return home. Often what keeps people in the country they emigrate to isn't some deep love of their new life, but circumstance - children, too little money to go home, too much pride to return home having admitted they've failed or familial expectations for money to be sent home. The vast majority of immigrants to the developed world live in near poverty.
I often find that even British people's perception of the US and Canada is quite different to the reality that I've experienced. It's driven by the media portrayal and perhaps brief holidays. For people from very different backgrounds that gulf will be far, far greater.
The second is that the level of immigration controls a country sets directly impact the net migration into the country. Regardless of how many people want to live somewhere, if they don't let you in then it doesn't really matter. It's very hard to use immigration as a metric without compensating for that somehow.
I often find that even British people's perception of the US and Canada is quite different to the reality that I've experienced. It's driven by the media portrayal and perhaps brief holidays. For people from very different backgrounds that gulf will be far, far greater.
The second is that the level of immigration controls a country sets directly impact the net migration into the country. Regardless of how many people want to live somewhere, if they don't let you in then it doesn't really matter. It's very hard to use immigration as a metric without compensating for that somehow.
And Russia is second in this comparison.
Where can I find more published information about this issue? Where are the immigrants coming from?
This was posted on Hacker News a while back, I'm sure there where interesting comments: http://peoplemov.in/
Original discussion here: https://news.ycombinator.com/item?id=5133098
The rankings for migrant destinations are:
USA 42,788,029
RUSSIAN FED. 12,270,388
GERMANY 10,758,061
SAUDI ARABIA 7,288,900
CANADA 7,202,340
UNITED KINGDOM 6,955,738
SPAIN 6,900,547
FRANCE 6,684,842
AUSTRALIA 5,522,408
INDIA 5,436,012
Data is sourced from the World Bank
Original discussion here: https://news.ycombinator.com/item?id=5133098
The rankings for migrant destinations are:
USA 42,788,029
RUSSIAN FED. 12,270,388
GERMANY 10,758,061
SAUDI ARABIA 7,288,900
CANADA 7,202,340
UNITED KINGDOM 6,955,738
SPAIN 6,900,547
FRANCE 6,684,842
AUSTRALIA 5,522,408
INDIA 5,436,012
Data is sourced from the World Bank
Oh, I see the source at the earlier Hacker News discussion also lists Russia as a high-ranked country for people moving out. That was my curiosity about the issue: this town is full of immigrants from Russia.
From that article:
Top emigration countries
MEXICO 11,859,236
INDIA 11,360,823
RUSSIAN FED. 11,034,681
CHINA 8,344,726
UKRAINE 6,525,145
BANGLADESH 5,384,875
PAKISTAN 4,678,730
So, yes, I was speaking of NET immigration, for which the United States still seems easily to be the top-ranked country. The point in another reply is correct that laws about immigration limit freedom of movement to some extent, but that is also revelatory about what people desire. The United States doesn't have to do anything to keep people already here in, but its government has been called on by some domestic voters to make it hard for just anyone to come in from outside.
From that article:
Top emigration countries
MEXICO 11,859,236
INDIA 11,360,823
RUSSIAN FED. 11,034,681
CHINA 8,344,726
UKRAINE 6,525,145
BANGLADESH 5,384,875
PAKISTAN 4,678,730
So, yes, I was speaking of NET immigration, for which the United States still seems easily to be the top-ranked country. The point in another reply is correct that laws about immigration limit freedom of movement to some extent, but that is also revelatory about what people desire. The United States doesn't have to do anything to keep people already here in, but its government has been called on by some domestic voters to make it hard for just anyone to come in from outside.
And here's just one obvious shouting indication in the data of how complex reality is. We see that the USA is the number one destination country in worldwide immigration upthread. Then we see in the parent that Mexico is the number one country for emigration.
But over the past five years, in every single year, according to the Pew Hispanic Center, there have been more people moving from the USA to Mexico than the reverse. And Russia is high up on both lists; that demands some kind of explanation, too.
You're going to need a lot more detailed information about what is really motivating migration before you can use numbers like that to measure the desirability of any nation for residence or citizenship.
But over the past five years, in every single year, according to the Pew Hispanic Center, there have been more people moving from the USA to Mexico than the reverse. And Russia is high up on both lists; that demands some kind of explanation, too.
You're going to need a lot more detailed information about what is really motivating migration before you can use numbers like that to measure the desirability of any nation for residence or citizenship.
If you take <a href="http://databanksearch.worldbank.org/DataSearch/LoadReport.as... World Bank net migration data for 2010</a> Russia is still near the top while the second place is taken by UAE which seem to be just a little worse than the USA J
United States 4 954 924,00
United Arab Emirates 3 076 634,00
Spain 2 250 005,00
Italy 1 998 926,00
Russian Federation 1 135 737,00
Keep in mind that about immigration perception in Russia:
Russians don't consider most Ukrainians/Belarussians immigrants (I assume americans would not consider canadians who just happened to work in the USA immigrants as well)
But peoplemov.in statistics wildly understate the number of immigrants from Central Asia who often aren't properly accounted for or just live here illegaly because of porous borders.
Russians don't consider most Ukrainians/Belarussians immigrants (I assume americans would not consider canadians who just happened to work in the USA immigrants as well)
But peoplemov.in statistics wildly understate the number of immigrants from Central Asia who often aren't properly accounted for or just live here illegaly because of porous borders.
Central Asia, Ukraine/Belarus, Transcaucasus countries, Moldova.
The reason rankings exist is because often, in the end, a decision MUST be made. Are current rankings possibly worse than a random draw? - conceivably, in some cases, but that doesn't have to be the case.
Your concerns about ranking-context and the general difficulty of ranking are, however, well placed. Solving the ranking problem, for the general case, is a first-class problem on the same level of difficulty as computer vision or language processing.
Personally, I think the world deserves a better approach to solving general ranking.
Your concerns about ranking-context and the general difficulty of ranking are, however, well placed. Solving the ranking problem, for the general case, is a first-class problem on the same level of difficulty as computer vision or language processing.
Personally, I think the world deserves a better approach to solving general ranking.
Eventually I realized that the only reason the companies pushed so hard, why they insisted so strongly on rankings and scores over information and analysis, was because it made it not their problem anymore. They didn’t have the credentials to pull 50 ‘good’ countries from 100 uncategorised ones, so they used us to push the responsibility away. ‘It’s not me saying Bolivia is an 8.2,’ they could tell their boss. ‘A human rights NGO said it was. Making shoes there is totally approved.
What else would you expect the corporation to do? Eventually, someone in that organization has to make a decision, and that requires the complex information about countries' human rights situations to be summarized in a simpler way -- for which a ranking is one approach.
It sounds like they wanted the NGO to do this summarization. If the NGO didn't do it, then they'd get someone else or from within their organization to do it.
Something like rankings are inevitable. They fulfill a huge information need. The fact there are lots of poorly done rankings out there won't change that. This just means, it's even more important to make useful rankings that reflect good information.
(And besides rankings, there are perhaps coarser-grained options, like making several tiers of groupings, only asserting each is an equivalence class ... or finer-grained options, like giving each country scores in three different categories. Etc. But in all cases, the incredibly rich and complex source information has to be summarized somehow.)
What else would you expect the corporation to do? Eventually, someone in that organization has to make a decision, and that requires the complex information about countries' human rights situations to be summarized in a simpler way -- for which a ranking is one approach.
It sounds like they wanted the NGO to do this summarization. If the NGO didn't do it, then they'd get someone else or from within their organization to do it.
Something like rankings are inevitable. They fulfill a huge information need. The fact there are lots of poorly done rankings out there won't change that. This just means, it's even more important to make useful rankings that reflect good information.
(And besides rankings, there are perhaps coarser-grained options, like making several tiers of groupings, only asserting each is an equivalence class ... or finer-grained options, like giving each country scores in three different categories. Etc. But in all cases, the incredibly rich and complex source information has to be summarized somehow.)
I share the author's aversion to reducing complex information to rankings. At the same time, at first reflection it seems that countries would have less incentives than universities to "juke their stats" (to use the expression in the article). Of course, economic indicators, crime statistics and the like are liable to be fudged for political purposes, but my impression is that countries don't compete with each other on rankings the way universities do.
While I can't say whether this is true on a larger scale, you may be interested to learn that Paul Kagame, the ruler of Rwanda, put together a team specifically for the purpose of maximizing his country's ranking on the World Bank's ease-of-doing-business index.
[1] http://www.nytimes.com/2013/09/08/magazine/paul-kagame-rwand...
[1] http://www.nytimes.com/2013/09/08/magazine/paul-kagame-rwand...
That's a great example. I just hope that at least the changes they made to maximize Rwanda's ranking on the ease-of-doing-business index were genuinely beneficial rather than merely cosmetic.
College rankings are foolish.
That said, at least ranking organizations give a real opinion. If your job is to assess human rights violations, you should numerically assess them. Qualitative rankings are a weak form of analysis.
Someone has to pick the country, why not the NGO tasked with doing so?
That said, at least ranking organizations give a real opinion. If your job is to assess human rights violations, you should numerically assess them. Qualitative rankings are a weak form of analysis.
Someone has to pick the country, why not the NGO tasked with doing so?
I generally agree, it is easy to define good human rights for a country.
But this gets hairy when you start evaluating how bad a non-perfect country is.
Consider press freedom.
Today, e.g. Turkey and Russia keeps the press down and stop dissenting voices. How much worse is that than e.g. Sweden, where the press goes in perfect lock step in certain areas -- few dissenting voices are allowed. (Most of that is self censure. Who choose what is censored and how? Only journalists know. Maybe. Part of this is cultural, too. How much? I have no clue.) Note that this one-way thinking is quite a bit different in the rest of Scandinavia.
Or a very intolerant country might be OK to live in, for a homosexual or someone of the "wrong" faith. Live and let live, the "different" would just get lots of curious questions.
Compare that with my present home country, which is a bit homophobic. The only openly gay guy I've met was a friend of a work friend. He seems to exercise to be a good street fighter. I hope that is just a coincidence. :-)
But this gets hairy when you start evaluating how bad a non-perfect country is.
Consider press freedom.
Today, e.g. Turkey and Russia keeps the press down and stop dissenting voices. How much worse is that than e.g. Sweden, where the press goes in perfect lock step in certain areas -- few dissenting voices are allowed. (Most of that is self censure. Who choose what is censored and how? Only journalists know. Maybe. Part of this is cultural, too. How much? I have no clue.) Note that this one-way thinking is quite a bit different in the rest of Scandinavia.
Or a very intolerant country might be OK to live in, for a homosexual or someone of the "wrong" faith. Live and let live, the "different" would just get lots of curious questions.
Compare that with my present home country, which is a bit homophobic. The only openly gay guy I've met was a friend of a work friend. He seems to exercise to be a good street fighter. I hope that is just a coincidence. :-)
'How much worse is that than e.g. Sweden, '
I'm going to go ahead and say that Turkey and Russia are still, pretty clearly, worse in this area, and you could probably quantify this. It's not good that Sweden's press isn't quite as diversified as it could be, but I think we can both agree that the constitutional protections in Sweden are objectively better than those in Turkey and Russia.
It might be hard to quantify these differences, but that doesn't mean it's impossible, and it doesn't mean that attempts to do so are even nearly worthless.
I'm going to go ahead and say that Turkey and Russia are still, pretty clearly, worse in this area, and you could probably quantify this. It's not good that Sweden's press isn't quite as diversified as it could be, but I think we can both agree that the constitutional protections in Sweden are objectively better than those in Turkey and Russia.
It might be hard to quantify these differences, but that doesn't mean it's impossible, and it doesn't mean that attempts to do so are even nearly worthless.
Well, according to the Committee to Protect Journalists, today Turkey tops the world in the number of imprisoned journalists: http://cpj.org/imprisoned/2012.php
No journalists are jailed in Russia at the moment, but Russia comes in 4th in the world in the number of journalists killed since 1992 with 56 deaths: http://cpj.org/killed/ Turkey is 13th with 21 deaths.
Statistics are indeed useful when we're separating these sort of extremes. The article gives the example of Norway being less corrupt than Angola, asserting that you "don't need an index" to tell you that. But comparing Norway and Denmark is where rankings are less useful, according to the author. The argument is not that it is always impossible to compare different countries in quantitative terms.
No journalists are jailed in Russia at the moment, but Russia comes in 4th in the world in the number of journalists killed since 1992 with 56 deaths: http://cpj.org/killed/ Turkey is 13th with 21 deaths.
Statistics are indeed useful when we're separating these sort of extremes. The article gives the example of Norway being less corrupt than Angola, asserting that you "don't need an index" to tell you that. But comparing Norway and Denmark is where rankings are less useful, according to the author. The argument is not that it is always impossible to compare different countries in quantitative terms.
My point is that the author of the paper wants to be paid to work for an NGO but doesn't believe in actually solidifying their opinion. Listing a bunch of stuff is not the same as saying this list means you should a) Build a product in this country, or B) You should not build a product.
Essentially he is being paid for very basic research which is not useful without an expert in the field. He wants a guy at Big Corp. to make the final decision so he can wash his hands of it.
I don't support this type of thinking. I would prefer that the author openly select the best country and stand by his opinion. Maybe it doesn't make sense to rank country A #132 and country B #135, but it definitely makes sense to give them a rating in several categories and draw a distinct line.
Or possibly, there is something I don't understand about the business.
Essentially he is being paid for very basic research which is not useful without an expert in the field. He wants a guy at Big Corp. to make the final decision so he can wash his hands of it.
I don't support this type of thinking. I would prefer that the author openly select the best country and stand by his opinion. Maybe it doesn't make sense to rank country A #132 and country B #135, but it definitely makes sense to give them a rating in several categories and draw a distinct line.
Or possibly, there is something I don't understand about the business.
To defend the author's position, I would be wary of providing a single ranking because the metrics are going to be different for different needs. Say you're ranking different lodging options at a holiday destination. You can come up with a single ranking for all the options, whether they are hotels, hostels, bed & breakfast inns. But it would be meaningless except to weed out the really bad options unless you knew what the preferences would be in terms of price, location, atmosphere, decor, etc. Such an index could still be useful, perhaps, but I see where the author is coming from.
I be honest, as far as I'm concerned you're in the PR business anyway so roll with it.
How does removing business from a country help? Will work place health and safety go up now your employers are getting less money and jobs from overseas?
More people are unemployed, so the child death rate will increase as they can't afford healthcare.
The concept that companies would chose Norway, one of the richest counties in the world over one of the poorest is pretty evil.
How does removing business from a country help? Will work place health and safety go up now your employers are getting less money and jobs from overseas?
More people are unemployed, so the child death rate will increase as they can't afford healthcare.
The concept that companies would chose Norway, one of the richest counties in the world over one of the poorest is pretty evil.
For a different set of rankings created (pulled?) from somewhere else: http://www.topuniversities.com/university-rankings/world-uni...
Commentary with a British focus: http://www.bbc.co.uk/news/education-24024767
Commentary with a British focus: http://www.bbc.co.uk/news/education-24024767
Unfortunately, if the US News' rankings suffer from certain problems, these rankings suffer much more heavily. These rankings compare universities that have far more quantitative differences than the US News' rankings; they compare sets of universities that don't share a language, structure, culture, etc...
I agree that university rankings suck, and I started a website to address it: http://www.collegeanswerz.com/.
[deleted]
What's the matter, tired of ranking at the bottom?
But sometimes progress can only be made by focusing on "simple" indicators -- things like childhood mortality rates, whether applied to a country or to a single hospital.
For a college to decide it wants to go from #200 to #190 in a particular ranking is meaningless, but going from #200 to #20 is going to be meaningful. Likewise the president of a country saying, "it's atrocious that we're one of the 20 most corrupt, or red-tape-laden, countries" -- it can provide the necessary impetus to change and improvement.
These rankings are certainly not "entirely unsubstantiated" -- they're certainly very far from perfect, but don't throw the baby out with the bathwater.