Technically, I think it's not that instrumental goals tend to converge, but rather that there are instrumental goals which are common to many terminal goals, which are the so-called "convergent instrumental goals".
Some of these goals are ones which we really would rather a misaligned super-intelligent agent not to have. For example:
You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.
As the pre-amble to the statement says: they kept the statement limited and succinct as there may be disagreement between the signatories about the exact nature of the risk and what to do about it.
The emergence of something significantly more intelligent than us whose goal are not perfectly aligned with ours poses a pretty clear existential risk. See, for example, the thousands of species made extinct by humans.
It's essentially when information that was used to identify the suspect is presented as confirmation of their guilt.
> The witness described a tall, white, black-haired man with a tattoo on his right ankle. Only .001% of the population match this description, so there is only a .001% chance that the suspect is innocent.
Of course, this ignores the fact that they compared the description against a database of one million people, among whom you would expect to find 10 coincidental matches.
For the same reason more data is required to select a model than to validate a model in machine learning.
Also, the way mathematics is taught at high school might not prepare you well for the way you use it at university.
Maths in high school focuses more on applying methods (taking a derivative etc.); maths at university (even on a CS course) is more about constructing mathematical models and arguments.
It can almost feel like a completely different subject.
Yes. It is odd that in the British system at age 16 you can choose to study mathematics and physics and literally nothing else. (I blame this for my poor history knowledge.)