AI evals are becoming intractable for the same reasons that measuring human general intelligence has been intractable for 150 years: the constructs resist decomposition, the benchmarks saturate, and the goalposts move. The reasons are the same because the thing being measured is the same. The difficulty is the evidence.
Something is happening right now that echoes the trends Clay Shirky was observing in those early days of Web 2.0. Perhaps it will turn out differently this time.
Skill at “steering” coding assistants may soon be the quality most sought after in software engineers, systems administrators and the like. It's a balance between a light touch and a firm focus.
Our entire system for measuring the value of cognitive work is denominated in human time. But not even machines realize they’re not on human time. That’s a problem.
We regularly have 92% - 93% participation in federal elections here in Australia. Having one next weekend, and already record numbers of pre-poll votes.
De-Risking AI", Wisely AI's latest white paper, highlights and explains five new risks of Generative AI tools: anthropomorphising; malicious and commercially protected training data; hallucinations; privacy, data security and data sovereignty; and prompt attacks. The white paper addresses each in detail, and suggest strategies to mitigate these risks. It's part of our core mission to "help organisations use AI safely and wisely."
Ask yourself if this is work you _want_ to do. What purpose does it serve - not just those you're teaching, but yourself? Can this be something you learn from too? Can it help open doors?
One thing to remember: people value work that they pay for. Something that is given freely has no value. And that's not something that can be changed.
If you do say no, say no as politely as possible. Make the no an opportunity, rather than a closing of a door.
Yes, I've been at two startup weekends (Sydney and Melbourne). They're amazing. Come prepared to pitch your idea (if you have one) and prepared to work on your idea (if it gets up) or with a team on someone else's idea.
The winners generally have something more than a good pitch at the end of the weekend. A demo of some sort says a lot more, to the judges, than a lot of fancy slides.