LLMs with a verification layer work great (code with tests)
I know my field quite well and I can one-shot many useful things. I can't trust any of it but I can trust tests and verification tools.
When you set up your Claude Science instance you can see that they're connecting to crossref, semantic scholar, pubmed, ArXiv, FDA. They instruct the LLM to validate citations.
My testing with this technique indicates that method they seem to be using (rag with an instruction to check sources) will reduce the confabulation rate for citations from the base rate 50-60% for regular models (e.g. regular Claude) to 5-15% (depending on how they implemented it).
On the one hand this is way better. On the other hand it's just good enough that your spot check will look good and your work will still contain hallucinations (which is probably worse than obviously bad).
Getting to zero confabulation would require a different process. (stand-alone validation engine running in parallel in real-time which is hard but not impossible.)
Quite beneficial to keep fake science out of the scholarly record. The good news is that bad papers will show multiple signals of misconduct.
This paper was not retracted for citation fraud, but a citation check gives it a zero score (desk reject)
"Overall Score
0 / 100
High Risk
Feedback Summary
The citation analysis identified significant issues verifying the existence of cited references and references with potentially weak topical alignment.
Document-Level Findings 3
Source location and matching issues
We were able to verify 117 of 144 references against sources. 1 not found, 26 potential mismatches.
One or more references exhibit patterns associated with higher citation risk, which can arise from citation errors or automated reference generation."
We did a test of GPT5 yesterday. We asked it to generate a synopsis of a scientific topic and cite sources. We then checked those sources. GPT5 still hallucinated 65% of the citations.
It did things like:
Make up the paper title
Make up the authors for a real paper title
Mix a real title and a real journal
If it can't even reference real papers it certainly can't be trusted to match up claims of fact with real sources.
Current AI tools generate citations that LOOK real but ARE fake. This might not be solvable inside the LLM. If anyone could do it, it'd be OpenAI. (OK maybe I'm giving them too much credit, but they have a crap-ton of money and seem to show a real interest in making their AI better)
If it can't be done in the LLM we can't trust LLMs basically ever.
I suppose there's a pretty big loophole here. Doing it outside the LLM but INSIDE the LLM product would be good enough.
The first AI tool to incorporate that (internal citation and claim checking) will win because if the AI can check itself and prevent hallucinated garbage from ever reaching the user we can start to trust them and then they can do everything we've been promised.
Until that day comes we can't trust them for anything.
I can't decide who should be more mortified the publisher or the author.
The good news is that publishers will soon have access to automated citation checking.
The problem is that LLMs are just convincing enough that people DO trust them which is sort of a problem since AI slop is creeping into everything.
What can be done to solve it (while not perfect) is pretty powerful. You can force feed them the facts (RAG) and then verify the result. Which is way better than trusting LLMs while doing neither of those things (which is what a lot of people do today anyway). See the recent 5 cases of lawyers getting in trouble for ChatGPT hallucinating citations of case law.
LLMs write better than most college students so if you do those two things (RAG + check) you can get college graduate level writing with accurate facts... and that unlocks a bit of value out in the world.
Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. The good news is that it hasn't been solved yet so if someone wants to solve it there might be money on the table.
I'd buy a mini PC on craigslist for $100-200. You can usually get one with 16gb of ram and i5, a 500gb ssd for that. Install a debian based linux on it (ubuntu is very common and common is good for training) Be sure to get one with the power adapter if you go ebay.
Configure it fresh from install using ansible. Prefer ansible built-ins when possible. If you're calling shell scripts you're probably doing it wrong. Create named user accounts and create keys for that user. grant your named user sudo nopasswd and disable password login. Named users are preferable for security audit purposes. ("Shit who logged in at 2PM on tuesday and deleted critical resources?", "dunno it was the 'ubuntu' user", "great I wish we logged in as ourselves so we at least had some way of knowing who's account got compromised")
Learn how to troubleshoot ssh login problems (-v), did you know that you can't log in if the permissions on the authorized keys file are wrong?
Enable and configure unattended-upgrades. Learn how UFW works and learn how to show iptables rules. Learn how to check which services are running, which are listening, which tcp connections are active.
Learn how to write a systemd start/stop script (it's like 5 lines, super easy)
Learn how to tail and grep logs to diagnose problems. Learn how to use find and grep.
When using ssl you'll probably use let's encrypt and you'll probably want to renew automagically with certbot and dns verification.
Create an elk stack, ship some logs to it. (maybe loki too)
Hold off on k8s for now. It is advanced and you're wasting your time and your most precious resource (your ability to work through frustration) if you beat your head against it too early. When you learn k8s assume the cluster is already set up and learn to deploy a single service (there's minikube and k3s for this sort of thing). Anyone running k8s already has a lot of k8s skills (or they shouldn't be using k8s to begin with). Contribute in other ways. If they don't have a lot of k8s skills and they're using it 1) they don't need a junior, they need a senior and 2) they need to get onto something simple and easy like ecs. And if they already screwed up that bad they probably need to move to heroku. (but i digress)
Run through some command line practice, learn awk, sed, the general way commands are structured (command, options, arguments). Learn to chain commands. Learn vi (did you know you can call arbitrary shell commands?). Learn how to replace lines in config files with sed, learn how to concatenate content to the end of a file, learn how to do that for a file owned by root (hint, look into tee)
Learn how to deploy code with github actions
Learn Terraform
Take the free online AWS solutions architect training courses and begin taking practice exams. The other clouds are clones. Azure with a preference for active directory, ghoogle with a preference for unnecessary complexity and pedantic bullshit.
Skip configuring a mail server, if you're doing that you're probably doing it wrong. (hint, you'll be hitting a mail sending service. Several of the small ones have a free tier which is great for learning). You'll need some more info here but you can search it up.
When learning bash scripting focus on readability and maintainability.
The Google style guide is a great reference.
https://google.github.io/styleguide/shellguide.html
In bash there are 5 ways to do something, the best way is the most readable way.
The person who thanks you may very well be yourself.
While we're on the topic, Google's SRE book is free online and epic. Learn the section on postmortems backwards and forwards. If you got a job and only brought the google postmortem template and culture with you you could improve nearly every company in the world.
Same with improving their code testing and deployment.
Same with doing simple cloud security and cost accountability tasks (like reviewing network perimeter and cost control suggestions in trusted advisor, finding unpatched servers and patching them, resolving dependbot vulnerability notices, etc.)
Find a community where people are pretty chill and ask.
You'd be surprised how often simply asking for what you want works.
(perhaps that is lesson #1)
Post some contact info in your bio, maybe someone here is looking for a mentee.
I was tested recently and my doctor said I had dangerously low Vitamin D. It was so low they were going to proscribe high dose supplements. I eat very healthy, mostly salads and a bit of dairy and fish here and there. I bet a lot of people didn't get enough sunlight during covid. Also people with darker skin need more sun exposure to get enough. (Among Black people in snowy climates something like 70% of the population have a Vitamin D deficiency)
If you take over-the counter supplements and follow the instructions you're unlikely to overdose.
I'm fascinated by Vitamin D and it's effect on the immune system. Particularly the association between deficiency and increased risk of cancer and cardiovascular disease. We put together a tool that scrapes semantic scholar, summarizes and key word extracts and then maps keywords in relation to each other. We chose that topic as one of our sample dashboards.
One weird thing I noticed was that Vitamin D supplementation, while it improves outcomes in terms of cancer, does not seem to improve risk of cardiovascular disease. Maybe it's just that people who get sufficient Vitamin D from sunlight are more likely to get outdoors and exercise ;-) But it made me wonder if there's some additional benefit of sun exposure, some vitamin we haven't discovered yet.
Weirdly, it seems that the equations for time work in either direction. Questions of causality aside it might be possible to get something to move backwards in time. Probably not anything living though. Or rather not anything you’d like to keep living.