Is this level of fear typical or reasonable? If so, why doesn’t Anthropic / AI code gen providers offer this type of service? Hard to believe Anthropic is not secure in some sense — like what if Claude Code is already inside some container-like thing?
Is it actually true that Claude cannot bust out of the container?
I clicked 3 different links failing at answering that question for myself and then I stopped caring, though not enough to pass up one-upping your comment.
What is your goal? if d1, d2, d3, etc is the dataset over which you're trying to optimize, then the goal is to find some best performing d_i. In this case, you're not evaluating. You're optimizing. Your acquisition function even says so: https://rentruewang.github.io/bocoel/research/
And in general if you have an LLM that performs really well on one d_i then who cares. The goal in LLM evaluation is to find a good performing LLM overall.
Finally, it feels that your Abstract and other snippets sound like an LLM wrote them.
The webpage discusses a Bayesian approach to experimentation, focusing on interpreting and extrapolating experimental results, mainly in a tech environment aiming to maximize user retention. It addresses challenges like the inference problem, the extrapolation problem, the explore-exploit problem, and a culture problem within tech companies around misuse of experiments. The author suggests providing decision-makers with benchmark statistics to help them estimate true effects of different policies, and discusses a model of experimentation dealing with observed and true effects along with the noise in experiments 1 .
Same idea here? Larger models do a better job forgetting their training data and dropping their semantic priors. Perhaps another way of thinking through this is that larger models learn new information and drop old information faster. https://arxiv.org/abs/2303.03846