I've spent tons of time evaluating o1-preview on SWEBench-Verified.
For one, I speculate OpenAI is using a very basic agent harness to get the results they've published on SWEBench. I believe there is a fair amount of headroom to improve results above what they published, using the same models.
For two, some of the instances, even in SWEBench-Verified, require a bit of "going above and beyond" to get right. One example is an instance where the user states that a TypeError isn't properly handled. The developer who fixed it handled the TypeError but also handled a ValueError, and the golden test checks for both. I don't know how many instances fall in this category, but I suspect its more than on a simpler benchmark like MATH.
I like to jokingly call founder mode: "fine-grained multi-level oversight". Others might call it the derogatory "micromanagement".
That doesn't mean I control every decision, or that I don't give people space to be creative. What it means is: for whatever is most important for the business, I get involved with the details. The goal is that when I move out of that area, the team I worked with is able to operate closer to founder mode than when I started.
The issue is that vision fundamentally can't be communicated by telephone, or all at once. You're trying to get to a point on the map that most people can't see. The path to it is the integration of all of the tiny decisions everyone makes along the way.
If you only course correct from the highest level you'll never get there.
You can do some amount of iterative and visual data transformation in Weave now. But maybe like 10% of what you can do with Pandas.
Pandas is awesome and lingua franca in data science, unseating it would be an incredible feat if anyone ever pulls that off.
The core data structure in Weave, ArrowWeaveList, has some advantages over a dataframe. It can represent arbitrarily nested data, store references to other object types that can't be stored in arrow (like images), and be visualized in the Weave UI.
For now, we just want to make it really easy to go back and forth.
Hi! I'm Shawn, founder/CTO at Weights & Biases. We've been working on Weave for a couple years now, and it powers core parts of wandb.ai.
It's a UI toolkit built for programmers that can be reprogrammed from the UI itself. You might call it a "yes-code UI" :)
I'm very happy that we were able to release Weave under Apache2 yesterday, and there is a lot of cool new technology in here that we haven't had a chance to describe yet! I'll just try to give a quick tech summary for now.
There are three core components in Weave: Types, Ops, and Panels.
Panels are UI elements that register to render a given type:
@weave.type()
class BertvizHeadView(weave.Panel):
id = "BertvizHeadView"
input_node: weave.Node[huggingface.ModelOutputAttention]
@weave.op()
def render(self) -> weave.panels.PanelHtml:
html = bertviz_head_view(self.input_node)
return weave.panels.PanelHtml(html)
And they can render other panels.
Panels can expose editable expressions (compositions of Weave ops) to the user, to give them control over what data transforms happen.
We call it Weave because it "Weaves a compute graph through the UI".
Weave includes a vectorizing DAG execution engine built on apache arrow, serialization and data versioning capabilities, and batteries Panels like Table and Plot.
We'll be writing and sharing a lot more now that the release is done. In the meantime, happy to answer any questions!
Wow! What a joy to click on the comments and find a positive comment at the top. Thanks for writing this and thanks HN upvoters for expressing your gratitude.
It’s also surprising at first that infinite series can add up to a specific, finite number. If I go halfway to X, halfway again, and so on, where do I end up?
All next realities are weighted based on their probability given the current one. The probabilities are based on our knowledge of the equations of physics. Some realities are more likely than others and adding this all up results in an average outcome. At the smallest scales the average outcome is not actually what happens, but time is a great equalizer.
Finally! All you ungrateful users who expect your search engines to pore through billions of pages in milliseconds without complaining are going to have to show some humility for once.
The response here makes me think most commenters don’t have experience with this particular footgun.
To clarify: Python can gc your task before it starts, or during its execution if you don’t hold a ref to it yourself.
I can’t think of any scenario in which you’d want that behavior, and it is very surprising as a user.
Python should hold a ref to tasks until they are complete to prevent this. This also “feels right”, in that if I submit a task to a loop, I’d think the loop would have a ref to the task!
It’d be interesting to dig up the internal dev discussions to see why the “fix this” camp hasn’t won.
"are any of my recent blog drafts unpublished and nearly ready to go?"