Easily 5-10x or even more in certain special cases (when it'd take me a lot of upfront effort to get context on some problem domain). And it can do all the "P2"s that I'd realistically never get to. There was a day where I landed 7 small-to-medium-size pull requests before lunch.
There are also cases where it fails to do what I wanted, and then I just stop trying after a few iterations. But I've learned what to expect it to do well in and I am mostly calibrated now.
The biggest difference is that I can have agents working on 3-4 parallel tasks at any given point.
Not to downplay the issue you raise but I haven't noticed this.
Every iteration I make on the prompts only make the request more specified and narrow and it's always gotten me closer to my desired goal for the PR. (But I do just ditch the worse attempts at each iteration cycle)
Is it possible that reasoning models combined with the actual interaction with the real codebase makes this "prompt fragility" issue you speak of less common?
I work at OpenAI (not on Codex) and have used it successfully for multiple projects so far. Here's my flow:
- Always run more than one rollout of the same prompt -- they will turn out different
- Look through the parallel implementations, see which is best (even if it's not good enough), then figure out what changes to your prompt would have helped nudge towards the better solution.
- In addition, add new modifications to the prompt to resolve the parts that the model didn't do correctly.
- Repeat loop until the code is good enough.
If you do this and also split your work into smaller parallelizable chunks, you can find yourself spending a few hours only looping between prompt tuning and code review with massive projects implemented in a short period of time.
I've used this for "API munging" but also pretty deep Triton kernel code and it's been massive.
For those who are interested in getting involved with online math circles (as parents or potential instructors), check out https://theglobalmathcircle.org (Jeremy, the author of this post graduated from our training program)
This isn't accurate. The bottleneck in very-large-scale-training BY FAR is communication between devices. If you have a million CPUs, the communication cost will be significantly higher than a thousand A100s (perhaps in the order of 100x or even more). So this is only possible to replicate with very dense and high compute chips with extremely fast interconnect.
OP here. I help run the Global Math Circle. Happy to answer any questions.
Here's a short blurb about our approach and history:
---
The GMC approach
Our approach is to treat math education as an accessible mystery. We combine a small group of children and an experienced guide. They go deep for ~8 weeks seeking to understand the insight behind the mystery. It’s an awesome experience where once a week the children share ideas and debate with each other what the next question should be, discovering deep mathematics along the way. The guide contributes minimal input because the discussion is led by the students, helping them learn leadership skills along the way. We never turn anyone away for lack of funds. All of our prices for circles and leader training are sliding scale and we regularly accept kids who can’t pay at all.
---
Our History
The Global Math Circle (originally named “The Math Circle”), the earliest math circle in the US, was founded in Boston, MA in 1994 by Robert and Ellen Kaplan[1], authors of such books as Out of the Labyrinth[2] and The Art of the Infinite[3]. The intention, in Bob's words, "Small groups meet with their leader, each seeing and talking with all; and as always, the leader no more than posing a deep and exciting question (an accessible mystery) and the students probing it collegially together. They come up with examples, counter-examples, insights and proofs: intense fun."
In 2015, the GMC moved online and has been running online circles and leader training institutes since. A group of enthusiasts have been helping Bob and Ellen, and with Bob's passing in 2022, continue to operate the GMC on the same principles that guided the organization from its inception. In addition to our main presence across the United States, GMC actively collaborates with groups around the world to spread our approach, from Senegal[4] to Brazil[5] and from Kenya[6] to Columbia[7].
The Netherlands has a digital government sign-in ("DigID") which works very well, secured using text messaging or 2 factor auth. With it you view all of your official documents across all branches of government. It allows for delegating permissions (say between a married couple). It's really great and shows that with competence these systems work out great.
Does Horizon also solve "optimistic updates"? If so I'd love to learn more details. For comparison, Meteor keeps a local datastore that updates immediately when data is mutated and then reconciled with the real database.
We were trying to keep our API surface area small with one way to load data into components, but you're right -- we should probably /also/ add a ES6 base class as a second option, and let the people choose which they prefer.
A lot of React developers still prefer mixins -- react-router recently switched from mixins to ES6 classes and then changed their mind "until ES6 classes have better answers to replace what mixins do (like decorators).": https://github.com/rackt/react-router/blob/master/UPGRADE_GU...
At Meteor, we're building a complete open-source platform for easily building best-of-class, fast, web and mobile apps. If you've never heard about Meteor, check out the original screencast: https://www.youtube.com/watch?v=fsi0aJ9yr2o (needless to say, lots have changed since then -- most notably a security model :)
Meteor touches all parts of an application stack (client-side templating[1], live database updates[2], a websocket-based sync protocol[3], an "all you need" build tool[4] that can build mobile and web apps, an in-house "transparent reactive programming" library[5], a cloud hosting environment[6] built on Docker and Kubernetes, and the list goes on -- all with a very very strong emphasis on great developer experience).
Learn more about Meteor's subprojects here: http://meteor.com/projects. Most of these projects are current edge new technology -- we develop new systems when the existing open source ones aren't good enough (either because of how they're implemented or because we can supply a better developer experience)
Moreover, almost all of our work is open source which means working here builds your GitHub profile -- always a good thing. Remote is OK. Learn more at http://https://www.meteor.com/jobs/, or happy to answer questions at [email protected].
"If you think about it, if we tried to build all of the above on top of npm, it would be npm in name only. Even if Meteor packages were in npm, you wouldn't use the npm tool to find or install packages (you'd use a wrapper that implemented the 'release snapshots', 'curation', 'repeatability' considerations), nor could you drop them directly into existing npm applications (per 'client vs server' and 'asset bundling')."
Hi, I work at Meteor. Our office is in San Francisco, CA but remote is ok for some positions. We're hiring engineers, designers, community organizers, recruiters, and more. Interns welcome, and we'll sponsor visas for great candidates.
===
Meteor is an open-source framework for building powerful modern web apps. We have a single ambitious goal: create the best way to build application software, and make it accessible to everyone. The Meteor community is growing fast, with new production apps launching every week.
We're backed by top investors, including Andreessen Horowitz, Matrix, Y Combinator, Ron Conway, Maynard Webb, and others.
We've got a healthy attitude towards work/life balance, an excellent team, and a great office in SOMA (though some of the positions support full time remote work).
HTMLBars is an excellent project and indeed does share many design principles with Blaze. They also differ in a few ways. For example, HTMLBars plans to never support full inline expressions in templates, whereas that is an explicit future goal for Blaze. So even though parts of HTMLBars could have been used in Blaze, it wouldn't give us the flexibility needed to take Blaze in the direction we think is best for our users.
React and Blaze both compile templates or components into an intermediate representation for their HTML structure (React's JSDOM; Blaze's HTMLJS). A lot of the difference in how they decide what DOM changes are necessary.
React re-renders components when data changes and diffs the resulting JSDOM. The diff algorithm is fast and has simple hooks to make it faster by letting you compare state and component properties.
Blaze doesn't diff the resulting DOM structure. Instead, Blaze diffs the data the components depend on. That data comes from reactive functions built on top of Deps, our dependency tracking system. When the component first renders, dependencies are tracked between data sources and rendered DOM elements (dependencies are automatically cleaned up when the component is taken off the page). Therefore when data changes, Blaze can directly update the relevant elements.
There is some overhead in Deps, our dependency tracking system, but Deps has been intentionally designed for supporting this efficiently (eg batching updates in a "flush" stage).
As for performance, our benchmarks have generally found that React outperforms Blaze when many elements change and Blaze outperforms React when few elements change (which is more common for typical operations on many web apps). We also have plans to improve performance on initial rendering.
There are some other difference worth noting: While a later version of Blaze will support easy APIs for re-useable components, React already has one. And the React component model is well-thought out, complete and in production use whereas the Blaze component model at the moment is only the implementation behind templates.