i think that’s exactly the core of the question. There is meaningful decorrelation across models in some domains, like language for instance. but i agree that it’s much less obvious for reasoning.
I don’t think more models is necessarily better, and one of our research questions is also: when does adding another reasoning path actually add useful information, and when is a single strong path better than an ensemble?
exactly, if you ensemble heavily small uncorrelated models (while each being expert on its task) you can get really interesting resutls.
on agentic and coding what's make the problem even deeper is the granularity. how and when to use each model and at which layer of abstraction (session, goal, task, turn/tool calling). this is also something we are working on actively!
there are and this what you optimize for. ensemble learning has a long literature on this. you want models that have the most diverse pool of capabilities so they complement each other. in verifiable tasks or classification this is straightfroward but a bit tricker in open ended text gen or CoT. this is one of our core research question
thanks to everyone for taking the time to try Echo and share feedback, this is precisely why i wanted to launch early.
i am going to try to address a couple of topics that came up often:
- i'll keep publishing stronger evals, including more difficult coding and agentic benchmarks, to map out more precisely the differences with sota
- the public eval dashboard will keep expanding and be updated (very open to more benchmark suggestions as well!)
- some people found issues in the eval dashboard ui and the sign up flow, should be now all fixed in prod
some important precisions as well:
- NO credit card is required to try Echo
- each acount includes 10$ of free credits to try on both the API and the chat
on the approach itself: the idea i'm exploring is more broader than model routing, i'm looking at how to allocate inference efficiently across open-weight models, deciding not only which models to use, but also how much computation a request deserves and how intermediate work should be combined.
ensembling by itself is not new. since random forests and probably even before in statistics/classic ml we knew that bringing multiple models together can outperform individual ones. the interesting problem for Echo is how to model and leverage this without paying the full ensemble cost at each request.
while there are conceptual similarities with systems like Fusion or Fugu, the architecture and optimization objective are different.
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
Thanks for checking the individual rows. HumanEval+ is one small code slice, not the whole basis for the launch claim. The public evaluator currently contains 907 rows across seven benchmark families, and matched SWE-bench Verified and BigCodeBench runs are the next code evidence being added.
You also found a real UI bug: the inspector should show both stored answers and currently does not in some rows. We are fixing that.
On the row you reran: the page records a frozen matched run. It does not claim that Fable is incapable of solving that prompt on another run. We are adding repeated matched trials and making run count and variance explicit. Your rerun is exactly the kind of external check the row-level page is intended to make possible.
The broader result remains: Echo is competitive with Fable across the evaluated task mix at materially lower measured inference cost. We are filling in the harder agentic-code evidence now rather than asking anyone to infer it from HumanEval+.
You are right that the privacy wording was too broad. We are fixing it now so it states explicitly that Echo does not use customer prompts, files, chats, or outputs to train or fine-tune models. We are updating the matching Terms language at the same time.
Echo also starts users with free credit and does not require a credit card to try it. The current signup flow did not make that clear enough, so we are fixing that presentation too.
I don’t think more models is necessarily better, and one of our research questions is also: when does adding another reasoning path actually add useful information, and when is a single strong path better than an ensemble?