No this doesn't currently use a model for valuation. It uses recursive exploration(with a stockfish like early terminition) and keeps track of the reward and regret at each node. With brute force on number of cores and compact memory layout I was able to get reasonable convergence with 6 way rings.
I do have some plans to release a follow on with some ml components ala Pluribus.
Be careful with initial impressions of metrics. We as humans have a heavy tenancy to anchor to our first judgments or impression. We see a win and assume the win is long term, with no downsides, and dependent on the new information/change.
So combine that with the Hawthorne effect and new business or health initiatives that can look great simply because participants notice change and notice the increased attention. However many human patterns have a tendency to regress to the mean.
Personally I have seen this a lot with developer tools and DevOps. A new SEV/incident/disaster happens and everyone rushes to create or onboard to a tool that would help. Around the office everyone raves about it and is sure that it would fix all issues. And the number of commits goes up, or the number of SEV's in an area decreases for a while. People were paying attention, after a while the tool starts to slow down or not be as used. It's got rough edges that weren't seen or scenarios that were supposed to be supported never get fully integrated. Eventually the patterns regress, but with more tools and more complexity.
Early game bluffs are essentially lies that you tell through the rest of the streets. In order to keep your opponents from knowing when you have premium starting hands, it's required to play some ranges, sometimes as if they were a different range. E.g., 10% of the time, I will bluff and act like I have AK, KK, AA, QQ. On the next street, I will need to continue that; otherwise, it becomes not profitable (opponents only need to wait one bet to know if I am bluffing). I have to evolve the lie as well. If cards come out that make my story more or less likely/profitable/possible, then I need to adjust the lie, not revert to the truth or the opponent's truth.
To see that LLMs aren't capable of this, I present all of the prompt jailbreaks that rely on repeated admonitions. And that makes sense if you think about the training data. There's not a lot of human writing that takes a fact and then confidently asserts the opposite as data mounts.
LLMs produce the most likely response from the input embeddings. Almost always, the easiest is that the next token is in agreement of the other tokens in the sequence. The problem in poker is that a good amount of the tokens in the sequence are masked and/or controlled by a villain who is actively trying to deceive.
Also, notice that I'm careful to say LLM's and not generalize to all attention head + MLP models. As attention with softmax and dot product is a good universal function. Instead, it's the large language model part that makes the models not great fits for poker. Human text doesn't have a latent space that's written about enough and thoroughly enough to have poker solved in there.
No the widths are not wide enough to explore. The number of possible game states can explode beyond the number of atoms in the universe pretty easily, especially if you use deep stacks with small big blinds.
For example when computing the counterfactual tree for 9 way preflop. 9 players have up to 6 different times that they can be asked to perform an action (seat 0 can bet 1, seat 1 raises min, seat 2 calls, back to seat 0 raises min, with seat 1 calling, and seat 2 raising min, etc). Each of those actions has check, fold, bet min, raise the min (starting blinds of 100 are pretty high all ready), raise one more than the min, raise two more than the min, ... raise all in (with up to a million chips).
(1,000,000.00 - 999,900.00) ^ 6 times per round ^ 9 players That's just for pre flop. Postflop, River, Turn, Showdown. Now imagine that we have to simulate which cards they have and which order they come in the streets (that greatly changes the value of the pot).
As for LLMs being great at range stats, I would point you to the latest research by UChicago. Text trained LLMs are horrible at multiplication. Try getting any of them to multiply any non-regular number by e or pi. https://computerscience.uchicago.edu/news/why-cant-powerful-...
Don't get what I'm saying wrong though. Masked attention and sequence-based context models are going to be critical to machines solving hidden information problems like this. Large Language Models trained on the web crawl and the stack with text input will not be those models though.
> Why wouldn't something like an RL environment allow them to specialize in poker playing, gaining those skills as necessary to increase score in that environment?
I think an RL environment is needed to solve poker with an ML model. I also think that like chess, you need the model to do some approximate work. General-purpose LLMs trained on text corpus are bad at math, bad at accuracy, and struggle to stay on task while exploring.
So a purpose built model with a purpose built exploring harness is likely needed. I've built the basis of an RL like environment, and the basis of learning agents in rust for poker. Next steps to come.
To play GTO currently you need to play hand ranges. (For example when looking at a hand I would think: I could have AKs-ATs, QQ-99, and she/he could have JT-98s, 99-44, so my next move will act like I have strength and they don't because the board doesn't contain any low cards). We have do this since you can't always bet 4x pot when you have aces, the opponents will always know your hand strength directly.
LLM's aren't capable of this deception. They can't be told that they have some thing, pretend like they have something else, and then revert to gound truth. Their egar nature with large context leads to them getting confused.
On top of that there's a lot of precise math. In no limit the bets are not capped, so you can bet 9.2 big blinds in a spot. That could be profitable because your opponents will call and lose (eg the players willing to pay that sometimes have hands that you can beat). However betting 9.8 big blinds might be enough to scare off the good hands. So there's a lot of probiblity math with multiplication.
Deep math with multiplication and accuracy are not the forte of llm's.
First being the hidden information, you don't know your opponents hand holdings; that is to say everyone in the game has a different information set.
The second is that there's a variable number of players in the game at any time. Heads up games are closer to solved. Mid ring games have had some decent attempts made. Full ring with 9 players is hard, and academic papers on it are sparse.
The third is the potential number of actions. For no limit games there's a lot of potential actions, as you can bet in small decimal increments of a big blind. Betting 4.4 big blinds could be correct and profitable, while betting 4.9 big blinds could be losing, so there's a lot to explore.
Text trained LLM's are likely not a good solution for optimal play, just as in chess the position changes too much, there's too much exploration, and too much accuracy needed.
CFR is still the best, however, like chess, we need a network that can help evaluate the position. Unlike chess, the hard part isn't knowing a value; it's knowing what the current game position is. For that, we need something unique.
I'm pretty convinced that this is solvable. I've been working on rs-poker for quite a while. Right now we have a whole multi-handed arena implemented, and a multi-threaded counterfactual framework (multi-threaded, with no memory fragmentation, and good cache coherency)
With BERT and some clever sequence encoding we can create a powerful agent. If anyone is interested, my email is: [email protected]
They would need to lie, which they can't currently do. To play at our current best, our approximation of optimal play involves ranges. Thinking about your hand as being any one of a number of cards. Then imagine that you have combinations of those hands, and decide what you would do. That process of exploration by imagination doesn't work with an eager LLM using huge encoded context.
I am the author/maintainer of rs-poker ( https://github.com/elliottneilclark/rs-poker ). I've been working on algorithmic poker for quite a while. This isn't the way to do it. LLMs would need to be able to do math, lie, and be random. None of which are they currently capable.
We know how to compute the best moves in poker (it's computationally challenging; the more choices and players are present, the more likely it is that most attempts only even try at heads-up).
With all that said, I do think there's a way to use attention and BERT to solve poker (when trained on non-text sequences). We need a better corpus of games and some training time on unique models. If anyone is interested, my email is elliott.neil.clark @ gmail.com
I think 'Batteries Included' would interest you, then. Like this, it's installable on AWS. It's a whole platform PaaS + AI + more built on open source. So Kubernetes is at the core, but with tons of automation and UI. Dev environments are Kubernetes in Docker (Kind-based).
While I was at FB (it wasn't Meta then), I saw what a superpower the infrastructure is there. Product engineers build things of a scale in days. While I was there, I got to be tech lead for several different teams (2x distributed dbs, 1x Dev Efficiency, 1x Ads), some of which are called out by name here.
Shout out to the HBase and ZippyDB teams! This is the first public acknowledgment that ZippyDB was converged upon.
It's also super cool to see the Developer Efficiency pushes called out. 10,000 Services pushed daily, or every commit is so impressive.
I started the company because I wanted to give the infrastructure team that FAANG companies have to smaller enterprises. Most of the best infrastructure is open source but too complicated to use or maintain. So we've built a full platform that will run on any Kubernetes cluster, giving a company a push-button infrastructure with everything built on open source. So you get Heroku with single sign-on and no CLI needed. Or you get a full RAG stack with model hosting on your EKS cluster.
Since most of the services and projects we're building on top of are open source, we wanted to give the code to the world while being sustainable in the long term as a team. I had also been a part of Cloudera, and I had seen the havoc that open core had on the long-term success of Hadoop. So, I wanted something different for licensing. We ended up with a license that somewhat resembles the FSL but fixes its major (in my opinion) problem. We don't use the competing use clause instead opting for a total install size requirement.
I'm happy to chat with anyone about this, my email is in my profile. Good Luck nd I hope it works for you.
When MS rolls up, they say we are charging for your usage of the MS database, Office, Outlook, Microsoft Windows 11, and the security promises. They are explicit that developing with and on Microsoft allows you access to the ecosystem. So the total bill is high, but part of that bill is a gateway into everyone else using Office, Outlook, Excel, Visual Studio, or SharePoint. The world runs on Excel and MS enterprise sales know that. They are negotiating a contract for one-of-a-kind software and access to the world of MS.
Redhat rolls up saying we want to charge you. They don't get to say that if you don't pay, the company will lose access to the software or the ecosystem. They don't get to say they are gatekeepers to other Linux users. Redhat can't claim to be giving the database or the development environment; everyone thinks they are free. If you stop paying Redhat, you probably can find an almost package for package compatible alternative in a rolling release (source: watched that happen multiple times CentOS, et al). So instead Redhat sells a contract for service, support, and indemnity. Those are great products and Red Hat will continue for a long time. They will just have very different staying power when contracts are renewed. They will have very different revenue growth.
It's not how I want it to be, just how I see it.
Source: Worked at MS and have friends who are former Redhat.
I agree the contract should be clear up front. Changing expectations later is a big problem. People want to give away the software for a while, using it as a loss leader to get attention while not being honest about their later need for money to fund the ongoing concern.
I was starting Batteries Included and had been writing it in Elixir. I want to give back to the community, show how to use Phoenix/Live view, and be transparent about what users are running, etc. However, I also know that if this will work long-term, I can not give it away to everyone forever. So it's better to be honest about things as early as possible.
We paid a very smart lawyer to draft the best compromise we could as early as possible.
This means we can develop in the open here: https://github.com/batteries-included/batteries-included while also giving it away long term and still being honest that this will require some long-term revenue stream. That revenue stream will come from the companies using it on larger installs.
Linux powers just about every major datacenter in there world. Every ML model was trained on Linux. However if you tried to make a company as powerful and successful as Microsoft you would fail.
Red Hat is the only company that has really made a living off of Linux. Even then, their contracts are orders of magnitudes less than the exact same customers will be paying Microsoft.
Linux is successful and remarkable and every company sees the value in having it around. So there's a shared mutual need. However, that doesn't mean that anyone can make it into more than barely scraping by as a going business.
- The anchoring principal. Once you have set the that the software is free, humans expect it to be free forever and all related things are judged off the initial impressions' price bucket. Humans will never want to pay for it later. It's judged worthless.
Open-core and closed-source addons and support models have misaligned principles. The community wants things to be easy to use and opinionated, while the OSS company wants to include as many customers as possible who need or want your help with their niche choices.
- Sustainability is awful. If you start an open source project you're either going to burn yourself and the community out, or require funding to do it as a day job. So, if you want the project not to stop early, you need money to pay for developers to make software better.
- Larger companies want something opinionated but rarely what's good for most of the community. So eventually when big tech/big industry is paying for developers to work on the project, there's a point where the large company will want their cake and the community is hostage. Do that enough times and the large company forks internally and the community fractures or withers out.
Source: I was at Cloudera while the Big Data craze took off. Then, I did open source for large tech.
I do have some plans to release a follow on with some ml components ala Pluribus.