How I built a poker-bot(codingthewheel.com)
codingthewheel.com
How I built a poker-bot
http://www.codingthewheel.com/archives/how-i-built-a-working-poker-bot
8 comments
I don't think I agree with you on no-limit vs. limit. There's a pretty simple strategy you can use to be a marginal winner at low stakes no-limit (or especially no-limit sit and goes). It' called short stacking, and essentially involves buying in for the minimum and playing mainly all-in or fold (with some preflop calling).
You have to leave the table as soon as you get past 1.5x the minimum buy-in though. It was mainly done on Party Poker, which was the only site dumb enough to let you cash out and then return to the table for the min again. Any other site either made you bring back what you left with or wait an hour.
On a site with enough tables in action, though, even that would not be a limitation. A bot could do fairly well at it.
Also I can tell you from first hand experience that breaking even at low stakes NL sit and goes can be done with an AI consisting of nothing but if/then statments. I did this with WinHoldem back in the days before it was banned from everywhere. I wasn't able to replicate that success with limit. The main problem with WinHoldem was that it didn't join or leave tables on its own (also, it would often bug out when the tournament got heads up), which meant that the time spent managing it made it less profitable than just playing myself.
That's why we tried to build a bot ourselves. Maybe I'll blog about that some time.
You have to leave the table as soon as you get past 1.5x the minimum buy-in though. It was mainly done on Party Poker, which was the only site dumb enough to let you cash out and then return to the table for the min again. Any other site either made you bring back what you left with or wait an hour.
On a site with enough tables in action, though, even that would not be a limitation. A bot could do fairly well at it.
Also I can tell you from first hand experience that breaking even at low stakes NL sit and goes can be done with an AI consisting of nothing but if/then statments. I did this with WinHoldem back in the days before it was banned from everywhere. I wasn't able to replicate that success with limit. The main problem with WinHoldem was that it didn't join or leave tables on its own (also, it would often bug out when the tournament got heads up), which meant that the time spent managing it made it less profitable than just playing myself.
That's why we tried to build a bot ourselves. Maybe I'll blog about that some time.
Yes, I looked up a few poker bots yesterday and those sites said that the bots are generally more profitable on no-limit with experienced (predictable) players. I can dig up the links again if anyone wants.
The reason limit is easier is just that there is less variables to account for much like it's easier to write a bot to play poker with fewer cards.
While I will agree sit and gos are all strategy that strategy is dependent on putting your opponent on reasonable ranges which a bot like this won't do.
Bots and winning bots are possible(stick most casual people up against the UofA bots for example), but I find it hard to believe this one is one of them.
While I will agree sit and gos are all strategy that strategy is dependent on putting your opponent on reasonable ranges which a bot like this won't do.
Bots and winning bots are possible(stick most casual people up against the UofA bots for example), but I find it hard to believe this one is one of them.
That's wrong too. If you think there are fewer variables in limit hold'em to figure when making your decision, you don't understand the game.
If you went to a no-limit tournament and just pushed all of your chips into the middle at every single opportunity, it's been shown that you're only a slight loser. At the WSOP main event, you'd be a better player than at least 1/3 of the field.
This is not true if you were to play in a limit tournament and raise every time. You'd probably be the worst player in the field.
The fact that it's less complicated is why no-limit is the tournament game of choice, but high stakes live poker is generally limit.
If you went to a no-limit tournament and just pushed all of your chips into the middle at every single opportunity, it's been shown that you're only a slight loser. At the WSOP main event, you'd be a better player than at least 1/3 of the field.
This is not true if you were to play in a limit tournament and raise every time. You'd probably be the worst player in the field.
The fact that it's less complicated is why no-limit is the tournament game of choice, but high stakes live poker is generally limit.
Nope. NL is a much more complex game than limit. This is covered fairly well in the U-of-A literature. Also some of the two plus two NL books describe why NL is more complex.
Just because there is a simple strategy (shortstacking push/fold) that can be applied successfully against weak players, that doesn't mean that NL is a simpler game.
When you are trying to model the game there is a lot more to consider in NL. In Limit at each point in a hand and at each future point in a hand there are only limited actions that your opponent can take - raise, call, check, fold. In NL they can do any of these actions but also for any amount - push, raise pot, raise 1/2 pot etc. So the problem space for NL is much larger than for Limit. Bluffing is a much bigger part of NL and bluffing is hard to model.
Playing profitable limit poker is a fairly mechanical, mathematical process. You can't force people out of a hand with a big bet. It basically boils down to betting when you're probably ahead, folding when you're probably behind and drawing when the pot odds are favorable.
The U of A bots can play limit reasonably successfully against decent players. A decent player can easily beat their bots at NL.
Building a bot that can profit at the microstakes tables might be easier for NL than limit because you can employ a push/fold short stacking strategy like you describe. But building a bot that can survive against better players would be much harder for NL than Limit.
Just because there is a simple strategy (shortstacking push/fold) that can be applied successfully against weak players, that doesn't mean that NL is a simpler game.
When you are trying to model the game there is a lot more to consider in NL. In Limit at each point in a hand and at each future point in a hand there are only limited actions that your opponent can take - raise, call, check, fold. In NL they can do any of these actions but also for any amount - push, raise pot, raise 1/2 pot etc. So the problem space for NL is much larger than for Limit. Bluffing is a much bigger part of NL and bluffing is hard to model.
Playing profitable limit poker is a fairly mechanical, mathematical process. You can't force people out of a hand with a big bet. It basically boils down to betting when you're probably ahead, folding when you're probably behind and drawing when the pot odds are favorable.
The U of A bots can play limit reasonably successfully against decent players. A decent player can easily beat their bots at NL.
Building a bot that can profit at the microstakes tables might be easier for NL than limit because you can employ a push/fold short stacking strategy like you describe. But building a bot that can survive against better players would be much harder for NL than Limit.
There are fewer variables when factoring in your opponent modeling. I'm sure being who you are you are very familiar with the UofA research and I'm pretty sure it backs up this statement.
Pushing every hand isn't right, it's pushing every hand with certain cards that Sklansky taught which admittedly is only a slight loser but a loser none the less.
I'm not saying limit holdem isn't a complex deep game, all I'm saying is that when it comes to bots no-limit bots are a lot harder to get right(in opponent modelling primarily which is a big part of bots these days).
I think no-limit is the tournament game of choice because it's seen by the general public as being the more exciting of the two(not that I agree but thats another discussion).
Pushing every hand isn't right, it's pushing every hand with certain cards that Sklansky taught which admittedly is only a slight loser but a loser none the less.
I'm not saying limit holdem isn't a complex deep game, all I'm saying is that when it comes to bots no-limit bots are a lot harder to get right(in opponent modelling primarily which is a big part of bots these days).
I think no-limit is the tournament game of choice because it's seen by the general public as being the more exciting of the two(not that I agree but thats another discussion).
I still disagree and think that limit is, in general, a much more complex game. But even if it's not, I think you're thinking of building a bot that's a world class player. That may be harder at no-limit, I'm not really sure.
But if you're building from the standpoint of breaking even at low stakes, it's far easier to do for no-limit, especially sit and goes. It's literally a couple hundred lines of if/then statements.
But if you're building from the standpoint of breaking even at low stakes, it's far easier to do for no-limit, especially sit and goes. It's literally a couple hundred lines of if/then statements.
It's interesting, but to be honest, if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures. The potential winnings are much higher, it's perfectly legal, and due to the depth of the markets such programs are capable of making much more money than poker bots. It's also doable, as funds like Renaissance Technologies (http://en.wikipedia.org/wiki/Renaissance_Technologies) and D. E. Shaw have demonstrated.
if you're smart enough to make a successful poker bot, you're probably smart enough to write a program to trade stocks or other liquid instruments like government bonds or commodity futures
I'm not so sure about that. I think the main reason is: there's a lot more "stupid money" at the poker table than there is in the market.
The bot as mentioned doesn't seem to do anything particularly fancy -- it just looks at the cards on the table and figures out your odds of winning based on that. This is information which is easily extracted mathematically, but which most players don't have direct access to. Good players will have a pretty good idea of these probabilities, whereas lousy players won't. If you can find yourself on a table full of lousy players, you're at a huge advantage.
In the market, on the other hand, there's only one poker table, and all the best players in the world are already sitting at it. Any time statistical arbitrage opportunities arise in prices, you can bet that D.E. Shaw and Rennaisance Technologies have already noticed it and moved a few billion dollars that way. If you want to win, you don't just have to be smarter than six guys in their basements doing calculations in their heads, you have to be smarter than thousands of overpaid quants with supercomputers.
I'm not so sure about that. I think the main reason is: there's a lot more "stupid money" at the poker table than there is in the market.
The bot as mentioned doesn't seem to do anything particularly fancy -- it just looks at the cards on the table and figures out your odds of winning based on that. This is information which is easily extracted mathematically, but which most players don't have direct access to. Good players will have a pretty good idea of these probabilities, whereas lousy players won't. If you can find yourself on a table full of lousy players, you're at a huge advantage.
In the market, on the other hand, there's only one poker table, and all the best players in the world are already sitting at it. Any time statistical arbitrage opportunities arise in prices, you can bet that D.E. Shaw and Rennaisance Technologies have already noticed it and moved a few billion dollars that way. If you want to win, you don't just have to be smarter than six guys in their basements doing calculations in their heads, you have to be smarter than thousands of overpaid quants with supercomputers.
Fair enough. You're probably right. I don't play online poker (and am a poor casual player) so I don't know what the level of skill is on those websites. I was just making an assumption based on how difficult I imagined it would be for a computer to play very good poker.
A few years ago, I wrote a program that would screen-scrape to read all cards showing and then do a Monte Carlo simulation to randomly complete the hand and compute what % of times my cards would win (along with pot odds). It took a second or so to do the simulation, so I could use it to make decisions in real time. It was a fun project, but I soon stopped using it. Your odds against random cards are not at all your actual odds, because most players fold weaker cards. What matters most is who you're playing against.
Anyway, the article, although it describes well the techniques for programmatically interacting with a poker client, is majorly misleading because this is not the hard part of making a bot. The hard part is figuring out whether, and how much, to bet. He alludes to this as "the million dollar question", but then omits to even mention it under "What skills will I need to write a bot?" Perhaps he'll address this in later articles?
If you look at the last screen shot, it's clear that he built a UI for an end-user to specify the actual logic for poker play. ("Call if it has not been raised, I'm later than 7th", etc.) His "bot" doesn't actually make decisions. It collects rules from an end-user, calls those to decide what to do, and passes the result on to the poker client. In other words, the hard part of the job - writing a program to play poker - has been passed to somebody else. Maybe that somebody else is the author, creating the algorithm through his UI... but then why not just code it?
There are serious researchers who have made credible bots (he links to them), bots which would (and, for all we know, do) make money on poker sites. But when I looked at it, the literature was not nearly detailed enough to code up their algorithms. (Also, they were working with simplified versions of poker, since full no-limit holdem is a hard nut to crack). What they didn't use were the "extensible rules-based systems" which this author claims "are a lot more powerful than you think". Count me skeptical. Good players eat such systems for snacks. Maybe they make money in low-stakes games against bad players, but I'd like to see evidence.
In short: never mind the dll injection, show us your bankroll over 100,000 hands!
Anyway, the article, although it describes well the techniques for programmatically interacting with a poker client, is majorly misleading because this is not the hard part of making a bot. The hard part is figuring out whether, and how much, to bet. He alludes to this as "the million dollar question", but then omits to even mention it under "What skills will I need to write a bot?" Perhaps he'll address this in later articles?
If you look at the last screen shot, it's clear that he built a UI for an end-user to specify the actual logic for poker play. ("Call if it has not been raised, I'm later than 7th", etc.) His "bot" doesn't actually make decisions. It collects rules from an end-user, calls those to decide what to do, and passes the result on to the poker client. In other words, the hard part of the job - writing a program to play poker - has been passed to somebody else. Maybe that somebody else is the author, creating the algorithm through his UI... but then why not just code it?
There are serious researchers who have made credible bots (he links to them), bots which would (and, for all we know, do) make money on poker sites. But when I looked at it, the literature was not nearly detailed enough to code up their algorithms. (Also, they were working with simplified versions of poker, since full no-limit holdem is a hard nut to crack). What they didn't use were the "extensible rules-based systems" which this author claims "are a lot more powerful than you think". Count me skeptical. Good players eat such systems for snacks. Maybe they make money in low-stakes games against bad players, but I'd like to see evidence.
In short: never mind the dll injection, show us your bankroll over 100,000 hands!
"Count me skeptical. Good players eat such systems for snacks."
You're thinking about it wrong. As he mentions, your bot doesn't play against good players. It plays .5/1 or 1/2 nl and makes a shit ton of money if it just breaks even. There aren't good players at those levels, because anyone who was good would win and quickly move up. You only need all of your if/then statements to add up to being better than the worst 5% of poker players, and that's not that hard to do. I know only because I've done it.
Thats why extensible rules based systems are better than you think. They can get you to that with a good amount of work. Especially in short stack nl games. They're clearly not going to win at a $5/$10 nl table or a $215 SNG. But they don't need to to rake in millions per year.
You're thinking about it wrong. As he mentions, your bot doesn't play against good players. It plays .5/1 or 1/2 nl and makes a shit ton of money if it just breaks even. There aren't good players at those levels, because anyone who was good would win and quickly move up. You only need all of your if/then statements to add up to being better than the worst 5% of poker players, and that's not that hard to do. I know only because I've done it.
Thats why extensible rules based systems are better than you think. They can get you to that with a good amount of work. Especially in short stack nl games. They're clearly not going to win at a $5/$10 nl table or a $215 SNG. But they don't need to to rake in millions per year.
Also I should mention, that with the exception of heads up, good players (at least by my definition of the word good) eat even the UofA's AI for snacks.
He mentioned it being a series of articles, maybe the next one will cover logic? I dunno.
I was involved in a fairly elaborate pokerbot effort. Believe me, the mechanics of playing the game are the easy part. The hard part is writing the poker logic. Check this out if you want an intro to the "basic" concepts:
Lokibot: http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html
The bot described in that paper lost about 8c a hand at the $1 tables. After extensive modifications, we got it to the point where it was losing about 2c a hand. This when Bush made poker illegal in the US, and we killed the project.
Lokibot: http://www.cs.ualberta.ca/~jonathan/Grad/papp/thesis.html
The bot described in that paper lost about 8c a hand at the $1 tables. After extensive modifications, we got it to the point where it was losing about 2c a hand. This when Bush made poker illegal in the US, and we killed the project.
> none of the online poker sites have really stepped up to the plate and either a) made bots legal (similar to the way that they're legal on Internet Chess Club) or b) put effective prevention measures in place.
That's a tough choice for a casino. Allow automation and you create a ghetto which discourages casual users. Disallow automation and you encourage an escalating "arms race".
Regardless, the techniques discussed could be detected using anti-virus techniques. I'm surprised that native clients don't perform integrity checks or is this the state of play in the "arms race"?
It would be fairly simple to disable integrity checks using methods to circumvent software DRM. Then it would be a matter of implementing distributed state to determine and ignore circumvented clients.
That's a tough choice for a casino. Allow automation and you create a ghetto which discourages casual users. Disallow automation and you encourage an escalating "arms race".
Regardless, the techniques discussed could be detected using anti-virus techniques. I'm surprised that native clients don't perform integrity checks or is this the state of play in the "arms race"?
It would be fairly simple to disable integrity checks using methods to circumvent software DRM. Then it would be a matter of implementing distributed state to determine and ignore circumvented clients.
Interestingly, the casino's financial interest, since it makes its money only on the rake, is to have as many players or bots as possible. In fact, from their perspective, an infinity of bots battling each other should be just fine--bots scale better than humans. From Ultimate bet's perspective, bots are great:
It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until UltimateBet is rich.
It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until UltimateBet is rich.
I considered this possibility but a casino which is too open to bots could become marginalised. Real players would be scalped too easily and bot writers would be playing a zero-sum game. Both factors would reduce casino turnover and profit.
So, it is in a casino's interest to leave the whole issue unresolved and with a level of tolerance which is in-step with competitors.
So, it is in a casino's interest to leave the whole issue unresolved and with a level of tolerance which is in-step with competitors.
I agree to an extent, however, I would be most happy owning a bot battleground, personally. It would dispense with all that time wasted on "thinking" and reduce the speed to a processor/internet lag dependent game. Bots are no more zero sum than people, they are just faster. I love the idea of a money making machine that scales with processor cores/mhz.
Can't wait to read the rest of the series. Promises to have a bit of everything: OCR, screen-scraping, rules engine, WinAPI hooks, DLL injection, and that's just part one!
Really Awesome, now I will apply to WoW.
Convincing others that your goldfarming bot is actually human seems like a highly challenging variant of the Turing test.
Not sure it's so hard, first you introduce some randomness and inefficiencies in the gathering process so that it's more credible that it's a fallible human that's doing it.
And as a second crucial step, you hook groups of bots into a "monitoring computer", at which an actual human sits and responds "No, I'm not a bot lol." when the supposedly fatal question arrives from an admin. You can even outsource this to the third world.
And as a second crucial step, you hook groups of bots into a "monitoring computer", at which an actual human sits and responds "No, I'm not a bot lol." when the supposedly fatal question arrives from an admin. You can even outsource this to the third world.
Mechanical Turk!
You've never played WoW, have you? "Randomness" is a dead giveaway of a bot. Other players will grief you, report you, and you will be banned quickly.
Just the computer vision aspect of it alone would be amazingly hard.
Efficient goldfarming (e.g., with a frost mage that can take on 15-30 monsters at once) involves carefully coordinating a complicated series of actions where timing is critical and reacting quickly and intelligently when things go wrong is the difference between living and dying. It involves making complicated judgments about relative positioning, reasoning about surroundings, cause and effect, and other things that require human-level intelligence.
If you can make a bot that can seek out and kill monsters in a non-trivial manner without dying all the time there's no reason you couldn't program a robot to replace a wide variety of menial real-world jobs. The consequences would be tremendous and using it to make $20/day off the Internet would be retarded because you could license the technology for billions.
Just the computer vision aspect of it alone would be amazingly hard.
Efficient goldfarming (e.g., with a frost mage that can take on 15-30 monsters at once) involves carefully coordinating a complicated series of actions where timing is critical and reacting quickly and intelligently when things go wrong is the difference between living and dying. It involves making complicated judgments about relative positioning, reasoning about surroundings, cause and effect, and other things that require human-level intelligence.
If you can make a bot that can seek out and kill monsters in a non-trivial manner without dying all the time there's no reason you couldn't program a robot to replace a wide variety of menial real-world jobs. The consequences would be tremendous and using it to make $20/day off the Internet would be retarded because you could license the technology for billions.
Bots don't require computer vision or any kind of sensory input; they can extract the native data they need straight out of the application's memory space. Effectively gaming bots have "perfect information". So the bots actually have more timely and accurate information than humans do, they can react quicker than humans do, and they can react as intelligently as your programming skill allows.
Check out http://www.easyuo.com. That application allows you to easily automate Ultima Online. Whilst deciphering WoW's application memory would probably be harder, an equivalent program is certainly possible (it probably already partially exists in various forms), and intelligent bots are a simple step after that.
Unfortunately, once the memory accessing has been abstracted away, programming a competent WoW bot is not a difficult task. The WoW bot doesn't have to be as awesome as the best human players, nor need it be ultra-efficient. What it lacks in ability it makes up in working without fatigue indefinitely.
Check out http://www.easyuo.com. That application allows you to easily automate Ultima Online. Whilst deciphering WoW's application memory would probably be harder, an equivalent program is certainly possible (it probably already partially exists in various forms), and intelligent bots are a simple step after that.
Unfortunately, once the memory accessing has been abstracted away, programming a competent WoW bot is not a difficult task. The WoW bot doesn't have to be as awesome as the best human players, nor need it be ultra-efficient. What it lacks in ability it makes up in working without fatigue indefinitely.
Bots don't require computer vision or any kind of sensory input; they can extract the native data they need straight out of the application's memory space.
I'm pretty sure WoW keeps track of all applications running on your computer. I think the way people do it is by using a hub and packet sniffers on a separate computer, although I imagine WoW traffic could easily be encrypted... so shrug.
I'm pretty sure WoW keeps track of all applications running on your computer. I think the way people do it is by using a hub and packet sniffers on a separate computer, although I imagine WoW traffic could easily be encrypted... so shrug.
Or you just use a virtual machine - of which WoW is unaware.
A bit like the matrix.
A bit like the matrix.
All this hypothetical talk is very nice, but there actually is a wowbot that Blizzard has been unable to effectively use countermeasures against, called MMOGlider. They've decided to simply sue after repeated failure to solve it technically.
I'm unmasked, I'm utterly clueless about WoW.
And if randomness is a dead giveaway of a bot, then don't implement it?... I was suggesting to add it but if it actually ampers your goals then make a fully deterministic bot...
And if randomness is a dead giveaway of a bot, then don't implement it?... I was suggesting to add it but if it actually ampers your goals then make a fully deterministic bot...
The point of bots is that they don't have to be efficient, just marginally effective and able to run with limited or no supervision. The solution to inefficient bots is easy: create more instances.
but therein lies the rub: it's easy for the administrators of the game to realize that no real person in the right mind would sit in the same location for 3 days and go after the same creatures over and over.
Make the bot change locations and target creatures from time to time?... The point is that whatever heuristics are discovered to pinpoint bots won't be effective very long because of arms-race dynamics.
You are taking this a bit too seriously, I think.
Although I do agree that with that technology you are better off licensing it.
Although I do agree that with that technology you are better off licensing it.
Not if you use a hunter class character. They have the reputation of being a bit slow :-)
awesome article.
I wonder mattmaroon's thoughts on this.
I wonder mattmaroon's thoughts on this.
If your bot can play 14 limit tables at a time with even the slightest % edge, you're going to make money. However, if you added more advanced player profiling algorithms into your bot, you're edge would likely increase and allow you to set it up at higher stakes.