How I made $500k with machine learning and high frequency trading(jspauld.com)
jspauld.com
How I made $500k with machine learning and high frequency trading
http://jspauld.com/post/35126549635/how-i-made-500k-with-machine-learning-and-hft
318 comments
This. I love crazy projects and Show HN's until the cows come home, but this one is dangerous that I must repeat the warning to others.
I cannot emphasize how important it is to understand that people who trade using price action (http://en.wikipedia.org/wiki/Price_action_trading ) are just speculating based on where they expect the price to move. It's no different than people who play Texas Hold'em online and speculate what cards others have based on betting patterns. If you get good at spotting the patterns (like this guy did) you can go on a winning streak, but when the game changes (as it did for this individual after 2009) then you either go home or go broke.
This guy found one edge in 2009. It won him 500k. Fantastic. More than any edge ever won me. But, the market has changed so much since then, with HFT becoming so prevalent (http://www.theverge.com/2012/8/7/3226187/high-frequency-trad... ) that please be careful before you follow this course. His code is unlikely to be worth much today unmodified, and when you modify it you'll realize, as I have, that when the other players have access to the order books and can jump the line you have no chance in the game in 2012.
One last nit: Please, please post recent data when you talk about projects like this. 2009-2010 is 3 years ago. Since then there was significant turmoil in the US, Asia, and the EU. How are these returns relevant for today?
I cannot emphasize how important it is to understand that people who trade using price action (http://en.wikipedia.org/wiki/Price_action_trading ) are just speculating based on where they expect the price to move. It's no different than people who play Texas Hold'em online and speculate what cards others have based on betting patterns. If you get good at spotting the patterns (like this guy did) you can go on a winning streak, but when the game changes (as it did for this individual after 2009) then you either go home or go broke.
This guy found one edge in 2009. It won him 500k. Fantastic. More than any edge ever won me. But, the market has changed so much since then, with HFT becoming so prevalent (http://www.theverge.com/2012/8/7/3226187/high-frequency-trad... ) that please be careful before you follow this course. His code is unlikely to be worth much today unmodified, and when you modify it you'll realize, as I have, that when the other players have access to the order books and can jump the line you have no chance in the game in 2012.
One last nit: Please, please post recent data when you talk about projects like this. 2009-2010 is 3 years ago. Since then there was significant turmoil in the US, Asia, and the EU. How are these returns relevant for today?
This is a bit off-topic, but it's actually quite feasible to get a real edge in Hold'em, and it's not just about spotting other people's patterns.
To start with, there's simple probability: knowing the odds of making you hand vs. the payoff in the pot, or the chance of winning with various starting hands. This is pretty basic but a lot of low-stakes players screw it up. If you get it right, their mistakes are your gain.
At a more advanced level, game theory comes into play, using bluffs and so on. The game is complex enough that it's not completely solved, and it's an active area of research. The University of Alberta is doing a lot of working developing poker bots using game theory. By playing a good strategy, you can prevent other players from exploiting your patterns.
Only after you've got a good grasp on all that should you really think much about exploiting a particular player's weaknesses. The Alberta guys are doing work on that part too. Exploitative play can improve your profit but also makes you more vulnerable.
For a good overview of this stuff, the book Mathematics of Poker by Ankenman and Chen is a good place to start.
I agree though that HFT is awfully competitive these days. If I had to choose between the two I'd play poker.
To start with, there's simple probability: knowing the odds of making you hand vs. the payoff in the pot, or the chance of winning with various starting hands. This is pretty basic but a lot of low-stakes players screw it up. If you get it right, their mistakes are your gain.
At a more advanced level, game theory comes into play, using bluffs and so on. The game is complex enough that it's not completely solved, and it's an active area of research. The University of Alberta is doing a lot of working developing poker bots using game theory. By playing a good strategy, you can prevent other players from exploiting your patterns.
Only after you've got a good grasp on all that should you really think much about exploiting a particular player's weaknesses. The Alberta guys are doing work on that part too. Exploitative play can improve your profit but also makes you more vulnerable.
For a good overview of this stuff, the book Mathematics of Poker by Ankenman and Chen is a good place to start.
I agree though that HFT is awfully competitive these days. If I had to choose between the two I'd play poker.
I have done HFT, played heads up semi professionally, and for my bachelors thesis wrote a paper on a PLO playing bot. I studied Alberta's research and it is phenomenal. The parallels that emerge between HFT and a pokerbot is essentially that the architectures of both systems are kind of same and the details are kind of orthogonal. The edge in Hold'em is kind of gone. The 1/2 games right now are as tough as the 25/50 games 5 years ago. PLO is still pretty exploitable though. The same is true with HFT. Most strategies in HFT no longer work but there are definitely ones that still give you a lot of edge-you just have to think harder : )(just like three betting preflop and cbetting the flop doesnt work anymroe).
> just like three betting preflop and cbetting the flop doesnt work anymore.
So true, and frustrating. But, the legalization of online play could bring back another boom at least for a couple of years.
So true, and frustrating. But, the legalization of online play could bring back another boom at least for a couple of years.
Thanks for posting this so I didn't have too.
Poker is "a game of skill with an element of luck" and should not be confused with say, gambling on roulette or the outcome of a coin toss.
Poker is "a game of skill with an element of luck" and should not be confused with say, gambling on roulette or the outcome of a coin toss.
The same could be said of HFT. Traders are "skilled" at having nanosecond access to the orderbook, having their servers co-located in the same rack space as the exchange itself. They are also "skilled" at recognizing a price movement nanoseconds before it actually happens and getting their order in just in time.
But the HFT game changes and you have to keep up. Just as a poker player from 10 years ago would not survive in the game today without adapting his style of play.
But the HFT game changes and you have to keep up. Just as a poker player from 10 years ago would not survive in the game today without adapting his style of play.
Hey, I didn't actually intend this to be a course. I do not make any money in the market right now so am certainly not qualified to teach a course on it. And of course, if I was making money in the market I wouldn't have posted this at all. So please everyone remember that. These comments have made me realize it's probably for the best if I do not post the source code. Basically you are competing against armies of PHDs who are buying buildings next to the exchange so they can get their executions slightly faster. It is indeed surprising to me that I was able to make money in the first place. But I do know for a fact that I did make money and I also know that I was not at risk of losing a bunch of money. As mentioned the most I lost in one day was $2000. That's all I was risking.
FWIW, I couldn't see your pnl chart.
I develop algorithmic strategies for a living, and my first reaction to reading your post was skepticism. I'm skeptical for two reasons. (1) because your methods are so unconventional in an industry where convention rules, and (2) because of the time frame of your success, which happened to be one of the more impressive market recoveries in history.
I can't tell you how many people I've worked with who fail to isolate the source of their pnl (myself included at times). This is key. It's important to benchmark your strategy against other stupid ones that you know don't have edge. When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Doing this will truly help isolate whether or not luck is involved. When you say that the number and size of your trades justifies the strategy's validity, that's just wrong. You could do 1000 trades in a day: buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
I make all traders benchmark their work against a series of other strategies that I know have no edge, even though they, at times, can appear to have edge.
Now, I'm not saying you didn't have legitimate edge, but you do your readers a disservice by omitting relevant stats and discussions like that.
I develop algorithmic strategies for a living, and my first reaction to reading your post was skepticism. I'm skeptical for two reasons. (1) because your methods are so unconventional in an industry where convention rules, and (2) because of the time frame of your success, which happened to be one of the more impressive market recoveries in history.
I can't tell you how many people I've worked with who fail to isolate the source of their pnl (myself included at times). This is key. It's important to benchmark your strategy against other stupid ones that you know don't have edge. When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Doing this will truly help isolate whether or not luck is involved. When you say that the number and size of your trades justifies the strategy's validity, that's just wrong. You could do 1000 trades in a day: buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
I make all traders benchmark their work against a series of other strategies that I know have no edge, even though they, at times, can appear to have edge.
Now, I'm not saying you didn't have legitimate edge, but you do your readers a disservice by omitting relevant stats and discussions like that.
> I'm skeptical for two reasons. (1) because your methods are so unconventional in an industry where convention rules, and (2) because of the time frame of your success,
I was also in this business, and there's nothing unconventional about his methods. It would, in fact, closely describe the methods of more than one shop I'm familiar with. (Except they WERE able to overcome the declines). And the 3-6 month indicator lifetime looks eerily familiar.
And these places are anything but "convention rules" - it's "creativity rules, before our competitors get creative enough".
> When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Assuming the OP is telling the truth, there is no equivalent "long equities" strategy that would make 1500% profit over 6 months (%3000 annualized), with a max drawdown of 20% ($2000 on $10000 - but his max drawdown was probably closer to 5% than to 20%). You are welcome to demonstrate that there is.
Sounds to me like you are doing low frequency strategies; it's a completely different ballgame than HFT. He's done 400,000 trades, half of them long, half of them short. It might have been luck, and he might have been riding something underlying the equities, but this is NOT equivalent to being long equities. He might have found a way to get non-linear leverage (rather than prediction). But that's also worth a lot of money in the right hands.
> buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
That may be (I wasn't trading in 2009-2010, and don't remember the movements or the required margins), but that would have had much higher volatility (and days with much more than $2000 loss) than the OP had. (Assuming, of course, he is telling the truth)
I was also in this business, and there's nothing unconventional about his methods. It would, in fact, closely describe the methods of more than one shop I'm familiar with. (Except they WERE able to overcome the declines). And the 3-6 month indicator lifetime looks eerily familiar.
And these places are anything but "convention rules" - it's "creativity rules, before our competitors get creative enough".
> When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Assuming the OP is telling the truth, there is no equivalent "long equities" strategy that would make 1500% profit over 6 months (%3000 annualized), with a max drawdown of 20% ($2000 on $10000 - but his max drawdown was probably closer to 5% than to 20%). You are welcome to demonstrate that there is.
Sounds to me like you are doing low frequency strategies; it's a completely different ballgame than HFT. He's done 400,000 trades, half of them long, half of them short. It might have been luck, and he might have been riding something underlying the equities, but this is NOT equivalent to being long equities. He might have found a way to get non-linear leverage (rather than prediction). But that's also worth a lot of money in the right hands.
> buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
That may be (I wasn't trading in 2009-2010, and don't remember the movements or the required margins), but that would have had much higher volatility (and days with much more than $2000 loss) than the OP had. (Assuming, of course, he is telling the truth)
Thank you. You are right - I should clarify things by saying my program had no directional bias. It was a 50/50 split of longs/shorts. Are there other stats I'm missing?
Given a max loss of 2k, we already know the Sharpe Ratio was pretty good.
2009-10 was more than just a huge rally, it was also a period where vol and skew were massively mispriced. I know this is high frequency, but like I alluded to, you need to make sure that what you're doing isn't replicating the pnl profile of low frequency strategies.
So, how did you perform relative to vol sellers? From the market bottom to the end of 2010, the max daily loss for a vol seller was about 3x average daily pnl, and >80% winners. So your returns do sound better, but not incredibly.
But, even if you failed to perform as well as vol selling did over the same period, that doesn't negate the strategy's validity. If returns were not correlated, then it's safe to say that you weren't just inadvertently shorting vol.
So, start there, work out a regression comparing your daily returns to someone selling vol. Do the same with moving average strategies. Mocking up a simple market making back test versus an ES beta is hard, but that too would be a something to test against. I don't expect you to do any of this, and I'm not going to bother to either. I'm just saying that a complete discussion of this subject would include that information.
2009-10 was more than just a huge rally, it was also a period where vol and skew were massively mispriced. I know this is high frequency, but like I alluded to, you need to make sure that what you're doing isn't replicating the pnl profile of low frequency strategies.
So, how did you perform relative to vol sellers? From the market bottom to the end of 2010, the max daily loss for a vol seller was about 3x average daily pnl, and >80% winners. So your returns do sound better, but not incredibly.
But, even if you failed to perform as well as vol selling did over the same period, that doesn't negate the strategy's validity. If returns were not correlated, then it's safe to say that you weren't just inadvertently shorting vol.
So, start there, work out a regression comparing your daily returns to someone selling vol. Do the same with moving average strategies. Mocking up a simple market making back test versus an ES beta is hard, but that too would be a something to test against. I don't expect you to do any of this, and I'm not going to bother to either. I'm just saying that a complete discussion of this subject would include that information.
vol
Guess you are familiar with http://en.wikipedia.org/wiki/Survivorship_bias? In 2009, there were probably tonnes of people trying to exploit the market using similar low-tech methods as you. Even if all of them were at best break-even, some of them likely made a lot of money on their unprofitable algorithms by pure chance thanks to the size of the cohort. Those few blogged about it and those who lost money didn't. :) I'm not saying that you just were lucky (please dont take this as criticism) - survival bias is just one of those things that always come to mind when people write about how they broke the market or when some investor is presenting his incredibly smart investment strategy that has netted him millions.
It's a great point and seems like a very smart thing to keep in mind. I think in my case, based on the statistics involved, the odds that my success was luck just seems astronomically small. But, guess I'm biased in my own way :)
By luck and skill you found a temporary systematic bias that other players missed. It was even luckier that you found it without a lot of upfront losses. But you could have made many attempts and not found any bias and overall lost money and gave up. If lots of people are losing small sums to find these biases, then it may be that the expected return of trying to find biases is zero or negative.
At best HFT is a near zero sum game. It isn't creating value for customers. It isn't making the world a better place.
It is an unfortunate flaw of our economic system that so many smart people put so much effort into playing zero sum games with each other.
At best HFT is a near zero sum game. It isn't creating value for customers. It isn't making the world a better place.
It is an unfortunate flaw of our economic system that so many smart people put so much effort into playing zero sum games with each other.
An example of your last point.
I know a very good engineer, who used to design innovative chips for 4G/LTE mobile telephony. These chips contributed to the market position of one of today's leading mobile phone manufacturers.
Today, this engineer is designing ASICs for high frequency trading (basically a specialised Ethernet switch, with all extra logic stripped out, so packets go through a few nanoseconds faster).
HFT isn't a zero sum game. It's sucking resources away from productive disciplines into an unproductive discipline, so making a net negative contribution.
I know a very good engineer, who used to design innovative chips for 4G/LTE mobile telephony. These chips contributed to the market position of one of today's leading mobile phone manufacturers.
Today, this engineer is designing ASICs for high frequency trading (basically a specialised Ethernet switch, with all extra logic stripped out, so packets go through a few nanoseconds faster).
HFT isn't a zero sum game. It's sucking resources away from productive disciplines into an unproductive discipline, so making a net negative contribution.
Why are you ignoring HFT's positive contribution?
Could you please elaborate what that contribution is?
It's always the same bullshit excuse: "providing liquidity". It's just that you pretty much need to be another HFT bot to partake in that liquidity.
From what I understood, this contribution is not about making stuff nanoseconds faster, but about how this pushes spreads down. Anyone doing any trading will be happier to see the spreads smaller, wouldn't he?
Note: by spreads I mean the difference between buy and sell prices. I don't know if there is a special word for it in this context.
Note: by spreads I mean the difference between buy and sell prices. I don't know if there is a special word for it in this context.
Exactly. HFT reduces counterparty risk for market makers (because with HFT, it's much more likely that there will be a counterparty for any given trade). This enables the market makers to reduce their bid-ask spreads; the profit from the bid-ask spread is what covers the risk a market maker faces from their market clearing obligations.
Do you know of any data on the size of the spreads over time?
How about efficiency? People call the liquidity providing aspects of HFT 'bullshit', but computers have vastly reduced the manpower necessary to manage a market.
Each futures pit used to have hundreds of traders, who required several assistants/support and commanded a huge salary. Many firms needed multiple traders in a pit, just to be able to make sure they could provide liquidity to all possible market participants. Today, a couple strategists with a small team of programmers can cover dozens of futures markets at once.
The same principle holds across bond, FX, equity and options markets alike. HFT has supplanted a terribly inefficient market with a better one. Is it perfect or even good? Probably not, but it's magnitudes better than the traditional method.
Each futures pit used to have hundreds of traders, who required several assistants/support and commanded a huge salary. Many firms needed multiple traders in a pit, just to be able to make sure they could provide liquidity to all possible market participants. Today, a couple strategists with a small team of programmers can cover dozens of futures markets at once.
The same principle holds across bond, FX, equity and options markets alike. HFT has supplanted a terribly inefficient market with a better one. Is it perfect or even good? Probably not, but it's magnitudes better than the traditional method.
An argument can also be made that this is a net negative contribution, as instead of a market employing hundreds of people, it's only employing dozens. Ergo, more unemployed people. While this is good for the market's owners and those currently employed to trade there, it is bad for the economy as a whole.
You're on Hacker News, but you think that destroying jobs with technological innovation is a bad thing?
There are plenty of arguments for its contribution. I don't need to repeat them.
Please help me understand this better?
With a deep understanding of markets and trading I fail to see why you see 'luck' as an explanatory variable is inversely correlated with the frequency of your trades (notwithstanding the effect of trading expenses)?
From what I have gleaned the following seems to be true: 1. Your algorithms worked (made money) 2. Then your algorithms did not work, but you could not figure out why
If you do not know why something stopped working it seems unlikely that you had a full understanding of why it was working in the first place. Without understanding the nature of the predictive value of the algorithm while it was working, its success seems to be good fortune.
Your algorithm could have shown a systematic correlation to any number of factors that could have created strong performance over several months. Performance would then be attributed to accidentally 'timing' a favorable market.
I know you feel differently, what am I missing?
And either way - kudos on the $500k.
With a deep understanding of markets and trading I fail to see why you see 'luck' as an explanatory variable is inversely correlated with the frequency of your trades (notwithstanding the effect of trading expenses)?
From what I have gleaned the following seems to be true: 1. Your algorithms worked (made money) 2. Then your algorithms did not work, but you could not figure out why
If you do not know why something stopped working it seems unlikely that you had a full understanding of why it was working in the first place. Without understanding the nature of the predictive value of the algorithm while it was working, its success seems to be good fortune.
Your algorithm could have shown a systematic correlation to any number of factors that could have created strong performance over several months. Performance would then be attributed to accidentally 'timing' a favorable market.
I know you feel differently, what am I missing?
And either way - kudos on the $500k.
I think you undersell yourself - kudos to your success. I'd back a hacker with a plan (and a cash flow crises perhaps) over an army of PhDs any day! Maybe the course you should think about teaching is how to how to orgainse such a high-quality hack as you've described in the article :)
I'm currently building a semi-high frequency trading solution and the problem I run into is the sheer breadth of expertise you need to get it all happening. Modern chip design, low-latency, lock-free concurrent messaging, fault-tolerant system design, adaptive learning algorithms, k-means clustering and broker APIs are just a smattering of the ideas I'm trying to get across to make progress. For me, algorithm creation comes more easily than reading about and implementing a broker interface.
There is certainly armies of PhDs out there backed by big money but they exist behind heavily guarded intellectual property walls. An open source HFT/Algo/Automated trading platform that brings a hacker sensibility to this problem domain would be seriously competitive.
I'm currently building a semi-high frequency trading solution and the problem I run into is the sheer breadth of expertise you need to get it all happening. Modern chip design, low-latency, lock-free concurrent messaging, fault-tolerant system design, adaptive learning algorithms, k-means clustering and broker APIs are just a smattering of the ideas I'm trying to get across to make progress. For me, algorithm creation comes more easily than reading about and implementing a broker interface.
There is certainly armies of PhDs out there backed by big money but they exist behind heavily guarded intellectual property walls. An open source HFT/Algo/Automated trading platform that brings a hacker sensibility to this problem domain would be seriously competitive.
Thank you for posting this. Very interesting to read.
Perhaps posting the source code would not be a good idea, but posting more details would be welcome so that people interested could follow their own path to automated trading.
Perhaps posting the source code would not be a good idea, but posting more details would be welcome so that people interested could follow their own path to automated trading.
I didn't see it in the article and I'm sorry if I missed it... but did you mention what the initial bankroll was?
The Instagram guys found an edge. It won them 730m. Fantastic. More than any edge ever won by me. But the market has changed so much since then, please be careful before you follow this course.
You are not wrong, but what you wrote here is applicable to any success story posted on HN.
Caveat lector. Always.
You are not wrong, but what you wrote here is applicable to any success story posted on HN.
Caveat lector. Always.
I think with the automated trading example, it makes it seem much easier for anyone to dip their cup in the stream.
When you think Facebook/Instagram, you think "Damn, those guys got lucky as hell". When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Sure, it may take months to lose your shirt selling a photo service to Face/Goog/Apple. You can lose everything overnight with automated trading.
My father used to trade commodities for a living in the pit at the CME many moons ago, and when I was growing up I would be his technical side when he was trading out of our suburban Chicago home (setting up FM receiver/satellite dish/etc for real time quote data, staying up late nights with him running through trading scenarios in Tradestation on Win3.1 with data downloaded in bulk from Knight Ridder, and so forth).
Something very important I learned from him was: "The market can stay irrational longer than you can stay solvent." With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
When you think Facebook/Instagram, you think "Damn, those guys got lucky as hell". When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Sure, it may take months to lose your shirt selling a photo service to Face/Goog/Apple. You can lose everything overnight with automated trading.
My father used to trade commodities for a living in the pit at the CME many moons ago, and when I was growing up I would be his technical side when he was trading out of our suburban Chicago home (setting up FM receiver/satellite dish/etc for real time quote data, staying up late nights with him running through trading scenarios in Tradestation on Win3.1 with data downloaded in bulk from Knight Ridder, and so forth).
Something very important I learned from him was: "The market can stay irrational longer than you can stay solvent." With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
> You can lose everything overnight with automated trading.
I'll take it ad absurdum: You can lose everything in a second by not looking left and right while crossing the road. Or even by looking left and right while crossing the road, when someone else is driving recklessly.
It is possible to attempt HFT with not much more risk than stating a new InstaFaceGoogApple service. Put $10,000 in your margin account, and use a broker that practices proper margin checking. Tada! You're not going to lose more than $15,000 over that. (Yes, you can lose more than you put in your margin account, but not by much).
While that's more, upfront, than InstaFaceGoogApple, it is comparable to the 4 months of salary that you're going to forfeit while building the InstaFace service. And unlike most InstaFace apps, you have immediate market feedback, which can only be a good thing.
Note: Instagram did have immediate feedback from the public at large, forcing them to scale much earlier than they expected - but they did not have a feedback as to the financial value of their proposition. In fact, it wouldn't take much for instagram worth to be zero. Read, e.g. http://www.jamesaltucher.com/2011/02/my-name-is-james-a-and-... - a $100M acquisition back then is like a $300M acquisition in today's valuations; not Instagram but definitely nothing to ignore.
> With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
That's true. But you still have to remember that 90% of startups fail, and of those that succeed, many are just moderately successful. And yet no one keeps yelling "but most startups lose money!" at every HN story.
> When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Which is what we should address, and these "it's a gamble" warning do not. When you see Suzanne Vega singing, you might think "Hey, it can't be that hard to sing". Many people do. And yet, they grow out of it, usually without trying to publish an album (and failing). This should be no different.
I'll take it ad absurdum: You can lose everything in a second by not looking left and right while crossing the road. Or even by looking left and right while crossing the road, when someone else is driving recklessly.
It is possible to attempt HFT with not much more risk than stating a new InstaFaceGoogApple service. Put $10,000 in your margin account, and use a broker that practices proper margin checking. Tada! You're not going to lose more than $15,000 over that. (Yes, you can lose more than you put in your margin account, but not by much).
While that's more, upfront, than InstaFaceGoogApple, it is comparable to the 4 months of salary that you're going to forfeit while building the InstaFace service. And unlike most InstaFace apps, you have immediate market feedback, which can only be a good thing.
Note: Instagram did have immediate feedback from the public at large, forcing them to scale much earlier than they expected - but they did not have a feedback as to the financial value of their proposition. In fact, it wouldn't take much for instagram worth to be zero. Read, e.g. http://www.jamesaltucher.com/2011/02/my-name-is-james-a-and-... - a $100M acquisition back then is like a $300M acquisition in today's valuations; not Instagram but definitely nothing to ignore.
> With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
That's true. But you still have to remember that 90% of startups fail, and of those that succeed, many are just moderately successful. And yet no one keeps yelling "but most startups lose money!" at every HN story.
> When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Which is what we should address, and these "it's a gamble" warning do not. When you see Suzanne Vega singing, you might think "Hey, it can't be that hard to sing". Many people do. And yet, they grow out of it, usually without trying to publish an album (and failing). This should be no different.
>>You can lose everything in a second by not looking left and right while crossing the road. Or even by looking left and right while crossing the road, when someone else is driving recklessly.
The point is you are lured into crossing the road, when you absolutely didn't have to.
The point is you are lured into crossing the road, when you absolutely didn't have to.
How are you "lured" by reading an article about someone who successfully crossed the road, any more than you are "lured" into a singing career by reading about Adele or "lured" into building an instagram clone?
No broker is offering the ability to engage in HFT for 10 grand.
In order to open an account with the neccessary infrastructure to engage in HFT one must be an accredited investor (typically means having a net worth of 1 million dollars or more) and the cheapest brokers typically require a minimum deposit of $500,000.
Not to mention the overall costs including hardware, co-location, market data and other vendor costs are on the order of 45-50k a month.
With $10,000 you can't even open up a normal day trading account as the law required a minimum deposit of 25k.
In order to open an account with the neccessary infrastructure to engage in HFT one must be an accredited investor (typically means having a net worth of 1 million dollars or more) and the cheapest brokers typically require a minimum deposit of $500,000.
Not to mention the overall costs including hardware, co-location, market data and other vendor costs are on the order of 45-50k a month.
With $10,000 you can't even open up a normal day trading account as the law required a minimum deposit of 25k.
Not true. I have a commodities trading account I use to trade corn, soybeans, and hogs. Minimum opening balance was $5K (Tradestation). Scottrade and others have a minimum of $500. Now if you're talking margin accounts, sure, you're going to need more.
> "The market can stay irrational longer than you can stay solvent."
As long as we're quoting Keynes, let's also remember this gem from a letter he wrote to the regents of King's College about the performance of their endowment's portfolio (which Keynes managed).
"The management of stock exchange investments of any kind is a low pursuit, having very little social value and partaking (at its best) of the nature of a game of skill, from which it is a good thing for most members of our Society to be free; whereas the justification of Worlaby and Elsham lies in its being a constructive and socially beneficial enterprise, where we exercise a genuine entrepreneurial function, in which many of our body can be reasonably and usefully interested. I welcome the fact that the Estates Committee-to judge from their poker faces and imperturbable demeanour-do not take either gains or losses from the Stock Exchange too gravely-they are much more depressed or elated (as the case may be) by farming results. But it may be useful and wise nevertheless, to analyse from time to time what is being done and the principles of our policy."
Edit: Worlaby and Elsham was a farm that the endowment owned.
As long as we're quoting Keynes, let's also remember this gem from a letter he wrote to the regents of King's College about the performance of their endowment's portfolio (which Keynes managed).
"The management of stock exchange investments of any kind is a low pursuit, having very little social value and partaking (at its best) of the nature of a game of skill, from which it is a good thing for most members of our Society to be free; whereas the justification of Worlaby and Elsham lies in its being a constructive and socially beneficial enterprise, where we exercise a genuine entrepreneurial function, in which many of our body can be reasonably and usefully interested. I welcome the fact that the Estates Committee-to judge from their poker faces and imperturbable demeanour-do not take either gains or losses from the Stock Exchange too gravely-they are much more depressed or elated (as the case may be) by farming results. But it may be useful and wise nevertheless, to analyse from time to time what is being done and the principles of our policy."
Edit: Worlaby and Elsham was a farm that the endowment owned.
Wow that is a gem. Keynes was a giant. Wish the political parties wouldn't run from him.
A wonderful quote, but this is the only google result for it. Can you provide a source?
http://www.capitalideasonline.com/articles/index.php?id=2049 claims that it's from a "Memorandum for the Estates Committee, King's College, Cambridge" dated the 8th of May, 1938.
"You can lose everything overnight with automated trading."
Didn't your father teach you about "Stop orders"
You can't have your algorithm cranking away without supervision. And to be extra sure, lots of testing and LIMITS.
Limit the amount and value of orders.
With stocks, worst case: you lose the face value of stocks in your portfolio
Derivatives: you can lose more, even 'infinite liability' (still, it's constrained by the stock market inertia)
Didn't your father teach you about "Stop orders"
You can't have your algorithm cranking away without supervision. And to be extra sure, lots of testing and LIMITS.
Limit the amount and value of orders.
With stocks, worst case: you lose the face value of stocks in your portfolio
Derivatives: you can lose more, even 'infinite liability' (still, it's constrained by the stock market inertia)
Stop orders don't guarantee execution or any specific limit to the loss; During a flash crash, you'll realize that a "10% stop loss" order CAN become a 50% loss.
This. I'm very familiar with stop orders. They're useless once the market goes to hell (the exact conditions you need them in).
I don't disagree with you, still, 50% losses are not 100% losses (or even more than 100% losses)
So I guess they have a role in limiting losses (which had they not been there would be much bigger)
So I guess they have a role in limiting losses (which had they not been there would be much bigger)
I love this comment :)
What pains me is just this year I've heard in 3 separate occasions for 3 separate startup businesses {industries: ['transportation', 'social', and 'mobile ads']} people propose "Let's do the Instagram strategy." It may be obvious to you and me how absurd that sounds, but there are a non-trivial number of people who blindly follow headlines.
Edit: I agree with @toomuchtodo. It's just too easy to risk with HFT that the warning is needed here more than elsewhere.
What pains me is just this year I've heard in 3 separate occasions for 3 separate startup businesses {industries: ['transportation', 'social', and 'mobile ads']} people propose "Let's do the Instagram strategy." It may be obvious to you and me how absurd that sounds, but there are a non-trivial number of people who blindly follow headlines.
Edit: I agree with @toomuchtodo. It's just too easy to risk with HFT that the warning is needed here more than elsewhere.
Agreed... in essence this is why we're all here. As entrepreneurs we all educated risk takers, and we realize any venture is essentially gambling if there is no edge. At any time, there could be a new idea that pushes any one HFT algorithm (or mobile photo sharing app, or words with friends clone) past the established mindshare into blue ocean territory. When that time comes, do you want to be caught with your pants down, lumbering under the excuse that you thought the oceans were too red for you to bother?
This is a very mean and unconstructive comment to someone who made the impressive achievement of building his own automated trading system and actually making money from it. I've started calling out comments like this one, because they cause a bad environment for useful discussion.
The only argument in your comment that isn't your own unfounded opinion is that market makers make money from people who execute trades. But this is true by definition.
The traders who "gleefully picked off all those trades" weren't outsmarting anyone, they were simply profiting from the difference in the asking and offering price in the market. This is the role of a market maker, and actually makes it cheaper for people like OP to execute a large number of trades. So even though this comment sounds like a sensible rebuttal of the linked article, it doesn't really say anything at all.
Again, sorry for creating a negative reply and contributing to a bad tone, but I really the right thing is to call out these kinds of replies. They discourage honest sharing and discussion.
yajoe's comment is an example of how to criticize a post like this in a useful way. http://news.ycombinator.com/item?id=4748989
The only argument in your comment that isn't your own unfounded opinion is that market makers make money from people who execute trades. But this is true by definition.
The traders who "gleefully picked off all those trades" weren't outsmarting anyone, they were simply profiting from the difference in the asking and offering price in the market. This is the role of a market maker, and actually makes it cheaper for people like OP to execute a large number of trades. So even though this comment sounds like a sensible rebuttal of the linked article, it doesn't really say anything at all.
Again, sorry for creating a negative reply and contributing to a bad tone, but I really the right thing is to call out these kinds of replies. They discourage honest sharing and discussion.
yajoe's comment is an example of how to criticize a post like this in a useful way. http://news.ycombinator.com/item?id=4748989
While writing his own trading system is a decent accomplishment, due to things such as an overall rising market in the time period involved and survivorship bias, the original author is likely to be completely mistaken about the reason for his winnings.
Given that he might convince other people to engage in high tech gambling in a less-favorable market than the one he operated in, strong words are called for in this case.
Given that he might convince other people to engage in high tech gambling in a less-favorable market than the one he operated in, strong words are called for in this case.
You could argue this, but in that case your arguments have to hold water and not just be a cursory dismissal. Ref. yajoe's much more thorough reasoning.
It is only gambling in the sense that any business is gambling: Your customers might stop coming tomorrow because the fad wore off, or a competitor provides a better/cheaper/hipper alternative.
(And indeed, living is gambling. It's all just a matter of the risk/reward portfolie).
But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/trade) - which makes it one of the best businesses one could ever have.
You can't live without gambling - by e.g. going to be a salaried employee for Yahoo rather than Google or that weird newfangled "TheFacebook" thingy back in 2004, was a gamble.
jspauld, statistically speaking, has made less of a gamble there than almost anyone else posting on HN.
> So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.
True. But that's true for every single success story posted on HN, reddit, or USAToday.
(And indeed, living is gambling. It's all just a matter of the risk/reward portfolie).
But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/trade) - which makes it one of the best businesses one could ever have.
You can't live without gambling - by e.g. going to be a salaried employee for Yahoo rather than Google or that weird newfangled "TheFacebook" thingy back in 2004, was a gamble.
jspauld, statistically speaking, has made less of a gamble there than almost anyone else posting on HN.
> So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.
True. But that's true for every single success story posted on HN, reddit, or USAToday.
No. Every business has a risk element, but what makes this gambling is that there is no good or service being produced. It's a game of trying to outguess the other players, with one trader's gain being another trader's loss (relative to market returns).
Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
> No. Every business has a risk element, but what makes this gambling is that there is no good or service being produced.
I was not aware that this is what defines gambling. And "no service produced" is certainly wrong by accepted economic theory - arbitrageurs provide a price discovery service for everyone; they get rewarded for exposing the inefficient prices, even though it is done through market mechanics rather than a specific customer.
OP appears to be a statistical arbitrageur - which is the same concept, except that it includes a shift in time or space (and incurs risk). You might not be interested in this price discovery service, but other people are paying for it with their wallet. (And it's mostly the market makers who pay for this with reduced profits)
> one trader's gain being another trader's loss (relative to market returns).
That's not true in investing in general - when shares have time to appreciate or depreciate, it is definitely not a zero sum game. Everyone can win, or everyone can lose, or anything in between (it all depends on your time range, and your measure of loss or profit. The "non-zero-sum" element arrives partly from companies using operating profit to buy back their own shares).
> Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
That's only true if all players are hf players. If there is sufficient non-HF activity, then the zero-sum argument does not hold.
(I'm not saying that it's not a good approximation - in most time scales, in most scenarios, it is - but it is not the mathematical truth you imply it is)
I was not aware that this is what defines gambling. And "no service produced" is certainly wrong by accepted economic theory - arbitrageurs provide a price discovery service for everyone; they get rewarded for exposing the inefficient prices, even though it is done through market mechanics rather than a specific customer.
OP appears to be a statistical arbitrageur - which is the same concept, except that it includes a shift in time or space (and incurs risk). You might not be interested in this price discovery service, but other people are paying for it with their wallet. (And it's mostly the market makers who pay for this with reduced profits)
> one trader's gain being another trader's loss (relative to market returns).
That's not true in investing in general - when shares have time to appreciate or depreciate, it is definitely not a zero sum game. Everyone can win, or everyone can lose, or anything in between (it all depends on your time range, and your measure of loss or profit. The "non-zero-sum" element arrives partly from companies using operating profit to buy back their own shares).
> Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
That's only true if all players are hf players. If there is sufficient non-HF activity, then the zero-sum argument does not hold.
(I'm not saying that it's not a good approximation - in most time scales, in most scenarios, it is - but it is not the mathematical truth you imply it is)
Futures, which I assume was the original poster's instrument of choice, are a zero sum game by definition as every contract is an agreement between two parties: buyer and seller.
Only if you assume all players only ever use futures. But make an interest synthetic contract (short future long underlying) and you're out of the zero sum regime again. And it's enough that one actor is not inside the zero sum regime to make that apply to the whole game.
Again, it's a great approximation most of time and over most time periods and asset classes, but it is NOT axiomatic in the way most people believe it is.
Remember: as long as there is a way to inject or withdraw more capital into the system (through whatever asset class, as they are all interconnected), the sum is not identically zero.
Just assume one of the stocks is a gold mining company that works efficiently. The share value rises, and the shares are redeemable for the gold, without anyone having to lose anything (except mother earth)
Again, it's a great approximation most of time and over most time periods and asset classes, but it is NOT axiomatic in the way most people believe it is.
Remember: as long as there is a way to inject or withdraw more capital into the system (through whatever asset class, as they are all interconnected), the sum is not identically zero.
Just assume one of the stocks is a gold mining company that works efficiently. The share value rises, and the shares are redeemable for the gold, without anyone having to lose anything (except mother earth)
As a professional poker player, analyst and journalist, and being fairly well-read on classifications of gambling vs skill game in different jurisdictions, I have not before come across a definition of gambling that was rooted in the idea that "no good or service is being produced."
Any zero sum or negative expected return conctract would meet this definition (quantitatively). if it floats your boat (makes you hallucinate etc), of course that might be considered a good or service. so you're right, if I was a lawyer those words would be loose language.
So consider an insurance company. This company is engaging in transactions (providing insurance) in which one party will make $X and the other will lose an equal amount of dollars. Would you say that this company is providing no service?
It sounds like you are making the argument that this is zero-sum game, but whether something is zero-sum depends on your utility function. If the players are risk averse, then a transaction like buying insurance can yield positive utility for both participants.
Many trading strategies are performing a service in similar (but more complicated) ways.
It sounds like you are making the argument that this is zero-sum game, but whether something is zero-sum depends on your utility function. If the players are risk averse, then a transaction like buying insurance can yield positive utility for both participants.
Many trading strategies are performing a service in similar (but more complicated) ways.
[deleted]
Very true, every time you hire someone or spend money on Adwords you're gambling that you'll net more in $$$ or at least lifestyle improvement (in the case of a new employee) than your cost. That is, if you're in business to make a profit, not just spending OPM to build your brand. Even acquiring a customer is a gamble.
Exactly. I've played millions (!) of real-money online poker hands and won some money in the process (not anywhere near what most people who've played that many hands did).
Most people don't understand that, when you're able to recognize patterns, playing millions of hands while never exposing more than 1% of your bankroll on any deal is not "gambling" but "printing money" (a tiny amount of money in my case compared to consulting but that is not the point).
At the same time the very fact that obviously (seen most of the posts here) most people don't understand basic bankroll management, risk management, standard deviation, expected value, variance, etc. means there are probably quite some opportunities out there to make money for those who do understand that ; )
Most people don't understand that, when you're able to recognize patterns, playing millions of hands while never exposing more than 1% of your bankroll on any deal is not "gambling" but "printing money" (a tiny amount of money in my case compared to consulting but that is not the point).
At the same time the very fact that obviously (seen most of the posts here) most people don't understand basic bankroll management, risk management, standard deviation, expected value, variance, etc. means there are probably quite some opportunities out there to make money for those who do understand that ; )
I think the real problem (as with so many problems) is definitional. What does "gambling" actually mean?
If I tried to kludge together a definition, I might come up with something like:
>Risking the loss of something of value in exchange for the possibility of gaining something of greater value in a situation where the determining factor of losing value or gaining value is random chance
The problem with a definition like this is that, as others have pointed out, it applies to vast realms of human endeavor, from founding a company to playing the lottery. It also includes no distinction between risks with a positive expected value and risks with a negative expected value.
If a lottery has ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
What we really need is a word that only refers to gambling in situations with an expected value less than or equal to zero.
If I tried to kludge together a definition, I might come up with something like:
>Risking the loss of something of value in exchange for the possibility of gaining something of greater value in a situation where the determining factor of losing value or gaining value is random chance
The problem with a definition like this is that, as others have pointed out, it applies to vast realms of human endeavor, from founding a company to playing the lottery. It also includes no distinction between risks with a positive expected value and risks with a negative expected value.
If a lottery has ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
What we really need is a word that only refers to gambling in situations with an expected value less than or equal to zero.
For the sake of clarity, I meant:
>If a lottery has a total of ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
>If a lottery has a total of ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
> What we really need is a word that only refers to gambling in situations with an expected value less than or equal to zero.
I believe we call that gambling.
I believe we call that gambling.
I'm a pretty risk averse guy and my typical reaction is to figure out why something won't work. One of my pet excuses is that assuming markets are efficient, someone must have already figured this out/ come up with a better exploit, etc. Most of the time I'm right. What troubles me is encapsulated in the following parable:
A UChicago economist and graduate student are walking across campus. The student says ... hey ... there is a hundred dollar bill on the ground! The economist scoffs and says no there isn't ... if there was one, someone must have picked it up already.
Sometimes I catch myself thinking this way. I have to remind myself that (a) markets aren't perfect, and (b) the real world has huge asymmetries in information, ideas, and perhaps willpower (by this, I mean while 100 people might think of a great idea, not all will attempt to implement it; even then, people will differ in execution).
That said, you're likely right. This trading strategy will likely lose money today :-p
A UChicago economist and graduate student are walking across campus. The student says ... hey ... there is a hundred dollar bill on the ground! The economist scoffs and says no there isn't ... if there was one, someone must have picked it up already.
Sometimes I catch myself thinking this way. I have to remind myself that (a) markets aren't perfect, and (b) the real world has huge asymmetries in information, ideas, and perhaps willpower (by this, I mean while 100 people might think of a great idea, not all will attempt to implement it; even then, people will differ in execution).
That said, you're likely right. This trading strategy will likely lose money today :-p
It's easy to fall into that mindset. And in fact, that mindset is right back where I am now. The only reason I had the gall to attempt this in the first place was the the simple fact that I was making money at the time (in 2008) 'manually' day trading the Russell 2000. I thought this 'should not be possible' so I figured there's no reason not to try an automated program.
People will tell you that you were just a lucky monkey. But you could have run your algorithm on past data, for hundreds or thousands of fake portfolios, to tell, statistically, what the odds of your algorithm being simply lucky are.
In early 2000s I wrote a machine learning algorithm that beat the S&P 100 with over 1 trillion to 1 odds against it being luck. It predicted a full trading day in advance. But that was all on paper at trading firms' puny costs; unlike you I couldn't beat retail costs. It's amazing that you could do that. For that reason alone I think it's highly likely that you were a skilled monkey.
Also like you, nobody in the industry was interested in my code, even after an industry magazine watched it for 3 months and found it gave "stellar" performance. The few people I was able to discuss it with told me point blank that it was impossible to do it skillfully (efficient market theory), so they assumed it was a hoax or the algorithm was just lucky.
In early 2000s I wrote a machine learning algorithm that beat the S&P 100 with over 1 trillion to 1 odds against it being luck. It predicted a full trading day in advance. But that was all on paper at trading firms' puny costs; unlike you I couldn't beat retail costs. It's amazing that you could do that. For that reason alone I think it's highly likely that you were a skilled monkey.
Also like you, nobody in the industry was interested in my code, even after an industry magazine watched it for 3 months and found it gave "stellar" performance. The few people I was able to discuss it with told me point blank that it was impossible to do it skillfully (efficient market theory), so they assumed it was a hoax or the algorithm was just lucky.
What did you end up doing with your code? Would you be able to run it today with the low-cost broker APIs?
The code sits in one of my archive folders. I ran it for a few years, perhaps to 2004, and saw the market steadily becoming more efficient, lowering my results (like the OP did). It may well be that it no longer predicts skillfully or profitably. As I recall, to beat the market the costs had to be very low, like pennies per trade, with no bid/ask spread, which I understood to be possible for large trading firms.
OK, cool. There are some places that offer equity trading for ~$0.005/share, but that says nothing about overcoming bid/ask. Looks like the OP did that by throwing a bit of market making into the mix.
If you make that many trades and your total market exposure at any given moment is small yet you consistently make a net profit then you've found an edge. At which point it's not really gambling any more, it's just making money!
If your net exposure is small, but that's only because you're offsetting various positions then you're probably picking up nickels in front of the volatility steamroller & if you stay in the market long enough you'll get squashed at some point.
If your net exposure is small, but that's only because you're offsetting various positions then you're probably picking up nickels in front of the volatility steamroller & if you stay in the market long enough you'll get squashed at some point.
I tried to address this concern at the start of my post. If you have some idea of how I manipulated the statistics I'd be happy to respond.
Having said that I can agree that my case is pretty unusual and that everyone should beware of attempting to do something like this. Even for myself I couldn't do it now. (There is a reason I turned my program off.)
Having said that I can agree that my case is pretty unusual and that everyone should beware of attempting to do something like this. Even for myself I couldn't do it now. (There is a reason I turned my program off.)
Ie., if you made 500k in one period, but then lost 50k a year for the next 10 years, you've made a net profit of 0. All I know is that you had one good run, similar to how some mutual funds have a good run for a while.
How much did you spend before you "tuned" it? How much did you spend afterwards? What were the tax consequences of your trades? Did you make exactly 500k? Have you traded at all since then?
You mentioned that you occasionally "sat in" and took some large losing positions. Were these on purpose? Bugs? Was your exposure actually much higher than you thought? Was limiting contract size enough risk management?
How much did you spend before you "tuned" it? How much did you spend afterwards? What were the tax consequences of your trades? Did you make exactly 500k? Have you traded at all since then?
You mentioned that you occasionally "sat in" and took some large losing positions. Were these on purpose? Bugs? Was your exposure actually much higher than you thought? Was limiting contract size enough risk management?
In my final month I lost $600. I didn't include this on the chart to keep things simpler visually. Since then I have not traded and the reason is that it was abundantly clear that my program was no longer working. That's why I shut it off.
With regard to tuning I may have lost $1000 or so but as I wrote in the article I was able to build a backtesting model that accurately simulated live trading. So once I had that I could basically use it to verify I had sufficient edge to make a profit after covering my commissions.
My risk exposure was very low. When I said large losing positions this meant like $600. But the bottom line is I had a daily stop loss of $3000 enforced at my broker. The most I ever lost was around $2000.
Anyway, there is not really some hidden thing that I am not telling people. It does bug me a bit that your comment is at the top given that it says I'm manipulating statistics and was actually one of the guys that the quants gleefully picked off. I think it's unlikely I traded much with other HFT systems but if I did they certainly lost money.
With regard to tuning I may have lost $1000 or so but as I wrote in the article I was able to build a backtesting model that accurately simulated live trading. So once I had that I could basically use it to verify I had sufficient edge to make a profit after covering my commissions.
My risk exposure was very low. When I said large losing positions this meant like $600. But the bottom line is I had a daily stop loss of $3000 enforced at my broker. The most I ever lost was around $2000.
Anyway, there is not really some hidden thing that I am not telling people. It does bug me a bit that your comment is at the top given that it says I'm manipulating statistics and was actually one of the guys that the quants gleefully picked off. I think it's unlikely I traded much with other HFT systems but if I did they certainly lost money.
If you do release the source, what's the best way to be notified of this? Your Twitter account looks pretty active? I'm particularly interested in your risk management strategies (this is where my previous efforts fell short).
Why couldn't you do it now?
Well I could try. But it's not going to be any easier now than it was in 2010. For four months I tried everything I could think of to keep it profitable but in the end nothing worked so I had to shut it off.
I have two theories why it stopped working. I think the market sped up. Latencies are always getting lower and your strategy that worked at 10 ms didn't work with players that are at 1 ms. Also, you might have been gamed because your strategy was easily predictable or you were putting too many trades through the same broker/exchange/etc.
I run an HFT group, and what he describes isn't what we'd call "retail". He was doing a number of things that professional shops do, including making markets to avoid paying the spread
and paying attention to queue position to predict execution
With a bit of luck and a good partner, this guy could have built a sustainable business.
With a bit of luck and a good partner, this guy could have built a sustainable business.
I don't get you haters. sure the title is a bit misleading because he never really discusses what his alphas were. But its a pretty good high level description of the architecture of a hft system. I was a quant at GS and these are not the retail investors you pick off. You have your own set of alphas and most of them are meant to pick on mom and pops clicking away at home. This guy didn't reveal his strategy but nevertheless the graph shows his strategy had a significant edge. The lifetime of a strategy also looks like that. It is another thing that his title for the post is kind of off.
There's a sentence in this article that is critical and yet very easy to overlook: the author had 2 years experience daytrading manually. That already gave him a lot of knowledge of how the markets work and where an edge might be found.
I think that if someone is a good programmer and has some mathematical chops and has that kind of experience daytrading, taking a shot at automated trading is probably a reasonable thing for them to do. Without all of that background, you're right, they're almost certain to lose money.
I think that if someone is a good programmer and has some mathematical chops and has that kind of experience daytrading, taking a shot at automated trading is probably a reasonable thing for them to do. Without all of that background, you're right, they're almost certain to lose money.
I disagree. The point of the article was the show the steps required to develop a statistical advantage in the market place. If you develop a robust model and are very diligent in how it executes and learns, you can be successful. However, what you say about market structure is true. It goes through periods of stability, followed by abrupt changes. Any model that a trader has developed has been developed on such a short time-scale of market activity, that it can turn out to be a bad sample size. Depends on the scale of time and trades.
You know, that is a really good argument.
Except two things:
1) He didn't lose money, he made 500k.
2) If this worked reliably, you would be out of a job.
Except two things:
1) He didn't lose money, he made 500k.
2) If this worked reliably, you would be out of a job.
Er... it doesn't work reliably, and by definition, cannot work reliably. If market inefficiencies exist to be exploited, then someone is ultimately getting the short end of every stick.
Correct. The far more common story is "how I lost $Xk on the stock market". Probably most people with programming/AI skills have tried their luck with the stock picking problem at one time or another. I have a few times, but only in simulation. Even using very elaborate machine learning methods and a lot of training data, making money from automated trades is a difficult problem, and my impression is that it's very much like betting on horses or football games.
How is it like horses and football games? Don't the latter have a lot more people playing with their emotion rather than utilizing an algorithm? Something that you can take advantage of?
I once worked for a software shop, and part of my job was writing trading code in a proprietary language for customers, who ranged from low end day traders to 8 figure annual revenue hedge funds. I had access to all kinds of tools, and saw many a varied strategy. There's quite a lot of money to be made selling solutions. I won't day trade.
What kind of solutions?
It's more a risk management game, not particularly gambling, just look at EUR/AUD: http://stooq.com/q/?s=euraud&c=3y&t=l&a=ln&b...
Would you dare to "predict" the direction of this FX rate movement in the next month? Then it's a matter of calculating potential profit/loss factor and adjusting your trade value. And yes, as with any high risk investment, putting all your eggs in one bucket is not a brilliant idea. Just like taking all your savings to Vegas.
Would you dare to "predict" the direction of this FX rate movement in the next month? Then it's a matter of calculating potential profit/loss factor and adjusting your trade value. And yes, as with any high risk investment, putting all your eggs in one bucket is not a brilliant idea. Just like taking all your savings to Vegas.
"Glorified gambling" is a pretty accurate description of this. However, that isn't necessarily a bad thing. If you have a situation where you can gamble with a long-term positive edge, then the proper strategy is to play with as much money and for as long as possible. The determining factor here is whether or not the combination of a particular investor's strategy, algorithm, and ability to execute will give them a long-term edge over others in the market - not whether or not this may be a risky activity in the short-term.
[deleted]
Gambling can be done intelligently and profitably. Don't be so quick to label gambling as a pitfall to be avoided at all costs. I am not a financial expert, but as someone with a background in math/probability based gambling strategies, I can see similarities in the financial markets.
[deleted]
No. This is not. Not ONE gambling. It is thousands of thousands of gamblings with a consistent winning ratio. cannot be oversimplified to a symmetric chance of win or lose.
Uh, if you look at his daily pnl charts, it looks like gambling with some extremely great odds, he rarely looses any money. That pattern is typically associated with HFT, If you can do many small trades and your strategy really has positive expected value you'll get great returns. I don't expect it would work now though, the HFT market is much more competitive these days.
It could have easily been called "how i lost 500k with machine learning".
If you've really worked in that field than it's very surprising you've never heard about what professional poker players call bankroll management (and they "stole" the concept from professional traders).
The whole point is that you can --either if you gain an edge or get lucky-- win big. Very big. But you're never exposing a large part of your funds in the process.
Maybe OP had an overall "stop loss" at, say, $10K. Had he had five minus $2K days in a row at the start, he'd be out. He wouldn't be broke. He wouldn't be without a car and without a bank account. He just would have lost $10K.
But there's no upper limit as to how much you can win.
All you need is discipline and sound bankroll management.
And, yes, I've won a five-digits figure (hence not anywhere near what OP did) real $$$ at online poker. Starting from $0.01/$0.02 small blind/big blind tables and then working my way up using bankroll management.
It's assymetric. You're foolish if you think that succesful traders who won $x were as likely to lose $x. This is simply not how it works and it's very well explained in OP's article. He's detailing what his maximal daily exposure was and it was tiny compared to what he made.
If you've really worked in that field than it's very surprising you've never heard about what professional poker players call bankroll management (and they "stole" the concept from professional traders).
The whole point is that you can --either if you gain an edge or get lucky-- win big. Very big. But you're never exposing a large part of your funds in the process.
Maybe OP had an overall "stop loss" at, say, $10K. Had he had five minus $2K days in a row at the start, he'd be out. He wouldn't be broke. He wouldn't be without a car and without a bank account. He just would have lost $10K.
But there's no upper limit as to how much you can win.
All you need is discipline and sound bankroll management.
And, yes, I've won a five-digits figure (hence not anywhere near what OP did) real $$$ at online poker. Starting from $0.01/$0.02 small blind/big blind tables and then working my way up using bankroll management.
It's assymetric. You're foolish if you think that succesful traders who won $x were as likely to lose $x. This is simply not how it works and it's very well explained in OP's article. He's detailing what his maximal daily exposure was and it was tiny compared to what he made.
Well put.
Risk management is probably the single most important thing to understand in trading. Unfortunately, it's something that lots of people learn late, if ever.
Folks get caught up in the romantic notion of betting it all and winning big, but end up losers. Meanwhile, the consistent winners they aspire to be are exposing perhaps 0.2% of their roll at a time.
Risk management is probably the single most important thing to understand in trading. Unfortunately, it's something that lots of people learn late, if ever.
Folks get caught up in the romantic notion of betting it all and winning big, but end up losers. Meanwhile, the consistent winners they aspire to be are exposing perhaps 0.2% of their roll at a time.
I like the distinction between risk management and the romantic notion of betting it all and winning big. I see the same pattern in other areas as well, e.g. developers who focus on automated testing and other who think it is heroic to modify production code at 2:00 am. I think of it as the cowboys versus the tax accountants.
I actually thought about making that analogy, but it seemed unnecessary as the analogs are just so common.
Though, one thing I think is a bit unique to trading is prevalence of folks who preach without practicing. Just about anyone with a brokerage account can rattle off the same short list of critical do's and don'ts, but very few actually follow them.
Though, one thing I think is a bit unique to trading is prevalence of folks who preach without practicing. Just about anyone with a brokerage account can rattle off the same short list of critical do's and don'ts, but very few actually follow them.
I have some friends that have made a lot of money playing poker. The analogy is very good. But, in poker you have to be willing to deal with much bigger ups and downs than what I had to deal with. You can hit a bad streak and it can hurt - even if you are in fact a really skilled player and do in fact have an edge.
With my program I didn't really have bad streaks because my P&L was averaged out over thousands of trades per day.
With my program I didn't really have bad streaks because my P&L was averaged out over thousands of trades per day.
Not always true. There are plenty of ways to minimize risk.
I played HUSNGs for a living for several years, and I could play three tables at a time (about 9 games per hour) with a 60% winrate. That's incredibly low variance--my graph over the long-term was better than a 45 degree incline.
I played HUSNGs for a living for several years, and I could play three tables at a time (about 9 games per hour) with a 60% winrate. That's incredibly low variance--my graph over the long-term was better than a 45 degree incline.
Yes this is the point I was going to write myself. I played online poker for four years and won over $100,000. I wasn't very good compared to the top 5% of players at my stakes, but I was much better at bankroll management, tilt control, and all of the other soft skills. The way I structured my bankroll made it actually impossible to go broke as well.
Stop-losses are not as effective or nearly as simple as they are described in typical financial media.
http://falkenblog.blogspot.com/2011/02/stop-loss-myth.html (with additional citations in the blog)
http://falkenblog.blogspot.com/2011/02/stop-loss-myth.html (with additional citations in the blog)
I don't get that. It would be true if he just made a few trades, but the author claimed to be making 2000 trades a day. Over a period of months winning that wouldn't qualify as blind luck.
Definitely not blind luck; but boosted by the fact that SPY was on a massive bull run at that time.
This is really cool, any way you cut it.
This is really cool, any way you cut it.
>Over a period of months winning that wouldn't qualify as blind luck.
Why not?
Why not?
It's simple statistics. The author was up 4k/day over 120 days. He doesn't say what his daily volatility was, but let's assume 2k (which squares pretty well with his claim that his worst day was a 2k loss).
With a quick bit of R code, we can simulate his PnL over 120 days multiple times, assuming he has no skill, and see what the probability of him being up 4k/day is. I'll use a t-distribution with 3 degrees of freedom, which allows big up and down swings (again, accentuating the effect of luck).
With a quick bit of R code, we can simulate his PnL over 120 days multiple times, assuming he has no skill, and see what the probability of him being up 4k/day is. I'll use a t-distribution with 3 degrees of freedom, which allows big up and down swings (again, accentuating the effect of luck).
> pnl <- c()
> for (i in 1:1000) pnl[i] <- mean(2000 * rt(120, df=3))
> mean(pnl > 4000)
0.0
By that definition you could start claiming everything as blind luck. Why bother doing anything at all, let luck do the work.
I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines.
I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-che...
He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:
> The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.
So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?
I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-che...
He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:
> The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.
So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?
Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?
'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess.
HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.
--
If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.
HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.
--
If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.
There is an air of either incredibility or sheer jealousy in these comments. Nevertheless, I just wanted to tell the OP that he did a great job. Thanks for sharing. I work in the finance industry as a quantitative software developer, and it certainly is not an easy job for one person to do. In fact, I tried (independent of my professional work) doing this myself, and I ended up losing a lot of money. If people are trying to do this, please please be careful. Big companies, like ones I have worked at, have technical and human resources that are vastly more powerful.
thanks quant123 ;)
Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck.
I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.
The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.
I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.
I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.
The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.
I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.
Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset.
I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.
Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.
Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.
I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.
I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.
Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.
Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.
I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.
In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating).
In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions).
> Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures.
That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt.
Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.
In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions).
> Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures.
That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt.
Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.
Some US venues impose fees for having low fill rate. As far as I know, most US HFT firms' strategies are not based on creating latency by rapidly submitting and canceling orders, but the bad behavior of some players is much more visible than the orderly behavior of others.
When I say limiting trades, I mean naively saying 'I will have at most x positions outstanding'. Each trade has an associated risk (variance), that interacts in complicated ways in a portfolio, which I'm sure you know.
silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).
You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.
silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).
You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.
> interacts in complicated ways in a portfolio, which I'm sure you know.
Yes. And assumptions about this are bound to break at the most inopportune moment, see e.g. AIG, which I already referred to. Read about the "copula model" disaster, as your statement indicates you are unaware of its details. https://en.wikipedia.org/wiki/David_X._Li#CDOs_and_Gaussian_...
> 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return)
This is true if and only if the long and short are in the same exchange, AND exchange rules allow netting long vs. short deterministically. Otherwise, you have counterparty risk. E.g., you can be long SPY and short SP contract (in equal underlying), which would theoretically mean your only exposure is interest rate changes (and sometimes not even that!)
However, since this is in different exchanges, it might happen that during a flash crash, your SPY position will be liquidated for insufficient margin at a low price, but then the price bounces back, and you've lost money on a perfectly hedged position.
OP's model (limiting exposure and assuming the worst, if I understand correctly) is not statistically efficient use of margin, but it's way better at actually managing risk than any statistical model.
Yes. And assumptions about this are bound to break at the most inopportune moment, see e.g. AIG, which I already referred to. Read about the "copula model" disaster, as your statement indicates you are unaware of its details. https://en.wikipedia.org/wiki/David_X._Li#CDOs_and_Gaussian_...
> 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return)
This is true if and only if the long and short are in the same exchange, AND exchange rules allow netting long vs. short deterministically. Otherwise, you have counterparty risk. E.g., you can be long SPY and short SP contract (in equal underlying), which would theoretically mean your only exposure is interest rate changes (and sometimes not even that!)
However, since this is in different exchanges, it might happen that during a flash crash, your SPY position will be liquidated for insufficient margin at a low price, but then the price bounces back, and you've lost money on a perfectly hedged position.
OP's model (limiting exposure and assuming the worst, if I understand correctly) is not statistically efficient use of margin, but it's way better at actually managing risk than any statistical model.
I don't know the exact definition of HFT but I did run my algorithm from a server collocated with my broker close to the exchange. I modelled lag time in simulation and not having it collocated certainly would have hurt.
Take a look at the VIX from 2009 until today and you'll understand why you stopped making money.
http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;
very interesting, a spike in sept-dec 2009 that roughly follows his PNL...
very interesting, a spike in sept-dec 2009 that roughly follows his PNL...
Without a ton of volatility, any homebrew HFT is going to lose to commissions and spread. Basically, he was trading in one of the few periods where is was possible to make money.
I've been considering trying HFT myself for a while. I'm competent with Machine Learning and am a Software Developer by day, so I can program and can sysadmin well enough to get something up and running without any trouble at all.
But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering.
It seems like the logical course of single-programmer HFT trading being:
- Find sample data
- Build your trading program using sample data
- When you're happy: connect to live API and set your trading program loose
- Iterate.
However, the first step and the third step seem like the ones which require the most research.
Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program?
Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.
But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering.
It seems like the logical course of single-programmer HFT trading being:
- Find sample data
- Build your trading program using sample data
- When you're happy: connect to live API and set your trading program loose
- Iterate.
However, the first step and the third step seem like the ones which require the most research.
Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program?
Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.
Nothing in HFT is free. People doing this for a living use precision time protocol in a colocated data center to build their own timestamps. However you can get started by buying ITCH, OpenBook, etc... data which has the full market depth feeds for the various exchanges. There are a couple of brokers out there specializing in the space.
If the easy part was building a working model either you got incredibly lucky or the model is wrong.
If the easy part was building a working model either you got incredibly lucky or the model is wrong.
Find sample data: https://quantquote.com/
API (Colo): http://www.lightspeedinstitutional.com/automated-trading/
Quality sample data and historical data to backtest is hard.
Symbols change, stocks split, dividends are announced etc.
Symbols change, stocks split, dividends are announced etc.
I run a 12 person HFT group in Denver. This is an excellent description of how an individual can bootstrap themselves into success. Great story, and nicely presented.
The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities."
Which doesn't really tell you much.
The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities."
Which doesn't really tell you much.
Thank you. I'm looking back through my code and there are really a lot of indicators. It would get pretty technical to explain them. They are all explainable it's just that each one corresponds to slightly different market conditions and I just didn't want to get into it.
Fair enough. Most people considering trying this probably have a few ideas for indicators.
But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.
But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.
Yes, I would find this very interesting. Doesn't matter if the indicator is now defunct.
Thanks for an excellent post. I would like to see any one indicator explained in detail as well. I am sure it would be of interest to a lot of people.
How well are you guys doing? How much money did you make?
There is a coursera course called "Computational Investing, Part I" that I am taking that aims to build a market trading simulator to test a trading model. It just started so it's not too late to join.
https://www.coursera.org/course/compinvesting1
https://www.coursera.org/course/compinvesting1
A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems.
Worst case he runs out of capital over a period of weeks.
He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.
Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).
In the short term however (sub-decade) - they can't price jack.
Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.
There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).
Both work. Both work well. And will continue to do so as long as markets exist.
Worst case he runs out of capital over a period of weeks.
He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.
Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).
In the short term however (sub-decade) - they can't price jack.
Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.
There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).
Both work. Both work well. And will continue to do so as long as markets exist.
This reminds me of an AMA from a few years ago. Really interesting if you are into stuff like this:
http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...
http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...
thanks for posting hadn't seen that
Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.
CourseTalk. CourseTalk! http://coursetalk.org .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.
By the way, no offense meant by the advertising thing. I enjoyed the article. Although, as a technical person, would've enjoyed more details on the code and algorithms.
This article is missing a crucial piece of data: what was the initial investment.. earning 500k with 10k initial investment is genious, with 10M initial investment it's just another year on the stock market.
This is a different style of trading than what investors do. He said that he was never more than a few contracts in. A single contract needs $10K day trading margin usually (depends on time frame and specific contract, but it's a reasonable estimate). Therefore, if he was never more than 10 contracts long or short at the same time, the risk was of $100K.
Making $500K on $100K over 6 months is good business.
Making $500K on $100K over 6 months is good business.
You are correct. I think I had to have $10-30k in my account. I was extremely low risk so they weren't concerned.
Especially considering that from it's lowest point in 2009 through the end of 2010, the S&P500 rose by 71%. Need to know what the starting capital was to be able to figure out if his return beat the market.
The charts show he was trading between Jun 2009 and Oct 2010. How much of his gains could be attributed to the market recovery in general? The Dow went from about 7000 to 11000, the Russell from about 600 to 800.
Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.
Longs and shorts were 50/50 and my program showed no preference for up or down days. High volatility and high volume was what it liked.
I guess the real question is: what was your alpha in that timeframe?
I'm probably showing my ignorance here but what do you mean by alpha? And how is it quantified?
Alpha is how much excess return you had over the market (or risk free return(
E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return]
If you made 20% when market was up 10%, you have positive alpha.
That is why everyone in the investment community is 'seeking alpha'.
E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return]
If you made 20% when market was up 10%, you have positive alpha.
That is why everyone in the investment community is 'seeking alpha'.
You don't need a bias to accidentally make money when the market is overall moving up, do you? Picking stocks by throwing darts while blindfolded will, on average, make you money in a market that's moving up.
Great point, here is a chart of the Russel 2000, incredible growth during that period.
http://finance.yahoo.com/q/bc?s=%5ERUT&t=5y&l=on&...
That's about 50% gain. Unless he put in $1m that doesn't completely explain his ROI.
Considering he was making 1000-4000 trades per day, I assume he put more than $1M. Trade fees alone would eat a huge chunk of money. The cheapest trade I can find is $3.95 (optionshouse.com).
So the fees will be $3950 - $15,800 per day.
$15,800 - $79,000 per week.
So the fees will be $3950 - $15,800 per day.
$15,800 - $79,000 per week.
He was trading futures. You need $10K in margin per future. He states somewhere that he started with $30K or so, and I estimate his "margin at risk" at any point was $100K or less.
And he surely wasn't paying retail prices - you can go down to around $0.50/future if you know what you are doing.
(In general, futures have 1:5 to 1:20 leverage)
And he surely wasn't paying retail prices - you can go down to around $0.50/future if you know what you are doing.
(In general, futures have 1:5 to 1:20 leverage)
Ok, at $0.5/future, the fees are
$500 - 2,000 per day
$2,500 - 10,000 per week
$130,000 - $520,000 per year
That's just the fees. He must make that much just to break even. If you can make that much starting with $30K, you're a supermotherfucking genius.
$500 - 2,000 per day
$2,500 - 10,000 per week
$130,000 - $520,000 per year
That's just the fees. He must make that much just to break even. If you can make that much starting with $30K, you're a supermotherfucking genius.
You should look more deeply into how these things work. Let's take the DAX Futures for example (which he was trading). All numbers are in EUR.
One "tick" (minimal movement) is worth 12.5 EUR. At volume you pay 0.5 EUR, IIRC, but let's assume you pay 1 EUR in fees, everything included.
If you bought and sold at the same price, you lost 1 EUR/trade. This is the cost of business.
If you bought, and sold after a favorable 1 tick movement, (e.g. bought at 4013.0 and sold at 4013.5), you're 10.5 EUR richer - 12.5 on the difference, minus 1 for each trade (one buy, one sell).
If you bought and sold after an unfavorable one tick movement (e.g. bought at 4013.5 and sold at 4013.0), you're 14.5 EUR poorer - 12.5 on the difference, and 1 for each trade (one buy one sell).
OP averaged $2/trade over 200,000 trades; that means he had 2/3 right calls, and 1/3 wrong calls or so if he only traded dax and only had 1 tick moves.
He was very smart, but you're looking at it wrong - the fees are the cost of doing business, much like salaries are the cost of producing software. In finance, you rarely care about revenue or "notional" (which can easily run into the trillions per year for a small trader - for ~1 eur, you get 75,000 eur in notional value on the dax).
You just roll the fees up-front into your choices when thinking about it, and it all makes much more sense.
(Not trying to take away from OPs very commendable achievement - just trying to give the common perspective on how to view this)
One "tick" (minimal movement) is worth 12.5 EUR. At volume you pay 0.5 EUR, IIRC, but let's assume you pay 1 EUR in fees, everything included.
If you bought and sold at the same price, you lost 1 EUR/trade. This is the cost of business.
If you bought, and sold after a favorable 1 tick movement, (e.g. bought at 4013.0 and sold at 4013.5), you're 10.5 EUR richer - 12.5 on the difference, minus 1 for each trade (one buy, one sell).
If you bought and sold after an unfavorable one tick movement (e.g. bought at 4013.5 and sold at 4013.0), you're 14.5 EUR poorer - 12.5 on the difference, and 1 for each trade (one buy one sell).
OP averaged $2/trade over 200,000 trades; that means he had 2/3 right calls, and 1/3 wrong calls or so if he only traded dax and only had 1 tick moves.
He was very smart, but you're looking at it wrong - the fees are the cost of doing business, much like salaries are the cost of producing software. In finance, you rarely care about revenue or "notional" (which can easily run into the trillions per year for a small trader - for ~1 eur, you get 75,000 eur in notional value on the dax).
You just roll the fees up-front into your choices when thinking about it, and it all makes much more sense.
(Not trying to take away from OPs very commendable achievement - just trying to give the common perspective on how to view this)
Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster.
The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.
The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.
There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck.
Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.
Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.
I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.
Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.
Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.
I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.
agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).
If anyone has questions for me happy to answer as best I can.
I have one question:
Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?
EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?
Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?
EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?
I tried at one point to do it with Ninjatrader. The programming skills for the trading software is not complicated. What is complicated is tweaking it so it will make money, there are tons of indicators out there and many people have tried this with neural networks and the like. A lot of effort is put into it. I personally thought there were smarter people than me who barely had an edge. I honestly didn't think it was within the grasp of a single programmer nowadays, but this author has proved me wrong. He of course has much more sophisticated algorithms than what I was attempting.
> You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible.
A profitable predictor is a much, much harder problem.
At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough.
Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.
A profitable predictor is a much, much harder problem.
At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough.
Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.
At a place like Goldman Sachs, you don't need quants or predictors. You call up the CEO of a company you want to post record profits, and you tell them if they don't do absolutely desperate, self-destructive things (screwing employees and customers for immediate gains), you will crash their stock and destroy their entire company. That CEO then has a choice. Be the one who killed the company, or be the one who kept it running for a few more weeks and delivered a record quarter that made Goldman Sachs happy.
Once you have the assets and capacity to actively manipulate the price of any stock at will, the market is a VERY different animal and no longer need to be understood at all. You simply force its hand.
Once you have the assets and capacity to actively manipulate the price of any stock at will, the market is a VERY different animal and no longer need to be understood at all. You simply force its hand.
>Why doesn't every hacker do this to make extra money?
I wondered the same thing and tried to answer it for myself not that long ago.
In short, it's hard, time-consuming, stressful and costly.
1) Your code has to work going forward. Coming up with something that works in backtesting is easy, doing it moving forward while staying within your risk envelope and considering all the associated costs is hard.
2) Closely related to #1. Whatever you put together is surely going to need plenty adaptation and oversight. It's a very fluid problem, you're just one player among countless others. How much time do you have in the day? Where is your capital coming from? For most people, the source of capital is a job that they probably can't ignore while fiddling with a trading program that would have to be remarkably successful to replace that income.
3) Everything costs money. You need a certain amount of capital to start with and there are all sorts of running costs. From the basic costs of execution, to market data and eventually co-location if you get that far. The fact that the OP kicked off with an amount that's essentially the barest minimum for any kind active trading is exceptional.
4) Can you maintain discipline and continue executing on all of the above while losing money? How many people struggle to simply get up on time, control their desire to check a particular website or to get over and move on from some frustration? Even if there's automation involved, trading will test your mental fortitude.
I wondered the same thing and tried to answer it for myself not that long ago.
In short, it's hard, time-consuming, stressful and costly.
1) Your code has to work going forward. Coming up with something that works in backtesting is easy, doing it moving forward while staying within your risk envelope and considering all the associated costs is hard.
2) Closely related to #1. Whatever you put together is surely going to need plenty adaptation and oversight. It's a very fluid problem, you're just one player among countless others. How much time do you have in the day? Where is your capital coming from? For most people, the source of capital is a job that they probably can't ignore while fiddling with a trading program that would have to be remarkably successful to replace that income.
3) Everything costs money. You need a certain amount of capital to start with and there are all sorts of running costs. From the basic costs of execution, to market data and eventually co-location if you get that far. The fact that the OP kicked off with an amount that's essentially the barest minimum for any kind active trading is exceptional.
4) Can you maintain discipline and continue executing on all of the above while losing money? How many people struggle to simply get up on time, control their desire to check a particular website or to get over and move on from some frustration? Even if there's automation involved, trading will test your mental fortitude.
"Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?"
I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.
I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.
1.)Getting a server close to the exchange
2.)Finding a good predictor. Predicting up or down is easy on paper. Finding a predictor that a.)beats the spread, b.)factors in lag time to execution, and c.)factors in commission, is hard.
3.)Market regime changes. What works a few months ago isn't guaranteed to work now.
2.)Finding a good predictor. Predicting up or down is easy on paper. Finding a predictor that a.)beats the spread, b.)factors in lag time to execution, and c.)factors in commission, is hard.
3.)Market regime changes. What works a few months ago isn't guaranteed to work now.
Because it's gambling. Some are better gamblers than others, but no individual can consistently have more ups than downs over a period of years.
"I was making a lot of money but now I've stopped" is the same thing as "I was lucky until I wasn't". Making a living by gambling pretty much sucks, which is why most hackers don't do it.
(You'd think that something as complex as markets would attract hackers trying to "figure it out". The problem is that due to the changing nature of the other participants, all hacks are temporary. Makes for great blog posts, but not a long term strategy.)
"I was making a lot of money but now I've stopped" is the same thing as "I was lucky until I wasn't". Making a living by gambling pretty much sucks, which is why most hackers don't do it.
(You'd think that something as complex as markets would attract hackers trying to "figure it out". The problem is that due to the changing nature of the other participants, all hacks are temporary. Makes for great blog posts, but not a long term strategy.)
You are correct that no individual can. But firms can. And some do. Its not gambling. Its not heading down to the roulette wheel. If someone puts on millions of trades and wins a statistically significant portion of them you would have to say its not gambling. Its trading with a statistical edge.
To really be able to do this with a lot of success, you need to be able to do intraday trades, which would make you a day trader under SEC rules, which also requires $25,000 in your brokerage account.
You have to have the capital for the server and access to the data feeds, as well as time to burn. Also, you need to find finance interesting enough to spend time with it.
Wouldn't the term "Statistical Arbitrage" be a more apt description of what you were doing?
If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".
If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".
It wasn't anything over the internet. I had a server rented at my broker who were situated close to the exchanges in Chicago and had direct lines. For sure I was not the fastest but only behind by a couple milliseconds perhaps.
Ok, that wasn't clear to me.
You talked about programming hotkeys (and then automating the hotkeys) so I assumed this was running on your desktop.
You talked about programming hotkeys (and then automating the hotkeys) so I assumed this was running on your desktop.
oh right.. well in the very beginning it was.
How much does that cost?
Thanks for the post. Is this the software you used?
http://www.ctsfutures.com/t4.aspx
Would be even more awesome if you could share some code on github.
http://www.ctsfutures.com/t4.aspx
Would be even more awesome if you could share some code on github.
Yes, however, I forgot to mention that pretty early on I converted my program to use an API from https://www.tradingtechnologies.com. It's odd, but I can't remember exactly why. I think it was simply because I found a broker who could offer me a lower commission rate and they only supported TT. With regard to posting code yes I may do that. We'll see.
XTAPI, or did you write something to talk to a FIX gateway?
(former TT employee)
(former TT employee)
I'm pretty sure it was XTAPI. Is that the simpler one? I used the simpler one. Whatever the case.. the code that dealt with the API was trivial compared to the rest of my program. So good work!
>With regard to posting code yes I may do that. We'll see.
What does it depend on?
What does it depend on?
If I was correct in guessing that he used XTAPI (via X_TRADER Pro), IIRC that costs around $1k/mo depending on who you go through for the front end license. I was only a developer so I'm not sure what licensing costs are for gateways, but those are the servers you connect to in order to place orders on the exchanges. He was trading the Russel on ICE, and DAX on Xetra (maybe Eurex?), so he would have needed two gateways.
If he posts the code, you're a long way from running it.
If he posts the code, you're a long way from running it.
My fixed costs were $2000/month. I think it was $1800 for TT plus $200 server rent so damn.. I was getting overcharged $800!! oh well :)
Your full package of $2k/mo sounds about right from what I remember, after you factor in server, access, licenses.
Do you think you got lucky or that your skill as a trader made you this money? Or rather, which of these do you think mattered most?
1) Being connected to friends who taught me a scalping style of day trading
2) Being really good at designing algorithms.
3) I only wish I could have started automated trading back in 2001
Do you know of any resources/books to learn trading/HFT in general?
How do (did) you cope with increased stress level? Trading futures, especially in a an automated way, can easily drain your margin unless the algorithm is really well tested for edge cases.
My automated program was much less stressful than trading manually. The best was going to Hawaii, waking up, and having the entire day done. I was making like 6k every day on that vacation. The best!! :)
why do you think your technique stopped working over time?
I wonder if running your program nowadays can have same successful results as two years ago.. anyone is going to try? :)
Don't do this with your own money. Found a startup building HFT tools, and then raise money for it, and use other people's money to test your tools.
If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do.
This is risk management.
If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do.
This is risk management.
I started the infrastructure for this kind of thing a while ago. It's BSD licensed.
It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems.
Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at:
http://home.rbcarleton.com/rbc/software/smalltalk/VSA/
I won't try to advertise it as a complete solution, but it might be the start of one.
It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems.
Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at:
http://home.rbcarleton.com/rbc/software/smalltalk/VSA/
I won't try to advertise it as a complete solution, but it might be the start of one.
Wait. This is not HFT. There's a huge difference between automated and high frequency trading. What he does is only automated scalping at best (or at the fastest).
Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders.
HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.
Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders.
HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.
quite true. but HFT has become a buzzword like "the cloud" and even many financial industry specialists claim to do HFT but in fact they are just market making. auto-scalping or even hedge funds that day trade with robots are not HFT.
this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y
even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns.
it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.
this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y
even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns.
it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.
Is there a way to do this with Python or Ruby? I could just as well program this in C#, but I have a friend who can code a little, but doesn't really need everything in C# to do what he wants. The value add of offering the simulator, including the taking into account the bid/ask prices and a stochastic model for latency. Combine this with a web based code editor and easy hosting, and I think this would be a viable product.
Nope, there is absolutely no way to do this with python or ruby. :)
Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks:
http://www.rapidquant.com/features
A C++ library with python bindings: http://quantlib.org/index.shtml
And a low level library for data analysis: http://pandas.pydata.org
Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks:
http://www.rapidquant.com/features
A C++ library with python bindings: http://quantlib.org/index.shtml
And a low level library for data analysis: http://pandas.pydata.org
If you use their library/SDK/Framework.... how do you actually trade, do you still need a brokerage? Sorry a bit new to this field.
Yep, almost any of them have an API these days. Your software would make the list of trades which is uploaded and executed...
Oh, I forgot about another backtesting framework for python. It has a built in IDE as well.
http://www.quantopian.com
Oh, I forgot about another backtesting framework for python. It has a built in IDE as well.
http://www.quantopian.com
I work at quantopian. We are working on broker integration, so you will be able to trade algos you develop live.
Our backtesting engine, Zipline, is opensource - https://github.com/quantopian/zipline
Zipline was unveiled at PyData NYC, and the presentation materials are here: https://app.quantopian.com/posts/hello-from-pydata
Our backtesting engine, Zipline, is opensource - https://github.com/quantopian/zipline
Zipline was unveiled at PyData NYC, and the presentation materials are here: https://app.quantopian.com/posts/hello-from-pydata
Sure, his program could easily have been done in almost any language. He wasn't competing on speed, which might have excluded languages like Python.
Looking at your first chart there, is there a reason (other than market conditions) you were making significantly more at the end of '09 than mid '10?
To be honest I don't know exactly what happened. My theory is that over time more and more market participants started integrating the types of analysis I was doing which rendered my program ineffectual. It's a pretty normal pattern that there is some inefficiency in the market and over time it disappears.
Being a machine learning program, how much of it did you tell it to forget?
Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between?
(I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)
Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between?
(I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)
This. If the 2009 model worked well, did you try letting your algorithm to 'forget' the 2010 data and see if the model worked better?
It's possible that your algorithm is sensitive to market volatility.
Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.
I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.
I am curious as to exactly why the profitability decreased steadily and rather rapidly all the way to ~zero. The article doesn't seem to expound on that unless I missed something.
Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.
Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.
another comment explained: http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;
I don't think anyone was adapting to what I was doing in particular but rather simply adapting to the opportunities in the market.
Thanks. Makes sense... I guess those opportunities tend to get ironed out rather fast then.
Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingly responsible for his returns, despite the frequency of trades).
We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with?
Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.
We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with?
Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.
1. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k.
2. Author has stated elsewhere than he began with $10k in seed capital.
3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number.
4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped.
Edited to fix numbers.
2. Author has stated elsewhere than he began with $10k in seed capital.
3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number.
4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped.
Edited to fix numbers.
1. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns.
2. ok.
3. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant.
4. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.
The high point of my trading was October 2009 when I made almost 100k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. Unfortunately by this point I guess I’d implemented all my best ideas because nothing I tried seemed to help much.
It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.
It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.
Would you be able to open source any of the code behind your trading system? Maybe not the "secret sauce", but it would be interesting to see how you processed the data feeds, modeled the data, entered orders, etc.
I certainly could open source it. I may just wait a bit on the off chance that somebody wants to purchase it.
Relax, nobody is buying your program.
Other than sheer luck, the most plausible explanation for your diminishing returns is that you found a strategy that worked _at that point in time_, other people copied it (starting with your broking firm), and as that strategy became more common your ability to make money disappeared.
I work in the industry, this happens all the time. Trading strategies have a shorter half life than you may think.
Other than sheer luck, the most plausible explanation for your diminishing returns is that you found a strategy that worked _at that point in time_, other people copied it (starting with your broking firm), and as that strategy became more common your ability to make money disappeared.
I work in the industry, this happens all the time. Trading strategies have a shorter half life than you may think.
that would require some proof that it works today.
Not really.
A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights.
If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.
A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights.
If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.
Yes i'm pretty sure it wouldn't work today. By buying the code I realistically mean hiring me to work for them based on what I achieved.
The guys you want to work for (2sigma, RenTec, Jane Street, Susquehanna, ...) are unlikely to call you up as a result of this blog post / hacker news exposure/discussion.
If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of-friends are infinitely more successful hiring strategies, for both sides.
(Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails).
the Nuclear Phynance message board is probably a better place to look for business offers.
If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of-friends are infinitely more successful hiring strategies, for both sides.
(Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails).
the Nuclear Phynance message board is probably a better place to look for business offers.
1. Hook
2. Line
3. ...
Profit!
2. Line
3. ...
Profit!
[deleted]
Trust me, you earned that much because of your luck. Otherwise Andrew Ng would have partnered with another finance professor and they would have been the richest people on earth!! Imagine trading with their expert systems on global markets.
I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!
I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!
I like to trade Forex using mql4, any suggestion? Thanks.
You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?
To be honest I'd given up and moved on to other stuff well before I actually shut down the program.
Thanks for the post, it's very inspirational.
Could you explain this part, specifically what do you mean by "bucket"?
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
Could you explain this part, specifically what do you mean by "bucket"?
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
Thanks for the post, it's very inspirational.
Could you explain this part, specifically what do you mean by "bucket".
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
Could you explain this part, specifically what do you mean by "bucket".
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
This is an activity that adds no value to our world.
It is pretty clear from his own graph that this stopped working in october '10. which was an eternity ago in terms of financial markets. algorithmic trading has increased manifold since then, so finding another arbitrage opportunity like he did is only going to be more difficult.
Cool article but I hope people don't start trying to follow this path. Ask yourself -- why did he stop?
yes. and yes.
I don't understand trading enough to even understand many of the terms in the article, however I'm curious to one thing: is it possible that a program could be written specifically to exploit yours? And/or is that a potential reason it became unprofitable?
Have you ever thought of making a trading system that would buy tons of stock when a flash crash happens? It is going to happen again. If your system is ready and you buy before they shut the market down or roll back orders you could make a hefty profit.
The difficulty is in identifying what is a 'flash crash' (i.e. a temporary downward blip in prices caused by computer or human error) and what is a genuine downward price movement.
If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses.
Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses.
Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
> The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash).
That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.
This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash).
That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.
Nitpick: When exchanges have busted trades there is a "at or below $XX.XX" condition as well as the "between XX:XX and YY:YY" condition.
In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood).
Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.
In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood).
Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.
The prototypical example of why a tax on financial transactions is urgently needed.
Excellent article on this posted here: http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-mo...
Isn't profit meaningless without knowing initial investment? I read the first few paragraphs and got bored. Why not say upfront what the bankroll was to start?
edit: I found what I was looking for in the comments.
edit: I found what I was looking for in the comments.
Just to save someone else from having to look for it like we both did, his initial investment was
$10k loan from dad. Built it up to 30k trading manually before my automated program went live.@Author ? Can you create an online course and teach us all? Thanks
Hopefully not buried too deep, but any books recommended for getting into day-trading, either manual, or algorithmic? Kindle preferred, but definitely not the deciding factor.
Might've been what it was a couple of years ago but this post, dated today, is the perfect advertisement for the author's current business.
Well done!
PS: no sarcasm intended, it truly is an excellent advert.
Well done!
PS: no sarcasm intended, it truly is an excellent advert.
I am skeptical for two reasons:
1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it.
There should be more to this story.
1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it.
There should be more to this story.
Can I recommend that you read the article and you will find therein the answers you seek!
Out of interest, how much capital did you start off with?
$10k loan from dad. Built it up to 30k trading manually before my automated program went live.
What were your average transaction costs per trade? 1000 to 4000 trades per day, lets say 2500 on average, translates to about 625K transactions per year. I assume you did not have to pay something like ETrades 3$ commission per future contract, which would result in almost 2M of fees per year.
If you're interested in working on this kind of stuff in the San Francisco area, send a resume to Headlands Technologies. [email protected]
What contracts did you trade? Looks like you did futures contracts? Did you look into forex at all, or was this strictly equites.
I was trading stock market index futures: Russell 2000, DAX, and a bit of Nasdaq. I experimented with other stuff as well.
This might be a dumb question but... what system does your algorithm interface with?
Last time I checked, NASDAQ and NYSE don't exactly publish API's
Last time I checked, NASDAQ and NYSE don't exactly publish API's
Looks like it's written in C# - is that correct?
It stated in the article it was C#.
"In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."
"In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."
Yes, that's what I was going off of. I was curious about that aspect of it, if it had stayed C# or what.
Actually I believe when I switched from T4 to TT it was C++ I was using.
Doesn't the post explicitly say it was?
Just because you CAN do it doesn't mean you SHOULD. Even if you don't think of it as "gambling", you're still taking in tons of money without providing any tangible benefit to society.
If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.
If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.
I didn't lose any sleep but I do indeed hope to contribute more to society with future projects.
"Since I'm no longer running my [half million dollar] program I’m happy to tell all"
This does not ring true.
This does not ring true.
I could do the same with poker. Or I could lose it all. That variance thing is a bitch.
Very cool article. Gave a detail explanation of the ENTIRE process. Thanks!
I'm glad to see a healthy respect for investment among the hacker community. It's traders like this who commit to nearly a full 10 seconds of ownership that are the backbone of economic growth for this country.
I think there is a healthy respect for computer code that can let loose upon the world and make money.
Your missing the point that owning part of a company for 10 seconds brings zero social value to the economy. Which I also tend to agree with. Slightly off-topic from the point of the article, which was that he achieved it without backing of a fairly large institution.
I didn't miss that point - I just chose not to address it. Lots of things add zero (or negative) social value to the economy including Farmville or the gazillionth Instagram clone. That doesn't mean they're not worth doing.
And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.
And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.
True it adds no social value, but it is arguably the ultimate hacker's game :) You can actually add a little social value by placing only orders that increase liquidity; those where someone else "takes out" your standing offer or bid. That and the fact that your profits are taxed as ordinary income in the US for equities at least.
how is it "machine learning"? A few if-else doesn't make it "machine learning", or does it? - @curationary
It takes data (last 4 weeks of market update data) and learns how to predict price movements and how to make profitable trades.
I'm sceptical that HFT is good for the public. What did you do to ensure your system wouldn't make a flash crash worse?
Why is it his business to defend other market participants?
I didn't say it was. But everyone should be in the business of not being evil and not exasperating the problems of others. In trading that means not contributing to flash crashes.
I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.
I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.
How much did you start with?
Amazing, love your project
A rising tide lifts all boats. "Price move predictions" sounds as effective as tea leaf reading.
And nothing of value was created.
Just kidding. It was a fascinating article. Thanks for sharing.
Just kidding. It was a fascinating article. Thanks for sharing.
[deleted]
of course no firm would respond to his noob low yield model.
The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.
Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.
Kudos though for taking on the task of learning how to code and making money with ML.
The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.
Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.
Kudos though for taking on the task of learning how to code and making money with ML.
This is such an odd comment. What motivates you to write such a comment? And what makes my system amateur and noob in your eyes?
Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it).
He trade other people's money, using other people's (probably employer's) account and resources, I suppose.
His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.)
The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.)
But this is just my guess.
He trade other people's money, using other people's (probably employer's) account and resources, I suppose.
His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.)
The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.)
But this is just my guess.
This is not even close to an accurate summary. He never stated that he had any employer backing, and he wasn't collecting market making fees. In fact he was paying brokerage fees which is the exact opposite.
He used his own money to test his algos? Come on.
Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system.
This is why my guess is that he was an employee.
Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system.
This is why my guess is that he was an employee.
An employee of what? He states above that he was paying roughly $200/month for a server and $1800/month for the software/data connections to his broker.
Prior to setting up my automated trading program I’d had 2 years experience as a “manual” day trader. - mom's and pap's 401k?
No, he said he backtested them against data he collected from the feeds. He paid for the feeds just like everyone else does with his own money.
For me it all looks like a bad-written fiction, and the source is just a blog post.
And that's the beauty of it - take it or leave it.
Basically this is a guy who does what many other have done, look no further than elitetrader.com, but smart enough to add "machine learning" to his story headline, which is the hottest buzz term on the street this year, everyone and his mom talks about machine learning last I checked.
So, it is just plain SEO for his startup - a primitive on-line course aggregator.
While this was quite fascinating, I couldn't see this form of trading as anything but a zero-sum game. Some players win, the other lose, like in any other game.
Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?
Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?
Okay, enough with the downvotes. What's wrong with my question? I didn't mean to bash a field about which I know next to nothing. My question is genuine.
Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had.
Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth.
But.
The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum.
Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had.
Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth.
But.
The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum.
Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair.
Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.
I refine my position here: http://news.ycombinator.com/item?id=4752742
I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.
I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.
It doesn't really matter why anyone is in the market. The fact is that they are posting prices to the limit order book, and in doing so providing liquidity and accepting risk and providing a service. The only way the OP could get executed passively is by offering a price equal to or better than the market price, yes? So someone got a better price (or more liquidity at the prevailing price) by him being there.
There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.
Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.
The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.
It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.
Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.
There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.
Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.
The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.
It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.
Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.
Okay, thanks. A few minor points.
I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.
I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.
We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.
By itself, closing a deal doesn't mean you provided value. It means the other party thinks you provided value to them. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)
If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray you are not the sucker. Locally, it's totally zero-sum.
Now maybe the whole system facilitates real transactions (with non-speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.
I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.
I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.
We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.
By itself, closing a deal doesn't mean you provided value. It means the other party thinks you provided value to them. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)
If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray you are not the sucker. Locally, it's totally zero-sum.
Now maybe the whole system facilitates real transactions (with non-speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.
You're still missing it. Every single additional limit order in the market creates real value. If I want to buy 500 shares of company A, and one market participant is offering 250 at $10. Another is offering 250 at $11. It is going to cost me 250 x 10 + 250 x 11 = $5250. Now, the "short term speculator" you are complaining about shows up and puts his tiny little order in, he is offering 1 share at $10. Now the equation if you want to execute is 251 x 10 + 249 x 11 = $5249. He/she just saved you $1. One real dollar. As someone who wants to get a transaction done on a market, more people posting bids and offers cannot possibly hurt you. Only help you. You have the complete freedom to decide to meet their offered price, to cross the spread, to pick up that carton of milk at the 7-11 and walk to the till and make the transaction.
Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.
In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.
How wide are bid/offer spreads these days? How can that possibly be a bad thing?
Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.
In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.
How wide are bid/offer spreads these days? How can that possibly be a bad thing?
(Edit: "limit order" looks like technical jargon. If so, I don't know what it means. Maybe you didn't say what I thought you said)
In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.
On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?
My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non-speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it harder, for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?
Now maybe a good speculator tends to provide a good service to non-speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. (Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)
In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.
On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?
My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non-speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it harder, for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?
Now maybe a good speculator tends to provide a good service to non-speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. (Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)
Addendum: rereading your posts, I see that you defined "limit order". So now I understand what you said.
> Every single additional limit order in the market creates real value.
Bullshit. You don't cover the main case, where 2 speculators make a deal. Show me how value is created (not just transferred) in this case.
> Every single additional limit order in the market creates real value.
Bullshit. You don't cover the main case, where 2 speculators make a deal. Show me how value is created (not just transferred) in this case.
I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side of the trade, that I'll have to pay an enormous spread or be left holding a position I want to be out of. An international corporation can be more confident of planning their cashflows knowing that the FX market will always be liquid, and will be quoting at most 2 ticks wide in the majors 24 hours a day every day. This is the oil of an economy, the freedom to allocate capital when you want, where you want and pay the smallest possible friction costs to do so.
Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.
" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.
To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.
Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.
" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.
To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.
My definition of value is simple: something is valuable when it manages to raise people's utility functions (right now, that's about as rigorous as I can be). A trade that happens to be closed by a plain old investor is valuable. If it's another speculator, however, the value is zero. Only consequences ultimately matter.
But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.
> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?
The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.
When Pareto Optimum and Nash Equilibrium are at odds, life sucks.
But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.
> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?
The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.
When Pareto Optimum and Nash Equilibrium are at odds, life sucks.
It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time.
I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money.
So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.