Algorithmic surrealism: A slow-motion guide to high-frequency trading(suitpossum.blogspot.com)
suitpossum.blogspot.com
Algorithmic surrealism: A slow-motion guide to high-frequency trading
http://suitpossum.blogspot.com/2015/06/high-frequency-trading-guide.html
35 comments
> This is one of the better overviews of HFT I have come across.
Really, I found it to be long on words and almost completely devoid of any content what so ever.
I mean this in all seriousness, What specifically can one point to in this article that has any actual content and isnt' just a collection of links to other sites for common definitions.
At no point did the article actually say how HFT firms design their systems or even how their algorithms might work.
> t hits on all the points I consider to be important (having worked in HFT for over six years)
I too am in the industry:)
Really, I found it to be long on words and almost completely devoid of any content what so ever.
I mean this in all seriousness, What specifically can one point to in this article that has any actual content and isnt' just a collection of links to other sites for common definitions.
At no point did the article actually say how HFT firms design their systems or even how their algorithms might work.
> t hits on all the points I consider to be important (having worked in HFT for over six years)
I too am in the industry:)
I concur with this sentiment. Especially:
> Really, I found it to be long on words and almost completely devoid of any content what so ever.
But I don't work in the financial industry. Still, the article offered nothing new to me
> Really, I found it to be long on words and almost completely devoid of any content what so ever.
But I don't work in the financial industry. Still, the article offered nothing new to me
It's not technical, that's true, but I really appreciate the philosophical take. It describes the foundation of everything and how the industry works from a high level very well. It was fun for me to read.
> the perception of constant "war".
It's not a perception though. Short-term trading opportunities are zero-sum. To win trades you either must undercut your competitor by trading for less edge, or beat him to realizing the opportunity by either predicting better or getting faster, usually some combination of all those. All of that means effectively working your ass off to take food off another man's table before he takes it from yours. That probably happened less in the pit days because there was a camaraderie between the locals facing off against paper. Nowadays I never see the guy I'm trying to beat into the queue. We aren't going out for beers. It's just business.
I'd say being successful in this business feels less like war and more like riding a bear. You never get to rest and there's a constant fear that someone, somewhere is hungrier and smarter than you are, plus the bear might snap and eat you.
>Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas.
The better firms tend to hire people out of school and train them up so they can do novel research. It's a smarter long-term play. I agree that few people are up to the task, even the best and brightest. It does amuse me that some guy a couple blocks away probably runs a super similar model with a different cool code-name and we battle each other for years in Sisyphean fashion. I'll never even know his name. A zipless fuck.
I don't like the author's characterization of HFTs as "parasites." Someone can be a middle-man and still add value. Market users don't need to trade with HFT dealers. They can easily work their own orders or wait around in a block trading facility hoping to find a natural match instead. For most of them, it's a bad trade-off. Even though the HFTs make money on average, trading with them costs less than becoming an expert in market microstructure, finding good hedges, facing non-execution risk, etc. CarMax makes money when they buy/sell your car in the same way, but it's usually more convenient or cheaper than doing it yourself.
In a way, HFT is like insurance in that they only provide a valuable service when they lose, but overall people prefer paying for certainty. Almost every winning trade I make is one where two natural counter parties could have found one another had they waited a short period of time or were in the right place/product, but I lose on a huge % of trades as well. If HFTs weren't in a competitive environment or if one firm had a clairvoyant pricing model, this would be problematic but unfortunately for me, that's not the case.
It's not a perception though. Short-term trading opportunities are zero-sum. To win trades you either must undercut your competitor by trading for less edge, or beat him to realizing the opportunity by either predicting better or getting faster, usually some combination of all those. All of that means effectively working your ass off to take food off another man's table before he takes it from yours. That probably happened less in the pit days because there was a camaraderie between the locals facing off against paper. Nowadays I never see the guy I'm trying to beat into the queue. We aren't going out for beers. It's just business.
I'd say being successful in this business feels less like war and more like riding a bear. You never get to rest and there's a constant fear that someone, somewhere is hungrier and smarter than you are, plus the bear might snap and eat you.
>Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas.
The better firms tend to hire people out of school and train them up so they can do novel research. It's a smarter long-term play. I agree that few people are up to the task, even the best and brightest. It does amuse me that some guy a couple blocks away probably runs a super similar model with a different cool code-name and we battle each other for years in Sisyphean fashion. I'll never even know his name. A zipless fuck.
I don't like the author's characterization of HFTs as "parasites." Someone can be a middle-man and still add value. Market users don't need to trade with HFT dealers. They can easily work their own orders or wait around in a block trading facility hoping to find a natural match instead. For most of them, it's a bad trade-off. Even though the HFTs make money on average, trading with them costs less than becoming an expert in market microstructure, finding good hedges, facing non-execution risk, etc. CarMax makes money when they buy/sell your car in the same way, but it's usually more convenient or cheaper than doing it yourself.
In a way, HFT is like insurance in that they only provide a valuable service when they lose, but overall people prefer paying for certainty. Almost every winning trade I make is one where two natural counter parties could have found one another had they waited a short period of time or were in the right place/product, but I lose on a huge % of trades as well. If HFTs weren't in a competitive environment or if one firm had a clairvoyant pricing model, this would be problematic but unfortunately for me, that's not the case.
>The better firms tend to hire people out of school and train them up so they can do novel research. It's a smarter long-term play.
I completely agree with you there. There are a handful of firms that do this consistently and they are leading the pack.
> I don't like the author's characterization of HFTs as "parasites."
Also agree. I think it's a bit harsh.
I completely agree with you there. There are a handful of firms that do this consistently and they are leading the pack.
> I don't like the author's characterization of HFTs as "parasites."
Also agree. I think it's a bit harsh.
> To win trades you either must undercut your competitor by trading for less edge, or beat him to realizing the opportunity by either predicting better or getting faster, usually some combination of all those.
On the other hand, at least in theory, the rest of the market benefits from smaller spread (a lot), and faster execution time (a bit).
It's a shame about the sup-penny rule. (https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm...) Without it, we'd get more of the former and less of the latter.
On the other hand, at least in theory, the rest of the market benefits from smaller spread (a lot), and faster execution time (a bit).
It's a shame about the sup-penny rule. (https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm...) Without it, we'd get more of the former and less of the latter.
I trade in stocks that aren't constrained by the tick size. Latency still matters a lot.
I agree the rule isn't sensible though. Much of the fragmentation in the US is driven by exchanges with inverted rebate structures (maker pays, taker gets paid to hit/take) + midpoint trading that are effectively a back-door way to quote sub-penny in stocks where the true competitive spread should be less than 1c.
I agree the rule isn't sensible though. Much of the fragmentation in the US is driven by exchanges with inverted rebate structures (maker pays, taker gets paid to hit/take) + midpoint trading that are effectively a back-door way to quote sub-penny in stocks where the true competitive spread should be less than 1c.
> One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly.
So exactly like Silicon Valley?
So exactly like Silicon Valley?
Successful ideas in general work this way, but I am impressed with the SPEED in which this dispersion takes place in the trading world. All this brain and computing power cycles through ideas as quickly as possible to find new sources of income. It's a giant complex network trying to find an ever-changing optimal solution to the current market state. Quite incredible.
[deleted]
Having zero real knowledge of trading in any fashion at all, recently I've been wondering if there was a niche for "medium-speed trading." I know I can't get close to the exchanges, and I don't have the capital to hire experienced trader/engineers to develop the latest algorithms.
Basically, is there a slice of the pie in trading much faster than humans, but much slower than HST?
It's an academic exercise, but one I've been toying with.
Basically, is there a slice of the pie in trading much faster than humans, but much slower than HST?
It's an academic exercise, but one I've been toying with.
Yes, there’s a "frequency spectrum" of sorts and HFT is at the short end of it. Here, your models monitor and react to trading tick-by-tick, and execution speed is paramount. It’s also where you have the fiercest arms race for the fastest systems, colocation, FPGAs, etc. Next up comes what’s often called “statistical arbitrage,” where you have models that no longer look at tick-by-tick trading but may look at what happens at 30-second, 1-minute, or 5-minute windows. Here you have more interesting relations emerge between stocks and the market, e.g., what does IBM do relative to the tech index, or relative to MSFT, etc. (Such cross sectional relationships don’t seem to matter as much in HFT.) Actually stat-arb was the domain where the earliest statistical approaches such as "pairs trading" emerged. Next up come models that trade daily (or less frequently), and here you begin to see the long-short market-neutral relative-value type of approaches, where some quantitative [mutual] funds may operate. Next up will be the traditional mutual funds, and beyond that you have your Warren Buffett’s, etc.
One thing to keep in mind is that the higher your trading frequency, the smaller the price moves you can hope to capture, which limits how much capital you can deploy in your models. This is why HFT models are usually small in size but have high Sharpe ratios. As you reduce your trading frequency, you can expect to capture larger price movements and deploy more capital but you’ll also be exposed to more of the vicissitudes of the general market, so your Sharpe ratio will decline. Market participants usually carve themselves a happy spot on this frequency spectrum and stay there. I don’t know of any firm who is successful at every spot.
One thing to keep in mind is that the higher your trading frequency, the smaller the price moves you can hope to capture, which limits how much capital you can deploy in your models. This is why HFT models are usually small in size but have high Sharpe ratios. As you reduce your trading frequency, you can expect to capture larger price movements and deploy more capital but you’ll also be exposed to more of the vicissitudes of the general market, so your Sharpe ratio will decline. Market participants usually carve themselves a happy spot on this frequency spectrum and stay there. I don’t know of any firm who is successful at every spot.
I think you are misconstruing holding period or predictive horizon with latency sensitivity. Many HFTs are looking at statistical relationships like the ones you mention to compute a fair price for making markets. The only trades where HFTs hold positions sub-second on average are pure arbitrages. Like you mention, there simply isn't enough price movement within that timeframe to generate a profit.
All the inputs to their pricing change rapidly, so their order prices must change quickly as well, but they can end up carrying risk for long periods of time. The Australian regulator looked at HFT activity in their markets, mind you probably less sophisticated than US stocks, and found the average holding period was 42 minutes: http://tabbforum.com/opinions/hft-concerns-are-overstated
All the inputs to their pricing change rapidly, so their order prices must change quickly as well, but they can end up carrying risk for long periods of time. The Australian regulator looked at HFT activity in their markets, mind you probably less sophisticated than US stocks, and found the average holding period was 42 minutes: http://tabbforum.com/opinions/hft-concerns-are-overstated
I'm not sure the average holding period is a useful thing to measure. For eXample, if you are trading a spread it's how quickly you put on the second leg that matters, and who cares how long you hold the pair for. So, 42 minutes has nothing to do with it.
Yes. Quantopian.
Not HFT by any definition (unless your definition of HFT is on the order of minutes), but purely algorithmic and data-driven. You can backtest your algorithms with up to 13 years of historical data, live trade your algorithm with paper money, and even link your algorithm to a broker and trade your algorithms with real money.
It's the single most disruptive financial service I know of, and I've had tremendous success with it, even without linking a brokerage account (just checking it daily for trade signals in my super-low-frequency algorithm on paper money).
https://www.quantopian.com/
Not HFT by any definition (unless your definition of HFT is on the order of minutes), but purely algorithmic and data-driven. You can backtest your algorithms with up to 13 years of historical data, live trade your algorithm with paper money, and even link your algorithm to a broker and trade your algorithms with real money.
It's the single most disruptive financial service I know of, and I've had tremendous success with it, even without linking a brokerage account (just checking it daily for trade signals in my super-low-frequency algorithm on paper money).
https://www.quantopian.com/
Where's your value add, your USP? If you're not going to be the fastest then you can't make money as a market-neutral market maker. You have to be able to offer a better price, by predicting the future value better than other market participants.
For the "faster than a human, slower than HST" niche all I can think of is the traders who automatically react to press releases, twitter and the like. If you're getting data that no-one else has and it affects the price, it doesn't matter whether you're as fast as HST. (Of course if you get competitors who are doing the same thing you still have to be faster than them).
Beyond that most stuff happens at human speed - which doesn't mean you can't do market analysis with algorithms, competing with humans. But speed is always going to be a factor - even if we're talking about e.g. a multi-month analyst investigation that figures out that company X is really a massive fraud (there are trading firms who make their business figuring this stuff out), that still becomes worthless if your competitor finishes their investigation a day earlier than yours.
For the "faster than a human, slower than HST" niche all I can think of is the traders who automatically react to press releases, twitter and the like. If you're getting data that no-one else has and it affects the price, it doesn't matter whether you're as fast as HST. (Of course if you get competitors who are doing the same thing you still have to be faster than them).
Beyond that most stuff happens at human speed - which doesn't mean you can't do market analysis with algorithms, competing with humans. But speed is always going to be a factor - even if we're talking about e.g. a multi-month analyst investigation that figures out that company X is really a massive fraud (there are trading firms who make their business figuring this stuff out), that still becomes worthless if your competitor finishes their investigation a day earlier than yours.
Since people still make money trading manually, yes, there's room at all levels.
> "People are routinely worried about harmless things, and routinely completely unworried about incredibly harmful things."
God what a quote. I'm stealing this and using it everywhere I can. It basically sums up my entire attitude toward humanity.
God what a quote. I'm stealing this and using it everywhere I can. It basically sums up my entire attitude toward humanity.
Bad quote, I agree, but it does express how risk management in the HFT world is often short sighted.
> if you'd like to support my ongoing Creative Commons writing, please consider buying me a virtual beer.
In the spirit of the article's talk about financialization, I wonder if there's yet a way to buy the author virtual beer options?
In the spirit of the article's talk about financialization, I wonder if there's yet a way to buy the author virtual beer options?
It's only going to get worse (or better) depending on how you look at it. So many behavioral factors will be included to shake stops and squeeze shorts at just the right time to make huge profits from the bounce. It's part of the reason as a trader I'm now just chasing Momentum, ignoring most other signals as noise. So much so, I'm building an app that supports discovering these momentum breakouts. In the unlikely event anyone is interested - www.mometic.com
[deleted]
I have some experience with quantitative investment coding and so am frequently asked to apply for HFT jobs.
I don't pursue it because I came to regard the practice as unethical.
I don't pursue it because I came to regard the practice as unethical.
Care to explain why? (unethical)
Aside from stating your own skills, employability, and high standard of ethics, your post is lacking useful info about HFT.
Aside from stating your own skills, employability, and high standard of ethics, your post is lacking useful info about HFT.
I just spent 15 minutes Googling this guy because his site in his profile was interesting. He humblebrags about employability, but he's incapable of finding work. He also makes bomb threats: http://blog.up.co/2012/04/30/not-even-bmob-threats-could-det...
No, I don't make bomb threats. I was using Twitter to lecture the Portland Startup Weekend on engineering ethics.
I resigned in protest from an HMI/SCADA vendor because of the poor code quality of their product.
I resigned in protest from an HMI/SCADA vendor because of the poor code quality of their product.
I see. Thanks for your side of the story. You are a very interesting person! I'm going to read more of your site later.
Thank you.
From time to time I toy with the idea of suing that joker for libel, but the reason I don't is that I support the notion that the best response to hate speech is more speech.
From time to time I toy with the idea of suing that joker for libel, but the reason I don't is that I support the notion that the best response to hate speech is more speech.
last semester a chicago HFT group came to my school (Wisconsin-Madison) and spent a short time trying to tell us why they weren't unethical.
They didn't convince me at all. The gist of what they were saying is that they're merely making the market more efficient. That's such bullshit. The money they're making is surely coming from someone's pocket, and that probably means the middle class day-trader rather than the NYC firms with billions to spend on diversification. Later on in their presentation their tone made me feel that they were so extremely proud of what they were doing. It really irked me.
They didn't convince me at all. The gist of what they were saying is that they're merely making the market more efficient. That's such bullshit. The money they're making is surely coming from someone's pocket, and that probably means the middle class day-trader rather than the NYC firms with billions to spend on diversification. Later on in their presentation their tone made me feel that they were so extremely proud of what they were doing. It really irked me.
More likely their money comes from impatient traders who pay the spread to them. They compete for that spread with other firms like them.
The impatient spread-crosser is happy to be filled and move on to doing whatever it is they do for a living rather than playing trader. You can send a marketable order for a liquid ETF to any exchange in the US and at worst you'll be a tick away from the NAV. You don't have to compute the basket value. You don't have to watch the order books for 500 stocks on 10 exchanges. You don't have to monitor the futures markets or trade in them. One click and someone else does all that work for a penny. How is that not valuable?
What specifically did you find unethical about them?
The impatient spread-crosser is happy to be filled and move on to doing whatever it is they do for a living rather than playing trader. You can send a marketable order for a liquid ETF to any exchange in the US and at worst you'll be a tick away from the NAV. You don't have to compute the basket value. You don't have to watch the order books for 500 stocks on 10 exchanges. You don't have to monitor the futures markets or trade in them. One click and someone else does all that work for a penny. How is that not valuable?
What specifically did you find unethical about them?
>and that probably means the middle class day-trader rather than the NYC firms with billions to spend
First of all there just aren't that many middle class day-traders, and they don't trade in very large size. If they were the targets of HFT, the industry would be absolutely tiny.
Second, the small day traders just don't have enough volume to cause significant price impact. They just can't move the market enough for the HFTs to profit off of it.
Third, you'd be surprised at how unsophisticated and careless many huge institutional investors are. And sometimes they're just willing to pay extra to get in or out very quickly.
First of all there just aren't that many middle class day-traders, and they don't trade in very large size. If they were the targets of HFT, the industry would be absolutely tiny.
Second, the small day traders just don't have enough volume to cause significant price impact. They just can't move the market enough for the HFTs to profit off of it.
Third, you'd be surprised at how unsophisticated and careless many huge institutional investors are. And sometimes they're just willing to pay extra to get in or out very quickly.
That's their typical and only way to justify it, indeed. I also feel like they are clueless idiots fighting for small profits and serving a made up ideal of free market. This free market cannot exist because, when left to itself, it collapses. To remedy that, it's government funded and the money that goes into those Wall Street's geniuses is simply taxpayers'.
Aside from firms that have payment for order flow relationships, the US stock markets are basically the pure capitalism you read about in an economics textbook: Low barriers to entry, undifferentiated product (my quote is as good as anyone else's = best price wins), fierce competition.
Maybe it is today, but as soon as it starts to collapse (subprimes and friends), the textbook model has to be saved by external interventions. And so on and so forth. Long term, I doubt pure liberal free markets take us anywhere.
One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly.
Either way, it was a great school for learning how to program and use statistics effectively.
As my interviewer at Ronin told me after a I failed the interview (we both knew it): "This is all a game, you just need to learn the rules"