Bootstrapping an Ultra Low Latency Trading Firm, Part 2(veyronb.wordpress.com)
veyronb.wordpress.com
Bootstrapping an Ultra Low Latency Trading Firm, Part 2
http://veyronb.wordpress.com/2011/08/01/bootstrapping-an-ultra-low-latency-trading-firm-part-2/
50 comments
There is a signficant cost to obtain data and most of the seemingly simple trades don't make much money anymore. Purchasing realtime market data is very expensive, and you generally need to make 1000$ per trading day to break even. It's not easy in this economic climate, especially if you are trading your own money.
Amongst other things, dark pools and alternative venues have sucked a lot of liquidity that would traditionally hit the market.
The reason the information is available on Wikipedia is because people like me contributed to many of the wiki articles on the topic :)
The profitable opportunities themselves are fleeting and tiny, which means only the first few traders actually get the trades they want. This, coupled with the sheer number of firms (in the hundreds) forces traders to do what it takes to minimize their delays, including obtaining lower latency market data, high performance servers and proximity. And yes, some exchanges (like NYSE) definitely play favorites.
Amongst other things, dark pools and alternative venues have sucked a lot of liquidity that would traditionally hit the market.
The reason the information is available on Wikipedia is because people like me contributed to many of the wiki articles on the topic :)
The profitable opportunities themselves are fleeting and tiny, which means only the first few traders actually get the trades they want. This, coupled with the sheer number of firms (in the hundreds) forces traders to do what it takes to minimize their delays, including obtaining lower latency market data, high performance servers and proximity. And yes, some exchanges (like NYSE) definitely play favorites.
What happens to the industry when market access and data is openly available at an affordable price and a speed/scale comparable to what professional traders currently have access to? And why hasn't this been done yet? As an outsider to the financial industry, it looks like this information/access is slowly becoming commoditized (IB, Lime, etc.)... is there going to be chaos when any HN reader can begin low latency trading (or mining market data) as easily as they could get started on a service like Twilio?
Why are so many exchanges still private? Why do you have to pay exorbitant amounts of money for an exchange seat or high-quality exchange access? Why do some exchanges have designated market makers? Are these systems actually effective, or is this just nepotism among bankers?
Disclaimer: I have no idea what I'm talking about; I'm legitimately curious about this stuff. Hope I'm not coming across as too brash/angry. :-)
Why are so many exchanges still private? Why do you have to pay exorbitant amounts of money for an exchange seat or high-quality exchange access? Why do some exchanges have designated market makers? Are these systems actually effective, or is this just nepotism among bankers?
Disclaimer: I have no idea what I'm talking about; I'm legitimately curious about this stuff. Hope I'm not coming across as too brash/angry. :-)
All of your questions will be resolved if you understand one key fact: "Exchanges are companies". If you want to know why they are not government entities, imagine the GOP backlash if they tried to own and run an exchange (I'm not sure how democrats would respond, but republicans would definitely cry wolf). They are businesses, which is why:
- market access and data will not be openly available at a reasonable price (you have to spend extra money if you want to redistribute the data). Yes, this is a profit center for the exchanges
- Exchange seats will be limited and cost money (profit center)
- Colocation will always cost money (people are willing to pay to colocate)
As far as DMMs are concerned, there is a legal obligation to place orders and offer a buy and sell price. More specifically, they are required to be at the inside bid or offer (buying at the best buy or selling at the best sell price) most of the time. This ensures that there's a reasonable counterparty when some other trader wants to buy or sell. There is some economic incentive to be a DMM, but they provide a real service to the markets.
A few example exchange companies that list on public exchanges:
NDAQ: NASDAQ OMX Group lists
NYX: NYSE Euronext
CME: CME Group (Chicago Mercantile Exchange)
- market access and data will not be openly available at a reasonable price (you have to spend extra money if you want to redistribute the data). Yes, this is a profit center for the exchanges
- Exchange seats will be limited and cost money (profit center)
- Colocation will always cost money (people are willing to pay to colocate)
As far as DMMs are concerned, there is a legal obligation to place orders and offer a buy and sell price. More specifically, they are required to be at the inside bid or offer (buying at the best buy or selling at the best sell price) most of the time. This ensures that there's a reasonable counterparty when some other trader wants to buy or sell. There is some economic incentive to be a DMM, but they provide a real service to the markets.
A few example exchange companies that list on public exchanges:
NDAQ: NASDAQ OMX Group lists
NYX: NYSE Euronext
CME: CME Group (Chicago Mercantile Exchange)
Regarding corporate exchanges: they may be private companies, but they are heavily regulated and considered vital pieces of our economy. How can we sanely say something is financially vital to our country, and then exclude 99.9% of the population from it?
Regarding exchange business models: What about a transactional business model? Give people free access to the market, cut out the brokerage middle men, and charge players to place transactions?
Regarding DMMs: if there's a financial incentive to add liquidity as a DMM, why wouldn't the market naturally fill the role? Why auction the role off to the highest bidder?
Regarding exchange business models: What about a transactional business model? Give people free access to the market, cut out the brokerage middle men, and charge players to place transactions?
Regarding DMMs: if there's a financial incentive to add liquidity as a DMM, why wouldn't the market naturally fill the role? Why auction the role off to the highest bidder?
"How can we sanely say something is financially vital to our country, and then exclude 99.9% of the population from it?" <-- How can we declare a private bank "too big to fail"? I mean, no we can't, but this isn't the only insane situation here.
"What about a transactional business model? Give people free access to the market, cut out the brokerage middle men, and charge players to place transactions?" <-- who fronts the cost of the exchange? Electricity, rent, salaries, etc. Amortizing the cost through trading fees isn't fair to those who trade, because there would be users of data who don't end up paying at all. I think if you accept that they are private businesses, the pricing model makes sense.
" why wouldn't the market naturally fill the role?" Let's say the market starts falling rapidly. In a pure-market system, no one would be able to sell. The DMM obligation ensures that, at the very least, some people would be able to sell at some reasonable price. Think about insurance companies: you make a small amount of money most of the time, and every once in a while you need to make a large payout.
"What about a transactional business model? Give people free access to the market, cut out the brokerage middle men, and charge players to place transactions?" <-- who fronts the cost of the exchange? Electricity, rent, salaries, etc. Amortizing the cost through trading fees isn't fair to those who trade, because there would be users of data who don't end up paying at all. I think if you accept that they are private businesses, the pricing model makes sense.
" why wouldn't the market naturally fill the role?" Let's say the market starts falling rapidly. In a pure-market system, no one would be able to sell. The DMM obligation ensures that, at the very least, some people would be able to sell at some reasonable price. Think about insurance companies: you make a small amount of money most of the time, and every once in a while you need to make a large payout.
In the heat of criticizing the financial industry, I forgot to say that I'm enjoying your posts, and I think what you're working on is really cool. :-)
A long time ago, there was a startup called Digital Island. They were a small(er) exchange (or is the term "market maker"?), but they handled about 20% of NASDAQ's daily volume.
The good thing was, they would let you buy their data post-facto (not realtime). At the end of the day, you could download their entire set of completed orders for the day. The cost was minimal ($30/month, IIRC).
But they were bought out by NASDAQ, and NASDAQ shut down this data-sharing program. Which, to me, reinforces the point that the powers-that-be don't want to lose control of the information; it's a pay-to-play system, and information is one of the privileges to paying the big bucks.
One would assume that being a heavily regulated system, it would be expected that all orders would be open to the public (at least the historical ones), to ensure transparency. But one would be mistaken.....
The good thing was, they would let you buy their data post-facto (not realtime). At the end of the day, you could download their entire set of completed orders for the day. The cost was minimal ($30/month, IIRC).
But they were bought out by NASDAQ, and NASDAQ shut down this data-sharing program. Which, to me, reinforces the point that the powers-that-be don't want to lose control of the information; it's a pay-to-play system, and information is one of the privileges to paying the big bucks.
One would assume that being a heavily regulated system, it would be expected that all orders would be open to the public (at least the historical ones), to ensure transparency. But one would be mistaken.....
> "powers-that-be don't want to lose control of the information"
This. What happens when they inevitably do?
This. What happens when they inevitably do?
They won't. To lose control of the information, the way it is _generated_ must become decentralized. While this may happen (e.g., it does in foreign exchange markets to some extent), it will not be _this_ kind of information, but something completely different.
They are probably pretty convinced that if your only competitive advantage is restricting access to information, you are doing it right.
I mean, isn't capitalism basically wrapped around the idea that without competition, businesses will act in their own best interest, which is diametrically opposed to the interests of consumers? That is to say, given the choice any "smart" business will leap at the chance to lock out competition, because that is in the best interests of the business (if it can get away with it)?
Anyway, what this is really about is getting the information faster. I don't really see anything wrong with that; some information is just time sensitive, and a good informant will practically be in the business of getting you information faster than his competitors... I see this as much the same thing.
I mean, isn't capitalism basically wrapped around the idea that without competition, businesses will act in their own best interest, which is diametrically opposed to the interests of consumers? That is to say, given the choice any "smart" business will leap at the chance to lock out competition, because that is in the best interests of the business (if it can get away with it)?
Anyway, what this is really about is getting the information faster. I don't really see anything wrong with that; some information is just time sensitive, and a good informant will practically be in the business of getting you information faster than his competitors... I see this as much the same thing.
No, capitalism is not a zero sum game, if that is what you're implying.
That's not what I mean. A corporation and its customers can both profit- one does not have to do poorly for the other to do well.
I only mean that a corporation can arguably do better (for itself) if it doesn't worry about things like regulations or competitors. A classic, classic example is dumping sewage into the ocean. Companies continue to do it even when it's legislated against.
I only mean that a corporation can arguably do better (for itself) if it doesn't worry about things like regulations or competitors. A classic, classic example is dumping sewage into the ocean. Companies continue to do it even when it's legislated against.
It's not so much restricting access to information as getting and acting on it quicker.
In "the olden days" there were very similar trades done by people who (for example) could get information about tea clippers from China arriving in England. Now, it's automated, but the idea is the same.
In "the olden days" there were very similar trades done by people who (for example) could get information about tea clippers from China arriving in England. Now, it's automated, but the idea is the same.
It's about who can afford to buy the fastest hardware on the fastest pipes
I think you underestimate the skill required to do that. I mean, Google is not king of search just because they can afford bigger datacentres than Bing. Search is not just "looking stuff up" that anyone with access to Wikipaedia could figure out in a few days, even tho' there is an article on MapReduce!
I think you underestimate the skill required to do that. I mean, Google is not king of search just because they can afford bigger datacentres than Bing. Search is not just "looking stuff up" that anyone with access to Wikipaedia could figure out in a few days, even tho' there is an article on MapReduce!
You might want to check out this paper by Alexander D. Wissner-Gross and Freer: http://www.alexwg.org/publications/PhysRevE_82-056104.pdf
Neat!
I'm not sure "the public" wants access to microsecond trade order data. I mean, I could send it to my mom's laptop, but she'd probably be pissed.
It's the market makers who are fulfilling these orders and the strike prices are already set once the HFT algos get in on the action (right?). I'm assuming that if the MMs had a problem with it, they'd have the clout to do something.
It's the market makers who are fulfilling these orders and the strike prices are already set once the HFT algos get in on the action (right?). I'm assuming that if the MMs had a problem with it, they'd have the clout to do something.
From part 1:
Question: Why don’t more people do it?
Answer: People do try, but its not as easy as it looks.Nice post veyron. Please keep it up.
For those interested, here's some other good reading on this topic while you wait for veyron's next post :)
- http://www.technologyreview.com/blog/mimssbits/25308/ Using textual analysis
- http://www.institutionalinvestor.com/Article/2593339/Markets... Article on HFT
- http://news.ycombinator.com/item?id=1517339 Ask HN: Is it feasible to do high-frequency trading as an individual?
- http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat... I bootstrapped an automated trading firm ama
- http://quant.stackexchange.com/
- http://bitstat.wordpress.com/
For those interested, here's some other good reading on this topic while you wait for veyron's next post :)
- http://www.technologyreview.com/blog/mimssbits/25308/ Using textual analysis
- http://www.institutionalinvestor.com/Article/2593339/Markets... Article on HFT
- http://news.ycombinator.com/item?id=1517339 Ask HN: Is it feasible to do high-frequency trading as an individual?
- http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat... I bootstrapped an automated trading firm ama
- http://quant.stackexchange.com/
- http://bitstat.wordpress.com/
Don't ask who is losing on the other side of your trades but think instead who is your customer. What is the service that you provide and how it helps them.
If you're making a profit it means you bought inventory when the price was below fair value (your customers didn't need it and wanted to sell as fast as possible) and you sold it when the price was above (your customers really needed the shares right now). The net benefit to everybody is that the volatility is lower, as you moved the price down when it was too high and moved it up when it was too low.
The market efficiency is higher too: a lot less capital is required to establish fair prices as market reacts immediately to any imbalance.
It also makes the spread lower and makes buying and selling stock cheaper for your customers. Only a few years ago market makers and specialists would chicken out at the first sign of trouble and would widen the spread between bid and ask prices. Crossing the spread is a huge part of your overall expense of trading. Unfortunately very few investors understand full impact of it on their returns and don't appreciate your contribution.
Execution time is better now. Even during flash crash it was possible to buy and sell with retail brokers, where's I still remember times in 2001 when retail broker market orders sometimes took minutes to fill.
If you're making a profit it means you bought inventory when the price was below fair value (your customers didn't need it and wanted to sell as fast as possible) and you sold it when the price was above (your customers really needed the shares right now). The net benefit to everybody is that the volatility is lower, as you moved the price down when it was too high and moved it up when it was too low.
The market efficiency is higher too: a lot less capital is required to establish fair prices as market reacts immediately to any imbalance.
It also makes the spread lower and makes buying and selling stock cheaper for your customers. Only a few years ago market makers and specialists would chicken out at the first sign of trouble and would widen the spread between bid and ask prices. Crossing the spread is a huge part of your overall expense of trading. Unfortunately very few investors understand full impact of it on their returns and don't appreciate your contribution.
Execution time is better now. Even during flash crash it was possible to buy and sell with retail brokers, where's I still remember times in 2001 when retail broker market orders sometimes took minutes to fill.
"If you're making a profit it means you bought inventory when the price was below fair value (your customers didn't need it and wanted to sell as fast as possible), and you sold it when the price was high (your customers really needed the shares right now)."
It's a bit more complicated than that, because it's fully possible that the seller in a trade is selling above his fair value and the buyer in that trade is buying below her fair value. Supply/demand certainly plays into it (a trader with a very large position is definitely willing to take some price hit if they are able to quicky their position), which will invariably lead to a discussion of utility functions and slowly bore everyone :P
It's a bit more complicated than that, because it's fully possible that the seller in a trade is selling above his fair value and the buyer in that trade is buying below her fair value. Supply/demand certainly plays into it (a trader with a very large position is definitely willing to take some price hit if they are able to quicky their position), which will invariably lead to a discussion of utility functions and slowly bore everyone :P
> invariably lead to a discussion of utility functions and slowly bore everyone
But that's the crux of the matter - both sides of the trade are getting some utility gain (otherwise the trade would not happen) and thus it is not a zero sum game.
It is still zero sum in short term dollars, which just obscures the subject for the people who equate utility with dollars.
But that's the crux of the matter - both sides of the trade are getting some utility gain (otherwise the trade would not happen) and thus it is not a zero sum game.
It is still zero sum in short term dollars, which just obscures the subject for the people who equate utility with dollars.
No, they are not getting utility gain.
Recently I was selling an netbook, aiming to get about £100. A friend told me she was wanting something, and looking to spend about £150. We split the difference and went £125.
It would not have been a utility gain if someone had quickly jumped between us, given me £100, sold the laptop to her for £150 and kept the £50 for themselves. We would both have ended up out of pocket.
The whole point of exchanges is that they connect buyers and sellers. Parasites living making a living by being faster than anyone else are not helping anyone.
If there was a level playing field, and everyone went at the same speed, and they could still make money, then I could believe they were adding some utility.
Recently I was selling an netbook, aiming to get about £100. A friend told me she was wanting something, and looking to spend about £150. We split the difference and went £125.
It would not have been a utility gain if someone had quickly jumped between us, given me £100, sold the laptop to her for £150 and kept the £50 for themselves. We would both have ended up out of pocket.
The whole point of exchanges is that they connect buyers and sellers. Parasites living making a living by being faster than anyone else are not helping anyone.
If there was a level playing field, and everyone went at the same speed, and they could still make money, then I could believe they were adding some utility.
It's more subtle than that. If you wanted to make the sale now and your friend wanted to buy next week when she got paid, then an intermediary would be adding value, purely by being willing to act more quickly. The difference would cover them sitting on the laptop for a week.
It is yet more sophisticated for some HFT trading.
Continuing the analogy: Some trading is as if there's a person standing between the two people trying to make a deal and can hear the offers before each party can react and then make their own offers to each party to capture some of the difference (re: HFT profit).
Explanation: In some forms of HFT the key advantage is that they can buy the market data faster than other people so they can see upcoming trades before other participants (undoubtedly there are better citations but here's a decent NYT article): http://dealbook.nytimes.com/2010/06/11/opening-up-the-market...
Continuing the analogy: Some trading is as if there's a person standing between the two people trying to make a deal and can hear the offers before each party can react and then make their own offers to each party to capture some of the difference (re: HFT profit).
Explanation: In some forms of HFT the key advantage is that they can buy the market data faster than other people so they can see upcoming trades before other participants (undoubtedly there are better citations but here's a decent NYT article): http://dealbook.nytimes.com/2010/06/11/opening-up-the-market...
High-speed traders are living in the millisecond range. Do you think the people doing trading on the order of seconds are pleased they are paying to save milliseconds or not? I suspect not, but would be happy to be proved wrong.
What would really happen is that a second trader would enter the market, offer to buy the laptop for 110$, hold onto it and then sell it on for 140$. Actual prices vary and with enough competition, will settle to something that gives the trader still a nice profit and enough cash to insure against the risk of such an operation (the trader doesn't know up-front when he can actually onload the goods and how far the price might move meanwhile).
Of course, if you're not desperate on selling right now and or the risk for the trader is too high (i.e. the minimum price you'll sell right now is higher than what the trader is willing to offer you right now), nothing happens.
Concluding, traders do add utility to a market. And as others have already said, with high freq trading it mostly results in massively reduces spreads (and the temporal utility as outlined in the laptop example essentially disappears).
Of course, if you're not desperate on selling right now and or the risk for the trader is too high (i.e. the minimum price you'll sell right now is higher than what the trader is willing to offer you right now), nothing happens.
Concluding, traders do add utility to a market. And as others have already said, with high freq trading it mostly results in massively reduces spreads (and the temporal utility as outlined in the laptop example essentially disappears).
These high-frequency traders are not parasites. They are providing liquidity. Essentially, they are providing you a service. If your friend had not come along, perhaps you would have been holding onto your netbook for a lot longer, or it may not have sold at all.
If you want to see what happens when liquidity dries up, you only have to look to the Flash Crash. Yes, the whole thing started due to an erroneous trade, but the fall was greatly magnified when all of the computers were pulled from the market when they couldn't make sense of things. If this type of trading were to be banished tomorrow, the market would tank so quickly that people would be begging for the computers to be flipped back on. The genie is already out of the bottle, and it can't go back in without considerable pain to everyone around it.
If you want to see what happens when liquidity dries up, you only have to look to the Flash Crash. Yes, the whole thing started due to an erroneous trade, but the fall was greatly magnified when all of the computers were pulled from the market when they couldn't make sense of things. If this type of trading were to be banished tomorrow, the market would tank so quickly that people would be begging for the computers to be flipped back on. The genie is already out of the bottle, and it can't go back in without considerable pain to everyone around it.
Dont forget that a lot of trades didn't clear.
I read somewhere that something like 30% of trades used to fall through because of paperwork mixups and other problems before electronic exchanges.
How would you like to think you sold at a profit only to find out that the trade didn't clear and now you're in the hole because the stock bottomed out?
I read somewhere that something like 30% of trades used to fall through because of paperwork mixups and other problems before electronic exchanges.
How would you like to think you sold at a profit only to find out that the trade didn't clear and now you're in the hole because the stock bottomed out?
The worst part was May 6 2010 (the flash crash). The exchanges and SEC arbitrarily threw out a number 60%, and people who managed to buy at 59% discount got to keep their (very) profitable trade and people who managed to buy at 61% discount lost only one leg of the trade. The process was not transparent, at all :/
I know some people who made a boatload (all trades cleared), and others who lost a boatload (one part of their trade was cleared, but the other part was broken, and they had to go back and buy back at a very disadvantageous price)
I know some people who made a boatload (all trades cleared), and others who lost a boatload (one part of their trade was cleared, but the other part was broken, and they had to go back and buy back at a very disadvantageous price)
Is it feasible to apply these basic HFT principles to the Bitcoin market? I assume it would be easier than the "real" markets because you aren't competing against the big players.
Wow, downvotes for asking a question?
The first example is called "pairs trading". Try extending out the math to multiple sets.
I once wrote an ADR arb detector, which even took into account the forward contract for the currency trade.
I once wrote an ADR arb detector, which even took into account the forward contract for the currency trade.
This is fun reading. It reminded me of another blog, "WK's High Frequency Trading Blog" (howtohft.wordpress.com) which got about 9 posts in with some great details, and then went silent.
I found it in my Google Reader feedlist, and the last post was from March, but Wordpress now says the blog has been deleted. Anyone know what happened it?
Veyron, I'm glad you're writing these posts. It's cool to have HFT demystified.
I found it in my Google Reader feedlist, and the last post was from March, but Wordpress now says the blog has been deleted. Anyone know what happened it?
Veyron, I'm glad you're writing these posts. It's cool to have HFT demystified.
No idea what happened to it, but I found this backup if anyone is interested:
http://www.opensourcetrader.com/author/wk-selph/
http://www.opensourcetrader.com/author/wk-selph/
I personally think he was writing those so that someone would stumble across them and hire him. The new employer must have asked him to take them down.
115 days ago, you posted that you had quit your job and struck out on your own. Is this the project you've been working on since then?
A friend and I built an iPhone app in pursuit of a new concept, and we did a few field tests, but Apple did not approve the app. At that point, although we could have continued pushing, we decided it was easier to stop the project and part ways. I decided I didn't want to go back and work for someone else (blame HN for being bitten by the startup bug :)
These are great! Keep posting 'em! :)
I'm more interested in the tech than the economics, but it's still a good read.
I'm more interested in the tech than the economics, but it's still a good read.
You can buy off the shelf HFT solution from http://www.stoneridgetechnology.com/products/pci-e-developme...
There are all kinds of companies selling technology solutions for everything (as targeted as FPGA-driven feed data processors; as all-encompassing as the solution you presented). Most of them are crap and oftentimes have no paying customers. Hopefully, I will be discussing the state of the financial tech industry in a later post.
What services would you use for executing the trades?
I don't know why I wrote this field off as boring several years ago. This is absolutely fascinating (from a technical perspective, I mean)
[deleted]
What makes these strategies profitable is exclusive access to information - not actual investing prowess. It's about who can afford to buy the fastest hardware on the fastest pipes in the closest proximity to the exchanges and top tier e-brokers. It's frequently even about who you know.
If your only competitive advantage is systemically restricting the public's access to information, you're doing it wrong.
(but my inner nerd thinks this is super cool, and my inner Gordon Gekko wants to get started today)