Algorithmic Trading: A Practical Tale for Engineers(toptal.com)
toptal.com
Algorithmic Trading: A Practical Tale for Engineers
http://www.toptal.com/data-science/algorithmic-trading-a-practical-tale-for-engineers
8 comments
I can attest to the fact that this sort of thing can be done consistently and successfully for thousands of trades over months' time.
Then you know that forex is a bank driven market; these little "statistical arbitrages" (which should be a 4 letter word) are subject to scale. When they break, they drain you dry. The "street" is littered with failed FX traders. It is the crack rock of the consumer level game because you can muster such phenomenal leverage (it isn't regulated nearly as much as equities).
At the bank level, they can see who the counter party is and they track their interactions with you. If you milk them too much, they will block you and your trade is dead. I can attest to that.
What is worse is the consumer-level brokers are essentially running a game against you.
Caveat Trado
Then you know that forex is a bank driven market; these little "statistical arbitrages" (which should be a 4 letter word) are subject to scale. When they break, they drain you dry. The "street" is littered with failed FX traders. It is the crack rock of the consumer level game because you can muster such phenomenal leverage (it isn't regulated nearly as much as equities).
At the bank level, they can see who the counter party is and they track their interactions with you. If you milk them too much, they will block you and your trade is dead. I can attest to that.
What is worse is the consumer-level brokers are essentially running a game against you.
Caveat Trado
My only experience is with equities, but yes forex is both highly leveraged and poorly regulated.
Even in equities, your brokerage is not your friend: if you place a limit order, there's nothing to stop them from placing the same order just a bit closer to the market and using you as their free stop loss.
Even in equities, your brokerage is not your friend: if you place a limit order, there's nothing to stop them from placing the same order just a bit closer to the market and using you as their free stop loss.
I'm not sure I understand what you are saying. How does the brokerage make money off of that. If they are improving the price you haven't lowered their risk profile at all.
If they are taking the other side of your trade and the market doesn't reflect that, they've actually given you a free roll.
If they are taking the other side of your trade and the market doesn't reflect that, they've actually given you a free roll.
Suppose you want to buy XYZ for $1.00. You place your bid and are the highest price against a $1.03 offer.
Evil broker places a $1.01 bid and gets a fill.
He immediately offers $1.02.
Case 1: The market runs up, he makes his tick on essentially zero commissions. On some markets, you earn commissions for providing liquidity.
Case 2: The market falls off, he smacks your bid and only loses a penny.
This is edge.
EDIT: When I say "the market", he could base his actions off an index. As for "free roll", sure... but there is still an impact on liquidity and you're in a situation where you are more likely to get a fill whenever the market is going against you.
Evil broker places a $1.01 bid and gets a fill.
He immediately offers $1.02.
Case 1: The market runs up, he makes his tick on essentially zero commissions. On some markets, you earn commissions for providing liquidity.
Case 2: The market falls off, he smacks your bid and only loses a penny.
This is edge.
EDIT: When I say "the market", he could base his actions off an index. As for "free roll", sure... but there is still an impact on liquidity and you're in a situation where you are more likely to get a fill whenever the market is going against you.
Is that legal trading anywhere? That seems like it would fall into front running definitions in all the jurisdictions I know of.
Legal? No. Does it happen: yes.
OK, but to my other reply. If they are going to blatantly front run, why would they even bother with the markets. Just spoof market behavior to you and take your money. Much easier.
If this behavior is impacting your execution why are you still using "Evil Broker"?
If this behavior is impacting your execution why are you still using "Evil Broker"?
You mean, provide a false market?
They do that in FX (you're trading their market; it is up to you to know what is going on beyond their walls).
As for spoofing other markets, you can get away with front running.
They do that in FX (you're trading their market; it is up to you to know what is going on beyond their walls).
As for spoofing other markets, you can get away with front running.
I'll grant trading FX is for crazy people.
As someone who's experience is way more on the commodities futures side of the house, I'm shocked that blatant front running is an expected cost of business in equities trading. Even in the highly monopolized CME world front running is considered beyond the pale.
I will say (in my experience in the commodities markets), 9 times out of 10 folks who claim that they are being front run are actually just unaware of how sophisticated their competition is when it comes to micro-book structure optimization.
As someone who's experience is way more on the commodities futures side of the house, I'm shocked that blatant front running is an expected cost of business in equities trading. Even in the highly monopolized CME world front running is considered beyond the pale.
I will say (in my experience in the commodities markets), 9 times out of 10 folks who claim that they are being front run are actually just unaware of how sophisticated their competition is when it comes to micro-book structure optimization.
FX is a great market, but not for the reasons people typically think. The problem lies in brokers for the mass market (the little guy). If you're at a billion dollar hedge fund, you have access to an entirely different class of brokers (and those markets are full of intrigue...)
I second your claim ("9 times out of 10") for CME. Human nature is to assign a complex answer ("they're reading the tape") to something very simple ("my model is shit").
That said, front running happens on CME, too. Again, it is the same construction: consumer sending orders through a nefarious broker.
I second your claim ("9 times out of 10") for CME. Human nature is to assign a complex answer ("they're reading the tape") to something very simple ("my model is shit").
That said, front running happens on CME, too. Again, it is the same construction: consumer sending orders through a nefarious broker.
EDIT: Adding this here since I can't reply to you kasey.
Why trade with Evil Broker? Don't. I suppose you can argue the difference between trading and investing.
Now, if you want to trade, well, just know they're going to fuck you and lubricate appropriately.
Why trade with Evil Broker? Don't. I suppose you can argue the difference between trading and investing.
Now, if you want to trade, well, just know they're going to fuck you and lubricate appropriately.
[deleted]
Imagine your algorithm says that a certain non-volatile instrument isn't going to go much below 5.00 and often goes above 5.15. You place a limit order in advance to buy at 5.00 (market orders don't guarantee price) and if you get filled you'll be hoping to short at 5.15 or so later. "Evil Brokerage" comes in and offers to buy at 5.01. Remarkably, you observe that any time you pull your 5.00 order, their 5.01 order disappears.
The instrument heads toward your 5.00 target. If their order turns out to provide all the necessary liquidity at that moment, the sellers never reach you, and the instrument turns around and "Evil Brokerage" soon sell for higher. Alternatively, if the instrument is on a move and it going to go against you too, brokerage sell to you and they only lose 0.01/share. That's why I call the scenario a free stop loss.
The instrument heads toward your 5.00 target. If their order turns out to provide all the necessary liquidity at that moment, the sellers never reach you, and the instrument turns around and "Evil Brokerage" soon sell for higher. Alternatively, if the instrument is on a move and it going to go against you too, brokerage sell to you and they only lose 0.01/share. That's why I call the scenario a free stop loss.
I get that, but again, doesn't that fall under illegal trading behavior? If we are going to assume that the brokerages are rampantly front running, why assume they are even telling the truth about execution.
At that point they can just tell you that your trades executed that way without the bother of actually going into the market.
Further, why would you stay on an execution platform that is so obviously bad for you?
At that point they can just tell you that your trades executed that way without the bother of actually going into the market.
Further, why would you stay on an execution platform that is so obviously bad for you?
I don't think it's illegal, not even really front-running. That'd be them turning a price profit on filling your order, not just using it to avoid a potential loss.
Also, you're right, at a minimum, via a subsidiary it's possible for the brokerage to "fill" your order without it ever reaching the exchanges, I think as long as it's inside the national bid-ask price spread at that moment. They pitch this as a feature - it also lets them avoid exchange fees. Then they can manage their overall risk separately.
Yes, I stopped trying to make those trades years ago. At one time though it was a remarkably consistent way to succeed on a small scale.
Also, you're right, at a minimum, via a subsidiary it's possible for the brokerage to "fill" your order without it ever reaching the exchanges, I think as long as it's inside the national bid-ask price spread at that moment. They pitch this as a feature - it also lets them avoid exchange fees. Then they can manage their overall risk separately.
Yes, I stopped trying to make those trades years ago. At one time though it was a remarkably consistent way to succeed on a small scale.
I'll admit I haven't looked at every broker, but every exchange that I know of, you have to opt in for orders to not go to the other exchanges.
This frequently is a feature for both sides of the equation due to lower fees, better execution rules, rebates, priority, etc.
I certainly know that I would never opt in to a platform that consistently provided worse execution (especially in the equities space where there are so many choices).
This frequently is a feature for both sides of the equation due to lower fees, better execution rules, rebates, priority, etc.
I certainly know that I would never opt in to a platform that consistently provided worse execution (especially in the equities space where there are so many choices).
> You place a limit order in advance to buy at 5.00 (market orders don't guarantee price) and if you get filled you'll be hoping to short at 5.15 or so later.
Minor nitpick.
Since you've bought the stock first, your second trade will be a sell and not a short, unless you overfill on the second leg, in that case you'll have a partial sell and partial short.
Minor nitpick.
Since you've bought the stock first, your second trade will be a sell and not a short, unless you overfill on the second leg, in that case you'll have a partial sell and partial short.
Correct. In this simple algo scenario the position is assumed to be long 1X or short 1X at any given time. After the first trade, all reversals trades are then either buy 2X or sell 2X until the very last trade.
Very true. Case in point, Ameritrade's odd marriage with Citadel... think they might be reading the tape?
https://www.tdameritrade.com/forms/AMTD2055.pdf
https://www.tdameritrade.com/forms/AMTD2055.pdf
Interesting, so, you are saying that the right approach (plus a religious avoidance of leveraged positions) is generally a low risk investment? What about trading fees, won't those progressively eat into you as your model continues to fizzle out at thousands of trades per month?
Trading costs have to part of your model, but at Interactive Brokers for example you only pay $0.005/share (with a $1 minimum) to trade. At that rate you can try to trade for an average of pennies per share (including winners and losers). When an algorithm fizzles, my experience is that it stops making small amounts of money per trade on average and starts losing small amounts of money per trade on average, with more volatility.
As you see then, you may automate trading itself, but then your discretion is shifted from deciding what stock to buy or sell short to when to turn on or off an algorithm. The advantage I suppose is that an algorithm isn't necessarily directly tied to market direction.
As you see then, you may automate trading itself, but then your discretion is shifted from deciding what stock to buy or sell short to when to turn on or off an algorithm. The advantage I suppose is that an algorithm isn't necessarily directly tied to market direction.
Ah, thanks for the explanation! I've stayed completely away from forex, so I don't know much about it.
I'm just going to leave this here.
I worked at a FX broker as a software engineer. When nerds first read about this stuff we all have the same knee-jerk reaction: "I write code. I'm smart. I could build a robot trader and make a Walter White pile of cash."
In short, I would recommend you stay the hell away from FX. It's bullshit. It's not like the any type of trading (algorithmic, quantative, manual, good luck charm) you see in stockes. I do thank the blog author for mentioning that his system never seemed to work long term. That's the key element most people can't accept.
Here are a few notes from my time in FX:
FX is traded on margin, and it's easy to lose all your money. Nearly every single person will lose all their money in FX, and those who claim success are riding a temporary wave. Even "experts" in the field can't hold big gains much more than a few weeks or months at best.
A few years back our company bought a trading system we'd been marketing like crazy and had customers buy into. Senior management decided to invest a 6-figure sum in this system with the intent of using the profits for employee bonuses. The day it crashed, not only did we have angry customers but our employees were pretty depressed.
FX trading systems are hype. Always. You'll hear the term "holy grail" in reference to trading strategies. That should be enough to keep you away. The real money is made by selling systems to idiots and writing books about it. As a matter of fact, you'll notice a lot of forums and blogs have referral links to brokers ;).
A common sentiment I heard tossed around traders and employees goes something like, "rather than trade FX, just go to Vegas and play roulette. At least you get free beers."
The technology is god-awful. You'd be blown away if you saw how shit-tacular the backend is that handles your money. I thought it was just the place I worked, but nope.
The company I worked for is very open and compliant, which I found admirable. This has continued to be their focus, so that's cool. Many brokers are corrupt, particularly if they're not in the US. Brokers have been caught increasing margins against traders, stop-gapping trades and worse. In the US the NFA is insane so they do a decent job of regulating, but it seems every year or so they nab a broker on something. Most NFA fines range in the millions so we took this seriously.
I have more war stories, but that's enough. Best advice: do not trade FX. I realize that may rile some of you, so be it. I wouldn't put my money near that shit.
I worked at a FX broker as a software engineer. When nerds first read about this stuff we all have the same knee-jerk reaction: "I write code. I'm smart. I could build a robot trader and make a Walter White pile of cash."
In short, I would recommend you stay the hell away from FX. It's bullshit. It's not like the any type of trading (algorithmic, quantative, manual, good luck charm) you see in stockes. I do thank the blog author for mentioning that his system never seemed to work long term. That's the key element most people can't accept.
Here are a few notes from my time in FX:
FX is traded on margin, and it's easy to lose all your money. Nearly every single person will lose all their money in FX, and those who claim success are riding a temporary wave. Even "experts" in the field can't hold big gains much more than a few weeks or months at best.
A few years back our company bought a trading system we'd been marketing like crazy and had customers buy into. Senior management decided to invest a 6-figure sum in this system with the intent of using the profits for employee bonuses. The day it crashed, not only did we have angry customers but our employees were pretty depressed.
FX trading systems are hype. Always. You'll hear the term "holy grail" in reference to trading strategies. That should be enough to keep you away. The real money is made by selling systems to idiots and writing books about it. As a matter of fact, you'll notice a lot of forums and blogs have referral links to brokers ;).
A common sentiment I heard tossed around traders and employees goes something like, "rather than trade FX, just go to Vegas and play roulette. At least you get free beers."
The technology is god-awful. You'd be blown away if you saw how shit-tacular the backend is that handles your money. I thought it was just the place I worked, but nope.
The company I worked for is very open and compliant, which I found admirable. This has continued to be their focus, so that's cool. Many brokers are corrupt, particularly if they're not in the US. Brokers have been caught increasing margins against traders, stop-gapping trades and worse. In the US the NFA is insane so they do a decent job of regulating, but it seems every year or so they nab a broker on something. Most NFA fines range in the millions so we took this seriously.
I have more war stories, but that's enough. Best advice: do not trade FX. I realize that may rile some of you, so be it. I wouldn't put my money near that shit.
Amen,
Forex has less fundamental information than other markets,incredible leverage and is subject to (legal) political manipulation at the highest level (see Sovereign wealth funds). In addition relationships I have found holding for stocks/indices do not transfer to the forex market at all.
Easily the most difficult market. I think it is probably crackable quantitively but only for a more limited set of strategies than the standard market.
This is actually an incredibly important point.
The FX market is a zero sum game and due to that its' incredibly cutthroat.
The FX market is a zero sum game and due to that its' incredibly cutthroat.
Very neat story. Thanks for writing it up. The nuts and bolts of this kind of operation are fascinating. I know a lot of people out there have trading ideas, but don't really know how to implement them.
If you just finished the post and you want to try coding up a trading algorithm for free, check out http://www.quantopian.com. It uses Python, not MQL, and provides free data and backtesting. (Yes, I work at Quantopian)
If you just finished the post and you want to try coding up a trading algorithm for free, check out http://www.quantopian.com. It uses Python, not MQL, and provides free data and backtesting. (Yes, I work at Quantopian)
I posted this on topal too, but I want to get your opinions also:
I studied a bit of market theory in college and learned about channel trading. The Efficient Market Hypothesis is just that - a "Hypothesis" - not a law. Plenty of academics have argued against it but they don't teach this in undergrad courses so as not to confuse students (this is seriously the answer I was given when I asked my teachers). Some people like Nassim Taleb are calling for a BAN on teaching entry level finance courses to MBAs who go out and gamble vast amounts of money based on incorrect theories.
I always thought that algo trading would be a good fit for channel strategies since the strategy is recursive in nature.
Does anyone have any pointers on how to implement channel type of strategies (as opposed to Moving Average strategies)? I'd be willing to chat about deeper into this topic.
FYI: Some (old) research shows that Exponential MA strategies make more and even out perform buy and hold strategies without taking into account tax advantages. If you have ever read a "Random Walk Down Wall Street" and don't believe in this "crap," it's actually in the footnotes of the sources the author cites.
I studied a bit of market theory in college and learned about channel trading. The Efficient Market Hypothesis is just that - a "Hypothesis" - not a law. Plenty of academics have argued against it but they don't teach this in undergrad courses so as not to confuse students (this is seriously the answer I was given when I asked my teachers). Some people like Nassim Taleb are calling for a BAN on teaching entry level finance courses to MBAs who go out and gamble vast amounts of money based on incorrect theories.
I always thought that algo trading would be a good fit for channel strategies since the strategy is recursive in nature.
Does anyone have any pointers on how to implement channel type of strategies (as opposed to Moving Average strategies)? I'd be willing to chat about deeper into this topic.
FYI: Some (old) research shows that Exponential MA strategies make more and even out perform buy and hold strategies without taking into account tax advantages. If you have ever read a "Random Walk Down Wall Street" and don't believe in this "crap," it's actually in the footnotes of the sources the author cites.
Channels are boundary conditions on an estimation of a price or a price itself,effectively a form of error bar. Your statistically looking at central tendencies and dispersion.
No. That's another trading method. I know what you are talking about there too, but that's not how I see the strategy working.
What I found in my research is that the statistical analysis being done on markets is an approximation of the underlying phenomena. Yes - central tendencies and dispersion work 90% of the time, but they hugely under price tail events. This is the basis of Nassim Taleb's trading strategy.
What I see really happening is a recursive channel pattern at different time scales. As the recursion plays out it "zooms in" on the correct price of a security and the central tendencies and dispersion can be analyzed through traditional statistics. However, when a market correction takes place it "zooms out" by stepping back through previous market cycles and then statistical models break down. It's very weird to explain. Essentially, I see the market being a predictable Fractal pattern. Check out Mandelbrot's work on Finance to get some insight. I just disagree with his (and Nassim's) conclusion that these can't be predicted.
I've always been interested in trying to back test my theory but I didn't have the technical chops and didn't know where to start looking.
Thanks for your thoughts.
What I found in my research is that the statistical analysis being done on markets is an approximation of the underlying phenomena. Yes - central tendencies and dispersion work 90% of the time, but they hugely under price tail events. This is the basis of Nassim Taleb's trading strategy.
What I see really happening is a recursive channel pattern at different time scales. As the recursion plays out it "zooms in" on the correct price of a security and the central tendencies and dispersion can be analyzed through traditional statistics. However, when a market correction takes place it "zooms out" by stepping back through previous market cycles and then statistical models break down. It's very weird to explain. Essentially, I see the market being a predictable Fractal pattern. Check out Mandelbrot's work on Finance to get some insight. I just disagree with his (and Nassim's) conclusion that these can't be predicted.
I've always been interested in trying to back test my theory but I didn't have the technical chops and didn't know where to start looking.
Thanks for your thoughts.
Diregard Taleb,the man cant trade (buying OTM puts and hoping for the world to end is not a trading strategy.)You are looking at basically a three regime model,(standard statistical events,macro breakdown,and "dont know").The problem is "fractals" is not a decent way to distinguish the three states,so now you have 3 problems. For what its worth I think Mandelbrot was onto to something but its a long way away from a trading strategy.You could decompose movements over different frequencies ("Fractal Self Similarity) and test for some sort of state wise relationship between recent micro and macro movements.Im largely sure this will exist but extracting a leading indicator out of this may be difficult.
I know this is late: but you get what I'm talking about:
You could decompose movements over different frequencies ("Fractal Self Similarity) and test for some sort of state wise relationship between recent micro and macro movements.
That's where the channel patterns come in - they are self similar at different time scales. The rules are pretty well known, but getting them coded is a different story. I always find MACD strategies which are only approximations.
I have taken different approach. I decided that the best way is to train by own brain how to trade. Basically my approach was to trade as much as possible and have results, without waiting for the market to let me know how I was doing.
So I wrote my own program called turtrades (test your trades). It basically presentments you with random stock in random past date and lets you trade or pass. Doing this I am using real historical market data and train my brain to spot winning patters and I get instant result.
For those interested you can find it here http://www.turtrades.com
Yes it is as selfish plug but program is free and I have nothing to gain.
Yes it is as selfish plug but program is free and I have nothing to gain.
I think there was a Bloomberg article at the front page of HN a few weeks ago talking about how algorithmic trading was producing less and less profits - can't find the source/link right now.
Is that still true? Or is algorithmic trading on a rebound?
Is that still true? Or is algorithmic trading on a rebound?
Algorithmic trading and high frequency trading are orthogonal concepts.
High frequency trading typically refers to two concepts. 1) how long you hold your position, 2) how quickly you can send/cancel existing orders. Though I'll hedge and say there isn't a really accepted definition of high frequency.
Algo trading on the other hand refers to using a computer to automate trading strategies. The typical example is a pair trade with Ford and GM. When they deviate you sell one and buy the other, when the prices come back together you reverse the trade and sell the first leg and buy the second.
Algo trading is here to stay. High frequency has typically been more arbitrage focused and its followed the pattern that happens to all arbitrage opportunities and gone away as more people got in on the action.
High frequency trading typically refers to two concepts. 1) how long you hold your position, 2) how quickly you can send/cancel existing orders. Though I'll hedge and say there isn't a really accepted definition of high frequency.
Algo trading on the other hand refers to using a computer to automate trading strategies. The typical example is a pair trade with Ford and GM. When they deviate you sell one and buy the other, when the prices come back together you reverse the trade and sell the first leg and buy the second.
Algo trading is here to stay. High frequency has typically been more arbitrage focused and its followed the pattern that happens to all arbitrage opportunities and gone away as more people got in on the action.
High-frequency algorithms are producing less profits. Other forms of algorithmic trading are growing.
You have to be careful with terms in this space (and many of them aren't even well defined).
There are algorithmic trading strategies that are high frequency, but not low latency for instance. Or low latency strategies that are not high frequency.
In my experience what people are talking about when they say that high frequency algorithms are producing less profits, what they really mean is that low latency arbitrage algorithms are not making as much profit.
Lots of high frequency algorithms remain profitable.
There are algorithmic trading strategies that are high frequency, but not low latency for instance. Or low latency strategies that are not high frequency.
In my experience what people are talking about when they say that high frequency algorithms are producing less profits, what they really mean is that low latency arbitrage algorithms are not making as much profit.
Lots of high frequency algorithms remain profitable.
Thanks for the comments!, this is just the tip of a huge iceberg down the water... As you'll see the logic is very basic, and the system is rather blind; but the main idea was to post this simple example because I noticed how unknown is this subject among programmers.
From this point forward, the imagination is the limit when talking about trading systems creation.
Any thoughts on how to implement a channel trading algorithm?
Very nice post. I love the illustrations.
Found it really interesting, I'm one of those that know nothing about FX, this has been very enlightening to me.
In case discussion goes cynical here, I can attest to the fact that this sort of thing can be done consistently and successfully for thousands of trades over months' time. These days though with HFT and "intelligent routing" your broker is your opponent in the equities world at least. When conditions become unfavorable to an algorithm things tend to fizzle out rather than wipe you out as long as you're not leveraged to the hilt. The point about optimizing is key: the best algorithm has the minimum number of parameters to tweak otherwise you're just curve-fitting. Whether you add value to the world is anyone's opinion.
Proceed at your own risk of time and money.