Friday, 4 April 2014

Flash Boys - Misleading information

Flash Boys?  I read the book last night.  Appalling.  I found it a well wide of the mark. It vilifies and accuses inaccurately. The pulsating vehemence of its message makes it a singly dangerous book.
I love Michael Lewis' storytelling. He really knows how to write a page-turner for a geek like me.

Working in an investment bank in 90s, Liars Poker had a cult status with the currency note based serial number poker he made infamous being pervasive with many traders in Sydney too. Without Moneyball there wouldn't have been a Soccernomics book to read. Soccernomics made watching football richer and, as a Liverpool fan, I'm especially liking soccer just now. So thank you, Mr Lewis.

The Big Short. Terrific read.  Hmm, but what happened to Paulson? A good story but not quite the full picture. No doubt about it though, Lewis writes books that many people find fun to read.

Nonetheless, Flash Boys is a stunningly dangerous piece of misinformation. The truth is quite a bit different to the words you'll read beyond its red cover.

I know just enough to be dangerous about this as I am a former HFT. I've traded a million index option contracts in a day in Korea on a system built from the ground up.  Back, in 2010, when people still used Blackberries, I got a call from Joanne, my broker in Toronto, half a world away, surprising me with the news that I had traded a bit over 13% of RIM at TMX for the day. Really? Some little tinpot firm on the other side of the planet mixing it up with big boys? Lewis is right there, you never know what strange firm may be responsible for a bunch of trading. The little firm I worked in had quite a few years of trading with only a few down days. Not quite as impressive as those firms with one or zero down days, but I'll take it. A team of mine also built a prototype sub 2-microsecond exchange matching engine and I'm the inventor on a somewhat dubious hardware matching engine patent. I like matching engines. I left HFT to do some cool tech but got trampled on spectacularly badly in court. In summary, I think I have just enough knowledge to know that there is quite a lot I really don't know. Though maybe just enough to call Mr Lewis out on some facets of his book. Just enough to be dangerous.

The IEX guys sound like great lads but let's start by looking at how to win on their platform. Their price is delayed fairly. OK. You are lucky enough know the price is going to be always delayed and thus stale at match point. The stale pricing is guaranteed by many miles of fibre in a shoebox. Great! Try to use all your inputs to deduce an impression of a better price in real time. Throw it at them. Send in limits or IOCs at your biased price and see if you land a trade in their time warp.  Better information exists on the outside of the delayed world.  Use it.  It's still a race. Slower exchanges are always problematic as the world knows better. IEX is no different. You can be played by better information or decisions. You don't eliminate the speed race. Hmm, it's not really much different to any other venue in that regard. There is always an arb in a world that has space and time as a feature.

In theory, there is no difference between theory and practice. Unfortunately, the real world does intervene on our clever ideas. It is wrong to think that there is not a continuum of probabilities and latency where views that seem absolute can't be replaced by a bit of educated guesswork with a virtual latency gain. Things are just not as simple as Lewis makes out. Some people think bunching trades into chunky time slices may be the solution. Such time point based auctions are not the answer. Even if you traded with an auction to the minute there is still a game of maximising your information at the deadline with latency tricks. So even slowing down trading to minutes becomes a latency game. A stupidly inefficient one, but still latency sensitive. Unintended consequences abound. You really do have to be careful what you wish for.

The clever thing IEX seem to have done is simply make a co-location space the size of New Jersey that includes all the other exchanges. It is still a game where speed matters and variances exist. It's just a bit different. Other market structure trade techniques still apply to the IEX context too, it just seems those get ignored in the story as inconveniences.

The IEX guys mean well. I just believe the ultimate basis for their thinking is, unfortunately, a bit wrong. Not a lot wrong.  Just a bit wrong. They are smart guys but I feel they have been skewered by their misunderstanding of Thor and HFT. When you have a hammer, everything looks like a nail. (Sorry, couldn't resist.) The result is they have run off on a bit of a tangent. It sounds a nice system, but I suspect it is not quite what they intended. There are many good features and reasons why it may make sense to use IEX but you need to get over the idea that it is perfectly fair and cannot be "gamed."

Clearly, "Thor" was a great tool. It was effective. However, I don't think the SEC guys were really dishonest in questioning Thor's goodness relative to the idea of over-provisioning liquidity at multiple venues. Poor form from Lewis in that regard. Tearing down ideas without proper consideration is never a good thing. The liquidity at various venues was real and could be hit individually. It wasn't fake. You must remember that the HFT market makers are scared rabbits fighting against adverse selection but needing to be stoic and fearless to maintain priority and sufficient size. It is dangerous picking up pennies in front of the market steamroller in the name of efficiency. The market maker may dare to put out more liquidity than they are comfortable with in an effort to cover all the bases. Odds are it is safe enough as they shouldn't get hit all at once. Thor! Thwack! Ouch! HFT market making is a thankless and tough job. Perhaps only exceeded by the thanklessness of being an SEC official that is wrongly ripped into by Mr Lewis.

Think about it. If some dude or dudette comes along and cleans your over-provisioned liquidity clock, then, as a market maker, you'll have to adapt and put out less liquidity or otherwise change style.  Is the gaming of the simultaneous orders with Thor really better than the over provisioning of liquidity? Thor's net result is to force there to be less simultaneous liquidity at venues and more bias toward particular venues. Is that a good result? I can see both arguments. There is certainly a yin and yang there.  It is naive to consider just one point of view.

IEX should be congratulated for their endeavours and especially their impressive ethics recounted in the book.  They have certainly tried very hard and it looks a decent enough solution as an exchange. There are also plenty of other ideas about making better exchanges. Then again, perhaps Lewis hasn't really explained it properly.

There are certainly great points in the book, such as the reference to the silliness of some of the inane order types. Some order types do seem to border on the ridiculous. As always, hindsight is a wonderful thing. However, reading the book you also wouldn't know that microwave wireless on the Chicago to New York link pre-dated Spread's cute fibre perhaps as far back as 2009. You wouldn't know that BATS was also the fastest exchange back in 2009 with a round trip of about 443 microseconds.  That was an important feature. Fastest makes a difference. You wouldn't know that there are natural advantages causing liquidity attraction in a game theoretic sense for the fastest exchange. I was horrified by the depiction of the flash crash. Frankly, Lewis' depiction of the May 2010 flash crash circumstances bordered on negligence. It's all a bit loose in the fact department.

There are lots of myths and folklore in trading.  Some are frighteningly unchallenged.  Did you know solely investing in index funds is a really dangerous idea? If everyone did it, there'd be no price discovery. The market would fail. That is an obvious statement but many would find it confronting on a first read. For some ideas, you don't need data, just as Einstein didn't really ride a light beam. It is wrong of firms like Nanex to ignore theoretical arguments, just as it is wrong to also ignore the data from the system. Many of the ideas that need confronting are indeed already well understood. It is important to remember a couple of key ideas: markets are best left alone to be the wonderfully efficient mechanisms they are, despite the motivations of their participants; and, complex systems, including human beings, need controls and limits. These principles are in conflict and need constant re-balancing but such ideas _and_ data can guide us.

Why has Lewis gone so wrong here? What's the agenda?

When I was 14 I read a story in the newspaper that I wanted to do Information Science at university. I didn't know what that was. I'm pretty sure I said I wanted to be a pilot.  Wrong message. I learnt at an early age that the truth of an article is often inversely proportional to how close you are to the story. Likewise, I find that I've been at least close enough to the HFT story such that Flash Boys doesn't stand up well to scrutiny.

Flash Boy's message was clear, but I don't know what the Lewis agenda is.  Perhaps he's just got it wrong? Against that, it reads like he has an agenda. Lewis needs to tell us how stuffed up the whole US equity trading system is and how you are being ripped off at every turn. Controversy sells. Hopefully, Lewis is just honestly wrong and not really being Machiavellian and trying to sell a boatload of books. He has been accused of just trying to whip up a frenzy to sell his book but I for one don't believe that. He is just pretty badly wrong in my view. I'm sure he'd disagree. Though it was annoying, I found it a fun read. Just don't take it too seriously. I had similar thoughts reading both Lance Armstrong's autobiography and Flash Boys, "Should it be in the fiction or non-fiction section?"

The investor, big and small, insto and retail, needs to remember that there has never been a better time in history to trade and get great pricing on executions.

Equities trading in the US is far from perfect. The system has a lot of good features and quite a few things that need some fixing. It always will, but it's pretty damned good. Flash Boys uses many woeful anecdotes to wrongfully undermine confidence in the national market system. Proceed with caution. Flash boys is a dangerous book.


  1. Liquidity, schmidity. It's still front-running at whatever speed, a service that costs the buyer without providing any benefit.

    1. what liquidity did they provide.... it was there already for the HFT guys to trade and get in front of. they only increased volume on the exchanges thus increasing exchange fees for the exchanges.

    2. Sorry, I'm not sure I understand what you're saying?

  2. More Competition. More quotes. Narrower spreads. Better prices. Better prices are a benefit, no? Or do you prefer worse prices?

    Think of the whole planet and not just New Jersey for a second. It's just the same but with bigger gaps between venues. The prices aren't always in sync. There are delays and discontinuities all over the system. Arbitrage makes it look to be a magically connected whole and serves the purpose of providing efficient pricing. Just because you don't understand it doesn't mean it's wrong. Front-running is illegal and an offensive term. Great care should be taken when using that term. Sure there are issues, but I think it's remarkable that the system is as good as it is.

    1. I think the point of the book is, How can prices be "better" if they are really just pretend, or undermined by traders who have a tech advantage that effectively lets them cut in line? Then they aren't really prices at all, they are mirages. That doesn't help anyone.

    2. You must have missed the part of the book then were Lewis described Brad's RBC experience of being able to hit all the orders individually. It was thus shown it was possible to hit all the orders at the same time with Thor. The volumes posted were not mirages.

      Think about it from a market maker's point of view. If you have forty venues to service, you can offer to buy and sell in all but you may not be comfortable with supporting all that volume on one side. So if you get hit on one side in multiple venues or at one venue with sufficient size, your natural inventory model says, I need to back off otherwise I'm taking on too much risk.

      The other main point is that there is no right to expect that an order would not have a price impact. That is the point of a market afterall. It should just be fair.

      We can agree that selling a billion shares of IBM should probably result in IBM shares being lower in price. They are unwanted by someone. So the finesse to that question is, "Where is the threshold for a minimum fluctuation?" I have a bunch of text books on my shelf that attempt to argue answers for that which get some of the way there. Simple answers really are naive.

      There are improper practices, and those should be policed, but the behaviour described in Flash Boys as vanishing orders, that could be "gamed" by Thor, and adjustments to prices is simply just the way markets should work.

      HFT market makers are scared of large orders, typically from instos. They don't want to get caught holding too much stock in the face of a large order that should have a price impact. Insto's are scared of HFT because they think they are somehow being gamed and the price impact is unfair. Both think the other is dangerous but you only hear one side in FlashBoys as the HFTs have no voice.

      Brad got carried away with Thor and thought he had a solution to a problem he does not fully understand. No one can truly understand it all as it keeps evolving but Brad has his own rose coloured glasses on.

      Part of the reason (only part) that HFTs like interacting with retail flow is not because it is dumb flow. It is often very smart. Importantly, it is not big blocks and thus it gets viewed as safe. An HFT can't be hit for six by a huge block or an insider view. A market maker lives in fear of adverse selection.

      "Better" is an academic quality question which is difficult to define. Smaller spreads are one sign of a better market. Markets are better due to HFT market makers.

      I don't believe IEX is really part of the solution. It has some good ideas and some real flaws. I could describe the exchange and some regulations that would be perhaps be more ideal but I'm not sure I'd be doing any future employer any favours as I dance for food.

  3. So the whole world should just invest in the same infrastructure to eliminate being "arbitraged" (aka front run) by some guys who think that they should be entitled to billions in profit because they can game the system? The real issue here isn't retail investors but institutional ones that get horrible execution because people like manoj think that their hypothetical utopian capital markets are efficient and anybody who disagrees is wrong. He came out of this as the 2nd biggest loser after obrien who outright lied. The nanex data shows that this crap is getting out of control and the collateral damage is huge. Now the pompous assholes think that currencies should be screwed with. After all, why hold only of the global population hostage to their .01%er game when they can screw over everybody by astroturfing entire counties' currency. It is all in the name of capitalism and utopian markets and perfect execution, who cares if every single persons day to day purchases of imported or exported goods are affected.

    You cannot help but think that these people think far too highly about their egos than what is actually just. Hopefully they get shut fown soon.

    1. An amusing rant, but a rant nonetheless. It just sounds like you just don't like markets. Even China saw the error in that thinking.

      As an insto with a large block, you should be getting advice and help from an expert, a broker, if you need help in making an execution efficient. Go upstairs and deal with that particular set of separate problems. If it was easy the ancient Sumerians would have solved it. Even when NYSE did most of the US volume, it was a dumb idea to just throw your huge order into the market.

      Markets are a bit like democracy. They have their problems, but they are better than anything else we have tried.

  4. And your blithe dismissal is no better than manoj's pompous douchenozzleness. I do not hate the market as I am part of it. However, your vlind adherance to your hubris is astounding. Just because you can game a system and screw investors doesn't mean you should. Further, even I'd you do and shouldn't doesn't mean you shouldn't be prosecuted for breaking the law. This is what will happen to front runners, hopefully. Funny you should mention democracy. Do you think what is happening is democracy? I doubt even 10% of the country would. That, my arrogant anf seld absorbed friend, is democracy. Not your silly notions of what you should be able to get away with.

    1. Apologies for the delay in approving your post as I was out grocery shopping with my youngest daughter.

      I'm not trying to say the market is perfect. The idea of of dozens of market places and a bevy of complex order types spread over the NMS fabric is quite crazy and confronting. The system is a monster.

      Trying to place a large block into such a messy fabric is a nasty thought. That is a real problem. TCA, algos and other tools help institutions to do this but it really shouldn't be as complex as it is. I think there is still an important role for brokers for exactly those reasons. It's a hard problem.

      My main point is specific to the Flash Boys book. I think that though Brad and Michael mean well, I don't agree with some of their hypotheses and I don't like the way the story was told by omitting or misinterpreting a few of the facts. Some specific examples of what they refer to as rigging, such as the "thor" example, I found misguided.

      Regulations are part of the problem as the US seems over prescriptive. Europe has nice principles in market reg that don't seem properly enforced. Canada and Australia seem to have a better balance, probably due to the advantage of reviewing the rest of the world before writing their own rules. That's just my pretty naive impression though, as I don't follow that space closely enough.

      I'd be happy to engage in a debate in the comment section about a specific practice if you want but I can't countenance a broad "the market is rigged" debate.

  5. Excellent article, perfectly on point.
    Point 1- trading costs, liquidity and execution have never been better for the "small investor". Dismiss him/her from the discussion of who HFT might be harming.
    Point 2- yeah, big buy or sell orders cause alarm bells to ring and as you said if you have a big block to sell you better have a plan to move it without everyone knowing you want to sell. That's just like any asset and any market.
    Point 3- HFT is certainly hard to explain to anyone and harder to defend as being good,legal,not front running and any attempt to explain it only makes the practice look worse.
    Point 4- US markets are the best in the world and as you say, not perfect, but better than all others.
    Point 5- HFT costs traders some fraction of a penny, it's a cost of doing business. Considering commissions-retail and institutional have dropped by about 99% since the early 70s, I'd say it has worked out well for all investors.
    Point 6- Michael Lewis is trying to sell books, that is his agenda. Case closed.

  6. A couple of interesting takes on Flash Boys that go into more detail than I do:

    An Amazon review
    A blog regarding bad journalism from Lewis

    1. An all too rare informed TV debate regarding Lewis' Flash Boys from Reuters via Youtube. One self termed whistleblower, Haim Bodek, and one mid-sized HFT practitioner, Manoj Narang.

    2. ‘Flash Boys’ Complete Nonsense: CMCRC’s Aitken

      From Tabb Forum, "CMCRC CEO Professor Michael Aitken, who has conducted multiple academic studies on HFT, says there is no evidence presented in the coverage stimulated by the release of Michael Lewis’s “Flash Boys” that securities markets are rigged."

      Youtube clip of Prof Aitken.

  7. Consumed the whole Flash Boys book this week. I am now not clear whether the HFT firms USE the infrastructure, or ARE the infrastructure. I have the feeling that you're in the camp that believes they ARE the infrastructure, or to put it simply, trading wouldn't be nearly as efficient without them. It seems to me that they are now what bonds the individual exchanges into effectively one big exchange. The opportunity is there to go from one exchange to another and take liquidity from one place and put it into another, yes? So they are the glue that has become necessary because there are so many exchanges. As a result of the fragmentation, there is now a new moving part to the market, and these guys do deserve to get paid somehow. The price is pennies per share, and they don't charge anything directly, they just make it through arbitrage they know will be there. This is the nicest way I can describe what's happening. I have the impression it doesn't need to work this way, it just does.

    I think what people find offensive is that HFTs haven't been blessed as part of the official market infrastructure, yet they're there and making money at an egregious clip. It appears risk-less, yet I am sure that that's the result of very careful execution and planning. A bumbling HFT could potentially lose out, but they're probably mostly smart organizations that have it all figured out in simulations and have tested their code before actually using it in the market.

    If someone were rash enough to legislate HFT firms and Dark Pools out of existence, what would happen?

    What I found disturbing (from the book) was the fact that new order types were invented to keep the HFT firms happy, that there are "make or take" incentives created that seem potentially manipulative, and that order flow is sold from my broker to a bank, who in turn sells it again to HFTs, and I really have no choice how or where my order gets executed.

    I also find it appalling that the SIP runs on a decade old Windows 2003 machine (this fact not from the book), and that the NASDAQ wants to rid themselves of responsibility for it. It's the consolidated quote system for the greatest market in the world and it runs on a ten year old machine running Win2003? LOL! Jeezus.. it's a wonder we don't shut down every day. What if the SIP was upgraded and updated more quickly? It seems that would cut down on the HFTs ability to use arbitrage. To a cynic, this would explain why nobody has upgraded it!

    As far as IEX, I wish them the best. I'm glad they're there. I'm sure they will still be gamed. I don't think their Thor counter attack on HFTs is unwarranted or to be squelched. It looks like another step in market evolution.

    My prediction: bloviating congressmen will call HFTs and bank officials forward and nothing will come of all this. I wouldn't mind if they mandated that exchanges and brokers allow the little guy to optionally send their order to an exchange of their choice, however.

    1. @Unknown

      That is quite an insightful comment. Thanks.

      1) HFTs USE or ARE the infrastructure?

      Excellent question. In terms of USE or ARE the infrastructure, I think a little analogy may help. I served a bit of a trading apprenticeship back in the early nineties at Bankers Trust in Sydney. I worked nights for about six months with a voice box to the CBOT 30 year t-bond pit. You could shout down orders to the floor guys from Sydney and hear the Chicago pit at the same time. Now, the biggest nights were typically non-farm payroll nights on the first Friday of the month. If an employment number came out that was a bit different to consensus, all hell would break loose in the pit. The amazing thing would be that it could be trading a big figure, or 32 thirty seconds or ticks, apart at different parts of the pit at the same time! If you looked at the bids and offers on Reuters it looked quite orderly but it that wasn’t reality. Essentially the price discovery taking place from all those traders, acting like mini market places within the t-bond market, was kind of messy.

      In terms of US equity venues, mainly in New Jersey but even in Canada, Mexico, Brazil, there needs to be some force to keep prices in line. Traders who see different prices in different markets will jump all over it to try to get a “risk-less” profit and it all evolves from there. Though it is never risk-less as the legs of the trade are not atomic. Arbitrage is important to keep things well priced. Indeed, no-arbitrage theory is also a fundamental theory that drives many derivative pricing models too. The potential to arbitrage forces prices into line, but someone has to actually do it.

      However, it is not quite so simple with the US equities as Reg National Market System (NMS) also having an effect. You have to route your customers order to the best price. If you trade unusually outside of the the National Best Bid Offer (NBBO), which is provided by the SIP, you get audited by the SEC with a, “Please explain!” So Reg NMS should prevent the t-bond example of crazy simultaneous pricing but with some burden on inter-market routing. Who should really do the inter-market routing is an interesting question.

      It is not really pennies per share that HFT makers gain. Think sub-pennies. In the RIM example from my experience I gave, I think it was a pretty high take from memory at around 0.5 cents per share. In the Korean example it was typically more like 10% of a tick (sometimes 50%, sometimes 5%).

      However, there is any number of strategies being played out by various participants. The so called “rebate” trade is such that traders are happy to trade at the same price and make nothing from the trade if the rebates they can get for posting liquidity exceed any fees they pay for any takes they are forced to do. If you can do a lot of volume you may be happy with 0.05 cent a share. It is a game of tiny tiny edges and trying to get enough volume to make it worthwhile. Some exchanges offer market maker programs, such as the DMM at NYSE, where benefits to execution exists in return for taking on some guarantees to your responsibilities to quote. Much HFT market maker activity replaced old school DMMs and other similar programmes as they turned out to be better at making markets even without the special benefits. You’ve seen the old specialist firms be road kill to some extent with GS rumoured to be looking to unload their $6.5B acquisition of Leeds and Kellogg in 2000 to IMC for $30M. Creative destruction indeed. That is a clear example of the reduction in value the middle-man due to the increased competition from HFT market makers. This old blog entry describes this essential truth about how HFT market makers make money.


    2. So I think the HFT you refer to mainly USE the infrastructure. Market maker programmes sometimes are such that they ARE an essential part of the infrastructure. The emergent property of it all is price discovery, or THE MARKET, just like the individual traders in the c-bot pit at the other end of my voice box in the nineties.

      2) Legislate HFT and Dark Pools out of existence? Interesting idea, except trying to find a definition of HFT is problematic. HFT is an emergent term for proprietary traders who trade a lot with quite a few variations put forward for the definition. HFT market makers are one subset of HFT. Also, if some one does a lot of arbitrage then they are an HFT even if they would only “take” prices. By banning HFT you’d be saying that when markets are out of sync then they shouldn’t be brought back together. That is a hard idea to put forward. You need speculators in a market to make it efficient. I’d also prefer speculators that make very little on each trade than speculators that make a lot on each trade. More competition is better.

      Dark Pools are a vexed subject. What if they were called Price Improvement Pools? The trouble with dark pools is that on a theoretical level they subvert price discovery. You have the same problem with passive index funds. Dark Pools provide a useful service in that blocks can meet without price impact if done correctly. So, both Index Funds and Dark Pools provide a useful benefit but subvert price discovery which means they can only exists as a fraction of a properly functioning market place before they lead to unintended consequences.

      3) Order types. Your point on the order types is well founded I think. However some of these come about from pressure to serve a liquidity provider. If you rely on trading at the same price in various places, aka the rebate trade, you are very sensitive to your execution costs. You only want to post liquidity and not take it, thus an exchange invents post-only type trades to serve that purpose. It helps people focus on posting liquidity to trade against and may indeed be healthier for a market place. You can understand the motivation for some of the orders but they can seemingly quickly get out of hand and really strange, especially when considering the myriads of routing options that are often embedded in them too.

      It must be remembered that it can be hard to get markets to function well. The CME FX futures launched in the mid 70s but didn't really become truly liquid until the mid 2000s, around 30 years later. Exchanges are sensitive to find ways to encourage the posting of liquidity and build markets hence the variations on order types. Sometime they go too far IMHO, such as the flash order types that were used by some equity market places for a while. You could understand the justification as the exchanges involved were trying to reward market makers but they did seem subversive to the idea of an open market and thus a bridge too far.

      You’re right about the routing issue. Brokers need to route your order to the best price under Reg NMS. Most HFTs sign a document away from this and use flags in their orders to prevent routing or to choose explicit routing policies as they want direct control to achieve some kind of certainty. Routing also adds a further cost to your order as someone gets paid for the routing. Knowing how to route your own order well is problematic and most smaller investors may indeed be best by leaving that decision up to their broker.


    3. 4) SIP

      SIP is an interesting issue. It is always behind the game. There is not much incentive to update it. I'd imagine that running it would be just like being a goal-keeper in soccer. People only notice the screw-ups and not the saves. Making the SIP better would help things but it is always delayed by definition and we’re only talking about matters of degree. From what you say it does sound a bit creaky indeed.

      HFTs fight the same issue in co-location too. The truth of the auction is in the matching engine. The data feed, even the direct fast one in the co-lo, is a rear view mirror into the truth of pricing. There is no fundamental truth across all the market places, nor even in a single market place. It is all shades of grey in the price discovery.

      Making the SIP faster would help tighten things up as the NBBO is used to police the quality of executions for customers as long as there is recognition that perfect timely execution is not always possible.

      5) IEX

      Essentially it’s a slow exchange with a really big co-lo (a bunch of data centres in New Jersey). It has simple order types which is good, but so do others, such as Nasdaq Ouch. It has an ownership model biased to one element of the market that is different but not necessarily better.

      IEX also doesn't have rebates for posting liquidity. That is, no maker-taker pricing. That is an OK choice but it is a choice. If you think about it, someone posting volume passively is giving a tiny option to the market. They are at risk of being traded through and losing money. How do they get paid for that option? They hope to earn a benefit by earning a bit of the bid-ask spread more often than not, or, to encourage the risk of placing that micro-option the exchange could rebate them a fee. That is, rebates may encourage the posting of liquidity and thus may improve the health of a market place. IEX chose not to do that and expect the makers to earn their rewards through the spread, which is one choice.

      I’m not a fan of the IEX model as I don’t see the need for its specific circumstances, but I also don’t see that it harms the market.

      I agree with you that Thor is a reasonable counter attack, but it is that. It is an attack. The widespread use of simultaneous multi-party trades would cause changes in behaviour just as other trade strategies do. Indeed, it is not too different to a fixed income desk coordinating calls to other desks to place a large order to minimise leakage and grab what liquidity they can. However, it is important to see that Thor-style routing is just another element of the “game” and not necessarily better or worse than other legitimate strategies.

      6) Congress. That is way beyond my simple comprehension.

  8. A very well written article and comments. I have just read Flash Boys and in my view it was very poorly researched and provided a one-sided view with baseless accusations. It is based purely on Katsuyama's view and therefore his marketing. The only thing I would add / echo to your comments are:

    1) Front running - everybody keeps using this term but in fact what Katsuyama was experiencing was liquidity fading, HFT firms were not going ahead and buying the shares on the other exchanges but pulling their quotes once they had traded on the first exchanges. It's a bi-product of fragmentation, if we move back towards a central limit order book then we won't have this problem but we then have a lack of competition and potentially a monopoly which as history has shown us keeps prices high and doesn't invest in the platform. There is no right / wrong solution to this just an understanding that it is the trade off you make.

    2) THOR - Katsuyama & Lewis continually imply that most brokers use sequential cost effective routers which is simply not the case. RBC had terrible technology and hit rates of 70% because Katsuyama had instructed the system be configured to go in entirety to one venue first. Just because he did something stupid doesn't mean everyone else was doing the same thing. He then moved to a simultaneous spray and got rates up to mid 90's percent. Still a bit lower than most other brokers at the high 90's because RBC had not invested in their routing architecture. To close this gap they came up with THOR which was a clever alternative to investing in their connectivity. Unfortunately taking liquidity in displayed markets is only a small subset of what institutional investors need from an electronic trading firm and Katsuyama found that a one trick pony outfit couldn't get paid across the board and such he decided to solve the fragmentation issue by setting up another exchange.

    3) IEX - yes the delay they have introduced will stop market data latency arbitrage in their dark pool, several other broker run dark pools employ various techniques to protect against this. Outside of that the delay is meaningless, as you said slow exchanges have always existed and they don't stop you getting picked off. A really poor analogy (apologies) is like having an automated revolving door at the entrance to your building, which is set to a fixed rotation speed and slows the rate at which people can enter/leave the building. It doesn't effect how quickly people get from their building to yours though, so speed is still an advantage. Katsuyama has also suddenly become the authority on good and bad interactions with Dark Pools, and unsurprisingly RBC (whose dark pool strategy he no doubt had a hand in designing) is considered good. They have no idea on what the brokers are doing in other dark pools and can therefore not opine as to whether the job they are doing is good or bad, that's for the clients to decide.

    4) Order types - whilst I agree with you that the proliferation of order types have in most part been produced to provide more efficient liquidity provision, they have also introduced complexity which can be gamed and potentially the market could to without - knowingly at the potential cost of slightly less liquidity available at any one point in time. This again is a trade off the market will decide upon as it evolves.

    1. Interesting comment, thanks.

      I think our thinking is alike in many regards. You clearly have a little more expertise than me though.

      We're certainly on the same page with order types. I can see the justification for their accretion but I wholeheartedly agree that their proliferation is troubling. I certainly know I've managed to find weird ways of using particular order types to eek out edges. An exchange user needs to eliminate disadvantage, so you have no choice but to do so.

      The hard thing to figure out is what should be done which would still allow innovation with respect to order types. It seems retrograde to simply stop innovation. Maybe we should stick to a fundamental set of simple order types and mandate that all other order types be implemented with those elemental order types only. That would not be enough as it doesn't eliminate the latency advantage an internal exchange based state machine may have. For example, an iceberg can easily be replicated with a simple state machine but is it really fair to allow those slightly more complex orders into the core? They certainly can be convenient for customers though. Some exchanges are already architected this way where they layer more complex protocols on a simpler inner core.

      At the other extreme we could allow direct graph execution of atomic ops (so it always terminates in <N steps) as an order type in the exchange and let people do what they will. Curious idea. Not a big fan.

      I like the atomic orders idea, perhaps with SEC controlled review, and allow layered protocols at the same architectural level as client interfaces so there is no inherent advantage. I like this as it seems a bad idea to prevent exchanges providing customer convenience as long as it can be fair.

      A bigger idea is that a group of diverse professionals form a group with enough influence (aka market share) that they can agree to a set of standards and schedules to make markets better. The group could simply insist that no member trade with any body who trades with anyone that does not comply to the "code" (to borrow from Flash Boys). That way the "code" is voluntary yet viral.

      Perhaps it is time for the industry to take its fate into its own hands?

  9. I need to read the article in depth to better understand your point but this part catched my attention:
    "The clever thing IEX seem to have done is simply make a co-location space the size of New Jersey that includes all the other exchanges"

    In my understanding of the book an IEX solution the point is that they want to provide an exchnage where, if you place your order as an investor, no HFT or anyone else gets a competitive advantage from whitin the exchange. Outside the exchange is still a wild world, the rules of the faster still apply but inside the exchange it is fair. I imagine a fence, a 350us courtyard where if you see that you can buy 10000 Intel at 21$ you can actually buy that. If an HFT has that 10000@21$ he cannot cancel it as he will do if he gets a signal and no other better price is there. My understanding of the front-running described in the book is that I try to buy 10000@21$ and by going to BATS and buy 100@21 I tell to HFT I'm there for a buy, they front-run me to other exchanges and wait for someone with qty@20.9$. They buy that qty and sell it back to me at 21$ or cancel their orders before I arrive.
    Inside the IEX courtyard they cannot do that anymore, outside they can still do it. So it is pretty clear to me what the IEX solution bring to the game: I can buy at 20.9 if it is inside IEX and HFT can't do anything about it.
    Am I missing something or have a distorted view?

    1. (1)
      Most modern exchanges have a co-lo and a matching engine. IEX is no different if you think of its equidistant cables as simply making its entry points a big co-lo space.

      Once you realise it is no different to any other exchange, just slower, you see all the normal "games" apply. It is a race to get an order in. It is a race to cancel an order to prevent adverse selection. It is a race. There is a real world outside no matter how it is spun. Think of any news, such as an interest rate cut, including market prices from other venues, it is a race to adjust.

      The simple order types IEX have are a boon and make it even clearer it is just like a normal exchange, just slower. The Nasdaq protocols, Ouch / UFO, are neat and simple too.

      So the buy side owns it. That's one segment. How does it stay fair for the other speculators that assist it to function? Markets need short, medium and long term speculators. How does it stay fair for the companies that are having their stocks traded?

      The large investors are playing it dumb. If they were cleverer they'd realise the economics of markets are such that they should be using the HFT crazy competitiveness to help themselves. Zen, Art of War, judo, whatever. Use them against themselves.

      HFT is hyper competitive. Smaller margins constantly squeezing each other out. Use that force against them. HFT's actually don't like fast, low jitter level playing fields with simple order types as they find it hard to get an edge on the other HFTs. They do like the set-up, as it is simple. They also like the environment as it reduces risks as they can select, hedge, cancel quicker and do more trades. More trades makes the odds better. Make a hyper competitive exchange, one with really low latency and the buy side wins. The buy side just think they don't.

      Algo's are here to stay. You're never going to find a way in a public market place to place 10M shares in one order without impact. It has to be a bespoke hidden match with no information leakage or broken down into pieces.

      Make a really low latency exchange and there is more natural liquidity from HFTs. If you have a 10 microsecond exchange compared to a 1000 microsecond exchange, you do a hundred transactions whilst the other exchange does one. People can hedge out fills and take on more risk. It is safer. There is more natural liquidity from sophisticated investors via correlations and inter-market trades. HFTs will get hit, hedge and repost liquidity. Liquidity replenishes to the point of price impact without even needing to go to another market.

      The fastest exchange, that is also fair, is the natural place for liquidity as it is the most attractive for managing risk. All other things being equal, which they never, liquidity would tend to grow at such an exchange. It is just a natural game theoretic outcome from competition and risk.

      It's not a message the buy side can generally stomach but the smarter buy side get it.

    2. (2)
      More competition is what we all fear as some fail and some succeed. That is a market. All the participants in the market place are the same in that regard. The buy side fear the rest of the buy side getting better alpha and taking their customers. The HFT guys and dolls fear getting squeeze out by better quants, tech and strategy. It is not just each other they fear.

      PFOF, dark pools and index funds are all parasitic to the market but with some benefits.

      There are too many venues. It feels like 5 is not enough and 20 is too many. Pick a number. Perhaps licenses should be like spectrum or some other limited resource sharing scheme.

      Maybe order types should be centrally controlled so there can only be a certain number, including some experimental ones? How can we innovate on order types and still control them?

      Exchanges need their own 15c3. Risk should be checked on every order.

      Perhaps market solutions could be used instead of some regulation. 15c3 does not always work in reality. Knight were fined by the SEC over it. What if direct access required an insurance premium to be paid by someone with a significantly larger, independent balance sheet to you? Premiums and risk assessments would focus the mind and operational risks.

      Whilst Flash Boys is great marketing. Marketing doesn't solve real problems.

      There is a lot to be done, but IEX is, well, a nice try, an honest and ethical nice try.

    3. Thank you for sharing your insights. I agree with some parts of your reasoning, the parts that I think I understand, but I also feel that your focus is on the overall market while the IEX solution is for a particular problem, some type of HFT. Inside IEX there is no race, that's it. HFT is blind there. In my understanding, it makes all of the difference that the cables are long: if you learn about a better price you cannot get out of the IEX courtyard fast enough to use that information.

      To call Flash Boys "great marketing" can only be done if you overestimate the IEX solution and throw a lot bigger problem into their hands, conclude that it's still not fixed it and then thinking that they did not fixed anything if they don't fix the "real problem". Separation of the problems will help people better understand the "real problem" and I see IEX as a first step into that direction.

      I might be a marketing victim but what I really enjoyed is that a guy (backed by buy-side) made this step from the inside. It was, and it still is, neither easy nor risk-free and I'm more willing to focus on what he achieved and what still needs to be done than to dillute that and then say, for internal balance: "There is a lot to be done, but IEX is, well, a nice try, an honest and ethical nice try". Changes will always come in small steps and the traction this thing gets it is a chance to real change.

      Keep in mind that any theorethical solution has the disadvantage that it does not yet exist. The practical ones exist, so let's give IEX a bit more credit for what they achieved until now.

    4. Thanks for the comment. I appreciate the debate.

      Sorry that I haven’t explained myself very well. I’ll try harder. My wife is a school teacher and shakes her head in disgust when I try to explain anything. There is always a good chance I'm just wrong. So, as long as you have low expectations, you shouldn’t be disappointed.
      It may be a good idea to read point 2) from Anonymous 8 Apr 2014 02:59:00 above.
      Katsuyama’s use of Thor was not really a new idea to the rest of the market. Indeed, it was a technique he was perhaps a little slow to arrive at. It was just one of a vast number of algos.
      Another way of thinking of him saving money with Thor is that he was harming his customers without it. Where did the magic saving of money come from he said he saved his clients with Thor? It was provisioned by those HFT market makers. The HFT market makers were saving him money, just as they were also saving other brokers money on their executions when those brokers were using their own less or more sophisticated algos. You can look at it two ways: he saved his customers money with Thor thanks to the HFT market makers; or, he was behind the times previously and costing his customers money.
      I’m not alone in thinking that the demonstration of Thor is actually a pretty good example of the benefit of HFT market making.
      Bloomberg’s Matt Levine wrote a nice article that showed how you could reinterpret Lewis’ book with HFT as the hero, rather than the villain. I certainly don’t think of the HFT market makers as heroes. They are profit-motivated capitalists that make the market better DESPITE their motivations. That’s the magic of markets. They generally work due to the competition they facilitate despite the motivations of the participants.
      Getting back to your example regarding IEX. My explanation was bad enough that you missed my point, so I’ll try again. You say words the effect of, “no one can get a competitive advantage from within the exchange” and “outside the exchange it is a wild world” which is true. It is true for all exchanges isn’t it? Orders come from the outside. The race is from the outside. IEX is still a race.
      You’re right if you see 10000@21$ at IEX and you send in your order to match, no one else can see that trade and move their prices at the other exchanges. I wonder if any other exchanges are slow enough to also have that feature? IEX may also be able to route that order to other exchanges to perhaps grab similar prices. That is not new though. You can still do that by structuring your trade timing and routing appropriately in the world before IEX.  All IEX is doing is embedding a particular algo into the platform. That is a choice. A fixed one. What value does IEX provide? I’m struggling to see it.
      All of the outside world signaled trades are still races.
      You’ve also just highlighted the essence of the major problem with slow exchanges. Trading on them puts you at greater risk because your information is older and thus liquidity providers are more susceptible to adverse selection. You can guess what happens in that case. If a market maker is more susceptible to adverse selection, they have to quote less aggressively. Spreads widen as a consequence and prices and liquidity will end up being worse than it would otherwise.
      Inside any exchange or ATS there is no race. There is a race outside. By being slow, IEX can route to other marketplaces without leaking the trade. Algos already can do that. Remember, one dangerous natural consequence of a slower exchange is pressure towards less liquidity and wider spreads.
      I’m trying hard, but I'm still struggling to see the value IEX adds.
      Consider this Gedankenexperiment. Say an exchange, called F, trades with a 2 second round trip time(RTT). Another, called S, trades with a 2 hour RTT. Say the market has been stable for the last 2 hours. You’d expect the prices in both to be similar.


    5. An event occurs which is positive for the market.

      Traders go to hit the offers in both F & S. You won’t know about S for a while. A few seconds pass. The market keeps going up at F. You don’t really know what is happening at S as it takes 2 hours to find out. Fire in more orders into S at what may be an appropriate price using F as the benchmark, as no one knows better. It is still a race to get those orders into S before others as S still has time priority and you’re not going to be alone in your view. You can guestimate your S fills and hedge, or not, by those estimates. Where would you hedge? You’d probably try F. S gives you uncertainty and you’re less likely to want to trade there but it is still a race.

      If you’re making markets in S. You’re getting pretty nervous as you can get filled and not know it for a long time. That is, you can’t hedge for a long time. You’re inclined to back off on your pricing to cope with this. You’re also going to offer less volume as you’re uncomfortable with the risk. However, you can cope to the degree you’re fast enough and your analytics can suggest or model likely outcomes. You try to react faster and cancel before your stale, bad prices can hit as it is still a race. You can try to take stale prices by reacting quicker than others can cancel. It is still a race, just a lot messier. At S there is a lot of guesswork, uncertainty and games of ducks and drakes going on. Hopefully now you can imagine with a two hour wait it can get real messy trading at S.

      Traders will have more confidence in trading in F. Hedging, trading, cancelling is more certain. F also becomes the benchmark for prices. Traders will use F to shoot trades at S trying to get value. F becomes the benchmark and the “go to” place for liquidity and hedging. S becomes a bit of a lottery or a shooting gallery with lousy prizes.

      You may be able to see there is a whole world of pain in modelling what may be happening at S based on: empirical histories to guide likely events, the volume at price levels, and the expected behaviour of other participants in reaction to the real world as represented by F, et cetera.

      You can probably also appreciate how complex and uncertain the modelling and operations can get with lots of exchanges, different bandwidths, latencies, order types, protocols and other traders. It is kind of amusing, or sad, depending on which hat you’re wearing, how HFT market makers are climbing all over each other with greater and greater sophistication to make less and less.

      I hope you may be able to understand a little better how a faster exchange has a natural tendency to attract more liquidity and better prices.

      IEX is a slow exchange by design. In the world of the nanosecond, this matters.

      The IEX delay model embeds a fixed choice of algorithmic approach for a client that would otherwise be available anyway.

      Now, NYSE was a slow exchange for years but maintained good share. There are plenty of other aspects to the quality of a market place but competitive pressure is always being applied to a slower exchange. It is more natural for the liquidity to end up in the better risk environment of a faster exchange.

      It’s actually not too hard to build a 10 microsecond RTT exchange with simple order types. Two of my teams have built two prototypes in the last seven years. One with FPGA hardware and one with software using 10G NICs. I’m pretty sure BATS, Nasdaq and plenty of others could too if they put their minds to it. Technically, in 2014, it is just doable to be able to build a sub-microsecond client to exchange RTT experience that would support a few thousand traders’ systems.


    6. I argue that a faster exchange is a better solution for the buy side. It is kind of funny how HFTs’ natural instincts are to like a faster exchange and the buy side’s natural instinct is to recoil in fear. It should be the complete opposite. Funny world.

      At the end of the day though, I’m only guessing at the behaviour of IEX. I’ve read Flash Boys, read through their ATS rules and order types but that hardly gives me a full understanding.

      I’m all ears as I’d really like to understand why IEX is a better market place. I’m not the smartest kid on the block, and I do have a prejudiced history, but I hope that I’m at least rational enough to get the concepts if someone can explain it to me a little better. I’m obviously having trouble in getting it. Please just type slowly.


      PS: I'm a bit loose with my language so I may be being hypocritical, but please, please be careful with the term, “front-running.” It is illegal. Arbitrage and adjusting to new information is not front-running. That is just how markets work. Trading has price impacts. Markets are price discovery machines.

  10. Gell-Man Amnesia
    Murray Gell-Man won a nobel prize for physics and was at Caltech with Richard Feynmann. Michael Crichton describes it.

    “Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
    In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.”

    ― Michael Crichton

    You wouldn't have a board of directors audit committee without a qualified accountant. It seems the same isn't true for the oversight of decisions involving either the expenditure on, or policy and regulation surrounding anything whose manifestation is one more more computers.

    I'd compare this to the NSA debacle along the lines of "intelligent people coming out with obviously, bizarrely, ridiculous pronouncements that don't result in the expected adverse consequences to their reputation for intelligence." Our grandchildren will laugh at us, at least I hope they will...

    1. I like your Gell-Man Amnesia reference. I don't think my wife and I have ever had an argument where it doesn't feature. Naturally, I'm always wrong.

      I hope I'm remembering the Flash Boys period correctly but you never know I may have my own convenient biases too.

      Dan Ariely had interesting things to say about his otherwise good doctor turning into a bit of beast when his motivations were conflicted by his need for a research participant in The Honest Truth About Dishonesty. We're all cognitively biased meat machines. I know it's wrong to think I can overcome my biases and prejudices but hopefully I can be aware of them.

      Scott Adams had this nice philosophy that the world was put together over a few hundred years by a handful of really smart people and somehow it all works but no one truly understands how. I think we saw manifestations of that with the GFC when the Fed reached deep into the rule book and did things most of us didn't even really know were possible at the time.

      The markets feels a bit like that. There are weird things in the dusty corners. I find the fearful thing is what we all know as the systematic risk to the system we saw in the May 2010 Flash Crash. I'm sure there is more that we could do there but there is also an element of the impossible I suspect.

      I doubt there is any way to make the chaotic system formally robust but more robust is a noble goal. If only we could consult with Mandelbrot. I would think that crashes should never be excluded, just calmed, as such price corrections are certainly necessary in world where Animal Spirits and bubbles are impossible to eliminate.

      I'm pretty optimistic about water, food, energy, life extension, blah blah but not so much about nukes and the doomsday clock. A particular constant in Drake's equation scares me. So, I'm with you. I hope our grandchildren laugh at least us much at us as my children already do :-)

  11. A tight spread does not necessarily make for a good price. A good price is execution price times the number of shares you want to buy or sell. HFT shops go home flat. They don't trade on fundamentals. So what drives all that volume? They jump in front of orders, both price and market orders and then transact with the original order. It is not illegal and one might even say it is not unethical. But that's what they do. Lewis may have made numerous errors and may have no solution to offer, but he got this part right.

    1. Some HFT shops go home flat. Not all.

      Think of a pair trade. Two stocks dancing where the view would be if they are out of sync, sell one, buy the other, and then hold until in sync or at loss point.

      Now, do that with N securities with better math. Do it within a sector, market, across asset classes, whatever and you end up with a lot of trades which may have absolute positions overnight as a statistical arbitrage. That is still HFT.

      Adjusting for new information and altering the price is just what markets do. Price discovery insists that if buyers want to buy more than sellers want to sell then prices should go up.

      There are plenty nefarious activities that go on but being so broad in accusation is not sensible. Specific circumstances need proper review and not the ignorant hand of Lewis.

  12. Just get rid of the HFT's and make a level playing field for all.
    Have one exchange - no dark pools.
    No time advantages for anyone.

    And maybe the SEC could really do their job.

    Sometimes simpler is better!

    1. Simpler in text does not translate to real life unfortunately.

      One exchange means monopoly rents which is why we need more venues and price competition. Thirteen exchanges is not too bad, but forty odd dark venues or ATSes and over two hundred internalisers is a crazy situation especially when combined with PFOF.

      There is always a time advantage, even if that is implicit by the maximal use of information to a deadline. You're not going to eliminate the technology war. That is a good thing due to the efficiencies we now all benefit from.

      The system is far from perfect so perhaps the SEC could do better but no one is perfect so we should be careful about casting stones at someone else's glasshouse. The system has improved over the last twenty years and we should thank the SEC for that at least. I really felt sorry for the scorn Lewis poured upon the SEC official in Flash Boys who, quite correctly in my mind, pointed out that Thor was not necessarily pure goodness relative to multi-venue HFT market making.

      Lewis has poured uninformed manure onto the embers that will smoulder for a long time with much heat, little light, with a hint of stench. It is sad he has used his considerable profile for such damage. There has never been a better time to get great executions for all traders, especially Joe Blow on Main Street.

      I agree simpler would be better. Just not too simple. We have to be careful what we wish for.