Tuesday, 8 April 2014

Flash Boys' sinister ending: who owns the new evil microwave tower?

At the conclusion of Flash Boys, Lewis holds out a challenge to find who owns the tower with the mysterious FCC license plate number 1215095.

"The application to use the tower to send a microwave signal had been filed in July 2012, and it had been filed by ... well, it isn't possible to keep any of this secret anymore. A day's journey in cyberspace would lead anyone who wished to know it into another incredible but true Wall Street story, of hypocrisy and secrecy and the endless quest by human beings to gain a certain edge in an uncertain world." 
 Part of the last paragraph of Flash Boys

This final piece of the narrative paints this particular microwave New New Thing as the next, somewhat sinister, step in the evolution of speed. Spread Networks was one piece of secretive infrastructure used to help rig the markets. Now it is the turn of evil microwave. Who owns this new form of evil to help the HFT pirates continue to rig the financial markets?

The internet has reacted! A bit of frenzied internet sleuthing has subsequently taken place over a number of blogs, tweets and web sites. A number of parties have reached the same conclusion.

Tradeworx! Mr Narang confirmed it to the WSJ. They're the new bandits!

Sigh... I don't think so. Read on.

I found the ending of Flash Boys one of the saddest bits of the Lewis book. I feel it was sad for journalism, unless you support deceptive journalism.

Mr Lewis goes to great pains to weave a narrative around Spread Networks. I think you'd have to agree it is a key part of the plot. He then throws out the New New Thing of microwave at the conclusion of the book with clear sinister overtones. You could almost hears the Jaws theme in the background. Good people have been hunting down who could it be doing this New New Thing of evil that was the evolution of speed after Spread Networks? A participatory cliff hanger to ferret out the perpetrators was a neat literary device.

Surely it would have been responsible journalism for Lewis, especially when writing on the scale of a book, a Lewis book nonetheless, to “google” the topic so he could have found out that his narrative was wrong and RF microwaves had been used on the Illinois – New Jersey link since 2009, BEFORE spread networks went live.

It took me less than thirty seconds to get a reference to a credible microwave 2009 story. Here's an article I've tweeted previously from the Chicago Tribune from 2012.
"He [Benti] said the microwave network starts at 350 E. Cermak, ends at another telecom hotel at 165 Halsey St. in Newark, N.J., and went live in the fourth quarter of 2009."
Did he ask Spread Networks? Lewis can remember hand movements and idle moments from conversations of meetings with enough clarity to include much minutiae in the book. Maybe Spread Networks didn't know about microwave when Lewis talked to them?

From page two of the Chicago Tribune article,
"Spread Networks CEO David Barksdale dismissed such criticism. Yes, data travels faster through the air than through fiber-optic lines, but severe weather events can disrupt microwave signals, and those signals can't carry as much data at one time as fiber can."
I think they knew.

Is it shoddy journalism from Mr Lewis? Perhaps he didn't want these particular facts to get in the way of his significant plot device. Now, if Lewis had included the microwave links from 2009 in the story, it wouldn't have necessarily changed his message. He should have been more aware and not so inaccurate. If Lewis can't get the basics of his main Flash Boys narrative right, you need to take everything else with a pinch of salt.

Also, why is Tradeworx copping flak for this? This fact of Tradeworx owning microwave was been public for sometime. Wired ran a high profile story in 2012 that even had a big coloured box with the Tradeworx name on it. There was even a price tag! Hardly a complete secret.

From the wired article with Tradeworx hidden in plain sight
In the words of the great Lleyton Hewitt, “Come on!”

There is no news here, just scuttlebutt and harmful innuendo. Lewis is implying Tradeworx is rigging the market. They are not. They're just collateral damage in Lewis' missteps.

How can we trust the story telling if the main plot device is so twisted out of shape?

Ought Lewis have known more about microwave?

Jim Barksdale is the investment force behind Spread Networks. Is he also an investor in IEX? Can't be sure, but his fellow ex-Netscape CEO Jim Clark is on the IEX board. Obviously these guys are acquaintances of Lewis from the New New Thing days. A Barksdale, David, remains CEO of Spread. Lewis could have asked them. Any of them. Based on the Chicago Tribune story, they ought to have known.

Lewis didn't even need his contacts here. He could have clicked "search."

Incompetent or deceptive?

You decide.


An example of the disclosed use of Spread Networks fibre:


Getco use of Spread from their S-4:
"Colocation and data line expenses increased $18.9 million (52.0%) to $55.2 million in 2010 from $36.3 million in 2009 primarily due to the introduction of Spread Networks, which is a fiber optic line that transmits exchange and market data between Chicago and New York, and the build out of GETCO’s Asia-Pacific colocations and data lines."

Monday, 7 April 2014

HFT - two choices: making money every day or oblivion

If an HFT makes money every day the system must be rigged!

It's not rigged.  It's just math!

I see since Flash Boys came out, HFTs have been copping flak for making money consistently.

Virtu had one down day in 1200 or so days. It must be rigged.

Well, the simple fact is that if you don't make money every day, then you're doing it wrong!

You do, or you die. It's that simple.

HFT implies that you do a lot of trades. If you do a lot of trades with a smidge of favourability, you should always be winning. It's just simple mathematics. The law of big numbers. A systematic screw-up for a day may cause an odd loss but that is a different risk category. You seriously wouldn't be doing a zillion trades of anything without an overall positive expectation. The law of big numbers also works against you. If you're on the wrong side of the fragile line you're practically guaranteed to lose.

This is not to say that you have a "cooked" system where every trade is a winner.

Let's look at some numbers:

Say you have a trade that makes you $0.01 per trade when you get it right and you lose $0.011 per trade when you get it wrong.

Let's now assume you do 10,000 trades a day.

What winning trade % do you need to win at for almost every day to be a winner?

Let's look:
  • 51% means roughly 99.7% of days are losers, a bit over 0.2% of days are winners
You probably wouldn't be happy with that.
  • 53% means 89% of days are winners
  • 54% means 99.9% of days are winners - you lose one day in four years
  • 55% means you really shouldn't lose
The percentages are tight. It is a matter of getting the profit / loss distribution for your trade outcomes right so a lot of trades make it hard to lose money.

Fundamentally, if you're a high frequency trader, you're doing a lot of trades. That's pretty much the definition of high frequency, otherwise you're a low frequency trader. A lot of trades are silly if you don't have a positive expectation. If you have a positive expectation then a lot of trades means you really should have a very high chance of winning on every day even if each trade's win chance is a bit like the toss of a coin.

Let's look at the same situation if you only did 100 trades a day. A hundred is quite a few but not really high frequency.
  • 51% trade win probably = about 38% of days are winners
  • 54% = about 62% of days are winners
  • 55% = about 69% of days are winners
Not quite as good as having a lot of trades at 55%. However at 51% you were previously pretty much guaranteed to lose with lots of trades but you may live a few days longer with only 100 trades per day. This is also why if you want to bet on red or black at roulette, where the house has an edge thanks to zero, you should just make one large single bet to maximise your chances of not losing.

You can see that lots of trades help improve the guarantees. You win or lose more consistently.

The hard thing about HFT is that if it was easy, everyone would do it. There are no easy trades with the level of sophistication that exists in mature markets. It is challenging and you are constantly being pushed to the margins as someone else is always prepared to make a little less and take all your edge if they can. HFT market makers fight compulsively over the scraps and win by making less than another team. It's a bare knuckle fight against other market makers where your reward is to make less money than your opponent if you win. You in turn get beaten by someone prepared to make even less. You are scared of large trades or better informed traders that will trade through you and screw with your distribution of returns. There is little room for error.

It is simple. An HFT should be making money every day or they will be going out of business. There is money or there is oblivion. There is never any forgiveness, just paranoia.  HFTs are terrified of adverse selection. Worse still, other market makers are trying desperately to replace you by being a fraction better. Lots of smart firms are always shutting down in HFT land. It is a tough business.

So, what's up with traders making money every day?

It's not rigged.  It's just math.


PS: Survivorship bias for HFT types means that all should have consistent profits if they remain in business for an extended period of time. I find that an interesting thought.

Earlier description of how HFT market makers make money
Earlier reasoning as to why investors and speculators can both win in markets


Footnote:  I spent quite a few years as a positioning style trader with a global portfolio of currency, short and long interest rates, equity indices, and some commodities.  I made money every year but it was a battle.  Each day was a 48% chance of a win. So, overall I was usually wrong. I was successful because the average winning day was 1.6 times the average losing day. It was pretty scary though, just missing your dozen best days in a year would be the difference between an annual profit or an annual loss. It was also quite funny when people asked for a comment on the markets as it was hard to explain that with my trading I was more likely to be wrong than right for any given day.

HFT is pretty similar. It is also just a game of profit and loss distribution. It is a lot easier on the psychology. Instead of waiting for a whole year to see if things work out, you should know at the end of each day with a large enough number of trades. Even worse for the psychology is that as a position style trader normally the profits are positively autocorrelated which means that if you are losing you expect to continue losing. Worse, you can't do anything about because if you over trade, resulting in extra frictional costs, you're only guaranteeing an eventual loss. HFT has better daily psychology but higher paranoia as there is a lot that can go wrong.


Footnote: This is vastly simplified but correct in essence.  Profits and loss tend to be positively autocorrelated which makes things more volatile in practice. Distributions are nowhere near as simple which is why there are PhDs running around with computers attached to their fingers.

Friday, 4 April 2014

Flash Boys - Misleading information

Flash Boys?  I read the book last night.  Appalling.  I found it a well wide of the mark. It vilifies and accuses inaccurately. The pulsating vehemence of its message makes it a singly dangerous book.
I love Michael Lewis' storytelling. He really knows how to write a page-turner for a geek like me.

Working in an investment bank in 90s, Liars Poker had a cult status with the currency note based serial number poker he made infamous being pervasive with many traders in Sydney too. Without Moneyball there wouldn't have been a Soccernomics book to read. Soccernomics made watching football richer and, as a Liverpool fan, I'm especially liking soccer just now. So thank you, Mr Lewis.

The Big Short. Terrific read.  Hmm, but what happened to Paulson? A good story but not quite the full picture. No doubt about it though, Lewis writes books that many people find fun to read.

Nonetheless, Flash Boys is a stunningly dangerous piece of misinformation. The truth is quite a bit different to the words you'll read beyond its red cover.

I know just enough to be dangerous about this as I am a former HFT. I've traded a million index option contracts in a day in Korea on a system built from the ground up.  Back, in 2010, when people still used Blackberries, I got a call from Joanne, my broker in Toronto, half a world away, surprising me with the news that I had traded a bit over 13% of RIM at TMX for the day. Really? Some little tinpot firm on the other side of the planet mixing it up with big boys? Lewis is right there, you never know what strange firm may be responsible for a bunch of trading. The little firm I worked in had quite a few years of trading with only a few down days. Not quite as impressive as those firms with one or zero down days, but I'll take it. A team of mine also built a prototype sub 2-microsecond exchange matching engine and I'm the inventor on a somewhat dubious hardware matching engine patent. I like matching engines. I left HFT to do some cool tech but got trampled on spectacularly badly in court. In summary, I think I have just enough knowledge to know that there is quite a lot I really don't know. Though maybe just enough to call Mr Lewis out on some facets of his book. Just enough to be dangerous.

The IEX guys sound like great lads but let's start by looking at how to win on their platform. Their price is delayed fairly. OK. You are lucky enough know the price is going to be always delayed and thus stale at match point. The stale pricing is guaranteed by many miles of fibre in a shoebox. Great! Try to use all your inputs to deduce an impression of a better price in real time. Throw it at them. Send in limits or IOCs at your biased price and see if you land a trade in their time warp.  Better information exists on the outside of the delayed world.  Use it.  It's still a race. Slower exchanges are always problematic as the world knows better. IEX is no different. You can be played by better information or decisions. You don't eliminate the speed race. Hmm, it's not really much different to any other venue in that regard. There is always an arb in a world that has space and time as a feature.

In theory, there is no difference between theory and practice. Unfortunately, the real world does intervene on our clever ideas. It is wrong to think that there is not a continuum of probabilities and latency where views that seem absolute can't be replaced by a bit of educated guesswork with a virtual latency gain. Things are just not as simple as Lewis makes out. Some people think bunching trades into chunky time slices may be the solution. Such time point based auctions are not the answer. Even if you traded with an auction to the minute there is still a game of maximising your information at the deadline with latency tricks. So even slowing down trading to minutes becomes a latency game. A stupidly inefficient one, but still latency sensitive. Unintended consequences abound. You really do have to be careful what you wish for.

The clever thing IEX seem to have done is simply make a co-location space the size of New Jersey that includes all the other exchanges. It is still a game where speed matters and variances exist. It's just a bit different. Other market structure trade techniques still apply to the IEX context too, it just seems those get ignored in the story as inconveniences.

The IEX guys mean well. I just believe the ultimate basis for their thinking is, unfortunately, a bit wrong. Not a lot wrong.  Just a bit wrong. They are smart guys but I feel they have been skewered by their misunderstanding of Thor and HFT. When you have a hammer, everything looks like a nail. (Sorry, couldn't resist.) The result is they have run off on a bit of a tangent. It sounds a nice system, but I suspect it is not quite what they intended. There are many good features and reasons why it may make sense to use IEX but you need to get over the idea that it is perfectly fair and cannot be "gamed."

Clearly, "Thor" was a great tool. It was effective. However, I don't think the SEC guys were really dishonest in questioning Thor's goodness relative to the idea of over-provisioning liquidity at multiple venues. Poor form from Lewis in that regard. Tearing down ideas without proper consideration is never a good thing. The liquidity at various venues was real and could be hit individually. It wasn't fake. You must remember that the HFT market makers are scared rabbits fighting against adverse selection but needing to be stoic and fearless to maintain priority and sufficient size. It is dangerous picking up pennies in front of the market steamroller in the name of efficiency. The market maker may dare to put out more liquidity than they are comfortable with in an effort to cover all the bases. Odds are it is safe enough as they shouldn't get hit all at once. Thor! Thwack! Ouch! HFT market making is a thankless and tough job. Perhaps only exceeded by the thanklessness of being an SEC official that is wrongly ripped into by Mr Lewis.

Think about it. If some dude or dudette comes along and cleans your over-provisioned liquidity clock, then, as a market maker, you'll have to adapt and put out less liquidity or otherwise change style.  Is the gaming of the simultaneous orders with Thor really better than the over provisioning of liquidity? Thor's net result is to force there to be less simultaneous liquidity at venues and more bias toward particular venues. Is that a good result? I can see both arguments. There is certainly a yin and yang there.  It is naive to consider just one point of view.

IEX should be congratulated for their endeavours and especially their impressive ethics recounted in the book.  They have certainly tried very hard and it looks a decent enough solution as an exchange. There are also plenty of other ideas about making better exchanges. Then again, perhaps Lewis hasn't really explained it properly.

There are certainly great points in the book, such as the reference to the silliness of some of the inane order types. Some order types do seem to border on the ridiculous. As always, hindsight is a wonderful thing. However, reading the book you also wouldn't know that microwave wireless on the Chicago to New York link pre-dated Spread's cute fibre perhaps as far back as 2009. You wouldn't know that BATS was also the fastest exchange back in 2009 with a round trip of about 443 microseconds.  That was an important feature. Fastest makes a difference. You wouldn't know that there are natural advantages causing liquidity attraction in a game theoretic sense for the fastest exchange. I was horrified by the depiction of the flash crash. Frankly, Lewis' depiction of the May 2010 flash crash circumstances bordered on negligence. It's all a bit loose in the fact department.

There are lots of myths and folklore in trading.  Some are frighteningly unchallenged.  Did you know solely investing in index funds is a really dangerous idea? If everyone did it, there'd be no price discovery. The market would fail. That is an obvious statement but many would find it confronting on a first read. For some ideas, you don't need data, just as Einstein didn't really ride a light beam. It is wrong of firms like Nanex to ignore theoretical arguments, just as it is wrong to also ignore the data from the system. Many of the ideas that need confronting are indeed already well understood. It is important to remember a couple of key ideas: markets are best left alone to be the wonderfully efficient mechanisms they are, despite the motivations of their participants; and, complex systems, including human beings, need controls and limits. These principles are in conflict and need constant re-balancing but such ideas _and_ data can guide us.

Why has Lewis gone so wrong here? What's the agenda?

When I was 14 I read a story in the newspaper that I wanted to do Information Science at university. I didn't know what that was. I'm pretty sure I said I wanted to be a pilot.  Wrong message. I learnt at an early age that the truth of an article is often inversely proportional to how close you are to the story. Likewise, I find that I've been at least close enough to the HFT story such that Flash Boys doesn't stand up well to scrutiny.

Flash Boy's message was clear, but I don't know what the Lewis agenda is.  Perhaps he's just got it wrong? Against that, it reads like he has an agenda. Lewis needs to tell us how stuffed up the whole US equity trading system is and how you are being ripped off at every turn. Controversy sells. Hopefully, Lewis is just honestly wrong and not really being Machiavellian and trying to sell a boatload of books. He has been accused of just trying to whip up a frenzy to sell his book but I for one don't believe that. He is just pretty badly wrong in my view. I'm sure he'd disagree. Though it was annoying, I found it a fun read. Just don't take it too seriously. I had similar thoughts reading both Lance Armstrong's autobiography and Flash Boys, "Should it be in the fiction or non-fiction section?"

The investor, big and small, insto and retail, needs to remember that there has never been a better time in history to trade and get great pricing on executions.

Equities trading in the US is far from perfect. The system has a lot of good features and quite a few things that need some fixing. It always will, but it's pretty damned good. Flash Boys uses many woeful anecdotes to wrongfully undermine confidence in the national market system. Proceed with caution. Flash boys is a dangerous book.