Monday, 28 January 2013

Self-driving cars versus HFT

Self-driving cars should be taken seriously now that Mercedes is talking seriously about it, is a summary view of the Fairfax newspaper published comments of NSW Roads Minister Duncan Gay[1]

Here in lies the problem.

Lives are at stake. It doesn’t seem enough to go by reputation alone.

Consider a stream of traffic crossing an intersection.  Crossing the streams may not be healthy.  One rogue car travelling orthogonal to the flow flies into the fray causing fear, fatality and forensic overtime.  It’s all effed up.  The litigious blame game car manufacturers are so frightened of begins.

There are surprisingly similar parallels to HFT here for me.  A rogue algo or HFT system takes out a busy security, exchange or market.  The blame game also necessarily begins.

Reputable enough shouldn’t be a sufficient condition.  The value of the damage, whether it be a single life or the economic confidence in a market system, is significant.  Clearly the risk to life is a more present danger than that of economic liberty but the metaphor works as a thought experiment.  Perhaps advocates of self-driving cars could learn from the experiences in market systems, including the failures that have already taken place just as market participants can learn from the nascent thinking around self-driving cars.

The SEC15c3-5 market rules prescribe risk control systems to be in place to disallow certain unpleasant behaviours.  Those risk systems are not necessarily the independent systems you may expect though.  There is no real detailed specification of their manifestation nor independence.  Perhaps there should be some independence of systems mandated to stop rogue orders even if just internal independence.  Kill switches triggered by drop-copy powered independent risk are like crumple zones and air bags that minimise the damage but only if the accident impact is constrained enough.  Running into a wall at 200km/h ain’t gonna be pretty but travelling at 200km/h on an autobahn is safe enough with the right vehicle and circumstances. The system has to be well designed and tested as a poorly designed kill switch would be next to useless.  No point the air bag coming out a minute after the accident. Controlling the unaccounted for trades within kill switch kill time, the accident impact window, is obviously important. Auditable, accountable and independent it must be.

An immediate death system for traders if they turn violently in an uncontrolled manner with their high speed trading and blow out their risk would certainly create an interesting attention focus.  Perhaps too much of a parallel to the driving analogy but not doubt an increase in accountability, perhaps by diversifying the penalty, may assist. Traders being paid to take an call option on their future profit is certainly a dangerous enough game.

Car manufacturers are fearful of the costs of damage, via litigation, of getting things wrong.  This is a mixed blessing as it stunts innovation but encourages safety.  It's hard to get the balance right. Perhaps market systems could learn from this.  What if every trading participant had to have a certificate of currency from an insurance firm for a bazillion dollars to cover the rogue electrons or photons?  Conceivably there may be good old fashioned self interest at work making the markets function as a safer place.  Smart insurers could make such insurance inexpensive by insisting on and auditing control systems.  Unfortunately, there is always the black swan risk in face of the inevitable poisson boredom between accidents where complacency festers amongst insurers in both oversight and premia.  Perhaps the private sector, with the right incentives to not get it wrong and the added diversity and creativity could provide an advance on mono-culture regulation.  This may be an advance in accountability or it could be just more bureaucracy.  We need to do the impossible and get the balance right but having the knobs available to tune the instrument would be a start.

Collision avoidance and mitigation in an automated car is somewhat analogous to a concurrently run risk powered kill switch.  Some independence of collision mitigation system or systems in the car's automated driving framework would be useful so that they don’t stuff up at the same time.

In my car, I’d like not just separate sensors but n-versioned systems where the versions are truly independent to automate crash prevention.  I’m happy enough to trust a single vendor, such as Mercedes or Google, with my driving system and get one efficient voice, but I want rigorous and independent safety oversight to protect my family in that car.  N-versioned safety should be prescribed, similar to space, air or medical systems where independent thinkers build to the same specification and majority voting on the decisions takes place as it did on the space shuttle.  When my life is at stake, I'd like a third independent opinion.  It doesn't have to be onerous if we amortise the cost over many vehicles.

History has shown that formal methods, such as program proofs, and diversity through techniques such as n-versioning, can build more reliable systems, though they have never really scaled to the reliability heights that were expected.  Errors in proofs and code are often correlated.  Independent thinkers often make similar errors in building systems.  There are simply not the pure multipliers of independence of error you’d hope for.  There is also the reliance on components in the systems that do not have the same rigorous methodology. A provably correct system is no solace if your batteries may overheat or your o-rings malfunction.  Systems and components matter. Engineering still counts. If it were easy to solve, there wouldn't be a problem.  The bottom line still remains that safer is better regardless of such shortcomings.

If the roads were intelligent and could disable or control traffic with accident prevention mechanisms, that could be a good safety overlay.  Imagine big brother radio signals controlling your car's safety as an independent arbiter.  Cue safety barriers popping up out of the road to prevent unintended crossings of the flow. I'm not a great driver, being easily distracted behind the wheel, but I worry, perhaps wrongly, more about other drivers that I have no control over.  

This does seem a bit expensive for the road system.  However, for exchanges, the economics are different and it should be just good housekeeping.  Sure the cars / market participants need to do the right thing, but if the cost effectiveness works for the infrastructure to have additional safety, why not?  Rogue begone.

Multiple independent systems and the correct financial incentives to invest in risk mitigation are required.  It seems we're not quite there yet.  I think both self-driving car hopefuls and parties responsible for market structure could learn a bit from each other.

Fake safety is dangerous, just as uncontrolled speed may be.  Try saying, "whale oil beef hooked" too fast and you'll understand[2].

All the best in trading safety.  Double entendre intended.


[1]  Whilst the FairFax original articles have been modified and I thought my memory deceived me,  the original lives on in the AAP records.

[2] Fake irish lyrics

Thursday, 24 January 2013

Hedging – losses as profits that feed traders

“Aristotle maintained that women have fewer teeth than men; although he was twice married, it never occurred to him to verify this statement by examining his wives' mouths.” 

Hedging is the domain of ducks and drakes where a loss may be a profit and nothing may be a loss.  There are happy losers and happy winners and, if you turn up, you too may be happy.  Let me meander through this babbling brook of words to clarify if only for the reason I find it mildly therapeutic.

Most good financial professionals understand hedging pretty well but I believe there is quite a bit of misunderstanding around the market effects and consequences for market structures.

As a corporate with a bunch of cashflows, the idea of locking in key component costs seems very practical.  Whether that be oil for a transport company like Coca-Cola, interest rate differential locks for a prime broker,  downside tails for an equity fund manager, matching costs to revenues is sensible.  This part we all understand well.  The choice of doing nothing is also a hedging strategy and this is well understood in finance but remains a barrier for many corporates.

Corporates’ hedging tactics are a strategic choice based around industry practice.  If you under hedge, perhaps by doing nothing, or over hedge, relative to your competitors, then you have a different cost structure and different paths of potential outcomes lay before you. The smarter professional when presenting a hedging or financial analysis to a client will also try to paint the industry landscape in this regard which is difficult as it is often fuzzy if known at all.

It’s an awkward game you can’t avoid as you are already playing even if you refuse to lay down an explicit bet.

Now comes the fun part.  The risk transfer process or hedging process is conducted in a market and you need to execute as efficiently as possible.  Physical agreements and derivatives and those that trade them come to the risk translation party.

The biggest myth in all of finance then enters the door:
            Markets are a zero sum game.

If you’re going to be hedging aren’t you then just falling victim to the trading sharks swimming in the tank?  As an investment bank, what's your edge and should you really being bothering to participate in a zero sum game which is a slightly negative sum game after fees?

The vast majority of finance professionals have this belief in the slightly negative sum game and also there are often important and consequential choices based around this myth such as closing down a whole IB division.  I’ve seen it done, perhaps you have too.

This is such an enormous wrong in finance it makes my head spin.  Aristotle should have counted his wife's teeth.

A market where buyers and sellers are matched and there are no transaction costs is a neatly closed system where for every winner in that market there is an offsetting loser.  However I’m yet to find a market that has no association with a related market or real assets.  In the real world, such closed systems don't exist.  Examples are welcome if you have one.

Consider a simple futures market on a stock index. As viewed simply, it is a closed system that is a slightly negative sum game but it relates to a real asset market that has wealth effects, the stock market.  If you’re fast asleep, invested in the physical index and wake up the next day with the market up, you, and many others, have just participated in a wealth creation event.  The stock market is not a zero sum game.  There are real assets with associated wealth affects related to the tide going out and rising again. Markets are linked.  Direct and statistical arbitrage helps make it efficient.  Every market is thus related to such wealth effects to some degree.  The futures market on the stock index is linked to the wealth effect in the physical.  There is no truly zero sum game.

The big reason for traders and banks to participate in seemingly zero sum markets is that you have a lot of happy losers.  There are real needs for hedging and those needs are acted upon.  Say a portfolio manager takes a 10% hedge on their $10B portfolio.  Mr Market goes up 20%.  The portfolio manager is going to be quite excited about losing $200M on her hedge as she is making $2B on the unhedged portion.  A very happy loser indeed.

Almost everyone hedging in business is a partial hedger.  Thus losses on a hedge make almost everyone in business a happy loser. It may be competitive but there is potentially happiness for all. 

There exist plenty of profit opportunities in finance due to this potential for happy outcomes.  The fundamental role of speculation is to make this risk transfer process as efficient as possible.  The more speculation, the more quoting, the more efficient the process.  Machiavellian taxes motivated by popular hype towards financial processes harm efficiencies and thus harm real businesses and the real economy.  Indeed such taxes can kill markets and thus make industries uncompetitive due to poor risk transfer mechanisms which is a consequence later opined to be unintended even though it was clear there was to be collateral damage.  I can hear the excuses now, "No ma'am, I did not expect the staff and pilots to lose their jobs as the airline could no longer fly as it couldn't compete on fuel hedging due to the transaction tax I introduced into parliament."  The short termism in taxing the unpopular for a political win is quite the moral hazard.

How do you profit from happy losers?  

I mentioned to my middle daughter who is heading off to her first year of university at ANU that part of the secret of success in first year uni is just turning up.  It’s a pretty easy beat after the torture of high school and getting the grades required to claim entry. Many of her friends will loaf, party and otherwise find excuses to abuse the unstructured freedom and skip class.  Just turning up is most of the battle.

Just turning up is half the battle in finance too.  Markets are not zero sum games and there are plenty of happy losers who don’t want to make money on their hedges.  There is an important role here for the speculator and trader in this risk transfer process.  It is not just a dog eat dog, art of war battle where for every winner there is a loser.  There are real benefits and plenty of win-wins that drive the whole financial eco-system.

Turn up.  Take the other side of a happy loser and everybody wins.

Shock.  Horror.  The financial system has a purpose. 

Saturday, 19 January 2013

HFT’s dirty little secret – where the money comes from

“Truth always rests with the minority, and the minority is always stronger than the majority, because the minority is generally formed by those who really have an opinion, while the strength of a majority is illusory, formed by the gangs who have no opinion — and who, therefore, in the next instant (when it is evident that the minority is the stronger) assume its opinion… while truth again reverts to a new minority.” 

As I meander through some of the whacko press that gets shoved into peoples faces, I’m continually surprised at the lack of understanding of HFT.  The voices in the press capable of intelligent and critical thought are being drowned out by the violent words of their not so savvy colleagues. HFTs are painted as the nasty people who bushwhack investors as they pass by and yet nothing could be further from the truth.

You’d think it was a massively complex area, too hard to work out fact from fiction, but it's not.  Even though the concepts are indeed straightforward and not at all hard to discern, there continues to be weird false debates that just recirculate time and time again.  HFT animals are poorly understood. It seems pointless to even try to stem the tide but let me try to stick my finger in the dyke wall and meander through.

What drives the bad press?  The disenfranchised complain and the press picks it up, or the disenfranchised become the press.  The investors, buy side and retail read the press and feel disadvantaged and indeed threatened by the misstatements and untruths. The maddening crowd bays for blood and death via taxes, law and regulation.

So who is really being hurt? 

It’s simple.  Follow the money.

Overall an HFT maker wins by having a better price. That’s it. Buy higher or sell lower. To be successful you have to make LESS money than the pricer you’re replacing by offerring better prices.  Spreads compress.  The market is more efficient.  So, market makers make less.  Other users, call them, for want of a better word, "investors", make MORE money. Simple enough, right?  

So the dirty little secret is that HFT eats its own.  They take money off and displace each other. The best bid they better, or the aggressive ask they advance, is most often just pushing aside another HFT market maker. It's an easy idea to understand and just capitalism 101 at work. There is a tendency for new young offspring firms to eat their parents if they survive their birth.  Creative destruction at its finest.  As a group, market makers make less over time as a natural consequence.  Asymptotically the nett profit goes to zero. Expect the clever, small, and thus low cost to prosper only by being the new benchmark in efficiency. It's a tough business. 

Again, the newly disenfranchised complain loudly about the soon to be disenfranchised and that is the sore echo we all hear.

The real winner is the market.  More for the user.  Retail, the buy side, investors, the true risk consumers, have not ever had the ease of entry and exit they do today.  HFT cares little about medium or long term alpha.  A few seconds is long enough.  The real investment profits are all there for the new asset owner, laying in front of them bleating, with tight spreads the like of which have never been seen before.  The consequences are obvious.  As a tiny or huge investor, today, right now, is the best time in all of history to transact.  You wouldn't think it from reading the papers.

Too many quotes?  A crazy idea. The more the better. Better pricing is not something to be ashamed of.  More competition in pricing is a good thing. Messaging taxes and transaction taxes harm society as they only make the markets less efficient and are the toxic response of the naïve.

HFTs have transient lives helping the rest of the market as they get destroyed by the more agile.  We should thank them for the social good they do indeed bring to markets despite their motivations.  You should seek out an HFT, thank them and make it soon as they may not be around for long. 

Hug an HFT before it's too late.  It may be the only thanks they get.

Thursday, 17 January 2013

Which is the world’s fastest switch for 10G Ethernet?

[June 2016 update: Metamako MetaMux48 is just over 80ns, the new leader in switching latency. April 2015 update: Metamako's MetaMux32 is the new king at 99ns, Gnodal & Zeptonics passed away, RIP]

As always, it depends on your application but here are the headline numbers and an order of latency merit:
  1. Zeptonics ZeptoMux          130 ns
  2. Gnodal GS7200               150 ns
  3. Cisco Nexus 3548            190 ns
  4. Mellanox SX1016             250 ns
  5. Arista 7150                 350 ns
  6. Gnodal GS7200               500 ns
Yes, Gnodal is listed twice for good reason.

It's important to note there is a further bunch of interesting switches, that may have exactly the features you need, from Juniper, IBM/Blade, Fujitsu and elsewhere that are in that 500-900ns range but that is the “old” fastest and so last year :-).  The products in the list above are the new kids on the block and are the current 2013 latency champions.

Now one interesting thing, to me, is that I believe you can make a case for all of the above switches for specific circumstances.  Let me meander a little on each.

1.         ZeptoMux is the fastest way on the planet to get two 10G Ethernet packets from two (or twenty three) different wires onto one wire.  Not NxN connectivity, just a 23 ports to 1 port, and back again, funnel.  The client ports can’t talk to each other.  It is the fastest way in the world to talk to a financial market or risk service or any other concentrated service but it is not a generic fully connected switch.  It can be integrated into Layer 2 and Layer 3 solutions (such as CME). Really low jitter with 130ns +/- 6ns for all packet sizes.  A few less nanoseconds on the way back from the service. I’m biased and vested so you have to take my meanderings here with a grain of salt here as my company makes this piece of glorious magic ;-)
2.         Gnodal is a regular NxN switch like everything  else on the list except the ZeptoMux.  This switch is special as it has a crazily low latency interconnect between Gnodal switches and hence it has great specs for large fabrics.  The down side is it is not really 150ns as this is a minimal packet length thing as the first hop onto a Gnodal fabric is store and forward and hence dependent on packet length.  This means that for a bigger packet, such as for some trading environments, you may actually be 200ns or 500ns port to port. Depending on your packet size Gnodal may actually be the slowest on this list.  That’s why I gave it two entries.  Just goes to highlight the best solution for you is dependent on your specific circumstances.
3.         The Cisco 3548 in Warp mode (reduced CAM == smaller network) runs at 190ns or perhaps even just a bit under.  There is an impressive Cisco chip “Monticello” in the box thanks to the work of 500 or 600 clever engineers over nearly two years that makes all of this possible I’m told.  Based on the published specs, I think this is the real gem of the lot as a fully functional switch.  Cost is a bit of an issue as I see this model, CISCO Model #: N3K-C3548P-FA-L3A, has a list price of $66k and then you have to pay for the SFP+ modules.  If money is no object and you have a small network, packets are not just minimum length, and you require NxN connectivity, Cisco is the fastest.
4.         Mellanox have a neat product with their SwitchX family that will get better over time with improved firmware and functionality and I'd argue it is a better value proposition than the Cisco 3548.  I’d buy this one if I had a latency sensitive but not a latency critical application and its feature set, which is more limited than the Cisco, suited my application. The next generation of Mellanox SwitchX-2 products may provide a bit of a latency improvement and is something to watch out for.  For value and performance I like the balance of this product best for a general NxN switch.
5.         The Arista 7150 series is a great product.  However they have fallen behind in the speed stakes.  Arista relies on Broadcom Trident family silicon and Intel/Fulcrum silicon for their product set with the 7150 using a newer Fulcrum chipset.  I'm a big fan of the very cleverly designed Fulcrum chips.  However, Arista's reliance on third party silicon is restrictive and it should be no surprise that all the four vendors in front of Arista on this list control more of their hardware IP than Arista does. Where Arista really shines is their EOS and rich feature set which is something they do indeed control.  EOS is a rich, pleasant environment and Andy Bechtolsheim is a legend so why not use an Arista 7150 if the nanoseconds don’t need to be counted so closely?  If the switches were all the same speed I think I’d choose this one for the software set it offers, but they’re not the same speed.

So, if you’re a trader and you need a switch to focus multiple lines onto one order or risk line, you need to use a ZeptoMux or you may come second and miss the trade.  Be aware that most applications in the data centre are not so “focus” oriented on a specific service point and thus don’t really work this way and you need one of the other more traditional NxN switches instead.  Horses for courses.

If you’re not dollar conscious, it is hard to beat the specs on the impressive looking Cisco Nexus 3548.  If you have a large HPC fabric then perhaps Gnodal may suit better with its super fast inter-switch latencies if your packets are appropriately sized.  Mellanox is a good value buy if you want fast but not fastest and keep in mind it may be faster for your app than Gnodal due to its cut-through.  The Arista is for latency sensitive but not latency critical apps and helped by being a nicely featured product and perhaps a good option for a cloud due to its pleasant OS.  Now you can perhaps see why I think there is actually a place and time for all of these products. 

So consistent with the Lake Wobegon effect, all switches are above average and everyone is a winner!

Stating the obvious, you can't consider these things in a vacuum and your application and environment is the context that makes a choice make sense.  It's not always about speed, sometimes features matter.  My customers want to be first past the post and care about their competitive advantage, so I care about speed, consistent speed, and nothing but the speed.

So, batten down the hatches and load the clichés.  Pack me up a truck of me very own ZeptoMux so I can go a huntin'.  Yeeha!