Monday, 18 June 2018

US Finra ATS stats update: June 18 release for May 28

The UBS bounce bank continues but remains off the 21% peak. Morgan Stanley strength persists. Instinet remains close to the top ten. Barclays has a good week.

Luminex promotes being 26th in the list with a straight face: "Luminex May 2018 Recap."

TradeWeb completed 164 trades for the week. Should they just use a pen and notebook? Who says they don't, I guess?

--Matt.

(click to enlarge)
(click to enlarge)

(click to enlarge)

(click to enlarge)


Sunday, 17 June 2018

The accidental HFT firm - Russian edition

Russian hackers have infiltrated Meanderful HQ. I can't vouch for the efficacy, but it looks not a small effort. If your Russian language skills are better than my non-existent skills, you may have a gander here: "Однажды в HFT-компании…":

(click to enlarge)
Happy trading,

--Matt.

____________



Saturday, 16 June 2018

IPCC wrong on climate change & SEC wrong on IEX

Made with Remarkable!

“Do the best you can until you know better. Then when you know better, do better.”
Maya Angelou

I’ve long been troubled by the IPCC report on climate change. It seems, to me, a fairly clear dereliction of duty to arrive at the conclusions it did. Mainly I find it is due to a lack of science in the report. Time is proving my thoughts quite correct 1 , 2 .

The IPCC underestimated climate change quite badly. Models were included that were obviously worse than others. Those bad models largely pulled the conclusions in the same direction - down ; especially in terms of the confidence interval. We are now finding all sorts of unknown aspects, ice shelf collapse, sea base methane release, etcetera are much worse than anticipated. The IPCC modelling has been shown to be flawed. Climate change is much worse than what we have been told.

Part of the problem is political interference with the science. No doubt the IPCC felt pressure to be as conservative as possible. They included a broad range of models that should have been excluded. Politricks won. Truth lost.

Pseudoscience is a complex and dangerous beast. I hope this IPCC example illustrates the point. I am also pretty confident the IPCC didn’t intentionally wish bad science upon itself.

SEC wrong on IEX

The WSJ is reporting a disturbing headline today,

“Study Finds ‘Speed Bumps’ Help Protect Ordinary Investors” by Cezary Podkul.

Hey, what?

That’s absurd. There is simply no basis in reality for such a claim. A speed bump in such a style harms market transparency, increases transaction costs, adds complexity, and piles uncertainty on to an already contentious market structure 5 .

“The evidence seems to be consistent with IEX’s speed bump resulting in improved market quality for some stocks,” Mr. Hu said in an interview.

Dr Hu to you, even if he has no Tardis. Show the foolish kid with a PhD some respect.

The first line in the WSJ’s article by Mr Podkul is,

Stock markets should take a break. Literally.”

So, is it the WSJ at fault for such drivel, Mr Podkul, or has Dr Hu run out on a limb too far and been caught in a little hyperbolic malpractice?

Then a few hours later the WSJ changes the first line to,

“A brief delay in stock trading can help protect ordinary investors from high-frequency traders, according to a study by an economist at the Securities and Exchange Commission.”

That’s a little less jingoistic at least.

You may read Dr Hu’s paper at SSRN, Intentional Access Delays, Market Quality, and Price Discovery: Evidence from IEX Becoming an Exchange

Here is the abstract for you:

click to enlarge

The startlingly dumb thing that jumps out at you is the preposterous claim that IEX improves price discovery. IEX harms price discovery. That’s its very mission. IEX penalises participants with extortionate fees if they dare move the lit price. The contortions Dr Hu’s paper has to go through to arrive at an improving price discovery conclusion are embarrassing.

Evil IEX speed-bump

click to enlarge

The nature of IEX is that it favours dark liquidity, recently over 75% of its total handled shares, according to IEX’s own statistics is dark. Let’s remind ourselves of this simple fact:

IEX Market Share May 2018

click to enlarge

The debate around IEX outweighs its impact due to the perversions IEX’s model has introduced to the market. IEX represents the thin end of the wedge in that it pulls the NMS into a state where less public price discovery is promoted, transactions cost more due to IEX’s very high fees, and the financial community: funds, brokers, and traders, are hit with greater complexity and risk.

Despite the harm IEX has done and continues to do to the market, IEX’s impact is pretty irrelevant due to its small market share:

IEX Market Share May 2018

click to enlarge

Small mercies are small mercies.

Mr David Weisberger points out some aspects of IEX’s lousy price discovery business on from 17 th October 2017 within his blog, “IEX ignores DATA (again) to market their exchange” ,

Notice that IEX is dead last in executed percentage, with a fill rate of 1.8%, more than 75% below NYSE and 65% lower than NASDAQ. Readers should note that this is the best data available[1] as it counts only shares accepted by the markets which are priced at the NBBO when the order was received.

Mr Weisberger gives one view on how bad IEX’s execution quality is:

Worse Executions

click to enlarge

He then concludes:

This data, empirically proves that IEX does not provide better fill probability. It proves the opposite, that investors who believe IEX’s false narrative are being misled and are likely suffering with higher execution costs as a result. (When orders are unfilled, it creates opportunity costs. That, of course, is the harm of allowing their misleading marketing to continue unabated) This data is surprising to people that believe Mr. Ryan’s assertion that it should be better to be alone at the NBBO at IEX, instead of being toward the end of the queue at another exchange. I find that amusing, since the IEX narrative assumes that traders look at each individual exchange separately, like checkout lines in a supermarket. That might have been true at the NYSE pre-Reg NMS, but today’s market participants use routing technology that evaluates the composite queue across exchanges. Therefore, providing a higher certainty of execution tends to drive routing decisions, meaning that IEX, with their infrequent NBBO participation and high average spread, is not the fast line at the supermarket as they claim.

Of course, neither the SEC, nor even the mainstream media seem to care about IEX’s continued use of factually wrong and misleading narratives. To my perception, IEX management continues to behave more like carnival barkers than responsible regulators. At some point, for the sake of all the investors that believe every word they say, my hope is that either they will change or the SEC will force them to.

Let’s get back to Dr Hu’s paper.

The paper highlights much of what is wrong with the academic approach to the financial industry. You take a bias and false premise and try to fit a certain outcome. It helps a lot if you use mathematics and twist the work of others. This is precisely what Dr Hu does in his paper with or without a sonic screwdriver. He is young and just out of university, so I guess he will learn in time. Now is the time to learn from this paper.

IEX is dark and expensive and tends to execute at the midpoint. Dr Hu arrives at the conclusion that if you benchmark the midpoint oriented IEX against the midpoint it does OK. No shit Sherlock. That’s the point of a dark pool that is parasitic to price discovery. There is no shame in that, but a public price discovery exchange engine it should not be. After all, what is the point of a public exchange without meeting that fundamental public need?

There are many asset managers this kind of exchange that isn’t an exchange does not suit. Some asset managers would prefer to accrete the spread by being passive. They hope not to regret too many things if the market runs away from them. IEX is a particularly bad place to trade if that is your schtick.

An aspect of Dr Hu’s paper I find particularly troubling is the hiding he does behind fancy greek symbols and mathematics hoping to reassure the reader there is a scientific point to his work. The good doctor is suffering from the same malaise as the IPCC faced: bad analysis. It is a bit different to the IPCC as the modelling and statistical approach was pretty flawed at the old IPCC, but here it is more about Dr Hu’s assumptions and improper market dynamics interpretations. No shame in being wrong. It needs to be called out though. Hopefully, as a relatively recently minted postgraduate, he will learn to consider the broader picture and not shape his approach to any particular agenda in the future.

I mentioned in a tweet that if you apply the lipstick of a Kalman filter along with its state space modelling to a pig, you still have a pig. This is what I’m talking about here:


Dr Hu p27

click to enlarge


Dr Hu p28

click to enlarge


The key point here is the estimation part. Dr Hu is just modelling midpoints. You’ll see the same midpoint bias in the Informational Efficiency Measures Dr Hu uses.

Should we gloss over the free rider problem? Let’s not, page 3:

The overall improvement in price discovery is harder to reconcile. In his Comment Letter, Eric Budish argues that a weakness of IEX’s market structure is that it only prevents latency arbitrage for (non-displayed) pegged orders, and does not help displayed limit orders which contribute to price discovery. As such, IEX’s market design can only succeed so long as it is able to “free-ride” off of other exchanges’ price discovery. Indeed, when I decompose exchanges’ contributions to price discovery using a state-space model, I find that IEX’s information share does not increase as it transitions from an ATS to an exchange. Hence, the improvements in price discovery are not directly attributable to IEX. Instead, it may be that the speed bump affects other forms of mechanical or regulatory arbitrage beyond “quote sniping,” which impacts price discovery.

One could conclude that that price discovery has improved in the market as a whole despite IEX. Perhaps it may have improved even more if IEX didn’t exist. This is where Dr Hu perhaps needs a little more market experience. A PhD and a short stint with the SEC does not prepare you to understand this kind of thing I guess.

So, the paper is a bit screwy. We could arrive at the view that Dr Hu is a young guy. Give him a break. He’ll do better next time. I think that is reasonable, but it is the consequences that are unreasonable.

Wall Street Journal

That the WSJ suddenly reported hysterical headlines from flawed reports is disturbing to me,

Study Finds ‘Speed Bumps’ Help Protect Ordinary Investors

SEC economist’s conclusions bolster models like upstart exchange IEX, which aims to slow ultrafast traders

A paper is made up of it journalists and editors. It was eventually pretty obvious to me that the article’s author, Mr Cezary Podkul, is an unabashed IEX fanboy. It is also interesting that some of the tones of the original article have changed. A slight concession but not enough as the appalling headline remains. A few lines of at the top of the article varied:

Removed:

- Stock markets should take a break. Literally.

Changed:

- That is the message in a paper published on Wednesday by an economist at the Securities and Exchange Commission who concluded that a brief delay in stock trading would protect ordinary investors from high-frequency traders.

+ A brief delay in stock trading can help protect ordinary investors from high-frequency traders, according to a study by an economist at the Securities and Exchange Commission.

Changed:

- The finding broadly endorses the business model of IEX Group Inc., an upstart exchange that slows down trading with a so-called speed bump that pauses inbound orders for 350 microseconds before relaying them to its exchange for execution. IEX also delays outbound updates to its market data feed.

+ The study, published Wednesday, bolsters the case for the business model of IEX Group Inc., an upstart exchange that slows down trading with a “speed bump” that pauses inbound orders for 350 microseconds before relaying them to its exchange for execution. IEX also delays outbound updates to its market data feed.

Changed:

- When it sought SEC approval to become an exchange in 2015, IEX said the delay would prevent rapid-fire traders from racing ahead of typical investors and unfairly profiting off the speed advantage. That claim was disputed by high-tech market maker Citadel, which warned the SEC that allowing speed bumps would “increase transaction costs for all investors.”

+ When it sought SEC approval to become an exchange in 2015, IEX said the delay would prevent rapid-fire traders from racing ahead of typical investors and unfairly profiting off the speed advantage. That claim was disputed by high-tech market maker Citadel LLC, which warned the SEC that allowing speed bumps would “increase transaction costs for all investors.” The study found a decrease in trading costs for some stocks.

The changes improved the quality of the article. The article remains an over the top representation from a fundamentally “challenged” report.

Mr Cezary Podkul 3 is quite an impressive young journalist winning awards here and there for various pieces. Some very good work in there, especially on his contributions to “The Rent Racket” and “Tobacco Bonds” stories.

He left his bias in plain sight though with this tweet:

Citadel tweet from Cezary Podkul

click to enlarge

You know the old saying: if you’re young and not a socialist then you have no heart, and if you’re old and not a conservative then you have no brain. It springs to mind that Mr Podkul’s heart is misplaced from time to time.

It’s kind of funny to consider the Citadel warning Mr Podkul refers to in the tweet that has come to pass:

  • harm market transparency: 75-80% dark is harmful you’d have to say
  • increase transaction costs for investors: have you seen IEX’s expensive fees?
  • add enormous complexity: it’s a mess, thank you IEX
  • add uncertainty: you can’t even reconcile your fees, remember the mea culpa when IEX had to send out refunds, their crumbling quote indicator is often wrong, you don’t know what you are going to be charged as it is subject to IEX’s interpretation (often wrong) of market conditions

Good highlight Mr Podkul. It turns out Citadel was right on the money. Are you really that blind?

I had the temerity to point out how I felt about this flawed article and report in a couple of tweets. Mr Chris Nagy showed how mature the debate around the IEX issue is:

Chris Nagy tweet

It’s a bit of a childish tweet from Mr Nagy 6 , but as CEO for Healthy Markets he is normally more astute and reasoned so we’ll give him a pass.

SEC problems

I think this naive paper by Dr Hu highlights a few problems at the SEC since Mrs Mary Jo White ran the show. The whole ICO and virtually currency debacles are getting quite stupid when SEC spokespeople think it is reasonable to call Ethereum not a security. I don’t think history will be kind to such an incorrect, incoherent, and indefensible point of view.

The SEC seemed a good home for the CFTC, especially given the overlapping fines and jurisdictions, e.g. FX, and challenges the CFTC often face. Perhaps the CFTC should be taking over the SEC as the SEC seems to be acting the junior partner in these days of daze.

Look at the kind of ridiculous debate around the SEC’s fee pilot. I’ve been steering clear of that issue as it is one of those heated debates where one side is so obviously wrong it should be damned obvious. Alas, it is not judging by the number of comment letters.

The set up of the pilot is just all wrong. How can you have such an impactful study without a reasonable and testable null hypothesis and hypothesis? The most likely outcome of the study is that you will have arguments over the data and the impact on other aspects of the market, such as with all the ATS’s. Is there a way to measure the impact of passive liquidity not being traded with? No. Is there full transparency on routing and beneficiaries? No.

Market data fees are a real problem as the direct fees have been subject to unconscionable rates of increase over the years. Marketplaces can get away with such piracy as the demand is inelastic. You gotta have the data.

Transaction fees are different. You have a lot of choices. There is also a reasonable argument to be had for higher fees for increased price provision at best and for stoking competition with the PFOF community who can effectively quote at sub-penny 4 increments providing prices public exchanges simply can’t compete with. The size of a particular fee is perhaps an ill-considered priority just as the fee pilot is an ill-conceived pilot.

You have to ask yourself, how did this fee pilot get so advanced. It looks like the SEC can be bullied by the funds and banks who want lower fees. Everyone should be aware of the politricks involved.

The SEC should be smarter. The SEC needs to be smarter.

The SEC certainly should not have published with their tentative imprimatur rubbish papers like Dr Hu’s. If you do, you’ll get hypocritical headlines like, “SEC wrong on IEX” (see above.) You see, I can publish scandalously like the WSJ too :P.

Happy trading,

–Matt.


PS: At least one senior exec from a financial company has some sympathy with my position:

Matt tweed

click to enlarge


Older IEX meanderings



Footnotes

  1. Conservative nature of IPCC reports

  2. Greater future global warming inferred from Earth’s recent energy budget ”, Patrick T. Brown & Ken Caldeira, Nature volume 552, pages 45–50 (07 December 2017

    click to enlarge

  3. Cezary Podkul , Senior Reporter at The Wall Street Journal, Previous: ProPublica, Reuters, The Washington Post, Education: Columbia University - Graduate School of Journalism

  4. Meanderful: Sub-pennies rule!

  5. Meanderful: Speed-bump 101

  6. Chris Nagy, Co-founder & Director Healthy Markets


Wednesday, 13 June 2018

Kickstarter book is a go - "The accidental HFT firm"

Thanks to some generous backers on Kickstarter, "The accidental HFT firm" book project is a go. The basic printing and proofreading costs are covered.

Thank you all.

I'm very interested in your feedback on topics which I'll try not to ignore. I certainly have intentions for the book but welcome input.  I've created a page to capture thoughts on the content and what probably won't be in it unless asked for. Please check it out and leave a comment or email me at matthurd@acm.org if you have a suggestion or criticism.

No, it won't be a rehash of the blog. I hope it will contain some content that may help, surprise, and delight.

Thanks again,

--Matt.


Shiller PE Ratio

(click to enlarge, source)
Price earnings ratio based on average inflation-adjusted earnings from the previous 10 years, known as the Cyclically Adjusted PE Ratio (CAPE Ratio), Shiller PE Ratio, or PE 10.

What could possibly go wrong? Nothing to see here. Move right along.

--Matt.

Tuesday, 12 June 2018

Negative latency

Made with Remarkable!

Negative latency is impossible. There is no such thing.

OK. So, why are you meandering about it then?

The impossible made other people millions of dollars over the years when my team and I gave them negative latency.

But you said it was impossible?

The fun stuff always is.

What are we actually measuring?

The details of what we’re measuring and its importance is so often overlooked.

In “The Accidental HFT firm” I glossed over measuring the latency although it featured prominently in the tale.

I mentioned the “ < 2ms ” being scrawled on the window of my old employer as the team’s new latency target. I then mentioned the first cut of what I termed the “fast trading engine” (FTE) reduced this measure of latency to fewer than five microseconds. This was a fair bit under the two millisecond target. I also mentioned that my employer at the time did not take into account the network stack or operating system overhead. The measurements we are talking about here are just the application latency measurements. A few years later, six I think, I interviewed a new employee from the same old place and they were still measuring latency the same incorrect way.

From “The Accidental HFT firm,”

“This was a time in which 2-3 GHz processors were new. Our performance measurement approach was a bit lame. We measured performance internally within the application rather than properly from the external network. This was fine when talking about two millisecond time frames as the network stack was in the 50 to 100 microsecond zone in those days of yore. The key point to remember here is that when you can do more than a billion instructions a second, that translates to more than two million instructions for two milliseconds. Frankly, it was just criminally wrong that even a millisecond, over a million instructions, was burnt on simple trading tasks. If it takes you a million sequential steps to decide on a trade, retire now! A modern processor can do thousands of instructions in a microsecond. Every nanosecond is sacred.”

Let’s look at a basic diagram:

Simple latency diagram for a trading engine So, the referred to latency is:

$$ t_{app\ latency} = t_{app\ out} - t_{app\ in} $$

FWIW this is normally measured by a high resolution timer such as provided by the operating system, or by using C++’s chrono library,

auto start_time = std::chrono::high_resolution_clock::now();

// do work...

auto current_time = std::chrono::high_resolution_clock::now();
uint64_t nanos = std::chrono::nanoseconds(current_time - start_time).count();
or by a roll your own Intel rdtsc or rdtscp instruction such as,

inline uint64_t rdtsc() {
  uint32_t low, high;
  asm volatile ("rdtsc" : "=a" (low), "=d" (high));
  return (uint64_t) high << 32 | low;
}
which has a bevy of caveats in terms of accuracy, to flush the pipeline or not, overhead, technology, and history. Don’t get me started ;-)

As mentioned, this measurement is both pretty useful and pretty useless. On the one hand it represents what a software engineer may control. On the other hand if you can do your work in 500 nanoseconds and the rest of the OS and network stack takes 50 microseconds aren’t you barking up the wrong tree?

So t app latency is interesting but not too useful. What we really want to measure is the so-called tick to trade latency, let’s call it t tick to trade :

$$ t_{tick\ to\ trade} = t_{Order} - t_{MD} $$

Simple tick to trade latency diagram for a trading engine

Here we want to measure the latency from port to port, or wire to wire. We wish to use some kind of arrangement where we may scoop the market data packet up off the line on the way in to record t MD and then record the order as it goes out to market, or our simulator in test, as t Order .

Most people who read meanderful understand this already but I need to set this up so we can be precise. Reality matters in the world of negative latency.

What is zero latency?

It is unlikely that your processing overhead will be zero, but if it was, you’d expect:

$$ t_{tick\ to\ trade} = t_{Order} - t_{MD} = 0\\\therefore t_{Order} = t_{MD} $$ Now it is impossible for those to be equal you’d think, especially if we are measuring correctly to the nanosecond.

Packets and wires are things too!

Consider a 100Mbps Ethernet UDP market data feed coming from the KRX like it used to do. Say your packet is 500 bytes long, or 4,000 bits. At 100 Mbps, a bit takes 1/100,000,000 seconds, or 10 nanoseconds. Your packet takes 4,000 x 10 nanoseconds or 40 microseconds of wire time.

A fibre optic cable allows 5 microseconds of propagation every 1 km. If you have a cable greater than 8 km long your packet, for a while, would only exist on the cable. It’s cute to think of the cable storing the packet in this way. It reminds me of the mecury delay lines used in the early computers. Your data centre's fibre length will be shorter than 8km. Some of the packet will be on the cable, some will be on the transmitting host, and some will be in the receiving host.

You’d like to get a timestamp to be more accurate than 40 microseconds, so it begs the question about how does the timestamping and packet buffering work on the host?

A packet on the wire

In the diagram above, the packet is 40 microseconds long. Parts may be on the receiver, on the wire, and in the sender all at the same time. Similarly, when we send an order in response, part of the order packet ends up in our machine and some lives on the wire.

Market data and order packets in flight simultaneously

The question here is when is, or what part of, the packet is timestamped on the way in, and what about on the way out? We're executing thousands of instructions per microsecond on our host. Details matter.

What would tcpdump do?

tcpdump , the common packet capture utility, delegates the timestamp responsibility to libpcap . libpcap ‘s normal behaviour is to get a clock for when the packet has arrived as part of its delegation to the kernel driver. Pcap will mark when it starts to send the packet out also via the driver. Wireshark also delegates the timestamping to libpcap just as tcpdump does. Windows is similar in spirit.

This is interesting and concerning as we know the packet is 40 microseconds in length due to the 500 bytes, or 4,000 bits at 100Mbps. Our t tick to trade reported by our packet capture with the impossible zero processing delay means that the first bit of the packet has been in the machine for 40,000 nanoseconds before a response was considered. Yikes! The application will only be notified in the usual UDP or TCP case once the packet has arrived, perhaps later if you don’t have the right network and kernel options set.

You can find the gory details of how this works here for completeness but I suggest you skip the next indented bit, Kernel documentation network timestamping :

SOF_TIMESTAMPING_RX_HARDWARE:

Request rx timestamps generated by the network adapter.

SOF_TIMESTAMPING_RX_SOFTWARE:
Request rx timestamps when data enters the kernel. These timestamps are generated just after a device driver hands a packet to the kernel receive stack.

SOF_TIMESTAMPING_TX_HARDWARE:
Request tx timestamps generated by the network adapter. This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SOFTWARE:
Request tx timestamps when data leaves the kernel. These timestamps are generated in the device driver as close as possible, but always prior to, passing the packet to the network interface. Hence, they require driver support and may not be available for all devices. This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_SCHED:
Request tx timestamps prior to entering the packet scheduler. Kernel transmit latency is, if long, often dominated by queuing delay. The difference between this timestamp and one taken at SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent of protocol processing. The latency incurred in protocol processing, if any, can be computed by subtracting a userspace timestamp taken immediately before send() from this timestamp. On machines with virtual devices where a transmitted packet travels through multiple devices and, hence, multiple packet schedulers, a timestamp is generated at each layer. This allows for fine grained measurement of queuing delay. This flag can be enabled via both socket options and control messages.

SOF_TIMESTAMPING_TX_ACK:
Request tx timestamps when all data in the send buffer has been acknowledged. This only makes sense for reliable protocols. It is currently only implemented for TCP. For that protocol, it may over-report measurement, because the timestamp is generated when all data up to and including the buffer at send() was acknowledged: the cumulative acknowledgment. The mechanism ignores SACK and FACK. This flag can be enabled via both socket options and control messages.

At least the pcap timestamps are measuring some of the stack, even if not all. I hope you find this consideration of which bits of bits a bit disturbing. What is the point of a clever 500 nanosecond processing time even if you have a fast 1 to 2 microsecond SolarFlare card if the packet is being coalesced into a buffer for 40,000 damned nanoseconds before you get told?! We have to do better than this.

A good start to measuring t tick to trade better is to use an external measurement device that doesn’t rely on pcap behaviour and settings. You can see in the above list of pcap's settings, the library supports co-operating hardware. Pcap will extract a hardware timestamp by curious packet magic. Often the checksum of the Ethernet frame is used. Perhaps your solution may just add the timestamping data at the end of the Ethernet frame.

A timing card, such as an Endace DAG will stamp your packets from the wire:

Endace DAG example

It is the normal convention for hardware to punch the timestamp into the timing mechanism for the start of the packet. This is different to the normal receive (rx) stamp in the kernel as you now know. It is now the same end, being the start, used by pcap on the transmit (tx) side of the sending the order. To a degree, if we ignore some network stack overhead fuzziness.

If we do this and get our packet traces up in Wireshark, we’ll see at least a 40 microsecond difference for the t tick to trade latency for hardware compared to software pcap. The hardware timestamping is what we really want. We need this improved understanding of our latencies.

Endace cards default to timestamping at the start of the packet, as is the usual convention. Most of their cards do have the option to make this at the end of the packet. This is sometimes useful. For example you may want to measure properly the overhead of software pcap. If you timestamp via both software and hardware and have a mechanism to correlate the time bases you can learn useful things about your network stack.

We’ve just moved the goalposts to make our life harder with zero latency becoming 40 microseconds. It's much harder to wrangle our thinking into a negative latency way forward now. Truth in measurement is better than fooling ourselves. One of the beautiful aspects of proprietary trading I've always enjoyed is that only the science and engineering matters. Everything has a purpose. Marketing, colour, philosophical fluff may entertain some but it not important to the ultimate meritocracy that is the field of proprietary trading. With many thousands of very clever people opposing your quest, intellecutal honesty is paramount. Honest, fair dinkum, negative latency is an achieveable thing. You’ll soon know how to do this for a variety of circumstances.

One problem with the Endace DAG cards is not only are they expensive, you’ll also need network taps or splitters plus a number of ports to capture anything. You will also have to keep jiggling around your connections to reconnect and measure different parts of your network. Add to that the simple fact that the cards randomly timestamp to a window over within 6ns which translates, due to the jitter and clock boundary conditions, to being really only accurate to within 10 to 20 nanoseconds. This is a pain in the accuracy ass, especially when dealing with 50 nanosecond minimum length packets on a 10Gbps wire or fibre. What a pain!

A much better and more economical solution is one of the Metamako devices that can split, tap, timestamp accurately to 1ns all on many ports at once.

Metamako MetaConnect 48

Also, you can repatch by UI or API remotely. If you do this stuff seriously and you’re not using Metamako gear for your trading, you’re doing something wrong. Deutsche Börse is doing it right :

You can get by with tcpdump but it is awkward and prone to unreliability. If you do go down that path, remember the “packet length” in time to make the adjustments manually and do pin the capture task on its own processor with suitably high priority. Rolling your own analysis tools is useful. I highly recommend libtins as higher level way of easing yourself into some pleasant C++ to augment the use of libpcap if you're so inclined beyond the plain plane.

Back to getting negative latency… are we now measuring the right thing?

Nope, it is actually worse than you think

We now know for this 500 byte 100Mbps example we are 40us behind the game when we get told by the OS the packet has arrived.

It is much worse though. We have to do two things here. We have to understand what is important in getting our order to hit before others and how the data is actually getting to us from the exchange and back to the exchange.

All exchanges in the world are currently software based. Some use Infiniband and some use Ethernet in their inner rings. What matters is getting your order first to the matching engine that is running the auction for your symbol. What does getting there first mean, getting all of your packet, to the last byte, to the matching engine. You may have to traverse a number of switches, machines / gateways, firewalls, cables, protocol translations, routers, rate limiters, all sorts, but the final arbiter is normally the matching engine.

CME’s auction is a little a different for the iLink gateways in that a timestamp at the Market Segment Gateways is put place using a SolarFlare card's WODA hardware on message receipt. This is the vital point where the ordering for the auction is confirmed. (As has been confirmed to me June 13.)

What is important is that a full message, normally a network packet, has to arrive at some exchange element, normally the matching engine, but sometimes the gateway, and that sets your priority.

Your market data is a rear view mirror view on to that data. It doesn’t matter if it is the SIP or a direct data feed. All you are seeing a version of history that used to exist at the auction site. There is no certainty about the current state of the auction when you trade, just the hopes and dreams of a trader ready to be crushed by the boot of reality.

In the example we are using I’m drawing on the old KRX as a proxy. I’m using round numbers rather than the real ones but the idea is there.

Let’s have a closer look at the premises at the broker’s site:

The market data path is now complicated. In this example, it is coming in on fibre, travelling via some telecommunications gear to come out as interleaved channelised layers that are feed through some T1 or E1 modems to come out as Ethernet. Fundamentally though, the data is only coming in at 256kbps. How long is our packet really?

4,000 bits at 1/256,000 or ~3.9 microseconds per bit for 15.625 milliseconds in length. That’s a helluva holy whole hole in the thinking about timing. If you click on the diagram above you will see the trading engine market data input time is now labelled t MD 0 .

At t MD 2 we’re getting a 256kbps feed. If that ends up going out at 100Mbps then it will store the packet until complete and then send out on the Ethernet’s wire. With no overhead , it would take 15,625 microseconds to store before forwarding the packet onto the next step. Plus we need to add the 40 microseconds for the wire time. Next, add the overhead of the low-end Cisco store and forward switch (a 29xx series) being used, say 20 microseconds, plus another 40 microseconds of wire time.

What we would traditionally measure as zero latency with tcpdump on the trading server is really 40 us latency with an external measurement from the front of the packet. Which is really at least 100 us at t MD 1 . Or 15,725 us at t MD 2 . And perhaps this means 15,825 us of latency as measured at t MD 3 at the telco gear interface to the building’s fibre (generously).

Our market data latency is worse then we think here. Let’s sit on that for a minute and think about it later. Recollecting the waste is making my head hurt.

Transmission tricks

Turning now to the other side of the equation, the TX. We normally start to measure as the packet starts going out of the trading engine. Our zero latency libpcap is at least 40 microseconds in reality with proper external measurement. Let’s assume that even though we are talking 100Mbps Ethernet there is a constraining 256kbps channel for the order line, as there was in a version of history at the KRX. This may be a physical channel, a real rate limited channellised group via some E1 or T1 line, or a reserving rate bucket limiter on a Cisco device somewhere in the bowels of the KRX campus datacentre which we’ll consider the same even though they are not.

Let's assume our order packet is also 500 bytes or 4,000 bits to keep the numbers simple. It too would take 15,625 microseconds to transition through the channel at 256kbps.

Stuff that. We know there is a TCP/IP header plus some KRX standard crapola that we need to send before each message. It is just TCP so there is no need to send an order message all in the same packet , so let’s send one TCP packet followed by another to complete the order message. Here we are costing an extra TCP/IP header but saving on the KRX order message. Let’s say it is 20 bytes or 160 bits from the order we save. That’s about 3.9us per bit, tongue to the side, tilt the head, and we save 625 microseconds if we send that section of the order before we even know we have to send an order.

Is it a true saving? Yes. Our packets will be joined by the matching engines, really the gateway process in this case, and they will really, truly, cross-my-heart be there 625 microseconds earlier.

Calculating the effective latency

Without any RX trick, we have just shaved 625 microseconds off our latency. Let's say we’re as slow as a wet week in Huonville. We take as long as 20 microseconds to process the market data and produce an order, or really an order completion.

Let's look at the timing. That is 40 microseconds of waiting for the packet in our normal buffer, plus 20 microseconds of glacial processing, less 625 microseconds we have saved via order specualtion, for a nett result of:

-565 microseconds.

Wait, what? Is that really a true comparison of a tick to trade where we measure the t MD to t Order .

It is. Folks we’ve gone negative. We have time travelled thanks to the speculation on the start of the order packet. This is because what matters is when the matching engine's reference point receives the last byte of the message, not the first byte.

It is an implied negative latency. This is the only real way to compare it to your normal tick to trade timing which would be a zero latency or 40 microsecond latency under magical no-overhead conditions. With negative latency, your order really will arrive faster and your hit rate really does go up. It is a thing; not a theoretical construct.

For this case with the ye olde KRX, my hit rates actually went up better than expected. This was likely because the KRX used store and forward switches so smaller packets traversed quicker. Also, sending speculative packets lightened your bandwidth at critical times getting you a little more ahead at each store and forward stage, kind of a nice upwardly beneficial percolating effect.

Now you can read more about the packet slicing, tapping at t MD 3 tricks and the TCP out of order sequence numbering, and IP fragmentation tricks along with the crazy ARIA cipher CBC hacking I got up to in The accidental HFT firm .

The real truth was that the RX and TX trickery saved us milliseconds in those ancient times. This was gradually whittled away as the exchange increased the bandwidth at various stages, but it remained very healthy for many years. I expect I’ll talk about a bunch of further tricks in the book.

A backlash against negative latency

When I started up Zeptonics after the accidental HFT firm, we marketed a KRX gateway that had better and different tricks to that old HFT firm. We also expanded the market data capture to slice after the bid field arrived, before the ask field. As you now know by hacking the telco fibre coming into the building we could get a very nice early read as we rode the earliest possible bit. There was 75 microseconds between the bid and ask fields even though they were both early in the packet. It was useful to know the bid in advance.

Near the start of 2010 Lime Brokerage in NY started marketing their broking service as a nett negative latency service. It wasn’t really. The fine print was comparing their service to what they thought you could do as they surmised they would have less overhead. “Less overhead than you” Lime called negative latency. The argument makes a bit of sense. They are saying using their service instead of building your own saves you latency whereas a normal broker would be slower than your custom system. That kind of make sense but I disagree with the use of the term. They would have slowed me down for sure.

Here I am using negative latency to mean faster than if you took no time at all to respond, not Lime's misrepresentation. This is both accurate and reasonable even if it’s a little hard to get your head around. More money from better hit rates doesn’t lie.

Lime sullied the "negative latency" term a little. When Zeptonics started on a similar marketing track we advertised, initially, a negative latency of at least 100 microseconds. Reality was better, but selling 100 microseconds was hard enough. We heard a lot of people saying to us, “Yeah, right, F * off…” Fair enough really. It is an odd idea. Plug your own trading engine into this gateway and suddenly you'll have less latency to the exchange and improved hit rates. What the funk? Do something extra and it takes less time? Sounds a bit daft, doesn't it? If something sounds too good to be true, it must be, right? Just imagine if we had tried to tell them the truth that it was a multiple of 100 microseconds faster than zero. Have I got an ICO for you!

So, we started referring to it as “trade acceleration” which feedback told us was less confronting,

“ZeptoAccess KRX provides direct trading access to the Korea Exchange (KRX) at a speed that is 100 microseconds faster than a direct wire connection. This somewhat counter-intuitive result - a gateway that’s faster than having no gateway at all - is made possible by harnessing patent-pending “trade acceleration” technology developed by Zeptonics.”

Here is a snapshot from the wayback machine from, well, wayback, or 2012 to be precise:

So why is a crusty old exchange relevant to me?

I can hear you mumbling, “So, you need some old crusty exchange with ancient T1/E1 lines like the KRX? Dude, move on already. Stop wasting my time.”

No. Many people have done similar. No, not waste your time, but achieve some trade acceleration through technological trickery.

Nomura had a nice FPGA solution for the Tokyo Stock Exchange. They used a slightly different technique. The protocol was 100Mbps or 1Gbps Ethernet to a virtual TSE session via TCP/IP . They would speculate and use a CRC invalidation to kill the packet at the Ethernet layer if they didn’t have an order. Network stacks were a bit slower in those days. I heard on the grapevine that Nomura’s tick to trade was around 2.3 microseconds, much faster than anyone else could achieve at the time. Pretty slow by my standards at least.

As an example, think about trading at BATS in the ye olde daze of 1Gbps. BATS had a native FIX 4.2 protocol back then. In 2009 they were the fastest exchange on the planet with a 443 microsecond RTT. How quaint ;-)

A FIX 4.2 single order message starts off like this:

The standard header has a few compulsory fields such as FIX4.2 as the BeginString , a BodyLength , MsgType , SenderCompID , TargetCompID plus whatever unchanging fields following the standard header in the rest of the order message.

It may not be inconceivable to save 25 bytes in an Ethernet frame. That is not much at 1Gbps. It is only 200 bits or 200 nanoseconds. You’ll need an FPGA to do it, but being 200 nanoseconds ahead of everyone else may be worth it. Perhaps you don’t need to wait for 100 bytes of the 200 byte market data frame, you can save another 800 bits or 800 nanoseconds. Now we’re suddenly talking about a negative latency or trade acceleration of a full microsecond. Real money.

Today's markets are mostly 10G which is much harder. Short packets are 50ns on the wire. A net save on a similar protocol to the BATS example would only be 100 nanoseconds, but perhaps worth it to some. Savings are savings. Faster is faster. Speed is money. Three word sentences end here.

For modern FPGAs in trading, time in the SERDES, serialising on and off the wire, tends to dominate processing rather than doing work. Tick to trade for a good FPGA system on 10Gbps Ethernet is under 100 nanoseconds, mainly in the SERDES. Perhaps you'll find negative latency remains possible if your protocol and IT team are kind to you.

I hope this helps explain what negative latency is. It doesn’t exist. It is just that we can move our frame of reference so our latency bounds are shifted. The implied latency in comparison is negative. This is just because of the limitations of our original point of view. Good engineers will optimise the problem you give them. Give them a broader view so they just don't optimise an isolated problem that keeps you behind in the latency race. The best engineers step back, if allowed, to view the full extent of the battlefield.

Think outside the trading engine box.



Happy trading,

–Matt.



Feed back on making this easier and better to read would be most welcome. I expect I’ll expand on this and add to it, in a more narrative form in the planned book, The accidental HFT firm If you’re interested in such details and real world war stories please think about supporting the Kickstarter for the book .

Monday, 11 June 2018

Shortwave trading - Who is that in Bob and Sniper's blog?

Bob and Alexandre, aka Sniper, provide the great pictures of the two Shortwave or HF antennae. They allude to the addresses but don't explicitly name the owners. Well, here are their linked FCC documents showing who the antennae belong to:

LineChard Capital LLC via Google Earth
(click to enlarge)
As you can see from FCC experimental form Bob and Sniper refer to:
FCC Form 442 - highlighting Richard Reeuwijk's name
(click to enlarge)
Richard Reeuwijk is the contact:

Richard's LinkedIn page - LineChard Capital (ex Optiver)
(click to enlarge)
As Sniper points out the Elburn Illinois towers are a bit better hidden until you see the property records:
Sniper's link to the image DB (45W160 Beith Road)
(click to enlarge)
Zooming out from the image above
Compare and contrast with Bob and Sniper's land records:


(click to enlarge)
So, which trading firm is at 600 W Chicago Ave?

Well, there may be more than one, as it is a big building, but this one may stand out to you:

Jump's Bloomberg profile
(click to enlarge)
I can understand why Sniper and Bob don't mention the company names explicitly as it may be impolite if you talk to people at those firms. I tend not to mention such data about the names of firms who chat to me. So, just let me know about your HF trading sites and I'll make sure I don't meander about your Shortwave Trading meanderings ;-)

Frankly, the only disturbing thing I see here is the rate of the rate increase for Jump's land: almost doubling in six years. Yikes. Government piracy at its worst!

Happy trading,

--Matt.




Sunday, 10 June 2018

The accidental HFT firm - the book

A Kickstarter for your consideration.


Korean trading. Fun and games with Japanese infrastructure. The Toronto - New Jersey interlisted arb. How not to negotiate multi-million dollar deals. Reverse engineering the ASX. Building the world's fastest (at the time) network switch and matching engine. Killing canaries. The ATS that wasn't. Lots of reading fun to be had by all.

An inspirational bit of "generative art" - I hope I'll be quicker

Condensation Cube, Plexiglas and water;
Hirshhorn Museum and Sculpture Garden,
begun 1965, completed 2008 by Hans Haacke 
I've gratefully received much positive feedback after writing in Automated Trader and then blogging my brief account of starting an HFT firm somewhat accidentally, "The accidental HFT firm."

Some well-wishers have requested that it be fleshed out into a book form with quite a bit more detail. This is the project to meet their requests.

Whilst it feels a bit of a self-indulgent, vanity project, I believe, from the requests I've received, there may be genuine interest. The murky dark corners of finance that are never spoken about, even in hushed tones, can be quite intriguing. Even those in the industry may like others to understand what they do but can't really talk about.

HFT needs a voice - warts and all.
"Now we were sending out orders
before the data packet arrived," --Matt.

Then again, perhaps it's just therapy. I'm OK with that.

Happy trading,

--Matt.
______________________

Risks and challenges


Procrastination and writer's block are the chief challenges. I need to produce 50,000 to 100,000 words that are not all of dubious quality.

Originally the 12,000 words took only a fairly long week. I expect a better account with more depth will take some months of writing, revision, restructuring, and starting again. I've never done this before, so I don't really know how I'll travel. I hope the time frame is reasonable. I think it is, but am uncertain.

Saturday, 9 June 2018

IEX May 2018 update

IEX remains the dark and expensive public exchange you should avoid dining at. Unless, of course,  you enjoy fake pretentious marketing where the reality devoured is devoid of any relationship to the veritable untruths hawked by the vendor.

IEX market share has been fairly static since April 2017. It continues to hover around the 2.2 to 2.5% mark.
(click to enlarge)
IEX has gone from a staggering unlit percentage of around 80% to a staggering unlit percentage of around 75%. AFAIK the SEC has still not thought about reversing its incorrect decision to grant IEX a license. Public price discovery be damned.

(click to enlarge)

Greater share each month continues to be correlated with being darker. The more you feed it, the eviler it gets.  Just don't feed it after midnight, or before for that matter.

(click to enlarge)
 A simpler monthly summary.
(click to enlarge)


The market infection continues.

Happy trading,

--Matt.

Wednesday, 6 June 2018

Top non-isp meanderers from the last year

Meander
(source, click to enlarge)
After clocking over 400,000 meanderers recently - a relatively tiny number in Internet speak, I thought I'd look at the popular non-ISP readers from the webby logs. Now, it is not too relevant as the vast majority come through ISPs or networks that disguise the reader, but it was an interesting exercise - to me at least:

Here is an ordered list of the first few hundred of biggest meanderers over the last year and bit.

Thank you for your interest, however misplaced that may be,

--Matt.

_____________


  • amazon technologies inc.
  • susquehanna investment group
  • microsoft corp
  • amazon.com inc.
  • dean witter financial services
  • two sigma investments llc
  • imc-chicago llc
  • morgan stanley group inc.
  • intel corporation
  • citadel investment group l.l.c.
  • jpmorgan chase & co.
  • hudson river trading llc
  • barclays capital
  • knight capital group inc
  • quantlab
  • operations & compliance network llc
  • susquehanna international group limited
  • bank of america
  • drw holdings
  • nasdaq inc.
  • chicago board of trade
  • ubs ag
  • lightower fiber networks i llc
  • bloomberg financial market
  • credit-suisse-group
  • deutsche bank
  • cinnober financial technology ab
  • microsoft corporation
  • u.s. securities & exchange commission
  • rbc capital markets corporation
  • optiver holding b.v.
  • amazon corporate llc
  • imc (international marketmakers combination)
  • intercontinentalexchange inc.
  • exponential-e ltd.
  • citigroup
  • akuna capital
  • citicorp global information network
  • fidelity investments
  • deutsche boerse ag
  • royal bank of canada
  • thinktech inc
  • wolverine trading llc
  • alliancebernstein l.p.
  • spark new zealand trading ltd
  • bats exchange inc.
  • instinet
  • getco llc
  • point72 asset management lp
  • samsungsds inc.
  • australia and new zealand banking group ltd
  • bnp paribas north america inc.
  • direct edge
  • wayport inc.
  • cisco systems inc.
  • jacobs levy equity management inc.
  • xilinx incorporated
  • flow traders b.v.
  • thomson reuters u.s. llc
  • wilfrid laurier university
  • the university of melbourne
  • new york university
  • oracle corporation
  • stanford university
  • themis trading
  • toronto dominion bank
  • tower research capital
  • abc arbitrage asset management
  • apple inc.
  • bnp paribas sa
  • vatic labs operations llc
  • columbia university
  • imcpacific-sy2-sipc3-00-1
  • massachusetts institute of technology
  • nordea bank denmark
  • avago technologies u.s. inc.
  • vultr holdings llc
  • aqr capital management
  • mellanox-technologies-ltd
  • university of cambridge
  • alibaba.com llc
  • australian academic and research network
  • morgan stanley & co intl plc
  • nasdaq technology ab
  • tbricks ab
  • the goldman sachs group inc.
  • cambridge
  • monkey brains
  • university of washington
  • virtu financial llc
  • northrop grumman corp.
  • parallax volatility advisers llc
  • sg americas operational services inc.
  • concord energy
  • cutler group lp
  • dimensional fund advisors
  • flow traders us llc
  • hewlett-packard company
  • nvidia corporation
  • spark new zealand trading limited
  • taiwan academic network
  • vmware inc.
  • cantor fitzgerald & co.
  • capital one financial corporation
  • crabel capital management llc
  • european bank for reconstrcution and development
  • google llc
  • gramian
  • kcg europe limited
  • millenium partners l.p.
  • northeastern university
  • thomson reuters (legal) inc.
  • tower research capital in
  • university of chicago
  • intercontinental exchange inc.
  • london trust media inc
  • london westend datacenter
  • synopsys inc.
  • amanah tech inc.
  • andrews & arnold ltd
  • google inc.
  • hewlett packard enterprise company
  • marshall wace asset management ltd
  • navy network information center (nnic)
  • seoul national university
  • the bloomsbury computing consortium
  • university of new south wales
  • university of sydney
  • wellington management company llp
  • barclays global investors
  • colt technology services group limited
  • dod network information center
  • dow jones-telerate
  • gelber group llc
  • millennium partners l.p.
  • monash university
  • raytheon company
  • softbank corp.
  • the bank of new york mellon corporation
  • the vanguard group inc.
  • university of florida
  • university of illinois
  • weeden & co. l.p.
  • advanced micro devices inc.
  • blackstone administrative serv
  • bluefin trading llc
  • cibc world markets
  • commerzbank ag
  • freehills
  • grantham mayo van otterloo
  • htg capital partners llc
  • mako
  • nokia corporation
  • northwestern university
  • oxford university
  • the toronto stock exchange
  • tower research capital llc
  • tradeweb llc
  • uber technologies inc
  • allston trading
  • commonwealth bank of australia
  • frostbyte consultancy
  • kabel baden-wuerttemburg gmbh & co. kg
  • kordia ltd
  • liberty mutual group
  • new york mercantile exchange
  • pricewaterhouse coopers ltd
  • tudor investment corporation
  • university of california los angeles
  • university of toronto
  • university of virginia
  • arm ltd cambridge 1
  • carnegie mellon university
  • chicago trading company
  • ebay inc
  • georgia institute of technology
  • itg derivatives llc.
  • itiviti group ab
  • johann wolfgang goethe-universitaet frankfurt
  • juniper networks inc.
  • kbs group
  • korea stock exchange
  • mass mutual
  • mckinsey & company inc.
  • motorola solutions inc.
  • national aeronautics and space administration
  • national australia bank limited
  • optiver australia pty ltd #2
  • princeton university
  • rochester institute of technology
  • state farm mutual automobile insurance company
  • sun trading llc
  • tyler capital limited
  • university of adelaide
  • university of colorado
  • university of michigan
  • university of notre dame
  • university of texas at austin
  • x t x markets ltd
  • yale university
  • bny brokerage
  • bridgewater associates lp
  • cray inc.
  • department of parliamentary services
  • equinix inc.
  • european central bank
  • fisher investments 1450
  • freescale semiconductor inc.
  • government of singapore investment corporation
  • guavatech inc.
  • harris corporation
  • harris trust & savings bank
  • harvard university
  • hynix semiconductor inc.
  • juniper networks international b.v.
  • lockheed martin corporation
  • man investments limited
  • middlebury college
  • moody s investors service
  • palantir technologies inc.
  • paypal inc.
  • snl financial
  • suntrust service corporation
  • supermicro
  • teachers insurance and annuity association of america
  • tibra global services
  • university of california san diego
  • university of minnesota
  • wave data centers llc
  • wilmer cutler pickering hale
  • zayo group eu limited
  • abacus group llc
  • abc arbitrage
  • acc-equation technologies inc
  • apollo global management
  • arm ltd austin tx
  • bae systems applied intelligence us corp.
  • banco central do brasil
  • burson-marsteller
  • canaccord genuity inc.
  • charles schwab & co. inc.
  • concepts ict holding b.v.
  • delta ltd.
  • department of finance (wa)
  • emc corporation
  • fisher investments
  • fts2001/commodity futures trading commission
  • graham capital management
  • headlands technologies llc
  • hyundai international merchant bank
  • invitech megoldasok zrt.
  • johns hopkins university applied physics laboratory
  • kabel baden-wuerttemberg gmbh & co. kg
  • korea advanced institute of science and technology
  • krypt technologies
  • london stock exchange plc
  • macquarie holdings (usa) inc.
  • nasdaqomx
  • nomura america services llc
  • pdt partners llc
  • providus capital pty ltd
  • solarflare communications
  • squarepoint ops llc
  • svyaz-holding ltd.
  • t. rutkausko i.i. baltsoft software
  • texas a&m university
  • the new york times company
  • the royal bank of scotland
  • trading technologies
  • universitaet heidelberg
  • university of maryland
  • university of rochester
  • university of utah
  • upc ceska republika s.r.o.
  • upc slovakia
  • william blair & company
  • windtre s.p.a
  • xangrila incorporated
  • abu dhabi investment authority
  • adobe systems inc.
  • aegon usa inc.
  • allstate insurance company
  • alpine investment management l
  • anchorage capital group
  • archelon llc
  • argonne national laboratory
  • ashurst australia services pty limited
  • australia and new zealand banking group limited (anz)
  • bain and company inc.
  • bank of montreal
  • bats trading limited
  • bids trading l.p.
  • boston university
  • brevan howard asset management llp
  • calamos advisors llc
  • california institute of technology
  • california polytechnic state university
  • cerner corporation
  • cftc
  • chicago stock exchange
  • clemson university
  • coinbase inc
  • cornell university
  • dartmouth college
  • davis polk & wardwell
  • disney worldwide services inc.
  • dv trading llc
  • electronics and telecommunications research institute
  • excell group plc
  • fidessa corporation
  • finovesta gmbh
  • first rand bank
  • ford motor company
  • ftip003358740 nyse technologies inc
  • general dynamics advanced information systems inc.
  • general motors llc
  • hanyang university
  • hibernia networks uk customer space
  • ibm
  • internet initiative japan inc.
  • katholieke universiteit leuven
  • korea university
  • lendingclub corporation
  • ludwig-maximilians-universitaet muenchen
  • macquarie bank
  • magna international inc.
  • magnetar capital llc
  • maya trombik
  • mcgill university
  • melita plc
  • mentor graphics corporation
  • morrison & foerster
  • nobis technology group llc
  • nomura network 1
  • nomura network 2
  • northwestern polytechnic university
  • nsw department of education
  • o reilly media inc.
  • old mission capital
  • oracle svenska ab
  • palo alto networks techops
  • pantor engineering ab
  • pricewaterhouse coopers services ltd
  • pricewaterhousecoopers llp
  • qbe insurance group ltd
  • queensland university of technology
  • richard fleischman and associates
  • royal bank of canada trust corporation limited
  • sac affaires vl
  • samsung research america inc.
  • savings bank of the russian federation (sberbank)
  • schonfeld tools llc.
  • sfr sa
  • sigma edge ltd
  • simon fraser university
  • state street imswest
  • swiss federal institute of technology zurich
  • telecity london customer - millennium capital management ltd
  • the pennsylvania state university
  • the university of manchester
  • timber hill llc
  • trayport limited
  • union bank of california
  • universitaet mannheim
  • university of bristol
  • university of california
  • university of texas at dallas
  • university of waterloo
  • university of wisconsin madison
  • us department of the treasury
  • xambala inc
  • xilinx
  • xtx markets ltd
  • accenture llp
  • accenture services pvt ltd
  • analysis group inc.
  • analysts international
  • apg groep n.v.
  • apn holdings nz limited
  • apple inc
  • arista networks inc.
  • atomic data centers internal and employee networks
  • australian stock exchange
  • australian wealth management
  • avant inc.
  • avnet inc.
  • axa financial inc.
  • baader bank ag
  • balyasny europe asset management
  • bank for international settlements
  • banque nationale du canada
  • baxter healthcare corporation
  • bis ltd
  • blackberry limited
  • blazenet technologies ltd
  • blue ridge websoft llc
  • bnp paribas partners for innovation (suisse) sarl
  • brown university
  • brownstone investment group llc
  • cadence design systems inc.
  • caiw diensten b.v.
  • canada pension plan investment board
  • case western reserve university
  • cern - european organization for nuclear research
  • chevron corporation
  • chicago board options exchange
  • chimera capital
  • church pension group services corporation
  • cicada limited
  • cirrus logic inc.
  • city of atlanta dept of av - aep lan
  • clearpool technologies llc
  • cme operations limited
  • colt-uk-wan-106
  • credit agricole s.a.
  • crosseas capital services
  • crossland llc
  • daimler ag
  • danske bank a/s
  • dekabank deutsche girozentrale
  • dell inc.
  • deloitte touche tohmatsu services inc.
  • department of premier and cabinet (victoria)
  • dow jones & company inc.
  • duke university
  • duquesne capital management
  • e*trade
  • eaton vance management
  • emory university
  • equinix singapore
  • ernst & young services pty ltd
  • exegy inc.
  • fbi criminal justice information systems
  • fermi national accelerator laboratory (fermilab)
  • fiat information technology excellence and methods s.p.a
  • fixnetix inc
  • flextronics international usa inc.
  • fortinet inc.
  • fraunhofer-gesellschaft zur foerderung der angewandten forschung
  • ftip003456620 bank of england
  • fulcrum networks ltd
  • globalfoundries u.s. inc.
  • hbk investments
  • hsbc banking and financial services
  • illinois tool works
  • imperial college london
  • indian institute of technology kanpur
  • indian institute of technology madras beside adyar cancer instit
  • investec
  • investment company institute
  • itg
  • jackson memorial hospital public health
  • koscom
  • kpmg
  • kpmg llp
  • kpmg luxembourg
  • kyivstar pjsc
  • landmark plc
  • latrobe university
  • lattice semiconductor
  • lawrence livermore national laboratory
  • lmax limited
  • london school of economics and political science
  • longwood medical and academic area (lma)
  • loomis sayles & company l p
  • los alamos national laboratory
  • marsh inc.
  • microchip technology
  • micron technology
  • microsoft - uk
  • microsoft limited
  • mitsui & co. commodity risk management inc.
  • morgan stanley australia
  • morgan stanley lan
  • morningstar inc.
  • mos enterprises
  • mugler ag
  • nanyang polytechnic
  • national center for atmospheric research
  • national chung cheng university
  • national institutes of health
  • nationwide mutual insurance company
  • natixis north america llc
  • network of nvidia corporation
  • neuberger berman
  • newton abbot
  • nodisto it llc
  • nordea bank ab
  • nrma insurance ltd
  • nvidia-asia
  • o grady peyton int l
  • oao vitebskiy oblastnoy techno-torgoviy center garant
  • ohio state university
  • open systems solutions inc.
  • orc group ab
  • otc markets group inc
  • pacific alternative asset man
  • paul weiss rifkind wharton
  • paxio inc.
  • pico quantitative trading llc
  • precious treasure pte ltd
  • pricewaterhousecoopers australia
  • prudential
  • pure storage inc.
  • queens university belfast
  • raymond james financial inc.
  • raytheon australia
  • saab ab
  • sahyadri hospitals limited
  • sandia national laboratories
  • sap america inc.
  • schroder investment management limited
  • scotia mcleod inc.
  • secure dragon llc.
  • shenzhen tencent computer systems company limited
  • state university of new york at stony brook
  • sumitomo mitsui banking corporation
  • super micro2
  • synerwealth financial ltd
  • t-systems international gmbh
  • t. rowe price associates inc.
  • tasmanian government
  • texas instruments
  • the boeing company
  • the capital group companies inc.
  • the children s hospital of philadelphia
  • the george washington university
  • the hong kong university of science and technology
  • the interchurch center/columbia university
  • the johns hopkins medical institutions
  • the st. paul travelers companies inc.
  • time warner inc.
  • tiscali uk limited
  • u.s. house of representatives
  • union bank of switzerland
  • united technologies research center
  • unitedhealth group incorporated
  • universite de geneve
  • university of arizona
  • university of cincinnati
  • university of nebraska at omaha
  • university of southampton
  • wal-mart stores inc
  • western digital corporation
  • weta digital ltd
  • wideopenwest finance llc
  • wolverine trading technologies llc
  • xilinx (ap) pte ltd
  • abn amro clearing sydney pty ltd
  • adobe systems incorporated
  • airbus defense and space gmbh
  • airbus group inc.
  • allianz technology se
  • allied irish banks plc
  • ally financial inc.
  • amtrust north america inc.
  • aon corporation
  • arbiter partners
  • ares management llc
  • aristotle capital management llc
  • arizona state university
  • arnold & porter
  • arrowstreet capital l.p.
  • auckland college of education
  • australian broadcasting commission
  • australian centre for advanced computing
  • automated systems inc.
  • avea iletisim hizmetleri a.s.
  • b & p fund service ab
  • baidu usa llc
  • banco credit suisse (brasil) s.a
  • bank of china
  • bank of england
  • beneficial data processing corp.
  • bexleyheath
  • binghamton university
  • blackrock financial management inc.
  • bloomberg l.p.
  • bnp paribas cib arbitrage
  • board of trade clearing corporation
  • bowling green state university
  • boy scouts of america
  • british airways plc
  • brno university of technology
  • brooklyn institute of art and
  • broomhill
  • brown brothers harriman & co.
  • bucher-guyer ag
  • bullseye capital
  • calyon america services inc.
  • cambridge incubator inc.
  • canon information systems research australia pty ltd
  • capula investment management llp
  • cell c (pty) ltd
  • centurylink technology
  • chinatrust commercial bank
  • cineca consorzio interuniversitario
  • cjsc vtb capital
  • closed joint stock company moscow interbank currency exchange
  • coles myer ltd
  • college of charleston
  • colorado state university
  • columbia ventures corporation
  • commonwealth department of foreign affairs and trade
  • corporate risk solutions inc
  • csc australia pty ltd
  • cust_deloitte
  • delaware life insurance co
  • department of human services
  • detronics s.r.o.
  • deutsches elektronen-synchrotron desy
  • drw holdings llc
  • epic systems corporation
  • eulex - euroope union rule of law mission
  • european synchrotron radiation facility
  • exegy incorporated
  • exxon mobil corporation
  • facebook inc
  • fidelity national information services inc.
  • finance canada and treasury board secretariat
  • fixnetix ltd
  • flinders university
  • flowroute inc
  • fortress group - burys st edmonds
  • fraunhofer-institut fuer nachrichtentechnik
  • freie universitaet berlin
  • freshminds talent ltd
  • fujitsu limited
  • fulcrum asset management
  • general motors corp.
  • george mason university
  • gfis gmbh
  • globalflows inc.
  • google incorporated
  • government of singapore
  • great point trading llc
  • gsa capital
  • guggenheim services llc
  • hankuk university of foreign studies computer center
  • hauser und wirth ltd.
  • highbridge capital management llc.
  • hsbc bank usa
  • huawei technologies duesseldorf gmbh
  • huntington bancshares inc.
  • hutchin hill capital lp
  • hvsl1-wacker llc dba serendipity labs
  • ibasis inc.
  • icap management services limited
  • indian institute of technology delhi
  • indiana university
  • ing financial markets
  • institut fuer informatik der tu muenchen rechnerbetriebsgruppe b
  • institut superieur d informatique de modelisation
  • intelligent technologies s.a.
  • international business machines corporation
  • international house
  • interxion holding nv
  • intuit inc.
  • investec bank ltd
  • jsc tinkoff credit systems bank
  • juniper networks australia
  • keybank national association
  • kirkland & ellis llp
  • kohler company
  • korbank s. a.
  • kpmg luxembourg s.c.
  • laurion capital management lp
  • lazard freres
  • leidos inc.
  • lek securities corporation
  • linkopings universitet
  • lord abbett & co.
  • macquarie bank european internet range
  • marex financial ltd
  • marian court college
  • marvell semiconductor inc.
  • mcafee inc.
  • mellon bank
  • menlo security inc.
  • mercer (us) inc.
  • mesirow financial
  • mirae asset global investments (usa) llc
  • moore capital management
  • national taiwan ocean university
  • new york life insurance company
  • noaa
  • non state educational institution educational scientific and exp
  • norges bank
  • north central college
  • northern territory government
  • nsw dept of commerce ref. ac3
  • ocado ltd
  • och ziff capital management sold by adam phones limited
  • office of the chief information officer
  • options london
  • orix usa corporation
  • panasonic avionics corporation
  • paradigma tecnologico s.l.
  • paramount pictures
  • parliament of victoria
  • pathpartner technology co
  • pccw limited
  • petroleo brasileiro s.a. - petrobras
  • pioneer pa space
  • plaid technologies inc.
  • politechnika warszawska
  • powszechna kasa oszczednosci bank polski sa
  • raiffeisen informatik gmbh
  • rand financial services inc.
  • rapid7 llc
  • rbs securities inc.
  • reed elsevier
  • renaissance technologies corp.
  • riotgames
  • rockefeller group technology solutions inc.
  • rockley photonics
  • rockwell automation inc.
  • rockwell collins inc.
  • ronin capital llc
  • royal bank of scotland
  • rutgers university
  • rwth aachen university
  • saint-petersburg state university of information
  • sap se
  • scottrade inc.
  • silesian university of technology computer centre
  • societe francaise du radiotelephone s.a.
  • solar turbines inc. mz g1
  • southpaw asset management lp
  • spectrum health
  • spot trading
  • square inc.
  • st. luke s roosevelt hospital center
  • state library of queensland
  • stockholm university
  • sun trading
  • suny rockland community college
  • swedish meteorological and hydrological institute
  • syracuse university
  • tcr holdings limted.
  • technion - technolog institute
  • technische universitaet carolo-wilhelmina zu braunschweig
  • technische universitaet hamburg-harburg
  • technische universiteit eindhoven
  • telecity group customer - goodbody stock brokers
  • the chicago board options exchange
  • the czech national bank
  • the manufacturers life insurance company
  • the mathworks incorporated
  • the university of edinburgh
  • the university of hong kong
  • the world bank group
  • thyssenkrupp business services gmbh
  • tiscali italia s.p.a.
  • tonbridge
  • tpsa departament osrodek informatyki
  • traditum group llc
  • trevena inc
  • tsinghua university
  • twitter
  • twitter inc.
  • u.s. department of energy
  • ukrchermetautomatica ltd.
  • united group limited
  • united kingdom atomic energy authority
  • united nations logistics base
  • universidad politecnica de madrid
  • universitaet stuttgart
  • university of california davis
  • university of california riverside
  • university of connecticut
  • university of dayton
  • university of leicester
  • university of massachusetts at boston
  • university of miami
  • university of michigan information and technology services
  • university of north carolina at chapel hill
  • university of pittsburgh
  • university of south carolina
  • university of southern california
  • university of surrey
  • university of tasmania
  • university of western ontario
  • us dept of justice
  • usda office of operations
  • vanderbilt university
  • victus capital management
  • viking global investors lp
  • virtual machine solutions llc
  • vwd vereinigte wirtschaftsdienste ag
  • washington post
  • washtenaw community college
  • westpac banking corporation nz
  • worcester polytechnic institute
  • xiamen university