Saturday, 13 December 2014

Sec. 309. Procedures for the retention of incidentally acquired communications

The only thing necessary for the triumph of evil is that good men do nothing.

H. R. 4681 Intelligence Authorization Act for Fiscal Year 2015.

Passed with 325 votes to 100.

Yep, it's pretty onerous and Orwellian, as is usual. It has slipped though Congress with scant attention and virtually no debate.

Section 309 is the most damning element. It contains enough outs and weasel words to allow the US intelligence community continue to break the US constitution forever. As mentioned, it is most troubling that this Act, with such a terrible Section, seemed to slip through Congress without too many people paying attention to the detail. Where was the debate?

The text is reproduced below, but here are a couple of highlights:
(1) Covered communication 
The term covered communication means any nonpublic telephone or electronic communication acquired without the consent of a person who is a party to the communication, including communications in electronic storage. 
A covered communication shall not be retained in excess of 5 years, unless— 
(iii) the communication is enciphered or reasonably believed to have a secret meaning; 
(vi) retention is necessary for technical assurance or compliance purposes... 
(vii) retention for a period in excess of 5 years is approved by the head of the element of the intelligence community responsible for such retention...
So, what do they do if they want to keep stuff for as long as they like? They can keep it forever by simply approving the retention themselves using (vii) above. They could keep anything just for technical assurance or for their own deemed compliance reasons (vi). Also, (iii) could easily be misinterpreted to provide their own basis for whatever they wish to do.

Of course, if you're one of the planet's 6.7 billion non-US citizens, there is no change. Your data continues to be owned by the greatest back-up and archival facility ever known to man:
(iv) all parties to the communication are reasonably believed to be non-United States persons
What kind of parliament would let something like this slip through with virtually no debate? Distracted by Christmas, budget approvals and the torture debate? Quite clever of the intelligence community to manage the political agenda with enough clout to get this through. It reinforces the idea that security is a game of unrelenting persistence, patience and scheming. Well done NSA and friends.

Eisenhower would be turning in his grave.


PS: A late burst from Congress managed to get 100 people to provide futile resistance but it was too little, too late. The NO vote was split fairly evenly between Democrats and Republicans.

PPS: Tim Berners-Lee on why internet freedom is important.


Sec. 309. Procedures for the retention of incidentally acquired communications

In this section:
Covered communication
The term covered communication means any nonpublic telephone or electronic communication acquired without the consent of a person who is a party to the communication, including communications in electronic storage.
Head of an element of the intelligence community
The term head of an element of the intelligence community means, as appropriate—
the head of an element of the intelligence community; or
the head of the department or agency containing such element.
United States person
The term United States person has the meaning given that term in section 101 of the Foreign Intelligence Surveillance Act of 1978 (50 U.S.C. 1801).

Procedures for covered communications
Requirement to adopt
Not later than 2 years after the date of the enactment of this Act each head of an element of the intelligence community shall adopt procedures approved by the Attorney General for such element that ensure compliance with the requirements of paragraph (3).
Coordination and approval
The procedures required by paragraph (1) shall be—
prepared in coordination with the Director of National Intelligence; and
approved by the Attorney General prior to issuance.
The procedures required by paragraph (1) shall apply to any intelligence collection activity not otherwise authorized by court order (including an order or certification issued by a court established under subsection (a) or (b) of section 103 of the Foreign Intelligence Surveillance Act of 1978 (50 U.S.C. 1803)), subpoena, or similar legal process that is reasonably anticipated to result in the acquisition of a covered communication to or from a United States person and shall permit the acquisition, retention, and dissemination of covered communications subject to the limitation in subparagraph (B).
Limitation on retention
A covered communication shall not be retained in excess of 5 years, unless—
the communication has been affirmatively determined, in whole or in part, to constitute foreign intelligence or counterintelligence or is necessary to understand or assess foreign intelligence or counterintelligence;
the communication is reasonably believed to constitute evidence of a crime and is retained by a law enforcement agency;
the communication is enciphered or reasonably believed to have a secret meaning;
all parties to the communication are reasonably believed to be non-United States persons;
retention is necessary to protect against an imminent threat to human life, in which case both the nature of the threat and the information to be retained shall be reported to the congressional intelligence committees not later than 30 days after the date such retention is extended under this clause;
retention is necessary for technical assurance or compliance purposes, including a court order or discovery obligation, in which case access to information retained for technical assurance or compliance purposes shall be reported to the congressional intelligence committees on an annual basis; or
retention for a period in excess of 5 years is approved by the head of the element of the intelligence community responsible for such retention, based on a determination that retention is necessary to protect the national security of the United States, in which case the head of such element shall provide to the congressional intelligence committees a written certification describing—
the reasons extended retention is necessary to protect the national security of the United States;
the duration for which the head of the element is authorizing retention;
the particular information to be retained; and
the measures the element of the intelligence community is taking to protect the privacy interests of United States persons or persons located inside the United States.

Sunday, 7 December 2014

FINRA ATS NSM Tier 1 weekly stats - week ended 10 Nov - updated 1 Dec

IEX is now #4. That Lava Flow at #6 is interesting in the context of their decision to shutdown in 2015. Credit Suisse and UBS remain at #1 and #2.

ITG is #12. One has to wonder if ITG customers know that "agency only" means that a customer's order for some names could possibly be traded against the ITG proprietary trading book which is hidden in plain sight? At least with Credit Suisse and Goldman Sachs you know of their potential to be on the other side of your trade. They don't mislead. Sometimes ITG discloses their prop trading activity in their 10-Q's, such as,
"Interlisted arbitrage trading, which accounts for most of our other revenues, generated $1.8 million in the third quarter of 2009, significantly down from the $4.4 million achieved in the comparable 2008 quarter due to a decrease in Canadian and U.S. interlisted trading volumes and an increase in competition in this area." Source: ITG - 10-Q for the fiscal period ended September 30, 2009
That, to me at least, seems to blur the common understanding of "agency only." Does this Bloomberg story hint that ITG's reputation is really as an agency-only shop without prop? I wonder what the SEC and a certain NY AG thinks? Maybe surprised customers could claim their trade fees from ITG back for the last decade or two?

Seven ATS platforms have "per trade" sizes well in excess of 1,000 shares and IEX isn't one of them. That is, despite their #FlashBoys hope to be a platform suitable for large institutional blocks, you still need to use HFT techniques to trade IEX well. IEX seem to be doing a good job but it is perhaps a little different to what Michael Lewis envisaged.

Happy trading,


Name Shares
Per trade
CROS CREDIT SUISSE SECURITIES (USA) LLC 332,102,591 11.25% 1,799,961 12.67% 185
UBSA UBS SECURITIES LLC 327,029,127 11.08% 2,299,498 16.18% 142
DBAX DEUTSCHE BANK SECURITIES INC. 195,254,900 6.61% 1,108,814 7.80% 176
IEXG IEX SERVICES LLC 191,720,074 6.50% 678,625 4.78% 283
MSPL MORGAN STANLEY & CO. LLC 171,558,600 5.81% 777,522 5.47% 221
FLOW LAVAFLOW, INC. 165,601,276 5.61% 834,201 5.87% 199
JPMX J.P. MORGAN SECURITIES LLC 163,801,415 5.55% 741,248 5.22% 221
SGMA GOLDMAN SACHS EXECUTION & CLEARING, L.P. 155,152,765 5.26% 839,811 5.91% 185
MLIX MERRILL LYNCH, PIERCE, FENNER & SMITH INCORPORATED 154,230,349 5.22% 672,986 4.74% 229
BIDS BIDS TRADING L.P. 130,247,400 4.41% 365,785 2.57% 356
KCGM KCG AMERICAS LLC 121,968,993 4.13% 709,040 4.99% 172
ITGP ITG INC. 121,452,700 4.11% 440,372 3.10% 276
EBXL LEVEL ATS 96,458,000 3.27% 616,258 4.34% 157
LATS BARCLAYS CAPITAL INC. 91,596,213 3.10% 551,491 3.88% 166
ICBX INSTINET, LLC 81,872,226 2.77% 338,540 2.38% 242
APOG CITADEL SECURITIES LLC 44,677,617 1.51% 246,816 1.74% 181
DLTA DEALERWEB INC. 43,896,832 1.49% 85 0.00% 516,433
XSTM NATIONAL FINANCIAL SERVICES LLC 42,367,701 1.44% 116,281 0.82% 364
LTPL CREDIT SUISSE SECURITIES (USA) LLC 38,513,986 1.30% 246,434 1.73% 156
PDQM PDQ ATS, INC. 37,457,924 1.27% 166,735 1.17% 225
MSTX MORGAN STANLEY & CO. LLC 34,003,400 1.15% 201,219 1.42% 169
LQNT LIQUIDNET, INC. 31,477,100 1.07% 676 0.00% 46,564
IATS INTERACTIVE BROKERS LLC 29,974,679 1.02% 95,669 0.67% 313
PULX STATE STREET GLOBAL MARKETS, LLC 25,872,846 0.88% 2,063 0.01% 12,541
NYFX CONVERGEX EXECUTION SOLUTIONS LLC 25,629,175 0.87% 38,386 0.27% 668
CXCX CITIGROUP GLOBAL MARKETS INC. 24,109,306 0.82% 140,051 0.99% 172
VRTX CONVERGEX EXECUTION SOLUTIONS LLC 17,684,595 0.60% 76,793 0.54% 230
LQFI CITIGROUP GLOBAL MARKETS INC. 13,384,483 0.45% 163 0.00% 82,113
XIST INSTINET, LLC 13,256,200 0.45% 18,602 0.13% 713
BOOK BLOOMBERG TRADEBOOK LLC 9,380,298 0.32% 41,439 0.29% 226
LQNA LIQUIDNET, INC. 7,942,400 0.27% 3,634 0.03% 2,186
MSRP MORGAN STANLEY & CO. LLC 2,499,700 0.08% 8,015 0.06% 312
AQUA AQUA SECURITIES L.P. 1,700,850 0.06% 181 0.00% 9,397
LEHM BARCLAYS CAPITAL INC. 1,438,933 0.05% 29 0.00% 49,618
RCSL RIVER CROSS SECURITES, LLLP 1,333,732 0.05% 5,581 0.04% 239
JEFX JEFFERIES EXECUTION SERVICES, INC. 706,586 0.02% 2,888 0.02% 245
FNBR FOLIOFN INVESTMENTS, INC. 425,410 0.01% 11,067 0.08% 38
WDNX WEEDEN & CO.L.P. 110,419 0.00% 167 0.00% 661
PROS PRO SECURITIES, L.L.C. 100 0.00% 1 0.00% 100

Total 2,951,782,201 100.00% 14,209,505 100.00% 208

FINRA ATS stats from June 2014 FWIW.
Original FINRA data

Sunday, 30 November 2014

Ode to Teaching

It's another end to a school year. For months, the world over, an army of teachers have been toiling in the confines of the education factories leading to a final preparatory push and some exhaustion.

There is often much personal and career sacrifice that goes with the decision to be a teacher, and, to continue as a teacher. It is not a profession renown for its pay, chefs on duty, foosball tables in the break out room, nor, sadly, community respect. Stock options are fairly rare. Dedicated professionals are haunted by the cruel and excruciating epitaph echoed by far too many, "Can't do, teach."

A warm thank you from their students sometimes makes all the difference.

A lovely reward for a year's work.
(click to enlarge - broken out below)

As many teachers do, at year end my wife appreciatively received some small gifts and kind words. Carolyn often pecks on her keyboard late into the night. Rises early for gym, the cross town commute, and the 7:30am "extra" tutes. Every day is a pretty long day. The occasional aggressive and threatening parent at a parent-teacher interview challenges the spirit. Each year, every semester, every class, is an opportunity of relentless importance. The wheels never stop. I'm proud of my caring wife and her choice to be a teacher. It been a tough couple of years with twenty years of her teaching superannuation being stolen, a cancer scare, suicide attempts and subsequent hospitalisation in the direct family. Some days aren't easy. Sometimes a note of thanks, like this one, makes that life choice of being a teacher completely worthwhile. 

Hopefully my wife doesn't see this too quickly so that I don't get told to take this note down. She's not one for a fuss or embarrassment. Yes, this card is a fabulous reflection on my wife, but it also shines the same light on the dedication of all her fellow professionals. When we read the unadulterated thoughts of appreciative students, we can all appreciate the importance and fine work that too often is hidden in the classroom. It's a wonderful profession for which we should all give thanks.

I hope this is a nice reminder that in the lives of our children and in the progression and advancement of our society:

Teaching Matters

A respectful token to energize the soul
(click to enlarge - broken out below)

One card for one soul can provide more heat than a fistful of dollars. 

If you know a teacher, take a moment and say a simple thanks. A little kindness can travel far.

Sunday, 23 November 2014

SC14 view from afar

Supercomputing 14 in New Orleans has wrapped up. Despite the stasis in the top 500 echelons some meanderings of interest to traders continue to emerge from HPC.

DE Shaw's Anton 2 - click to enlarge
(The Anton 2 Chip: A 2nd Generation ASIC for Molecular Dynamics Simulation Hot Chips 2014)

The BIG announcement for me for the week was Xilinx's SDAccel. Every latency focused trading firm is going to have to evaluate this framework from a functionality and productivity point of view. There have been quite a few different toolkits for doing this kind of work over the years but this feels a game changer.

I've previously been involved with Impulse-C and found it an interesting and useful product from a good company. However, it was largely not used in my shop as the types and pragmas required in the code essentially removed the C'ness which made it yet another C-like foreign language in practical terms. We found it more productive to just go straight to VHDL code at the end of the day, so the Impulse-C, though impressive, became shelf-ware. Perhaps if we had more of a computational burden it may have been more useful, but much of the work at the time was tricky I/O and thus VHDL mattered more.

The nicest implementation of C-like hardware languages I've reviewed over the years was Handel-C. Lovely CSP / Occam / Pi-calculus like constructs. Quite elegant code. Not C though, but this was not that important for me at the time. The killer was that the platform implementation targets were limited and the performance of the code was somewhat, er, um, challenged. Limited performance is not what you're looking for when biting off hardware implementations. Impulse-C targeted VHDL or Verilog so it had the advantage that the specific platform compiler could optimize. Handel-C tried to do more and targeted the RTL level directly but it didn't quite do the terrific job the native compilers could do. I think they lost an opportunity there.

There have been a bunch of others over the years. For example, Mitrion-C was bundled with Cray's hybrid FPGA supercompute platform for a while (Cray XD1(TM) supercomputer) and SGI's RASC. No stand outs for success. C-like programming for FPGA platforms is hard.

Hopefully, Xilinx's SDAccel can pick up some success. I think the big game changer is both the support and weight of Xilinx behind the platform and the availability of partial reconfiguration on the fly. It remains to be seen whether the intrusions of specific types, code styles and the dreaded pragmas overwhelms the benefits a high level language brings, but it sounds quite the advance to me. I'm optimistic. An impressive aspect from the demos was the occasional improvement possible over raw VHDL from both clock and area perspectives. When combined with SDNet and their QPI toolkit, extremely compelling architectures may be possible with small development teams. Let's hope the C-like approach works for FPGAs this time. All trading firms will need to have a look at this kind of framework if truly low latency is what makes them tick.

Some other tech from SC14

Intel had some interesting announcements, not all at SC14. The further development of the Knights platform to Knights Hill is very compelling, especially when combined with HMCC memory. With the progression from Corner, Landing to Hill, there is plenty of geographical growth possible. I can't wait to see what the Mountain platform or 2030 Galaxy platform looks like ;-)

The release of HMCC 2.0 and the support of various vendors, including Xilinx, is exciting for a few reasons, but the main one for me is the indication of continued growth in 3D, or 2D stacking, manufacturing. Intel was also talking about this for Flash memory, just as Samsung has recently done, and Intel presaged packaging solutions with around 10TB of Flash in SSDs in only a couple of years. NVidia also previously announced stacking of memory with their Pascal GPU platform. NVidia's Volta, announced for Summit and Sierra, will continue that. Continued development of innovative packing will drive densities higher and costs down though at a cost of accessibility as it seems this is a bit like the new PCB but only accessible to the very few.

D E Shaw's Anton 2 chip was an exciting platform to read about. I'm not sure if it won the Bell prize, but if it didn't, I think it probably should have (* update: it did). One notable takeaway from reading their paper is that so much more is still possible with further overlap, out of order speculation and processes beyond 40nm and 2B gates. I can't wait to see Anton 3. It is good to see a firm be prepared to build a custom ASIC. Their last one, the original Anton, was done by an e-beam house in Japan but the markings on this chip are from Korea, so I wonder who they are using to fab there? Whilst it is for molecular dynamics, I'm sure every trader would love to have such awesome power at their finger tips for "Hardware Support for Fine-Grained Event-Driven Computation." Though I'm sure their research motives are altruistic, I'm also sure a clever trading firm would find a use for that ;-)

100Gb connectivity continues to mature with announcements from Mellanox of a sub-90ns 36 port 100Gbps IB switch and with the Invea-tech platform, previously mentioned, being demonstrated at SC14. I must say I find the 25G Ethernet movement, also on demo at SC14 running "under 100ns" platform to platform, a little more practical as a future path for now.

Mellanox, in addition to their 100Gbps IB/Eth card, also joined SolarFlare in having an FPGA NIC solution. It is an alluring solution that looks like it may be better than the SolarFlare for latency as it supports FPGA fabric on either or both of the network and PCIe facing sides. The press doesn't declare the flavour of FPGA nor the tools available. We'll have to wait to understand it better to see if it can compete with a vanilla network connected FPGA NIC for low latency trading.

The Power8 CAPI platform had its first official platform release with Nallatech at the show. A good latency reduction technique to be aware of. I think I'd be focusing on the Xilinx / Intel QPI solution myself though.

Some of my old friends at Metamako were at SC14 next to the Xilinx booth with their compelling platform. If you want the fastest way to get two packets from two wires onto the one wire at 10G, say for facing an exchange, risk gateway or other financial choke point, such as a mandated firewall, then their new MetaMux 32 reigns supreme. It is the fastest way at ~100ns to get two packets onto the same wire at 1G / 10G speeds. This is in additional to their fancy-smancy layer 1 reconfig tricks. I think the biggest use case for me for Metamako is simply using their MetaConnect platform just for timestamping though. You could save a bunch of money using one of their MetaConnects, or perhaps even the MetaMux, to timestamp and collect packet information rather than using expensive timestamping cards but I'm not sure people realise how cool that is from a non-intrusive tap reconfig and cost saving point of view. I'd really like a MetaConnect to script up for dynamic network reconfig and running performance unit tests for financial apps in the test lab.

A mystery for me is the intriguing Intel Omni-Scale platform. Just enough details have been released to know that it is going to be interesting but not enough to really know what it is all about. So, keep your eye out for developments there. Intel have long had the opportunity to kill off much of the NPU, custom processor and even perhaps the entire NIC market by incorporating compelling networking interfaces into the coherent level of their microprocessors. It is not clear when and how this may happen with Omni-Scale. Vendors relying on NIC card revenue should be starting to shift a little uncomfortably in their seats. A big leap in network latency reduction could be at hand but there remains an opportunity for a firm like Cavium or AMD to gazump Intel on the low latency network and compute combination. Direct fabric integration, not the kludgy PCIe SoC kind, is long overdue for mainstream processors. Maybe this is at last on its way.

Perhaps not so relevant to trading apps, the announcement of Summit and Sierra was intriguing with their reliance on nVidia's Volta with stacked RAM and NVLink. It is certainly a nice win for the IBM Power 9 platform even if the headline performance is really reliant on nVidia rather than the Power 9. It also just shows how hard an ExaFLOP is, at least for LinPack, as both of don't quite get to ExaFLOP performance, so the ExaFLOP race remains on.

Happy trading,


{NB: Scant public details from Intel on Knights Hill (optical connect, stacked memory, faster) and Omni-Scale/Path.}

Monday, 17 November 2014

Alpha Data - PCIe 3.0 Xilinx UltraScale 10/40G

I was browsing the exhibitors at SC14 which is on this week in New Orleans. I noticed that I had missed this nice looking Xilinx Kintex UltraScale ADM-PCIE-KU3 from Alpha Data which was news way back in April 2014.
Source: Alpha Data - Data sheet for ADM-PCIE-KU3
(click to enlarge)
Kintex UltraScale is now going mainstream so perhaps just noticing it nowish is cutely serendipitous. I'm not sure about the position of the SMA connector for timing as it looks a little inconvenient in this picture though the picture was an early release. Hopefully it has moved. PCIe 3.0 allowing direct access to Intel uP cache memory (data direct), from Sandy Bridge onwards, would allow lower latency than competing PCIe 2.X solutions. PCIe 2 solutions really should be avoided in this day and age if you're going to the trouble of eeking out the little bits of latency you can.

Simple is good. It is indeed a pretty simple board with up to 8 x 10G lines via break out cables or 2 x 40G, plus additional RAM. Not sure too many in trading would find the SATA ports useful but some may for direct recording to disk but I can't imagine too many bothering.

It looks a good choice and worth checking out at their booth at SC14 this week if you're after a pretty clean Kintex UltraScale design for PCIe 3.0 goodness.

I still think I'd prefer a Zynq solution, such as the Hitech Gobal board HTG-Z7-PCIE-HH or one of the not yet released Arria 10 ARM SoC based boards (e.g. BittWare’s A10PHQ). Still, I'd think a Zynq 7045 would be preferable to the 7100 on the HTG as I'd feel better trading better transceivers for less FPGA fabric, so I'm yet to find the goldilocks board for me. I'm interesting in hearing about boards that you might have found that are just right for your network oriented application. Drop me a mail.

Happy trading,


PS: Nearly two year old FPGA summary from this blog
PPS: Terasic FPGA board previously mentioned


The mystery regarding the PPS placement mentioned above is solved. Here is a picture of the Alpha Data card from SC14 via Xilinx's blog:

Picture of Alpha Data card from Xilinx Forum
(Click to enlarge)

This shows the SMA being exposed to the outside world via the PCIe bracket. It's a bit of an ugly hack but, hey, it works. Overall the card looks a pretty satisfying solution especially when teamed with the particularly awesome SDAccel framework Xilinx announced at SC14.  This particular card was one of the launch devices. SDaccel is a game changer, especially for trading and HFT. All low latency trading firms will need to review SDAccel from both a functionality and productivity point of view.


Saturday, 8 November 2014

Don't buy an ExaNIC from ExaBlaze

[Update: Solarflare is suing Exablaze for patent infringement]

I've seen more dubious PR from the nasty people at Zomojo/ExaBlaze recently.

I thought it worthwhile warning people once more before they get sucked into ExaBlaze's reality distortion field.

Specifically: why their ExaNIC cards are not a smart idea; and, why you should steer well away from such wasteful expenditure.

It would be easy to be seduced by the pitch for the ExaNIC but the pitch is hollow with parts simply straight out lies. It's not your fault if you've been deceived into purchasing one.

There are two potential reasons for looking at an ExaNIC:
   1) A low latency network card, and
   2) An even lower latency FPGA trading solution.

Neither make sense. Let's meander through why.

Is it a good NIC?

In the first case, the ExaNIC card is feature poor and it has little advantage over a much better solution from Mellanox or SolarFlare. Mellanox[2] and SolarFlare[1][3] are quality NICs that have rich feature sets, kernel bypass and trusted implementations with open source drivers. Also, Chelsio and Myricom are worthy of some consideration if you have specific features you need.

You should be concerned about the signal integrity of the cards from ExaBlaze as the initial versions had poor to little signal integrity on the networking side which is a difficult problem to spot and solve. It's fine when it works but potentially devastating when you're having to solve problems. ExaBlaze push silly claims about latencies quoting 60 byte packets when the minimum frame length on 1G and 10G is 64 bytes. A 60 byte frame is illegal and referred to as a "runt" frame. To top it all off, the ExaNIC cards are expensive. Sure, buy one to play with if you must, but it makes no sense to invest in the card as a NIC solution.

If a quantum of a hundred nanoseconds is really important to you then, regardless of any NIC, you should be considering an FPGA solution to avoid the PCIe latency cost.

So, what about the ExaNIC as an FPGA solution for trading?

Just don't.

As an FPGA solution it is a much simpler and clearer case to weigh up. It is one of the slowest 10G or 1G Xilinx FPGA solutions you could possibly use. Yes, one of the slowest. If you are going to go all the way to an FPGA solution, which is an expensive way to go, then make sure you do it right.

The fundamental limiting factor in the ExaNIC x4 is that the network facing SERDES are off chip and slower than using native on-chip Xilinx transceivers. ExaNIC as an FPGA path is a dumb solution for low-latency. There are plenty of other excellent FPGA boards out there so you have no reason to waste your money. If you have been fooled into purchasing an ExaNIC on the mistaken belief that it was a fast way to connect to the market and trade then you may be entitled to your money back. They are lying to their customers about their high-latency architecture by referring to it as low-latency.

"It hung in the air in exactly the way that bricks don't."

Look at the following picture from the ExaBlaze web site:

(click to enlarge)
The standard way an FPGA NIC card works, except an ExaNIC doesn't. The SERDES for ExaNIC x4 are not on the FPGA, they are external, an extra hop, and higher latency. Don't be deceived.
The diagram clearly shows the network SERDES to be on-chip with respect to the FPGA. This is a lie for the ExaNIC x4 and it may also be a lie for the ExaNIC x2 if it is a similar design. The SERDES on the ExaNIC x4 is an external mux-demux chip and a slower solution than an on-chip FPGA transceiver. Why would a vendor use a higher latency architecture? To use a cheaper FPGA and save money. Don't be fooled. If you have fallen for the ExaBlaze misrepresentation, go get your money back is all I can suggest.

In summary, as NICs go, there is better quality, functionality and more reasonable prices from Mellanox and SolarFlare NICs, so avoid ExaNIC. As an FPGA solution, it is just a dumb idea to invest in an architecturally slow solution. You can't make it fast. It is slow by design. Get a real FPGA board from a trustworthy vendor as there are plenty of good alternatives. The combined ARM & FPGA SoC solutions as NICs are my favourites at the moment even though the ARMs are a little underwhelming. The SoC FPGA fabric to uP hop beats PCIe latencies for interesting hybrid solutions.

You should try to buy equipment from a trustworthy vendor. Matt Chapman annoyed a lot of people with his little video at Inside HPC where he dances around a product that Zeptonics developed and he claimed as his own. What a schmuck. Greg Robinson used stolen IP from ITG when he was sacked as CEO of ITG Australia to compete on the trading front at Zomojo with ITG's prop trading (Canada / NY interlisted arb specifically) and told multiple staff (so it's not just my claim) not to talk to Bill Burdett about it as Bill was still on the Board of directors of NYSE listed ITG Inc. Bill is a Zomojo beneficial shareholder. Then you have my little continuing dispute with them where I hope for a miracle one day that will see my faith in justice restored with some kind of judicial remedy that passes the smell test. Exablaze / Zomojo lied to either the court or their customers claiming Chapman was no longer associate with the trading at Zomojo to customers and yet at the same time claiming to the court he was an officer of Zomojo. Those statements are mutually exclusive, so both couldn't be true. Zomojo/Exablaze lied to the court many times. Zomojo and Exablaze are nasty people. Do you want to buy from a vendor that not only trades against you but misleads you? Caveat emptor.

If you've bought an ExaNIC, you should use their misleading representations as a reason to return the NICs and get your money back. Fortunately for you, there are plenty of good people in the industry making better product. It's simple. Exablaze serves no useful purpose. Use better products.

Happy trading,


[Update Sunday 2014-11-23]

PS: My speculation the ExaNIC x2 works the same way as the ExaNIC x4 with an external _SLOW_ mux-demux SERDES requiring an extra hop is confirmed. Xilinx point out the x2 board uses a Kintex 7-160T FPGA which has a maximum of 8 high speed transceivers. As the card has 8 x PCIe 2 (yeah, no PCIe 3.0), it has no transceivers left for facing the network.  So, it is confirmed, the x2 has a similar architecture to the ExaNIC x4 and ExaBlaze are lying to their customers about the design on their public website. The architecture is slow by design. Use a better product.

[1] Some Solarflare 10GbE Server Adapter Features

• LSO, LRO, GSO large packet offloads • TCP/UDP/IP checksums offloads • Line rate packet
filtering • Receive Side Scaling (RSS) • Accelerated Receive Flow Steering (RFS) • NetQueue,
VMQ, SR-IOV • 256 Multicast filters • Jumbo Frames (9KB) • 4096 VLANs/port • PXE boot,
iSCSI boot • IEEE 802.3ae – 10 Gigabit Ethernet • IEEE 802.3an – 10GBASE-T • IEEE
802.3ad – Link Aggregation and Failover • IEEE 802.1Q, 802.1p – VLAN tags, priority • IEEE
802.3x – Pause

Operating Systems: RHEL 5, 6, 7; MRG; SLES 10, 11; SLERT; other Linux; Windows Server 2003, 2003 R2, 2008, 2008 R2, 2012, 2012 R2; OS X v10.6, 10.7, 10.8, 10.9; Solaris x86 10, 11; ESX 3.5, 4.x, ESXi 5.x; KVM; Windows Hyper-V; XenServer 5.x, 6.0.All server adapters support: SR-IOV, 127 VFs per port,1024 vNICs per port.

[2] Some Mellanox NIC features

– IEEE Std 802.3ae 10 Gigabit Ethernet
– IEEE Std 802.3ba 40 Gigabit Ethernet
– IEEE Std 802.3ad Link Aggregation
– IEEE Std 802.3az Energy Efficient Ethernet
– IEEE Std 802.1Q, .1P VLAN tags and priority
– IEEE Std 802.1Qau Congestion Notification
– IEEE Std 802.1Qbg
– IEEE P802.1Qaz D0.2 ETS
– IEEE P802.1Qbb D1.0 Priority-based Flow
– IEEE 1588v2
– Jumbo frame support (9600B)
– VXLAN and NVGRE - A Framework for
Overlaying Virtualized Layer 2 Networks over
Layer 3 Networks. Network Virtualization
hardware offload engines 
– Single Root IOV
– Address translation and protection
– Dedicated adapter resources
– Multiple queues per virtual machine
– Enhanced QoS for vNICs
– VMware NetQueue support
– RDMA over Converged Ethernet
– TCP/UDP/IP stateless offload
– Intelligent interrupt coalescence
– Remote boot over Ethernet
– Remote boot over iSCSI
– MPI, Platform MPI
– PCIe Base 3.0 compliant, 1.1 and 2.0 compatible
– 2.5, 5.0, or 8.0GT/s link rate x8
– Auto-negotiates to x8, x4, x2, or x1
– Support for MSI/MSI-X mechanisms
– Interoperable with 10/40GbE Ethernet switches.
Interoperable with 56GbE Mellanox Switches.
– Passive copper cable with ESD protection
– Powered connectors for optical and active cable
– QSFP to SFP+ connectivity through QSA module
– Citrix XenServer 6.1
– RHEL/CentOS 5.X and 6.X, Novell SLES10 SP4;
SLES11 SP1 , SLES 11 SP2, OEL, Fedora 14,15,17,
Ubuntu 12.04
– Windows Server 2008/2012/2012 R2
– FreeBSD
– OpenFabrics Enterprise Distribution (OFED)
– OpenFabrics Windows Distribution (WinOF)
– VMware ESXi 4.x and 5.x

[3] Some specific SolarFlare features

Product Number
Standards & Compliance
IEEE 1588 v2
IEEE 802.3ae
IEEE 802.3ad
IEEE 802.1Q
IEEE 802.1p
IEEE 802.3x
RoHS Compliant
5.9W (typical)

Precision Packet Time Stamping
7.5ns resolution
Stable Precision Oscillator
Stratum 3 compliant; short term drift < 3.7*10-7 in 24 hours
Server Clock Synchronization Accuracy
Sub 200ns
1PPS-input circuit
Rising edge active, TTL into 50Ω
1PPS-output circuit
Rising edge on-time, TTL into 50Ω
I/O Virtualization
2048 guest OS protected vNICs; 240 virtual functions;
16 physical functions
PCI Express
PCIe 3.0 x8 @ 8.0 GT/s
SFC9120 10G Ethernet Controller
Supports high-performance 10GbE
SFP+ Support
Supports optical & copper SFP/SFP+ modules; Direct-Attach,
Fiber (10G or 1G), 1G/10G combo
1000BASE-T SFP Support
Supports 1G 1000BASE-T SFP modules
Low Latency
Cut-through architecture/intelligent interrupt coalescing
Receive Side Scaling (RSS)
Distributes IPv4, IPv6 loads across all CPU cores;
MSI-X minimizes interrupt overhead
Hardware Offloads
TSO, LRO, GSO; IPv4/IPv6; TCP, UDP checksums
Adapter Teaming/Link Aggregation
LACP for redundant links & increased bandwidth
(compatible with MLAG)
Jumbo Frames
9216 byte MTU for performance
Enhanced Tuning
Adaptive interrupt moderation
IP Flow Filtering
Hardware directs packets based on IP, TCP, UDP headers
Advanced Packet Filtering
4096 multicast filters; 4096 VLANs/port; adaptive TCP/UDP/IP,
MAC, VLAN, RSS, RFS filtering; Accelerated Receive Flow
Steering (RFS)
Intel QuickDataTM
Uses host DMA engines to accelerate I/O
Remote Boot
PXE, iSCSI boot; unattended installation
Virtualization Support
VMware ESXi; Microsoft Hyper-V; XenServer; Linux KVM;
Operating Systems
RHEL 5, 6, 7, MRG; SLES 10, 11, SLERT; other Linux;
Windows Server 2008 R2, 2012, 2012 R2