Saturday, 8 November 2014

Don't buy an ExaNIC from ExaBlaze

[Update: Solarflare is suing Exablaze for patent infringement]

I've seen more dubious PR from the nasty people at Zomojo/ExaBlaze recently.

I thought it worthwhile warning people once more before they get sucked into ExaBlaze's reality distortion field.

Specifically: why their ExaNIC cards are not a smart idea; and, why you should steer well away from such wasteful expenditure.

It would be easy to be seduced by the pitch for the ExaNIC but the pitch is hollow with parts simply straight out lies. It's not your fault if you've been deceived into purchasing one.

There are two potential reasons for looking at an ExaNIC:
   1) A low latency network card, and
   2) An even lower latency FPGA trading solution.

Neither make sense. Let's meander through why.

Is it a good NIC?

In the first case, the ExaNIC card is feature poor and it has little advantage over a much better solution from Mellanox or SolarFlare. Mellanox[2] and SolarFlare[1][3] are quality NICs that have rich feature sets, kernel bypass and trusted implementations with open source drivers. Also, Chelsio and Myricom are worthy of some consideration if you have specific features you need.

You should be concerned about the signal integrity of the cards from ExaBlaze as the initial versions had poor to little signal integrity on the networking side which is a difficult problem to spot and solve. It's fine when it works but potentially devastating when you're having to solve problems. ExaBlaze push silly claims about latencies quoting 60 byte packets when the minimum frame length on 1G and 10G is 64 bytes. A 60 byte frame is illegal and referred to as a "runt" frame. To top it all off, the ExaNIC cards are expensive. Sure, buy one to play with if you must, but it makes no sense to invest in the card as a NIC solution.

If a quantum of a hundred nanoseconds is really important to you then, regardless of any NIC, you should be considering an FPGA solution to avoid the PCIe latency cost.

So, what about the ExaNIC as an FPGA solution for trading?

Just don't.

As an FPGA solution it is a much simpler and clearer case to weigh up. It is one of the slowest 10G or 1G Xilinx FPGA solutions you could possibly use. Yes, one of the slowest. If you are going to go all the way to an FPGA solution, which is an expensive way to go, then make sure you do it right.

The fundamental limiting factor in the ExaNIC x4 is that the network facing SERDES are off chip and slower than using native on-chip Xilinx transceivers. ExaNIC as an FPGA path is a dumb solution for low-latency. There are plenty of other excellent FPGA boards out there so you have no reason to waste your money. If you have been fooled into purchasing an ExaNIC on the mistaken belief that it was a fast way to connect to the market and trade then you may be entitled to your money back. They are lying to their customers about their high-latency architecture by referring to it as low-latency.

"It hung in the air in exactly the way that bricks don't."

Look at the following picture from the ExaBlaze web site:

(click to enlarge)
The standard way an FPGA NIC card works, except an ExaNIC doesn't. The SERDES for ExaNIC x4 are not on the FPGA, they are external, an extra hop, and higher latency. Don't be deceived.
The diagram clearly shows the network SERDES to be on-chip with respect to the FPGA. This is a lie for the ExaNIC x4 and it may also be a lie for the ExaNIC x2 if it is a similar design. The SERDES on the ExaNIC x4 is an external mux-demux chip and a slower solution than an on-chip FPGA transceiver. Why would a vendor use a higher latency architecture? To use a cheaper FPGA and save money. Don't be fooled. If you have fallen for the ExaBlaze misrepresentation, go get your money back is all I can suggest.

In summary, as NICs go, there is better quality, functionality and more reasonable prices from Mellanox and SolarFlare NICs, so avoid ExaNIC. As an FPGA solution, it is just a dumb idea to invest in an architecturally slow solution. You can't make it fast. It is slow by design. Get a real FPGA board from a trustworthy vendor as there are plenty of good alternatives. The combined ARM & FPGA SoC solutions as NICs are my favourites at the moment even though the ARMs are a little underwhelming. The SoC FPGA fabric to uP hop beats PCIe latencies for interesting hybrid solutions.

You should try to buy equipment from a trustworthy vendor. Matt Chapman annoyed a lot of people with his little video at Inside HPC where he dances around a product that Zeptonics developed and he claimed as his own. What a schmuck. Greg Robinson used stolen IP from ITG when he was sacked as CEO of ITG Australia to compete on the trading front at Zomojo with ITG's prop trading (Canada / NY interlisted arb specifically) and told multiple staff (so it's not just my claim) not to talk to Bill Burdett about it as Bill was still on the Board of directors of NYSE listed ITG Inc. Bill is a Zomojo beneficial shareholder. Then you have my little continuing dispute with them where I hope for a miracle one day that will see my faith in justice restored with some kind of judicial remedy that passes the smell test. Exablaze / Zomojo lied to either the court or their customers claiming Chapman was no longer associate with the trading at Zomojo to customers and yet at the same time claiming to the court he was an officer of Zomojo. Those statements are mutually exclusive, so both couldn't be true. Zomojo/Exablaze lied to the court many times. Zomojo and Exablaze are nasty people. Do you want to buy from a vendor that not only trades against you but misleads you? Caveat emptor.

If you've bought an ExaNIC, you should use their misleading representations as a reason to return the NICs and get your money back. Fortunately for you, there are plenty of good people in the industry making better product. It's simple. Exablaze serves no useful purpose. Use better products.

Happy trading,

--Matt.


[Update Sunday 2014-11-23]

PS: My speculation the ExaNIC x2 works the same way as the ExaNIC x4 with an external _SLOW_ mux-demux SERDES requiring an extra hop is confirmed. Xilinx point out the x2 board uses a Kintex 7-160T FPGA which has a maximum of 8 high speed transceivers. As the card has 8 x PCIe 2 (yeah, no PCIe 3.0), it has no transceivers left for facing the network.  So, it is confirmed, the x2 has a similar architecture to the ExaNIC x4 and ExaBlaze are lying to their customers about the design on their public website. The architecture is slow by design. Use a better product.
______

[1] Some Solarflare 10GbE Server Adapter Features

• LSO, LRO, GSO large packet offloads • TCP/UDP/IP checksums offloads • Line rate packet
filtering • Receive Side Scaling (RSS) • Accelerated Receive Flow Steering (RFS) • NetQueue,
VMQ, SR-IOV • 256 Multicast filters • Jumbo Frames (9KB) • 4096 VLANs/port • PXE boot,
iSCSI boot • IEEE 802.3ae – 10 Gigabit Ethernet • IEEE 802.3an – 10GBASE-T • IEEE
802.3ad – Link Aggregation and Failover • IEEE 802.1Q, 802.1p – VLAN tags, priority • IEEE
802.3x – Pause

Operating Systems: RHEL 5, 6, 7; MRG; SLES 10, 11; SLERT; other Linux; Windows Server 2003, 2003 R2, 2008, 2008 R2, 2012, 2012 R2; OS X v10.6, 10.7, 10.8, 10.9; Solaris x86 10, 11; ESX 3.5, 4.x, ESXi 5.x; KVM; Windows Hyper-V; XenServer 5.x, 6.0.All server adapters support: SR-IOV, 127 VFs per port,1024 vNICs per port.

[2] Some Mellanox NIC features

ETHERNET
– IEEE Std 802.3ae 10 Gigabit Ethernet
– IEEE Std 802.3ba 40 Gigabit Ethernet
– IEEE Std 802.3ad Link Aggregation
– IEEE Std 802.3az Energy Efficient Ethernet
– IEEE Std 802.1Q, .1P VLAN tags and priority
– IEEE Std 802.1Qau Congestion Notification
– IEEE Std 802.1Qbg
– IEEE P802.1Qaz D0.2 ETS
– IEEE P802.1Qbb D1.0 Priority-based Flow
Control
– IEEE 1588v2
– Jumbo frame support (9600B)
OVERLAY NETWORKS
– VXLAN and NVGRE - A Framework for
Overlaying Virtualized Layer 2 Networks over
Layer 3 Networks. Network Virtualization
hardware offload engines 
HARDWARE-BASED I/O VIRTUALIZATION
– Single Root IOV
– Address translation and protection
– Dedicated adapter resources
– Multiple queues per virtual machine
– Enhanced QoS for vNICs
– VMware NetQueue support
ADDITIONAL CPU OFFLOADS
– RDMA over Converged Ethernet
– TCP/UDP/IP stateless offload
– Intelligent interrupt coalescence
FLEXBOOT™ TECHNOLOGY
– Remote boot over Ethernet
– Remote boot over iSCSI
PROTOCOL SUPPORT
– Open MPI, OSU MVAPICH, Intel MPI, MS
– MPI, Platform MPI
– TCP/UDP
– iSER, NFS RDMA
– uDAPL
PCI EXPRESS INTERFACE
– PCIe Base 3.0 compliant, 1.1 and 2.0 compatible
– 2.5, 5.0, or 8.0GT/s link rate x8
– Auto-negotiates to x8, x4, x2, or x1
– Support for MSI/MSI-X mechanisms
CONNECTIVITY
– Interoperable with 10/40GbE Ethernet switches.
Interoperable with 56GbE Mellanox Switches.
– Passive copper cable with ESD protection
– Powered connectors for optical and active cable
support
– QSFP to SFP+ connectivity through QSA module
OPERATING SYSTEMS/DISTRIBUTIONS
– Citrix XenServer 6.1
– RHEL/CentOS 5.X and 6.X, Novell SLES10 SP4;
SLES11 SP1 , SLES 11 SP2, OEL, Fedora 14,15,17,
Ubuntu 12.04
– Windows Server 2008/2012/2012 R2
– FreeBSD
– OpenFabrics Enterprise Distribution (OFED)
– OpenFabrics Windows Distribution (WinOF)
– VMware ESXi 4.x and 5.x

[3] Some specific SolarFlare features

Product Number
SFN7322F
Standards & Compliance
IEEE 1588 v2
IEEE 802.3ae
IEEE 802.3ad
IEEE 802.1Q
IEEE 802.1p
IEEE 802.3x
RoHS Compliant
Power
5.9W (typical)

Precision Packet Time Stamping
7.5ns resolution
Stable Precision Oscillator
Stratum 3 compliant; short term drift < 3.7*10-7 in 24 hours
Server Clock Synchronization Accuracy
Sub 200ns
1PPS-input circuit
Rising edge active, TTL into 50Ω
1PPS-output circuit
Rising edge on-time, TTL into 50Ω
I/O Virtualization
2048 guest OS protected vNICs; 240 virtual functions;
16 physical functions
PCI Express
PCIe 3.0 x8 @ 8.0 GT/s
SFC9120 10G Ethernet Controller
Supports high-performance 10GbE
SFP+ Support
Supports optical & copper SFP/SFP+ modules; Direct-Attach,
Fiber (10G or 1G), 1G/10G combo
1000BASE-T SFP Support
Supports 1G 1000BASE-T SFP modules
Low Latency
Cut-through architecture/intelligent interrupt coalescing
Receive Side Scaling (RSS)
Distributes IPv4, IPv6 loads across all CPU cores;
MSI-X minimizes interrupt overhead
Hardware Offloads
TSO, LRO, GSO; IPv4/IPv6; TCP, UDP checksums
Adapter Teaming/Link Aggregation
LACP for redundant links & increased bandwidth
(compatible with MLAG)
Jumbo Frames
9216 byte MTU for performance
Enhanced Tuning
Adaptive interrupt moderation
IP Flow Filtering
Hardware directs packets based on IP, TCP, UDP headers
Advanced Packet Filtering
4096 multicast filters; 4096 VLANs/port; adaptive TCP/UDP/IP,
MAC, VLAN, RSS, RFS filtering; Accelerated Receive Flow
Steering (RFS)
Intel QuickDataTM
Uses host DMA engines to accelerate I/O
Remote Boot
PXE, iSCSI boot; unattended installation
Management
SNMP, ACPI v3.0
Virtualization Support
VMware ESXi; Microsoft Hyper-V; XenServer; Linux KVM;
SR-IOV
Operating Systems
RHEL 5, 6, 7, MRG; SLES 10, 11, SLERT; other Linux;
Windows Server 2008 R2, 2012, 2012 R2



5 comments:

  1. thanks for the heads up... we'll get the word out...

    -Joe
    Affirmed Systems

    ReplyDelete
  2. Thanks Joe.

    Fortunately there are a couple of dozen FPGA NIC vendors and a handful of L1 crosspoint / L2 mux-switch vendors that do better product. So, there is no need to bother worrying about ExaBlaze unless you're in the unfortunate position where you've trusted them and been deceived such that you've bought into their improper marketing.

    --Matt.

    ReplyDelete
  3. I think ExaBlaze are heading for oblivion regardless. Mellanox have killed them off completely with their FPGA NIC and 90ns switch.

    ReplyDelete
  4. I'm not sure I know enough but the X10 card to saying anything too sensible but there are some clear inferences to make and questions to ask.

    Remember the primary reason to avoid Exablaze still exists. Greg Robinson and Matt Chapman have proved to be liars to the courts and their customers. They are both, in effect, unconvicted criminals. They have been acting as patent trolls threatening firms around the world with dodgy patents. Exablaze continued to trade they hid this from their customers & engaged in putting Matt Chapman forward as both an officer of Zomojo's trading company and CTO of Exablaze. So much for NDAs. You can't trust them which is sad to say about an Australian tech firm but thankfully they are the nasty exception in an otherwise good tech community.

    Current PR about being the first card under a microsecond is not a new claim. Exablaze claimed the same previously back in 2013 for their X2 release. I'm not sure if you can be first twice. Are they right now, or were they right then, or neither?

    The X10 was announced a while ago but it seemed to have problems. You'll see in the current PCB image Exablaze released, the PCB is marked as V2 which implies the first one had issues.

    For the X10, it looks like they have ditched the external SERDES which makes their card similar to all the other vendors. This also implicitly acknowledges their misleading past when they lied about their architecture. It is hard to tell anything much without closer examination. Their Link product has decent signal integrity because they didn't design that part of it. Their original X2 and X4 had signal integrity issues, especially over copper, and it is unclear if those problems continue with the X10. I know this about the X2 and X4 as I was there at the design time but I can't tell from the public information what they have done with X10. Given their damning history, it may have been wise to burn some picoseconds and include a CDR to nip that criticism in the bud, but I don't know if they've included one on the FPGA to SFP+ path on the ExaNIC X10. Ultrascale probably doesn't need that for many use cases, but Exablaze don't design well so they should have included better signal integrity to ease their customers concerns.

    There are lots of great vendors out there. People should either use a Mellanox or Solar Flare if they want a good NIC. If you care enough about 100ns of performance there are plenty of good FPGA vendors to buy PCIe cards from including Invea-tech, Nallatech, Alpha-Data, DiniGroup, Bittware, Enyx, 4DSP, Hitech Global, PLDA, SR Labs, ... and many more.

    Some boards are better than others. Look at this HTG board. I've had very good and quite bad experience with HTG boards, largely good (they do produce one of the standard Xilinx kits for example), but you'd have to check that specific card out in detail to really know. The SMA connector on the board looks awkward and I don't like that at all but the Z-Ray stuff is very cool. The Alpha-Data Kintex UltraScale board is a good one and I'm sure there are others.

    My current preferred solution for FPGA trading would be a Metamako MetaApp thanks to its layer 1 matrix, timing, x86_64 on board, and external PCIe support. The time everything capability plus the ability to bridge with just a single digit nanosecond latency is hard to beat.

    An ideal solution for PCIe card trading would be a Zynq Ultrascale based solution but that, and the kind of additional component support, would really depend on the kind of network environment, trading, and algo you need to implement.

    The question is pretty simple, why would you bother with an untrustworthy patent troll like Exablaze that has lied to customers and mislead about their technology in the past?

    Get a better vendor.

    ReplyDelete