Thursday, 17 January 2013

Which is the world’s fastest switch for 10G Ethernet?

[June 2016 update, again updated in March 2017: Metamako MetaMux48 is ~69ns, the new leader in switching latency. April 2015 update: Metamako's MetaMux32 is the new king at 99ns, Gnodal & Zeptonics passed away, RIP]

As always, it depends on your application but here are the headline numbers and an order of latency merit:
  1. Zeptonics ZeptoMux          130 ns
  2. Gnodal GS7200               150 ns
  3. Cisco Nexus 3548            190 ns
  4. Mellanox SX1016             250 ns
  5. Arista 7150                 350 ns
  6. Gnodal GS7200               500 ns
Yes, Gnodal is listed twice for good reason.

It's important to note there is a further bunch of interesting switches, that may have exactly the features you need, from Juniper, IBM/Blade, Fujitsu and elsewhere that are in that 500-900ns range but that is the “old” fastest and so last year :-).  The products in the list above are the new kids on the block and are the current 2013 latency champions.

Now one interesting thing, to me, is that I believe you can make a case for all of the above switches for specific circumstances.  Let me meander a little on each.

1.         ZeptoMux is the fastest way on the planet to get two 10G Ethernet packets from two (or twenty three) different wires onto one wire.  Not NxN connectivity, just a 23 ports to 1 port, and back again, funnel.  The client ports can’t talk to each other.  It is the fastest way in the world to talk to a financial market or risk service or any other concentrated service but it is not a generic fully connected switch.  It can be integrated into Layer 2 and Layer 3 solutions (such as CME). Really low jitter with 130ns +/- 6ns for all packet sizes.  A few less nanoseconds on the way back from the service. I’m biased and vested so you have to take my meanderings here with a grain of salt here as my company makes this piece of glorious magic ;-)
2.         Gnodal is a regular NxN switch like everything  else on the list except the ZeptoMux.  This switch is special as it has a crazily low latency interconnect between Gnodal switches and hence it has great specs for large fabrics.  The down side is it is not really 150ns as this is a minimal packet length thing as the first hop onto a Gnodal fabric is store and forward and hence dependent on packet length.  This means that for a bigger packet, such as for some trading environments, you may actually be 200ns or 500ns port to port. Depending on your packet size Gnodal may actually be the slowest on this list.  That’s why I gave it two entries.  Just goes to highlight the best solution for you is dependent on your specific circumstances.
3.         The Cisco 3548 in Warp mode (reduced CAM == smaller network) runs at 190ns or perhaps even just a bit under.  There is an impressive Cisco chip “Monticello” in the box thanks to the work of 500 or 600 clever engineers over nearly two years that makes all of this possible I’m told.  Based on the published specs, I think this is the real gem of the lot as a fully functional switch.  Cost is a bit of an issue as I see this model, CISCO Model #: N3K-C3548P-FA-L3A, has a list price of $66k and then you have to pay for the SFP+ modules.  If money is no object and you have a small network, packets are not just minimum length, and you require NxN connectivity, Cisco is the fastest.
4.         Mellanox have a neat product with their SwitchX family that will get better over time with improved firmware and functionality and I'd argue it is a better value proposition than the Cisco 3548.  I’d buy this one if I had a latency sensitive but not a latency critical application and its feature set, which is more limited than the Cisco, suited my application. The next generation of Mellanox SwitchX-2 products may provide a bit of a latency improvement and is something to watch out for.  For value and performance I like the balance of this product best for a general NxN switch.
5.         The Arista 7150 series is a great product.  However they have fallen behind in the speed stakes.  Arista relies on Broadcom Trident family silicon and Intel/Fulcrum silicon for their product set with the 7150 using a newer Fulcrum chipset.  I'm a big fan of the very cleverly designed Fulcrum chips.  However, Arista's reliance on third party silicon is restrictive and it should be no surprise that all the four vendors in front of Arista on this list control more of their hardware IP than Arista does. Where Arista really shines is their EOS and rich feature set which is something they do indeed control.  EOS is a rich, pleasant environment and Andy Bechtolsheim is a legend so why not use an Arista 7150 if the nanoseconds don’t need to be counted so closely?  If the switches were all the same speed I think I’d choose this one for the software set it offers, but they’re not the same speed.

So, if you’re a trader and you need a switch to focus multiple lines onto one order or risk line, you need to use a ZeptoMux or you may come second and miss the trade.  Be aware that most applications in the data centre are not so “focus” oriented on a specific service point and thus don’t really work this way and you need one of the other more traditional NxN switches instead.  Horses for courses.

If you’re not dollar conscious, it is hard to beat the specs on the impressive looking Cisco Nexus 3548.  If you have a large HPC fabric then perhaps Gnodal may suit better with its super fast inter-switch latencies if your packets are appropriately sized.  Mellanox is a good value buy if you want fast but not fastest and keep in mind it may be faster for your app than Gnodal due to its cut-through.  The Arista is for latency sensitive but not latency critical apps and helped by being a nicely featured product and perhaps a good option for a cloud due to its pleasant OS.  Now you can perhaps see why I think there is actually a place and time for all of these products. 

So consistent with the Lake Wobegon effect, all switches are above average and everyone is a winner!

Stating the obvious, you can't consider these things in a vacuum and your application and environment is the context that makes a choice make sense.  It's not always about speed, sometimes features matter.  My customers want to be first past the post and care about their competitive advantage, so I care about speed, consistent speed, and nothing but the speed.

So, batten down the hatches and load the clichés.  Pack me up a truck of me very own ZeptoMux so I can go a huntin'.  Yeeha!

No comments:

Post a Comment