Chelsio’s Terminator 5 ASIC offers a high performance, robust third‐generation implementation of RDMA (Remote Direct Memory Access) over 40Gb Ethernet – iWARP. The Terminator series adapters have been field proven in numerous large clusters, including a 1300‐node cluster at Purdue University. Chelsio supports several Message Passing Interface (MPI) implementations through integration within the OpenFabrics Enterprise Distribution (OFED), which has in-boxed Terminator drivers since release 1.2. Continuing on the performance curve established by T3 and T4, the T5 design cuts T4’s end-to-end RDMA latency in half to about 1.5µs. Benchmark reports available below show that T5 offers performance that meets or exceeds the fastest FDR InfiniBand speeds in real‐world applications.
GPUDirect RDMAThe GPUDirect Technology eliminates CPU bandwidth and latency bottlenecks using RDMA transfers between GPUs and other PCIe devices. GPUDirect is well suited to using and benefiting from Chelsio’s T5 iWARP RDMA over Ethernet because of their characteristics and performance requirements. Network DirectThe Chelsio Network Direct (ND) driver is Windows Hardware Quality Lab (WHQL)-certified by Microsoft as a solution for High Performance Computing (HPC) clusters running Windows 2008 R2. SMB DirectA key feature of Windows Server 2012, SMB Direct allows the SMB storage protocol to natively leverage RDMA enabled NICs. With fully offloaded RDMA support, Terminator 5 based adapters deliver large performance and efficiency gains to Windows users in a seamless, plug and play fashion. iWARP vs. RoCE/IBChelsio has published a number of reports and papers discussing iWARP’s performance and characteristics, comparing and contrasting it to the InfiniBand over Ethernet specification, also known as RDMA over Converged Ethernet (RoCE). RoCE FAQ: The pitfalls in RoCE answered with respect to iWARP.RoCE raises many questions when practical deployment issues and limitations are encountered, and the answers to these questions are universally cause of concern to potential users. RoCE: The Fine PrintThe promise of RoCE is to bring RDMA’s benefits to Ethernet. In the interest of truth in advertising, here is the missing fine print. RoCE: The Grand ExperimentUpon closer examination, the CE component of the name is revealed to be a misnomer at best, since in a dedicated fabric, the CE suite of protocols (also called DCB) effectively boils down to Ethernet’s PAUSE. A Rocky Road for RoCEBy throwing overboard critical pieces of the IB and TCP stacks which provide stability and scalability, RoCE shines at simple micro-benchmarks in back-to-back or similarly limited deployment scenarios. However, it stands to fail in large clustered application performance, where all its limitations would be exposed. Effectively, the protocol represents an attempt by InfiniBand vendors at enticing the customer with a good Ethernet clustering benchmark story, but switching to selling InfiniBand gear in the end. RoCE at a CrossroadsBy depending on a Layer-2 scheme of limited usability, RoCE lacks the essential requirements for scalability and reliability. In contrast iWARP is the no-risk high performance solution for 40Gb Ethernet clustering, leveraging TCP/IP’s mature and proven design, with the required congestion control, scalability and routability preserving existing hardware and requiring no new protocols, interoperability, or long maturity period. RoCE is Dead, Long Live RoIP?The move from raw Ethernet to IP based encapsulation leaves existing RoCE users with a number of questions regarding their investment and the future of the technology. GPUDirect over 40GbE iWARP RDMA vs. RoCEThis paper provides early benchmark results that illustrate the benefits of GPUDirect RDMA using Chelsio T5 Unified Wire Adapters. iWARP RDMA is shown to provide 30% higher throughput in comparison to InfiniBand over Ethernet (RoCE). Network Direct Chelsio 40GbE vs Mellanox 56G IBThis paper presents raw performance data comparing the Chelsio T580-LP-CR 40GbE iWARP adapter and Mellanox ConnectX-3 FDR InfiniBand adapter. Traditionally, InfiniBand had enjoyed a performance advantage in raw bandwidth and latency micro-benchmarks. However, with the latest 40GbE and 100GbE standards, Ethernet now shares the basic physical layer with IB and has essentially caught up on these basic metrics. NFS with iWARP at 40GbE vs. IB-FDRThis paper presents NFS over RDMA performance results comparing iWARP RDMA over 40Gb Ethernet and FDR InfiniBand (IB). The results show that Ethernet at 40Gb provides competitive performance to the latest IB speeds, while preserving existing equipment and without requiring a fabric overhaul and additional acquisition and management costs. 40Gb Ethernet: A Competitive Alternative to InfiniBandWith the availability of 40Gb Ethernet, the performance gap between Ethernet and InfiniBand options has been virtually closed. This paper provides three real application benchmarks running on IBM’s Rackswitch G8316, a 40Gb Ethernet aggregation switch, in conjunction with Chelsio Communications’ 40Gb Ethernet Unified Wire network adapter. This shows how iWARP offers comparable application level performance at 40Gbps with the latest InfiniBand FDR speeds. SMBDirect 40 GbE iWARP vs 56G InfinibandThis paper compares the Chelsio T5 Unified Wire Network Adapters with the Mellanox ConnectX-3 Adapters for performance over the range of I/O sizes on Windows Server 2012 R2. The results show that Ethernet can now do RDMA performance without the need for specialized equipment. Windows Server 2012 R2 SMB PerformanceThis paper provided performance results for SMB 3.0 running over Chelsio’s T5 RDMA enabled Ethernet adapter and compared it to Intel’s X520-DA2 non-RDMA server adapter. The results demonstrate the benefits of RDMA in improved performance and efficiency. SMBDirect over Ethernet using iWARP on Windows Server 2012 R2Using Chelsio T5 Unified Wire Network Adapters with industry leading RDMA over Ethernet (iWARP), Chelsio enables Microsoft Windows Server 2012 R2 to deliver superior performance over the range of I/O sizes rivaling that of competing technologies. Lowest UDP, TCP, and RDMA Over Ethernet LatencyIn the HPC Linux for Wall Street conference,NYC today, Chelsio demonstrated a user-mode UDP and TCP latency of 1.6 µs and 2.0 µs respectively. Using its WireDirect software suite, both numbers represented industry record performance. The software provides direct network access to user space and is binary compatible with existing TCP and UDP sockets applications. User-mode UDP showed 3 million messages/second with excellent jitter profile and no dropped packets. Similarly, user-mode TCP demonstrated 2.3 million messages/second with nearly zero packet jitter, thanks to the use of T5’s offload engine. These preliminary results are expected to improve before general availability. 40G SMB Direct RDMA Over Ethernet For Windows Server 2012Chelsio Communications, a leading provider of High Speed Ethernet Unified Wire adapters and ASICs, today announced that it will demonstrate 40Gb SMB performance on its new T5 ASIC this week at SNW Spring conference in Orlando, FL. The demonstration will show Microsoft’s SMB Direct running at line-rate 40Gb using iWARP. This will be the first demonstration of Chelsio’s T5 40G storage technology – a converged interconnect solution that simultaneously supports all of the networking, cluster and storage protocols. Chelsio offers a complete suite of drivers for Windows Server 2012, including NDIS, Network Direct for HPC applications, Network Direct Kernel for system services, iSCSI, FCoE and Hyper-V support for virtualized environments. This constitutes as one of the most comprehensive server adapter solutions that can unleash the full value of a Windows Server installation. LAMMPS, LS-DYNA, HPL, and WRF on iWARP vs. InfiniBand FDRThe use of InfiniBand as interconnect technology for HPC applications has been increasing over the past few years, replacing the aging Gigabit Ethernet as the most commonly used fabric. The main reason for preferring IB over 10Gbps Ethernet is it’s native support for RDMA, a technology that forms the basis for high performance MPI implementations. Today, a mature competitive RDMA solution over Ethernet – the iWARP protocol – is available and enables MPI applications to run unmodified over the familiar and preferred Ethernet technology. Offering the same API to applications and inboxed within the same middleware distributions, the technology can be dropped in seamlessly in place of the esoteric fabric. While current solutions are 10Gbps Ethernet-based, higher speed 40Gbps and 100Gbps implementations are slated for imminent availability. Nevertheless, as this paper shows with real application benchmarks, iWARP today offers competitive application level performance at 10Gbps against the latest FDR IB speeds. InfiniBand Migration to iWARPWith the advent of 40GbE, and the arrival of 100GbE, Ethernet today can match or exceed InfiniBand in raw speed. Coupled with mature iWARP implementations, this sets the stage for migrating compute clusters from legacy IB networks to Ethernet, without any performance penalty, while realizing all the economies of scale that an all‐Ethernet environment allows. InfiniBand’s Fifteen MinutesFinally, a key differentiator for IB and the main reason for its recent resurgence is the RDMA communication interface it provides. It allows very efficient communication, where most of the data transfer is handled silently by the adapter, without the involvement of the main CPU. Thus, it frees up the cycles for the host system to process useful application workloads. In the datacenter age, at a time where system efficiency and power savings are critical metrics, increased efficiency is directly translated into dollars – both in terms of CAPEX and OPEX. Although making use of RDMA requires rewriting of applications, the gained efficiencies offer sufficient return on investment in areas such as HPC, storage system back-end and some datacenter and cloud applications. iWARP SMB Direct Technology BriefUsing Chelsio’s Unified Wire Network Adapters with industry leading iWARP, Chelsio enables Microsoft Windows Server 2012 to deliver superior performance in the areas of high bandwidth and low CPU utilization rivaling that of competing technologies. iWARP Benchmarks with Arista SwitchArastra’s switch along with Chelsio’s adapter provides a high throughput, low latency 10 Gigabit Ethernet based solution. The switch delivers an outstanding balance of performance and value with key data center class features. The latency and throughput performance shown in this report demonstrate that 10 Gigabit Ethernet is well suited for operation in demanding clustering applications. Low Latency for High Frequency TradingChelsio’s Unified Wire Network adapter achieves all the requirements to make it ideal for low latency High Frequency Trading (HFT) operations. At the same time, when combined with iWARP, enabling NFSRDMA, LustreRDMA and similar protocols, the adapter makes for an ideal Unified Target adapter, simultaneously processing iSCSI, FCoE, TOE, NFSRDMA, LustreRDMA, CIFS and NFS traffic. Ultra Low Latency Data Center Switches with iWARP NICsFor years, InfiniBand was the dominant interconnect technology for HPC applications, but it has now been eclipsed by Ethernet as the preferred networking protocol where scalability and ultralow latency are required. Juniper Networks’ QFX3500 Switch is a high-performance, ultralow latency, 10GbE switch specifically designed to address a wide range of demanding deployment scenarios such as traditional data centers, virtualized data centers, high-performance computing, network attached storage, converged server I/O and cloud computing. HPC Converging on Low Latency iWARPHPC cluster architectures are moving away from proprietary and expensive networking technologies towards Ethernet as the performance/latency of TCP/IP continues to lead the way. InfiniBand, the once-dominant interconnect technology for HPC applications leveraging MPI and RDMA, has now been supplanted as the preferred networking protocol in these environments. High Frequency TradingHFT has transformed the investment landscape, now accounting for more than two thirds of all current trading volume. As the traffic volumes and complexity has grown, so has the consequences of inefficiencies in the network architecture. |
|
