Network Working Group David Melman Internet Draft Tal Mizrahi Intended status: Informational Marvell Expires: March 2012 Donald Eastlake Huawei September 11, 2011 FCoE over TRILL draft-mme-trill-fcoe-01.txt Abstract Fibre Channel over Ethernet (FCoE) and TRILL are two emerging standards in the data center environment. While these two protocols are seemingly unrelated, they have a very similar behavior in the forwarding plane, as both perform hop-by-hop forwarding over Ethernet, modifying the packet's MAC addresses at each hop. This document describes an architecture for the integrated deployment of these two protocols. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on March 11, 2012. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. Melman, et al. Expires March 11, 2012 [Page 1] Internet-Draft FCoE over TRILL September 2011 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction ................................................ 2 2. Abbreviations ............................................... 3 3. FCoE over TRILL ............................................. 4 3.1. FCoE over a TRILL Cloud................................. 4 3.2. FCoE over RBridge....................................... 5 3.2.1. FCRB .............................................. 5 3.2.2. Topology .......................................... 7 3.2.3. The FCRB Flow...................................... 8 4. Security Considerations..................................... 10 5. IANA Considerations ........................................ 10 6. Acknowledgments ............................................ 11 7. References ................................................. 11 7.1. Normative References................................... 11 7.2. Informative References................................. 11 1. Introduction Data center networks are rapidly evolving towards a consolidated approach, where Ethernet is used as the common infrastructure for all types of traffic. Storage traffic, which was traditionally dominated by the Fibre Channel (FC) protocol suite, is evolving towards Fibre Channel over Ethernet (FCoE), where native FC packets are encapsulated with an FCoE encapsulation over an Ethernet header. Traffic between two FCoE end nodes (ENodes) is forwarded through one or more FCoE Forwarders (FCF). An FCF takes a forwarding decision based on the Fibre Channel destination ID (D_ID), and enforces security policies between ENodes, also known as zoning. Once an FCF takes a forwarding decision, it modifies the source and destination MAC addresses of the packet, to reflect the path to the next hop FCF or ENode. FCFs use a routing protocol called Fabric Shortest Path First (FSPF) to find the optimal path to each destination. An FCF typically has one or more native Fibre Channel interfaces, allowing Melman, et al. Expires March 11, 2012 [Page 2] Internet-Draft FCoE over TRILL September 2011 it to communicate with native Fibre Channel devices, e.g., storage arrays. TRILL [RFCTRILL] is a protocol for transparent least cost routing, where RBridges forward traffic to their detination based on a least cost route, using a TRILL encapsulation header. RBridges forward TRILL-encapsulated packets based on the Egress RBridge Nickname in the TRILL header. An RBridge forwards a TRILL-encapsulated packet after modifying its MAC addresses to reflect the path to the next-hop RBridge, and decrementing a Hop Count field. TRILL and FCoE bear a strong resemblance in their forwarding planes. Both protocols take a forwarding decision based on protocol addresses above Layer 2, and modify the Ethernet MAC addresses on a per-hop basis. Each of the protocols uses its own routing protocol rather than using any type of bridging protocol such as spanning tree protocol [802.1Q] or the Shortest Path Bridging protocol [802.1aq]. FCoE and TRILL are both targeted at the data center environment, and their concurrent deployment is self-evident. This document describes an architecture for the integrated deployment of these two protocols. 2. Abbreviations ENode FCoE Node such as server or storage array EoR End of Row FC Fibre Channel FCF Fibre Channel Forwarder FCoE Fibre Channel over Ethernet FCRB Fibre Channel forwarder over RBridge FSPF Fabric Shortest Path First LAN Local Area Network RBridge Routing Bridge SAN Storage Area Network ToR Top of Rack TRILL Transparent Interconnection of Lots of Links Melman, et al. Expires March 11, 2012 [Page 3] Internet-Draft FCoE over TRILL September 2011 WAN Wide Area Network 3. FCoE over TRILL 3.1. FCoE over a TRILL Cloud The simplest approach for running FCoE traffic over a TRILL network is presented in Figure 1. The figure illustrates a TRILL-enabled network, where FCoE traffic is transparently forwarded over the TRILL cloud. The figure illustrates two ENodes, a Server and an FCoE Storage Array, an FCF, and a native Fibre Channel SAN connected to the FCF. FCoE traffic between the two ENodes is sent from the first ENode over the TRILL cloud to the FCF, and then back through the TRILL cloud to the second ENode. +---+ | |_________ | | \ ___ _ +---+ \/ \_/ \__ _ __ FCoE Storage _/ \ / \_/ \_ Array / TRILL / +---+ \_ \ (ENode A) \_ Cloud /________| |____/ SAN _/ / \ | | \__ _/ \__/\_ ___/ +---+ \_/ +---+ / \_/ FCF | |________/ | | +---+ Server (ENode B) Figure 1 The "Separate Cloud" Approach The configuration in Figure 1 separates the TRILL cloud(s) and the FCoE cloud(s). The TRILL cloud forwards FCoE traffic as standard Ethernet traffic, and appears to the ENodes and FCF as an Ethernet LAN. The main drawback of the Separate Cloud approach is that RBridges and FCFs are separate nodes in the network, resulting in more cabeling and boxes, and communication between Enodes usually requires two TRILL cloud traversals with twice as many hops. As mentioned above, data center networking is converging towards a consolidated and cost effective approach, where the same infrastructure and equipment is Melman, et al. Expires March 11, 2012 [Page 4] Internet-Draft FCoE over TRILL September 2011 used for both data and storage traffic, and where high efficiency and minimal number of hops are important factors when designing the network topology. 3.2. FCoE over RBridge 3.2.1. FCRB Rather than the Separate Cloud approach discussed in the previous subsection, an alternate approach is presented, where each switch incorporates both an FCF entity and an RBridge entity. This consolidated entity is referred to as FCoE-forwarder-over-RBridge (FCRB). Figure 2 illustrates an FCRB, and its main building blocks. An FCRB can be functionally viewed as two independent entities: o An FCoE Forwarder (FCF) entity. o An RBridge entity. The FCF entity is connected to one of the ports of the RBridge, and appears to the RBridge as a native Ethernet host. A detailed description of the interaction between the layers is presented in Section 3.2.3. Melman, et al. Expires March 11, 2012 [Page 5] Internet-Draft FCoE over TRILL September 2011 +--------------------+ |FCRB | | +-----------+ | | | FCF |-+ | | +-----+-----+ | | | | | | | +-----+-----+ | | | | RBridge | | | | +-+-+-+-+-+-+ | | | | | | | | | | | +---|-|-|-|-|-|---|--+ _ __ FCoE/ / | | | | | \ Native / \_/ \_ +---+ Ethernet / / | | | | \ FC \_ \ | |_________________/ / | | | | \______________/ SAN _/ | | / | | | | \__ _/ +---+ / | | | | \_/ FCoE Storage / | | | | FCoE / Ethernet Array / |_| | | over TRILL (ENode A) / / \_/ \__ / _/ \ +---+ / / TRILL / | |____________/ \_ Cloud / | | / \ +---+ \__/\_ ___/ Server \_/ (ENode B) Figure 2 FCRB Entity in the Network The FCRB entity maintains layer independence between the TRILL and FCoE protocols, while enabling both protocols on the same network. It is noted that FCoE traffic is always forwarded through an FCF, and cannot be forwarded directly between two ENodes. Thus, FCoE traffic between ENodes A and B in the topology in Figure 1 is forwarded through the path ENode A-->TRILL cloud-->FCF-->TRILL cloud-->ENode B Traffic between A and B in the topology in Figure 2 is forwarded through the path ENode A-->FCRB-->ENode B Melman, et al. Expires March 11, 2012 [Page 6] Internet-Draft FCoE over TRILL September 2011 Hence, the usage of FCRB entities allows TRILL and FCoE to use common infrastructure and equipment, as opposed to the Separate Cloud topology presented in Figure 1. 3.2.2. Topology The network configuration illustrated in Figure 3 shows a typical topology of a data center network. Servers are hierarchically connected through Top-of-Rack (ToR) switches, and End-of-Row (EoR) racks. The EoR switches to other clouds, such as an external WAN, or a native FC SAN. Melman, et al. Expires March 11, 2012 [Page 7] Internet-Draft FCoE over TRILL September 2011 _ __ _ __ / \_/ \_ / \_/ \_ \_ \ \_ \ ..... / SAN _/ _/ WAN _/ \__ _/ \ / \__ _/ \_/ \ / \_/ | \ / | | \ / | | \/ | EoR +----+_______/\_______+----+ FCoE over | | | | RBridge | | | | (FCRB) +----+ +----+ / \ / \ / \ / \ ToR +---+ +---+ +---+ +---+ FCoE over | | | | | | | | RBridge | | | | | | | | (FCRB) +---+ +---+ +---+ +---+ / \ / \ / \ / \ / \ / \ / \ / \ +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ Servers/ | | | | | | | | | | | | | | | | ENodes +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ A B C D E F G H Figure 3 FCoE over RBridge Topology Note that in the example in Figure 3 all the ToR and EoR switches are FCRB entities, but it is also possible for some of the network nodes to be pure RBridges, creating a topology where FCRBs are interconnected through TRILL clouds. 3.2.3. The FCRB Flow FCoE traffic sent between two ENodes, A and B, is transmitted through the ToR FCRB, since A and B are connected to the same ToR. Traffic between A and C must be forwarded through the EoR FCRB. Melman, et al. Expires March 11, 2012 [Page 8] Internet-Draft FCoE over TRILL September 2011 +--------+ +--------+ +--------+ +--------+ +--------+ | FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FCoE | | ENode | +--------+ +--------+ +--------+ | ENode | | | |RBridge |.....|RBridge |.....|RBridge | | | +--------+ +--------+ +--------+ +--------+ +--------+ |Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet| +--------+ +--------+ +--------+ +--------+ +--------+ Server ToR EoR ToR FCoE Storage ENode A FCRB FCRB FCRB Array ENode C Figure 4 Traffic between two ENodes - Example Figure 4 illustrates the traffic between ENodes A and C that are not connected to the same ToR. o FCoE traffic from A is sent to the ToR over the Ethernet interface. o The RBridge entity at the ToR forwards the packet to the FCF, analogous to forwarding between two Ethernet hosts. The FCF entity at the ToR takes a forwarding decision, and updates the destination MAC address of the packet to the address of the EoR FCF. The packet is then forwarded to the RBridge entity, where it is encapsulated in a TRILL header, and sent to the EoR FCRB. o The RBridge entity in the EoR FCRB, acting as the egress RBridge, decapsulates the TRILL header and forwards the FCoE packet to the FCF entity. The FCF takes a forwarding decision and updates the MAC address of the packet according to the next hop ToR. The packet is then forwarded to the RBridge and encapsulated with a new TRILL header. o At the ToR FCRB, the packet reaches the final egress RBridge, and the TRILL encapsulation is removed. The FCF then forwards the packet to the RBridge entity after updating its MAC addresses. The RBridge entity forwards the packet to the target ENode. Melman, et al. Expires March 11, 2012 [Page 9] Internet-Draft FCoE over TRILL September 2011 +--------+ +--------+ +---------+ +--------+ | FCoE |.....| FCF |.....| FCF |.....| FC | | ENode | +--------+ +----+----+ |protocol| | | |RBridge |.....| RB | | | stack | +--------+ +--------+ +----+ FC | | | |Ethernet|<===>|Ethernet|<===>|Eth | |<===>| | +--------+ +--------+ +----+----+ +--------+ Server ToR EoR Native FC ENode FCRB FCRB Storage Array Figure 5 Example Traffic between ENode & Native FC Storage Array Figure 5 illustrates traffic sent between an ENode and an FC Storage Array, following the network topology in Figure 3. o FCoE traffic from the ENode is sent to the ToR over the Ethernet interface. o The RBridge entity at the ToR forwards the packet to the FCF. The FCF entity at the ToR takes a forwarding decision and updates the destination MAC address of the packet to the address of the EoR FCF. The packet is then forwarded to the RBridge entity, where it is encapsulated in a TRILL header, and sent to the EoR FCRB. o The egress RBridge entity at the EoR FCRB decapsulates the TRILL header, and forwards the FCoE packet to the FCF entity. The packet is then forwarded as a native FC packet through the FC interface to the native FC node. 4. Security Considerations For general TRILL Security Considerations see [RFCTRILL]. For general FCoE Security Consideration see Annex D of [FC-BB-5]. There are no additional security implications imposed by this document. 5. IANA Considerations There are no IANA actions required by this document. RFC Editor: please delete this section before publication. Melman, et al. Expires March 11, 2012 [Page 10] Internet-Draft FCoE over TRILL September 2011 6. Acknowledgments This document was prepared using 2-Word-v2.0.template.dot. 7. References 7.1. Normative References [RFCTRILL] Perlman, R., Eastlake, D., Dutt, D., Gai, S., Ghanwani, A., "Routing Bridges (RBridges): Base Protocol Specification", RFC6325, July 2011. 7.2. Informative References [FC-BB-5] ANSI INCITS 462: Information Technology - Fibre Channel - Backbone - 5 (FC-BB-5). [802.1Q] "IEEE Standard for Local and metropolitan area networks - Virtual Bridged Local Area Networks", IEEE Std 802.1Q-2011, May 2011. [802.1aq] "IEEE Standard for Local and metropolitan area networks - Shortest Path Bridging", work in progress, June 2011. Authors' Addresses David Melman Marvell 6 Hamada St. Yokneam, 20692 Israel Email: davidme@marvell.com Tal Mizrahi Marvell 6 Hamada St. Yokneam, 20692 Israel Email: talmi@marvell.com Donald Eastlake 3rd Huawei Technologies Melman, et al. Expires March 11, 2012 [Page 11] Internet-Draft FCoE over TRILL September 2011 155 Beaver Street Milford, MA 01757 USA Phone: +1-508-333-2270 EMail: d3e3e3@gmail.com Melman, et al. Expires March 11, 2012 [Page 12]