Storage Maintenance (StorM) Working Group Mike Ko Internet Draft Huawei Symantec Intended status : Proposed Standard December 4, 2009 Expires : June 2010 Obsoletes: RFC 5046 iSCSI Extensions for RDMA Specification draft-ietf-storm-iser-00.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on June 4, 2010. Abstract iSCSI Extensions for RDMA provides the RDMA data transfer capability to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol such as the iWARP protocol suite. An RDMA-Capable Protocol provides RDMA Read and Write services, which enable data to be transferred directly into SCSI I/O Buffers without intermediate data copies. This document describes the extensions to the iSCSI protocol to support RDMA services as provided by an RDMA-Capable Protocol such as the iWARP protocol suite. This document obsoletes RFC5046 Ko Expires June 2010 [Page 1] Internet-Draft iSER Specification December 2009 Table of Contents 1 Definitions and Acronyms....................................6 1.1 Definitions.................................................6 1.2 Acronyms...................................................12 1.3 Conventions................................................14 2 Introduction...............................................15 2.1 Motivation.................................................15 2.2 Architectural Goals........................................16 2.3 Protocol Overview..........................................17 2.4 RDMA services and iSER.....................................18 2.4.1 STag......................................................18 2.4.2 Send......................................................19 2.4.3 RDMA Write................................................19 2.4.4 RDMA Read.................................................19 2.5 SCSI Read Overview.........................................20 2.6 SCSI Write Overview........................................20 2.7 iSCSI/iSER Layering........................................21 3 Upper Layer Interface Requirements.........................22 3.1 Operational Primitives offered by iSER.....................22 3.1.1 Send_Control..............................................23 3.1.2 Put_Data..................................................23 3.1.3 Get_Data..................................................23 3.1.4 Allocate_Connection_Resources.............................24 3.1.5 Deallocate_Connection_Resources...........................24 3.1.6 Enable_Datamover..........................................24 3.1.7 Connection_Terminate......................................25 3.1.8 Notice_Key_Values.........................................25 3.1.9 Deallocate_Task_Resources.................................25 3.2 Operational Primitives used by iSER........................26 3.2.1 Control_Notify............................................26 3.2.2 Data_Completion_Notify....................................26 3.2.3 Data_ACK_Notify...........................................27 3.2.4 Connection_Terminate_Notify...............................27 3.3 iSCSI Protocol Usage Requirements..........................27 4 Lower Layer Interface Requirements.........................29 4.1 Interactions with the RCaP Layer...........................29 4.2 Interactions with the Transport Layer......................30 5 Connection Setup and Termination...........................31 5.1 iSCSI/iSER Connection Setup................................31 5.1.1 Initiator Behavior........................................32 5.1.2 Target Behavior...........................................34 5.1.3 iSER Hello Exchange.......................................35 5.2 iSCSI/iSER Connection Termination..........................36 5.2.1 Normal Connection Termination at the Initiator............36 5.2.2 Normal Connection Termination at the Target...............37 5.2.3 Termination without Logout Request/Response PDUs..........37 Ko et al. Expires June 2010 [Page 2] Internet-Draft iSER Specification December 2009 6 Login/Text Operational Keys................................39 6.1 HeaderDigest and DataDigest................................39 6.2 MaxRecvDataSegmentLength...................................39 6.3 RDMAExtensions.............................................40 6.4 TargetRecvDataSegmentLength................................41 6.5 InitiatorRecvDataSegmentLength.............................41 6.6 OFMarker and IFMarker......................................42 6.7 MaxOutstandingUnexpectedPDUs...............................42 6.8 MaxAHSLength...............................................42 6.9 WriteAddressForSolicitedDataOnly...........................43 7 iSCSI PDU Considerations...................................44 7.1 iSCSI Data-Type PDU........................................44 7.2 iSCSI Control-Type PDU.....................................45 7.3 iSCSI PDUs.................................................45 7.3.1 SCSI Command..............................................45 7.3.2 SCSI Response.............................................47 7.3.3 Task Management Function Request/Response.................48 7.3.4 SCSI Data-out.............................................50 7.3.5 SCSI Data-in..............................................50 7.3.6 Ready To Transfer (R2T)...................................53 7.3.7 Asynchronous Message......................................55 7.3.8 Text Request & Text Response..............................55 7.3.9 Login Request & Login Response............................55 7.3.10 Logout Request & Logout Response........................55 7.3.11 SNACK Request...........................................56 7.3.12 Reject..................................................56 7.3.13 NOP-Out & NOP-In........................................56 8 Flow Control and STag Management...........................57 8.1 Flow Control for RDMA Send Message Types...................57 8.1.1 Flow Control for Control-Type PDUs from the Initiator.....57 8.1.2 Flow Control for Control-Type PDUs from the Target........60 8.2 Flow Control for RDMA Read Resources.......................61 8.3 STag Management............................................61 8.3.1 Allocation of STags.......................................62 8.3.2 Invalidation of STags.....................................62 9 iSER Control and Data Transfer.............................64 9.1 iSER Header Format.........................................64 9.2 iSER Header Format for iSCSI Control-Type PDU..............64 9.3 iSER Header Format for iSER Hello Message..................67 9.4 iSER Header Format for iSER HelloReply Message.............68 9.5 SCSI Data Transfer Operations..............................69 9.5.1 SCSI Write Operation......................................69 9.5.2 SCSI Read Operation.......................................70 9.5.3 Bidirectional Operation...................................70 10 iSER Error Handling and Recovery...........................71 10.1 Error Handling............................................71 10.1.1 Errors in the Transport Layer...........................71 Ko et al. Expires June 2010 [Page 3] Internet-Draft iSER Specification December 2009 10.1.2 Errors in the RCaP Layer................................72 10.1.3 Errors in the iSER Layer................................72 10.1.4 Errors in the iSCSI Layer...............................74 10.2 Error Recovery............................................76 10.2.1 PDU Recovery............................................76 10.2.2 Connection Recovery.....................................77 11 Security Considerations....................................78 12 IANA Considerations........................................79 13 References.................................................80 13.1 Normative References......................................80 13.2 Informative References....................................80 14 Summary of Changes from RFC 5046...........................82 14.1 Proposed Changes To Be Considered.........................83 14.2 Changes To Be Done........................................83 Appendix A.......................................................84 14.3 iWARP Message Format for iSER.............................84 14.3.1 iWARP Message Format for iSER Hello Message.............84 14.3.2 iWARP Message Format for iSER HelloReply Message........85 14.3.3 iWARP Message Format for SCSI Read Command PDU..........86 14.3.4 iWARP Message Format for SCSI Read Data.................87 14.3.5 iWARP Message Format for SCSI Write Command PDU.........88 14.3.6 iWARP Message Format for RDMA Read Request..............89 14.3.7 iWARP Message Format for Solicited SCSI Write Data......90 14.3.8 iWARP Message Format for SCSI Response PDU..............91 15 Appendix B.................................................92 15.1 Architectural discussion of iSER over InfiniBand..........92 15.2 The Host side of the iSCSI & iSER connections in Infiniband92 15.3 The Storage side of iSCSI & iSER mixed network environment93 15.4 Discovery processes for an InfiniBand Host................93 15.5 IBTA Connection specifications............................94 16 Acknowledgments............................................95 Ko et al. Expires June 2010 [Page 4] Internet-Draft iSER Specification December 2009 Table of Figures Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase...21 Figure 2 iSER Header Format.....................................64 Figure 3 iSER Header Format for iSCSI Control-Type PDU..........65 Figure 4 iSER Header Format for iSER Hello Message..............67 Figure 5 iSER Header Format for iSER HelloReply Message.........68 Figure 6 SendSE Message containing an iSER Hello Message........84 Figure 7 SendSE Message containing an iSER HelloReply Message...85 Figure 8 SendSE Message containing a SCSI Read Command PDU......86 Figure 9 RDMA Write Message containing SCSI Read Data...........87 Figure 10 SendSE Message containing a SCSI Write Command PDU....88 Figure 11 RDMA Read Request Message.............................89 Figure 12 RDMA Read Response Message containing SCSI Write Data.90 Figure 13 SendInvSE Message containing SCSI Response PDU........91 Figure 14 iSCSI and iSER on IB..................................92 Figure 15 Storage Controller with TCP, iWARP, and IB Connections93 Ko et al. Expires June 2010 [Page 5] Internet-Draft iSER Specification December 2009 1 Definitions and Acronyms 1.1 Definitions Advertisement (Advertised, Advertise, Advertisements, Advertises) - The act of informing a remote iSER Layer that a local node's buffer is available to it. A Node makes a buffer available for incoming RDMA Read Request Message or incoming RDMA Write Message access by informing the remote iSER Layer of the Tagged Buffer identifiers (STag, TO, and buffer length). Note that this Advertisement of Tagged Buffer information is the responsibility of the iSER Layer on either end and is not defined by the RDMA-Capable Protocol. A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message destined for the remote iSER Layer. Completion (Completed, Complete, Completes) - Completion is defined as the process by the RDMA-Capable Protocol layer to inform the iSER Layer, that a particular RDMA Operation has performed all functions specified for the RDMA Operation. Connection - A connection is a logical circuit between the initiator and the target, e.g., a TCP connection. Communication between the initiator and the target occurs over one or more connections. The connections carry control messages, SCSI commands, parameters, and data within iSCSI Protocol Data Units (iSCSI PDUs). Connection Handle - An information element that identifies the particular iSCSI connection and is unique for a given iSCSI-iSER pair. Every invocation of an Operational Primitive is qualified with the Connection Handle. Data Sink - The peer receiving a data payload. Note that the Data Sink can be required to both send and receive RCaP Messages to transfer a data payload. Data Source - The peer sending a data payload. Note that the Data Source can be required to both send and receive RCaP Messages to transfer a data payload. Datamover Interface (DI) - The interface between the iSCSI Layer and the Datamover Layer as described in [DA]. Datamover Layer - A layer that is directly below the iSCSI Layer and above the underlying transport layers. This layer exposes and Ko et al. Expires June 2010 [Page 6] Internet-Draft iSER Specification December 2009 uses a set of transport independent Operational Primitives for the communication between the iSCSI Layer and itself. The Datamover layer, operating in conjunction with the transport layers, moves the control and data information on the iSCSI connection. In this specification, the iSER Layer is the Datamover layer. Datamover Protocol - A Datamover protocol is the wire-protocol that is defined to realize the Datamover layer functionality. In this specification, the iSER protocol is the Datamover protocol. Event - An indication provided by the RDMA-Capable Protocol layer to the iSER Layer to indicate a Completion or other condition requiring immediate attention. Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming outstanding RDMA Read Requests that the RDMA-Capable Controller can handle on a particular RCaP Stream at the Data Source. For some RDMA-Capable Protocol layers, the term "IRD" may be known by a different name. For example, for InfiniBand, the equivalent for IRD is the Responder Resources. Invalidate STag - A mechanism used to prevent the Remote Peer from reusing a previous explicitly Advertised STag, until the iSER Layer at the local node makes it available through a subsequent explicit Advertisement. I/O Buffer - A buffer that is used in a SCSI Read or Write operation so SCSI data may be sent from or received into that buffer. iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of the SCSI Architecture Model of SAM-2 over TCP. iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data- type PDU and also not a SCSI Data-out PDU carrying solicited data is defined as an iSCSI control-type PDU. Specifically, it is to be noted that SCSI Data-out PDUs for unsolicited data are defined as iSCSI control-type PDUs. iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection. An iSCSI data-type PDU, when requested for transmission by the sender iSCSI Layer, results in the associated data transfer without the participation of the remote iSCSI Layer, i.e. the PDU itself is not delivered as-is Ko et al. Expires June 2010 [Page 7] Internet-Draft iSER Specification December 2009 to the remote iSCSI Layer. The following iSCSI PDUs constitute the set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU. iSCSI Layer - A layer in the protocol stack implementation within an end node that implements the iSCSI protocol and interfaces with the iSER Layer via the Datamover Interface. iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI Layer at the initiator and the iSCSI Layer at the target divide their communications into messages. The term "iSCSI protocol data unit" (iSCSI PDU) is used for these messages. iSCSI/iSER Connection - An iSER-assisted iSCSI connection. iSCSI/iSER Session - An iSER-assisted iSCSI session. iSCSI-iSER Pair - The iSCSI Layer and the underlying iSER Layer. iSER - iSCSI Extensions for RDMA, the protocol defined in this document. iSER-assisted - A term generally used to describe the operation of iSCSI when the iSER functionality is also enabled below the iSCSI Layer for the specific iSCSI/iSER connection in question. iSER-IRD - This variable represents the maximum number of incoming outstanding RDMA Read Requests that the iSER Layer at the initiator declares on a particular RCaP Stream. iSER-ORD - This variable represents the maximum number of outstanding RDMA Read Requests that the iSER Layer can initiate on a particular RCaP Stream. This variable is maintained only by the iSER Layer at the target. iSER Layer - The layer that implements the iSCSI Extensions for RDMA (iSER) protocol. iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and [MPA] when layered above [TCP]. [RDMAP] and [DDP] may be layered above SCTP or other transport protocols. Local Mapping - A task state record maintained by the iSER Layer that associates the Initiator Task Tag to the Local STag(s). The specifics of the record structure are implementation dependent. Ko et al. Expires June 2010 [Page 8] Internet-Draft iSER Specification December 2009 Local Peer - The implementation of the RDMA-Capable Protocol on the local end of the connection. Used to refer to the local entity when describing protocol exchanges or other interactions between two Nodes. Node - A computing device attached to one or more links of a network. A Node in this context does not refer to a specific application or protocol instantiation running on the computer. A Node may consist of one or more RDMA-Capable Controllers installed in a host computer. Operational Primitive - An Operational Primitive is an abstract functional interface procedure that requests another layer to perform a specific action on the requestor's behalf or notifies the other layer of some event. The Datamover Interface between an iSCSI Layer and a Datamover layer within an iSCSI end node uses a set of Operational Primitives to define the functional interface between the two layers. Note that not every invocation of an Operational Primitive may elicit a response from the requested layer. A full discussion of the Operational Primitive types and request-response semantics available to iSCSI and iSER can be found in [DA]. Outbound RDMA Read Queue Depth (ORD) - The maximum number of outstanding RDMA Read Requests that the RDMA-Capable Controller can initiate on a particular RCaP Stream at the Data Sink. For some RDMA-Capable Protocol layer, the term "ORD" may be known by a different name. For example, for InfiniBand, the equivalent for ORD is the Initiator Depth. Phase-Collapse - Refers to the optimization in iSCSI where the SCSI status is transferred along with the final SCSI Data-in PDU from a target. See section 3.2 in [RFC3720]. RCaP Message - One or more packets of the network layer comprising a single RDMA operation or a part of an RDMA Read Operation of the RDMA-Capable Protocol. For iWARP, an RCaP Message is known as an RDMAP Message. RCaP Stream - A single bidirectional association between the peer RDMA-Capable Protocol layers on two Nodes over a single transport-level stream. For iWARP, an RCaP Stream is known as an RDMAP Stream, and the association is created following a successful Login Phase during which iSER support is negotiated. Ko et al. Expires June 2010 [Page 9] Internet-Draft iSER Specification December 2009 RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that provides a reliable RDMA transport functionality, e.g., iWARP, InfiniBand, etc. RDMA-Capable Controller - A network I/O adapter or embedded controller with RDMA functionality. For example, for iWARP, this could be an RNIC, and for InfiniBand, this could be a HCA (Host Channel Adapter) or TCA (Target Channel Adapter). RDMA-enabled Network Interface Controller (RNIC) - A network I/O adapter or embedded controller with iWARP functionality. RDMA Operation - A sequence of RCaP Messages, including control Messages, to transfer data from a Data Source to a Data Sink. The following RDMA Operations are defined - RDMA Write Operation, RDMA Read Operation, Send Operation, Send with Invalidate Operation, Send with Solicited Event Operation, Send with Solicited Event and Invalidate Operation, and Terminate Operation. RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations to transfer ULP data between a Local Peer and the Remote Peer as described in [RDMAP]. RDMA Read Operation - An RDMA Operation used by the Data Sink to transfer the contents of a Data Source buffer from the Remote Peer to a Data Sink buffer at the Local Peer. An RDMA Read operation consists of a single RDMA Read Request Message and a single RDMA Read Response Message. RDMA Read Request - An RCaP Message used by the Data Sink to request the Data Source to transfer the contents of a buffer. The RDMA Read Request Message describes both the Data Source and the Data Sink buffers. RDMA Read Response - An RCaP Message used by the Data Source to transfer the contents of a buffer to the Data Sink, in response to an RDMA Read Request. The RDMA Read Response Message only describes the Data Sink buffer. RDMA Write Operation - An RDMA Operation used by the Data Source to transfer the contents of a Data Source buffer from the Local Peer to a Data Sink buffer at the Remote Peer. The RDMA Write Message only describes the Data Sink buffer. Remote Direct Memory Access (RDMA) - A method of accessing memory on a remote system in which the local system specifies the remote Ko et al. Expires June 2010 [Page 10] Internet-Draft iSER Specification December 2009 location of the data to be transferred. Employing an RDMA- Capable Controller in the remote system allows the access to take place without interrupting the processing of the CPU(s) on the system. Remote Mapping - A task state record maintained by the iSER Layer that associates the Initiator Task Tag to the Advertised STag(s). The specifics of the record structure are implementation dependent. Remote Peer - The implementation of the RDMA-Capable Protocol on the opposite end of the connection. Used to refer to the remote entity when describing protocol exchanges or other interactions between two Nodes. SCSI Layer - This layer builds/receives SCSI CDBs (Command Descriptor Blocks) and sends/receives them with the remaining command execute [SAM2] parameters to/from the iSCSI Layer. Send - An RDMA Operation that transfers the contents of a Buffer from the Local Peer to a Buffer at the Remote Peer. Send Message Type - A Send Message, Send with Invalidate Message, Send with Solicited Event Message, or Send with Solicited Event and Invalidate Message. SendInvSE Message - A Send with Solicited Event and Invalidate Message. SendSE Message - A Send with Solicited Event Message Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for an R2T PDU. The semantics for both types of sequence numbers are as defined in [RFC3720]. Session, iSCSI Session - The group of Connections that link an initiator SCSI port with a target SCSI port form an iSCSI session (equivalent to a SCSI I-T nexus). Connections can be added to and removed from a session even while the I-T nexus is intact. Across all connections within a session, an initiator sees one and the same target. Solicited Event (SE) - A facility by which an RDMA Operation sender may cause an Event to be generated at the recipient, if the recipient is configured to generate such an Event, when a Send with Solicited Event or Send with Solicited Event and Invalidate Message is received. Ko et al. Expires June 2010 [Page 11] Internet-Draft iSER Specification December 2009 Steering Tag (STag) - An identifier of a Tagged Buffer on a Node (Local or Remote) as defined in [RDMAP] and [DDP]. For other RDMA-Capable Protocols, the Steering Tag may be known by different names but will be herein referred to as STags. For example, for Infiniband, a Remote STag is known as an R-Key, and a Local STag is known as an L-Key, and both will be considered STags. Tagged Buffer - A buffer that is explicitly Advertised to the iSER Layer at the remote node through the exchange of an STag, Tagged Offset, and length. Tagged Offset (TO) - The offset within a Tagged Buffer. Traditional iSCSI - Refers to the iSCSI protocol as defined in [RFC3720] (i.e. without the iSER enhancements). Untagged Buffer - A buffer that is not explicitly Advertised to the iSER Layer at the remode node. 1.2 Acronyms Acronym Definition -------------------------------------------------------------- AHS Additional Header Segment BHS Basic Header Segment CO Connection Only CRC Cyclic Redundancy Check DDP Direct Data Placement Protocol DI Datamover Interface HCA Host Channel Adapter IANA Internet Assigned Numbers Authority IB Infiniband IETF Internet Engineering Task Force I/O Input - Output Ko et al. Expires June 2010 [Page 12] Internet-Draft iSER Specification December 2009 IO Initialize Only IP Internet Protocol IPoIB IP over Infiniband IPsec Internet Protocol Security iSER iSCSI Extensions for RDMA ITT Initiator Task Tag LO Leading Only MPA Marker PDU Aligned Framing for TCP NOP No Operation NSG Next Stage (during the iSCSI Login Phase) OS Operating System PDU Protocol Data Unit R2T Ready To Transfer R2TSN Ready To Transfer Sequence Number RDMA Remote Direct Memory Access RDMAP Remote Direct Memory Access Protocol RFC Request For Comments RNIC RDMA-enabled Network Interface Controller SAM2 SCSI Architecture Model - 2 SCSI Small Computer Systems Interface SNACK Selective Negative Acknowledgment - also Sequence Number Acknowledgement for data STag Steering Tag SW Session Wide Ko et al. Expires June 2010 [Page 13] Internet-Draft iSER Specification December 2009 TCA Target Channel Adapter TCP Transmission Control Protocol TMF Task Management Function TTT Target Transfer Tag TO Tagged Offset ULP Upper Level Protocol 1.3 Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Ko et al. Expires June 2010 [Page 14] Internet-Draft iSER Specification December 2009 2 Introduction 2.1 Motivation The iSCSI protocol ([RFC3720]) is a mapping of the SCSI Architecture Model (see [SAM2]) over the TCP protocol. SCSI commands are carried by iSCSI requests and SCSI responses and status are carried by iSCSI responses. Other iSCSI protocol exchanges and SCSI Data are also transported in iSCSI PDUs. Out-of-order TCP segments in the Traditional iSCSI model have to be stored and reassembled before the iSCSI protocol layer within an end node can place the data in the iSCSI buffers. This reassembly is required because not every TCP segment is likely to contain an iSCSI header to enable its placement and TCP itself does not have a built- in mechanism for signaling ULP message boundaries to aid placement of out-of-order segments. This TCP reassembly at high network speeds is quite counter-productive for the following reasons: wasted memory bandwidth in data copying, need for reassembly memory, wasted CPU cycles in data copying, and the general store-and-forward latency from an application perspective. TCP reassembly was recognized as a serious issue in [RFC3720], and the notion of a "sync and steering layer" was introduced that is optional to implement and use. One specific sync and steering mechanism, called "markers", was defined in [RFC3720] which provides an application- level way of framing iSCSI PDUs within the TCP data stream even when the TCP segments are not yet reassembled to be in-order. With these defined techniques in [RFC3720], a Network Interface Controller customized for iSCSI (SNIC) could offload the TCP/IP processing and support direct data placement, but most iSCSI implementations do not support iSCSI "markers", making SNIC marker-based direct data placement unusable in practice. The iWARP protocol stack provides direct data placement functionality that is usable in practice, and in addition, there is also interest in using iSCSI with other RDMA protocol stacks that support direct data placement, such as the one provided by InfiniBand. The generic term RDMA-Capable Protocol (RCaP) is used to refer to the RDMA functionality provided by such protocol stacks. With the availability of RDMA-Capable Controllers within a host system, which does not have SNICs, it is appropriate for iSCSI to be able to exploit the direct data placement function of the RDMA- Capable Controller like other applications. Ko et al. Expires June 2010 [Page 15] Internet-Draft iSER Specification December 2009 iSCSI Extensions for RDMA (iSER) is designed precisely to take advantage of generic RDMA technologies - iSER's goal is to permit iSCSI to employ direct data placement and RDMA capabilities using a generic RDMA-Capable Controller. In summary, iSCSI/iSER protocol stack is designed to enable scaling to high speeds by relying on a generic data placement process and RDMA technologies and products, which enable direct data placement of both in-order and out-of-order data. This document describes iSER as a protocol extension to iSCSI, both for convenience of description and also because it is true in a very strict protocol sense. However, it is to be noted that iSER is in reality extending the connectivity of the iSCSI protocol defined in [RFC3720], and the name iSER reflects this reality. When the iSCSI protocol as defined in [RFC3720] (i.e. without the iSER enhancements) is intended in the rest of the document, the term "Traditional iSCSI" is used to make the intention clear. 2.2 Architectural Goals This section summarizes the architectural goals that guided the design of iSER. 1. Provide an RDMA data transfer model for iSCSI that enables direct in order or out of order data placement of SCSI data into pre- allocated SCSI buffers while maintaining in order data delivery. 2. Not require any major changes to SCSI Architecture Model [SAM2] and SCSI command set standards. 3. Utilize existing iSCSI infrastructure (sometimes referred to as "iSCSI ecosystem") including but not limited to MIB, bootstrapping, negotiation, naming & discovery, and security. 4. Require a session to operate in the Traditional iSCSI data transfer mode if iSER is not supported by either the initiator or the target (not require iSCSI full feature phase interoperability between an end node operating in Traditional iSCSI mode, and an end node operating in iSER-assisted mode). 5. Allow initiator and target implementations to utilize generic RDMA-Capable Controllers such as RNICs, or implement iSCSI and iSER in software (not require iSCSI or iSER specific assists in the RCaP implementation or RDMA-Capable Controller). Ko et al. Expires June 2010 [Page 16] Internet-Draft iSER Specification December 2009 6. Require full and only generic RCaP functionality at both the initiator and the target. 7. Implement a light weight Datamover protocol for iSCSI with minimal state maintenance. 2.3 Protocol Overview Consistent with the architectural goals stated in section 2.2, the iSER protocol does not require changes in the iSCSI ecosystem or any related SCSI specifications. iSER protocol defines the mapping of iSCSI PDUs to RCaP Messages in such a way that it is entirely feasible to realize iSCSI/iSER implementations that are based on generic RDMA-Capable Controllers. The iSER protocol layer requires minimal state maintenance to assist an iSCSI full feature phase connection, besides being oblivious to the notion of an iSCSI session. The crucial protocol aspects of iSER may be summarized thus: 1. iSER-assisted mode is negotiated during the iSCSI login for each session, and an entire iSCSI session can only operate in one mode (i.e. a connection in a session cannot operate in iSER-assisted mode if a different connection of the same session is already in full feature phase in the Traditional iSCSI mode). 2. Once in iSER-assisted mode, all iSCSI interactions on that connection use RCaP Messages. 3. A Send Message Type is used for carrying an iSCSI control-type PDU preceded by an iSER header. See section 7.2 for more details on iSCSI control-type PDUs. 4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages are used for carrying control and all data information associated with the iSCSI data-type PDUs. See section 7.1 for more details on iSCSI data-type PDUs. 5. Target drives all data transfer (with the exception of iSCSI unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA Read Requests and RDMA Writes respectively. 6. RCaP is responsible for ensuring data integrity. (For example, iWARP includes a CRC-enhanced framing layer called MPA on top of TCP; and for Infiniband, the CRCs are included in the Reliable Connection mode). For this reason, iSCSI header and data digests are negotiated to "None" for iSCSI/iSER sessions. Ko et al. Expires June 2010 [Page 17] Internet-Draft iSER Specification December 2009 7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully supported by iSER. (However, see section 7.3.11 on the handling of SNACK Request PDUs.) 8. iSER requires no changes to iSCSI authentication, security, and text mode negotiation mechanisms. Note that Traditional iSCSI implementations may have to be adapted to employ iSER. It is expected that the adaptation when required is likely to be centered around the upper layer interface requirements of iSER (section 3). 2.4 RDMA services and iSER iSER is designed to work with software and/or hardware protocol stacks providing the protocol services defined in RCaP documents such as [RDMAP], [IB], etc. The following subsections describe the key protocol elements of RCaP services that iSER relies on. 2.4.1 STag An STag is the identifier of an I/O Buffer unique to an RDMA-Capable Controller that the iSER Layer Advertises to the remote iSCSI/iSER node in order to complete a SCSI I/O. In iSER, Advertisement is the act of informing the target by the initiator that an I/O Buffer is available at the initiator for RDMA Read or RDMA Write access by the target. The initiator Advertises the I/O Buffer by including the STag in the header of an iSER Message containing the SCSI Command PDU to the target. The base Tagged Offset is not explicitly specified, but the target must always assume it as zero. The buffer length is as specified in the SCSI Command PDU. The iSER Layer at the initiator Advertises the STag for the I/O Buffer of each SCSI I/O to the iSER Layer at the target in the iSER header of the SendSE Message containing the SCSI Command PDU, unless the I/O can be completely satisfied by unsolicited data alone. The iSER Layer at the target provides the STag for the I/O Buffer that is the Data Sink of an RDMA Read Operation (section 2.4.4) to the RCaP layer on the initiator node - i.e. this is completely transparent to the iSER Layer at the initiator. The iSER protocol is defined so that the Advertised STag is automatically invalidated upon a normal completion of the associated task. This automatic invalidation is realized via the SendInvSE Ko et al. Expires June 2010 [Page 18] Internet-Draft iSER Specification December 2009 Message carrying the SCSI Response PDU. There are two exceptions to this automatic invalidation - bidirectional commands, and abnormal completion of a command. The iSER Layer at the initiator is required to explicitly invalidate the STag in these cases, in addition to sanity checking the automatic invalidation even when that does happen. 2.4.2 Send Send is the RDMA Operation that is not addressed to an Advertised buffer by the sending side, and thus uses Untagged buffers on the receiving side. The iSER Layer at the initiator uses the Send Operation to transmit any iSCSI control-type PDU to the target. As an example, the initiator uses Send Operations to transfer iSER Messages containing SCSI Command PDUs to the iSER Layer at the target. An iSER layer at the target uses the Send Operation to transmit any iSCSI control-type PDU to the initiator. As an example, the target uses Send Operations to transfer iSER Messages containing SCSI Response PDUs to the iSER Layer at the initiator. 2.4.3 RDMA Write RDMA Write is the RDMA Operation that is used to place data into an Advertised buffer on the receiving side. The sending side addresses the Message using an STag and a Tagged Offset that are valid on the Data Sink. The iSER Layer at the target uses the RDMA Write Operation to transfer the contents of a local I/O Buffer to an Advertised I/O Buffer at the initiator. The iSER Layer at the target uses the RDMA Write to transfer whole or part of the data required to complete a SCSI Read command. The iSER Layer at the initiator does not employ RDMA Writes. 2.4.4 RDMA Read RDMA Read is the RDMA Operation that is used to retrieve data from an Advertised buffer on a remote node. The sending side of the RDMA Read Request addresses the Message using an STag and a Tagged Offset that are valid on the Data Source in addition to providing a valid local STag and Tagged Offset that identify the Data Sink. Ko et al. Expires June 2010 [Page 19] Internet-Draft iSER Specification December 2009 The iSER Layer at the target uses the RDMA Read Operation to transfer the contents of an Advertised I/O Buffer at the initiator to a local I/O Buffer at the target. The iSER Layer at the target uses the RDMA Read to fetch whole or part of the data required to complete a SCSI Write Command. The iSER Layer at the initiator does not employ RDMA Reads. 2.5 SCSI Read Overview The iSER Layer at the initiator receives the SCSI Command PDU from the iSCSI Layer. The iSER Layer at the initiator generates an STag for the I/O Buffer of the SCSI Read and Advertises the buffer by including the STag as part of the iSER header for the PDU. The iSER Message is transferred to the target using a SendSE Message. The iSER Layer at the target uses one or more RDMA Writes to transfer the data required to complete the SCSI Read. The iSER Layer at the target uses a SendInvSE Message to transfer the SCSI Response PDU back to the iSER Layer at the initiator. The iSER Layer at the initiator notifies the iSCSI Layer of the availability of the SCSI Response PDU. 2.6 SCSI Write Overview The iSER Layer at the initiator receives the SCSI Command PDU from the iSCSI Layer. If solicited data transfer is involved, the iSER Layer at the initiator generates an STag for the I/O Buffer of the SCSI Write and Advertises the buffer by including the STag as part of the iSER header for the PDU. The iSER Message is transferred to the target using a SendSE Message. The iSER Layer at the initiator may optionally send one or more non- immediate unsolicited data PDUs to the target using Send Message Types. If solicited data transfer is involved, the iSER Layer at the target uses one or more RDMA Reads to transfer the data required to complete the SCSI Write. The iSER Layer at the target uses a SendInvSE Message to transfer the SCSI Response PDU back to the iSER Layer at the initiator. The iSER Layer at the initiator notifies the iSCSI Layer of the availability of the SCSI Response PDU. Ko et al. Expires June 2010 [Page 20] Internet-Draft iSER Specification December 2009 2.7 iSCSI/iSER Layering iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer and the RCaP layer. Note that the RCaP layer may be composed of one or more distinct protocol layers depending on the specifics of the RCaP. Figure 1 shows an example of the relationship between SCSI, iSCSI, iSER, and the different RCaP layers. For TCP, the RCaP is iWARP. For Infiniband, the RCaP is the Reliable Connected Transport Service. Note that the iSCSI layer as described here supports the RDMA Extensions as used in iSER. +-------------------------------------+ | SCSI | +-------------------------------------+ | iSCSI | DI ------> +-------------------------------------+ | iSER | +---------+--------------+------------+ | RDMAP | | | +---------+ Infiniband | | | DDP | Reliable | Other | +---------+ Connected | RDMA- | | MPA | Transport | Capable | +---------+ Service | Protocol | | TCP | | | +---------+--------------+------------+ | | Infiniband | Other | | IP | Network | Network | | | Layer | Layer | +---------+--------------+------------+ Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase Ko et al. Expires June 2010 [Page 21] Internet-Draft iSER Specification December 2009 3 Upper Layer Interface Requirements This section discusses the upper layer interface requirements in the form of an abstract model of the required interactions between the iSCSI Layer and the iSER Layer. The abstract model used here is derived from the architectural model described in [DA]. [DA] also provides a functional overview of the interactions between the iSCSI Layer and the datamover layer as intended by the Datamover Architecture. The interface requirements are specified by Operational Primitives. An Operational Primitive is an abstract functional interface procedure between the iSCSI Layer and the iSER Layer that requests one layer to perform a specific action on behalf of the other layer or notifies the other layer of some event. Whenever an Operational Primitive in invoked, the Connection_Handle qualifier is used to identify a particular iSCSI connection. For some Operational Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data buffer associated with the requested or completed operation. The abstract model and the Operational Primitives defined in this section facilitate the description of the iSER protocol. In the rest of the iSER specification, the compliance statements related to the use of these Operational Primitives are only for the purpose of the required interactions between the iSCSI Layer and the iSER Layer. Note that the compliance statements related to the Operational Primitives in the rest of this specification only mandate functional equivalence on implementations, but do not put any requirements on the implementation specifics of the interface between the iSCSI Layer and the iSER Layer. Each Operational Primitive is invoked with a set of qualifiers which specify the information context for performing the specific action being requested of the Operational Primitive. While the qualifiers are required, the method of realizing the qualifiers (e.g., by passing synchronously with invocation, or by retrieving from task context, or by retrieving from shared memory, etc.) is implementation dependent. 3.1 Operational Primitives offered by iSER The iSER protocol layer MUST support the following Operational Primitives to be used by the iSCSI protocol layer. Ko et al. Expires June 2010 [Page 22] Internet-Draft iSER Specification December 2009 3.1.1 Send_Control Input qualifiers: Connection_Handle, BHS and AHS (if any) of the iSCSI PDU, PDU-specific qualifiers Return results: Not specified This is used by the iSCSI Layers at the initiator and the target to request the outbound transfer of an iSCSI control-type PDU (see section 7.2). Qualifiers that only apply for a particular control- type PDU are known as PDU-specific qualifiers, e.g., ImmediateDataSize for a SCSI Write command. For details on PDU- specific qualifiers, see section 7.3. The iSCSI Layer can only invoke the Send_Control Operational Primitive when the connection is in iSER-assisted mode. 3.1.2 Put_Data Input qualifiers: Connection_Handle, content of a SCSI Data-in PDU header, Data_Descriptor, Notify_Enable Return results: Not specified This is used by the iSCSI Layer at the target to request the outbound transfer of data for a SCSI Data-in PDU from the buffer identified by the Data_Descriptor qualifier. The iSCSI Layer can only invoke the Put_Data Operational Primitive when the connection is in iSER-assisted mode. The Notify_Enable qualifier is used to indicate to the iSER Layer whether or not it should generate an eventual local completion notification to the iSCSI Layer. See section 3.2.2 on Data_Completion_Notify for details. 3.1.3 Get_Data Input qualifiers: Connection_Handle, content of an R2T PDU, Data_Descriptor, Notify_Enable Return results: Not specified This is used by the iSCSI Layer at the target to request the inbound transfer of solicited data requested by an R2T PDU into the buffer identified by the Data_Descriptor qualifier. The iSCSI Layer can only invoke the Get_Data Operational Primitive when the connection is in iSER-assisted mode. Ko et al. Expires June 2010 [Page 23] Internet-Draft iSER Specification December 2009 The Notify_Enable qualifier is used to indicate to the iSER Layer whether or not it should generate the eventual local completion notification to the iSCSI Layer. See section 3.2.2 on Data_Completion_Notify for details. 3.1.4 Allocate_Connection_Resources Input qualifiers: Connection_Handle, Resource_Descriptor (optional) Return results: Status This is used by the iSCSI Layers at the initiator and the target to request the allocation of all connection resources necessary to support RCaP for an operational iSCSI/iSER connection. The iSCSI Layer may optionally specify the implementation-specific resource requirements for the iSCSI connection using the Resource_Descriptor qualifier. A return result of Status=success means the invocation succeeded, and a return result of Status=failure means that the invocation failed. If the invocation is for a Connection_Handle for which an earlier invocation succeeded, the request will be ignored by the iSER Layer and the result of Status=success will be returned. Only one Allocate_Connection_Resources Operational Primitive invocation can be outstanding for a given Connection_Handle at any time. 3.1.5 Deallocate_Connection_Resources Input qualifiers: Connection_Handle Return results: Not specified This is used by the iSCSI Layers at the initiator and the target to request the deallocation of all connection resources that were allocated earlier as a result of a successful invocation of the Allocate_Connection_Resources Operational Primitive. 3.1.6 Enable_Datamover Input qualifiers: Connection_Handle, Transport_Connection_Descriptor, Final Login_Response_PDU (optional) Return results: Not specified Ko et al. Expires June 2010 [Page 24] Internet-Draft iSER Specification December 2009 This is used by the iSCSI Layers at the initiator and the target to request that iSER-assisted mode be used for the connection. The Transport_Connection_Descriptor qualifier is used to identify the specific connection associated with the Connection_Handle. The iSCSI layer can only invoke the Enable_Datamover Operational Primitive when there was a corresponding prior resource allocation. The Final_Login_Response_PDU input qualifier is applicable only for a target, and contains the final Login Response PDU that concludes the iSCSI Login Phase. 3.1.7 Connection_Terminate Input qualifiers: Connection_Handle Return results: Not specified This is used by the iSCSI Layers at the initiator and the target to request that a specified iSCSI/iSER connection be terminated and all associated connection and task resources be freed. When this Operational Primitive invocation returns to the iSCSI layer, the iSCSI layer may assume full ownership of all iSCSI-level resources, e.g. I/O Buffers, associated with the connection. 3.1.8 Notice_Key_Values Input qualifiers: Connection_Handle, number of keys, list of Key-Value pairs Return results: Not specified This is used by the iSCSI Layers at the initiator and the target to request the iSER Layer to take note of the specified Key-Value pairs which were negotiated by the iSCSI peers for the connection. 3.1.9 Deallocate_Task_Resources Input qualifiers: Connection_Handle, ITT Return results: Not specified This is used by the iSCSI Layers at the initiator and the target to request the deallocation of all RCaP-specific resources allocated by the iSER Layer for the task identified by the ITT qualifier. The iSER Layer may require a certain number of RCaP-specific resources associated with the ITT for each new iSCSI task. In the normal course of execution, these task-level resources in the iSER Layer Ko et al. Expires June 2010 [Page 25] Internet-Draft iSER Specification December 2009 are assumed to be transparently allocated on each task initiation and deallocated on the conclusion of each task as appropriate. In exception scenarios where the task does not conclude with a SCSI Response PDU, the iSER Layer needs to be notified of the individual task terminations to aid its task-level resource management. This Operational Primitive is used for this purpose, and is not needed when a SCSI Response PDU normally concludes a task. Note that RCaP- specific task resources are deallocated by the iSER Layer when a SCSI Response PDU normally concludes a task, even if the SCSI Status was not success. 3.2 Operational Primitives used by iSER The iSER layer MUST use the following Operational Primitives offered by the iSCSI protocol layer when the connection is in iSER-assisted mode. 3.2.1 Control_Notify Input qualifiers: Connection_Handle, an iSCSI control-type PDU Return results: Not specified This is used by the iSER Layers at the initiator and the target to notify the iSCSI Layer of the availability of an inbound iSCSI control-type PDU. A PDU is described as "available" to the iSCSI Layer when the iSER Layer notifies the iSCSI Layer of the reception of that inbound PDU, along with an implementation-specific indication as to where the received PDU is. 3.2.2 Data_Completion_Notify Input qualifiers: Connection_Handle, ITT, SN Return results: Not specified This is used by the iSER Layer to notify the iSCSI Layer of the completion of outbound data transfer that was requested by the iSCSI Layer only if the invocation of the Put_Data Operational Primitive (see section 3.1.2) was qualified with Notify_Enable set. SN refers to the DataSN associated with the SCSI Data-In PDU. This is used by the iSER Layer to notify the iSCSI Layer of the completion of inbound data transfer that was requested by the iSCSI Layer only if the invocation of the Get_Data Operational Primitive (see section 3.1.3) was qualified with Notify_Enable set. SN refers to the R2TSN associated with the R2T PDU. Ko et al. Expires June 2010 [Page 26] Internet-Draft iSER Specification December 2009 3.2.3 Data_ACK_Notify Input qualifier: Connection_Handle, ITT, DataSN Return results: Not specified This is used by the iSER Layer at the target to notify the iSCSI Layer of the arrival of the data acknowledgement (as defined in [RFC3720]) requested earlier by the iSCSI Layer for the outbound data transfer via an invocation of the Put_Data Operational Primitive where the A-bit in the SCSI Data-in PDU is set to 1. See section 7.3.5. DataSN refers to the expected DataSN of the next SCSI Data-in PDU which immediately follows the SCSI Data-in PDU with the A-bit set to which this notification corresponds, with semantics as defined in [RFC3720]. 3.2.4 Connection_Terminate_Notify Input qualifiers: Connection_Handle Return results: Not specified This is used by the iSER Layers at the initiator and the target to notify the iSCSI Layer of the unsolicited termination or failure of an iSCSI/iSER connection. The iSER Layer MUST deallocate the connection and task resources associated with the terminated connection before the invocation of this Operational Primitive. Note that the Connection_Terminate_Notify Operational Primitive is not invoked when the termination of the connection was earlier requested by the local iSCSI Layer. 3.3 iSCSI Protocol Usage Requirements To operate in an iSER-assisted mode, the iSCSI Layers at both the initiator and the target MUST negotiate the RDMAExtensions key (see section 6.3) to "Yes" on the leading connection. If the RDMAExtensions key is not negotiated to "Yes", then iSER-assisted mode MUST NOT be used. If the RDMAExtensons key is negotiated to "Yes" but the invocation of the Allocate_Connection_Resources Operational Primitive to the iSER layer fails, the iSCSI layer MUST fail the iSCSI Login process or terminate the connection as appropriate. See section 10.1.3.1 for details. If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer MUST satisfy the following protocol usage requirements from the iSER protocol: Ko et al. Expires June 2010 [Page 27] Internet-Draft iSER Specification December 2009 1. The iSCSI Layer at the initiator MUST set ExpDataSN to 0 in Task Management Function Requests for Task Allegiance Reassignment for read/bidirectional commands, so as to cause the target to send all unacknowledged read data. 2. The iSCSI Layer at the target MUST always return the SCSI status in a separate SCSI Response PDU for read commands, i.e., there MUST NOT be a "phase collapse" in concluding a SCSI Read Command. 3. The iSCSI Layers at both the initiator and the target MUST support the keys as defined in section 6 on Login/Text Operational Keys. If used as specified, these keys MUST NOT be answered with NotUnderstood and the semantics as defined MUST be followed for each iSER-assisted connection. 4. The iSCSI Layer at the initiator MUST NOT issue SNACKs for PDUs. Ko et al. Expires June 2010 [Page 28] Internet-Draft iSER Specification December 2009 4 Lower Layer Interface Requirements 4.1 Interactions with the RCaP Layer The iSER protocol layer is layered on top of an RCaP layer (see Figure 1) and the following are the key features that are assumed to be supported by any RCaP layer: * The RCaP layer supports all basic RDMA operations, including RDMA Write Operation, RDMA Read Operation, Send Operation, Send with Invalidate Operation, Send with Solicited Event Operation, Send with Solicited Event & Invalidate Operation, and Terminate Operation. * The RCaP layer provides reliable, in-order message delivery and direct data placement. * When the iSER Layer initiates an RDMA Read Operation following an RDMA Write Operation on one RCaP Stream, the RDMA Read Response Message processing on the remote node will be started only after the preceding RDMA Write Message payload is placed in the memory of the remote node. * The RCaP layer encapsulates a single iSER Message into a single RCaP Message on the Data Source side. The RCaP layer decapsulates the iSER Message before delivering it to the iSER Layer on the Data Sink side. * When the iSER Layer provides the STag to be remotely invalidated to the RCaP layer for a SendInvSE Message, the RCaP layer uses this STag as the STag to be invalidated in the SendInvSE Message. * The RCaP layer uses the STag and Tagged Offset provided by the iSER Layer for the RDMA Write and RDMA Read Request Messages. * When the RCaP layer delivers the content of an RDMA Send Message Type to the iSER Layer, the RCaP layer provides the length of the RDMA Send message. This ensures that the iSER Layer does not have to carry a length field in the iSER header. * When the RCaP layer delivers the SendSE or SendInvSE Message to the iSER Layer, it notifies the iSER Layer with the mechanism provided on that interface. * When the RCaP layer delivers a SendInvSE Message to the iSER Layer, it passes the value of the STag that was invalidated. Ko et al. Expires June 2010 [Page 29] Internet-Draft iSER Specification December 2009 * The RCaP layer propagates all status and error indications to the iSER Layer. * For a transport layer that operates in byte stream mode such as TCP, the RCaP implementation supports the enabling of the RDMA mode after Connection establishment and the exchange of Login parameters in byte stream mode. For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode. * Whenever the iSER Layer terminates the RCaP Stream, the RCaP layer terminates the associated Connection. 4.2 Interactions with the Transport Layer After the iSER connection is established, the RCaP layer and the underlying transport layer are responsible for maintaining the Connection and reporting to the iSER Layer any Connection failures. Ko et al. Expires June 2010 [Page 30] Internet-Draft iSER Specification December 2009 5 Connection Setup and Termination 5.1 iSCSI/iSER Connection Setup During connection setup, the iSCSI Layer at the initiator is responsible for establishing a connection with the target. After the connection is established, the iSCSI Layers at the initiator and the target enter the Login Phase using the same rules as outlined in [RFC3720]. The connection transitions into the iSCSI full feature phase in iSER-assisted mode following a successful login negotiation between the initiator and the target in which iSER-assisted mode is negotiated and the connection resources necessary to support RCaP have been allocated at both the initiator and the target. The same connection MUST be used for both the iSCSI Login phase and the subsequent iSER-assisted full feature phase. iSER-assisted mode MUST be enabled only if it is negotiated on the leading connection during the LoginOperationalNegotiation Stage of the iSCSI Login Phase. iSER-assisted mode is negotiated using the RDMAExtensions= key. Both the initiator and the target MUST exchange the RDMAExtensions key with the value set to "Yes" to enable iSER-assisted mode. If both the initiator and the target fail to negotiate the RDMAExtensions key set to "Yes", then the connection MUST continue with the login semantics as defined in [RFC3720]. If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode. (For InfiniBand this will require an [IPoIB] type connection.) iSER-assisted mode is defined for a Normal session only and the RDMAExtensions key MUST NOT be negotiated for a Discovery session. Discovery sessions are always conducted using the transport layer as described in [RFC3720]. An iSER enabled node is not required to initiate the RDMAExtensions key exchange if its preference is for the Traditional iSCSI mode. The RDMAExtensions key, if offered, MUST be sent in the first available Login Response or Login Request PDU in the LoginOperationalNegotiation stage. This is due to the fact that the value of some login parameters might depend on whether iSER-assisted mode is enabled or not. iSER-assisted mode is a session-wide attribute. If both the initiator and the target negotiated RDMAExtensions="Yes" on the leading connection of a session, then all subsequent connections of the same session MUST enable iSER-assisted mode without having to exchange RDMAExtensions key during the iSCSI Login Phase. Ko et al. Expires June 2010 [Page 31] Internet-Draft iSER Specification December 2009 Conversely, if both the initiator and the target failed to negotiate RDMAExtensions to "Yes" on the leading connection of a session, then the RDMAExtensions key MUST NOT be negotiated further on any additional subsequent connection of the session. When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest and the DataDigest keys MUST be negotiated to "None" on all iSCSI/iSER connections participating in that iSCSI session. This is because, for an iSCSI/iSER connection, RCaP is responsible for providing error detection that is at least as good as a 32-bit CRC for all iSER Messages. Furthermore, all SCSI Read data are sent using RDMA Write Messages instead of the SCSI Data-in PDUs, and all solicited SCSI write data are sent using RDMA Read Response Messages instead of the SCSI Data-out PDUs. HeaderDigest and DataDigest which apply to iSCSI PDUs would not be appropriate for RDMA Read and RDMA Write operations used with iSER. 5.1.1 Initiator Behavior If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the initiator side, prior to sending the Login Request with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to allocate the connection resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive. The connection resources required are defined by implementation and are outside the scope of this specification. The iSCSI Layer may invoke the Notice_Key_Values Operational Primitive before invoking the Allocate_Connection_Resources Operational Primitive to request the iSER Layer to take note of the negotiated values of the iSCSI keys for the Connection. The specific keys to be passed in as input qualifiers are implementation dependent. These may include, but not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. To minimize the potential for a denial of service attack, the iSCSI Layer MUST NOT request the iSER Layer to allocate the connection resources necessary to support RCaP until the iSCSI layer is sufficiently far along in the iSCSI Login Phase that it is reasonably certain that the peer side is not an attacker. In particular, if the Login Phase includes a SecurityNegotiation stage, the iSCSI Layer MUST defer the connection resource allocation (i.e. invoking the Allocate_Connection_Resources Operational Primitive) to the LoginOperationalNegotiation stage ([RFC3720]) so that the resource allocation occurs after the authentication phase is completed. Ko et al. Expires June 2010 [Page 32] Internet-Draft iSER Specification December 2009 Among the connection resources allocated at the initiator is the Inbound RDMA Read Queue Depth (IRD). As described in section 9.5.1, R2Ts are transformed by the target into RDMA Read operations. IRD limits the maximum number of simultaneously incoming outstanding RDMA Read Requests per an RCaP Stream from the target to the initiator. The required value of IRD is outside the scope of the iSER specification. The iSER Layer at the initiator MUST set IRD to 1 or higher if R2Ts are to be used in the connection. However, the iSER Layer at the initiator MAY set IRD to 0 based on implementation configuration which indicates that no R2Ts will be used on that connection. Initially, the iSER-IRD value at the initiator SHOULD be set to the IRD value at the initiator and MUST NOT be more than the IRD value. On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be set to 0 since the iSER Layer at the initiator does not issue RDMA Read Requests to the target. Failure to allocate the requested connection resources locally results in a login failure and its handling is described in section 10.1.3.1. If the iSER Layer at the initiator is successful in allocating the connection resources necessary to support RCaP, the following events MUST occur in the specified sequence: 1. The iSER Layer MUST return a success status to the iSCSI Layer in response to the Allocate_Connection_Resources Operational Primitive. 2. After the target returns the Login Response with the T bit set to 1 and the NSG field set to FullFeaturePhase, and a status class of 0 (Success), the iSCSI Layer MUST invoke the Enable_Datamover Operational Primitive with the following qualifiers. (See section 10.1.4.6 for the case when the status class is not Success.): a. Connection_Handle that identifies the iSCSI connection. b. Transport_Connection_Descriptor which identifies the specific transport connection associated with the Connection_Handle. 3. The iSER Layer MUST send the iSER Hello Message as the first iSER Message. See Section 5.1.3 on iSER Hello Exchange. Ko et al. Expires June 2010 [Page 33] Internet-Draft iSER Specification December 2009 5.1.2 Target Behavior If the outcome of the iSCSI negotiation is to enable iSER-assisted mode, then on the target side, prior to sending the Login Response with the T (Transit) bit set to 1 and the NSG (Next Stage) field set to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to allocate the resources necessary to support RCaP by invoking the Allocate_Connection_Resources Operational Primitive. The connection resources required are defined by implementation and are outside the scope of this specification. Optionally, the iSCSI Layer may invoke the Notice_Key_Values Operational Primitive before invoking the Allocate_Connection_Resources Operational Primitive to request the iSER Layer to take note of the negotiated values of the iSCSI keys for the Connection. The specific keys to be passed in as input qualifiers are implementation dependent. These may include, but not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc. To minimize the potential for a denial of service attack, the iSCSI Layer MUST NOT request the iSER Layer to allocate the connection resources necessary to support RCaP until the iSCSI layer is sufficiently far along in the iSCSI Login Phase that it is reasonably certain that the peer side is not an attacker. In particular, if the Login Phase includes a SecurityNegotiation stage, the iSCSI Layer MUST defer the connection resource allocation (i.e. invoking the Allocate_Connection_Resources Operational Primitive) to the LoginOperationalNegotiation stage ([RFC3720]) so that the resource allocation occurs after the authentication phase is completed. Among the connection resources allocated at the target is the Outbound RDMA Read Queue Depth (ORD). As described in section 9.5.1, R2Ts are transformed by the target into RDMA Read operations. The ORD limits the maximum number of simultaneously outstanding RDMA Read Requests per RCaP Stream from the target to the initiator. Initially, the iSER-ORD value at the target SHOULD be set to the ORD value at the target. On the other hand, the IRD at the target MAY be set to 0 since the iSER Layer at the target does not expect RDMA Read Requests to be issued by the initiator. Failure to allocate the requested connection resources locally results in a login failure and its handling is described in section 10.1.3.1. Ko et al. Expires June 2010 [Page 34] Internet-Draft iSER Specification December 2009 If the iSER Layer at the target is successful in allocating the connection resources necessary to support RCaP, the following events MUST occur in the specified sequence: 1. The iSER Layer MUST return a success status to the iSCSI Layer in response to the Allocate_Connection_Resources Operational Primitive. 2. The iSCSI Layer MUST invoke the Enable_Datamover Operational Primitive with the following qualifiers: a. Connection_Handle that identifies the iSCSI connection. b. Transport_Connection_Descriptor which identifies the specific transport connection associated with the Connection_Handle. c. The final transport layer (e.g. TCP) message containing the Login Response with the T bit set to 1 and the NSG field set to FullFeaturePhase 3. The iSER Layer MUST send the final Login Response PDU in the native transport mode to conclude the iSCSI Login Phase. If the underlying transport is TCP, then the iSER Layer MUST send the final Login Response PDU in byte stream mode. 4. After receiving the iSER Hello Message from the initiator, the iSER Layer MUST respond with the iSER HelloReply Message to be sent as the first iSER Message. See section 5.1.3 on iSER Hello Exchange for more details. Note: In the above sequence, the operations as described in bullets 3 and 4 MUST be performed atomically for iWARP connections. Failure to do this may result in race conditions. 5.1.3 iSER Hello Exchange The first iSER Message sent by the iSER Layer at the initiator to the target MUST be the iSER Hello Message. The iSER Hello Message is used by the iSER Layer at the initiator to declare iSER parameters to the target. See section 9.3 on iSER Header Format for iSER Hello Message. In response to the iSER Hello Message, the iSER Layer at the target MUST return the iSER HelloReply Message as the first iSER Message sent by the target. The iSER HelloReply Message is used by the iSER Ko et al. Expires June 2010 [Page 35] Internet-Draft iSER Specification December 2009 Layer at the target to declare iSER parameters to the initiator. See section 9.4 on iSER Header Format for iSER HelloReply Message. In the iSER Hello Message, the iSER Layer at the initiator declares the iSER-IRD value to the target. Upon receiving the iSER Hello Message, the iSER Layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator. The iSER Layer at the target MAY adjust (lower) its ORD value to match the iSER-ORD value if the iSER-ORD value is smaller than the ORD value at the target in order to free up the unused resources. In the iSER HelloReply Message, the iSER Layer at the target declares the iSER-ORD value to the initiator. Upon receiving the iSER HelloReply Message, the iSER Layer at the initiator MAY adjust (lower) its IRD value to match the iSER-ORD value in order to free up the unused resources, if the iSER-ORD value declared by the target is smaller than the iSER-IRD value declared by the initiator. It is an iSER level negotiation failure if the iSER parameters declared in the iSER Hello Message by the initiator are unacceptable to the target. This includes the following: * The initiator-declared iSER-IRD value is greater than 0 and the target-declared iSER-ORD value is 0. * The initiator-supported and the target-supported iSER protocol versions do not overlap. See section 10.1.3.2 on the handling of the error situation. 5.2 iSCSI/iSER Connection Termination 5.2.1 Normal Connection Termination at the Initiator The iSCSI Layer at the initiator terminates an iSCSI/iSER connection normally by invoking the Send_Control Operational Primitive qualified with the Logout Request PDU. The iSER Layer at the initiator MUST use a SendSE Message to send the Logout Request PDU to the target. After the iSER Layer at the initiator receives the SendSE Message containing the Logout Response PDU from the target, it MUST notify the iSCSI Layer by invoking the Control_Notify Operational Primitive qualified with the Logout Response PDU. Ko et al. Expires June 2010 [Page 36] Internet-Draft iSER Specification December 2009 After the iSCSI logout process is complete, the iSCSI layer at the target is responsible for closing the iSCSI/iSER connection as described in Section 5.2.2. After the RCaP layer at the initiator reports that the Connection has been closed, the iSER Layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, invalidate the Local Mapping(s) (if any) that associate the ITT(s) used on that connection to the local STag(s) before notifying the iSCSI Layer by invoking the Connection_Terminate_Notify Operational Primitive. 5.2.2 Normal Connection Termination at the Target Upon receiving the SendSE Message containing the Logout Request PDU, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Control_Notify Operational Primitive qualified with the Logout Request PDU. The iSCSI Layer completes the logout process by invoking the Send_Control Operational Primitive qualified with the Logout Response PDU. The iSER Layer at the target MUST use a SendSE Message to send the Logout Response PDU to the initiator. After the iSCSI logout process is complete, the iSCSI Layer at the target MUST request the iSER Layer at the target to terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. As part of the termination process, the RCaP layer MUST close the Connection. When the RCaP layer notifies the iSER Layer after the RCaP Stream and the associated Connection are terminated, the iSER Layer MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mapping(s) (if any) that associate the ITT(s) used on that connection to the local STag(s) and the Advertised STag(s) respectively. 5.2.3 Termination without Logout Request/Response PDUs 5.2.3.1 Connection Termination Initiated by the iSCSI Layer The Connection_Terminate Operational Primitive MAY be invoked by the iSCSI Layer to request the iSER Layer to terminate the RCaP Stream without having previously exchanged the Logout Request and Logout Response PDUs between the two iSCSI/iSER nodes. As part of the termination process, the RCaP layer will close the Connection. When the RCaP layer notifies the iSER Layer after the RCaP Stream and the associated Connection are terminated, the iSER Layer MUST perform the following actions. Ko et al. Expires June 2010 [Page 37] Internet-Draft iSER Specification December 2009 If the Connection_Terminate Operational Primitive is invoked by the iSCSI Layer at the target, then the iSER Layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s) respectively. If the Connection_Terminate Operational Primitive is invoked by the iSCSI Layer at the initiator, then the iSER Layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mapping(s) (if any) that associate the ITT(s) used on the connection to the local STag(s). 5.2.3.2 Connection Termination Notification to the iSCSI Layer If the iSCSI/iSER connection is terminated without the invocation of Connection_Terminate from the iSCSI Layer, the iSER Layer MUST notify the iSCSI Layer that the iSCSI/iSER connection has been terminated by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking Connection_Terminate_Notify, the iSER Layer at the target MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local and Remote Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s) and the Advertised STag(s) respectively. Prior to invoking Connection_Terminate_Notify, the iSER Layer at the initiator MUST deallocate all connection and task resources (if any) associated with the connection, and invalidate the Local Mappings (if any) that associate the ITT(s) used on the connection to the local STag(s). If the remote iSCSI/iSER node initiated the closing of the Connection (e.g., by sending a TCP FIN or TCP RST), the iSER Layer MUST notify the iSCSI Layer after the RCaP layer reports that the Connection is closed by invoking the Connection_Terminate_Notify Operational Primitive. Another example of a Connection termination without a preceding logout is when the iSCSI Layer at the initiator does an implicit logout (connection reinstatement). Ko et al. Expires June 2010 [Page 38] Internet-Draft iSER Specification December 2009 6 Login/Text Operational Keys Certain iSCSI login/text operational keys have restricted usage in iSER, and additional keys are used to support the iSER protocol functionality. All other keys defined in [RFC3720] and not discussed in this section may be used on iSCSI/iSER connections with the same semantics. 6.1 HeaderDigest and DataDigest Irrelevant when: RDMAExtensions=Yes Negotiations resulting in RDMAExtensions=Yes for a session implies HeaderDigest=None and DataDigest=None for all connections in that session and overrides both the default and an explicit setting. 6.2 MaxRecvDataSegmentLength For an iSCSI connection belonging to a session in which RDMAExtensions=Yes was negotiated on the leading connection of the session, MaxRecvDataSegmentLength need not be declared in the Login Phase. Instead InitiatorRecvDataSegmentLength (as described in section 6.5) and TargetRecvDataSegmentLength (as described in section 6.4) keys are negotiated. The values of the local and remote MaxRecvDataSegmentLength are derived from the InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys even if the MaxRecvDataSegmentLength was declared during the login phase. In the full feature phase, the initiator MUST consider the value of its local MaxRecvDataSegmentLength (that it would have declared to the target) as having the value of InitiatorRecvDataSegmentLength, and the value of the remote MaxRecvDataSegmentLength (that would have been declared by the target) as having the value of TargetRecvDataSegmentLength. Similarly, the target MUST consider the value of its local MaxRecvDataSegmentLength (that it would have declared to the initiator) as having the value of TargetRecvDataSegmentLength, and the value of the remote MaxRecvDataSegmentLength (that would have been declared by the initiator) as having the value of InitiatorRecvDataSegmentLength. Note that RFC 3720 requires that when a target receives a NOP-Out request with a valid Initiator Task Tag, it responds with a NOP-In with the same Initiator Task Tag that was provided in the NOP-Out request. Furthermore, it returns the first MaxRecvDataSegmentLength bytes of the initiator provided Ping Data. Since there is no MaxRecvDataSegmentLength common to the initiator and the target in Ko et al. Expires June 2010 [Page 39] Internet-Draft iSER Specification December 2009 iSER, the length of the data sent with the NOP-Out request MUST not exceed InitiatorMaxRecvDataSegmentLength. The MaxRecvDataSegmentLength key is applicable only for iSCSI control-type PDUs. 6.3 RDMAExtensions Use: LO (leading only) Senders: Initiator and Target Scope: SW (session-wide) RDMAExtensions= Irrelevant when: SessionType=Discovery Default is No Result function is AND This key is used by the initiator and the target to negotiate the support for iSER-assisted mode. To enable the use of iSER-assisted mode, both the initiator and the target MUST exchange RDMAExtensions=Yes. iSER-assisted mode MUST NOT be used if either the initiator or the target offers RDMAExtensions=No. An iSER-enabled node is not required to initiate the RDMAExtensions key exchange if it prefers to operate in the Traditional iSCSI mode. However, if the RDMAExtensions key is to be negotiated, an initiator MUST offer the key in the first Login Request PDU in the LoginOperationalNegotiation stage of the leading connection, and a target MUST offer the key in the first Login Response PDU with which it is allowed to do so (i.e., the first Login Response PDU issued after the first Login Request PDU with the C bit set to 0) in the LoginOperationalNegotiation stage of the leading connection. In response to the offered key=value pair of RDMAExtensions=yes, an initiator MUST respond in the next Login Request PDU with which it is allowed to do so, and a target MUST respond in the next Login Response PDU with which it is allowed to do so. Negotiating the RDMAExtensions key first enables a node to negotiate the optimal value for other keys. Certain iSCSI keys such as MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T, ImmediateData, etc., may be negotiated differently depending on Ko et al. Expires June 2010 [Page 40] Internet-Draft iSER Specification December 2009 whether connection is in Traditional iSCSI mode or iSER-assisted mode. 6.4 TargetRecvDataSegmentLength Use: IO (Initialize only) Senders: Initiator and Target Scope: CO (connection-only) Irrelevant when: RDMAExtensions=No TargetRecvDataSegmentLength= Default is 8192 bytes Result function is minimum This key is relevant only for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection of the session. It is used by the initiator and the target to negotiate the maximum size of the data segment that an initiator may send to the target in an iSCSI control-type PDU in the full feature phase. For SCSI Command PDUs and SCSI Data-out PDUs containing non-immediate unsolicited data to be sent by the initiator, the initiator MUST send all non-Final PDUs with a data segment size of exactly TargetRecvDataSegmentLength whenever the PDUs constitute a data sequence whose size is larger than TargetRecvDataSegmentLength. 6.5 InitiatorRecvDataSegmentLength Use: IO (Initialize only) Senders: Initiator and Target Scope: CO (connection-only) Irrelevant when: RDMAExtensions=No InitiatorRecvDataSegmentLength= Default is 8192 bytes Result function is minimum Ko et al. Expires June 2010 [Page 41] Internet-Draft iSER Specification December 2009 This key is relevant only for the iSCSI connection of an iSCSI session if RDMAExtensions=Yes was negotiated on the leading connection of the session. It is used by the initiator and the target to negotiate the maximum size of the data segment that a target may send to the initiator in an iSCSI control-type PDU in the full feature phase. 6.6 OFMarker and IFMarker Irrelevant when: RDMAExtensions=Yes Negotiations resulting in RDMAExtensions=Yes for a session implies OFMarker=No and IFMarker=No for all connections in that session and overrides both the default and an explicit setting. 6.7 MaxOutstandingUnexpectedPDUs Use: LO (leading only), Declarative Senders: Initiator and Target Scope: SW (session-wide) Irrelevant when: RDMAExtensions=No MaxOutstandingUnexpectedPDUs= Default is 0 This key is used by the initiator and the target to declare the maximum number of outstanding "unexpected" iSCSI control-type PDUs that it can receive in the full feature phase. It is intended to allow the receiving side to determine the amount of buffer resources needed beyond the normal flow control mechanism available in iSCSI. An initiator or target should select a value such that it would not impose an unnecessary constraint on the iSCSI Layer under normal circumstances. The value of 0 is defined to indicate that the declarer has no limit on the maximum number of outstanding "unexpected" iSCSI control-type PDUs that it can receive. See sections 8.1.1 and 8.1.2 for the usage of this key. Note that iSER Hello and HelloReply Messages are not iSCSI control-type PDUs and are not affected by this key. 6.8 MaxAHSLength Use: LO (leading only), Declarative Ko et al. Expires June 2010 [Page 42] Internet-Draft iSER Specification December 2009 Senders: Initiator and Target Scope: SW (session-wide) Irrelevant when: RDMAExtensions=No MaxAHSLength= Default is 0 This key is used by the intiator and target to declare the maximum size of AHS in an iSCSI control-type PDU that it can receive in the full feature phase. It is intended to allow the receiving side to determine the amount of resources needed for receive buffering. An initiator or target should select a value such that it would not impose an unnecessary constraint on the iSCSI Layer under normal circumstances. The value of 0 is defined to indicate that the declarer has no limit on the maximum size of AHS in iSCSI control- type PDUs that it can receive. 6.9 WriteAddressForSolicitedDataOnly Use: LO (leading only), Declarative Senders: Initiator Scope: SW (session-wide) RDMAExtensions= Irrelevant when: RDMAExtensions=No Default is No This key is used by the intiator to declare to the target the origin of the Write Virtual Address used in the iSER header of an iSCSI control-type PDU. When set to No, the Write Virtual Address points to an I/O buffer that contains all the write data, including both the unsolicited and solicited data. When set to Yes, the Write Virtual Address points to an I/O buffer that only contains the solicited data. Ko et al. Expires June 2010 [Page 43] Internet-Draft iSER Specification December 2009 7 iSCSI PDU Considerations When a connection is in the iSER-assisted mode, two types of message transfers are allowed between the iSCSI Layer at the initiator and the iSCSI Layer at the target. These are known as the iSCSI data- type PDUs and the iSCSI control-type PDUs and these terms are described in the following sections. 7.1 iSCSI Data-Type PDU An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI layer, to take place between the peer iSCSI nodes in the full feature phase of an iSCSI/iSER connection. An iSCSI data-type PDU, when requested for transmission by the iSCSI Layer in the sending node, results in the data being transferred without the participation of the iSCSI Layers at the sending and the receiving nodes. This is due to the fact that the PDU itself is not delivered as-is to the iSCSI Layer in the receiving node. Instead, the data transfer operations are transformed into the appropriate RDMA operations which are handled by the RDMA-Capable Controller. The set of iSCSI data-type PDUs consists of SCSI Data-in PDUs and R2T PDUs. If the invocation of the Operational Primitive by the iSCSI Layer to request the iSER Layer to process an iSCSI data-type PDU is qualified with Notify_Enable set, then upon completing the RDMA operation, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Data_Completion_Notify Operational Primitive qualified with ITT and SN. There is no data completion notification at the initiator since the RDMA operations are completely handled by the RDMA-Capable Controller at the initiator and the iSER Layer at the initiator is not involved with the data transfer associated with iSCSI data-type PDUs. If the invocation of the Operational Primitive by the iSCSI Layer to request the iSER Layer to process an iSCSI data-type PDU is qualified with Notify_Enable cleared, then upon completing the RDMA operation, the iSER Layer at the target MUST NOT notify the iSCSI Layer at the target and MUST NOT invoke the Data_Completion_Notify Operational Primitive. If an operation associated with an iSCSI data-type PDU fails for any reason, the contents of the Data Sink buffers associated with the operation are considered indeterminate. Ko et al. Expires June 2010 [Page 44] Internet-Draft iSER Specification December 2009 7.2 iSCSI Control-Type PDU Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI Data-out PDU carrying solicited data is defined as an iSCSI control- type PDU. The iSCSI Layer invokes the Send_Control Operational Primitive to request the iSER Layer to process an iSCSI control-type PDU. iSCSI control-type PDUs are transferred using Send Message Types of RCaP. Specifically, it is to be noted that SCSI Data-Out PDUs carrying unsolicited data are defined as iSCSI control-type PDUs. See section 7.3.4 on the treatment of SCSI Data-out PDUs. When the iSER Layer receives an iSCSI control-type PDU, it MUST notify the iSCSI Layer by invoking the Control_Notify Operational Primitive qualified with the iSCSI control-type PDU. 7.3 iSCSI PDUs This section describes the handling of each of the iSCSI PDU types by the iSER Layer. The iSCSI Layer requests the iSER Layer to process the iSCSI PDU by invoking the appropriate Operational Primitive. A Connection_Handle MUST qualify each of these invocations. In addition, BHS and the optional AHS of the iSCSI PDU as defined in [RFC3720] MUST qualify each of the invocations. The qualifying Connection_Handle, the BHS and the AHS are not explicitly listed in the subsequent sections. 7.3.1 SCSI Command Type: control-type PDU PDU-specific qualifiers (for SCSI Write or bidirectional command): ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut PDU-specific qualifiers (for SCSI Read or bidirectional command): DataDescriptorIn The iSER Layer at the initiator MUST send the SCSI command in a SendSE Message to the target. For a SCSI Write or bidirectional command, the iSCSI Layer at the initiator MUST invoke the Send_Control Operational Primitive as follows: * If there is immediate data to be transferred for the SCSI write or bidirectional command, the qualifier ImmediateDataSize MUST be used to define the number of bytes of immediate unsolicited data Ko et al. Expires June 2010 [Page 45] Internet-Draft iSER Specification December 2009 to be sent with the write or bidirectional command, and the qualifier DataDescriptorOut MUST be used to define the initiator's I/O Buffer containing the SCSI Write data. * If there is unsolicited data to be transferred for the SCSI Write or bidirectional command, the qualifier UnsolicitedDataSize MUST be used to define the number of bytes of immediate and non- immediate unsolicited data for the command. The iSCSI Layer will issue one or more SCSI Data-out PDUs for the non-immediate unsolicited data. See Section 7.3.4 on SCSI Data-out. * If there is solicited data to be transferred for the SCSI Write or bidirectional command, as indicated by the Expected Data Transfer Length in the SCSI Command PDU exceeding the value of UnsolicitedDataSize, the iSER Layer at the initiator MUST do the following: a. It MUST allocate a Write STag for the I/O Buffer defined by the qualifier DataDescriptorOut. DataDescriptorOut describes the I/O buffer starting with the immediate unsolicited data (if any), followed by the non-immediate unsolicited data (if any) and solicited data. This means that the BufferOffset for the SCSI Data-out for this command is equal to the TO. This implies zero TO for this STag points to the beginning of this I/O Buffer. b. It MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) to the Write STag. c. It MUST Advertise the Write STag to the target by sending it as the Write STag in the iSER header of the iSER Message (the payload of the SendSE Message of RCaP) containing the SCSI Write or bidirectional command PDU. See section 9.2 on iSER Header Format for iSCSI Control-Type PDU. For a SCSI Read or bidirectional command, the iSCSI Layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorIn which defines the initiator's I/O Buffer for receiving the SCSI Read data. The iSER Layer at the initiator MUST do the following: a. It MUST allocate a Read STag for the I/O Buffer. b. It MUST establish a Local Mapping that associates the Initiator Task Tag (ITT) to the Read STag. Ko et al. Expires June 2010 [Page 46] Internet-Draft iSER Specification December 2009 c. It MUST Advertise the Read STag to the target by sending it as the Read STag in the iSER header of the iSER Message (the payload of the SendSE Message of RCaP) containing the SCSI Read or bidirectional command PDU. See section 9.2 on iSER Header Format for iSCSI Control-Type PDU. If the amount of unsolicited data to be transferred in a SCSI Command exceeds TargetRecvDataSegmentLength, then the iSCSI Layer at the initiator MUST segment the data into multiple iSCSI control-type PDUs, with the data segment length in all PDUs generated except the last one having exactly the size TargetRecvDataSegmentLength. The data segment length of the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength. When the iSER Layer at the target receives the SCSI Command, it MUST establish a Remote Mapping that associates the ITT to the Advertised Write STag and the Read STag if present in the iSER header. The Write STag is used by the iSER Layer at the target in handling the data transfer associated with the R2T PDU(s) as described in section 7.3.6. The Read STag is used in handling the SCSI Data-in PDU(s) from the iSCSI Layer at the target as described in section 7.3.5. 7.3.2 SCSI Response Type: control-type PDU PDU-specific qualifiers: DataDescriptorStatus The iSCSI Layer at the target MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorStatus which defines the buffer containing the sense and response information. The iSCSI Layer at the target MUST always return the SCSI status for a SCSI command in a separate SCSI Response PDU. "Phase collapse" for transferring SCSI status in a SCSI Data-in PDU MUST NOT be used. The iSER Layer at the target sends the SCSI Response PDU according to the following rules: * If no STags were Advertised by the initiator in the iSER Message containing the SCSI command PDU, then the iSER Layer at the target MUST send a SendSE Message containing the SCSI Response PDU. * If the initiator Advertised a Read STag in the iSER Message containing the SCSI Command PDU, then the iSER Layer at the target MUST send a SendInvSE Message containing the SCSI Response PDU. The header of the SendInvSE Message MUST carry the Read STag to be invalidated at the initiator. Ko et al. Expires June 2010 [Page 47] Internet-Draft iSER Specification December 2009 * If the initiator Advertised only the Write STag in the iSER Message containing the SCSI command PDU, then the iSER Layer at the target MUST send a SendInvSE Message containing the SCSI Response PDU. The header of the SendInvSE Message MUST carry the Write STag to be invalidated at the initiator. When the iSCSI Layer at the target invokes the Send_Control Operational Primitive to send the SCSI Response PDU, the iSER Layer at the target MUST invalidate the Remote Mapping that associates the ITT to the Advertised STag(s) before transferring the SCSI Response PDU to the initiator. Upon receiving the SendInvSE Message containing the SCSI Response PDU from the target, the RCaP layer at the initiator will invalidate the STag specified in the header. The iSER Layer at the initiator MUST ensure that the correct STag is invalidated. If both the Read and the Write STags were Advertised earlier by the initiator, then the iSER Layer at the initiator MUST explicitly invalidate the Write STag upon receiving the SendInvSE Message because the header of the SendInvSE Message can only carry one STag (in this case the Read STag) to be invalidated. The iSER Layer at the initiator MUST ensure the invalidation of the STag(s) used in a command before notifying the iSCSI Layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response. This precludes the possibility of using the STag(s) after the completion of the command thereby causing data corruption. When the iSER Layer at the initiator receives the SendSE or the SendInvSE Message containing the SCSI Response PDU, it SHOULD invalidate the Local Mapping that associates the ITT to the local STag(s). The iSER Layer MUST ensure that all local STag(s) associated with the ITT are invalidated before notifying the iSCSI Layer of the SCSI Response PDU by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. 7.3.3 Task Management Function Request/Response Type: control-type PDU PDU-specific qualifiers (for TMF Request): DataDescriptorOut, DataDescriptorIn The iSER Layer MUST use a SendSE Message to send the Task Management Function Request/Response PDU. Ko et al. Expires June 2010 [Page 48] Internet-Draft iSER Specification December 2009 For the Task Management Function Request with the TASK REASSIGN function, the iSER Layer at the initiator MUST do the following: * It MUST use the ITT as specified in the Referenced Task Tag from the Task Management Function Request PDU to locate the existing STag(s), if any, in the Local Mapping(s) that associates the ITT to the local STag(s). * It MUST invalidate the existing STag(s), if any, and the Local Mapping(s) that associates the ITT to the local STag(s). * It MUST allocate a Read STag for the I/O Buffer as defined by the qualifier DataDescriptorIn if the Send_Control Operational Primitive invocation is qualified with DataDescriptorIn. * It MUST allocate a Write STag for the I/O Buffer as defined by the qualifier DataDescriptorOut if the Send_Control Operational Primitive invocation is qualified with DataDescriptorOut. * If STags are allocated, it MUST establish new Local Mapping(s) that associate the ITT to the allocated STag(s). * It MUST Advertise the STags, if allocated, to the target in the iSER header of the SendSE Message carrying the iSCSI PDU, as described in section 9.2. For the Task Management Function Request with the TASK REASSIGN function for a SCSI Read or bidirectional command, the iSCSI Layer at the initiator MUST set ExpDataSN to 0 since the data transfer and acknowledgements happen transparently to the iSCSI Layer at the initiator. This provides the flexibility to the iSCSI Layer at the target to request transmission of only the unacknowledged data as specified in [RFC3720]. When the iSER Layer at the target receives the Task Management Function Request with the TASK REASSIGN function, it MUST do the following: * It MUST use the ITT as specified in the Referenced Task Tag from the Task Management Function Request PDU to locate the mappings that associate the ITT to the Advertised STag(s) and the local STag(s), if any. * It MUST invalidate the local STaq(s), if any, associated with the ITT. Ko et al. Expires June 2010 [Page 49] Internet-Draft iSER Specification December 2009 * It MUST replace the Advertised STag(s) in the Remote Mapping that associates the ITT to the Advertised STag(s) with the Write STag and the Read STag if present in the iSER header. The Write STag is used in the handling of the R2T PDU(s) from the iSCSI Layer at the target as described in section 7.3.6. The Read STag is used in the handling of the SCSI Data-in PDU(s) from the iSCSI Layer at the target as described in section 7.3.5. 7.3.4 SCSI Data-out Type: control-type PDU PDU-specific qualifiers: DataDescriptorOut The iSCSI Layer at the initiator MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorOut which defines the initiator's I/O Buffer containing unsolicited SCSI Write data. If the amount of unsolicited data to be transferred as SCSI Data-out exceeds TargetRecvDataSegmentLength, then the iSCSI Layer at the initiator MUST segment the data into multiple iSCSI control-type PDUs, with the DataSegmentLength having the value of TargetRecvDataSegmentLength in all PDUs generated except the last one. The DataSegmentLength of the last iSCSI control-type PDU carrying the unsolicited data can be up to TargetRecvDataSegmentLength. The iSCSI Layer at the target MUST perform the reassembly function for the unsolicited data. For unsolicited data, if the F bit is set to 0 in a SCSI Data-out PDU, the iSER Layer at the initiator MUST use a Send Message to send the SCSI Data-out PDU. If the F bit is set to 1, the iSER Layer at the initiator MUST use a SendSE Message to send the SCSI Data-out PDU. Note that for solicited data, the SCSI Data-out PDUs are not used since R2T PDUs are not delivered to the iSCSI layer at the initiator; instead R2T PDUs are transformed by the iSER layer at the target into RDMA Read operations. (See section 7.3.6.) 7.3.5 SCSI Data-in Type: data-type PDU PDU-specific qualifiers: DataDescriptorIn When the iSCSI Layer at the target is ready to return the SCSI Read data to the initiator, it MUST invoke the Put_Data Operational Ko et al. Expires June 2010 [Page 50] Internet-Draft iSER Specification December 2009 Primitive qualified with DataDescriptorIn which defines the SCSI Data-in buffer. See section 7.1 on the general requirement on the handling of iSCSI data-type PDUs. SCSI Data-in PDU(s) are used in SCSI Read data transfer as described in section 9.5.2. The iSER Layer at the target MUST do the following for each invocation of the Put_Data Operational Primitive: 1. It MUST use the ITT in the SCSI Data-in PDU to locate the remote Read STag in the Remote Mapping that associates the ITT to Advertised STag(s). The Remote Mapping was established earlier by the iSER Layer at the target when the SCSI Read Command was received from the initiator. 2. It MUST generate and send an RDMA Write Message containing the read data to the initiator. a. It MUST use the remote Read STag as the Data Sink STag of the RDMA Write Message. b. It MUST use the Buffer Offset from the SCSI Data-in PDU as the Data Sink Tagged Offset of the RDMA Write Message. c. It MUST use DataSegmentLength from the SCSI Data-in PDU to determine the amount of data to be sent in the RDMA Write Message. 3. It MUST associate DataSN and ITT from the SCSI Data-in PDU with the RDMA Write operation. If the Put_Data Operational Primitive invocation was qualified with Notify_Enable set, then when the iSER Layer at the target receives a completion from the RCaP layer for the RDMA Write Message, the iSER Layer at the target MUST notify the iSCSI Layer by invoking the Data_Completion_Notify Operational Primitive qualified with DataSN and ITT. Conversely, if the Put_Data Operational Primitive invocation was qualified with Notify_Enable cleared, then the iSER Layer at the target MUST NOT notify the iSCSI Layer on completion and MUST NOT invoke the Data_Completion_Notify Operational Primitive. When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER Layer at the target MUST notify the iSCSI Layer at the target when the data transfer is complete at the initiator. To perform this additional function, the iSER Layer at the target can take advantage of the operational ErrorRecoveryLevel if previously disclosed by the iSCSI Layer via an earlier invocation of the Notice_Key_Values Operational Primitive. There are two approaches that can be taken: Ko et al. Expires June 2010 [Page 51] Internet-Draft iSER Specification December 2009 1. If the iSER Layer at the target knows that the operational ErrorRecoveryLevel is 2, or if the iSER Layer at the target does not know the operational ErrorRecoveryLevel, then the iSER Layer at the target MUST issue a zero-length RDMA Read Request Message following the RDMA Write Message. When the iSER Layer at the target receives a completion for the RDMA Read Request Message from the RCaP layer, implying that the RDMA-Capable Controller at the initiator has completed processing the RDMA Write Message due to the completion ordering semantics of RCaP, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Data_Ack_Notify Operational Primitive qualified with ITT and DataSN (see section 3.2.3). 2. If the iSER Layer at the target knows that the operational ErrorRecoveryLevel is 1, then the iSER Layer at the target MUST do one of the following: a. It MUST notify the iSCSI Layer at the target by invoking the Data_Ack_Notify Operational Primitive qualified with ITT and DataSN (see section 3.2.3) when it receives the local completion from the RCaP layer for the RDMA Write Message. This is allowed since digest errors do not occur in iSER (see section 10.1.4.2) and a CRC error will cause the connection to be terminated and the task to be terminated anyway. The local RDMA Write completion from the RCaP layer guarantees that the RCaP layer will not access the I/O Buffer again to transfer the data associated with that RDMA Write operation. b. Alternatively, it MUST use the same procedure for handling the data transfer completion at the initiator as for ErrorRecoveryLevel 2. It should be noted that the iSCSI Layer at the target cannot set the A-bit to 1 if the ErrorRecoveryLevel=0. SCSI status MUST always be returned in a separate SCSI Response PDU. The S bit in the SCSI Data-in PDU MUST always be set to 0. There MUST NOT be a "phase collapse" in the SCSI Data-in PDU. Since the RDMA Write Message only transfers the data portion of the SCSI Data-in PDU but not the control information in the header, such as ExpCmdSN, if timely updates of such information is crucial, the iSCSI Layer at the initiator MAY issue NOP-Out PDUs to request the iSCSI Layer at the target to respond with the information using NOP- In PDUs. Ko et al. Expires June 2010 [Page 52] Internet-Draft iSER Specification December 2009 7.3.6 Ready To Transfer (R2T) Type: data-type PDU PDU-specific qualifiers: DataDescriptorOut In order to send an R2T PDU, the iSCSI Layer at the target MUST invoke the Get_Data Operational Primitive qualified with DataDescriptorOut which defines the I/O Buffer for receiving the SCSI Write data from the initiator. See section 7.1 on the general requirements on the handling of iSCSI data-type PDUs. The iSER Layer at the target MUST do the following for each invocation of the Get_Data Operational Primitive: 1. It MUST ensure a valid local STag for the I/O Buffer and a valid Local Mapping that associates the Initiator Task Tag (ITT) to the local STag. This may involve allocating a valid local STag and establishing a Local Mapping. 2. It MUST use the ITT in the R2T to locate the remote Write STag in the Remote Mapping that associates the ITT to Advertised STag(s). The Remote Mapping was established earlier by the iSER Layer at the target when the iSER Message containing the Advertised Write STag and the SCSI Command PDU for a SCSI Write or bidirectional command was received from the initiator. 3. If the iSER-ORD value at the target is set to 0, the iSER Layer at the target MUST terminate the connection and free up the resources associated with the connection (as described in 5.2.3) if it received the R2T PDU from the iSCSI Layer at the target. Upon termination of the connection, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Connection_Terminate_Notify Operational Primitive. 4. If the iSER-ORD value at the target is set to greater than 0, the iSER Layer at the target MUST transform the R2T PDU into an RDMA Read Request Message. While transforming the R2T PDU, the iSER Layer at the target MUST ensure that the number of outstanding RDMA Read Request Messages does not exceed iSER-ORD value. To transform the R2T PDU, the iSER Layer at the target: a. MUST derive the local STag and local Tagged Offset from the DataDescriptorOut that qualified the Get_Data invocation. b. MUST use the local STag as the Data Sink STag of the RDMA Read Request Message. Ko et al. Expires June 2010 [Page 53] Internet-Draft iSER Specification December 2009 c. MUST use the local Tagged Offset as the Data Sink Tagged Offset of the RDMA Read Request Message. d. MUST use the Desired Data Transfer Length from the R2T PDU as the RDMA Read Message Size of the RDMA Read Request Message. e. MUST use the remote Write STag as the Data Source STag of the RDMA Read Request Message. f. MUST use the Buffer Offset from the R2T PDU as the Data Source Tagged Offset of the RDMA Read Request Message. 5. It MUST associate R2TSN and ITT from the R2T PDU with the RDMA Read operation. If the Get_Data Operational Primitive invocation was qualified with Notify_Enable set, then when the iSER Layer at the target receives a completion from the RCaP layer for the RDMA Read operation, the iSER Layer at the target MUST notify the iSCSI Layer by invoking the Data_Completion_Notify Operational Primitive qualified with R2TSN and ITT. Conversely, if the Get_Data Operational Primitive invocation was qualified with Notify_Enable cleared, then the iSER Layer at the target MUST NOT notify the iSCSI Layer on completion and MUST NOT invoke the Data_Completion_Notify Operational Primitive. When the RCaP layer at the initiator receives a valid RDMA Read Request Message, it will return an RDMA Read Response Message containing the solicited write data to the target. When the RCaP layer at target receives the RDMA Read Response Message from the initiator, it will place the solicited data in the I/O Buffer referenced by the Data Sink STag in the RDMA Read Response Message. Since the RDMA Read Request Message from the target does not transfer the control information in the R2T PDU such as ExpCmdSN, if timely updates of such information is crucial, the iSCSI Layer at the initiator MAY issue NOP-Out PDUs to request the iSCSI Layer at the target to respond with the information using NOP-In PDUs. Similarly, since the RDMA Read Response Message from the initiator only transfers the data but not the control information normally found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates of such information is crucial, the iSCSI Layer at the target MAY issue NOP-In PDUs to request the iSCSI Layer at the initiator to respond with the information using NOP-Out PDUs. Ko et al. Expires June 2010 [Page 54] Internet-Draft iSER Specification December 2009 7.3.7 Asynchronous Message Type: control-type PDU PDU-specific qualifiers: DataDescriptorSense The iSCSI Layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorSense which defines the buffer containing the sense and iSCSI Event information. The iSER Layer MUST use a SendSE Message to send the Asynchronous Message PDU. 7.3.8 Text Request & Text Response Type: control-type PDU PDU-specific qualifiers: DataDescriptorTextOut (for Text Request), DataDescriptorIn (for Text Response) The iSCSI Layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorTextOut (or DataDescriptorIn) which defines the Text Request (or Text Response) buffer. The iSER Layer MUST use SendSE Messages to send the Text Request (or Text Response PDUs). 7.3.9 Login Request & Login Response During the login negotiation, the iSCSI Layer interacts with the transport layer directly and the iSER Layer is not involved. See section 5.1 on iSCSI/iSER Connection Setup. If the underlying transport is TCP, the Login Request PDUs and the Login Response PDUs are exchanged when the connection between the initiator and the target is still in the byte stream mode. The iSCSI Layer MUST not send a Login Request (or a Login Response) PDU during the full feature phase. A Login Request (or a Login Response) PDU, if used, MUST be treated as an iSCSI protocol error. The iSER Layer MAY reject such a PDU from the iSCSI Layer with an appropriate error code. If a Login Request PDU is received by the iSCSI Layer at the target, it MUST respond with a Reject PDU with a reason code of "protocol error". 7.3.10 Logout Request & Logout Response Type: control-type PDU PDU-specific qualifiers: None Ko et al. Expires June 2010 [Page 55] Internet-Draft iSER Specification December 2009 The iSER Layer MUST use a SendSE Message to send the Logout Request or Logout Response PDU. Section 5.2.1 and 5.2.2 describe the handling of the Logout Request and the Logout Response at the initiator and the target and the interactions between the initiator and the target to terminate a connection. 7.3.11 SNACK Request Since HeaderDigest and DataDigest must be negotiated to "None", there are no digest errors when the connection is in iSER-assisted mode. Also since RCaP delivers all messages in the order they were sent, there are no sequence errors when the connection is in iSER- assisted mode. Therefore the iSCSI Layer MUST NOT send SNACK Request PDUs. A SNCAK Request PDU, if used, MUST be treated as an iSCSI protocol error. The iSER Layer MAY reject such a PDU from the iSCSI Layer with an appropriate error code. If a SNACK Request PDU is received by the iSCSI Layer at the target, it MUST respond with a Reject PDU with a reason code of "protocol error". 7.3.12 Reject Type: control-type PDU PDU-specific qualifiers: DataDescriptorReject The iSCSI Layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorReject which defines the Reject buffer. The iSER Layer MUST use a SendSE Message to send the Reject PDU. 7.3.13 NOP-Out & NOP-In Type: control-type PDU PDU-specific qualifiers: DataDescriptorNOPOut (for NOP-Out), DataDescriptorNOPIn (for NOP-In) The iSCSI Layer MUST invoke the Send_Control Operational Primitive qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn) which defines the Ping (or Return Ping) data buffer. The iSER Layer MUST use SendSE Messages to send the NOP-Out (or NOP-In) PDU. Ko et al. Expires June 2010 [Page 56] Internet-Draft iSER Specification December 2009 8 Flow Control and STag Management 8.1 Flow Control for RDMA Send Message Types Send Message Types in RCaP are used by the iSER Layer to transfer iSCSI control-type PDUs. Each Send Message Type in RCaP consumes an Untagged Buffer at the Data Sink. However, neither the RCaP layer nor the iSER Layer provides an explicit flow control mechanism for the Send Message Types. Therefore, the iSER Layer SHOULD provision enough Untagged buffers for handling incoming Send Message Types to prevent buffer exhaustion at the RCaP layer. If buffer exhaustion occurs, it may result in the termination of the connection. An implementation may choose to satisfy the buffer requirement by using a common buffer pool shared across multiple connections, with usage limits on a per connection basis and usage limits on the buffer pool itself. In such an implementation, exceeding the buffer usage limit for a connection or the buffer pool itself may trigger interventions from the iSER Layer to replenish the buffer pool and/or to isolate the connection causing the problem. iSER also provides the MaxOutstandingUnexpectedPDUs key to be used by the initiator and the target to declare the maximum number of outstanding "unexpected" control-type PDUs that it can receive. It is intended to allow the receiving side to determine the amount of buffer resources needed beyond the normal flow control mechanism available in iSCSI. The buffer resources required at both the initiator and the target as a result of control-type PDUs sent by the initiator is described in section 8.1.1. The buffer resources required at both the initiator and target as a result of control-type PDUs sent by the target is described in section 8.1.2. 8.1.1 Flow Control for Control-Type PDUs from the Initiator The control-type PDUs that can be sent by an initiator to a target can be grouped into the following categories: 1. Regulated: Control-type PDUs in this category are regulated by the iSCSI CmdSN window mechanism and the immediate flag is not set. 2. Unregulated but Expected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism but are expected by the target. Ko et al. Expires June 2010 [Page 57] Internet-Draft iSER Specification December 2009 3. Unregulated and Unexpected: Control-type PDUs in this category are not regulated by the iSCSI CmdSN window mechanism and are "unexpected" by the target. 8.1.1.1 Control-Type PDUs from the Initiator in the Regulated Category Control-type PDUs that can be sent by the initiator in this category are regulated by the iSCSI CmdSN window mechanism and the immediate flag is not set. The queuing capacity required of the iSCSI layer at the target is described in section 3.2.2.1 of [RFC3720]. For each of the control- type PDUs that can be sent by the initiator in this category, the initiator MUST provision for the buffer resources required for the corresponding control-type PDU sent as a response from the target. The following is a list of the PDUs that can be sent by the initiator and the PDUs that are sent by the target in response: a. When an initiator sends a SCSI Command PDU, it expects a SCSI Response PDU from the target. b. When the initiator sends a Task Management Function Request PDU, it expects a Task Management Function Response PDU from the target. c. When the initiator sends a Text Request PDU, it expects a Text Response PDU from the target. d. When the initiator sends a Logout Request PDU, it expects a Logout Response PDU from the target. e. When the initiator sends a NOP-Out PDU as a ping request with ITT != 0xffffffff and TTT = 0xffffffff, it expects a NOP-In PDU from the target with the same ITT and TTT as in the ping request. The response from the target for any of the PDUs enumerated here may alternatively be in the form of a Reject PDU sent instead before the task is active, as described in section 6.3 of [RFC3720]. 8.1.1.2 Control-Type PDUs from the Initiator in the Unregulated but Expected Category For the control-type PDUs in the Unregulated but Expected category, the amount of buffering resources required at the target can be predetermined. The following is a list of the PDUs in this category: Ko et al. Expires June 2010 [Page 58] Internet-Draft iSER Specification December 2009 a. SCSI Data-out PDUs are used by the initiator to send unsolicited data. The amount of buffer resources required by the target can be determined using FirstBurstLength. Note that SCSI Data-out PDUs are not used for solicited data since the R2T PDU which is used for solicitation is transformed into RDMA Read operations by the iSER layer at the target. See section 7.3.4. b. A NOP-Out PDU with TTT != 0xffffffff is sent as a ping response by the initiator to the NOP-In PDU sent as a ping request by the target. 8.1.1.3 Control-Type PDUs from the Initiator in the Unregulated and Unexpected Category PDUs in the Unregulated and Unexpected category are PDUs with the immediate flag set. The number of PDUs in this category which can be sent by an initiator is controlled by the value of MaxOutstandingUnexpectedPDUs declared by the target. (See section 6.7.) After a PDU in this category is sent by the initiator, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs declared by the target. The target uses the value of MaxOutstandingUnexpectedPDUs that it declared to determine the amount of buffer resources required for control-type PDUs in this category that can be sent by an initiator. For the initiator, for each of the control-type PDUs that can be sent in this category, the initiator MUST provision for the buffer resources if required for the corresponding control-type PDU that can be sent as a response from the target. An outstanding PDU in this category is retired as follows. If the CmdSN of the PDU sent by the initiator in this category is x, the PDU is outstanding until the initiator sends a non-immediate control-type PDU on the same connection with CmdSN = y (where y is at least x) and the target responds with a control-type PDU on any connection where ExpCmdSN is at least y+1. When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the initiator MUST NOT generate any unexpected PDUs which otherwise it would have generated, even if it is intended for immediate delivery. Ko et al. Expires June 2010 [Page 59] Internet-Draft iSER Specification December 2009 8.1.2 Flow Control for Control-Type PDUs from the Target Control-type PDUs that can be sent by a target and are expected by the initiator are listed in the Regulated category. (See section 8.1.1.1.) For the control-type PDUs that can be sent by a target and are unexpected by the initiator, the number is controlled by MaxOutstandingUnexpectedPDUs declared by the initiator. (See section 6.7.) After a PDU in this category is sent by a target, it is outstanding until it is retired. At any time, the number of outstanding unexpected PDUs MUST not exceed the value of MaxOutstandingUnexpectedPDUs declared by the initiator. The initiator uses the value of MaxOutstandingUnexpectedPDUs that it declared to determine the amount of buffer resources required for control-type PDUs in this category that can be sent by a target. The following is a list of the PDUs in this category and the conditions for retiring the outstanding PDU: a. For an Asynchronous Message PDU with StatSN = x, the PDU is outstanding until the initiator sends a control-type PDU with ExpStatSN set to at least x+1. b. For a Reject PDU with StatSN = x which is sent after a task is active, the PDU is outstanding until the initiator sends a control-type PDU with ExpStatSN set to at least x+1. c. For a NOP-In PDU with ITT = 0xffffffff and StatSN = x, the PDU is outstanding until the initiator responds with a control-type PDU on the same connection where ExpStatSN is at least x+1. But if the NOP-In PDU is sent as a ping request with TTT != 0xffffffff, the PDU can also be retired when the initiator sends a NOP-Out PDU with the same ITT and TTT as in the ping request. Note that when a target sends a NOP-In PDU as a ping request, it must provision a buffer for the NOP-Out PDU sent as a ping response from the initiator. When the number of outstanding unexpected control-type PDUs equals MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the target MUST NOT generate any unexpected PDUs which otherwise it would have generated, even if its intent is to indicate an iSCSI error condition (e.g., Asynchronous Message, Reject). Task timeouts as in the initiator waiting for a command completion or other connection and session level exceptions will ensure that correct operational behavior will result in these cases despite not generating the PDU. This rule overrides any other requirements elsewhere which require that a Reject PDU MUST be sent. Ko et al. Expires June 2010 [Page 60] Internet-Draft iSER Specification December 2009 (Implementation note: SCSI task timeout and recovery can be a lengthy process and hence SHOULD be avoided by proper provisioning of resources.) (Implementation note: To ensure that the initiator has a means to inform the target that outstanding PDUs have been retired, the target should reserve the last unexpected control-type PDU allowable by the value of MaxOutstandingUnexpectedPDUs declared by the initiator for sending a NOP-In ping request with TTT != 0xffffffff to allow the initiator to return the NOP-Out ping response with the current ExpStatSN.) 8.2 Flow Control for RDMA Read Resources The total number of RDMA Read operations that can be active simultaneously on an iSCSI/iSER connection depends on the amount of resources allocated as declared in the iSER Hello exchange described in section 5.1.3. Exceeding the number of RDMA Read operations allowed on a connection will result in the connection being terminated by the RCaP layer. The iSER Layer at the target maintains the iSER-ORD to keep track of the maximum number of RDMA Read Requests that can be issued by the iSER Layer on a particular RCaP Stream. During connection setup (see section 5.1), iSER-IRD is known at the initiator and iSER-ORD is known at the target after the iSER Layers at the initiator and the target have respectively allocated the connection resources necessary to support RCaP, as directed by the Allocate_Connection_Resources Operational Primitive from the iSCSI Layer before the end of the iSCSI Login Phase. In the full feature phase, the first message sent by the initiator is the iSER Hello Message (see section 9.3) which contains the value of iSER-IRD. In response to the iSER Hello Message, the target sends the iSER HelloReply Message (see section 9.4) which contains the value of iSER-ORD. The iSER Layer at both the initiator and the target MAY adjust (lower) the resources associated with iSER-IRD and iSER-ORD respectively to match the iSER-ORD value declared in the HelloReply Message. The iSER Layer at the target MUST flow control the RDMA Read Request Messages to not exceed the iSER-ORD value at the target. 8.3 STag Management An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer used in an RDMA operation. The allocation and the subsequent invalidation of the STags are specified in this document if the Ko et al. Expires June 2010 [Page 61] Internet-Draft iSER Specification December 2009 STags are exposed on the wire by being Advertised in the iSER header or declared in the header of an RCaP Message. 8.3.1 Allocation of STags When the iSCSI Layer at the initiator invokes the Send_Control Operational Primitive to request the iSER Layer at the initiator to process a SCSI Command, zero, one, or two STags may be allocated by the iSER Layer. See section 7.3.1 for details. The number of STags allocated depends on whether the command is unidirectional or bidirectional and whether solicited write data transfer is involved or not. When the iSCSI Layer at the initiator invokes the Send_Control Operational Primitive to request the iSER Layer at the initiator to process a Task Management Function Request with the TASK REASSIGN function, besides allocating zero, one, or two STags, the iSER Layer MUST invalidate the existing STags, if any, associated with the ITT. See section 7.3.3 for details. The iSER Layer at the target allocates a local Data Sink STag when the iSCSI Layer at the target invokes the Get_Data Operational Primitive to request the iSER Layer to process an R2T PDU. See section 7.3.6 for details. 8.3.2 Invalidation of STags The invalidation of the STags at the initiator at the completion of a unidirectional or bidirectional command when the associated SCSI Response PDU is sent by the target is described in section 7.3.2. When a unidirectional or bidirectional command concludes without the associated SCSI Response PDU being sent by the target, the iSCSI Layer at the initiator MUST request the iSER Layer at the initiator to invalidate the STags by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. In response, the iSER Layer at the initiator MUST locate the STag(s) (if any) in the Local Mapping that associates the ITT to the local STag(s). The iSER Layer at the initiator MUST invalidate the STag(s) (if any) and the Local Mapping. For an RDMA Read operation used to realize a SCSI Write data transfer, the iSER Layer at the target SHOULD invalidate the Data Sink STag at the conclusion of the RDMA Read operation referencing the Data Sink STag (to permit the immediate reuse of buffer resources). Ko et al. Expires June 2010 [Page 62] Internet-Draft iSER Specification December 2009 For an RDMA Write operation used to realize a SCSI Read data transfer, the Data Source STag at the target is not declared to the initiator and is not exposed on the wire. Invalidation of the STag is thus not specified. When a unidirectional or bidirectional command concludes without the associated SCSI Response PDU being sent by the target, the iSCSI Layer at the target MUST request the iSER Layer at the target to invalidate the STags by invoking the Deallocate_Task_Resources Operational Primitive qualified with ITT. In response, the iSER Layer at the target MUST locate the local STag(s) (if any) in the Local Mapping that associates the ITT to the local STag(s). The iSER Layer at the target MUST invalidate the local STag(s) (if any) and the mapping. Ko et al. Expires June 2010 [Page 63] Internet-Draft iSER Specification December 2009 9 iSER Control and Data Transfer For iSCSI data-type PDUs (see section 7.1), the iSER Layer uses RDMA Read and RDMA Write operations to transfer the solicited data. For iSCSI control-type PDUs (see section 7.2), the iSER Layer uses Send Message Types of RCaP. 9.1 iSER Header Format An iSER header MUST be present in every Send Message Type of RCaP. The iSER header is located in the first 12 bytes of the message payload of the Send Message Type of RCaP, as shown in Figure 2. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Opcode| Opcode Specific Fields | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Opcode Specific Fields | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Opcode Specific Fields | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2 iSER Header Format Opcode - Operation Code: 4 bits The Opcode field identifies the type of iSER Messages: 0001b = iSCSI control-type PDU 0010b = iSER Hello Message 0011b = iSER HelloReply Message All other opcodes are reserved. 9.2 iSER Header Format for iSCSI Control-Type PDU The iSER Layer uses Send Message Types of RCaP to transfer iSCSI control-type PDUs (see section 7.2). The message payload of each of the Send Message Types of RCaP used for transferring an iSER Message contains an iSER Header followed by an iSCSI control-type PDU. The iSER header in a Send Message Type of RCaP carrying an iSCSI control-type PDU MUST have the format as described in Figure 3. Ko et al. Expires June 2010 [Page 64] Internet-Draft iSER Specification December 2009 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |W|R| | | 0001b |S|S| Reserved | | |V|V| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Write STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Write Virtual Address High | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Write Virtual Address Low | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Read STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Read Virtual Address High | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Read Virtual Address Low | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3 iSER Header Format for iSCSI Control-Type PDU WSV - Write STag Valid flag: 1 bit This flag indicates the validity of the Write STag field of the iSER Header. If set to one, the Write STag field in this iSER Header is valid. If set to zero, the Write STag field in this iSER Header MUST be ignored at the receiver. The Write STag Valid flag is set to one when there is solicited data to be transferred for a SCSI Write or bidirectional command, or when there are non-immediate unsolicited and solicited data to be transferred for the referenced task specified in a Task Management Function Request with the TASK REASSIGN function. RSV - Read STag Valid flag: 1 bit This flag indicates the validity of the Read STag field of the iSER Header. If set to one, the Read STag field in this iSER Header is valid. If set to zero, the Read STag field in this iSER Header MUST be ignored at the receiver. The Read STag Valid flag is set to one for a SCSI Read or bidirectional command, or a Task Management Function Request with the TASK REASSIGN function. Write STag - Write Steering Tag: 32 bits This field contains the Write STag when the Write STag Valid flag is set to one. For a SCSI Write or bidirectional command, Ko et al. Expires June 2010 [Page 65] Internet-Draft iSER Specification December 2009 the Write STag is used to Advertise the initiator's I/O Buffer containing the solicited data. For a Task Management Function Request with the TASK REASSIGN function, the Write STag is used to Advertise the initiator's I/O Buffer containing the non- immediate unsolicited data and solicited data. This Write STag is used as the Data Source STag in the resultant RDMA Read operation(s). When the Write STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive. Write Virtual Address High: 32 bits This field contains the high order bits of the Virtual Address for the SCSI Write command when the Write STag Valid flag is set to one. When the Write STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive. Write Virtual Address Low: 32 bits This field contains the low order bits of the Virtual Address for the SCSI Write command when the Write STag Valid flag is set to one. When the Write STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive. Read STag - Read Steering Tag: 32 bits This field contains the Read STag when the Read STag Valid flag is set to one. The Read STag is used to Advertise the initiator's Read I/O Buffer of a SCSI Read or bidirectional command, or a Task Management Function Request with the TASK REASSIGN function. This Read STag is used as the Data Sink STag in the resultant RDMA Write operation(s). When the Read STag Valid flag is zero, this field MUST be set to zero and ignored on receive. Read Virtual Address High: 32 bits This field contains the high order bits of the Virtual Address for the SCSI Read command when the Read STag Valid flag is set to one. When the Read STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive. Read Virtual Address Low: 32 bits This field contains the low order bits of the Virtual Address for the SCSI Read command when the Read STag Valid flag is set to one. When the read STag Valid flag is set to zero, this field MUST be set to zero and ignored on receive. Ko et al. Expires June 2010 [Page 66] Internet-Draft iSER Specification December 2009 Reserved: Reserved fields MUST be set to zero on transmit and MUST be ignored on receive. 9.3 iSER Header Format for iSER Hello Message An iSER Hello Message MUST only contain the iSER header which MUST have the format as described in Figure 4. iSER Hello Message is the first iSER Message sent on the RCaP Stream from the iSER Layer at the initiator to the iSER Layer at the target. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | | | | | 0010b | Rsvd | MaxVer| MinVer| iSER-IRD | | | | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 iSER Header Format for iSER Hello Message MaxVer - Maximum Version: 4 bits This field specifies the maximum version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document. MinVer - Minimum Version: 4 bits This field specifies the minimum version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document. iSER-IRD: 16 bits This field contains the value of the iSER-IRD at the initiator. Reserved (Rsvd): Reserved fields MUST be set to zero on transmit, and MUST be ignored on receive. Ko et al. Expires June 2010 [Page 67] Internet-Draft iSER Specification December 2009 9.4 iSER Header Format for iSER HelloReply Message An iSER HelloReply Message MUST only contain the iSER header which MUST have the format as described in Figure 5. The iSER HelloReply Message is the first iSER Message sent on the RCaP Stream from the iSER Layer at the target to the iSER Layer at the initiator. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | |R| | | | | 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD | | | |J| | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5 iSER Header Format for iSER HelloReply Message REJ - Reject flag: 1 bit This flag indicates whether the target is rejecting this connection. If set to one, the target is rejecting the connection. MaxVer - Maximum Version: 4 bits This field specifies the maximum version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document. CurVer - Current Version: 4 bits This field specifies the current version of the iSER protocol supported. It MUST be set to 10 to indicate the version of the specification described in this document. iSER-ORD: 16 bits This field contains the value of the iSER-ORD at the target. Reserved (Rsvd): Reserved fields MUST be set to zero on transmit, and MUST be ignored on receive. Ko et al. Expires June 2010 [Page 68] Internet-Draft iSER Specification December 2009 9.5 SCSI Data Transfer Operations The iSER Layer at the initiator and the iSER Layer at the target handle each SCSI Write, SCSI Read, and bidirectional operation as described below. 9.5.1 SCSI Write Operation The iSCSI Layer at the initiator MUST invoke the Send_Control Operational Primitive to request the iSER Layer at the initiator to send the SCSI Write Command. The iSER Layer at the initiator MUST request the RCaP layer to transmit a SendSE Message with the message payload consisting of the iSER header followed by the SCSI Command PDU and immediate data (if any). If there is solicited data, the iSER Layer MUST Advertise the Write STag in the iSER header of the SendSE Message, as described in section 9.2. Upon receiving the SendSE Message, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See section 7.3.1 for details on the handling of the SCSI Write Command. For the non-immediate unsolicited data, the iSCSI Layer at the initiator MUST invoke a Send_Control Operational Primitive qualified with the SCSI Data-out PDU. Upon receiving each Send or SendSE Message containing the non-immediate unsolicited data, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Data-out PDU. See section 7.3.4 for details on the handling of the SCSI Data-out PDU. For the solicited data, when the iSCSI Layer at the target has an I/O Buffer available, it MUST invoke the Get_Data Operational Primitive qualified with the R2T PDU. See section 7.3.6 for details on the handling of the R2T PDU. When the data transfer associated with this SCSI Write operation is complete, the iSCSI Layer at the target MUST invoke the Send_Control Operational Primitive when it is ready to send the SCSI Response PDU. Upon receiving a SendSE or SendInvSE Message containing the SCSI Response PDU, the iSER Layer at the initiator MUST notify the iSCSI Layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See section 7.3.2 for details on the handling of the SCSI Response PDU. Ko et al. Expires June 2010 [Page 69] Internet-Draft iSER Specification December 2009 9.5.2 SCSI Read Operation The iSCSI Layer at the initiator MUST invoke the Send_Control Operational Primitive to request the iSER Layer at the initiator to send the SCSI Read Command. The iSER Layer at the initiator MUST request the RCaP layer to transmit a SendSE Message with the message payload consisting of the iSER header followed by the SCSI Command PDU. The iSER Layer at the initiator MUST Advertise the Read STag in the iSER header of the SendSE Message, as described in section 9.2. Upon receiving the SendSE Message, the iSER Layer at the target MUST notify the iSCSI Layer at the target by invoking the Control_Notify Operational Primitive qualified with the SCSI Command PDU. See section 7.3.1 for details on the handling of the SCSI Read Command. When the requested SCSI data is available in the I/O Buffer, the iSCSI Layer at the target MUST invoke the Put_Data Operational Primitive qualified with the SCSI Data-in PDU. See section 7.3.5 for details on the handling of the SCSI Data-in PDU. When the data transfer associated with this SCSI Read operation is complete, the iSCSI Layer at the target MUST invoke the Send_Control Operational Primitive when it is ready to send the SCSI Response PDU. Upon receiving the SendInvSE Message containing the SCSI Response PDU, the iSER Layer at the initiator MUST notify the iSCSI Layer at the initiator by invoking the Control_Notify Operational Primitive qualified with the SCSI Response PDU. See section 7.3.2 for details on the handling of the SCSI Response PDU. 9.5.3 Bidirectional Operation The initiator and the target handle the SCSI Write and the SCSI Read portions of this bidirectional operation the same as described in Section 9.5.1 and Section 9.5.2 respectively. Ko et al. Expires June 2010 [Page 70] Internet-Draft iSER Specification December 2009 10 iSER Error Handling and Recovery RCaP provides the iSER Layer with reliable in-order delivery. Therefore, the error management needs of an iSER-assisted connection are somewhat different than those of a Traditional iSCSI connection. 10.1 Error Handling iSER error handling is described in the following sections, classified loosely based on the sources of errors: 1. Those originating at the transport layer (e.g., TCP). 2. Those originating at the RCaP layer. 3. Those originating at the iSER Layer. 4. Those originating at the iSCSI Layer. 10.1.1 Errors in the Transport Layer If the transport layer is TCP, then TCP packets with detected errors are silently dropped by the TCP layer and result in retransmission at the TCP layer. This has no impact on the iSER Layer. However, connection loss (e.g., link failure) and unexpected termination (e.g., TCP graceful or abnormal close without the iSCSI Logout exchanges) at the transport layer will cause the iSCSI/iSER connection to be terminated as well. 10.1.1.1 Failure in the Transport Layer Before RCaP Mode is Enabled If the Connection is lost or terminated before the iSCSI Layer invokes the Allocate_Connection_Resources Operational Primitive, the login process is terminated and no further action is required. If the Connection is lost or terminated after the iSCSI Layer has invoked the Allocate_Connection_Resources Operational Primitive, then the iSCSI Layer MUST request the iSER Layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. 10.1.1.2 Failure in the Transport Layer After RCaP Mode is Enabled If the Connection is lost or terminated after the iSCSI Layer has invoked the Enable_Datamover Operational Primitive, the iSER Layer MUST notify the iSCSI Layer of the connection loss by invoking the Connection_Terminate_Notify Operational Primitive. Prior to Ko et al. Expires June 2010 [Page 71] Internet-Draft iSER Specification December 2009 invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2. 10.1.2 Errors in the RCaP Layer The RCaP layer does not have error recovery operations built in. If errors are detected at the RCaP layer, the RCaP layer will terminate the RCaP Stream and the associated Connection. 10.1.2.1 Errors Detected in the Local RCaP Layer If an error is encountered at the local RCaP layer, the RCaP layer MAY send a Terminate Message to the Remote Peer to report the error if possible. (For iWARP, see [RDMAP] for the list of errors where a Terminate Message is sent.) The RCaP layer is responsible for terminating the Connection. After the RCaP layer notifies the iSER Layer that the Connection is terminated, the iSER Layer MUST notify the iSCSI Layer by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2. 10.1.2.2 Errors Detected in the RCaP Layer at the Remote Peer If an error is encountered at the RCaP layer at the Remote Peer, the RCaP layer at the Remote Peer may send a Terminate Message to report the error if possible. If it is unable to send the Terminate Message, the Connection is terminated. This is treated the same as a failure in the transport layer after RDMA is enabled as described in section 10.1.1.2. If an error is encountered at the RCaP layer at the Remote Peer and it is able to send a Terminate Message, the RCaP layer at the Remote Peer is responsible for terminating the connection. After the local RCaP layer notifies the iSER Layer that the Connection is terminated, the iSER Layer MUST notify the iSCSI Layer by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2. 10.1.3 Errors in the iSER Layer The error handling due to errors at the iSER Layer is described in the following sections. Ko et al. Expires June 2010 [Page 72] Internet-Draft iSER Specification December 2009 10.1.3.1 Insufficient Connection Resources to Support RCaP at Connection Setup After the iSCSI Layer at the initiator invokes the Allocate_Connection_Resources Operational Primitive during the iSCSI login negotiation phase, if the iSER Layer at the initiator fails to allocate the connection resources necessary to support RCaP, it MUST return a status of failure to the iSCSI Layer at the initiator. The iSCSI Layer at the initiator MUST terminate the Connection as described in Section 5.2.3.1. After the iSCSI Layer at the target invokes the Allocate_Connection_Resources Operational Primitive during the iSCSI login negotiation phase, if the iSER Layer at the target fails to allocate the connection resources necessary to support RCaP, it MUST return a status of failure to the iSCSI Layer at the target. The iSCSI Layer at the target MUST send a Login Response with a status class of 3 (Target Error), and a status code of "0302" (Out of Resources). The iSCSI Layers at the initiator and the target MUST terminate the Connection as described in Section 5.2.3.1. 10.1.3.2 iSER Negotiation Failures If the RCaP or iSER related parameters declared by the initiator in the iSER Hello Message is unacceptable to the iSER Layer at the target, the iSER Layer at the target MUST set the Reject (REJ) flag, as described in section 9.4, in the iSER HelloReply Message. The following are the cases when the iSER Layer MUST set the REJ flag to 1 in the HelloReply Message: * The initiator-declared iSER-IRD value is greater than 0 and the target-declared iSER-ORD value is 0. * The initiator-supported and the target-supported iSER protocol versions do not overlap. After requesting the RCaP layer to send the iSER HelloReply Message, the handling of the error situation is the same as that for iSER format errors as described in section 10.1.3.3. 10.1.3.3 iSER Format Errors The following types of errors in an iSER header are considered format errors: * Illegal contents of any iSER header field Ko et al. Expires June 2010 [Page 73] Internet-Draft iSER Specification December 2009 * Inconsistent field contents in an iSER header * Length error for an iSER Hello or HelloReply Message (see section 9.3 and 9.4) When a format error is detected, the following events MUST occur in the specified sequence: 1. The iSER Layer MUST request the RCaP layer to terminate the RCaP Stream. The RCaP layer MUST terminate the associated Connection. 2. The iSER Layer MUST notify the iSCSI Layer of the connection termination by invoking the Connection_Terminate_Notify Operational Primitive. Prior to invoking the Connection_Terminate_Notify Operational Primitive, the iSER layer MUST perform the actions described in Section 5.2.3.2. 10.1.3.4 iSER Protocol Errors The first iSER Message sent by the iSER Layer at the initiator MUST be the iSER Hello Message (see section 9.3). Likewise, the first iSER Message sent by the iSER Layer at the target MUST be the iSER HelloReply Message (see section 9.4). Failure to send the iSER Hello or HelloReply Message, as indicated by the wrong Opcode in the iSER header, is a protocol error. The handling of this error situation is the same as that for iSER format errors as described in section 10.1.3.3. If the sending side of an iSER-enabled connection acts in a manner not permitted by the negotiated or declared login/text operational key values as described in section 6, this is a protocol error and the receiving side MAY handle this the same as for iSER format errors as described in section 10.1.3.3. 10.1.4 Errors in the iSCSI Layer The error handling due to errors at the iSCSI Layer is described in the following sections. For error recovery, see section 10.2. 10.1.4.1 iSCSI Format Errors When an iSCSI format error is detected, the iSCSI Layer MUST request the iSER Layer to terminate the RCaP Stream by invoking the Connection_Terminate Operational Primitive. For more details on the connection termination, see Section 5.2.3.1. Ko et al. Expires June 2010 [Page 74] Internet-Draft iSER Specification December 2009 10.1.4.2 iSCSI Digest Errors In the iSER-assisted mode, the iSCSI Layer will not see any digest error because both the HeaderDigest and the DataDigest keys are negotiated to "None". 10.1.4.3 iSCSI Sequence Errors For Traditional iSCSI, sequence errors are caused by dropped PDUs due to header or data digest errors. Since digests are not used in iSER-assisted mode and the RCaP layer will deliver all messages in the order they were sent, sequence errors will not occur in iSER- assisted mode. 10.1.4.4 iSCSI Protocol Error When the iSCSI Layer handles certain protocol errors by dropping the connection, the error handling is the same as that for iSCSI format errors as described in section 10.1.4.1. When the iSCSI Layer uses the iSCSI Reject PDU and response codes to handle certain other protocol errors, no special handling at the iSER Layer is required. 10.1.4.5 SCSI Timeouts and Session Errors This is handled at the iSCSI Layer and no special handling at the iSER Layer is required. 10.1.4.6 iSCSI Negotiation Failures For negotiation failures that happen during the Login Phase at the initiator after the iSCSI Layer has invoked the Allocate_Connection_Resources Operational Primitive and before the Enable_Datamover Operational Primitive has been invoked, the iSCSI Layer MUST request the iSER Layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI Layer at the initiator MUST terminate the Connection. For negotiation failures during the Login Phase at the target, the iSCSI Layer can use a Login Response with a status class other than 0 (success) to terminate the Login Phase. If the iSCSI Layer has invoked the Allocate_Connection_Resources Operational Primitive and before the Enable_Datamover Operational Primitive has been invoked, the iSCSI Layer at the target MUST request the iSER Layer at the target to deallocate all connection resources by invoking the Ko et al. Expires June 2010 [Page 75] Internet-Draft iSER Specification December 2009 Deallocate_Connection_Resources Operational Primitive. The iSCSI Layer at both the initiator and the target MUST terminate the Connection. During the iSCSI Login Phase, if the iSCSI Layer at the initiator receives a Login Response from the target with a status class other than 0 (Success) after the iSCSI Layer at the initiator has invoked the Allocate_Connection_Resources Operational Primitive, the iSCSI Layer MUST request the iSER Layer to deallocate all connection resources by invoking the Deallocate_Connection_Resources Operational Primitive. The iSCSI Layer MUST terminate the Connection in this case. For negotiation failures during the full feature phase, the error handling is left to the iSCSI Layer and no special handling at the iSER Layer is required. 10.2 Error Recovery Error recovery requirements of iSCSI/iSER are the same as that of Traditional iSCSI. All three ErrorRecoveryLevels as defined in [RFC3720] are supported in iSCSI/iSER. * For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and no special handling by the iSER Layer is required. * For ErrorRecoveryLevel 1, see section 10.2.1 on PDU Recovery. * For ErrorRecoveryLevel 2, see section 10.2.2 on Connection Recovery. The iSCSI Layer may invoke the Notice_Key_Values Operational Primitive during connection setup to request the iSER Layer to take note of the value of the operational ErrorRecoveryLevel, as described in sections 5.1.1 and 5.1.2. 10.2.1 PDU Recovery As described in sections 10.1.4.2 and 10.1.4.3, digest and sequence errors will not occur in the iSER-assisted mode. If the RCaP layer detects an error, it will close the iSCSI/iSER connection, as described in section 10.1.2. Therefore, PDU recovery is not useful in the iSER-assisted mode. The iSCSI Layer at the initiator SHOULD disable iSCSI timeout-driven PDU retransmissions. Ko et al. Expires June 2010 [Page 76] Internet-Draft iSER Specification December 2009 10.2.2 Connection Recovery The iSCSI Layer at the initiator MAY reassign connection allegiance for non-immediate commands which are still in progress and are associated with the failed connection by using a Task Management Function Request with the TASK REASSIGN function. See section 7.3.3 for more details. When the iSCSI Layer at the initiator does a task reassignment for a SCSI Write command, it MUST qualify the Send_Control Operational Primitive invocation with DataDescriptorOut which defines the I/O Buffer for both the non-immediate unsolicited data and the solicited data. This allows the iSCSI Layer at the target to use recovery R2Ts to request for data originally sent as unsolicited and solicited from the initiator. When the iSCSI Layer at the target accepts a reassignment request for a SCSI Read command, it MUST request the iSER Layer to process SCSI Data-in for all unacknowledged data by invoking the Put_Data Operational Primitive. See section 7.3.5 on the handling of SCSI Data-in. When the iSCSI Layer at the target accepts a reassignment request for a SCSI Write command, it MUST request the iSER Layer to process a recovery R2T for any non-immediate unsolicited data and any solicited data sequences that have not been received by invoking the Get_Data Operational Primitive. See section 7.3.6 on the handling of Ready To Transfer (R2T). The iSCSI Layer at the target MUST NOT issue recovery R2Ts on an iSCSI/iSER connection for a task for which the connection allegiance was never reassigned. The iSER Layer at the target MAY reject such a recovery R2T received via the Get_Data Operational Primitive invocation from the iSCSI Layer at the target, with an appropriate error code. The iSER Layer at the target will process the requests invoked by the Put_Data and Get_Data Operational Primitives for a reassigned task in the same way as for the original commands. Ko et al. Expires June 2010 [Page 77] Internet-Draft iSER Specification December 2009 11 Security Considerations When iSER is layered on top of an RCaP layer and provides the RDMA extensions to the iSCSI protocol, the security considerations of iSER are the same as that of the underlying RCaP layer. For iWARP, this is described in [RDMAP] and [RDDPSEC]. Since iSER-assisted iSCSI protocol is still functionally iSCSI from a security considerations perspective, all of the iSCSI security requirements as described in [RFC3720] and [RFC3723] apply. If iSER is layered on top of a non-IP based RCaP layer, all the security protocol mechanisms applicable to that RCaP layer is also applicable to an iSCSI/iSER connection. If iSER is layered on top of a non-IP protocol, the IPsec mechanism as specified in [RFC3720] MUST be implemented at any point where the iSER protocol enters the IP network (e.g., via gateways), and the non-IP protocol SHOULD implement (optional to use) a packet by packet security protocol equal in strength to the IPsec mechanism specified by [RFC3720]. To minimize the potential for a denial of service attack, the iSCSI Layer MUST NOT request the iSER Layer to allocate the connection resources necessary to support RCaP until the iSCSI layer is sufficiently far along in the iSCSI Login Phase that it is reasonably certain that the peer side is not an attacker, as described in sections 5.1.1 and 5.1.2. Ko et al. Expires June 2010 [Page 78] Internet-Draft iSER Specification December 2009 12 IANA Considerations This document has no actions for IANA. Ko et al. Expires June 2010 [Page 79] Internet-Draft iSER Specification December 2009 13 References 13.1 Normative References [RFC5046] M. Ko et al., "iSCSI Externsions for Remote Direct Memory Access", RFC 5046, October 2007 [RFC3720] J. Satran et al., "iSCSI", RFC 3720, April 2004 [RFC3723] B. Aboba et al., "Securing Block Storage Protocols over IP", RFC 3723, April 2004. [RDMAP] R. Recio et al., "An RDMA Protocol Specification", RFC 5040, October 2007 [DDP] H. Shah et al., "Direct Data Placement over Reliable Transports", RFC 5041, October 2007 [MPA] P. Culley et al., "Marker PDU Aligned Framing for TCP Specification", RFC 5044, October 2007 [RDDPSEC] J. Pinkerton et al., "DDP/RDMAP Security", RFC 5042, October 2007 [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981 [RFC2119] Bradner, S., "Key Words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 13.2 Informative References [SAM2] T10/1157D, SCSI Architecture Model - 2 (SAM-2) [DA] M. Chadalapaka et al., "Datamover Architecture for iSCSI", RFC 5047, October 2007 [VERBS] J. Hilland et al., "RDMA Protocol Verbs Specification", RDMAC Consortium Draft Specification draft-hilland-iwarp-verbs- v1.0-RDMAC, April 2003 [IPSEC] S. Kent et al., "Security Architecture for the Internet Protocol", RFC 2401, November 1998 [IB] InfiniBand Architecture Specification Volume 1 Release 1.2, October 2004 Ko et al. Expires June 2010 [Page 80] Internet-Draft iSER Specification December 2009 [IPoIB] H.K. Chu et al, "Transmission of IP over InfiniBand", RFC 4391, March 2006 Ko et al. Expires June 2010 [Page 81] Internet-Draft iSER Specification December 2009 14 Summary of Changes from RFC 5046 1. Removed the requirement that a connection be opened in "normal" TCP mode and transitioned to zero-copy mode. The allows the spec to conform to existing implementation for both Infiniband and iWARP. Changes were made in sections 2, 3.1.6, 4.2, 5.1, 5.1.1, 5.1.2, 5.1.3, 10.1.3.4, and 11. 2. Added a clause in section 6.2 to clarify that the initiator must not send more than InitiatorMaxRecvDataSegmentLength worth of data when a NOP-Out request is sent with a valid Initiator Task Tag. Since InitiatorMaxRecvDataSegmentLength can be smaller than TargetMaxRecvDataSegmentLength, returning the original data in the NOP-Out request in this situation can overflow the receive buffer unless the length of the data sent with the NOP-Out request is less than InitiatorMaxRecvDataSegmentLength. 3. Added MaxAHSLength key in section 6.8 to set a limit on the AHS Length as proposed by Alex Nezhinsky. This is useful when posting receive buffers in knowing what the maximum possible message length is in a PDU which contains AHS. 4. Added WriteAddressForSolicitedDataOnly key in section 6.9 to indicate how the memory region will be used as proposed by Alex Nezhinsky. Alex argued that WSTAG (RKEY+VA for Infiniband) sent by the initiator with a WRITE SCSI command already accounts for the offset associated with the solicited data, and does not reference the entire data. The rationale is that the initiator sometimes treats the memory regions intended for unsolicited and solicited data differently, as they are carried out using different Infiniband mechanisms (Send from the initiator, RDMA from the target) that imply different registration modes. In contrast, the model implied by the iSER spec creators was that the memory occupied by data is treated as contiguous (or virtually contiguous, by means of scatter-gather mechanisms) and homogenous region. Adding a new key will allow coping with current implementations. 5. Added two 64-bit fields in iSER header in section 9.2 for the Read Virtual Address and the Write Virtual Address to conform to the Infiniband format as proposed by Bob Russell. This allows one implementation such as the OFED stack to be used in both the Infiniband and the iWARP environment. Ko et al. Expires June 2010 [Page 82] Internet-Draft iSER Specification December 2009 14.1 Proposed Changes To Be Considered 6. Alex Nezhinsky stated that iSER Hello and iSER HelloReply messages are not implemented in practice. He argued that we should update the spec to remove the requirement to exchange the iSER Hello and iSER HelloReply messages. 14.2 Changes To Be Done Update all diagrams to conform with the expanded iSER header format. Ko et al. Expires June 2010 [Page 83] Internet-Draft iSER Specification December 2009 Appendix A 14.3 iWARP Message Format for iSER This section is for information only and is NOT part of the standard. It simply depicts the iWARP Message format for the various iSER Messages when the transport layer is TCP. 14.3.1 iWARP Message Format for iSER Hello Message The following figure depicts an iSER Hello Message encapsulated in an iWARP SendSE Message. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0010b | Zeros | 0001b | 0001b | iSER-IRD | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6 SendSE Message containing an iSER Hello Message Ko et al. Expires June 2010 [Page 84] Internet-Draft iSER Specification December 2009 14.3.2 iWARP Message Format for iSER HelloReply Message The following figure depicts an iSER HelloReply Message encapsulated in an iWARP SendSE Message. The Reject (REJ) flag is set to 0. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0011b |Zeros|0| 0001b | 0001b | iSER-ORD | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7 SendSE Message containing an iSER HelloReply Message Ko et al. Expires June 2010 [Page 85] Internet-Draft iSER Specification December 2009 14.3.3 iWARP Message Format for SCSI Read Command PDU The following figure depicts a SCSI Read Command PDU embedded in an iSER Message encapsulated in an iWARP SendSE Message. For this particular example, in the iSER header, the Write STag Valid flag is set to zero, the Read STag Valid flag is set to one, the Write STag field is set to all zeros, and the Read STag field contains a valid Read STag. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0001b |0|1| All zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Read STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Read Command PDU | // // | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8 SendSE Message containing a SCSI Read Command PDU Ko et al. Expires June 2010 [Page 86] Internet-Draft iSER Specification December 2009 14.3.4 iWARP Message Format for SCSI Read Data The following figure depicts an iWARP RDMA Write Message carrying SCSI Read data in the payload: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Sink STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Sink Tagged Offset | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Read data | // // | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9 RDMA Write Message containing SCSI Read Data Ko et al. Expires June 2010 [Page 87] Internet-Draft iSER Specification December 2009 14.3.5 iWARP Message Format for SCSI Write Command PDU The following figure depicts a SCSI Write Command PDU embedded in an iSER Message encapsulated in an iWARP SendSE Message. For this particular example, in the iSER header, the Write STag Valid flag is set to one, the Read STag Valid flag is set to zero, the Write STag field contains a valid Write STag, and the Read STag field is set to all zeros since it is not used. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0001b |1|0| All zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Write STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Write Command PDU | // // | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10 SendSE Message containing a SCSI Write Command PDU Ko et al. Expires June 2010 [Page 88] Internet-Draft iSER Specification December 2009 14.3.6 iWARP Message Format for RDMA Read Request An iSCSI R2T is transformed into an iWARP RDMA Read Request Message. The following figure depicts an iWARP RDMA Read Request Message: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved (Not Used) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read Request) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read Request) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read Request) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Sink STag (SinkSTag) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Data Sink Tagged Offset (SinkTO) + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RDMA Read Message Size (RDMARDSZ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Source STag (SrcSTag) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Data Source Tagged Offset (SrcTO) + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 11 RDMA Read Request Message Ko et al. Expires June 2010 [Page 89] Internet-Draft iSER Specification December 2009 14.3.7 iWARP Message Format for Solicited SCSI Write Data The following figure depicts an iWARP RDMA Read Response Message carrying the solicited SCSI Write data in the payload: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Sink STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Sink Tagged Offset | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Write Data | // // | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 12 RDMA Read Response Message containing SCSI Write Data Ko et al. Expires June 2010 [Page 90] Internet-Draft iSER Specification December 2009 14.3.8 iWARP Message Format for SCSI Response PDU The following figure depicts a SCSI Response PDU embedded in an iSER Message encapsulated in an iWARP SendInvSE Message: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA Header | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Invalidate STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0001b |0|0| All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | All Zeros | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Response PDU | // // | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MPA CRC | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 13 SendInvSE Message containing SCSI Response PDU Ko et al. Expires June 2010 [Page 91] Internet-Draft iSER Specification December 2009 15 Appendix B 15.1 Architectural discussion of iSER over InfiniBand This section explains how an InfiniBand network (with Gateways) would be structured. It is informational only and is intended to provide insight on how iSER is used in an InfiniBand environment. 15.2 The Host side of the iSCSI & iSER connections in Infiniband Figure 14 defines the topologies in which iSCSI and iSER will be able to operate on an InfiniBand Network. +---------+ +---------+ +---------+ +---------+ +--- -----+ | Host | | Host | | Host | | Host | | Host | | | | | | | | | | | +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ |----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB IB| IB | IB | IB | IB | SubNet2 SWTCH +-v-----------v-----------v-----------v-----------v---------+ | InfiniBand Switch for Subnet1 | +---+-----+--------+-----+--------+-----+------------v------+ | TCA | | TCA | | TCA | | +-----+ +-----+ +-----+ | IB / IB \ / IB \ / \ +--+--v--+--+ | iSER | | iSER | | IPoIB | | | TCA | | | Gateway | | Gateway | | Gateway | | +-----+ | | to | | to | | to | | Storage | | iSCSI | | iSER | | IP | | Controller| | TCP | | iWARP | |Ethernet | +-----+-----+ +---v-----| +---v-----| +----v----+ | EN | EN | EN +--------------+---------------+----> to IP based storage Ethernet links that carry iSCSI or iWARP Figure 14 iSCSI and iSER on IB In Figure 14, the Host systems are connected via the InfiniBand Host Channel Adapters (HCAs) to the InfiniBand links. With the use of IB switch(es), the InfiniBand links connect the HCA to InfiniBand Target Channel Adapters (TCAs) located in gateways or Storage Controllers. An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated in IB protocols to either standard iSCSI, or iSER Messages for iWARP. An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP protocol, and in the iSCSI case, permits Ko et al. Expires June 2010 [Page 92] Internet-Draft iSER Specification December 2009 iSCSI to be operated on an IB Network between the Hosts and the [IPoIB] Gateway. 15.3 The Storage side of iSCSI & iSER mixed network environment Figure 15 shows a storage controller that has three different portal groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1). | | | | | | +--+--v--+----------+--v--+----------+--v--+--+ | | IB | |iWARP| | EN | | | | | | TCP | | NIC | | | |(TCA)| | RNIC| | | | | +-----| +-----+ +-----+ | | TPG-1 TPG-2 TPG-4 | | 9.1.3.3 9.1.2.4 9.1.2.6 | | | | Storage Controller | | | +---------------------------------------------+ Figure 15 Storage Controller with TCP, iWARP, and IB Connections The normal iSCSI portal group advertising processes (via SLP, iSNS, or SendTargets) are available to a Storage Controller. 15.4 Discovery processes for an InfiniBand Host An InfiniBand Host system can gather portal group IP address from SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via [IPoIB]. After obtaining one or more remote portal IP addresses, the Initiator uses the standard IP mechanisms to resolve the IP address to a local outgoing interface and the destination hardware address (Ethernet MAC or IB GID of the target or a gateway leading to the target). If the resolved interface is an [IPoIB] network interface, then the target portal can be reached through an InfiniBand fabric. In this case the Initiator can establish an iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand interface, using the Hardware Address (InfiniBand GID) obtained through the standard Address Resolution (ARP) processes. If more than one IP address are obtained through the discovery process, the Initiator should select a Target IP address that is on the same IP subnet as the Initiator if one exists. This will avoid Ko et al. Expires June 2010 [Page 93] Internet-Draft iSER Specification December 2009 a potential overhead of going through a gateway when a direct path exists. In addition a user can configure manual static IP route entries if a particular path to the target is preferred. 15.5 IBTA Connection specifications It is outside the scope of this document, but it is expected that the InfiniBand Trade Association (IBTA) has or will define: * The iSER ServiceID * A Means for permitting a Host to establish a connection with a peer InfiniBand end-node, and that peer indicating when that end-node supports iSER, so the Host would be able to fall back to iSCSI/TCP over [IPoIB]. * A Means for permitting the Host to establish connections with IB iSER connections on storage controllers or IB iSER connected Gateways in preference to [IPoIB] connected Gateways/Bridges or connections to Target Storage Controllers that also accept iSCSI via [IPoIB]. * A Means for combining the IB ServiceID for iSER and the IP port number such that the IB Host can use normal IB connection processes, yet ensure that the iSER target peer can actually connect to the required IP port number. Ko et al. Expires June 2010 [Page 94] Internet-Draft iSER Specification December 2009 16 Acknowledgments Credit goes to the authors of [RFC5046], M. Ko, M. Chadalapaka, J. Hufferd, U. Elzur, H. Shah, and P. Thaler for coming up with the first version of the iSER specification. Author's Address Mike Ko Huawei Symantec 20245 Stevens Creek Blvd. Cupertino, CA 95014, USA Phone: +1-408-510-7465 Email: michael@huaweisymantec.com Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Ko et al. Expires June 2010 [Page 95]