<?xml version="1.0" encoding="UTF-8"?>
  <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
  <!-- generated by https://github.com/cabo/kramdown-rfc version  (Ruby 3.1.2) --> version='1.0' encoding='UTF-8'?>

<!DOCTYPE rfc [
  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;">
]>

<?rfc strict="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc editing="no"?>
<?rfc tocompact="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>

<rfc xmlns:xi="http://www.w3.org/2001/XInclude"
ipr="trust200902"
docName="draft-ietf-nfsv4-scsi-layout-nvme-07"
number="9561"
category="std"
consensus="true"
submissionType="IETF" tocDepth="3"
obsoletes=""
updates=""
tocInclude="true"
sortRefs="true" symRefs="true">
symRefs="true"
version="3"
xml:lang="en"
>

  <front>
    <title abbrev="pNFS SCSI Layout for NVMe">Using the Parallel NFS (pNFS) SCSI Layout to access NVMe storage devices</title> Access Non-Volatile Memory Express (NVMe) Storage Devices</title>
    <seriesInfo name="RFC" value="9561"/>
    <author initials="C." surname="Hellwig" fullname="Christoph Hellwig" role="editor">
      <organization></organization>
      <organization/>
      <address>
        <email>hch@lst.de</email>
      </address>
    </author>
    <author initials="C." surname="Lever" fullname="Charles Lever">
      <organization abbrev="Oracle">Oracle Corporation</organization>
      <address>
        <postal>
          <country>United States of America</country>
        </postal>
        <email>chuck.lever@oracle.com</email>
      </address>
    </author>
    <author initials="S." surname="Faibish" fullname="Sorin Faibish">
      <organization>Opendrives.com</organization>
      <address>
        <postal>
          <street>11 Selwyn Road</street>
          <city>Newton</city>
          <region>MA</region>
          <code>02461</code>
          <country>United States of America</country>
        </postal>
        <phone>+1 617-510-0422</phone>
        <email>s.faibish@opendrives.com</email>
      </address>
    </author>
    <author initials="D." surname="Black" fullname="David L. Black">
      <organization>Dell Technologies</organization>
      <address>
        <postal>
          <street>176 South Street</street>
          <city>Hopkinton</city>
          <region>MA</region>
          <code>01748</code>
          <country>United States of America</country>
        </postal>
        <email>david.black@dell.com</email>
      </address>
    </author>

    <date year="2024" month="January" day="10"/> month="March"/>

    <area>Transport</area>
    <workgroup>NFSv4</workgroup>
    <keyword>NFSv4</keyword>

    <abstract>
      <t>This document specifies how to use the Parallel Network File System (pNFS)
Small Computer System Interface (SCSI) Layout Type to access storage devices
using the Non-Volatile Memory Express (NVMe) protocol family.</t>
    </abstract>
  </front>
  <middle>
    <section anchor="sec:intro"><name>Introduction</name> anchor="sec_intro">
      <name>Introduction</name>
      <t>NFSv4.1 (in <xref target="RFC8881"/>) target="RFC8881"/> includes a pNFS feature that allows
reads and writes to be performed by other means other than directing read and
write operations to the server.  Through use of this feature, the server,
in the role of metadata server server, is responsible for managing file and
directory metadata while separate means are provided for execution of to execute
reads and writes.</t>
      <t>These other means of performing file reads and writes are defined by
individual mapping types types, which often have their own specifications.</t>
      <t>The pNFS Small Computer System Interface (SCSI) layout <xref target="RFC8154"/> is a layout
type that allows NFS clients to directly perform I/O I&wj;/&wj;O to block storage devices
while bypassing the Metadata Server (MDS).  It is specified by using
concepts from the SCSI protocol family for the data path to the storage devices.</t>
      <t>NVM Express (NVMe), or the Non-Volatile Memory Host Controller Interface
Specification, is a set of specifications to talk to storage devices over
a number of protocols such as PCI Express (PCIe), Fibre Channel (FC) and
TCP/IP (FC),
TCP/IP, or Remote Direct Memory Access (RDMA) networking.  NVMe is currently
the by far dominant protocol predominantly used protocol to access PCIe Solid State Disks (SSDs),
and it is increasingly being adopted for remote storage access where it replaces to replace
SCSI-based protocols such as iSCSI.</t>
      <t>This document defines how NVMe Namespaces using the NVM Command Set <xref target="NVME-NVM"/>
exported by NVMe Controllers implementing the
NVMe Base specification <xref target="NVME-BASE"/> are to be used as
storage devices using the SCSI Layout Type.
The definition is independent of the underlying transport used by the
NVMe Controller and thus supports Controllers implementing a wide variety
of transports, including PCI Express, PCIe, RDMA, TCP TCP, and Fibre Channel.</t> FC.</t>
      <t>This document does not amend the existing SCSI layout document.  Rather,
it
defines how NVMe Namespaces can be used within the SCSI Layout by
establishing a mapping of the SCSI constructs used in the SCSI layout
document to corresponding NVMe constructs.</t>
      <section anchor="ssc:intro:reqlang"><name>Requirements anchor="ssc_intro_reqlang">
        <name>Requirements Language</name>

<t>The
        <t>
    The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
    "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>",
    "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>",
    "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be
    interpreted as described in BCP 14 BCP&nbsp;14 <xref target="RFC2119"/> <xref
    target="RFC8174"/> when, and only when, they appear in all capitals, as
    shown here.</t> here.
        </t>

      </section>
      <section anchor="ssc:intro:defs"><name>General anchor="ssc_intro_defs">
        <name>General Definitions</name>
        <t>The following definitions are provided for the purpose of providing
an appropriate included to provide context for the reader.</t>
        <dl>
  <dt>Client</dt>
          <dt>Client:</dt>
          <dd>
            <t>The "client" is the entity that accesses the NFS server's
resources.  The client may be an application that contains the
logic to access the NFS server directly directly, or it may be part of the operating
system that provides remote file system services for a set of
applications.</t>
          </dd>
          <dt>Metadata Server (MDS)</dt> (MDS):</dt>
          <dd>
            <t>The Metadata Server (MDS) is the entity responsible for coordinating
client access to a set of file systems and is identified by a server
owner.</t>
          </dd>
        </dl>
      </section>
      <section anchor="ssc:intro:conv"><name>Numerical anchor="ssc_intro_conv">
        <name>Numerical Conventions</name>
        <t>Numerical values defined in the SCSI specifications, e.g., specifications (e.g., <xref target="SPC5"/>, target="SPC5"/>) and the
NVMe specifications, e.g., specifications (e.g., <xref target="NVME-BASE"/>, target="NVME-BASE"/>) are represented using the same
conventions as those specifications wherein a 'b' suffix denotes a binary
(base 2) number, e.g., number (e.g., 110b = 6 decimal, decimal) and an 'h' suffix denotes a
hexadecimal (base 16) number, e.g., number (e.g., 1ch = 28 decimal.</t> decimal).</t>
      </section>
    </section>
    <section anchor="sec:slm"><name>SCSI anchor="sec_slm">
      <name>SCSI Layout mapping Mapping to NVMe</name>
      <t>The SCSI layout definition <xref target="RFC8154"/> references only a
few SCSI-specific concepts directly.  This document provides a mapping
from these SCSI concepts to NVM Express concepts that are used
when using the pNFS SCSI layout with NVMe namespaces.</t>
      <section anchor="ssc:volident"><name>Volume anchor="ssc_volident">
        <name>Volume Identification</name>
        <t>The pNFS SCSI layout uses the Device Identification Vital Product Data (VPD)
page (page code 83h) from <xref target="SPC5"/> to identify the devices used by
a layout. Implementations that use NVMe namespaces as storage devices
map NVMe namespace identifiers to a subset of the identifiers
that the Device Identification VPD page supports for SCSI logical
units.</t>
        <t>To be used as storage devices for the pNFS SCSI layout, NVMe namespaces
<bcp14>MUST</bcp14> support either the IEEE Extended Unique Identifier (EUI64) or
Namespace Globally Unique Identifier (NGUID) value reported in a Namespace
Identification Descriptor, the I/O I&wj;/&wj;O Command Set Independent Identify
Namespace Data Structure, data structure, and the Identify Namespace Data Structure, data structure,
NVM Command Set <xref target="NVME-BASE"/>. If available, use of the NGUID value is
preferred as it is the larger identifier.</t>

<t>Note:

        <aside><t>Note: The PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME have no equivalent
in NVMe and cannot be used to identify NVMe storage devices.</t> devices.</t></aside>
        <t>The pnfs_scsi_base_volume_info4 structure for an NVMe namespace
<bcp14>SHALL</bcp14> be constructed as follows:</t>

<t><list style="numbers">
        <ol spacing="normal" type="1"><li>
            <t>The "sbv_code_set" field <bcp14>SHALL</bcp14> be set to PS_CODE_SET_BINARY.</t>
          </li>
          <li>
            <t>The "pnfs_scsi_designator_type" field <bcp14>SHALL</bcp14> be set to
  PS_DESIGNATOR_EUI64.</t>
          </li>
          <li>
            <t>The "sbv_designator" field <bcp14>SHALL</bcp14> contain either the NGUID or
  the EUI64 identifier for the namespace.  If both NGUID and EUI64
  identifiers are available, then the NGUID identifier <bcp14>SHOULD</bcp14> be
  used as it is the larger identifier.</t>
</list></t>
          </li>
        </ol>

        <t>RFC 8154 <xref target="RFC8154"/> specifies the "sbv_designator" field as an XDR variable
length opaque&lt;&gt;. opaque&lt;&gt; (refer to Section <xref target="RFC4506" sectionFormat="bare" section="4.10"/> of RFC 4506 <xref target="RFC4506"/>). The length of that XDR opaque&lt;&gt; value (part of
its XDR representation) indicates which NVMe identifier is present.
That length <bcp14>MUST</bcp14> be 16 octets for an NVMe NGUID identifier and
<bcp14>MUST</bcp14> be 8 octets for an NVMe EUI64 identifier.  All other lengths
<bcp14>MUST NOT</bcp14> be used with an NVMe namespace.</t>
      </section>
      <section anchor="ssc:fencing"><name>Client anchor="ssc_fencing">
        <name>Client Fencing</name>
        <t>The SCSI layout uses Persistent Reservations (PRs) to provide client
fencing.  For this to be achieved, both the MDS and the Clients have to
register a key with the storage device, and the MDS has to create a
reservation on the storage device.</t>
        <t>The following sub-sections subsections provide a full mapping of the required
PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT SCSI commands <xref target="SPC5"/>
to NVMe commands which that <bcp14>MUST</bcp14> be used when using
NVMe namespaces as storage devices for the pNFS SCSI layout.</t>
        <section anchor="ssc:fencing:keys"><name>PRs anchor="ssc_fencing_keys">
          <name>PRs - Key Registration</name>
          <t>On NVMe namespaces, reservation keys are registered using the
Reservation Register command (refer to Section 7.3 of <xref target="NVME-BASE"/>)
with the Reservation Register Action
(RREGA) field set to 000b (i.e., Register Reservation Key) and
supplying the reservation key in the New Reservation Key (NRKEY)
field.</t>
          <t>Reservation keys are unregistered using the Reservation Register
command with the Reservation Register Action (RREGA) field set to
001b (i.e., Unregister Reservation Key) and supplying the reservation
key in the Current Reservation Key (CRKEY) field.</t>
          <t>One important difference between SCSI Persistent Reservations
and NVMe Reservations is that NVMe reservation keys always apply
to all controllers used by a host (as indicated by the NVMe Host
Identifier). This behavior is analogous to setting the ALL_TG_PT
bit when registering a SCSI Reservation key, Key, and it is always supported
by NVMe Reservations, unlike the ALL_TG_PT for which SCSI support is
inconsistent and cannot be relied upon.
Registering a reservation key with a namespace creates an
association between a host and a namespace. A host that is a
registrant of a namespace may use any controller with which that
host is associated (i.e., that has the same Host Identifier,
refer to Section 5.27.1.25 of <xref target="NVME-BASE"/>)
to access that namespace as a registrant.</t>
        </section>
        <section anchor="ssc:fencing:reg"><name>PRs anchor="ssc_fencing_reg">
          <name>PRs - MDS Registration and Reservation</name>
          <t>Before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the MDS
needs to prepare the volume for fencing using PRs. This is done by
registering the reservation generated for the MDS with the device
(see <xref target="ssc:fencing:keys"/>) target="ssc_fencing_keys"/>) followed by a Reservation Acquire
command (refer to Section 7.2 of <xref target="NVME-BASE"/>) with
the Reservation Acquire Action (RACQA) field set to 000b (i.e., Acquire)
and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive Access
- Registrants Only Reservation).</t>
        </section>
        <section anchor="ssc:fenceaction"><name>Fencing anchor="ssc_fenceaction">
          <name>Fencing Action</name>
          <t>In case of a non-responding client, the MDS fences the client by
executing a Reservation Acquire command (refer to Section 7.2 of <xref target="NVME-BASE"/>),
with the Reservation Acquire Action
(RACQA) field set to 001b (i.e., Preempt) or 010b (i.e., Preempt and
Abort), the Current Reservation Key (CRKEY) field set to the
server's reservation key, the Preempt Reservation Key (PRKEY) field
set to the reservation key associated with the non-responding client client,
and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive
Access - Registrants Only Reservation).
The client can distinguish I/O I&wj;/&wj;O errors due to fencing from other
errors based on the Reservation Conflict NVMe status code.</t>
        </section>
        <section anchor="ssc:recovery"><name>Client anchor="ssc_recovery">
          <name>Client Recovery after a Fence Action</name>
          <t>If an NVMe command issued by the client to the storage device returns
a non-retryable error (refer to the DNR bit defined in Figure 92 in
<xref target="NVME-BASE"/>), the client <bcp14>MUST</bcp14> commit all layouts that
use the storage device through the MDS, return all outstanding layouts
for the device, forget the device ID, and unregister the reservation
key.</t>
        </section>
      </section>
      <section anchor="ssc:caches"><name>Volatile write caches</name> anchor="ssc_caches">
        <name>Volatile Write Caches</name>
        <t>For NVMe controllers controllers, a volatile write cache is enabled if bit 0 of the
Volatile Write Cache (VWC) field in the Identify Controller Data
Structure, I/O data
structure, I&wj;/&wj;O Command Set Independent (see (refer to Figure 275 in <xref target="NVME-BASE"/>)
is set and the Volatile Write Cache Enable (WCE) bit (i.e., bit 00) in
the Volatile Write Cache Feature (Feature Identifier 06h)
(see
(refer to Section 5.27.1.4 of <xref target="NVME-BASE"/>) is set.
If a volatile write cache is enabled on an NVMe namespace used as a
storage device for the pNFS SCSI layout, the pNFS server (MDS) <bcp14>MUST</bcp14>
use the NVMe Flush command to flush the volatile write cache to
stable storage before the LAYOUTCOMMIT operation returns by using the
Flush command (see (refer to Section 7.1 of <xref target="NVME-BASE"/>).
The NVMe Flush command is the equivalent to the SCSI SYNCHRONIZE
CACHE commands.</t>
      </section>
    </section>
    <section anchor="sec:security"><name>Security anchor="sec_security">
      <name>Security Considerations</name>

      <t>NFSv4 clients access NFSv4 metadata servers using the NFSv4
protocol. The security considerations generally described in <xref target="RFC8881"/>
apply to a client's interactions with
the metadata server. However, NFSv4 clients and servers access
NVMe storage devices at a lower layer than NFSv4. NFSv4 and
RPC security are not directly applicable to the I/Os I&wj;/&wj;Os to data servers
using NVMe.
Refer to Section Sections <xref target="RFC8154" section="2.4.6" sectionFormat="bare">Extents Are Permissions</xref> and Section <xref target="RFC8154" section="4" sectionFormat="bare">Security Considerations</xref> of <xref target="RFC8154"/> for the
Security Considerations
security considerations of direct block access to block storage from NFS clients.</t>
      <t>pNFS with an NVMe layout can be used with NVMe transports (e.g., NVMe
over PCIe <xref target="NVME-PCIE"/>) that provide essentially no additional security
functionality. Or, pNFS may be used with storage protocols such as NVMe
over TCP <xref target="NVME-TCP"/> that can provide significant transport layer
security.</t>
      <t>It is the responsibility of those administering and deploying pNFS with
an NVMe layout to ensure that appropriate protection is deployed to that
protocol based on the deployment environment as well as the nature and
sensitivity of the data and storage devices involved.  When using IP-based
storage protocols such as NVMe over TCP, data confidentiality and
integrity <bcp14>SHOULD</bcp14> be provided for traffic between pNFS clients and NVMe
storage devices by using a secure communication protocol such as Transport
Layer Security (TLS) <xref target="RFC8446"/>. For NVMe over TCP, TLS <bcp14>SHOULD</bcp14> be used as
described in <xref target="NVME-TCP"/> to protect traffic between pNFS clients and NVMe
namespaces used as storage devices.</t>
      <t>A secure communication protocol might not be needed for pNFS with NVMe
layouts in environments where physical and/or logical security measures
(e.g., air gaps, isolated VLANs) provide effective access control
commensurate with the sensitivity and value of the storage devices and data
involved (e.g., public website contents may be significantly less sensitive
than a database containing personal identifying information, passwords,
and other authentication credentials).</t>
      <t>Physical security is a common means for protocols not based on IP. In environments where the security requirements for the storage
protocol cannot be met, pNFS with an NVMe layout <bcp14>SHOULD NOT</bcp14> be
deployed.</t>
      <t>When security is available for the data server storage protocol,
it is generally at a different granularity and with a different
notion of identity than NFSv4 (e.g., NFSv4 controls user access
to files, and NVMe controls initiator access to volumes).  As
with pNFS with the block layout type <xref target="RFC5663"/>,
the pNFS client is responsible for enforcing appropriate
correspondences between these security layers. In environments
where the security requirements are such that client-side
protection from access to storage outside of the layout is not
sufficient, pNFS with a SCSI layout on a NVMe namespace <bcp14>SHOULD
NOT</bcp14> be deployed.</t>
      <t>As with other block-oriented pNFS layout types, the metadata server
is able to fence off a client's access to the data on an NVMe namespace
used as a storage device.  If a metadata server revokes a layout, the
client's access <bcp14>MUST</bcp14> be terminated at the storage devices via fencing
as specified in <xref target="ssc:fencing"/>. target="ssc_fencing"/>.  The client has a
subsequent opportunity to acquire a new layout.</t>
    </section>
    <section anchor="sec:iana"><name>IANA anchor="sec_iana">
      <name>IANA Considerations</name>

<t>The document does not require any actions by IANA.</t>

</section>

  </middle>

  <back>

    <references title='Normative References'>

<reference anchor='RFC5663'>
  <front>
    <title>Parallel NFS (pNFS) Block/Volume Layout</title>
    <author fullname='D. Black' initials='D.' surname='Black'/>
    <author fullname='S. Fridella' initials='S.' surname='Fridella'/>
    <author fullname='J. Glasgow' initials='J.' surname='Glasgow'/>
    <date month='January' year='2010'/>
    <abstract>
      <t>Parallel NFS (pNFS) extends Network File Sharing version 4 (NFSv4) to allow clients to directly access file data on the storage used by the NFSv4 server. This ability to bypass the server for data access can increase both performance and parallelism, but requires additional client functionality for data access, some of which is dependent on the class of storage used. The main pNFS operations document specifies storage-class-independent extensions to NFS; this document specifies the additional extensions (primarily data structures) for use of pNFS with block- and volume-based storage. [STANDARDS-TRACK]</t>
    </abstract>
  </front>
  <seriesInfo name='RFC' value='5663'/>
  <seriesInfo name='DOI' value='10.17487/RFC5663'/>
</reference>

<reference anchor='RFC8154'>
  <front>
    <title>Parallel NFS (pNFS) Small Computer System Interface (SCSI) Layout</title>
    <author fullname='C. Hellwig' initials='C.' surname='Hellwig'/>
    <date month='May' year='2017'/>
    <abstract>
      <t>The Parallel Network File System (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The Small Computer System Interface (SCSI) layout type is defined in this document as an extension to pNFS to allow the use of SCSI-based block storage devices.</t>
    </abstract>
  </front>
  <seriesInfo name='RFC' value='8154'/>
  <seriesInfo name='DOI' value='10.17487/RFC8154'/>
</reference>

<reference anchor='RFC8881'>
  <front>
    <title>Network File System (NFS) Version 4 Minor Version 1 Protocol</title>
    <author fullname='D. Noveck' initials='D.' role='editor' surname='Noveck'/>
    <author fullname='C. Lever' initials='C.' surname='Lever'/>
    <date month='August' year='2020'/>
    <abstract>
      <t>This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol.</t>
      <t>This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661.</t>
    </abstract>
  </front>
  <seriesInfo name='RFC' value='8881'/>
  <seriesInfo name='DOI' value='10.17487/RFC8881'/>
</reference> IANA actions.</t>
    </section>
  </middle>
  <back>
    <references>
      <name>References</name>
      <references>
        <name>Normative References</name>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4506.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.5663.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8154.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8881.xml"/>

        <reference anchor="SPC5" > anchor="SPC5">
          <front>
            <title>SCSI Primary Commands-5</title>
    <author > Commands - 5 (SPC-5)</title>
            <author>
              <organization>INCITS Technical Committee T10</organization>
            </author>
            <date year="2019"/>
          </front>
          <seriesInfo name="ANSI INCITS" name="INCITS" value="502-2019"/>
        </reference>

        <reference anchor="NVME-BASE" target="https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-2.0c-2022.10.04-Ratified.pdf"> target="https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-2.0d-2024.01.11-Ratified.pdf">
          <front>
            <title>NVM Express Base Specification, Revision 2.0c</title>
    <author > Specification</title>
            <author>
              <organization>NVM Express, Inc.</organization>
            </author>
            <date year="2022" month="October"/> year="2024" month="January"/>
          </front>
<refcontent>Revision 2.0d</refcontent>
        </reference>

        <reference anchor="NVME-NVM" target="https://nvmexpress.org/wp-content/uploads/NVM-Express-NVM-Command-Set-Specification-1.0c-2022.10.03-Ratified.pdf"> target="https://nvmexpress.org/wp-content/uploads/NVM-Express-             NVM-Command-Set-Specification-1.0d-2023.12.28-Ratified.pdf">
          <front>
            <title>NVM Express NVM Command Set Specification, Revision 1.0c</title>
    <author > Specification</title>
            <author>
              <organization>NVM Express, Inc.</organization>
            </author>
            <date year="2022" month="October"/> year="2023" month="December"/>
          </front>
<refcontent>Revision 1.0d</refcontent>
        </reference>

        <reference anchor="NVME-TCP" target="https://nvmexpress.org/wp-content/uploads/NVM-Express-TCP-Transport-Specification-1.0c-2022.10.03-Ratified.pdf"> target="https://nvmexpress.org/wp-content/uploads/NVM-Express-TCP-Transport-Specification-1.0d-2023.12.27-Ratified.pdf">
          <front>
            <title>NVM Express TCP Transport Specification, Revision 1.0c</title>
    <author > Specification</title>
            <author>
              <organization>NVM Express, Inc.</organization>
            </author>
            <date year="2022" month="October"/>
  </front>
</reference>

<reference anchor='RFC8446'>
  <front>
    <title>The Transport Layer Security (TLS) Protocol Version 1.3</title>
    <author fullname='E. Rescorla' initials='E.' surname='Rescorla'/>
    <date month='August' year='2018'/>
    <abstract>
      <t>This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery.</t>
      <t>This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations.</t>
    </abstract>
  </front>
  <seriesInfo name='RFC' value='8446'/>
  <seriesInfo name='DOI' value='10.17487/RFC8446'/>
</reference>

<reference anchor='RFC2119'>
  <front>
    <title>Key words for use in RFCs to Indicate Requirement Levels</title>
    <author fullname='S. Bradner' initials='S.' surname='Bradner'/>
    <date month='March' year='1997'/>
    <abstract>
      <t>In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</t>
    </abstract> year="2023" month="December"/>
          </front>
  <seriesInfo name='BCP' value='14'/>
  <seriesInfo name='RFC' value='2119'/>
  <seriesInfo name='DOI' value='10.17487/RFC2119'/>
</reference>

<reference anchor='RFC8174'>
  <front>
    <title>Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words</title>
    <author fullname='B. Leiba' initials='B.' surname='Leiba'/>
    <date month='May' year='2017'/>
    <abstract>
      <t>RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.</t>
    </abstract>
  </front>
  <seriesInfo name='BCP' value='14'/>
  <seriesInfo name='RFC' value='8174'/>
  <seriesInfo name='DOI' value='10.17487/RFC8174'/>
<refcontent>Revision 1.0d</refcontent>
        </reference>

        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8446.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
      </references>

    <references title='Informative References'>
      <references>
        <name>Informative References</name>

        <reference anchor="NVME-PCIE" target="https://nvmexpress.org/wp-content/uploads/NVM-Express-PCIe-Transport-Specification-1.0c-2022.10.03-Ratified.pdf"> target="https://nvmexpress.org/wp-content/uploads/NVM-Express-PCIe-Transport-Specification-1.0d-2023.12.27-Ratified.pdf">
          <front>
            <title>NVMe over PCIe Transport Specification, Revision 1.0c</title>
    <author > Specification</title>
            <author>
              <organization>NVM Express, Inc.</organization>
            </author>
            <date year="2022" month="October"/> year="2023" month="December"/>
          </front>
<refcontent>Revision 1.0d</refcontent>
        </reference>

      </references>
    </references>
    <section numbered="false" anchor="acknowledgements"><name>Acknowledgements</name>

<t>Carsten Bormann ran the scripted conversion from the XML2RFC v2 format
used by anchor="acknowledgements">
      <name>Acknowledgements</name>
      <t><contact fullname="Carsten Bormann"/> converted an earlier drafts RFCXML v2 source for this document to the currently used a markdown source format.</t>
      <t>David Noveck provided ample feedback to various drafts of this document.</t>
    </section>

  </back>

<!-- ##markdown-source:
H4sIAAAAAAAAA71b63IaS5L+X09Ra/8wzAIWsix7FDMTBwOyiZEQA7LPeC+h
KJpCdKjp5nQ3yIRD77LPsk+2X2ZVdVc3yOec2LP7R4KmLplZefkyK7vdbouF
yvWFPD05PWufdNvdE7FIglit8WyRqmXeDnW+bMfLbHfWzoIsbEdqn2zzdrxb
6/bJO7G7kG9EoPILmeULkeWpVusLORreXopNeCEkHqdhgJ9f7XX2Ct+DZL3W
cZ6VT8I4CmNdfteLMA/jezyIE/qeJ0H5I74k643yV8Sjhd7kKzx5Q9+z/TrV
S2+DLEnz6pODNbLtvHxG2+ZhHoGmzxkokflKy4lKVRTpSI4vZ7Kxwd+mnPVn
I3nFAgEVUgWBzjI5/nKtwXaSqnstF3oX4qlQ83mqISyaWJm3TFKeIRREdyFv
UxVnGxAsHiEBDN6diYdH90mobb5K0gvRlmlC9JGskhQMmCPrr9IQO29W8pOO
osfwnsS5VmF0IVfB6qcoyzsLjclutEojnckrvdO0RpJiy5tUBZGW/SQFFSoP
kxi/OOrNjyzAbZynewgoDnO9kLMcapTJZCl7a40DV+XGwWobPHQi2uOnhOd3
IOqCiFmShrG8VOE8zFYFERsdL9JwpzMey1qkNc6m25UzHT3uYzlN1IIICXNQ
MdaPOROa6ntQfCGve0zkgtQKqn3effX7iM46S0PRT0mdlM0qIW391648775r
v+2etE/OTk/BTxhDwwYd+SFSwUNxJAO1CxfyqnzMDA5wPPJWB6s4iZL7EAri
8fjuHFLBQYNAelJw+SnZPITxDxjtvjt7/zsZXRB5nTnR9tMCRDGTIk7SNc5+
p8mCp5f9t+fnb+zH9923Z+7j+/dd+jib9N/Sf5iisRrW70karlW6hyat1ype
ZO23PMRpMH1uG2GMxv3R7cyIA8RFPCXMc63lLRwSjXReqvtn/pqBCw3TXCZm
ISl7Y2xpFrqQb09O23YsTGvY/tCbDS2BKr0nGa/yfJNdvH5NbuzbJoXZdkDK
68dNO0jiHP7p9XYTQcOy11igPTRD2h9UptuzjQ7CJegk22ifdk4C7HV62ume
dE7O2lM8XoZ60dkslr5IsIy0y0haRlaWackp/ESGT5IWfE5Q3iItOYqDTkU2
p6dtlhazjD9/BMf02R5ge6bzGvPdCvNvfhPz9NmuCFPOn5VD94+Rw21/8kfI
Acu0C9f8B0gB65Wu/v9QBmSlZ2fnF4JsxbNpls2kP/pDzALr6D9COlomCBKS
lvt/EE673UZcg9NF0BdC3K7CTAL7bAmdyMxsCqe5Sh4ptm9hslUYoPPHJH2Q
lyGi5Wyf5XptYYGYrTGElHyzzcGP/XEEAaZLFWjZIP/YdADgdr/RHnqoA4dt
gUDGkOaXJIIcsOO1Xidwrk6fGiS+ptykCQGkSC7VOoz2HfBFbK7DxQJhW7wk
ItJksQ1IlvL7y0wHFyE9ehKCEUanKxsIx9+/W/f+9NREXAui7QKiUAa9LLXK
tymJQ+USnCaPmQB2WWAAbPoxDSnWgKG5lhtwDLVDCJrvZQImUrnWOFiaG8tF
mOqAoJ6k6TRb8GyJmGugB69DvMPhQzU6Ut6u0mR7v+LzQDzL6dQsQS1vZAsK
z18JJtHAtc4Vzl/Z3yWmQXAbbBHOMYJQGFySuidqliRgosYQSHIupj+u6MdM
b6AHoNRwA+hGokckBae0lP6mgy3LOFkeyKZDyqaJfk8gINEKq6DgQKa0zUIv
AZZJnuBwEWLLLQLmWm02rCbQpYxoDFZYEQYrV2rHihumMnmMnV4bY7KUWEz6
25TWwH+rIIACT08kSmWfi5yVuVQMRstBFBLip7M0Eo32jlk5en3DuhIlwcOB
7hthz/cblRVWcO1OYmYOsnE9mDWhGKOcCHF2ywrHpiPguQKkB1CTNFnzEgxP
aqbCx0Y/8tobBfTlVK9KFGTmO3JjeC1pZx+z0U9JlkOwZGbwHGkpU1FzbSzI
DDERylA9KKZFRQ/0v0YP+0yhZLxdz7E46ZHlDNLYQg9URh61JJjcKwi+DAHp
KQWIY3izxmW/ySqPwPR6NCF2pqAeGj7gE3O89IyTakwH172mjI0PhJQ70jhw
sBBs0xTHHe0FCQTHsFQpHCv0WsGxFmKHAS88t8dOf5ZEoUWr2Dd7wEaz2SBr
tgRZAfwQTILOFMelFskmt9aWGkqdYOyKjzAuEJTj5w3gLdSJzr09V7TxoYxC
+rVTjwPG3EwUYAbHAPXZhpaTnmeuYZrv3x0Ge3oSCKQIY0YjeYlSFbDrehNp
2skuJXgEA8SKBrglCcnC5MgTGA/LYlSZqGtFSZyfbFKs6bDNM2MhLw1+4Uo0
5TnEMntVLIxvabTnVYpYzLuBj4JST62J+Xy1JYluaHD2PKNwo3CVcqeA4vO9
oB3dFojXJt7QOE9vEfihci0GTrRTRX0Pjy2BCOIEXghfF8yP/obMmBZleVg3
5iZAfYFLVhw3cvGjQw8Qt5zYH0OEn/hAyHDNOsvVPEL6aLh17tmKlgfDLQF7
IBJnZjF/IetMC3Zw1EGSmnjFgmGqyhXA/8uXMNhftjBWrq6Alvh+SwqBIJ/Z
IH+R6l8iPH8ybv9B7yXMFzHmxfXn2e2Llvkvxzf8eTr8x+fRdDigz7NPvaur
4oOwI2afbj5fDcpP5cz+zfX1cDwwk/FUVh6JF9e9r/iFzvHFzeR2dDPuXb0w
EvBPsdTykFwm9CA3yg4wEqTh3EjtQ3/y3//VPYOF/Ati0mm3+2cYiPnyHgkx
vsATxGa3JIbrMF8h7L3AuWh4J6xCwS9QmxBeFroGf5CtKF6SD4F0//TvJJn/
vJB/mQeb7tnf7ANiuPLQyazykGV2+ORgshHikUdHtimkWXlek3SV3t7Xyncn
d++hYDX6qGOgr0gOCg+RVZQI1pFZDVomFORJIRfe4AM0RHq92aabxGA28xuF
ZtgSTiBNNmlILp9zjG95MYcAEECfEH0GEOJC0qYvDJx4QX6LDRs+Jd9b0MGu
X5sfCHoYtPcq44pJlmxTiuCS1zHLwDb3pGGGlMi5W16N6FEho1WqeVGhJvBC
VnWPEtqAeoK+Ki18qcWz8T1XJxlY8QZWSpmLYAz77ABalT05ScPBAirFlVSS
3YujeMiK6uhvNbnVUXCQwCUgVltyrZQcy0mJUDxiDUqlOEIRpMBfDmxT0esx
5pOEfo23pgZFYDPe0fi6hkHuO0pIioE7FW0hCYd9fVdZxUktqTv3nRbMn2pS
T08tG5VstHpusBdaW6y+gAyQC2jDbmUkzRAGCE4WRCuSZFKP1RZ6kFORr+av
EA+Xy/AbqEdA4ixqDvGme9EgKCJPmxa6OXK63ZO5/Ks8x4QgBCg3PEA/X60O
1xIr/U3ZgdIs2D0/WBEI56/y9L1bkdTmZSVkFflDYkKLyQyzaG0NvRIzS+Tg
ZwGpXoLpmBEpeVkllvqRJ7addGQBxZ2tsCn6Lr+wiCJoCgfbszJwmkUMsQWw
LZ+zJ0hNkBbk7b0zLOvvlh0K4obpuIjyRlGB40GUHFmddkCMFXVHSBXPn/wM
ylt165zQgPFYfZEvFGfkxKTickAm2vgyGTTFhkJ2g/9SUVe+f7NqmsTFqTSx
bc1sbxKWAvGZtNDlYh05cpjLJREkGEqca+xyuKulXpB+bVhp3KlzBNu59QVE
h/ez4J1+wP5kIJnHAiqS5zHyIyerIrGFinF+6qPcg9yniC41+bfqLAoO13Y7
qUPOvGnmaDgcQoVyAr8Lqpf/si3JJZc5/Dw6P2vCqYsCBcqPUTIHYNgfGz/+
+Hk0aBqXRX7EYH92BsUCoiaPAeOZDZgzNQxKiv18YuTBczt175HD+jNjKMhl
EOvziqHy2aHimczF+EJo0FKqnQojgFmsW9RcoEHEpWUyzMSGrT81hxTmLsRE
VFZMPc2g5DmhIhwZzWR2NxjORh/Hvdub6d1t94QJrz4d966HpoYRJ5LwLbYk
KAB58gnTDEBygvpOTXz7OHYV54oe8TK7oxvNO/Kadzs29jsqk57JzAnIBN+4
pk3C4Le5B8EN5wYPZRdCdDsGq2Tz3R0Z8h0M5QVCpo4gZzebjAfUguP+zWB4
Nxve3n0YjXvTr51ygZJMeMXwHnE5Se+oyvLMaoi1VQmy/naqFJVLVVexgMe3
D3PQfMVIX3k170ALCyyEQ6WYpZwn5FV5Lh0RT6OrXs+FkIf2lCsnP13u6G1h
QfCcMJhzBD/WMUQlSWGpEqDKkm7+vBwUQRn5z8GUk1OiTEDf7sFMslEw9f/4
y9+MHN3TpfGqNKMYYe2iYSEgUsqMBxSogq2eCqsL8gBFxc5UUErGwaGdQDk7
drGbsi+bU6SXCVTPuk+npgfyo7KOm/L+2Iz6oeIIe0iHTHnS7GkdKCUYfvZ7
aBsdk0QYxC4vAQgo8pqguTTfjmAKDpcTaAVydJo31YQcbdhqTKZZkwzFggOL
SYVdDtReshKGruqsglWod3rRMmrIVcPBrHCLfVuONKXRRNBVakY1T2VS4tDO
qfqN0q3SWivFm1FBCrhdibQkWCbxkemdesaE4NnOdGBYdJwpudxGUb1ekJrE
fiEmw+lsNLsdjinNmw2nX4ZyNDZe8/CXm8+3DjCZC9gCQggH84pfjP45JTHH
W+Am8etw4dlAzEjqpcQJyrb8O6Q7ZWGnPpay53gB4VNieVNXKQB1X7w0zIJ0
c24+Rhee5ti9dOr4lA0OU3RwMyN5+a7zhqRcCXtNUajA0dV6PFU0ptPhx17T
Og7ry09OAN4bYUd3WuUEfxXIwBRaCYvY8hofcYVBl+KMAaFrs4Ewpn8ffm0K
3pec3THZbONj0jnKj3DS+S1cy2Nci5OTbsH152Lno3zLZ/kWHt99U0M+5L3P
vEvH+02sqbIIjEW15UW4tDkItDh/1FBg04lw3LFwUZlVreJuQguU+ZdDxYse
Ff0jJsiOuG7k1TldhVTJFZX9Gyor3LyrnJqV6VZAlMCx2TGZ0FzDLYUJO38V
K8DhZMuuBqJ2RWKJWH13+/FucivmCIRsqU7opuDIbNcUo+WSdMuCBcNwK64w
7YsBaC+Owgdd3Y/t3DgLk31bQA0IGMYEhqyYq7As1RFVBLabJO6IaYXQut6b
oOIlHcbDkiyEyrIkCM1Yd75WzJwh+xikZ57zQRLL1smnyhS4/R2oAETIVsV7
7yQNJYZVWkXwerSUpQIMWY3nTTgg2BqBufEpz7YlDvzO287pu063c/r2iPfx
K0xYuaSUwIksGTGh1jlXikoV50oy8VWg6myxDHztB40DpQMC2I3NiUzgwe/o
cO++3Fx9vh7eEVnSwGN3JWYCcMtFQxFrvchMhKaLUaM0dgppjN3UOiLQa5Wd
M/+YLomEr791h3jPFcncqyYSs4W7MjFINDKtIcqDiPLUtFHXGaYvlF7AoVX8
KEKcHp4Rby7qvtIuVrrKXv8fPwoQdnxTOGThL8ZtAY3p7dfJsLbE2cotMPwW
RJDpTtt7OdEuVIAAzg3VYbw1m05jHCzrBTXN0IqfQDNGMQzY5HswliRue7cP
tePn47Ww2pYL6QrEXIGzTh0T0u+UeOt4UK6KXDwj8jI6TVKt15uc8nl50j2p
P+fI3JvDpTVbvz0QuY0If7iCc92zmeXcNgfLTbzlRLncgX/0vE8hj6On87/X
KWHven9Vp7xiesBdHXzPtg2zFRcydJomCIyLLbsP5wq4psUZhrADzMWsBc8+
1f0kXkZhkLtsXuXbjKtjTp1tsjHVAd2FQ0hLg+YvGQpUlDy1Y0jDl0X24nQx
zLJtGaYtT0fbAKzLzIQzjjzdU7ZouPWUmotg46mkQO3Vry/Deyov/PkUX0RN
0/3NGZIH3A7JUMOgahMZhOtJqpGW2xYZa54tSyvPp8nASqwodi1RdD7YPAff
73XuPZKjgQEPJao8Bt469gKp6H8wzTwB0jHtCvzmC6R/adueK+BJUdA4mEth
QsckXYhuyZI8samRKPb6mcf3eXzjy899p98WUBalMO/Cmmphwiub/ajqxsHF
Htrpu7cyjOtRm5pPdF4kiUcJGzIXsvFzH/ZHfFibY5ZOqCAgnp17abuuGu6D
V3Q8OV81Tfyr4YuzI6HLENph/f9VeTOOqBeCXRFG1boOflCOLZ5m/k0UaXeh
xbzJJXzPqrBH8hf8wOKJQ1KRfPBVe2kEcwNpaMZV7yuSYLoOHd2WPWXOdIsO
IVak6sYVWUKQh2I0bu8Ize52rahVOi/A8ph9Hfc/TW/Go38bin6v/2lY5ODG
emjTbUr3cn1C04uiD87eyNhfXbte0VrlXj3gh7Vet0qnCr9G4LpfTCnLLcrV
TG9LA7uozF25b/d6AwWnQeYywFDyKjN39coWNgqcVKOpA4z8SK8EtGSNEUoR
LdmGKXGsiCvpikcSqEtJx9ghkaJyD6NdkmL5dNIv+SNkSilJcVNr71JJfewh
wQeYPjVPfLYFk8ig9MU69u/fnX6cds4657LBlwjgoIddkHGuEUxIBCbrdWPP
ROOZE26SjpXVSmtK4jl9wGDDhm2dswrAYdXruYNWsdlVinW26FZvZTE/lp04
smGuD/ntlLIz19oBNQ+TO/EvsiVdvsMpsdLE0IrFgi8KVVQcglhu48A8w7eO
vIEGMIX2Hr6kxp34YatWSRB1A1l68JEux/jeHow5iqjAyxctZIhFHxOrjHA0
QUijop5c3IeHRKCJM3TFqxbrMC4zV5wpokOUcCWjELGoiRh6ouOsbJb1mh2I
LasUlAbxWub2gmN70SdXgUVmGN+U6ngXpklsGmVgaPRCic1BYxMguMqE7XEE
u4IV2+DIZlYzqTCGi93pRUfKn8tL09HEtMyJH5+HdOfRMhvAlyxNRZkPmokh
z3DPylyU9GuNIqla0j2xy+43fveoq9YctLoVflwZLTN4bhu727VClI7e8h2r
K3YdhY01bq8QlYyDOzs7pxuwAqeU/GGQx4BrwKv5SF8nE3fYv5HB2O8xPHrx
CY3t/Qqz6/B+lUtbgaEE3cq4dAe8lwOUdOtTapTrntys9hl3YIC215hsb2dL
l7rWitQ7E9ZXqDCV92pDXXwZxWts+uWqN86apYtYLkntd0WfpgWAwrwYiNXI
OsoivKfAJCBztWJV+SAokFUSqnOq7FzYZjuHo4eRzLPQNRoRl9bpeE4Cjivi
Nny7rxYcWBSvy/0V9pKM7R7Rgb2bu3Ckh8WbFtTSS53L3GNnWljNlQq9sEDj
7ZEFqbaGklGGPnEyL4TMfcEkHgw23eJ8kIUd8iE7NzGadCh5P3KYuR/rU79Z
0OE2K8/S+ZQ1PMTvlnw2lJR9anRN55wZmGFHUmHE3fhVG64tLKz7GGrHpFkl
FOG47wq9ubyHKW8jlTr1sPXDYoAA+aYN356R6RSzSKGIcAaCGEVko0sd+CAI
CtCZtQrzLMdxGwzdHnrtUabqlVFHei8zNYtSasSuCdcuPlBCzv6G3rB7emqJ
AijbBPDIewqaFIyTaC+eiLJB1FRjnIsxnTPFEXDkyzqypiLi11SEsBP7TxNh
mbg2ARLhBTJGH6Us3GmSgyHTt1ZreQ9ZcQV3NAWmoOTpV+V6MOHuiWoSYlRO
2BtJT+V6BnVaW2Nxt5M0NM1cvIMn/MykJjV0SsmcQ4Vc3ALpSx/jljwWGnws
UxJFplS/DeRLcnXwVkqqd8mDLt+nYOpEfVt3T5cTyozZydp2m7pD3IXKVVyE
8t+O4Bjl38k+VbsiVya9owafX7bcFc6FfurKMXjf1t0UQstjeddHbxj1xr3j
yUuoYmXvfg8bta2ycRXeJQ8I7LSae4tpTu/PYode8BAnj8hO741uiu8X29h0
vOkFNuirlG4h5AdywzHyPWUvZLnHBsxzC1+aFRpLP/7z+uqU2gZ2p9K4b+Fu
crRKI0qy+XX44syLtxxMiF6r9GHBL9hwj6ldA5Sb13/HQA+w+gLsKOrMwsno
BTHFfgM+jG567C7u1aaiQV2I/wFd0piarT8AAA==

-->
</rfc>