NVMe storage: commands, PCIe lanes and queues

NVMe defines storage commands and queues, commonly carried over PCIe for local SSDs, while slot wiring and workload determine usable performance.

NVMe, short for Non-Volatile Memory Express, defines how host software submits storage commands and receives completions. Local NVMe SSDs commonly communicate over PCI Express. NVMe also has network transport specifications, so “NVMe” and “PCIe” are not interchangeable names for the entire standard.

Its queue-based design allows multiple outstanding operations without using the older SATA command interface. That is useful for parallel storage workloads, but an application must actually issue enough work to use that concurrency.

Three specifications on one product label

Label Question it answers
SSD What kind of storage medium is used?
NVMe What command interface is used?
PCIe generation and lane count What local transport capacity is available?
M.2, U.2 or an add-in card What physical arrangement is used?

M.2 does not guarantee NVMe. An M.2 slot can support particular protocols, lengths and keying arrangements, and an M.2 device can be SATA. Compatibility depends on the board and device specifications together.

Lanes establish a transport boundary

Consider PCIe 3.0 at 8 GT/s per lane with 128b/130b encoding. An original four-lane calculation gives:

8 × 10^9 transfers/s × 128/130 ÷ 8 × 4 lanes
≈ 3.938 × 10^9 bytes/s in one direction

This is an encoding-adjusted transport ceiling. PCIe packets, NVMe commands and the device's operation reduce useful file throughput. It is not an SSD benchmark.

A drive designed for four lanes may operate with fewer lanes or an older generation in a particular slot. Several devices can also share an upstream connection. The negotiated connection and topology matter more than the largest number printed on the box.

Outstanding commands and response time

With more than one command in flight, a device can work on operations in parallel or schedule them efficiently. A library scan, simultaneous file access and application-data activity may provide that work. One small read that must finish before the next starts still depends on its individual latency.

Increasing queue depth can increase aggregate throughput while also increasing the time a request waits. For an interactive server, those are different outcomes. A high-depth benchmark does not predict how quickly the next poster or database lookup will appear.

NVMe does not remove flash write management, storage caches or thermal behavior. It gives a capable storage device an efficient command and transport arrangement; the benefit still follows the actual work the server asks it to perform.

See Also

References