AtRackTec is Attractec’s building blocks and concepts for data centers and edge servers.
Dense Array of M.2 NVMEs in an EPYC Server
Introduction
Attractec was requested to propose a concept for NVME M.2 storage in a 1U half-width ruggedized EPYC 9005 server.
We propsed a concept of a dense array consisting of 32 NVME devices.
Value proposition
The array of M.2 NVME provides very high storage capacity (256 TB) with extremely high bandwidth (over 400 GB/s) while utilizing cost-effective readily-available NVME devices and maximizing usability with tool-free hot swapping through the front panel.
Illustration of concept
Preliminary specifications
| Devices | Up to 32x M.2 2280 |
| Storage capacity | Up to 256 TB (32 x 8 TB) |
| Interface | 32 to 128 lanes PCIe Gen 5 (x1 / x2 / x4 per device) |
| Theoretical bandwidth | 126 GB/s (32 lanes) to 504 GB/s (128 lanes) |
| Physical Footprint | 40 mm (h) x 100 mm (w) x 100 mm (d) |
| Total power | Approx. 300W (depending on NVME devices and utilization) |
| Features | Front panel hot-swapping Tool-free assembly Indicator LED per device |
Advantages over EDSFF
Traditional high-density storage in data centers is based on E3.S or E1.S form factors.
We believe that M.2 NVMEs may have several advantages that make them worthwhile -storage capacity, bandwidth, granularity, availability and cost per GB.
Storage capacity
Bandwidth
The NVME array can provide high bandwidth because 32 PCIe ports can be used independently and each can be Gen5. If 112 PCIe Gen 5 lanes are allocated to storage, the aggregate theoretical storage bandwidth is over 400 GB/s.
EDSFF has a much smaller number of devices which are usually limited to PCIe Gen 4, so it is difficult to achieve more than 50 GB/s in this setup.
Granularity
32 devices of relatively small capacity have the benefits of high granularity, among them gradual increase of capacity, physical data segregation, flexible redundancy and fail-over schemes and quick swapping of a failing device.
Market Availability
M.2 NVME devices are readily available from many sources.
EDSFF devices are less common. Purchasing often involve longer lead time.
Cost per GB
M.2 NVME market is highly competitive and benefits from economies of scale.
EDSFF is considered a premium enterprise product with higher cost per GB.
Variations
The concept can be extended to 10 PB in 4U (100 PB in a 42U rack) using more sophisticated cooling and somewhat reduced serviceability.