fast-file-zfs-beegfs

Architecture and configurations for Fred Hutch "Fast File" service, our HPC posix file system

View the Project on GitHub FredHutch/fast-file-zfs-beegfs

Home

Background

We decided to replace our existing Scaleout NAS solution for our primary posix file system with commodity equipment. Drivers are consolidation / reduction of the number of technologies that need to be supported (we have been running a Scratch file system using ZFS/BeeGFS for 4 years) and cost (Fred Hutch charges Investigators for Storage).

We would like to share our journey over the next few months. Please feel free to add questions or comments in this blog.

The project started out with purchasing some Hardware. We will be using the BeeGFS buddy mirror option which requires a mirror server for every primary storage server. The config has about 8PB raw which will shrink down to under 3PB usable space once installed. This seems like a waste at first but the entire hardware costs only about $2 / TiB usable / month when calculated over 5 years.

Hardware

Primary Storage

8 Server x 8 core 476TB raw disk, 256GB RAM each

  • Supermicro X10DRH-iT - Intel C612 Chipset - 10x SATA - Dual Intel 10G (RJ45)
  • 2 x Quad-Core Intel Xeon Processor E5-2637 v4 3.50GHz 15MB Cache (135W)
  • Supermicro SuperChassis 847BE1C-R1K28LPB - 4U - 36 x 3.5” SAS3/SATA
  • 16 x HYNIX 16GB PPC4-21300 2666MHz
  • 34 x 14TB SAS 3.0 12.0Gb/s 7200RPM - 3.5” - Ultrastar DC HC530 (512e)
  • 2 x 1.92TB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive
  • 2 x 240GB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive
  • LSI SAS 9300-8i SAS 12Gb/s PCIe 3.0 8-Port Host Bus Adapter

Mirror Storage

8 Server x 8 core, 476TB raw disk, 256GB RAM each

  • Supermicro X10DRH-iT - Intel C612 Chipset - 10x SATA - Dual Intel 10G (RJ45)
  • 2 x Quad-Core Intel Xeon Processor E5-2637 v4 3.50GHz 15MB Cache (135W)
  • SC847B 4U RM BLACK LP 1200W PWS
  • 16 x HYNIX 16GB PPC4-21300 2666MHz
  • 34 x 14.0TB SAS 3.0 12.0Gb/s 7200RPM - 3.5” - Seagate Exos X14 (512e/4Kn)
  • 1.92TB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive
  • 2 x 240GB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive
  • LSI SAS 9300-8i SAS 12Gb/s PCIe 3.0 8-Port Host Bus Adapter

Metadata Servers

2 Server x 36 core, 3TB Optane disk, 96GB RAM each

  • Supermicro Ultra SuperServer 6029U-TRT - 2U - 12x SATA - Dual 10G (RJ45)
  • 2 x Intel Xeon Gold 6154 Processor 18-core 3.00GHz 24.75MB Cache (200W)
  • 12 x 8GB PC4-21300 2666MHz DDR4 ECC Registered DIMM
  • LSI 9341-4I 12GB/S SAS RAID CA
  • 2 x 400GB HGST Ultrastar DC SS200 Series 2.5 SAS 12.0Gb/s Solid State Drive (3x DWPD)
  • 2 x 1.5TB Intel Optane SSD DC P4800X Series U.2 PCIe 3.0 x4 NVMe Solid State Drive

Management Servers (mgmt/admin)

2 Server x 4 core, 2TB disk, 64GB RAM each

  • Supermicro X11SSH-F - uATX - Intel C236 Chipset Dual Intel 1G (RJ45)
  • Quad-Core Intel Xeon Processor E3-1220 v6 3.00GHz 8MB Cache (72W)
  • 4 x 16GB PC4-21300 2666MHz DDR4 ECC UDIMM
  • Supermicro SuperChassis 825TQC-R740LPB - 2U - 8 x 3.5” SAS3/SATA - 740W Redundant
  • 2 x 960GB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive
  • LSI MegaRAID 9341-8i SAS 12Gb/s PCIe 3.0 8-Port Controller
  • Intel 10-Gigabit Ethernet Converged Network Adapter X540-T2 (2x RJ-45)

NAS / DR Gateway Servers

2 Server x 12 core, 8TB disk + spares, 768GB RAM each

  • Supermicro Ultra SuperServer 6029U-TRT - 2U - 12x SATA - Dual 10G (RJ45)
  • 2 x Intel Xeon Gold 6128 Processor 6-core 3.40GHz 19.25MB Cache (115W)
  • 24 x 32GB PC4-21300 2666MHz DDR4 ECC Registered DIMM
  • 4 x 14.0TB SAS 3.0 12.0Gb/s 7200RPM - 3.5” - Seagate Exos X14 (512e/4Kn)
  • 4 x 14TB SAS 3.0 12.0Gb/s 7200RPM - 3.5” - Ultrastar DC HC530 (512e)
  • 2 x 3.84TB Intel SSD D3-S4610 Series 2.5” SATA 6.0Gb/s Solid State Drive
  • 2 x 240GB Micron 5200 MAX Series 2.5” SATA 6.0Gb/s Solid State Drive