parallel FS for HPTC cluster

In a HPC cluster, one need parallel FS for the compute node to access the data

This weblog will try to summary the different parallel FS existing today.

  1. NFS: most universal parallel FS under IP , NFSv4 has many new features
    • state, An NFSv4 client uses state to notify an NFSv4 server of its intentions on a file: locking, reading, writing..
    • byte-range locking and share reservation
    • file delegation. An NFSv4 server can allow an NFSv4 client to access and modify a file in its’s own cache without sending any network request until another client try to access the same file
    • compound RPCs into single RPC request
    • protocal support for file migration and replication
    • combines disparate NFS protocols (stat, NLM, mount, ACL and NFS) into a single protocol
  2. QFS, SUN, run on Solaris/SPARC, require FC-SAN
    • separate metadata
    • DAU, 4k, 15k,32k,64K
    • Direct IO capability
    • SC support
  3. PVFS, PVFS2, on linux cluster, opensource
    • Distributed metadata
    • Supoort TCP/IP, infiniband , myrinet
    • User-controlled stripping of files across nodes
    • Multiple interfaces , MPI-IO via ROMIO
    • Support for heterogenous clusters
  4. GPFS, IBM, on AIX and linux support FC-SAN or SAN Simulation
    • Scalable FS
    • Parallel FS, store blocks of a single file in multiple nodes
    • Shared FS, use token management system to ensure data consistency
    • HA FS, provides automated recovery
    • Client side data caching
    • large blocksize options 16k, 64k, 256k,513k,1024k
    • Blocking-level locking, DLM
    • Data protection, metadata and user data can be replicated
  5. Lustre FS, on linux,
    • support TCP, Quadrics, Elan, Myrinet GM, SCI, SDP and Infiniband interconnect,
    • opensource
    • metadta server and Object storage server
    • support 1000+ nodes
  6. RedHat Global FS, on linux,
    • use SAN constructed with iSCSI or Fibre Channelt,
    • opensource
    • support direct IO
    • support 100+ nodes
    • has no single-point-of-failure
  7. CXFS, SGI, support IRIX, Solaris and Windown NT, require FC-SAN
  8. OpenAFS,
    • support Solaris sparc/x86, many other OS
    • opensource
  9. TerraGrid from Terrascale
    • clients running TerraGrid iSCSI Initiators
    • Storage nodes running TerrGrid iSCSI Targets
    • no need of metadata
    • buildin HA
    • support linux, amd and em64T

About laotsao 老曹

HopBit GridComputing LLC Rockscluster Gridengine Solaris Zone, Solaris Cluster, OVM SPARC/Ldom Exadata, SPARC SuperCluster
This entry was posted in HPTC. Bookmark the permalink.

One Response to parallel FS for HPTC cluster

  1. anonymous poster says:

    Do not forget about OpenAFS (, a distributed file system. No more of a parallel FS than NFS

Comments are closed.