Lustre

From HPC documentation portal
Jump to: navigation, search

Lustre is a scalable cluster file system, a high performance, parallel I/O file system. If the correct library for I/O is used (i.e. MPI-IO), reading and writing can be done in parallel from several nodes.

Lustre terminology and setup

  • MDS is the MetaData Server which handles the information about files and directories.
  • OST is the Object Storage Target, which is responsible for writing or reading the actual data to and from disk.

Hexagon setup

/work on Hexagon have 36 OSTs which are sharing the load of reading and writing data to disk, while one MDS handles the information about the files and directories created.

Lustre file striping defines the number of OSTs a file is written across. By default a file will be

  • striped over 2 OSTs
  • in chunks of 1MB

A user can use the lfs command to set the striping patterns of a single file, or a directory. Note that the changed striping can only take effect for newly created files or files that are copied (not moved) into the directory.

Useful sub commands of lfs are: getstripe, setstripe, df, and help.

To set the stripe size of a directory, use the syntax:

lfs setstripe --size X "dir"

Where X is the number of bytes per OSTs (defaults to 1M).

To set the stripe count of a directory, use the syntax:

lfs setstripe --count X "dir"

Where X is the number of OSTs to stripe over (defaults to 2, maximum is 36)

These two can of course be combined. Do NOT set the "start" or "index" OST it may damage performance of the file system and prematurely fill it up.

When a file has been created the striping is set. It is not possible to change this later unless it is copied to a directory with a different striping default. Files that needs a particular striping can also be created with the lfs command. If not stated otherwise the file will inherit the striping from the directory.

Tips

Here are some Lustre striping tips.

  • Many clients and many files: Do NOT stripe.
  • Many clients one file: Do stripe.
  • Some clients and few large files: Do stripe