We will start by defining the major system components from an administrator or user's perspective. A PVFS2 file system may consist of the following pieces (some are optional): the pvfs2-server, system interface, management interface, Linux kernel driver, pvfs2-client, and ROMIO PVFS2 device.
The pvfs2-server is the server daemon component of the file system. It runs completely in user space. There may be many instances of pvfs2-server running on many different machines. Each instance may act as either a metadata server, an I/O server, or both at once. I/O servers store the actual data associated with each file, typically striped across multiple servers in round-robin fashion. Metadata servers store meta information about files, such as permissions, time stamps, and distribution parameters. Metadata servers also store the directory hierarchy.
Initial PVFS2 releases will only support one metadata server per file system, but this restriction will be released in the future.
The system interface is the lowest level user space API that provides access to the PVFS2 file system. It is not really intended to be an end user API; application developers are instead encouraged to use MPI-IO as their first choice, or standard Unix calls for legacy applications. We will document the system interface here, however, because it is the building block for all other client interfaces and is thus referred to quite often. It is implemented as a single library, called libpvfs2. The system interface API does not map directly to POSIX functions. In particular, it is a stateless API that has no concept of open(), close(), or file descriptors. This API does, however, abstract the task of communicating with many servers concurrently.
The management interface is similar in implementation to the system interface. It is a supplemental API that adds functionality that is normally not exposed to any file system users. This functionality is intended for use by administrators, and for applications such as fsck or performance monitoring which require low level file system information.
The Linux kernel driver is a module that can be loaded into an unmodified Linux kernel in order to provide VFS support for PVFS2. Currently this is only implemented for the 2.6 series of kernels. This is the component that allows standard Unix applications (including utilities like ls and cp) to work on PVFS2. The kernel driver also requires the use of a user-space helper application called pvfs2-client.
pvfs2-client is a user-space daemon that handles communication between PVFS2 servers and the kernel driver. Its primary role is to convert VFS operations into system interface operations. One pvfs2-client must be run on each client that wishes to access the file system through the VFS interface.
The ROMIO PVFS2 device is a component of the ROMIO MPI-IO implementation (distributed separately) that provides MPI-IO support for PVFS2. ROMIO is included by default with the MPICH MPI implementation and includes drivers for several file systems. See http://www.mcs.anl.gov/romio/ for details.