Frequently Asked Questions about PVFS2
This section covers some basic questions for people who are unfamiliar with PVFS2.
PVFS2 is an open-source, scalable parallel file system targeted at production
parallel computation environments. It is designed specifically to scale to
very large numbers of clients and servers. The architecture is very modular,
allowing for easy inclusion of new hardware support and new algorithms. This
makes PVFS2 a perfect research testbed as well.
The ``V'' in PVFS stands for virtual. This is a holdover from the original
(PVFS1) project that built a parallel file system on top of local file
systems, which we still do now. It isn't meant to imply virtualization of
storage, although that is sort of what the file system does.
No, and we're not even sure what that means! The design of PVFS2 does
not depend on the design of the traditional *NIX Virtual Filesystem
Switch (VFS) layer, although we provide a compatibility layer that
allows access to the file system through it.
The PVFS2 Guide (http://www.pvfs.org/pvfs2/pvfs2-guide.html) has more
information on all of these components, plus a discussion of the system as a
whole, the code tree, and more.
This section covers questions related to particular system architectures,
operating systems, and other hardware.
Other than hardware supported by the Linux OS, no. PVFS2 uses
existing network infrastructure for communication and can currently
operate over TCP, Myrinet, and InfiniBand. Disk local to servers is
used for PVFS2 storage, so no storage area network (SAN) is required
either (although it can be helpful when setting up fault tolerant solutions;
see Section 7.
2.2 What architectures does PVFS2 support?
The majority of PVFS2 is POSIX-compliant C code that runs in user
space. As such, much of PVFS2 can run on most available systems. See
Question 2.5 for more information on particular
hardware.
The (optional) part of PVFS2 that hooks to the operating system on
clients must be written specifically for the particular operating
system. Question 2.4 covers this issue.
Yes! The ``language'' that PVFS2 uses to talk between clients and
servers is encoded in a architecture-independent format (little-endian
with fixed byte length parameters). This allows different PVFS2
components to interact seamlessly regardless of architecture.
2.4 Does running PVFS2 require a particular kernel or kernel
version?
You can run the userspace PVFS2 servers and administration tools on
every major GNU/Linux distribution out of the box, and we intend to
keep it that way.
However, the kernel module that allows client access to the PVFS2 system
does depend on particular kernel versions because it builds against
the running one (in the same manner as every other Linux module).
The kernel dependent PVFS2 client support has been written for Linux
kernel versions 2.4.19 (and greater) and 2.6.0 (and greater). At this
time only Linux clients have this level of support.
2.5 What specific hardware architectures are supported by the
PVFS2 kernel module?
To our knowledge, PVFS2 has been verified to be working on x86/IA-32,
IA-64, AMD64, PowerPC (ppc), and Alpha based GNU/Linux distributions.
No. The kernel module source included with PVFS2 is generally
targetted toward the official ``Linus'' kernels (found at kernel.org).
Patches for the PVFS2 kernel module code may be provided for major
distributions that have modified their kernel to be incompatible with
the officially released kernels. The best place to find out more
information about support for a kernel tied to a particular
distribution is on the PVFS2-developers mailing list.
No, this is currently not supported nor recommended.
At this time we have no plans for porting the code to operating
systems other than Linux. However, we do encourage porting efforts of
PVFS2 to other operating systems, and will likely aid in the
development.
This section covers issues related to installing and configuring PVFS2.
The PVFS2 Quick Start Guide
(http://www.pvfs.org/pvfs2/pvfs2-quickstart.html) provides an overview
of both a simple, single-server installation, and a more complicated,
multi-server configuration.
3.2 How can I store PVFS2 data on multiple disks on a single node?
There are at least two ways to do this.
In general the best solution to this problem is going to be to get the disks
logically organized into a single unit by some other OS component, then build
a file system on that single logical unit for use by the PVFS2 server on that
node.
There are a wide array of hardware RAID controllers that are capable of
performing this task.
The Multiple Devices (MD) driver is a software component of Linux that can be
used to combine multiple disk drives into a single logical unit, complete with
RAID for fault tolerance.
Using the Logical Volume Management (LVM) component of the Linux OS is another
option for this (see the HOWTO at
http://www.tldp.org/HOWTO/LVM-HOWTO.html). LVM would also allow you to
add or remove drives at a later time, which can be quite convenient. You
can of course combine the MD and LVM components in interesting ways as well,
but that's outside the scope of this FAQ.
There's an EVMS program that can be used for managing local storage; this
might be useful for setting up complicated configurations of local storage
prior to starting up PVFS2 servers.
A second solution would be to use more than one server on the same node, each
using a different file system to store its data. This might lead to resource
contention issues, so we suggest trying other options first.
If you do decide to run more than one PVFS2 server on the same node,
setting things up is as simple as setting up servers on different
nodes. Each will need its own entry in the list of Aliases and its
own server-specific configuration file, as described in the Quick Start
(http://www.pvfs.org/pvfs2/pvfs2-quickstart.html).
Absolutely! Any PVFS2 server can store either metadata, data, or both.
Simply allocate unique MetaHandleRanges for each server that you would like to
store metadata; the clients will handle the rest.
Unfortunately, no. While using multiple metadata servers distributes
metadata, it does not replicate or store redundant information across
these servers. For information on better handling failures, see
Section 7.
3.6 How many servers should I run?
Really, the answer is ``it depends'', but here are some factors you
should take into account.
Running multiple metadata servers might help if you expect to have have
a lot of small files. The metadata servers are not involved in data
access (file contents) but do have a role in file creation and lookup.
Multiple clients accessing different files will likely access different
metadata servers, so you could see a load balancing effect.
A good rule of thumb is you should run as many data servers as possible.
One common configuration is to have some nodes with very
high-performance disks acting as servers to the larger cluster. As you
use more servers in this configuraiton, the theoretical peak performance
of PVFS2 increases. The clients, however, have to make very large
requests in order to stripe the I/O across all the servers. If your
clients will never write large files, use a smaller number of servers.
If your clients are writing out gigantic checkpoint files or reading in
huge datasets, then use more servers.
It is entirely possible to run PVFS2 servers on the same nodes doing
computation. In most cases, however, you will see better performance if
you have some portion of your cluster dedicated to IO and another portion
dedicated to computation.
PVFS2 servers currently listen on one interface at a time. Multihome
support, where a PVFS2 server can accept connections from several
network interfaces, would be helpful in many situations. We plan to
implement this feature in the future.
The Linux automounter needs some help dealing with PVFS2's resource strings. A
typical mount command (on Linux 2.6) would look like this:
mount -t pvfs2 tcp://server0:3334/pvfs2-fs /mnt/pvfs2
The entry in the automount config file should look like this:
pvfs -fstype=pvfs2 :tcp://server0\:3334/pvfs2-fs
Note the backslash-escape of the colon before the port number. Without that
escape, the automounter will get confused and replace 'tcp://' with
'tcp:///'
3.9 Can I mount more than one PVFS2 file system on the same client?
Yes. However, when setting up the two file systems it is important that both
file systems have unique Name and ID values (in the
file system configuration file). This means that you can't simply make a copy
of the fs.conf generated by pvfs2-genconfig; you will need
to edit the files a bit. This editing needs to be performed before you
create the storage spaces!
Hans Reiser summarized the upgrade approach from reiserfs V3 to V4 with the following:
To upgrade from reiserfs V3 to V4, use tar, or sponsor us to write a convertfs.
Similarly, there are no tools currently provided by the PVFS2 team to
upgrade from PVFS1 to PVFS2, so tar is your best bet.
This section outlines some steps that will help the developers figure out what
has happened when you have a problem.
The best place to look for documentation on PVFS2 is the PVFS2 web site at
http://www.pvfs.org/pvfs2. Documentation (including this FAQ) is also
available in the doc subdirectory of the PVFS2 source distribution.
The first thing to do is to check out the existing documentation and see if it
addresses your problem. We are constantly updating documentation to clarify
sections that users have found confusing and to add to this document answers
to questions that we have seen.
The next thing to do is to check out the PVFS2 mailing list archives at
http://www.pvfs.org/pvfs2/lists.html. It is likely that you are not
the first person to see a particular problem, so searching this list will
often result in an immediate answer.
If you still haven't found an answer, the next thing to do is to mail the
mailing list and report your problem.
If you enjoy using IRC, you can also join us on irc.freenode.net in
the #pvfs2 channel.
First you will need to join the PVFS2 Users Mailing list at
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users. You
must be a user to post to the list; this is necessary to keep down the amount
of spam on the list.
Next you should gather up some information regarding your system:
- Version of PVFS2
- Version of MPI and MPI-IO (if you're using them)
- Version of Linux kernel (if you're using the VFS interface)
- Hardware architecture, including CPU, network, storage
- Any logs that might be useful to the developers
Including this information in your first message will help the developers most
quickly help you. You are almost guaranteed that if you do not include this
information in your first message, you will be asked to provide it in the
first reply, slowing down the process.
You should be aware that you are also likely to be asked to try the newest
stable version if you are not running that version. We understand that this
is not always possible, but if it is, please do.
Note: Please do not send your message to both the PVFS2 Users List and
the PVFS2 Developers List; the lists serve different purposes. Also, please
do not send your message directly to particular developers. By keeping
discussion of problems on the mailing lists we ensure that the discussion is
archived and that everyone has a chance to respond.
This section covers error conditions you might encounter, what they might mean,
and how to fix them.
First, make 100% sure you typed the mount command correctly. As discussed in
the PVFS2 quickstart, different mount commands are needed for linux-2.4 and
linux-2.6. A linux-2.6 mount command will look like this:
prompt# mount -t pvfs2 tcp://testhost:3334/pvfs2-fs /mnt/pvfs2
Under linux-2.4, the mount command looks slightly different:
prompt# mount -t pvfs2 pvfs2 /mnt/pvfs2 -o tcp://testhost:3334/pvfs2-fs
This error could also mean a pvfs2-client process is not running,
either because it was not started before the mount command, or was
terminated at some point. If you can reliably (or even intermittently)
cause the pvfs2-client to exit abnormally, please send a report to the
developers.
This error can also occur if you attempt to mount a second PVFS2 file system
on a client, where the new file system has the same name or ID as one
that is already mounted. If you are trying to mount more than one file system
on the same client and have problems, please see question
3.9.
Finally, be sure there are no typos in your command line, as this is
commonly the case!
On some systems, the pvfs2-server will start consuming 100% of the CPU after
you try to read or write a file to PVFS2. Please check to see if your
distribution has an updated glibc package. RHEL3, for example, will exhibit
this behavior with glibc-2.3.2-95.6, but not with the updated glibc-2.3.2-95.20
package.
This section covers issues related to the performance of PVFS2.
6.1 I ran Bonnie and/or IOzone and the performance is terrible.
Why? Is there anything I can do?
We designed PVFS2 to work well for scientific applications in a cluster
environment. In such an environment, a file system must either spend
time ensuring all client-side caches are in sync, or not use a cache
at all (which is how PVFS2 currently operates). The bonnie
and bonnie++ benchmarks read and write very small blocks -
on the order of 1K. These many small requests must travel from the
client to the server and back again. Without client-side caching,
there is no sane way to speed this up.
To improve benchmark performance, specify a bigger block size. PVFS2
has several more aggressive optimizations that can be turned on, but
those optimizations require that applications accessing PVFS2 can cope
with out-of-sync caches.
In the future, PVFS2 is looking to provide optional semantics for use
through the VFS that will allow some client-side caching to speed these
kinds of serial benchmarks up. By offering a way to explicitly sync
data at any given time or by providing 'close-to-open' semantics, these
kinds of caching improvements become an option for some applications.
Bear in mind that benchmarks such as IOzone and Bonnie were meant to
stress local file systems. They do not accurately reflect the types of
workloads for which we designed PVFS2. Furthermore, because of their
serial nature, PVFS2 will be unable to deliver its full performance.
Instead try running a parallel file system benchmark like IOR
(http://www.llnl.gov/asci/purple/benchmarks/limited/ior/).
See Question 6.1. If the program uses small block sizes to
access a PVFS2 file, performance will suffer.
Setting both (or either of) the TroveSyncMeta and
TroveSyncData options to no in the config file can
improve performance in some situations. By default these values are
both set to yes, which means that server will issue a sync
call after each metadata or data update, respectively. If you set the
value to no and the server is terminated unexpectedly, you will likely
lose data (or access to it). Also, PVFS2 has a transparent server
side attribute cache (enabled by default), which can speed up
applications which read a lot of attributes (ls, for
example). Playing around with the AttrCache* config file
settings may yield some performance improvements. If you're running a
serial application on a single node, you can also use the client side
attribute cache (disabled by default). This timeout is adjustable as
a command line argument to pvfs2-client.
In an environment where there is one client accessing a file on one
server, NFS will outperform PVFS2 in many benchmarks. NFS has
completely different consistency semantics, which work very well when
just one process accesses a file. There is some ongoing work that
will optionally offer similar consistency semantics for PVFS2, at
which point we will be playing on a level field, so to speak.
However, if you insist on benchmarking PVFS2 and NFS in a
single-client test, there are some immediate adjustments you can make.
The easiest way to improve PVFS2 performance is to increase the block
size of each access. Large block sizes help most file systems, but
for PVFS2 they make a much larger difference in performance than they
do for other file systems.
Also, if the TroveSyncMeta and TroveSyncData options
are set to no in your PVFS2 configuration file, the server
will sync data to disk only when a flush or close operation is called.
This option is set to yes by default, to limit the amount of
data that could be lost if a server is terminated unexpectedly. With
this option enabled, it is somewhat analogous to mounting your NFS
volume with the sync flag, forcing it to sync data after each
operation.
As a final note on the issue, if you plan on running application XXX,
or a similar workload, and the NFS consistency semantics are adequate
for what you're doing, then perhaps PVFS2 is not a wise choice of file
system for you. PVFS2 is not designed for serial workloads,
particularly one with small accesses.
Yes! However, the interaction between the PVFS2 servers and the local
file system hosting the storage space has not been fully explored.
No doubt a great deal of time could be spent on different file systems
and their parameters.
People have looked at sync performance for a variety of file systems.
Some file systems will flush all dirty buffers when fsync is
called. Other file systems will only flush dirty buffers belonging to
the file. See the threads starting at
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2004-July/000740.htmland at
http://www.parl.clemson.edu/pipermail/pvfs2-developers/2004-July/000741.html.
These tests demonstrate wide variance in file system behavior.
Interested users are encouraged to experiment and discuss their
findings on the PVFS2 lists.
If you're looking for a quick suggestion for a local file system type to use, we
suggest ext3 with ``journal data writeback'' option as a reasonable choice.
7 Fault Tolerance
This section covers issues related to fault tolerance in the context of PVFS2.
Systems can be set up to handle many types of failures for PVFS2. Given enough
hardware, PVFS2 can even handle server failure.
Yes. One of the benefits of the PVFS2 design is that client failures are not a
significant event in the system. Because there is no locking system in PVFS2,
and no shared state stored on clients in general, a client failure does not
affect either the servers or other clients.
Yes, if configured to do so. Multiple disks on each server may be used to
form redundant storage for that server, allowing servers to continue operating
in the event of a disk failure. See section 3.2 for more
information on this approach.
Yes, if your network has redundant links. Because PVFS2 uses standard
networks, the same approaches for providing multiple network connections to a
server may be used with PVFS2. Need a reference of some sort.
Yes. We currently have a recipe describing the hardware and software
needed to set up PVFS2 in a high availability cluster. Our method is
outlined in the `pvfs2-ha.{ps,pdf}' file in the doc subdirectory of the
PVFS2 distribution. This configuration relies on shared storage and
commodity ``heartbeat'' software to provide means for failover.
Software redundancy offers a less expensive solution to redundancy,
but usually at a non-trivial cost to performance. We are studying how
to implement software reduncancy with lower overhead, but at this time
we provide no software-only server failover solution.
This section covers issues related to accessing PVFS2 file systems.
The ROMIO MPI-IO implementation, as provided with MPICH2 and others, supports
PVFS2. You can find more information in the ROMIO section of the
pvfs2-quickstart: http://www.pvfs.org/pvfs2/pvfs2-quickstart.html#sec:romio
You can, yes, but you probably should not. The PVFS2 developers are not
likely to help you out if you do this and something gets messed up...
This section covers questions about managing PVFS2 file systems.
The default storage implementation for PVFS2 (called Trove DBPF for ``DB Plus
Files'') stores all file system data held by a single server in a single
subdirectory. In that subdirectory is a directory tree containing UNIX files
with file data and metadata.
This entire directory tree can be backed up in any manner you like and
restored if problems occur.
As a side note, this was not possible in PVFS1, and is one of the many
improvements present in the new system.
You can add and change the order of PVFS2 servers for an existing PVFS2 file
system. At this time, you must stop all the servers in order to do so.
To add a new server:
- Unmount all clients
- Stop all servers
- Edit your config file to:
- Add a new Alias for the new server
- Add a new DataHandleRange for the new server (picking a range you
didn't previously use)
- Deploy the new config file to all the servers, including the new one
- Create the storage space on the new server
- Start all servers
- Remount clients
To reorder the servers (causing round-robin to occur in a different relative
order):
- Unmount all clients
- Stop all servers
- Edit your config file to reorder the DataHandleRange entries
- Deploy the new config file to all the servers
- Start all servers
- Remount clients
Note that adding a new server will not cause existing datafiles to be
placed on the new server, although new ones will be (by default). Migration
tools are necessary to move existing datafiles (see
Question 9.3) both in the case of a new server, or if you
wanted to migrate data off a server before removing it.
9.3 Are there tools for migrating data between servers?
Not at this time, no.
PVFS2 uses a particular algorithm for calculating the free space on a file
system that takes the minimum amount of space free on a single server and
multiplies this value by the number of servers storing file data.
This algorithm was chosen because it provides a lower-bound on the amount of
data that could be stored on the system at that point in time.
If this value seems low, it is likely that one of your servers has less space
than the others (either physical space, or because someone has put some other
data on the same local file system on which PVFS2 data is stored). The
pvfs2-statfs utility, included with PVFS2, can be used to check the
amount of free space on each server, as can the karma GUI.
PVFS2 uses a 64-bit value for describing the offsets into files, so
theoretically file sizes are virtually unlimited. However, in practice other
system constraints place upper bounds on the size of files and file systems.
To best calculate maximum file and file system sizes, you should determine the
maximum file and file system sizes for the local file system type that you are
using for PVFS2 server storage and multiply these values by the number of
servers you are using.
This section discusses features that are not present in PVFS2 that are present
in some other file systems.
We didn't implement hardlinks, and there is no plan to do so. Symlinks are
implemented.
Private, read-only mmapping of files is supported. Shared mmapping of files
is not. Supporting this would force a great deal of additional infrastructure
into PVFS2 that would compromise the design goals of simplicity and
robustness. This ``feature'' was intentionally left out, and it will remain
so.
No. Currently PVFS2 does not intelligently place new files based on free
space. It's a good idea, and possible, but we have not done this yet. See
Section 11.1 for notes on how you could help get this
feature in place.
No. Locking subsystems add a great deal of shared state to a parallel file
system implementation, and one of the primary design goals was to eliminate
shared state in PVFS2. This results in a simpler, more fault tolerant
overall system than would have been possible had we integrated locking into
the file system.
It's possible that an add-on locking subsystem will be developed at some point;
however, there is no plan to build such a system at this time.
This section covers ways one could contribute to the PVFS2 project.
11.1 How can I contribute to the PVFS2 project?
There are lots of ways to directly or indirectly contribute to the PVFS2
project. Reporting bugs helps us make the system better, and describing your
use of the PVFS2 system helps us better understand where and how PVFS2 is
being deployed.
Even better, patches that fix bugs, add features, or support new hardware are
very welcome! The PVFS community has historically been a friendly one, and we
encourage users to discuss issues and exchange ideas on the mailing lists.
If you're interested in this type of exchange, we suggest joining the PVFS2
Developers List, grabbing the newest CVS version of the code, and seeing what
is new in PVFS2. See http://www.pvfs.org/pvfs2/developers.html for more
details.
This section answers questions regarding specific components of the
implementation. It is most useful for people interested in augmenting or
modifying PVFS2.
This section specifically covers questions about the BMI interface and
implementations.
Each BMI module is allowed to define its own maximum message size. See
BMI_tcp_get_info, BMI_gm_get_info, and
BMI_ib_get_info for examples of the maximum sizes that each of the
existing modules support. The maximum should be reported when you issue a
get_info call with the option set to BMI_CHECK_MAXSIZE.
Higher level components of PVFS2 perform these checks in order to make sure
that they don't choose buffer sizes that are too large for the underlying
network.
If the receive buffer is too small for the incoming message, then the
communication will fail and an error will be reported if possible. We
don't support any semantics for receiving partial messages or anything like
that. Its ok if the receive buffer is too big, though.
Frequently Asked Questions about PVFS2
This document was generated using the
LaTeX2HTML translator Version 2002-2-1 (1.71)
Copyright © 1993, 1994, 1995, 1996,
Nikos Drakos,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Ross Moore,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html -split 0 -show_section_numbers -nonavigation -init_file /tmp/testing/pvfs2-1.0.0/doc/latex2html-init pvfs2-faq.tex
The translation was initiated by on 2004-11-09
2004-11-09
|