Linux Files and File Systems
 

Files are the central concept behind Unix and Unix-like systems. Almost everything is treated as a file: all directories, pipes to other processes, the interface to hardware devices, even pointers to files (links). There are even virtual files that give a user access to kernel structures.
 
Whereas a file refers to a single entity, a file system describes the way files are stored on media. The term media includes the obvious hard or floppy disk and CD-ROMs, as well as a network service or the RAM of your machine. Each file system type implements different properties. Not every kind of file can reside on any file system, and not every file system type supports every medium.
 
Files have properties that determine the file type. A common property of all files is that they have a set of permissions, which is the designation that indicates which group it belongs to and who is the owner. Before we discuss file systems or permissions and ownerships, let's take a closer look at the types of files.
 

Regular files are used to store data on a file system. This is the "common sense" type of file, a container for persistent data.
 

Directories hold other files. Because modern file systems are organized in hierarchies, you need something to hold the different levels of the hierarchy. That is a directory's purpose. If you're not used to referring to directories as files, remembering that they are nothing other than files containing a list of other files is important.

 
Special device files are simply interfaces to a device. Two kinds exist: buffered device files and unbuffered device files. The buffered special device files are called block device files; the unbuffered ones are the character device files.
 

A pipe is a connection between two processes. It's treated like a file inside the application, yet doesn't have a representation in the file system. This type of file is interesting only if you mean to write an application. From a user level, pipes are an easy method to use to concatenate commands and give the output from one application as input to another one.
 

Named pipes are like pipes but are represented in the file system. They are also used for interprocess communication, but they can exist without any process accessing them.
 

Sockets are similar to pipes and perform the same functions. The difference is that sockets are used to communicate over a network. The details are important only if you want to use sockets in your application. Further details on sockets are beyond the scope of this book. You can find excellent sources that explain the concept in greater depth.
 

A hard link is a sequential entry in a directory structure for an existing file. It's like a new name for a file.
 

A soft link is a pointer to a file. We'll discuss the difference between hard and soft links later on in this chapter.
 
Executables are not a separate file type and are therefore not included in this list. In Unix systems, being executable or not is a property determined by the file permissions and does not depend on the type of file. Remember that some files can be executed only by designated users or user groups.
 

The main interface to the file system is the ls command. Details on this command can be found in the man pages. If you run ls -l in any directory, you receive a nine-column listing of the files in that directory. Let's focus on the first column as an example. The first column holds information about the files' type and permissions. The first character indicates the type of information and the remaining nine characters determine the permissions. Some implementations of ls mark files by using different colors. This use of color is convenient but not standard (they often distinguish files by extensions, too).
 
Example:

> ls -l 
total 104 
drwxr-xr-x   2 bb  users   1024 Nov  1 17:20 Build/ 
-rw-r--r--   1 bb  users  32120 Nov 20 19:57 Duke 
drwxr-xr-x   2 bb  users   1024 Nov  1 16:49 Mail/ 
drwxr-xr-x   2 bb  users   1024 Nov 23 23:00 SuSE-Book/ 
drwxr-xr-x   2 bb  users   1024 Nov 19 20:01 bin/ 
drwxr-xr-x   8 bb  users   1024 Oct 27 20:41 public_html/ 
-rw-r--r--   1 bb  users    190 Nov  8 23:33 quote 
drwxr-xr-x   2 bb  users   1024 Nov  1 13:52 tmp/ 

The preceding is the listing of a typical user home directory. You see that the character d marks directories, whereas a single dash (-) marks regular files.
 
One interesting feature is worth mentioning at this point. The second column tells you how many hard links (or names) refer to each file. As you can see, the hard link count is one for the regular files but is at least two for the directories. Why is that? Well, a directory has a name in the directory where it is contained, and has a reference to itself (.). That makes two. If a directory has subdirectories, all of these also have a reference to this directory. Take public_html as an example. The d is the first letter in the first row, which tells you that it's a directory, and the fact that it has eight references indicates that it has six subdirectories:
 

> ls -l public_html/ 
total 14 
drwxr-xr-x   6 bb  users   1024 Nov 19 20:53 Gallery/ 
drwxr-xr-x   2 bb  users   1024 Oct 27 20:41 Info/ 
drwxr-xr-x   2 bb  users   1024 Oct 27 20:41 Pilot/ 
drwxr-xr-x   2 bb  users   1024 Oct 13 22:50 Private/ 
-rw-r--r--   1 bb  users    577 Oct  3 19:07 docs.html 
drwxr-xr-x   2 bb  users   1024 Oct  3 16:20 fw-howto/ 
-rw-r--r--   1 bb  users    996 Oct  6 06:40 gallery.html 
drwxr-xr-x   2 bb  users   2048 Nov 19 20:55 gif/ 
-rw-r--r--   1 bb  users   1319 Oct  3 16:07 index.html 
-rw-r--r--   1 bb  users   1508 Oct 26 17:40 index1.html 
-rw-r--r--   1 bb  users    500 Oct  3 19:09 links.html 

That's enough about directories for now. Look through the following code to see what other kinds of files you can find.
 

> ls -l /dev 
[...] 
brw-rw----   1 root  disk    3,   0 Jul 26 12:45 /dev/hda 
brw-rw----   1 root  disk    3,   1 Jul 26 12:45 /dev/hda1 
brw-rw----   1 root  disk    3,   2 Jul 26 12:45 /dev/hda2 
brw-rw----   1 root  disk    3,   3 Jul 26 12:45 /dev/hda3 
brw-rw----   1 root  disk    3,   4 Jul 26 12:45 /dev/hda4 
brw-rw----   1 root  disk    3,   5 Jul 26 12:45 /dev/hda5 
brw-rw----   1 root  disk    3,   6 Jul 26 12:45 /dev/hda6 
brw-rw----   1 root  disk    3,   7 Jul 26 12:45 /dev/hda7 
brw-rw----   1 root  disk    3,   8 Jul 26 12:45 /dev/hda8 
brw-rw----   1 root  disk    3,   9 Jul 26 12:45 /dev/hda9 
crwx-w----   1 bb    tty     4,   1 Nov 23 23:19 /dev/tty1 
crw-rw----   1 root  tty     4,  11 Jul 26 12:45 /dev/tty11 
[...] 

Pipes are closely related to device special files. Like device special files, pipes don't refer to actual data stored in the file system. A pipe is used as a way to communicate between two processes. These processes can be threads of the same application, or two different applications in which the output of one application is piped as input to the other application. You can think of a named pipe as a regular pipe with a representation in the file system.
 
Whereas regular pipes are temporary and exist only as long as the processes that use the pipe exist, named pipes are more permanent fixtures.
 
If you think of a pipe as a jet stream of water, then a named pipe is like a hose. You can create a named pipe, fill it with data from one application, and read the data out of the named pipe with another application at the same time or later. The buffer capacity is not great. Remember, it's a hose, not a bucket (which, in this analogy, would be a regular file). Another name for named pipes is FIFO, which stands for First In First Out.
 
Named pipes are marked by a p in the ls -l output:
 

> ls -l /dev/initctl 
prw------- 1 root root 0 Dec 31 1989 /dev/initctl| 

Now that we've reviewed regular files, directories, device special files, and named pipes, we'll examine one last group that makes the list complete: links.
 
Links are used to provide more than one reference to a file. We examined this concept in the section on directories. Recall that a directory is no more than a special kind of file. Directories are linked together when hard links are created from the children to the parents. These hard links have special names and are created on the fly when a directory is created: . points to the directory itself, and .. points to the parent directory. It's commonly thought that . and .. are special characters that are treated in a special way. In truth, they are nothing other than hard links to directories.
 
Of course, hard links can point to regular files. A hard link is just a new entry in the directory file that points to the same inode as another entry. No way exists to distinguish between hard links and the files they point to. There is no original and copy. Recall that the hard link counter (the number given) tells the system when the actual data can be removed from the storage medium. Every time you delete a reference to a file, the hard link count is decreased. The data will be removed when the counter reaches zero. Makes sense, doesn't it?
 
There is one problem with hard links: they can point only to files within the same file system. We discuss file systems later on in this chapter, at which point you'll see why this is a hard limit to live with. The way around this limitation is to use soft links. Soft links are like hard links; they point to files. The difference is that soft links are files themselves. The only data they contain is a pointer to the filename they refer to. Refer to the /dev directory for an example:
 

> ls -l /dev/modem  
lrwxrwxrwx   1 root  root  9 Oct  3 18:09 /dev/modem -> /dev/cua1 

> ls -l /var/X11R6/bin/ 
lrwxrwxrwx 1 root root 24 Dec 4 16:38 X -> /usr/X11R6/bin/XF86_SVGA 

> ls -l /usr/spool 
lrwxrwxrwx 1 root root 12 Dec 17 21:21 /usr/spool -> ../var/spool 

Now that you know about the various file types, you're probably interested in learning how to create them. As with most things in the Unix world, no one way exists to accomplish this. You can create every type of file in many different ways. Every programming language provides different ways of integrating file access. It may be better if we leave it to the manuals of these languages to explain how they handle files. We'll provide a short overview of the shell commands that can be used to do this on the command line.
 
The easiest way to create a regular file is as follows:
 

> touch filename 

> mkdir directory 

> mknod /dev/cua1 c 5 65  

> mknod /dev/hda10 b 3 10 

> mknod /dev/initctl p 

> ln -s /dev/cua1 /dev/modem 

> ln file1 file2 

Now that we've reviewed the basics, single files, it's time to take this review to another level -- file systems. We've used the term file system quite frequently up to now without a working definition. Because file system has two different meanings, it can become quite confusing. When we talk about file systems, we usually think of directory trees. This is a hierarchical structure of directories that contain other directories and/or any kind of file.
 

> tar cvzf /dev/fd0 public_html/Info/ 
public_html/Info/ 
public_html/Info/index.html 
public_html/Info/bbatwork.jpg 
public_html/Info/resumee.html 

File systems organize files into logical hierarchical structures with directories, links, and so on. With kernel version 2.2, the Linux system supports twenty-one different file system types: minix, ext2, iso9660, msdos, umsdos, vfat, proc, nfs, smb, ncp, hpfs, sysv, adfs, amiga-fs, ROM-fs,NTFS, joilet, Apple Macintosh fs, QNX fs, Coda, and ufs.
 
A file system organizes the data on a block device. This can be a hard disk, a floppy disk, RAM, a flash RAM cartridge, or even a network link to a remote system. The file system itself doesn't know anything about the medium it uses. This is handled by the device driver. It's the job of the device driver to translate the address of a specific block to a physical position on a hard drive, a memory region, and so on.
 
The de facto standard file system on Linux systems is Extended 2 File System (ext2). It was designed by Remy Card as a successor to the Extended File System ext (also by Remy Card), which was dropped with the 2.2 kernel release.
 
The first file system for Linux was the Minix file system (Minixfs). It had many disadvantages. Partitions were limited to 64MB, filenames to 14 characters. The extended file system, introduced to the Linux world in April 1992, was mostly based on the Minixfs; it removed the previously mentioned limits, but still was far from the perfect file system Remy had in mind. He therefore decided to write a completely new file system, ext2. It added better space allocation management to extfs, exhibited better performance, allowed the use of special flags for file management, and was extensible. It's been available since 1993 and is the most successful Linux file system to date.
 
When the extfs was added to the Linux kernel, a new layer entered the system, the so-called Virtual File System (VFS) layer. With this layer, the file system and the kernel were no longer one unit. A well-defined interface between kernel and file system was created. VFS allows Linux to support many different file systems. All file systems appear identical to the kernel and to the programs running on the system. The VFS layer allows you to transparently mount different file systems at a time. Figure 1-1 shows the relation between the kernels VFS and the real filesystems.
 

In this section, we return to the ext2 file system and review its structure.
 
The ext2 file system is built on the premise that data is held in blocks of equal size on the storage medium. The block size can vary on different ext2 file systems, but within one file system, all blocks have the same length. The length is set when the file system is created by mke2fs. This strategy's disadvantage is that on average, half the block size is wasted for each file. Assuming that the block size is 1024 bytes, a file with 1025 bytes will use two blocks, just as a file with 2047 will. By handling the files this way, you reduce the CPU's workload.
 
Not all blocks on the file system hold actual file data. Some are used to hold information about the structure of the file system. Ext2 uses inode data structures for each file stored in the file system. An inode describes which blocks contain the data of the specified file, its type, owner, and group, the permissions and the date of the last access, the last modification, and its creation date. The inodes are held in inode tables. A directory is simply a file that contains pointers to the inodes of the files within this directory.
 
Figure 1-2 shows the format of an ext2 inode. You can see four fields at the beginning of the structure. The first field is for the mode (permission); the second holds the owner and group information; the third shows the size of the system in bytes; and the fourth shows the three timestamps. Following these fields are pointers to the data blocks of the file. There are 12 direct pointers and 3 indirect pointers with an indirection level of one, two, and three.
 

Now that we've had a look at the internals of the ext2 file system, it's time to talk about how it's used in the system. In Linux, as in all Unix-type systems, separate file systems are not accessed by device identifiers such as drive names or volume identifiers. You'll find a single hierarchical structure. Separate file systems are mounted into this hierarchy no matter the type of storage medium. Separate file systems are mounted into the existing hierarchy to mount points. A mount point is nothing more than a directory in the existing tree. If the directory already contains something, this information will be overlaid (and not accessible) when a file system is mounted to this directory. It'll be visible again when the overlaying tree is unmounted:
 

# touch /mnt/This-Directory-is-not-empty 
# ls /mnt/ 
.                            This-Directory-is-not-empty 
.. 
# mount -oro /dev/cdrom /mnt 
# ls /mnt/ 
.                             TRANS.TBL 
..                            full-names 
.S.u.S.E-disk-002.1998111916  suse 
# umount /mnt/ 
 ls /mnt/ 
.                            This-Directory-is-not-empty 
.. 

The layout of the directory tree is not arbitrary. The overall structure of the directory layout is defined in the File System Hierarchy Standard. If you look in the root of a SuSE Linux 6.0 system, you'll see the following directories:
 

boot   etc  lost+found  proc  tmp   cdrom  opt 
home   mnt  root        usr   bin   dev    lib 
sbin   var 

etc   root   bin lib   sbin   dev 

 
One part of the standard file system that deserves to be explained in more detail is the tree beginning with /proc. This file system is becoming the de facto method for Linux to handle process and system information. It is a nice example of the power of Linux's Virtual File System, a file system that does not really exist, either in the /proc directory or in its subdirectories. The content of files in the /proc is generated by the kernel at the moment a user reads them, representing information about running processes and kernel internals. Some files are writable, which allows the possibility of changing kernel parameters during runtime.