Cypris' lookout

Admin: Linux file server performance boost (ext3 version)

Sunday, July 11, 2010

Using a Linux for an office file server is a no-brainer: it’s cheap, you don’t have to worry about unmanageable license costs and it just works.

Default settings of most Linux distributions are however not optimal: they are meant to be as standard compliant and as general as possible so that everything works well enough regardless of what you do.

For a file server hosting large numbers of files, these default settings can become a liability: everything slows down as the number of files creeps up and it makes your once-snappy fileserver as fas as a sleepy sloth.

There are a few things that we can do to ensure we get the most of our server.

Checking our configuration

First, a couple of commands that will help us investigate the current state of our configuration.

• df will give us a quick overview of the filesystem:

  df -T
  Filesystem    Type   1K-blocks      Used Available Use% Mounted on
  /dev/md2      ext3    19840804   4616780  14199888  25% /
  tmpfs        tmpfs      257580         0    257580   0% /dev/shm
  /dev/md0      ext3      194366     17718    166613  10% /boot
  /dev/md4      ext3     9920532   5409936   3998532  58% /var
  /dev/md3      ext3      194366      7514    176817   5% /tmp
  /dev/md5      ext3    46980272  31061676  13493592  70% /data
  

• tune2fs will help us configure the options for each ext3 partition. If we want to check what is the current configuration of a given partition, says we want to know the current options for our /data mount:

  # tune2fs -l /dev/md5
  

  If I was using LVM as a Volume manager, I would type something like:

  # tune2fs -l /dev/VolGroup00/LogVol02
  

  This would give lots of information about the partition:

  tune2fs 1.40.2 (12-Jul-2007)
  Filesystem volume name:   <none>
  Last mounted on:          <not available>
  Filesystem UUID:          d6850da8-af6f-4c76-98a5-caac2e10ba30
  Filesystem magic number:  0xEF53
  Filesystem revision #:    1 (dynamic)
  Filesystem features:      has_journal resize_inode dir_index filetype 
                            needs_recovery sparse_super large_file
  Filesystem flags:         signed directory hash
  Default mount options:    user_xattr acl
  Filesystem state:         clean
  Errors behavior:          Continue
  ....
  

  The interesting options are listed under Filesystem features and Default mount options. For instance, here we know that the partition is using a journal and that it is using the dir_index capability, already a performance booster.

• cat /proc/mounts is useful to know the mounting options for our filesystem (just listed some interesting ones here):

  rootfs / rootfs rw 0 0
  /dev/root / ext3 rw,data=ordered 0 0
  /dev/md0 /boot ext3 rw,data=ordered 0 0
  /dev/md4 /var ext3 rw,data=ordered 0 0
  /dev/md3 /tmp ext3 rw,data=ordered 0 0
  /dev/md5 /data ext3 rw,data=ordered 0 0
  none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
  /dev/md4 /var/named/chroot/var/run/dbus ext3 rw,data=ordered 0 0
  

  The data=ordered mount parameter tells us of the journaling configuration for the partition.

Journaling

So what is journaling?
It’s one of the great improvements of ext3: a journal is a special log on the disk that keeps track of changes about to be made. It ensures that, in case of failure, the filesystem can quickly recover without loss of information.

There are 3 settings for the journalling feature:

• data=journal the most secure but also slowest option since all data and metadata is written to disk: the whole operation needs to be completed before any other operation can be completed. It’s sort of going to the bank for a deposit, filling the paperwork and making sure the teller puts the money in the vault before you leave.
• data=ordered is usually the default compromise: you fill-in the paperwork and remind the teller to put the money in the vault asap.
• data=writeback is the fastest but you can’t be absolutely sure that things will be done in time to prevent any loss if a problem occurs soon after you’ve asked for the data to be written.

In normal circumstances all 3 end-up the same way: data is eventually written to disk and everything is fine.
Now if there is a crash just as the data was written only option journal would guarantee that everything is safe. Option ordered is fairly safe too because the money should be in the vault soon after you left; most systems use this option by default.

If you want to boost your performance and use writeback you should make sure that:

• you have a good power-supply backup to minimise the risk of power failure
• you have a good data backup strategy
• you’re ok with the risk of losing the data that was written right before the crash.

To change the journaling option you simply use tune2fs with the appropriate option:

    # tune2fs -o journal_data_writeback /dev/md5

Mount options

Now that we’ve changed the available options for our partition, we need to tell the system to use them.
Edit /etc/fstab and add data=writeback to the option columns:

    /dev/md5     /data    ext3    defaults,data=writeback   1 2

Next time our partition is mounted, it will use the new option. For that we can either reboot or remount the partition:

    # mount - o remount /data

noatime option

There is another option that can have a very dramatic effect on performance, probably even more than the journaling options above.

By default, whenever you read a file the kernel will update its last access time, meaning that we end up with a write operation for every read!
Since this is required for POSIX compliance, almost all Linux distributions leave this setting alone by default.
For a file server, this can have such drastic consequence on performance.

To disable this time-consuming and not useful feature (for a file server), simply add the noatime option to the fstab mount options:

    /dev/md5     /data    ext3    defaults,noatime,data=writeback   1 2

Note that updating access times is sometimes required by some software, such as mail software (such as mutt). If you properly keep your company data in a dedicated partition, you can enable the mount options only for that partition and keep other options for the root filesystem.

dealing with errors in fstab

After doing the above on one of the servers, I realized that I made a typo when editing /etc/fstab.
This resulted in the root filesystem being mounted read-only, making fstab impossible to edit…

To make matters worse, this machine was a few thousand miles away and could not be accessed physically….

Remounting the root filesystem resulted in errors:

    # mount -o remount,rw /
    mount: / not mounted already, or bad option

After much trial and rebooting, this worked (you need to specify all mounting options, to avoid the wrong defaults from being read from etc/mtab`):

    # mount  -o rw,remount,defaults /dev/md2 /

After that, I could edit /etc/fstab and correct the typo…

Conclusions

How much these options will improve performance really depends on how your data is used: the improvements should be perceptible if your directories are filled with large amounts of small files.
Deletion should also be faster.

Entry Filed under  :  Linux,sysadmin