Some distribution, in particular RedHat/CentOS 5, do not allow us to select ext4 as a formatting option during setup of the machine, so you will initially have to use ext3 as file system (on top of LVM preferably for easy extensibility).

A small digression on partitioning

Remember to create separate partitions for your file data: do not mix OS files with data files, they should live on different partitions. In an enterprise environment, a minimal partition configuration for a file server could look like:

Hardware:

• 2x 160GB HDD for the OS
• 4x 2TB HDD for the data

The 160GB drives could be used as such:

• 200MB RAID1 partition over the 2 drives for /boot
• 2GB RAID1 partition over the 2 drives for swap
• all remaining space as a RAID1 partition over the 2 drives for /
  Note though that it is generally recommended to create additional partitions to further contain /tmp and /var.

The 2TB drives could be used like this:

• all space as RAID6 over all drives (gives us 4TB of usable space) for /data
• alternatively, all space as RAID5 over all drives (gives us 6TB of usable space) The point of using RAID6 is that it gives better redundancy than RAID5, so you can safely add more drives later without increasing the risk of failure of the whole array (which is not true of RAID5).

Moving to ext4

If you are upgrading an existing system, backup first!

Let’s say that your /data partition is an LVM volume under /dev/VolGroup01/LogVol00. First, make sure we have the ext4 tools installed on our machine, then unmount the partition to upgrade:

# yum -y install e4fsprogs
# umount /dev/VolGroup01/LogVol00

For a new system, create a large partition on the disk, then format the volume (this will destroy all data on that volume!).

# mkfs -t ext4 -E stride=32 -m 0 -O extents,uninit_bg,dir_index,filetype,has_journal,sparse_super /dev/VolGroup01/LogVol00
# tune4fs -o journal_data_writeback /dev/VolGroup01/LogVol00

Note: on a RAID array, use the appropriate -E stride,stripe-width options, for instance, on a RAID5 array using 4 disks and 4k blocks, it could be: -E stride=16,stripe-width=48

For an existing system, upgrading from ext3 to ext4 without damaging existing data is barely more complicated:

# fsck.ext3 -pf  /dev/VolGroup01/LogVol00
# tune2fs -O extents,uninit_bg,dir_index,filetype,has_journal,sparse_super /dev/VolGroup01/LogVol00
# fsck.ext4 -yfD /dev/VolGroup01/LogVol00

We can optionally give our volume a new label to easily reference it later:

# e4label /dev/VolGroup01/LogVol00 data

Then we need to persist the mount options in /etc/fstab:

/dev/VolGroup01/LogVol00    /data    ext4    noatime,data=writeback,barrier=0,nobh,errors=remount-ro    0 0

And now we can remount our volume:

# mount /data

If you upgraded an existing filesystem from etx3, you may want to run the following to ensure the existing files are using extents for file attributes:

# find /data -xdev -type f -print0 | xargs -0 chattr +e
# find /data -xdev -type d -print0 | xargs -0 chattr +e

Important notes

The mounting options we use are somewhat a bit risky if your system is not adequately protected by a UPS.
If your system crashes due to a power failure, you are more likely to lose data using these options than using the safer defaults.
At any rate, you must have a proper backup strategy in place to safeguard data, regardless of what could damage them (hardware failure or user error).

• The barrier=0 option disables Write barriers that enforce proper on-disk ordering of journal commits.
• The data=writeback and nobh go hand in hand and allow the system to write data even after it has been committed to the journal.
• The noatime ensures that the access time is not updated when we’re reading data as this is a big performance killer (this one is safe to use in any case).

References

• Mount options to improve ext4 file system performance
• Ext4 Howto
• Migrating a live system from ext3 to ext4 filesystem

Default settings of most Linux distributions are however not optimal: they are meant to be as standard compliant and as general as possible so that everything works well enough regardless of what you do.

For a file server hosting large numbers of files, these default settings can become a liability: everything slows down as the number of files creeps up and it makes your once-snappy fileserver as fas as a sleepy sloth.

There are a few things that we can do to ensure we get the most of our server.

Checking our configuration

First, a couple of commands that will help us investigate the current state of our configuration.

• df will give us a quick overview of the filesystem:

  # df -T
  Filesystem    Type   1K-blocks      Used Available Use% Mounted on
  /dev/md2      ext3    19840804   4616780  14199888  25% /
  tmpfs        tmpfs      257580         0    257580   0% /dev/shm
  /dev/md0      ext3      194366     17718    166613  10% /boot
  /dev/md4      ext3     9920532   5409936   3998532  58% /var
  /dev/md3      ext3      194366      7514    176817   5% /tmp
  /dev/md5      ext3    46980272  31061676  13493592  70% /data
  

• tune2fs will help us configure the options for each ext3 partition. If we want to check what is the current configuration of a given partition, says we want to know the current options for our /data mount:

  # tune2fs -l /dev/md5
  

  If I was using LVM as a Volume manager, I would type something like:

  # tune2fs -l /dev/VolGroup00/LogVol02
  

  This would give lots of information about the partition:

  tune2fs 1.40.2 (12-Jul-2007)
  Filesystem volume name:   <none>
  Last mounted on:          <not available>
  Filesystem UUID:          d6850da8-af6f-4c76-98a5-caac2e10ba30
  Filesystem magic number:  0xEF53
  Filesystem revision #:    1 (dynamic)
  Filesystem features:      has_journal resize_inode dir_index filetype 
                            needs_recovery sparse_super large_file
  Filesystem flags:         signed directory hash
  Default mount options:    user_xattr acl
  Filesystem state:         clean
  Errors behavior:          Continue
  ....
  

  The interesting options are listed under Filesystem features and Default mount options. For instance, here we know that the partition is using a journal and that it is using the dir_index capability, already a performance booster.

• cat /proc/mounts is useful to know the mounting options for our filesystem (just listed some interesting ones here):

  rootfs / rootfs rw 0 0
  /dev/root / ext3 rw,data=ordered 0 0
  /dev/md0 /boot ext3 rw,data=ordered 0 0
  /dev/md4 /var ext3 rw,data=ordered 0 0
  /dev/md3 /tmp ext3 rw,data=ordered 0 0
  /dev/md5 /data ext3 rw,data=ordered 0 0
  none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
  /dev/md4 /var/named/chroot/var/run/dbus ext3 rw,data=ordered 0 0
  

  The data=ordered mount parameter tells us of the journaling configuration for the partition.

Journaling

So what is journaling?
It’s one of the great improvements of ext3: a journal is a special log on the disk that keeps track of changes about to be made. It ensures that, in case of failure, the filesystem can quickly recover without loss of information.

There are 3 settings for the journalling feature:

• data=journal the most secure but also slowest option since all data and metadata is written to disk: the whole operation needs to be completed before any other operation can be completed. It’s sort of going to the bank for a deposit, filling the paperwork and making sure the teller puts the money in the vault before you leave.
• data=ordered is usually the default compromise: you fill-in the paperwork and remind the teller to put the money in the vault asap.
• data=writeback is the fastest but you can’t be absolutely sure that things will be done in time to prevent any loss if a problem occurs soon after you’ve asked for the data to be written.

In normal circumstances all 3 end-up the same way: data is eventually written to disk and everything is fine.
Now if there is a crash just as the data was written only option journal would guarantee that everything is safe. Option ordered is fairly safe too because the money should be in the vault soon after you left; most systems use this option by default.

If you want to boost your performance and use writeback you should make sure that:

• you have a good power-supply backup to minimise the risk of power failure
• you have a good data backup strategy
• you’re ok with the risk of losing the data that was written right before the crash.

To change the journaling option you simply use tune2fs with the appropriate option:

# tune2fs -o journal_data_writeback /dev/md5

Mount options

Now that we’ve changed the available options for our partition, we need to tell the system to use them.
Edit /etc/fstab and add data=writeback to the option columns:

/dev/md5     /data    ext3    defaults,data=writeback   1 2

Next time our partition is mounted, it will use the new option. For that we can either reboot or remount the partition:

# mount - o remount /data

noatime option

There is another option that can have a very dramatic effect on performance, probably even more than the journaling options above.

By default, whenever you read a file the kernel will update its last access time, meaning that we end up with a write operation for every read!
Since this is required for POSIX compliance, almost all Linux distributions leave this setting alone by default.
For a file server, this can have such drastic consequence on performance.

To disable this time-consuming and not useful feature (for a file server), simply add the noatime option to the fstab mount options:

/dev/md5     /data    ext3    defaults,noatime,data=writeback   1 2

Note that updating access times is sometimes required by some software, such as mail software (such as mutt). If you properly keep your company data in a dedicated partition, you can enable the mount options only for that partition and keep other options for the root filesystem.

dealing with errors in fstab

After doing the above on one of the servers, I realized that I made a typo when editing /etc/fstab.
This resulted in the root filesystem being mounted read-only, making fstab impossible to edit…

To make matters worse, this machine was a few thousand miles away and could not be accessed physically….

Remounting the root filesystem resulted in errors:

# mount -o remount,rw /
mount: / not mounted already, or bad option

After much trial and rebooting, this worked (you need to specify all mounting options, to avoid the wrong defaults from being read from etc/mtab`):

# mount  -o rw,remount,defaults /dev/md2 /

After that, I could edit /etc/fstab and correct the typo…

Conclusions

How much these options will improve performance really depends on how your data is used: the improvements should be perceptible if your directories are filled with large amounts of small files.
Deletion should also be faster.

This long article is fairly detailed but if your setup is similar it should be enough to get you going quickly.

• The basic setup
• Objectives
• Technologies
• Things to know
• Things to read
• Installing shorewall
• Preparing the system
• Shorewall configuration
• Installing our firewall rules
• DNS
• Initial conclusions
• References

The basic setup

Without further ado, this is the network configuration:

Notable things

We have 2 broadband connections:

• CYBR, a standard DSL line with a fixed IP 111.99.88.77 allocated through PPPoE.
• HKBN, a standard 100Mbps line with a fixed IP 30.40.50.62.

The network is split into different zones:

• the Internet zone, connected to our Firewall through interfaces eth0 (ppp0) and eth1.
• a Firewall zone, delimited by the firewall system itself
• a DMZ zone connected through interface eth2 for the servers we want to make visible from the Internet.
  The DMZ has its own private subnet delimited by 192.168.254.0/255.255.255.0.
• a LAN zone connected through interface eth3 so local computers can access the Internet and be protected from it.
  The DMZ has its own private subnet delimited by 192.168.0.0/255.255.255.0.

Objectives

What we want from our setup:

1. our firewall protects our DMZ and LAN from unwanted access.
2. our win server can host websites or other services.
3. our linux server can handle receiving and sending email or other services.
4. our firewall can handle incoming traffic from either ISP.
5. our firewall can load-balance local outgoing traffic across both ISP.
6. If one line fails, incoming traffic switches to the other line.
7. If one line fails, outgoing traffic switches to the other line.
8. Eventually, we want both the linux and win servers to be able to host different websites and we want the firewall to send incoming requests to the right server.

In this first article, I’ll present the setup for items 1-5.
The remaining topics will be the subject of subsequent articles of their own.

Technologies

The firewall is our primary subject. What is being discussed here is pretty much distribution-independent and should work on all flavours of Linux.

OS on the firewall system

I chose CentOS on the firewall.
Being an almost byte-for-byte identical copy of RedHat Enterprise Linux, all configuration will be identical on RedHat and its derivatives such as Fedora.

Firewall software, first try

When my firewall needs are simpler, I use the Stronger IP Firewall Ruleset from the Linux IP Masquerade HOWTO.
I started to modify the script to adapt it to my new Multi-ISP setup but things got complicated once I needed to debug routing tables.
I got it 80% of the way but tracing network connections and packet routing is complicated and time-consuming.
After a couple of days of peering into log files and wireshark capture screens, I gave up manual configuration and decided to go with something else.

Firewall software, final

The product I chose in the end is shorewall: it’s a very flexible firewall system that create the necessary iptable rules and configure most of the routing needs to properly handle complex network setup.
Shorewall is Open Source, very stable, has been out for a long time, is actively maintained and has lots of excellent documentation and examples.

Things to know

Before we get into the meat of the article, you should brush up on the following topics:

• You have some knowledge of Linux system administration, know how to configure network connections, know how to enable/disable/stop/start services, able to edit config files.
• Networking: you should know what a netmask is, what a gateway is, what a subnet is and have a passing understanding of IP classes, IP notation, what ports are for, what’s the difference between the tcp, udp, icmp protocols, what Dynamic Port Forwarding (DNAT) is, what Network Address Translation (NAT) is, what masquerading means.
• Some basic understanding of DNS and local host name resolving (using host.conf and resolv.conf)
• Some basic knowledge of what routing is for and how it works.
• Some knowledge of how the linux kernel handles network packets (NetFilter, basics of iptables).

You don’t need to be a specialist in any of these areas but any knowledge helps.
I’m far from being well versed into Netfilter and routing, it’s not often that I have to deal directly with these topics, but brushing up on these topics helped.

Things to read

Shorewall has very extensive documentation. So much so that it can be a bit daunting, not knowing where to start.
I found the following documents helpful to get me started:

• Setup Guide for multiple interfaces.
• Routing article
• Guide for handling Multi-ISP

Installing shorewall

Go to the download site list [http://shorewall.net/download.htm#Sites] and download the most appropriate binary package for your distribution.

If you get RPMs for RedHat systems, you only need to install (rpm -ivh) the following packages:

shorewall-4.X.X-X.noarch.rpm
shorewall-perl-4.X.X-X.noarch.rpm 

If you install from source, only download, compile and install the common, doc and perl packages.

Preparing the system

For shorewall to properly handle both our firewall and packet routing needs, we need to make sure that the other parts of the system are not interfering with it.

Internet lines

Make sure that your multiple internet lines are properly working on their own!

Disable routing

• Make sure that you don’t define a GATEWAY in the configuration of your network interfaces (in /etc/sysconfig/network-scripts/ifcfg-XXX) .
• If you use an (A)DSL connection, also set DEFROUTE=no if its ifcfg-XXX file as well.
• Remove the GATEWAY from the /etc/sysconfig/network file if there is one.
• Edit your /etc/sysctl.conf file and set net.ipv4.conf.default.rp_filter = 0.

Disable firewall

Disable the current firewall, for instance using the system-config-securitylevel helper tool.
Be careful if you’re directly connected to the Internet, you will be left without protection!
You can actually wait until shorewall is properly configured to disable the firewall.

Shorewall configuration

Shorewall uses a set of simple configuration files, all located under /etc/shorewall/. For exact detail of each configuration files, have a look at the list of man pages.

Zones

zones are probably the simplest configuration file.
Details in the zones man page. Here we just name the various zones we want our firewall to handle:

################################################################
#ZONE   TYPE          OPTIONS       IN                  OUT
#                                   OPTIONS             OPTIONS
fw      firewall
net     ipv4
loc     ipv4
dmz     ipv4

This just reflects our setup as highlighted in the diagram above.

Note that the fw zone is often referred to as the $FW variable instead in various configuration files.

Interfaces

Here we list all the network interfaces connected to our firewall and for which zone they apply.
Details in the interfaces man page.

################################################################
#ZONE   INTERFACE       BROADCAST       OPTIONS
net     ppp0            detect
net     eth1            detect
dmz     eth2            detect
loc     eth3            detect

Note that for our net zone, we list the 2 interfaces connected to our ISPs.
If you’re using PPPoE to connect, don;t use the interface name eth0 but use ppp0 instead.

Policy

The policy file tells shorewall which default actions should be taken when traffic is moving from one zone to another.
These default actions are taken if no other special action was specified in other configuration files.
View the policy file as a list of default actions for the firewall.
Details about this configuration file as in its man page.

################################################################
#SOURCE DEST    POLICY          LOG     LIMIT:      CONNLIMIT:
#                               LEVEL   BURST       MASK
net     net     DROP            info
loc     net     ACCEPT
dmz     net     ACCEPT
loc     dmz     ACCEPT
loc     $FW     ACCEPT
dmz     $FW     ACCEPT
$FW     net     ACCEPT
dmz     loc     DROP            info
net     all     DROP            info
all     all     DROP            info

Traffic from one zone to another needs to be explicitely ACCEPTed, REJECTed or DROPped.
For instance, loc net ACCEPT means that we allow all traffic from our local LAN to the Internet, while net all DROP means we don’t allow incoming traffic from the internet to anyone (remember this is the default action, in most cases we will override this for specific types of traffic in the rules file).
When we set the default action to DROP, we can tell shorewall to keep a trace of the details in the /var/log/messages log.

Providers

The providers file is generally only used in a multi-ISP environment.
Here we define how we want to mark packets originating from one ISP with a unique ID so we can tell the kernel to route these packets to the right interface.
Not doing this would get packets received from one interface to be routed to the default gateway instead.
The details of this configuration file are explained in the providers man page for it.

#############################################################################
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY      OPTIONS          COPY
CYBR  1      0x1  main      ppp0      -            track,balance=1  eth2,eth3
HKBN  2      0x2  main      eth1      30.40.50.61  track,balance=5  eth2,eth3

Note that the DUPLICATE columns tells shorewall that it should make a copy of the main default routing table for this particular routing table (called CYBR or HKBN depending on which ISP we refer to).
Packets are marked with number 0×1 or 0×2 so we can distinguish them during their travel through the system.
For PPPoE connections, don’t specify a GATEWAY since it’s most likely that your ISP didn’t give you one.

The most interesting part of this file are the OPTIONS: track means that we want the packets to be tracked as they travel through the system; balance tells the kernel that we want traffic coming out to be spread over both interfaces.
Additionally, we want HKBN to receive more or less 5 times more traffic than CYBR (note that this has no effect on reply packets).

The COPY columns will ensure that the routing tables created for CYBR and HKBN are copied for each internal interface, so our eth2 and eth3 interfaces know how to route packets to the right ISP.

Route Rules

For our purpsose, the route_rules file only describes how traffic should be routed through one or the other ISP we set up in /etc/shorewall/providers.
Details are in the route_rules file man page.

#####################################################################
#SOURCE             DEST               PROVIDER        PRIORITY
ppp0                -                  CYBR            1000
eth1                -                  HKBN            1000

Here we simply say that all traffic through the CYBR table should be sent to ppp0.
The PRIORITY is an ordering number that tell shorewall to consider this routing rule before it marks the packets. Since we know the packets originated from ppp0 or eth1 we don’t really need to mark them.

Masq

The masq file will contain the masquerading rules for our private interfaces: in essence, we want traffic from the local LAN and DMZ to be hidden behind our limited number of external IPs.
See the masq manpage for all the details.

#####################################################################
#INTERFACE              SOURCE           ADDRESS                 
# Ensure that traffic originating on the firewall and redirected via 
# packet marks always has the source IP address corresponding to the 
# interface that it is routed out of.
# See http://shorewall.net/MultiISP.html#Example1
ppp0                    30.40.50.62      111.99.88.77
eth1                    111.99.88.77     30.40.50.62
ppp0                    eth2             111.99.88.77
eth1                    eth2             30.40.50.62
ppp0                    eth3             111.99.88.77
eth1                    eth3             30.40.50.62

The first part ensures that the traffic coming out of our public interfaces but originating from the other is actually rewritten as originating from the right IP for the interface.
This ensures that packets leaving eth1 for instance don’t come out with the wrong source address of the other interface.
The second part of the ensures that packets from our LAN or DMZ leaving either public interfaces are doing so with the right IP address, so traffic from my desktop going through ppp0 for instance, will have its source address as 100.90.80.70.

Rules

This is the main file where we tell shorewall our basic configuration and how we want packets to be handled in the general case.
The /etc/shorewall/rules file contains the specific instructions on where to direct traffic that will override the default actions defined in the /etc/shorewall/policy file.

#####################################################################
#ACTION    SOURCE                DEST                   PROTO  
#                                                                     
SECTION NEW
# Drop and log packets that come from the outside but pretend 
# to have a local address
DROP:info  net:192.168.0.0/24    all
DROP:info  net:192.168.254.0/24  all

# Redirect incoming traffic to the correct server for WWW and email
DNAT       all                   dmz:192.168.254.20     tcp   www
DNAT       all                   dmz:192.168.254.10     tcp   110
DNAT       all                   dmz:192.168.254.10     tcp   143
DNAT       all                   dmz:192.168.254.10     tcp   25

In its most basic form, what we’ve just defined here is that we want all traffic from anywhere destined for port 80 (www) to be sent to our win server.
All mail traffic, POP3 (port 110), IMAP (port 143) and SMTP (port 25) is to be redirected to our linux server in the DMZ.

There are a few more useful rules that we can include, for instance, I want to be able to access my servers through either ISPs from home (IP 123.45.67.89) and disallow everyone else from accessing it.

#####################################################################
#ACTION    SOURCE                DEST                   PROTO  
#                                                                     
# Allow SSH to the firewall from the outside only from home

ACCEPT     net:123.45.67.89      $FW                    tcp   ssh
# Redirect input traffic to the correct server for RDP, VNC and SSH 
DNAT       net:123.45.67.89      dmz:192.168.254.10:22  tcp   2222
DNAT       net:123.45.67.89      dmz:192.168.254.10     tcp   5901
DNAT       net:123.45.67.89      dmz:192.168.254.20     tcp   3389

When I SSH to 30.40.50.62 or 100.90.80.70, on the normal port 22, I will access the firewall.
Now if I SSH to the non-standard port 2222, I will instead access the linux server.
Ports 5901 are for remoting through VNC on the linux machine, and port 3389 will be used for Remote Desktop connections to the win server.

To make sure my machines are up and running, I like to be able to ping them:

#####################################################################
#ACTION    SOURCE              DEST              PROTO  
#                                                                     
# Accept pings between zones
ACCEPT     dmz                 loc               icmp  echo-request
ACCEPT     loc                 dmz               icmp  echo-request

Note that ping will only work between the LAN and the DMZ and pinging my firewall from the Internet will result in the requests being silently dropped.
I usually prefer that configuration as it makes discovering the servers by random bots slightly less likely.

There are lots of other cool things we can do with forwarding but that will do for now.

shorewall.conf

The last file we’re going to look at is the main configuration file for shorewall.
See details about each option from the man page for shorewall.conf.

Most options are OK by default. The only ones that I have had to change are:

STARTUP_ENABLED=Yes
MARK_IN_FORWARD_CHAIN=Yes
FASTACCEPT=Yes
OPTIMIZE=1

The first option tells shorewall that we want it to start automatically when the system boots.
That’s not enough though, so make sure that the service will be started:

# chkconfig shorewall --levels 235 on

Installing our firewall rules

Shorewall configuration files need to be compiled without error before the firewall is actually loaded by shorewall.
The command:

# shorewall restart

will stop and recompile the current configuration.
If there are any errors, the current firewall rules will be unchanged.
There are lots of other commands that can be issued. Check the man page for a complete list.

If you use PPPoE, you will want the firewall to be restarted every time the line reconnects.
The simplest way is to create a file /etc/ppp/if-up.local with only a single line:

shorewall restart

DNS

There is one remaining issue with our firewall: if a user on the LAN attempts to access the web server by its name the request will probably fail.
Same for accessing our mail server: we can configure our desktop to connect to 192.168.254.10 to get and send emails, but on the laptop we would usually use something like pop.acme.com instead so we can read our emails from outside the office.

Similarly, trying to access www.acme.com hosted on the win server from the linux server will fail.

One solution is to route traffic through the firewall but that’s actually fairly complicated to setup properly.
The shorewall FAQ 2 discourages this and instead recommends the use of split-DNS: it’s very easy to setup and it works like a charm.

dnsmasq

Just install dnsmasq on the firewall. There are ready-made packages available for it and a simple yum install dsnmasq should suffice.

Dnsmasq provides a simple DNS forwarding and DHCP service. I had already configured dhcpd -which is already fairly simple to configure- on my firewall so I won’t need DHCP from dnsmasq but you can easily set it up if you want.

On the DNS side, dnsmasq can be told to first try to resolve hostnames by looking at the standard /etc/hosts file and then query the DNS servers defined in /etc/resolv.conf if necessary.

This simple trick means that we can:

• Keep our normal DNS service pointing to say 100.90.80.70 for www.acme.com so that people on the Internet will properly resolve their web requests to our win server.
• Add an entry in the firewall’s hosts file to point local clients to 192.168.254.20 instead.

To achieve this, simply edit /etc/hostsand add entries matching all your services:

# Acme's services. 
# One line for each DNS entries accessible from the Internet
192.168.254.20        acme.com
192.168.254.20        www.acme.com
192.168.254.10        pop.acme.com
192.168.254.10        mail.acme.com

dsnmasq configuration

Edit the /etc/dsnmasq.conf and uncomment or add the following lines:

# Never forward plain names (without a dot or domain part)
domain-needed
# Never forward addresses in the non-routed address spaces.
bogus-priv
# listen on DMZ and LAN interfaces
interface=eth2
interface=eth3
# don't want dnsmasq to provide dhcp
no-dhcp-interface=eth2
no-dhcp-interface=eth3

Then make sure that dsnmasq will start on boot:

# chkconfig dnsmasq --levels 235 on
# service dnsmasq restart

DNS resolution

There may be one last issue with DNS: in your /etc/resolv.conf you will have listed the DNS servers of one or both of your ISPs.
The problem is that some ISPs don’t allow access to their name servers from a network different than theirs.

The result is that each time any of the systems issues a DNS request it may fail and need to be sent to the next server instead, which may also fail and introduce delays in accessing named resources on the Internet.

One easy way out is to not use the ISPs DNS servers but instead only list the free OpenDNS name servers in your resolv.conf:

search acme.com
nameserver 208.67.222.222
nameserver 208.67.220.220

Then make sure that you disable DNS in your /etc/sysconfig/network-config/ifcfg-XXX configuration file for your PPPoE connection:

PEERDNS=no

Failure to do so will result in your /etc/resolv.conf file being rewritten with the DNS servers of one of your ISP every time you reconnect to them.

DHCP configuration

If you use dhcpd for local users, then you will need to make sure that its DNS server is set to the firewall’s:

# DHCP Server Configuration file.
ddns-update-style none;
ignore client-updates;

subnet 192.168.0.0 netmask 255.255.255.0 {
    option routers                  192.168.0.1;
    option subnet-mask              255.255.255.0;
    option domain-name              "acme.com";
    option domain-name-servers      192.168.0.1;
    range 192.168.0.200 192.168.0.250;
    default-lease-time 86400;
    max-lease-time 132000;
}

On your local machines that use DHCP, make sure to renew your IP.
All other machines should be configured to use 192.168.0.1 as their unique DNS server and the machines in the DMZ should have their DNS set to 192.168.254.1.

Unless you reboot, don’t forget and flush the local DNS cache of each machine:
On Windows, from the command line:

C:\> ipconfig /flushdns

On Mac, from the terminal:

bash-x.xxx$ dnscacheutil -flushcache

Initial conclusions

I believe this type of firewall setup is fairly common and I hope that the -rather long- article helped you get your own setup in place.
In the -much shorter- follow-up articles, we’ll make our system as redundant as possible so our web and email services stay online even when one of the broadband connections fails.

In the meantime, don’t hesitate to leave your comments and corrections below.

History

• 06FEB2009: added info about restarting the firewall on PPPoE reconnection and disabling ISP’s DNS resolution.
• 04FEB2009: initial release.

References

• Shorewall‘s main site.
• Shorewall’s [man page][manpage] list for all configuration files.
• OpenDNS free worldwide DNS service (and more).
• dnsmasq DNS forwarder (for easily setting up split-horizon-DNS).
• CentOS linux distribution, based on RedHat Enterprise Linux.

The Linux server-based application would scan the vast (100,000s of CAD drawings) and, depending on their format, attempt to create thumbnails and extract textual information from them and populate a database so we could easily find related drawings through a simple web-interface.

The software would present thumbnails of the drawings; when the drawing was in DWG or DXF format, the thumbnail would be an SVG vector representation of the original drawing instead of an image. That made the preview much more useful as you could zoom in and still retain enough detail to ascertain if the drawings was actually what you were looking for.

Other file formats (TIFF, HPGL plots, etc) were transformed into PNG image previews that were saved into 2 size: a small thumbnail, good enough to be displayed in a list, and a larger one that would show more details.

The server software was written in Perl, with some of the converters in C. Every night, the server would go through the whole tree of drawings on the filesystem, looking for drawings it hadn’t seen before and it would pass them to the appropriate plug-in for extracting text -where possible- and create thumbnails.

Memories of fun projects…

Download links:

• cad2svg: http://etc.nkadesign.com/Download/Cad2svg
• cad2text: http://etc.nkadesign.com/Download/Cad2text

RAID configurations explained

Doing this, I was able to remotely connect to the working server and access the disk, mount it and access the data.

Salvaging the data

Once one of the RAID1 drives was placed into the USB enclosure and connected to the other available Linux box it was easy to just remount the drives:

fdisk will tell us which partitions are interesting, assuming that /dev/sdc is our usb harddrive:

[root@fax ~]# fdisk -l /dev/sdc

Disk /dev/sdc: 81.9 GB, 81964302336 bytes
16 heads, 63 sectors/track, 158816 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes  

Device      Boot    Start         End      Blocks   Id  System
/dev/sdc1   *           1         207      104296+  fd  Linux raid autodetect
/dev/sdc2             208       20526    10240776   fd  Linux raid autodetect
/dev/sdc3           20527       22615     1052856   fd  Linux raid autodetect
/dev/sdc4           22616      158816    68645304    f  W95 Ext'd (LBA)
/dev/sdc5           22616      158816    68645272+  fd  Linux raid autodetect

We can’t simply mount the partitions, they need to be assembled into a RAID partition first:

[root@fax ~]# mdadm --assemble /dev/md6 /dev/sdc1 --run
mdadm: /dev/md6 has been started with 1 drive (out of 2).

The --run argument forces the RAID partition to be assembled, otherwise, mdadm will complain that there is only a single drive available instead of the 2 -or more- it would expect.

Now simply mount the assembled partition to make it accessible in /mnt for instance:

[root@fax ~]# mount /dev/md6 /mnt

We can now salvage our data by repeating this process for every partition.
Using RAID1 means you have at least 2 disks to choose from, so if one is damaged beyond repair, you may be lucky and the mirror one on the other drive should work.

If the drives are not physically damaged but they won’t boot, you can always use a pair (or more) of USB HDD enclosures and reconstruct the RAID arrays manually from another Linux box.

Planning for disasters

The lesson here is about planning: you can’t foresee every possible event and have contingencies for each one of them, either because of complexity or cost, but you can easily make your life much easier by planning ahead a little bit.

Most small businesses cannot afford dedicated IT staff, so they will usually end-up having the least IT-phobic person become their ‘system administrator’.
It’s your job as a consultant/technical support to ensure that they have the minimum tools at hand to perform emergency recovery, especially if you cannot intervene yourself quickly.

On-Site emergency tools

In every small business spare parts closet there should be at least:

• Whenever possible, a spare Linux box, even if it’s just using older salvaged components (like a decommissioned PC). Just have a generic Linux install on it and make sure it is configured so it can be plugged in and accessed from the network.
• a spare US$50 router, preferably pre-configured to be a temporary drop-in replacement for the existing router/firewall. Ideally, configure it to forward port 22 (SSH) to the spare Linux box so you can easily access the spare box from outside.
• USB external hard-drive enclosure.
• a spare PC power supply.
• some network cables, a couple of screwdrivers.

There are many more tools, such as rescue-CDs (like bootable Linux distributions), spare HDD, etc that can be kept but you have to remember that your point of contact need to be able to be your eyes and hands, so the amount of tools you provide should match their technical abilities.
Don’t forget to clearly label confusing things like network ports (LAN, WAN) on routers, cables and PCs.

The point is that if you can’t be on site within a short period of time, then having these cheap tools and accessories already on site mean that your customers can quickly recover just by following your instructions on the phone.
Once everything is plugged-in, you should be able to remotely carry-out most repairs.

Resources

• Linux software RAID1 setup
• Fedora System Recovery Week

Install the client-side highlighter

Download dp.SyntaxHighlighter. Uncompress its content under a new /skins/common/SyntaxHighlighter folder in your MediaWiki installation (don’t forget to make sure the files can be read by the web server; for instance, on Linux you may use chown apache.apache -R *).

In the skin template you are using for your MediaWiki site, insert the necessary code as required. In my example, I use the default /skins/MonoBook.php template into which I added the following:

Just before the closing </head> tag:
<!-- http://www.dreamprojections.com/syntaxhighlighter/ --> <link type="text/css" rel="stylesheet" href="/skins/common/SyntaxHighlighter/Styles/SyntaxHighlighter.css"></link>Just before the closing </body> tag:<!-- http://www.dreamprojections.com/syntaxhighlighter/ --> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shCore.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushCSharp.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushXml.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushSql.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushJScript.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushPhp.js"></script> <script language="javascript"> dp.SyntaxHighlighter.HighlightAll('code'); </script>

Note that you must include a reference to each source file corresponding to the type of programming language you want to highlight.
Have a look under the /skins/common/SyntaxHighlighter/Scripts/ folder to see which languages you can highlight; there are a lot more than the few I use on my site.

Install the WikiMedia extension

I’ve created a small extension to WikiMedia to allow us to enclose any source code in a new <codesyntax> tag. Click on the View Plain option below and copy-paste the following code into a new file that you will save under /extensions/syntaxhighlighter.php (again, make sure this is readable by the webserver).
<?php /************************************************************ ** Simple WikiMedia extension to create a new <codesyntax> tag that renders ** syntax highlighted through the dp.SyntaxHighlighter javascript helper ** See http://www.dreamprojections.com/syntaxhighlighter/ ** See http://meta.wikimedia.org/wiki/Extending_wiki_markup ** (c) Renaud Bompuis, 2007 ** ------------------------------------------------------------------------------ ** v1.0 - 20EFB2007 : first version. Uses the <pre> tag to ensure that the content of ** the <textarea> containing the source code will not be messed up by WikiMedia. ** There are certainly better ways to do this, but they seem more complicated than this ** (registering hooks, recording content before it is expanded and then replacing ** messed up content with the original) ************************************************************/ $wgExtensionFunctions[] = "wfSyntaxHighlighterExtension"; function wfSyntaxHighlighterExtension() { global $wgParser; $wgParser->setHook( "codesyntax", "renderCodeSyntax" ); } // The callback function for converting the input text to HTML output function renderCodeSyntax( $input, $argv, &$parser ) { $output = "<pre style='border:0;padding:0;margin:0;'>" ."<textarea name='code' class='".$argv["lang"]."'>" .Xml::escapeTagsOnly($input) ."</textarea></pre>" ; return $output; } ?>
Add the following line to the end of your LocalSettings.php file, right before the closing ?> tag.
require_once 'extensions/syntaxhighlighter.php';

Usage

To highlight code in your MediaWiki pages, just enclose your source code with the new <codesyntax> tag. This tag takes a lang attribute to specify the options that normally would be listed in the class attribute in the dp.SyntaxHighlighter documentation.

For example:
<codesyntax lang="c#"> string url = "<a href=\"" + someObj.getUrl() + "\" target=\"_blank\">l;"; // single line comments // second single line override protected void OnLoad(EventArgs e) { if(Attributes["class"] != null) { //_year.CssClass = Attributes["class"]; } // base.OnLoad(e); } </codesyntax>
Will display as:
string url = "<a href=\"" + someObj.getUrl() + "\" target=\"_blank\">"; // single line comments // second single line override protected void OnLoad(EventArgs e) { if(Attributes["class"] != null) { //_year.CssClass = Attributes["class"]; } // base.OnLoad(e); }

For more information on using dp.SyntaxHighlighter see:
http://www.dreamprojections.com/syntaxhighlighter/Usage.aspx