Posts filed under 'sysadmin'

Linux sysadmin: a short RAID trouble-shooting story

Linux I recently had an issue at a remote location (12000km away) where the old multi-purpose Linux server that had been working for the past 5 years wouldn’t boot again after a nasty power failure.
The server was used as a firewall, a local email store, a file server and a backup server, so its failure is a big deal for the small business that was using it.

RAID explained
RAID configurations explained
You can’t always have complete redundancy, so some amount of bad crash is to be expected over the years. Fortunately, I always construct my servers around a simple software RAID1 array and that leaves some hope for recovery.
In this instance, the server would start and then miserably fail in a fashion that would suggest a hardware failure of some sort. Not being able to be physically present and having no dedicated system admin on location, I directed the most knowledgeable person there to use a spare internet router to recover Internet connectivity and connect one of the disk to another Linux server (their fax server) through a USB external drive.

Doing this, I was able to remotely connect to the working server and access the disk, mount it and access the data.

Salvaging the data

Once one of the RAID1 drives was placed into the USB enclosure and connected to the other available Linux box it was easy to just remount the drives:

fdisk will tell us which partitions are interesting, assuming that /dev/sdc is our usb harddrive:

[root@fax ~]# fdisk -l /dev/sdc

Disk /dev/sdc: 81.9 GB, 81964302336 bytes
16 heads, 63 sectors/track, 158816 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes  

Device      Boot    Start         End      Blocks   Id  System
/dev/sdc1   *           1         207      104296+  fd  Linux raid autodetect
/dev/sdc2             208       20526    10240776   fd  Linux raid autodetect
/dev/sdc3           20527       22615     1052856   fd  Linux raid autodetect
/dev/sdc4           22616      158816    68645304    f  W95 Ext'd (LBA)
/dev/sdc5           22616      158816    68645272+  fd  Linux raid autodetect

We can’t simply mount the partitions, they need to be assembled into a RAID partition first:

[root@fax ~]# mdadm --assemble /dev/md6 /dev/sdc1 --run
mdadm: /dev/md6 has been started with 1 drive (out of 2).

The --run argument forces the RAID partition to be assembled, otherwise, mdadm will complain that there is only a single drive available instead of the 2 -or more- it would expect.

Now simply mount the assembled partition to make it accessible in /mnt for instance:

[root@fax ~]# mount /dev/md6 /mnt

We can now salvage our data by repeating this process for every partition.
Using RAID1 means you have at least 2 disks to choose from, so if one is damaged beyond repair, you may be lucky and the mirror one on the other drive should work.

If the drives are not physically damaged but they won’t boot, you can always use a pair (or more) of USB HDD enclosures and reconstruct the RAID arrays manually from another Linux box.

Planning for disasters

The lesson here is about planning: you can’t foresee every possible event and have contingencies for each one of them, either because of complexity or cost, but you can easily make your life much easier by planning ahead a little bit.

Most small businesses cannot afford dedicated IT staff, so they will usually end-up having the least IT-phobic person become their ‘system administrator’.
It’s your job as a consultant/technical support to ensure that they have the minimum tools at hand to perform emergency recovery, especially if you cannot intervene yourself quickly.

On-Site emergency tools

In every small business spare parts closet there should be at least:

  • Whenever possible, a spare Linux box, even if it’s just using older salvaged components (like a decommissioned PC). Just have a generic Linux install on it and make sure it is configured so it can be plugged in and accessed from the network.
  • a spare US$50 router, preferably pre-configured to be a temporary drop-in replacement for the existing router/firewall. Ideally, configure it to forward port 22 (SSH) to the spare Linux box so you can easily access the spare box from outside.
  • USB external hard-drive enclosure.
  • a spare PC power supply.
  • some network cables, a couple of screwdrivers.

There are many more tools, such as rescue-CDs (like bootable Linux distributions), spare HDD, etc that can be kept but you have to remember that your point of contact need to be able to be your eyes and hands, so the amount of tools you provide should match their technical abilities.
Don’t forget to clearly label confusing things like network ports (LAN, WAN) on routers, cables and PCs.

The point is that if you can’t be on site within a short period of time, then having these cheap tools and accessories already on site mean that your customers can quickly recover just by following your instructions on the phone.
Once everything is plugged-in, you should be able to remotely carry-out most repairs.


5 comments June 7th, 2008

MS Access: Restarting and compacting the database programmatically

Microsoft Access In my previous article about changing the MS Access colour scheme I had the need to allow the user to restart the database after the colour scheme was changed. (Article and Code Updated 13FEB2009.)

Being able to cleanly restart and compact the application is also useful in other instances:

  • Changes made to the environment
  • Recovering from errors (for instance after a network disconnection)
  • Forcing the user to re-log cleanly into the application
  • Automatically restarting a long-running application (for instance so that it may automatically compact on close and restart afresh with or without user intervention).

The problem is that you cannot -to the best of my knowledge- close and open again the same database from within MS Access itself. Most executables cannot do that and the way to solve the issue is usually to pass the control to another boostrap programme, close the main application and let the bootstrap programme re-open the main application again. I wanted a simple and clean way of using it. One that would not require shipping external dependencies.

How to use it

Download the sample database below, copy the Utilities module or just the Restart sub defined in it into your own application.

To use it, just call the Restart sub and the application will close and re-open. If you supply the optional Compact:=true parameter, the database will also be compacted during the restart process. This will work for normal databases (mdb/accdb) and also compiled (mde/accde) and runtime (accdr) databases as well.

Important note

If you want to use this code do not enable the Compact on Close option in Access for your database as the code doesn’t pick that up yet. Instead, you can either simply call restart Compact:=true on user action (for instance from a menu) or on other triggers, for instance when the database is being open and hasn’t been compacted for more than a week.

How it works

If you’re curious about the technical details, here is how it was put together. The main idea is that the MS Access database application has to be self-sufficient and restart itself by performing these steps:

  • create a small batch file
  • run the batch file, passing the path and extension of our database
  • close the main application
  • the running batch file would wait for the MS Access lock file to be removed
  • once the lock file disappears, we open the database after compacting it if required.

The key point here is that the batch file cannot just reopen the database right away: if the application is big or if it’s compacting on close for instance, it may take several seconds to actually close. The only moment we can be pretty sure that the database is effectively closed is when the lock file is deleted by MS Access.

The batch file is hard-wired in the Restart sub that does all the work:

SET /a counter=0
ping -n 1 -w 100 > nul
SET /a counter+=1
IF "!counter!"=="60" GOTO CLEANUP
"%~f1" "%~f2.%3" /compact
start " " "%~f2.%3"
del %0

When the application runs the batch file, it passes 4 arguments:

  • the full path to the MSAccess.exe executable (used for compacting the database)
  • the full path to the database without the extension
  • the database file extension without the leading “.”
  • the appropriate database lock file extension (laccdb or ldb).

This allows us to easily construct the path to either the database or the lock file at line 07 and 09. Line 08 is actually only inserted if we need to compact the database: it simply launches MSAccess.exe with the /compact command line switch.

The funny use of PING is actually a simple way to wait for some time before we check if the lock file is still there or not. There is no SLEEP or WAIT function provided by default in Windows so we have to be a bit creative and use the time-out option of the PING command trying to ping an nonexistent, but valid, IP address. Once the lock file has disappeared, we open the database at line 09 and then delete the batch file itself so we leave no leftovers.

The other thing of note is that we now use a counter to keep track of the number of times we checked the existence of the lock file. Once this counter reaches a pre-determined amount (60 by default, ~ 45 seconds) we consider that there is a problem and the database application didn’t close, so we just exit and delete the batch file.

DownloadDownload the (48KB) containing both an Access 2007 ACCDB and Access 2000 MDB test databases.

Other implementations

Code Updates

v1.2: 13FEB2009

  • Added optional parameter to compact the database during restart.

v1.1: 09AUG2008

  • Now a separate test database (used to be bundled with the Colour Scheme sample).
  • Added support for older Access versions (an Access2000 MDB is now included).
  • Corrected wrong lock file extension for accd* files.
  • Added a time-out feature after which the batch file will delete itself after a while if the Access lock file wasn’t released (for instance following a crash).
  • Added checks to delete the batch file if it has not deleted itself for some reason (for instance after a reboot).
  • The batch file now has a unique name based on the name of the database, allowing multiple databases to be restarted from the same directory.
  • Added license notice at top of source code.
  • Updated the article to reflect the changes.

v1.0: 06MAY2008

  • Original version

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 Unported License.

40 comments May 6th, 2008

“Apple’s OS Edge Is a Threat to Microsoft”. Really?

BusinessBusiness Week has a recent article where the author foresee the demise of Windows in favour of Apple’s OS.
Reading it, I couldn’t help thinking I was reading one of these overenthusiastic 1925 popular-science article promising us that within just a few years we would all use our own flying car to get to work.
Yeah, right, mine is parked right under my window.

The basic premise of the article is that Apple will attack the corporate market through the back-door, using the iPhone and its forthcoming development platform.
The author contend that development to the iPhone will drive interest in the Mac and allow Apple to displace Windows by offering more business-related applications that would eventually not require Microsoft’s OS running side-by-side in your Mac, freeing you of the dominance of the evil empire.
Please read the article first. Done?
Ok, let’s see…

First, let’s be nice: if Apple want to eat at Microsoft’s dominance of the OS market, then all the better: more competition will drive innovation and choice, which is always nice.
There is certainly some truth to the idea that Apple could offer some useful tools and technologies to the corporate world.

The problem is that author of the article was a bit too much of a Mc Fanboy(TM) to make his arguments compelling.

Let’s start direct quotes from the article.

Speaking of the ability of the Mac to quickly switch from a Windows application to a Mac by using a keyboard shortcut:

“Windows users, in the very near future, will be free to switch to Apple computers and mobile devices, drawn by a widening array of Mac software, without suffering the pain of giving up critical Windows-based applications right away.”

Useful as it may be, it’s forgetting one thing: if you need to be able to run your windows application under a Mac, you will need to give Microsoft money for the Windows license, making the newly found convenience of running the two OS together seamlessly a fairly expensive one, both in terms of computer resources and money.
While it will interest a lot of people and they may use their windows apps on Apple hardware is this really compelling enough to make it a viable migration path?

About the new Mac OS kernel:

“That kernel has proved easily adaptable across the entire Apple product line, from highly complex servers all the way down to the relatively simple iPod Touch. Such modularity allows Apple to add whatever functions are necessary for each product environment—all while maintaining cross-product compatibility.
By contrast, Microsoft has held on to an OS tethered to the 1980s, piling additions upon additions with each upgrade to Windows. With last year’s arrival of Vista, Windows has swollen to 1 billion bytes (a gigabyte) or more of software code. The “Mach” kernel of the Mac OS X, however, requires less than 1 million bytes (a megabyte) of data in its smallest configuration, expanding modestly with the sophistication of the application.”

All this is so silly and skewed that I wonder where to start.
First, kernel size is irrelevant: a basic kernel functionality does not make an operating system. Additionally, I doubt that we’re comparing the same functionality and for desktop or server applications this is utterly irrelevant.
For small devices Windows has its own flavours and they can quickly be deployed to almost any hardware with minimal effort.
Applications built for Windows built for mobile devices can be ported to Windows without too much trouble, and vice versa. Whatever platform you’re developing on, applications for small hardware are always an exercise in compromise: there is no way that you can just recompile an app for a different target and have it just work beautifully in any device. A mobile app must take care of power requirements, limited screen estate, memory and CPU limitations and specific usability issues like the presence of a touch-screen instead of a keyboard and mouse.
The point is that you have to take the platform into consideration when crafting software. Besides, using .Net makes it fairly easy to develop applications for any flavour of Windows. Additionally, whether the OS under the XBox is different or not from the desktop version of Windows is also irrelevant: you can develop games under the XNA platform that compile and work just the same under Windows as they do on the Xbox.

Now, for the argument regarding the “additions upon additions” made upon Windows that contribute to its bloat, it’s an easy shot to make and an unfair one: Apple has had no regards for its legacy applications and since its primary market was consumers, it could get away with making each of its major OS release incompatible with the previous one.
We can argue whether it was a good thing for Microsoft to keep all the quirks from previous releases of its operating system alive but we must not forget that in the corporate world, Windows has close to 100% market share on the desktop.
What that means is that Microsoft could not afford not supporting business applications across versions. it would have been suicidal to even envisage dropping compatibility.
If Apple ever get into that corporate market above a few percent it will have to guarantee compatibility between versions of its OS and programming tools if it wants to have any chance at being taken seriously at all.
When companies spend money developing an application that is critical to their business, they don’t want to hear about its supporting OS becoming incompatible every couple of years.
Apple would have to support and maintain compatibility for at least 10 years for every major OS version. So far Apple hasn’t shown that it was capable of that type of commitment.

Amipro 3.1 on my computerA small digression.
A couple of weeks ago I found some old files of mine on a floppy.
They were Ami Pro files, made at a time when Ami Pro 3.1 was the best word processor around.
Problem was that I could not safely open these files any longer: filters for various word processors would mangle the complex layout of the pages, making them un-usable. Out of curiosity, I found a full version of the original Amipro 3.1.
I had no expectations of being able to run that setup package. After all, Ami Pro 3.1 came out in 1994, before Windows 95 was even released!
Well, I clicked on that setup.exe file and watched the installation process go through… Everything went fine.
Surely, I thought, there is no way this is going to run under Windows XP. It’s going to crash for sure.
I double-clicked on the 16 colour icon and lo and behold, the whole thing actually ran! Flawlessly!
I was able to open my old files without any issue, save them under another format. Everything worked, miraculously.
That application was written at a time the Internet didn’t even exist yet I was able to install and run it without problem on a current operating system.
Back to our regular schedule.

“Despite Apple’s relative scarcity on corporate desktops, Mac laptops are already well accepted within the enterprise, with a market share of more than 20% and growing. For business travellers, the new MacBook Air, some three pounds lighter than comparable Windows-based laptops, already offers one huge advantage.”

First, Apple has a almost 20% market share on overall laptops sales only, not corporate sales.
Second, that figure is for the US only. Apple’s laptop market share in the world is certainly not bad, but it’s a quarter of that figure, making adoption of Apple laptops outside the US very small indeed.
Third, while Apple laptops are certainly sexy, they bring their own issues to businesses: lack of in-house serviceable parts, issues with making them fit into a complex infrastructure, hardware and software compatibility problems.
Bringing Macs into your organisation can be painful. Of course, it’s never impossible, but beyond simple setups, it takes time, energy and money to make stuff work seamlessly.
So I contend that, unless you have a serious commitment to Apple, most of the Macs getting into companies are for simple usage: fetching emails, browsing, making presentations and maybe Photoshop, although it seems that the application that was once the sole reason for some to use macs is now going to switch side, at least for a little while.
For Macs to become a first-class corporate platform would require that Apple makes a very serious commitment to cross-platform development to make applications work on Windows as well.
I seriously don’t think that the ability of Macs to run Windows side-by-side will do much: as mentioned above, the expense incurred offers no real benefit to businesses and having to support both platforms would increase the cost of ownership because of the upfront cost of buying and installing multiple OS, 15% higher average cost of Apple laptops compared to other similarly specified laptops and the cost of maintaining and supporting multiple hardware and OS stacks.
Slickness can only get you so far I suppose.

“While Mac desktops offer a growing number of superior features over Windows desktops”

Seriously dude, what superior features?

“Apple’s recently introduced Leopard servers compete in a market of unhappy Vista server buyers where Microsoft’s market share is only 40%. Leopard has a decent chance to expand from its small beachhead.”

Huh? Vista servers?
WTF is that?
You can’t be talking about Windows 2003 because it’s actually a rock-solid platform and there isn’t much to complain about.
Windows 2008 is just coming out and it looks just as promising.
Apple has some seriously nice server hardware, it’s so beautiful to look at all this engineering that it gets me all hot and bothered.
Seriously though, on the server side, Leopard would not be in competition with Windows but with *nix.
Server-side applications like ASP, Exchange, Sharepoint, SQL Server and the myriad other Microsoft-only server software will only work on Windows.
Apple is not going to compete as a platform for hosting these which means it will host services traditionally found on Linux/Unix/BSD systems.
In that *nix camp Apple would certainly be a good contender, although it would have to prove that it can compete with the low license and hardware cost of its competitors and offer more.

One more thing to remember: Apple is in the hardware business: it uses software to lock people into its hardware business.
I’m all for competition but at least using Linux or Microsoft products doesn’t lock me into a single hardware/software platform pair: I have orders of magnitude more choices when it comes to my servers, desktops and laptops than what Apple can offer me.
Apple products may be slick and beautifully engineered but they certainly don’t offer more freedom: chose an iPhone and get stuck with a single network provider, buy an iPod and get stuck with iTune, buy a mac and get stuck with Apple, limited software offerings, no serious games and limited hardware support.

One other thing to keep in mind is that Microsoft has acquired a huge weight in terms of software development power: new technologies have been pouring out of Redmond at a pace that is impossible to follow.
The number of developers and companies able to develop software for Windows platforms is wayyy above what Apple can dream of at the moment (we’re talking about millions of developer making a living off Windows).
Apple will need to make serious efforts to woo the huge amounts of developers it needs to bring enough business and non-business applications to its platforms before it can become a serious competitor to Windows in the corporate world.
Whether Apple has the capacity and will to do that instead of remaining a consumer-oriented company remains to be seen.

That being said I wouldn’t mind developing software for Apple platforms. I would love to be able to use a generic “surface laptop”, a sort of larger iPhone that could be used by sales people for taking orders and showing off new products.

Anyway, the point of all this was simply to offer a modest reality-check to an article that should have been a bit more measured and balanced in its fanboy-ish enthusiasm.

1 comment April 13th, 2008

MediaWiki: Formating and colouring Code

technology01.pngMediaWiki is the wiki software behind WikiPedia.
The issue, when using it as a software development tool, is formatting code in a pretty way. As we did with WordPress before, here are some details to make dp.SyntaxHighlighter work fairly seamlessly with MediaWiki.

Install the client-side highlighter

Download dp.SyntaxHighlighter. Uncompress its content under a new /skins/common/SyntaxHighlighter folder in your MediaWiki installation (don’t forget to make sure the files can be read by the web server; for instance, on Linux you may use chown apache.apache -R *).

In the skin template you are using for your MediaWiki site, insert the necessary code as required. In my example, I use the default /skins/MonoBook.php template into which I added the following:

Just before the closing </head> tag:
Just before the closing tag:

Note that you must include a reference to each source file corresponding to the type of programming language you want to highlight.
Have a look under the /skins/common/SyntaxHighlighter/Scripts/ folder to see which languages you can highlight; there are a lot more than the few I use on my site.

Install the WikiMedia extension

I’ve created a small extension to WikiMedia to allow us to enclose any source code in a new <codesyntax> tag. Click on the View Plain option below and copy-paste the following code into a new file that you will save under /extensions/syntaxhighlighter.php (again, make sure this is readable by the webserver).

Add the following line to the end of your LocalSettings.php file, right before the closing ?> tag.


To highlight code in your MediaWiki pages, just enclose your source code with the new <codesyntax> tag. This tag takes a lang attribute to specify the options that normally would be listed in the class attribute in the dp.SyntaxHighlighter documentation.

For example:

Will display as:

For more information on using dp.SyntaxHighlighter see:

2 comments February 20th, 2007

SysAdmin: When your computer becomes forgetful

BusinessSometimes your computer crashes without reason. It happens at any time, for no particular reason.
Other times you’re trying to install a new OS on a brand new PC and at some point, it fails, reboot itself or just hangs.

A couple years ago I had this really depressing experience with a brand new system I was building. All the components were newly bought, but installing the OS (a Linux distro) used to fail almost at the end of the process.
No matter how many times I tried, I could never get to the end of it.
By clever subterfuge I managed to get it installed only to have it crash on a regular basis.
It was unstable, unreliable and after two days wasted banging my head against the walls, I gave up…

Memtest86 Well, no for too long. A flash of inspiration came to me and I popped in the install CD and ran the only utility that was available at the prompt: memtest86.
That simple tool is a godsend (if I may appropriate the word from the believers. I promise to give it back). After running its various memory tests for 10 minutes it reported errors in one of the RAM sticks installed on the motherboard.
All that aggravation for a puny bit that was not remembering its state…
I promptly returned the RAM and tested the new one for a few hours until I was confident there was no issue with its chips and went my merry way to install the OS I so desperately needed. All went without a hitch.

So my advice is: go to the memtest86 website. Burn the bootable ISO and test your PC from time to time, especially if you have strange intermittent issues that you can’t pin down to a simple software problem.
You’d be amazed at the number of times I had to ditch a stick of RAM… memtest86 saved by sanity countless times.
By the way, consider donating a little bit if it helped you too. That’s always cheaper than a session with a shrink…

September 20th, 2006

Next Posts

Most Recent Posts



Posts by Month