Podcasts are a great way to get exposed to these new things: if you commute, jog, travel, you can easily use that time away from the computer to get better informed and reflect about our field.

I have a 20 minute walk between home and my office. That’s more than 3h of free time per week that I can spend thinking, listening to music or getting entertained and educated through relevant podcasts.

These are my preferred shows so far:

Hanselminutes
A 30 minute or so podcast about technologies, usually closely or distantly related to Microsoft development. Scott is an excellent interviewer and his blog is a must read too.

Herding Code
A 60 minute show with a bunch of well-known developers, K. Scott Allen, Kevin Dente, Scott Koon (Lazycoder), and Jon Galloway discussing wide-ranging topics with their guests.

.Net Rocks
The unavoidable 60 minute grand-father or all .Net podcasts. Funny and always informative. Also check out the related DnrTV for on-topic screencasts, excellent for getting started on a new technology.

The Polymorphic podcast
A nice podcast, the host, Craig Shoemaker, is a great inteviewer. Only drawback is that shows are not regularly produced.

The StackExchange podcast
Sometimes random, mildly irritating at times (guests sound quality, audio sync between guest, ramblings about Stackoverflow’s principles) but usually both enlightenning and funny. Topics revolve around Stackoverflow and the various StackExchange sites. It’s always worth listenting to Joel Spolsky, he tends to see clearly in an industry that often borders on the BS.

This Developer’s Life
Last but not least, a beautifully produced podcast that covers more philosophical topics through interviews and real-life experiences. It’s not about technology, it’s about the human aspect of being a coder, and what it means to be one.

Whichever you choose, any of these podcasts are bound to expand your knowledge, educate and entertain you!

After upgrading my development machine (Win7 x64) and a few clients (Windows 2008R2 x64) to SP1 (x86), I started to get strange issues:

• I use .Net libraries from my Access application and suddenly, even when not instantiating any .Net objects, Access would crash, usually on startup, but sometimes when opening the VBE.
  Decompiling and re-compacting the database would be OK, usually once, but the problem would reappear the next time I would restart the application.
  
• In the Runtime, I would get strange errors, such as The setting your entered isn’t valid for this property, or Action Failed Error Number: 2950, or Runtime Error 3270 Property not found.
  The strange thing about these errors is that they would occur in places that had not been modified for many releases of our application, parts that have been running without problem until now.
  
• Another weird issue was the systematic reset of our custom ribbon to its first tab. this could happen randomly, but most it could also be reproduced by simply opening a report as a tab page (that fill-in the whole MDI window). When closing that form, the first tab of the ribbon would select itself automatically. This wasn’t happening when closing pop-up windows.

After removing the Office and Runtime Service Pack 1, everything went back to normal.

A fix, finally!

A good 6 months after SP1 was released, Microsoft finally issued hotfix for the issue!
So, read about it on the MS Forums and get your KB2596585 hotfix, then decompile your database to clean it out.

Some distribution, in particular RedHat/CentOS 5, do not allow us to select ext4 as a formatting option during setup of the machine, so you will initially have to use ext3 as file system (on top of LVM preferably for easy extensibility).

A small digression on partitioning

Remember to create separate partitions for your file data: do not mix OS files with data files, they should live on different partitions. In an enterprise environment, a minimal partition configuration for a file server could look like:

Hardware:

• 2x 160GB HDD for the OS
• 4x 2TB HDD for the data

The 160GB drives could be used as such:

• 200MB RAID1 partition over the 2 drives for /boot
• 2GB RAID1 partition over the 2 drives for swap
• all remaining space as a RAID1 partition over the 2 drives for /
  Note though that it is generally recommended to create additional partitions to further contain /tmp and /var.

The 2TB drives could be used like this:

• all space as RAID6 over all drives (gives us 4TB of usable space) for /data
• alternatively, all space as RAID5 over all drives (gives us 6TB of usable space) The point of using RAID6 is that it gives better redundancy than RAID5, so you can safely add more drives later without increasing the risk of failure of the whole array (which is not true of RAID5).

Moving to ext4

If you are upgrading an existing system, backup first!

Let’s say that your /data partition is an LVM volume under /dev/VolGroup01/LogVol00. First, make sure we have the ext4 tools installed on our machine, then unmount the partition to upgrade:

# yum -y install e4fsprogs
# umount /dev/VolGroup01/LogVol00

For a new system, create a large partition on the disk, then format the volume (this will destroy all data on that volume!).

# mkfs -t ext4 -E stride=32 -m 0 -O extents,uninit_bg,dir_index,filetype,has_journal,sparse_super /dev/VolGroup01/LogVol00
# tune4fs -o journal_data_writeback /dev/VolGroup01/LogVol00

Note: on a RAID array, use the appropriate -E stride,stripe-width options, for instance, on a RAID5 array using 4 disks and 4k blocks, it could be: -E stride=16,stripe-width=48

For an existing system, upgrading from ext3 to ext4 without damaging existing data is barely more complicated:

# fsck.ext3 -pf  /dev/VolGroup01/LogVol00
# tune2fs -O extents,uninit_bg,dir_index,filetype,has_journal,sparse_super /dev/VolGroup01/LogVol00
# fsck.ext4 -yfD /dev/VolGroup01/LogVol00

We can optionally give our volume a new label to easily reference it later:

# e4label /dev/VolGroup01/LogVol00 data

Then we need to persist the mount options in /etc/fstab:

/dev/VolGroup01/LogVol00    /data    ext4    noatime,data=writeback,barrier=0,nobh,errors=remount-ro    0 0

And now we can remount our volume:

# mount /data

If you upgraded an existing filesystem from etx3, you may want to run the following to ensure the existing files are using extents for file attributes:

# find /data -xdev -type f -print0 | xargs -0 chattr +e
# find /data -xdev -type d -print0 | xargs -0 chattr +e

Important notes

The mounting options we use are somewhat a bit risky if your system is not adequately protected by a UPS.
If your system crashes due to a power failure, you are more likely to lose data using these options than using the safer defaults.
At any rate, you must have a proper backup strategy in place to safeguard data, regardless of what could damage them (hardware failure or user error).

• The barrier=0 option disables Write barriers that enforce proper on-disk ordering of journal commits.
• The data=writeback and nobh go hand in hand and allow the system to write data even after it has been committed to the journal.
• The noatime ensures that the access time is not updated when we’re reading data as this is a big performance killer (this one is safe to use in any case).

References

• Mount options to improve ext4 file system performance
• Ext4 Howto
• Migrating a live system from ext3 to ext4 filesystem

Default settings of most Linux distributions are however not optimal: they are meant to be as standard compliant and as general as possible so that everything works well enough regardless of what you do.

For a file server hosting large numbers of files, these default settings can become a liability: everything slows down as the number of files creeps up and it makes your once-snappy fileserver as fas as a sleepy sloth.

There are a few things that we can do to ensure we get the most of our server.

Checking our configuration

First, a couple of commands that will help us investigate the current state of our configuration.

• df will give us a quick overview of the filesystem:

  # df -T
  Filesystem    Type   1K-blocks      Used Available Use% Mounted on
  /dev/md2      ext3    19840804   4616780  14199888  25% /
  tmpfs        tmpfs      257580         0    257580   0% /dev/shm
  /dev/md0      ext3      194366     17718    166613  10% /boot
  /dev/md4      ext3     9920532   5409936   3998532  58% /var
  /dev/md3      ext3      194366      7514    176817   5% /tmp
  /dev/md5      ext3    46980272  31061676  13493592  70% /data
  

• tune2fs will help us configure the options for each ext3 partition. If we want to check what is the current configuration of a given partition, says we want to know the current options for our /data mount:

  # tune2fs -l /dev/md5
  

  If I was using LVM as a Volume manager, I would type something like:

  # tune2fs -l /dev/VolGroup00/LogVol02
  

  This would give lots of information about the partition:

  tune2fs 1.40.2 (12-Jul-2007)
  Filesystem volume name:   <none>
  Last mounted on:          <not available>
  Filesystem UUID:          d6850da8-af6f-4c76-98a5-caac2e10ba30
  Filesystem magic number:  0xEF53
  Filesystem revision #:    1 (dynamic)
  Filesystem features:      has_journal resize_inode dir_index filetype 
                            needs_recovery sparse_super large_file
  Filesystem flags:         signed directory hash
  Default mount options:    user_xattr acl
  Filesystem state:         clean
  Errors behavior:          Continue
  ....
  

  The interesting options are listed under Filesystem features and Default mount options. For instance, here we know that the partition is using a journal and that it is using the dir_index capability, already a performance booster.

• cat /proc/mounts is useful to know the mounting options for our filesystem (just listed some interesting ones here):

  rootfs / rootfs rw 0 0
  /dev/root / ext3 rw,data=ordered 0 0
  /dev/md0 /boot ext3 rw,data=ordered 0 0
  /dev/md4 /var ext3 rw,data=ordered 0 0
  /dev/md3 /tmp ext3 rw,data=ordered 0 0
  /dev/md5 /data ext3 rw,data=ordered 0 0
  none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
  /dev/md4 /var/named/chroot/var/run/dbus ext3 rw,data=ordered 0 0
  

  The data=ordered mount parameter tells us of the journaling configuration for the partition.

Journaling

So what is journaling?
It’s one of the great improvements of ext3: a journal is a special log on the disk that keeps track of changes about to be made. It ensures that, in case of failure, the filesystem can quickly recover without loss of information.

There are 3 settings for the journalling feature:

• data=journal the most secure but also slowest option since all data and metadata is written to disk: the whole operation needs to be completed before any other operation can be completed. It’s sort of going to the bank for a deposit, filling the paperwork and making sure the teller puts the money in the vault before you leave.
• data=ordered is usually the default compromise: you fill-in the paperwork and remind the teller to put the money in the vault asap.
• data=writeback is the fastest but you can’t be absolutely sure that things will be done in time to prevent any loss if a problem occurs soon after you’ve asked for the data to be written.

In normal circumstances all 3 end-up the same way: data is eventually written to disk and everything is fine.
Now if there is a crash just as the data was written only option journal would guarantee that everything is safe. Option ordered is fairly safe too because the money should be in the vault soon after you left; most systems use this option by default.

If you want to boost your performance and use writeback you should make sure that:

• you have a good power-supply backup to minimise the risk of power failure
• you have a good data backup strategy
• you’re ok with the risk of losing the data that was written right before the crash.

To change the journaling option you simply use tune2fs with the appropriate option:

# tune2fs -o journal_data_writeback /dev/md5

Mount options

Now that we’ve changed the available options for our partition, we need to tell the system to use them.
Edit /etc/fstab and add data=writeback to the option columns:

/dev/md5     /data    ext3    defaults,data=writeback   1 2

Next time our partition is mounted, it will use the new option. For that we can either reboot or remount the partition:

# mount - o remount /data

noatime option

There is another option that can have a very dramatic effect on performance, probably even more than the journaling options above.

By default, whenever you read a file the kernel will update its last access time, meaning that we end up with a write operation for every read!
Since this is required for POSIX compliance, almost all Linux distributions leave this setting alone by default.
For a file server, this can have such drastic consequence on performance.

To disable this time-consuming and not useful feature (for a file server), simply add the noatime option to the fstab mount options:

/dev/md5     /data    ext3    defaults,noatime,data=writeback   1 2

Note that updating access times is sometimes required by some software, such as mail software (such as mutt). If you properly keep your company data in a dedicated partition, you can enable the mount options only for that partition and keep other options for the root filesystem.

dealing with errors in fstab

After doing the above on one of the servers, I realized that I made a typo when editing /etc/fstab.
This resulted in the root filesystem being mounted read-only, making fstab impossible to edit…

To make matters worse, this machine was a few thousand miles away and could not be accessed physically….

Remounting the root filesystem resulted in errors:

# mount -o remount,rw /
mount: / not mounted already, or bad option

After much trial and rebooting, this worked (you need to specify all mounting options, to avoid the wrong defaults from being read from etc/mtab`):

# mount  -o rw,remount,defaults /dev/md2 /

After that, I could edit /etc/fstab and correct the typo…

Conclusions

How much these options will improve performance really depends on how your data is used: the improvements should be perceptible if your directories are filled with large amounts of small files.
Deletion should also be faster.

The problem is that testing for “blankness” can mean many different things to different types:

• For an Object type, the variable can be Nothing.
• For a String type, the string can have no content at all: "", vbNullString.
• For a Variant type, the string can have any of the following attributes or values:
  • it can be Missing if the variable is an unused optional parameter,
  • it can be Empty if it was never assigned,
  • it can be Null if, for instance it’s bound to a nullable field or unbound with no value,
  • it can be an empty string "", vbNullString.

When having to check these variables in code, it can be tiresome to have to go through testing some of these possibilities just to find out that your variable does or not not contains something useful, regardless of the type of variable you are using.

To avoid having to do all these tests, make the code a bit more tidy and allow me to move on to more important things, I use this small utility function quite often:

'----------------------------------------------------------------------------- ' True if the argument is Nothing, Null, Empty, Missing or an empty string . '----------------------------------------------------------------------------- Public Function IsBlank(arg As Variant) As Boolean Select Case VarType(arg) Case vbEmpty IsBlank = True Case vbNull IsBlank = True Case vbString IsBlank = (arg = vbNullString) Case vbObject IsBlank = (arg Is Nothing) Case Else IsBlank = IsMissing(arg) End Select End Function

So now I don’t have to worry so much about the type of the variable I’m testing when I want to know if it contains useful data: ... ' Here assume that CustomerReference is a control on a form. ' By using IsBlank() we avoid having to test both for Null and empty string. If IsBlank(CustomerReference) Then MsgBox "Customer Reference cannot be left blank." End If ...

Obviously, IsBlank() doesn’t replace the other tests but I found it to be more straightforward to use in most cases.

I’ve been filling the drive with pictures from my recent camera purchase and I couldn’t put it off any longer, so I bought the new drive and went online to find some good tutorial on how to crack open the Macbook Pro case.

After a few searches, I noticed that many people were referring to the iFixit.com website. It was very easy to find the tutorial I was looking for, I didn’t have to register, and each step was made very clear and simple.
It took no time to open the case and replace the drive.
I was very happy with that find.

Now, that’s not the end of the story.

A couple of days before I replaced the drive the left fan of the laptop suddenly became noisy. This would happen a few times a day, at random, and would last 10-20 minutes.
My only solution to get this repaired was to get to the local Apple service shop. Even though I knew exactly which part number was to be replaced, they still wanted me to:

• go across town to visit them so they could see for themselves what the problem was: annoying because the problem was intermittent so I may have to go for nothing.
• wait for the part to arrive a few days later.
• go back to leave the laptop
• go again to collect the repaired laptop the next day or so. So all in all: about 6h spend travelling back and forth + no laptop for a couple of day + the risk that some indiscreet technician start looking through my personal stuff.

Instead, I went back to the iFixit website:

• identified my machine
• found out the list of spare parts available from their store
• added the fan to my cart
• paid for it.
• found a guide that showed how to replace the part.

That took me all of 10 minutes; I placed my order on Thursday and the next Monday I received the part … halfway across the globe!

I also got a survey request from iFixit and left some comments, from which I got back two nice detailed email follow-ups, one from the CEO saying they were implementing my remarks as part of their site improvement efforts.

Well, I thought I would share this story. It’s not that often that you get excited by an online vendor that not only does its job well but goes beyond expectations.

Run-time Error '3155' ODBC--insert on a linked table  failed

It was driving me mad.
I could insert a simple record using SQL Server Management Studio, I could add new records to the table in datasheet mode within Access, but as soon as I tried to insert a record from code, whether using DAO recordset or executing the same SQL INSERT, it would miserably fail.

After a fair bit of investigation and tests, of which you can read the full account on the question I asked on StackOverflow, it turns out that this is a long-standing bug in the ODBC Driver (or Access).

Memo fields in Access are usually translated into nvarchar(MAX) in SQL Server by tools like SSMA.
Unfortunately, when you link tables having these fields using the SQL Server Client driver, these fields get incorrectly interpreted as string, even though they appear ok from the table design view.
It’s only if you try to insert something into the field, either text larger than 255 chars or NULL, that you get the error message.

So, the solution, at least in this case, is to revert to the older SQL Server ODBC driver instead, or use varchar() instead of nvarchar(), but if you’re dealing with Unicode, you have to stick with nvarchar().

References

• My question Inserting NULL in an nvarchar fails in MSAccess on StackOverflow.
• A reference I found on this issue: Ms Access linking table with nvarchar(max).

My problem is that currently, the Access application uses autoincrement numbers as surrogate Primary Keys (PK). Since I will need to synchronise the data over multiple remote sites, including occasionally disconnected clients, I can’t use the current autoincrement PK and will need to change to GUID.

To see for myself what could be the impact, I made a series of benchmarks.
This first part is fairly simple:

• Populate a Product table that contains 3 fields: ID, SKU and Designation with 1,000,000 records.
• Test natively on SQL Server and Access 2007.
• The records are inserted in transactions batches of 1000 records.
• I collect the time taken for each of these transactions and plot it.

Test setup

Nothing much to say about that:

All tests are performed on a dedicated Windows Server 2008 x64 rack running Access 2007 and SQL Server 2008 Standard (SP1) x64.

Test database

In SQL Server, we created a database with two tables ProductGUID and ProductInt:

CREATE TABLE ProductGUID ( ID UNIQUEIDENTIFIER DEFAULT NEWSEQUENTIALID(), SKU NVARCHAR(32) NOT NULL, Description NVARCHAR(255) NULL ); CREATE CLUSTERED INDEX ProdGUIDix ON ProductGUID(ID); GO CREATE TABLE ProductINT ( ID INT IDENTITY(1,1), SKU NVARCHAR(32) NOT NULL, Description NVARCHAR(255) NULL ); CREATE CLUSTERED INDEX ProdINTix ON ProductINT(ID); GO

For the table using a GUID, we use the NewSequentialID() instead of NewID() to create new keys. This is supposed to offer much better performance as the generated GUIDs are guaranteed to be sequential rather than random, resulting in better index performance on insertion.

For the Access version of the tables, we basically use the same definition, except that we used 4 tables:

• ProductINT: let Jet/ACE autonumbering create the sequential integer Primary Key.
• ProductINTRandom: let Jet/ACE autonumbering create the random integer Primary Key.
• ProductGUIDRandom: let Jet/ACE use its own internal GenGUID() for the key which generates random GUIDs instead of sequential ones.
• ProdcutGUIDSequential: call the Windows API (UuidCreateSequential) to create sequential ID instead.

SQL Server Test code

Using the SQL Server Management Studio, we performed the following test once for each table (resetting the database in-between): SET NOCOUNT ON; GO DECLARE @i INT = 1; WHILE (@i <= 1000) BEGIN DECLARE @tstart DATETIME2 = SYSDATETIME(); BEGIN TRAN DECLARE @a INT = 1; WHILE (@a <= 1000) BEGIN INSERT INTO ProductGUID (SKU,Description) VALUES ('PROD' + CONVERT(CHAR,@a), 'Product number ' + CONVERT(CHAR,@a)); SELECT @a = @a + 1; END; COMMIT TRAN; SELECT DATEDIFF(MILLISECOND, @tstart, SYSDATETIME()) AS timespan; SELECT @i = @i + 1; END; GO

Basically, we perform 1000 transactions each inserting 1000 records into the table ProductGUID or ProductINT.

Access 2007 Test code

To duplicate the same conditions, the following VBA code will perform 1000 transactions each inserting 1000 records.
Note that the recordset is opened in Append mode only.
The importance of this will be discussed in another article.

' Run this to inset 1,000,000 products in batches of 1000 ' In the given table Public Sub Benchmark(TableName as String, InsertSeqGUID as Boolean) Dim i As Integer For i = 1 To 1000 Insert1000Products TableName, InsertSeqGUID Next i End Sub ' Insert 1000 products in a table Public Sub Insert1000Products(TableName as String, InsertSeqGUID as boolean) Dim i As Long Dim db As DAO.Database Dim rs As DAO.Recordset Dim ws As DAO.Workspace Dim starttime As Long Dim timespan As Long Set ws = DBEngine.Workspaces(0) DoEvents starttime = GetClock ' Get the current time in ms ws.BeginTrans Set db = CurrentDb Set rs = db.OpenRecordset(TableName, dbOpenDynaset, dbAppendOnly) With rs For i = 1 To 1000 .AddNew If InsertSeqGUID Then !ID = "{guid {" & CreateStringUUIDSeq() & "}" !SKU = "PROD" & i !Description = "Product number " & i .Update Next i End With ws.CommitTrans rs.Close timespan = GetClock() - CDbl(starttime) Set rs = Nothing Set db = Nothing ' Print Elapsed time in milliseconds Debug.Print timespan DoEvents End Sub

We call this code to perform inserts on each of our Access tables:

• ProductINT table: we just insert data in the ProductINT table, letting Access create autonumber IDs.
• ProductINTRandom table: we just insert data in the ProductINTRandom table, letting Access create random autonumber IDs.
• ProductGUIDRandom table: we let Access create the Random GUID for the primary key.
• ProductGUIDSequential: we use the Windows API to create a sequential ID that we insert ourselves.

Test results

Without further ado, here are the raw results, showing the number of inserted record per second that we achieve for each test over the growing size of the database (here are only shown tests comapring Sequantial GUID and Autoincrement on SQL Server and Access, see next sections for the other results):

What we clearly see here is that performance when using autoincrement and Sequential GUID stays pretty much constant over the whole test.
That’s good new as it means that using Sequential GUIDs do not degrade performance over time.

As a side note, in this particular test, Access offers much better raw performance than SQL Server. In more complex scenarios it’s very likely that Access’ performance would degrade more than SQL Server, but it’s nice to see that Access isn’t a sloth.

Using Sequential GUID vs Autoincrement in Access

The results show that we do take a performance hit of about 30% when inserting Sequential GUID vs just using autonumbers.
We’re still getting good results, but that’s something to keep in mind.

In terms of CPU consumption, here is what we get:

Random PK, whether they are simple integer or GUID do consume substantially more CPU resources.

Using Sequential GUID vs Identity in SQL Server

Out-of-the box, SQL Server performs quite well and there is not much difference whether you’re using Sequential GUIDs or autoincrement PK.

There is however a surprising result: using Sequential GUIDs is actually slightly faster than using autoincrement!

There is obviously an explanation for this but I’m not sure what it is so please enlighten me :-)

CPU Consumption:

Using Random GUID vs Sequential GUID vs Random Autonumber in Access

So, what is the impact of choosing a Sequential GUID as opposed to letting Access create its own random GUIDs?

It’s clear that random GUIDs have a substantial performance impact: their randomness basically messes up indexing, resulting in the database engine having to do a lot more work to re-index the data on each insertion.
The good thing is that this degradation is pretty logarithmic so while it degrades over time, the overall performance remains pretty decent.
While GUIDs are larger than Random Integers (16 bytes vs 4 bytes) the actual performance of inserting records whose PK is a random integrer is actually worse than random GUID…

Provisional conclusions

Here we’ve check the baseline for our performance tests. In the next article, we’ll look exclusively at the performance of inserting data from a remote Access 2007 front end using our VBA code.

Having this baseline will allow us to check the performance overhead of using ODBC and letting Jet/ACE manage the dialogue with the SQL Server backend.

Feel free to leave your comments below, especially if you have any resources or information that would be useful.

Updates

• 16APR2009: added test of random autonumber as PK in Access.
• 13APR2009: Original Article

I’ve got this Dell R300 rack server that runs Windows Server 2008 that I dedicate to running IIS and SQL Server 2008, mostly for development purposes.

In my previous blog entry, I was trying some benchmark to compare the performance of Access and SQL Server using INT and GUID and getting some strange results.

Here are the results I was getting from inserting large amounts of data in SQL Server:

On the server, not using transactions makes the query run more than 8 seconds, at least an order of magnitude slower than it should!

I initially thought there was something wrong with my server setup but since I couldn’t find anything, I just spend the day re-installing the OS and SQL server, applying all patches and updates so the server is basically brand new, nothing else on the box, no other services, basically all the power is left for SQL Server…

Despair

When I saw the results for the first time after spending my Easter Sunday rebuilding the machine I felt dread and despair.
The gods were being unfair, it had to be a hardware issue and it had to be related to either memory or hard disk, although I couldn’t understand really why but these were the only things that I could see have such an impact on performance.

I started to look in the hardware settings:

And then I noticed this in the Policies tab of the Disk Device Properties :

Just for the lulz of it, I ticked the box, close the properties

And then tried my query again:

A nearly 35 fold increase in performance!

Moral of the story

If you are getting strange and inconsistent performance results from SQL Server, make sure you check that Enable advanced performance option.
Even if you’re not getting strange results, you may not be aware of the issue, only that some operations may be much slower than they should.

Before taking your machine apart and re-installing everything on it, check your hardware settings, there may be options made available by the manufacturer or the OS that you’re not aware of…

Lesson learnt.

While there are tools like SSMA that can help you move an Access database to SQL Server, a lot of the problems you’ll encounter can be solved before you even have to think about upsizing.
Abiding by a few simple rules will cost you nothing when creating your Access application but will save you a lot of headache if -when- the time comes to upsize.

So here are a few things to keep in mind.

Naming conventions

Access is pretty liberal about naming conventions and it will let you freely name your tables, columns indexes and queries. When these get moved to another database you’ll most probably be faced with having to rename them.
In some cases, you could actually create subtle bugs because something that used to work fine in Access may be tolerated in the new database but be interpreted differently.

• Do not use spaces or special characters in your data object names.
  Stick to characters in the range A through Z, 0 to 9 with maybe underscores _ somewhere in between (but not at the start or the end).
  Also try to respect casing wherever you reference this name (especially for databases like MySQL which are case-sensitive if the hosted on a Linux platform for instance).
  eg:
  Customer Order Lines (archive) should be CustomerOrderLines_Archive.
  Query for last Year's Turnover should be QueryLastYearTurnover.
  Index ID+OrderDate should become instead ID_OrderDate.

• Do not use keywords that are reserved or might mean something else whether they are SQL keywords or functions names:
  A column called Date could be renamed PurchaseDate for instance.
  Similarly, OrderBy could be renamed SortBy or PurchaseBy instead, depending on the context of Order.
  Failing to do so may not generate errors but could result in weird and difficult to debug behaviour.

• Do not prefix tables with Sys, USys, MSys or a tilde ~.
  Access has its own internal system tables starting with these prefixes and it’s best to stay away from these.
  When a table is deleted, Access will often keep it around temporarily and it will have a tilde as its prefix.

• Do not prefix Queries with a tilde ~.
  Access use the tilde to prefix the hidden queries kept internally as recordsource for controls and forms.

Database design

• Always use Primary keys.
  Always have a non-null primary key column in every table.
  All my tables have an autonumber column called ID. Using an automatically generated column ID guarantees that each record in a table can be uniquely identified.
  It’s a painless way to ensure a minimum level of data integrity.

• Do not use complex multivalue columns.
  Access 2007 introduced complex columns that can record multiple values.
  They are in fact fields that return whole recordset objects instead of simple scalar values. Of course, this being an Access 2007 only feature, it’s not compatible with any other database. Just don’t use it, however tempting and convenient it might be.
  Instead use a table to record Many-To-Many relationships between 2 tables or use a simple lookup to record lists of choices in a text field itself if you’re only dealing with a very limited range of multivalues that do not change.

• Do not use the Hyperlink data type.
  Another Access exclusive that isn’t available in other databases.

• Be careful about field lookups.
  When you create Table columns, Access allows you to define lookup values from other tables or lists of values.
  If you manually input a list of values to be presented to the user, these won’t get transferred when upsizing to SQL Server.
  To avoid having to maintain these lookup lists all over your app, you could create small tables for them and use them as lookup instead; that way you only need to maintain a single list of lookup values.

• Be careful about your dates.
  Access date range is much larger than SQL Server.
  This has 2 side-effects:
  1) if your software has to deal with dates outside the range, you’ll end-up with errors.
  2) if your users are entering dates manually, they could have made mistakes when entering the year (like 09 instead of 2009).
  Ensure that user-entered dates are valid for your application.

VBA

While most of your code will work fine, there are a few traps that will bomb your application or result in weird errors:

• Always explicitly specify options when opening recordsets or executing SQL.
  With SQL Server, the dbSeeChange is mandatory whenever you open a recordset for update.
  I recommend using dbFailOnError as well as it will ensure that the changes are rolled back if an error occurs. Dim rs as DAO.RecordSet ' Open for read/write set rs = db.OpenRecordSet("Stock", dbOpenDynaset, dbSeechanges + dbFailOnError) ' Open for read only set rs = db.OpenRecordSet("Stock", dbOpenSnapshot) ' Direct SQL execution CurrentDB.Execute "INSERT INTO ...", dbSeeChanges + dbFailOnError

• Get the new autonumbered ID after updating the record.
  In Access, autonumbered fields are set as soon as the record is added even if it hasn’t been saved yet.
  That doesn’t work for SQL Server as autonumbered IDs are only visible after the records have been saved. ' Works for Access tables only ' We can get the new autonumber ID as soon as the record is inserted rs.AddNew mynewid = rs!ID ... rs.Update ' Works for ODBC and Access tables alike ' We get the new autonumber ID after the record has been updated rs.AddNew ... rs.Update rs.Move 0, rs.LastModified mynewid = rs!ID

• Never rely on the type of your primary key.
  This is more of a recommendation but if you use an autonumbered ID as your primary key, don’t rely in your code or you queries on the fact that it is a long integer.
  This can become important if you ever need to upsize to a replicated database and need to transform your number IDs into GUID.
  Just use a Variant instead.

Parting thoughts

These simple rules will not solve all your problems but they will certainly reduce the number of issues you’ll be faced with when upsizing you Access application.
Using a tool like SSMA to upsize will then be fairly painless.

If you have other recommendations, please don’t hesitate to leave them in the comments, I’ll regularly update this article to included them.

References

• You will find lots of other bits of wisdom on this page: My random thoughts on SQL Server Upsizing from Microsoft Access by Tony, from Granite Consulting.
• Martin Green’s Office tips has a series of articles on Access to SQL Server migration.

One of the challenges is to upsize the current MS Access 2007 backend database to SQL Server 2008. If you try it from Access itself using the Upsizing Wizard, you may end up getting this error message:

The Upsizing Wizard only works with Microsoft SQL Server (Versions 6.50 SP5 or higher). Please log in to a SQL Server data source.

After spending some time fiddling around with SQL Server settings I couldn’t understand why I was still getting this error.
Turns out that the upsizing wizard is apparently sensitive to the version of SQL Server you’re using and it doesn’t consider SQL Server v10 (2008) as being later than v6.50…

This issue is in fact a blessing.
Microsoft provides a migration tool for upsizing MS Access database to SQL Server 2008 that’s orders of magnitude better than anything the basic wizard can do: the SQL Migration Assistant for Access, or SSMA.

SSMA lets you take a bunch of Access databases and move the tables and queries you choose to SQL Server, automatically linking them in your original database if you want.
It’s not just a one-off thing either: SSMA keeps track of the objects that where transferred and allows you to synchronise both schema and data as often as you need.

So here you are: do not use the basic MS Access Upsizing Wizard, download and use SSMA instead.

Strange COM Error

While SSMA works perfectly fine on my Windows 2008 x64 laptop, on my main Windows XP desktop it throws an exception when trying to load an Access database:

Unable to cast COM object of type ‘Microsoft.Office.Interop.Access.Dao.DBEngineClass’ to interface type ‘Microsoft.Office.Interop.Access.Dao._DBEngine’
… {00000021-0000-0010-8000-00AA006D2EA4}…

It was a COM error saying that the library for DAO couldn’t be loaded.

I couldn’t find any relevant information on the web.
After a while, I had a look at the DAO driver in
C:\Program Files\Common Files\Microsoft Shared\DAO\dao360.dll
and I noticed that the filename was in blue: that reminded me that I had set compression on the filesystem.

I disabled compression for the file and, magically, SSMA worked again…

Moral of the story: be careful about compressing your filesystem, some registered libraries and system files may work in unpredictable ways…

References

• SQL Server Migration Assistant Team Blog
• SQL Server Migration Assistant for Access main page.
• SQL Server Migration Assistant download. Note that you will have to register to get a license file that you need to save on your PC before you can use this free tool.
• “Access 2007 Upsizing to SQL Server 2008 Express in SQL Upsizing” discussion on Microsoft groups.
• “KB838594: Error message when you try to upsize your Access database by using the Upsizing Wizard” is not the problem here, but some may find the reference useful if you’re trying to upsize your MS Access database to a SQL Server database on a network.
• My random thoughts on SQL Server Upsizing from Microsoft Access from Tony’s Access MVP website, and his list of links to other resources.

Updates

• 23FEB2012: Added link to SSMA Team blog and updated download link to current version (5.2)
• 17MAR2009: Added section on the strange COM error.
• 14MAR2009: Added links to Tony’s Access MVP website.
• 05JAN2009: Original publication.

So whenever a book isn’t available here, I buy it online as a PDF.
It’s cheaper, saves postage and I can keep them handy on my desktop computer at work and my laptop.

I love Safari and being able to access such a large library of books online in such a flexible way is great, but if you’re not in the US, the experience is not always that pleasant, with the site sometimes becoming a bit too slow for comfort.

The publishers I regularly buy ebooks from are O’Reilly, when they make the PDF available, and Apress.

O’Reilly’s PDF may have your subscribtion details embedded at the bottom of each page.
It’s perfectly reasonable and doesn’t ruin the experience: I can still search, copy snippets to the clipboard and print the book if I need to keep an annotated copy.

Apress encrypt the PDF with your account email. Again, that’s fine by me, they don’t prevent me from using the book and it’s not a great annoyance to have to type the password to unlock the book.

Now comes Wrox (Wiley): they publish fine books and even have some available as ebooks.
The biggest issue I have though is that they assume that people who buy their books are all potential criminals:

• The book is as expensive in paper as it is in ebook format. That can’t be right: ebooks have zero reproduction cost while paper books have huge material costs.
• The ebook version needs to be registered with Adobe Digital Editions that locks it to a particular machine.
• You’re only allowed to download the ebook to 4 different PCs and you’ve got 2 weeks to do so.
  This seems fair, but it’s not: if I change OS or PC, I’ve already burnt 2 licenses.
• You can’t copy/paste more than a page at a time, up to 10 pages every week… that’s just a bit silly.
• Can’t copy any of the artwork, diagrams, etc.
• Doesn’t say anything about what happens if Adobe drops their particular DRM software or if I need to use the book on the next laptop I’ll buy a year from now.
• Adobe Digital Edition only supports Windows and Mac and a Sony reader. So using Linux (even though Wrox plublishes books about it) or a Kindle or any other mobile device is out of the question.

So the net advantage of buying an eBook from Wrox (Wiley) is: your guess is as good as mine.

Yeah, you can buy each chapter as a PDF: great value, at US$4.99 per chapter, the book is costing you nearly US$100. You can get the fully printed version for half the cost…
Still, I’ll concede that being able to download a particular chapter can be useful.

The bottom line is: if your book gets printed, it’s going to be pirated and distributed for free in ebook format within a couple of weeks of being published.
While thinking they are protecting their copyright, Wiley is in fact punishing people who buy their books.

I’ll stick with Apress and O’Reilly I think. At least I don’t feel treated like a thief.

12FEB2010 Update: Wrox is now trying DRM-free PDF ebooks. We can only applaud this decision. While the offering is still limited at the moment, most or all of their books will eventually be available DRM-free.

Bookshelf by Charles Kalpakian

This long article is fairly detailed but if your setup is similar it should be enough to get you going quickly.

• The basic setup
• Objectives
• Technologies
• Things to know
• Things to read
• Installing shorewall
• Preparing the system
• Shorewall configuration
• Installing our firewall rules
• DNS
• Initial conclusions
• References

The basic setup

Without further ado, this is the network configuration:

Notable things

We have 2 broadband connections:

• CYBR, a standard DSL line with a fixed IP 111.99.88.77 allocated through PPPoE.
• HKBN, a standard 100Mbps line with a fixed IP 30.40.50.62.

The network is split into different zones:

• the Internet zone, connected to our Firewall through interfaces eth0 (ppp0) and eth1.
• a Firewall zone, delimited by the firewall system itself
• a DMZ zone connected through interface eth2 for the servers we want to make visible from the Internet.
  The DMZ has its own private subnet delimited by 192.168.254.0/255.255.255.0.
• a LAN zone connected through interface eth3 so local computers can access the Internet and be protected from it.
  The DMZ has its own private subnet delimited by 192.168.0.0/255.255.255.0.

Objectives

What we want from our setup:

1. our firewall protects our DMZ and LAN from unwanted access.
2. our win server can host websites or other services.
3. our linux server can handle receiving and sending email or other services.
4. our firewall can handle incoming traffic from either ISP.
5. our firewall can load-balance local outgoing traffic across both ISP.
6. If one line fails, incoming traffic switches to the other line.
7. If one line fails, outgoing traffic switches to the other line.
8. Eventually, we want both the linux and win servers to be able to host different websites and we want the firewall to send incoming requests to the right server.

In this first article, I’ll present the setup for items 1-5.
The remaining topics will be the subject of subsequent articles of their own.

Technologies

The firewall is our primary subject. What is being discussed here is pretty much distribution-independent and should work on all flavours of Linux.

OS on the firewall system

I chose CentOS on the firewall.
Being an almost byte-for-byte identical copy of RedHat Enterprise Linux, all configuration will be identical on RedHat and its derivatives such as Fedora.

Firewall software, first try

When my firewall needs are simpler, I use the Stronger IP Firewall Ruleset from the Linux IP Masquerade HOWTO.
I started to modify the script to adapt it to my new Multi-ISP setup but things got complicated once I needed to debug routing tables.
I got it 80% of the way but tracing network connections and packet routing is complicated and time-consuming.
After a couple of days of peering into log files and wireshark capture screens, I gave up manual configuration and decided to go with something else.

Firewall software, final

The product I chose in the end is shorewall: it’s a very flexible firewall system that create the necessary iptable rules and configure most of the routing needs to properly handle complex network setup.
Shorewall is Open Source, very stable, has been out for a long time, is actively maintained and has lots of excellent documentation and examples.

Things to know

Before we get into the meat of the article, you should brush up on the following topics:

• You have some knowledge of Linux system administration, know how to configure network connections, know how to enable/disable/stop/start services, able to edit config files.
• Networking: you should know what a netmask is, what a gateway is, what a subnet is and have a passing understanding of IP classes, IP notation, what ports are for, what’s the difference between the tcp, udp, icmp protocols, what Dynamic Port Forwarding (DNAT) is, what Network Address Translation (NAT) is, what masquerading means.
• Some basic understanding of DNS and local host name resolving (using host.conf and resolv.conf)
• Some basic knowledge of what routing is for and how it works.
• Some knowledge of how the linux kernel handles network packets (NetFilter, basics of iptables).

You don’t need to be a specialist in any of these areas but any knowledge helps.
I’m far from being well versed into Netfilter and routing, it’s not often that I have to deal directly with these topics, but brushing up on these topics helped.

Things to read

Shorewall has very extensive documentation. So much so that it can be a bit daunting, not knowing where to start.
I found the following documents helpful to get me started:

• Setup Guide for multiple interfaces.
• Routing article
• Guide for handling Multi-ISP

Installing shorewall

Go to the download site list [http://shorewall.net/download.htm#Sites] and download the most appropriate binary package for your distribution.

If you get RPMs for RedHat systems, you only need to install (rpm -ivh) the following packages:

shorewall-4.X.X-X.noarch.rpm
shorewall-perl-4.X.X-X.noarch.rpm 

If you install from source, only download, compile and install the common, doc and perl packages.

Preparing the system

For shorewall to properly handle both our firewall and packet routing needs, we need to make sure that the other parts of the system are not interfering with it.

Internet lines

Make sure that your multiple internet lines are properly working on their own!

Disable routing

• Make sure that you don’t define a GATEWAY in the configuration of your network interfaces (in /etc/sysconfig/network-scripts/ifcfg-XXX) .
• If you use an (A)DSL connection, also set DEFROUTE=no if its ifcfg-XXX file as well.
• Remove the GATEWAY from the /etc/sysconfig/network file if there is one.
• Edit your /etc/sysctl.conf file and set net.ipv4.conf.default.rp_filter = 0.

Disable firewall

Disable the current firewall, for instance using the system-config-securitylevel helper tool.
Be careful if you’re directly connected to the Internet, you will be left without protection!
You can actually wait until shorewall is properly configured to disable the firewall.

Shorewall configuration

Shorewall uses a set of simple configuration files, all located under /etc/shorewall/. For exact detail of each configuration files, have a look at the list of man pages.

Zones

zones are probably the simplest configuration file.
Details in the zones man page. Here we just name the various zones we want our firewall to handle:

################################################################
#ZONE   TYPE          OPTIONS       IN                  OUT
#                                   OPTIONS             OPTIONS
fw      firewall
net     ipv4
loc     ipv4
dmz     ipv4

This just reflects our setup as highlighted in the diagram above.

Note that the fw zone is often referred to as the $FW variable instead in various configuration files.

Interfaces

Here we list all the network interfaces connected to our firewall and for which zone they apply.
Details in the interfaces man page.

################################################################
#ZONE   INTERFACE       BROADCAST       OPTIONS
net     ppp0            detect
net     eth1            detect
dmz     eth2            detect
loc     eth3            detect

Note that for our net zone, we list the 2 interfaces connected to our ISPs.
If you’re using PPPoE to connect, don;t use the interface name eth0 but use ppp0 instead.

Policy

The policy file tells shorewall which default actions should be taken when traffic is moving from one zone to another.
These default actions are taken if no other special action was specified in other configuration files.
View the policy file as a list of default actions for the firewall.
Details about this configuration file as in its man page.

################################################################
#SOURCE DEST    POLICY          LOG     LIMIT:      CONNLIMIT:
#                               LEVEL   BURST       MASK
net     net     DROP            info
loc     net     ACCEPT
dmz     net     ACCEPT
loc     dmz     ACCEPT
loc     $FW     ACCEPT
dmz     $FW     ACCEPT
$FW     net     ACCEPT
dmz     loc     DROP            info
net     all     DROP            info
all     all     DROP            info

Traffic from one zone to another needs to be explicitely ACCEPTed, REJECTed or DROPped.
For instance, loc net ACCEPT means that we allow all traffic from our local LAN to the Internet, while net all DROP means we don’t allow incoming traffic from the internet to anyone (remember this is the default action, in most cases we will override this for specific types of traffic in the rules file).
When we set the default action to DROP, we can tell shorewall to keep a trace of the details in the /var/log/messages log.

Providers

The providers file is generally only used in a multi-ISP environment.
Here we define how we want to mark packets originating from one ISP with a unique ID so we can tell the kernel to route these packets to the right interface.
Not doing this would get packets received from one interface to be routed to the default gateway instead.
The details of this configuration file are explained in the providers man page for it.

#############################################################################
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY      OPTIONS          COPY
CYBR  1      0x1  main      ppp0      -            track,balance=1  eth2,eth3
HKBN  2      0x2  main      eth1      30.40.50.61  track,balance=5  eth2,eth3

Note that the DUPLICATE columns tells shorewall that it should make a copy of the main default routing table for this particular routing table (called CYBR or HKBN depending on which ISP we refer to).
Packets are marked with number 0×1 or 0×2 so we can distinguish them during their travel through the system.
For PPPoE connections, don’t specify a GATEWAY since it’s most likely that your ISP didn’t give you one.

The most interesting part of this file are the OPTIONS: track means that we want the packets to be tracked as they travel through the system; balance tells the kernel that we want traffic coming out to be spread over both interfaces.
Additionally, we want HKBN to receive more or less 5 times more traffic than CYBR (note that this has no effect on reply packets).

The COPY columns will ensure that the routing tables created for CYBR and HKBN are copied for each internal interface, so our eth2 and eth3 interfaces know how to route packets to the right ISP.

Route Rules

For our purpsose, the route_rules file only describes how traffic should be routed through one or the other ISP we set up in /etc/shorewall/providers.
Details are in the route_rules file man page.

#####################################################################
#SOURCE             DEST               PROVIDER        PRIORITY
ppp0                -                  CYBR            1000
eth1                -                  HKBN            1000

Here we simply say that all traffic through the CYBR table should be sent to ppp0.
The PRIORITY is an ordering number that tell shorewall to consider this routing rule before it marks the packets. Since we know the packets originated from ppp0 or eth1 we don’t really need to mark them.

Masq

The masq file will contain the masquerading rules for our private interfaces: in essence, we want traffic from the local LAN and DMZ to be hidden behind our limited number of external IPs.
See the masq manpage for all the details.

#####################################################################
#INTERFACE              SOURCE           ADDRESS                 
# Ensure that traffic originating on the firewall and redirected via 
# packet marks always has the source IP address corresponding to the 
# interface that it is routed out of.
# See http://shorewall.net/MultiISP.html#Example1
ppp0                    30.40.50.62      111.99.88.77
eth1                    111.99.88.77     30.40.50.62
ppp0                    eth2             111.99.88.77
eth1                    eth2             30.40.50.62
ppp0                    eth3             111.99.88.77
eth1                    eth3             30.40.50.62

The first part ensures that the traffic coming out of our public interfaces but originating from the other is actually rewritten as originating from the right IP for the interface.
This ensures that packets leaving eth1 for instance don’t come out with the wrong source address of the other interface.
The second part of the ensures that packets from our LAN or DMZ leaving either public interfaces are doing so with the right IP address, so traffic from my desktop going through ppp0 for instance, will have its source address as 100.90.80.70.

Rules

This is the main file where we tell shorewall our basic configuration and how we want packets to be handled in the general case.
The /etc/shorewall/rules file contains the specific instructions on where to direct traffic that will override the default actions defined in the /etc/shorewall/policy file.

#####################################################################
#ACTION    SOURCE                DEST                   PROTO  
#                                                                     
SECTION NEW
# Drop and log packets that come from the outside but pretend 
# to have a local address
DROP:info  net:192.168.0.0/24    all
DROP:info  net:192.168.254.0/24  all

# Redirect incoming traffic to the correct server for WWW and email
DNAT       all                   dmz:192.168.254.20     tcp   www
DNAT       all                   dmz:192.168.254.10     tcp   110
DNAT       all                   dmz:192.168.254.10     tcp   143
DNAT       all                   dmz:192.168.254.10     tcp   25

In its most basic form, what we’ve just defined here is that we want all traffic from anywhere destined for port 80 (www) to be sent to our win server.
All mail traffic, POP3 (port 110), IMAP (port 143) and SMTP (port 25) is to be redirected to our linux server in the DMZ.

There are a few more useful rules that we can include, for instance, I want to be able to access my servers through either ISPs from home (IP 123.45.67.89) and disallow everyone else from accessing it.

#####################################################################
#ACTION    SOURCE                DEST                   PROTO  
#                                                                     
# Allow SSH to the firewall from the outside only from home

ACCEPT     net:123.45.67.89      $FW                    tcp   ssh
# Redirect input traffic to the correct server for RDP, VNC and SSH 
DNAT       net:123.45.67.89      dmz:192.168.254.10:22  tcp   2222
DNAT       net:123.45.67.89      dmz:192.168.254.10     tcp   5901
DNAT       net:123.45.67.89      dmz:192.168.254.20     tcp   3389

When I SSH to 30.40.50.62 or 100.90.80.70, on the normal port 22, I will access the firewall.
Now if I SSH to the non-standard port 2222, I will instead access the linux server.
Ports 5901 are for remoting through VNC on the linux machine, and port 3389 will be used for Remote Desktop connections to the win server.

To make sure my machines are up and running, I like to be able to ping them:

#####################################################################
#ACTION    SOURCE              DEST              PROTO  
#                                                                     
# Accept pings between zones
ACCEPT     dmz                 loc               icmp  echo-request
ACCEPT     loc                 dmz               icmp  echo-request

Note that ping will only work between the LAN and the DMZ and pinging my firewall from the Internet will result in the requests being silently dropped.
I usually prefer that configuration as it makes discovering the servers by random bots slightly less likely.

There are lots of other cool things we can do with forwarding but that will do for now.

shorewall.conf

The last file we’re going to look at is the main configuration file for shorewall.
See details about each option from the man page for shorewall.conf.

Most options are OK by default. The only ones that I have had to change are:

STARTUP_ENABLED=Yes
MARK_IN_FORWARD_CHAIN=Yes
FASTACCEPT=Yes
OPTIMIZE=1

The first option tells shorewall that we want it to start automatically when the system boots.
That’s not enough though, so make sure that the service will be started:

# chkconfig shorewall --levels 235 on

Installing our firewall rules

Shorewall configuration files need to be compiled without error before the firewall is actually loaded by shorewall.
The command:

# shorewall restart

will stop and recompile the current configuration.
If there are any errors, the current firewall rules will be unchanged.
There are lots of other commands that can be issued. Check the man page for a complete list.

If you use PPPoE, you will want the firewall to be restarted every time the line reconnects.
The simplest way is to create a file /etc/ppp/if-up.local with only a single line:

shorewall restart

DNS

There is one remaining issue with our firewall: if a user on the LAN attempts to access the web server by its name the request will probably fail.
Same for accessing our mail server: we can configure our desktop to connect to 192.168.254.10 to get and send emails, but on the laptop we would usually use something like pop.acme.com instead so we can read our emails from outside the office.

Similarly, trying to access www.acme.com hosted on the win server from the linux server will fail.

One solution is to route traffic through the firewall but that’s actually fairly complicated to setup properly.
The shorewall FAQ 2 discourages this and instead recommends the use of split-DNS: it’s very easy to setup and it works like a charm.

dnsmasq

Just install dnsmasq on the firewall. There are ready-made packages available for it and a simple yum install dsnmasq should suffice.

Dnsmasq provides a simple DNS forwarding and DHCP service. I had already configured dhcpd -which is already fairly simple to configure- on my firewall so I won’t need DHCP from dnsmasq but you can easily set it up if you want.

On the DNS side, dnsmasq can be told to first try to resolve hostnames by looking at the standard /etc/hosts file and then query the DNS servers defined in /etc/resolv.conf if necessary.

This simple trick means that we can:

• Keep our normal DNS service pointing to say 100.90.80.70 for www.acme.com so that people on the Internet will properly resolve their web requests to our win server.
• Add an entry in the firewall’s hosts file to point local clients to 192.168.254.20 instead.

To achieve this, simply edit /etc/hostsand add entries matching all your services:

# Acme's services. 
# One line for each DNS entries accessible from the Internet
192.168.254.20        acme.com
192.168.254.20        www.acme.com
192.168.254.10        pop.acme.com
192.168.254.10        mail.acme.com

dsnmasq configuration

Edit the /etc/dsnmasq.conf and uncomment or add the following lines:

# Never forward plain names (without a dot or domain part)
domain-needed
# Never forward addresses in the non-routed address spaces.
bogus-priv
# listen on DMZ and LAN interfaces
interface=eth2
interface=eth3
# don't want dnsmasq to provide dhcp
no-dhcp-interface=eth2
no-dhcp-interface=eth3

Then make sure that dsnmasq will start on boot:

# chkconfig dnsmasq --levels 235 on
# service dnsmasq restart

DNS resolution

There may be one last issue with DNS: in your /etc/resolv.conf you will have listed the DNS servers of one or both of your ISPs.
The problem is that some ISPs don’t allow access to their name servers from a network different than theirs.

The result is that each time any of the systems issues a DNS request it may fail and need to be sent to the next server instead, which may also fail and introduce delays in accessing named resources on the Internet.

One easy way out is to not use the ISPs DNS servers but instead only list the free OpenDNS name servers in your resolv.conf:

search acme.com
nameserver 208.67.222.222
nameserver 208.67.220.220

Then make sure that you disable DNS in your /etc/sysconfig/network-config/ifcfg-XXX configuration file for your PPPoE connection:

PEERDNS=no

Failure to do so will result in your /etc/resolv.conf file being rewritten with the DNS servers of one of your ISP every time you reconnect to them.

DHCP configuration

If you use dhcpd for local users, then you will need to make sure that its DNS server is set to the firewall’s:

# DHCP Server Configuration file.
ddns-update-style none;
ignore client-updates;

subnet 192.168.0.0 netmask 255.255.255.0 {
    option routers                  192.168.0.1;
    option subnet-mask              255.255.255.0;
    option domain-name              "acme.com";
    option domain-name-servers      192.168.0.1;
    range 192.168.0.200 192.168.0.250;
    default-lease-time 86400;
    max-lease-time 132000;
}

On your local machines that use DHCP, make sure to renew your IP.
All other machines should be configured to use 192.168.0.1 as their unique DNS server and the machines in the DMZ should have their DNS set to 192.168.254.1.

Unless you reboot, don’t forget and flush the local DNS cache of each machine:
On Windows, from the command line:

C:\> ipconfig /flushdns

On Mac, from the terminal:

bash-x.xxx$ dnscacheutil -flushcache

Initial conclusions

I believe this type of firewall setup is fairly common and I hope that the -rather long- article helped you get your own setup in place.
In the -much shorter- follow-up articles, we’ll make our system as redundant as possible so our web and email services stay online even when one of the broadband connections fails.

In the meantime, don’t hesitate to leave your comments and corrections below.

History

• 06FEB2009: added info about restarting the firewall on PPPoE reconnection and disabling ISP’s DNS resolution.
• 04FEB2009: initial release.

References

• Shorewall‘s main site.
• Shorewall’s [man page][manpage] list for all configuration files.
• OpenDNS free worldwide DNS service (and more).
• dnsmasq DNS forwarder (for easily setting up split-horizon-DNS).
• CentOS linux distribution, based on RedHat Enterprise Linux.

Here I present the tools I’ve tried and what I’ve finally settled on as possibly the ultimate answer to the problem of synchronising files and folders across multiple computers:

• rsync
• unison
• WinSCP
• General Backup tools
• Revision Control Systems
• Complex setup
• What we want from data synchronisation
• Live Mesh folders
• Conclusion
• References

rsync

I’ve tried rsync, which is a great Open Source tool to securely synchronise data either one-way or both-ways.
It’s very efficient with bandwidth as it only transfer blocks of data that have actually changed in a file instead of the whole file. It can tunnel traffic across SSH and I’ve got a few cronjobs set up between various servers to back-up files daily.

It’s only weaknesses are that:

• Every time it runs, it needs to inspect all files on both sides to determine the changes, which is quite an expensive operation.
• Setting up synchronisation between multiple copies of the data can be tricky: you need to sync your computers in pairs multiple times, which quickly becomes expensive and risky if you have the same copy across multiple computers.
• It doesn’t necessarily detect that files are in use at the time of the sync, which could corrupt them.

unison

It a folder synchronisation tool whose specific purpose is to address some of the shortcomings of rsync when synchronising folders between computers. It’s also a cross-platform Open Source tool that works on Linux, OS/X, Windows, etc.

Unison uses the efficient file transfer capabilities of rsync but it is better at detecting conflicts and it will give you a chance to decide which copy you want when a conflict is detected.

The issue though is that, like rsync, it needs to inspect all files to detect changes which prevents it from detecting and propagating updates as they happen.

The biggest issue with these synchronisation tools is that they tend to increase the risk of conflict because changes are only detected infrequently.

WinSCP

WinSCP Is an Open Source Windows GUI FTP utility that also allows you to synchonise folders between a local copy and a remote one on the FTP server.

It has conflict resolution and allows you to decide which copy to keep.

It’s great for what it does and allows you to keep a repository of your data in sync with your local copies but here again, WinSCP needs to go through each file to detect the differences and you need to sync manually each computer against the server, which is cumbersome and time consuming.

General Backup tools

There are lot more tools that fall into that category of backup utilities: they all keep a copy of your current data in an archive, on a separate disk or online. Some are great in that they allow you to access that data on the web (I use the excellent JungleDisk myself) but file synchronisation is not their purpose.

Now for some Captain Obvious recommendation: remember that file synchronisation is not a backup plan: you must have a separate process to keep read-only copies of your important data.
File synchronisation will update and delete files you modify across all your machines, clearly not what you want if you need to be able to recover them!

Revision Control Systems

Revision control software like cvs, subversion, git, etc are generally used to keep track of changes of source code files; however, they have also been used successfully to keep multiple copies of the same data in sync.
It’s actually exactly what I use for all my source code and associated files: I have a subversion server and I check-out copies of my software project folders on various computers.

After making changes on one computer, I commit the changes back to the server and update these changes on all other computers manually.

While great at keeping track of each version of your files and ideally suited to pure text documents like source code, using revision control systems have drawbacks that make them cumbersome for general data synchronisation:

• you need to manually commit and update your local copies against the server.
• not all of them are well suited to deal with binary files
• when they work with binary files, they just copy the whole file when it changed, which is wasteful and inefficient.

Revision Control System are great for synchronising source code and configuration files but using them beyond that is rather cumbersome.

Complex setup

All of the above solutions also have a major drawback: getting them to work across the Internet requires complex setup involving firewall configurations, security logins, exchange of public encryption keys in some cases, etc.

All these are workable but don’t make for friendly and piece-of-mind setup.

What we want from data synchronisation

I don’t know about you but what I’m looking for in a synchronisation tool is pretty straightforward:

• Being able to point to a folder on one computer and make it synchronise across one or multiple computers.
• Detect and update the changed files transparently in the background without my intervention, as the changes happen.
• Be smart about conflict detection and only ask me to make a decision if the case isn’t obvious to resolve.

Live Mesh folders

Enters Microsoft Live Mesh Folders, now in beta and available to the public. Live Mesh is meant to be Microsoft answer’s to synchronising information (note, I’m not saying data here) across computers, devices and the Internet.
While Live Mesh wants to be something a lot bigger than just synchronising folders, let’s just concentrate on that aspect of it.

Installing Live Mesh is pretty easy: you will need a Windows Live account to log-in but once this is done, it’s a small download and a short installation.

Once you’ve added your computer to your “Mesh” and are logged in you are ready to use Live Mesh:

• You decide how the data is synchronised for each computer participating in your Mesh:
  you’re in charge of what gets copied where, so it’s easy to make large folders pair between say your laptop and work desktop and not your online Live Desktop (which has a 5GB limit) or your computer at home. You’re in control.
• Files are automatically synchronised as they change across all computers that share the particular folder you’re working in.
  If the file is currently used, it won’t be synced before it is closed.
• If the other computers are not available, the sync will automatically happen as they are up again.
• There is no firewall setup: each computer knows how to contact the others and automatically -and uses- the appropriate network: transfers are local if the computers are on the same LAN or done across the Internet otherwise.
  All that without user intervention at all.
• Whenever possible, data is exchanged in a P2P fashion where each device gets data from all the other devices it can see, making transfers quite efficient.
• File transfers are encrypted so they should be pretty safe even when using unsafe public connections.
• If you don’t want to allow sync, say you’re on a low-bandwidth dialup, you can work offline.
• The Mesh Operating Environment (MOE) is pretty efficient at detecting changes to files. Unlike other systems, in most cases it doesn’t need to scan all files to find out which ones have been updated or deleted.

Some drawbacks

• It’s not a final product, so there are some quirks and not all expected functionalities are there yet.
• The Mesh Operating Environment (MOE) services can be pretty resource hungry, although, in fairness, it’s not too bad except that it slows down your computer’s responsiveness while it loads at boot time.
• You can’t define patterns of files to exclude in your folder hierarchy.
  That can be a bit annoying if the software you use is often creating large backup files automatically (like CorelDraw does) or if there are sub folders you don’t need to take everywhere.
• The initial sync process can take a long time if you have lots of files.
  A solution if you have large folders to sync is to copy them first manually on each computer and then force Live Mesh to use these specific folders: the folders will be merged together and the initial sync process will be a lot faster as very little data needs to be exchanged between computers.

Bear in mind that Live Mesh is currently early beta and that most of these drawback will surely be addressed in the next months.

Conclusion

I currently have more than 18GB representing about 20,000 files synchronised between 3 computers (work desktop, laptop and home desktop) using Live Mesh.

While not 100% there, Live Mesh Folder synchronisation is really close to the real thing: it’s transparent, efficient, easy to use and it just works as you would expect.

Now that Microsoft has released the Sync Framework to developers, I’m sure that other products will come on the market to further enhance data synchronisation in a more capable way.
In the meantime, Live Mesh has answered my needs so far.

References

• Wikipedia reference on file synchronisation.
• rsync home page and Wikipedia entry
• unison home page and Wikipedia entry
• JungleDisk online backup (cheap and very configurable, uses Amazon S3 for storage)
• Microsoft Live Mesh web site and Live Mesh Wikipedia entry

The good thing is when you delete a partition using the Windows Disk Management console it doesn’t actually delete your files, only the partition header.

With NTFS files systems, there is a backup at the end of the partition. The problem is how do you recover it?

I first looked at the instructions from Microsoft knowledge base article kb245725, downloaded the low-level sector editor Dskprobe but was getting no-where with it.

Searching google brings you to the usual list of recovery software that you can’t be sure will actually do the job until you fork $$ for them.
I’ve got nothing against paying for software but I’ve been bitten by false promises before.

My search ended up with TestDisk an OpenSource utility to manipulate and recover partitions that works on almost all platforms.
The user interface is DOS only, so it’s not pretty, not point-and-click user friendly but it has a fair amount of options and after fiddling around with it for 10 minutes, I was able to simply recover the backup boot sector and tada! all my files were back!

So, some recommendations when recovering lost partitions:

• Don’t panic! If you only deleted the partition (whichever type), chances are you’re likely to recover it or at least salvage the files.
• Obviously, be careful not to write anything over them, like recreating partitions and a file system.
• If you use a utility like TestDisk, don’t blindly follow the on-screen instructions. At first, it was telling me that I had 2 Linux partitions on the device (which used to be true) but it did not see the NTFS one. Then it thought I had a FAT partition only until I switched to the advanced options and inspected the boot partition.
  Just know enough about file systems to know what you’re looking for.
• Low-level tools are not for everyone, so if you’re not comfortable using them, don’t tempt your luck and try a paid-for recovery tool with an easier interface.

If you use TestDisk and you manage to recover your files, don’t forget to donate to encourage Christophe GRENIER, the author.

References

• KB245725: How To Recover an Accidentally Deleted NTFS or FAT32 Dynamic Volume.
• TestDisk data recovery utility for Windows, Linux, OS/X, etc
• PhotoRec Digital Picture and File Recovery Open Source utility from the same author.

With OpenOffice 3, things got trickier and applications that used to work will now fail. I’ve just spend nearly a day trying to get a mail-merge application I built to work again with the new version of OO.

Changes and limitations

Developing .Net applications with OpenOffice 3 now requires that the .Net framework 3.5 be installed.
Only one version of OpenOffice/StarOffice must be installed.

Getting the CLI assemblies in your project

OpenOffice used to ship with a bunch of CLI DLLs for manipulating Open Documents from .Net. With OpenOffice version 3, these DLLs are directly installed in the GAC and not available in the Program installation folder any longer, making them unavailable when you need to add them as references to your Visual Studio project.

The trick is to extract the DLLs from the installation CAB, then reference those and make sure you exclude them in your setup file so only the latest version installed on the user’s computer’s GAC will get used.

1. Open the installation folder where the installation files are extracted (usually left on your desktop under a folder like OpenOffice.org 3.0 (en-US) Installation Files during the OO installation process.
2. Open the openofficeorg1.cab file (using a utility like 7zip if necessary) and extract the files matching cli_*.dl.
3. Add the ones you need to your VS Project’s references and make sure the properties of these references have their Copy Local and Specific Version properties set to false.
4. If you have a setup project that added these references to the list of files, right-click each of them and select Exclude to make sure they won’t be packaged.

The reason for not deploying the DLLs is that they are very specific to a given OpenOffice version and the latest assemblies relevant to the user’s OpenOffice installation will already be deployed to the target machine’s GAC.
When .Net loads assemblies, if they are not included in the path of the application being launched, they will be loaded from the GAC if they can be found there.

Path and environment setup

Before being able to call OpenOffice from your application you now must set your application’s environment variables to the correct paths so the assemblies can find the actual OpenOffice library and program files.

Basically, you need to add to your PATH the path to the folder where the UNO java libraries reside. You also need to add a UNO_PATH environment variable that points to the program folder of OpenOffice. Basically, before any call to OpenOffice functions you must:

• Append to PATH something like C:\Program Files\OpenOffice.org 3\URE\bin.
• Create a UNO_PATH variable set to C:\Program Files\OpenOffice.org 3\program.

Because there is no guarantee that these paths will not change or are valid for all systems you must get them from specific keys located in the Registry:

• PATH is appended with the vaue of HKLM\SOFTWARE\OpenOffice.org\Layers\URE\1\UREINSTALLLOCATION to which you must append the bin directory.
• UNO_PATH is set to the content of the HKLM\SOFTWARE\OpenOffice.org\UNO\InstallPath key.

See the C# and VB.Net code below for working examples.

Special Considerations for x64 systems

My development machine runs Windows Server 2008 x64 and I’ve ran into some specific issues that you’re likely to encounter when deploying to a 64 bits OS.

OpenOffice is 32 bits only
That means that your .Net project must be set to target x86 systems only:
Open your Solution’s Configuration Manager and under Active solution platform click New… then:

Make sure you repeat this for both the Debug and Release configurations.

Registry keys are elsewhere
32 bit applications see their registry keys normally expected under:
HKEY_LOCAL_MACHINE\Software moved to:
HKEY_LOCAL_MACHINE\Software\Wow6432Node instead.
This of course creates issues when you’re trying to read a registry key that’s not where it should be…

The Code

The code below will allow you to correctly connect to OpenOffice 3 under 32 or 64 bit systems.
It reads the registry to find the proper paths and appends the PATH and creates the UNO_PATH environment variables expected by the the bootstrapper to find the OpenOffice program and libraries.

The code is built upon information and a test program made available by Marten Feldtmann on his blog (more information, in English, is available on OOoForum ).

Please let me know if this works for you or if you have any corrections. private void InitOpenOfficeEnvironment() { string baseKey; // OpenOffice being a 32 bit app, its registry location is different in a 64 bit OS if (Marshal.SizeOf(typeof(IntPtr)) == 8) baseKey = @"SOFTWARE\Wow6432Node\OpenOffice.org\"; else baseKey = @"SOFTWARE\OpenOffice.org\"; // Get the URE directory string key = baseKey + @"Layers\URE\1"; RegistryKey reg = Registry.CurrentUser.OpenSubKey(key); if (reg==null) reg = Registry.LocalMachine.OpenSubKey(key); string urePath = reg.GetValue("UREINSTALLLOCATION") as string; reg.Close(); urePath = Path.Combine(urePath, "bin"); // Get the UNO Path key = baseKey + @"UNO\InstallPath"; reg = Registry.CurrentUser.OpenSubKey(key); if (reg==null) reg = Registry.LocalMachine.OpenSubKey(key); string unoPath = reg.GetValue(null) as string; reg.Close(); string path; path = string.Format ("{0};{1}", System.Environment.GetEnvironmentVariable("PATH"), urePath); System.Environment.SetEnvironmentVariable("PATH", path); System.Environment.SetEnvironmentVariable("UNO_PATH", unoPath); }

In VB.Net: Private Sub InitOpenOfficeEnvironment() Dim baseKey As String ' OpenOffice being a 32 bit app, its registry location is different in a 64 bit OS If (Marshal.SizeOf(GetType(IntPtr)) = 8) Then baseKey = "SOFTWARE\Wow6432Node\OpenOffice.org\" Else baseKey = "SOFTWARE\OpenOffice.org\" End If ' Get the URE directory Dim key As String = (baseKey + "Layers\URE\1") Dim reg As RegistryKey = Microsoft.Win32.egistry.CurrentUser.OpenSubKey(key) If (reg Is Nothing) Then reg = Registry.LocalMachine.OpenSubKey(key) End If Dim urePath As String = CType(reg.GetValue("UREINSTALLLOCATION"),String) reg.Close urePath = Path.Combine(urePath, "bin") ' Get the UNO Path key = (baseKey + "UNO\InstallPath") reg = Microsoft.Win32.Registry.CurrentUser.OpenSubKey(key) If (reg Is Nothing) Then reg = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(key) End If Dim unoPath As String = CType(reg.GetValue(Nothing),String) reg.Close Dim path As String path = String.Format ("{0};{1}",System.Environment.GetEnvironmentVariable("PATH"),urePath) System.Environment.SetEnvironmentVariable("PATH", path) System.Environment.SetEnvironmentVariable("UNO_PATH", unoPath) End Sub

Updates

07MAY2009 -- Added reference link to OOo documentation.
03DEC2008 -- Added VB.Net translation. Thanks to Stefan for suggesting it.
30JAN2009 -- Added reference to Aleksandr Sazonov’s article on CodeProject (thanks for the the article).

References

• CLI Language Binding
• OOoForum thread on getting sample C# code to work with OpenOffice 3
• OOoForum thread on upgrading code from OpenOffice 2.4 to 3
• Programming with .Net quick start guide.
• CLI UNO bridge not working
• Cross-compatible spreadsheet processor in .Net for Excel and OOo Calc, article from Aleksandr Sazonov on CodeProject.
• Writing client programs -- from the OOo documentation

The ‘Microsoft.Jet.OLEDB.4.0′ provider is not registered on the local machine:

It may look otherwise, but this error is generally due to either of two thing:

• you don’t have Office 2007/2010 Jet drivers installed
• or you are running a 32 bit application in a default x64 environment.

The first issue is easy to solve, just download the Access 2010 Database Engine from Microsoft (works with Access 2007 databases as well).

For the second one, the fix is also easy enough:

• For Windows 2008: Navigate to Server Manager > Roles > Web Server (IIS) > Internet Information Services (IIS) Manager, then look under your machine name > Application Pool.
• For Windows 7: Navigate to Programs > Administrative Tools > Internet Information Services (IIS) Manager, then look under your machine name > Application Pool.

Under there you can call the DefaultAppPool’s advanced settings to change Enable 32-Bits Applications to True:

You may have to restart the service for it to take effect but it should work.

References

• Scott Hanselman’s blog entry about undestanding 64 vs. 32 bitness in .Net.
• Access 2010 Database Engine driver

Updates

• 10DEC2011: Updated driver link to use the Access 2010 engine.
• 03APR2010: Added instructions for Windows 7
• 12FEB2009: Added reference to Scott’s article.
• 28OCT2008: Original version

The Great

Hardware delight

Whether running OS/X or Windows 2008 I’ve got no major complaint about the performance of the machine. It’s fast, stable (except sometimes it’s not waking up from sleep or it does but the screen remains black). The screen is nice and vibrant, I just love the magnetic power connector and the small size of the power adapter. I have a few complaints though, see below.

OSX battery Power usage

For such a large and powerful laptop I’m pleasantly surprised with the duration of the battery under OSX: I’ve been able to watch videos for 3h, full screen, without trouble and overheating (although I would lower the screen brightness to reduce consumption).
I haven’t had such luck under Windows 2008 where I’ve been struggling to find the right power settings balance, but remember that’s a server OS and it’s not really meant to be run on a laptop.

The Ugly

The mouse

You wonder why Apple, in all its hardware expertise could design the mighty-mouse with a single big button that can still do right-clicking but can’t give us the same thing with the enormous single-button of the trackpad.
Now the new models -just released this week- have done away with the button entirely, which may be just as well although I’m curious about how well the drivers will work under Windows.

Mouse acceleration in OSX is pretty frustrating to me. When you’ve got a large screen, you’re endlessly shuffling the mouse to get that pointer in the right place. It feels slow, inaccurate and is extremely irritating after a while. The problem is even worse when you’re working in OSX under VMware Fusion: while it might still be usable under OSX, the difference is really severe and unnatural in Windows.
This does not happen under bootcamp though where mouse acceleration behaves as you would expect (for windows).
I’ve tried a number of utilities (iMouse , SteerMouse and others) but none gave me what I needed.

The keyboard

The keyboard feels great to type on, with a nice spring and softness. There are a few issues though:
Why is the Return key so small compared to the right Shift?
The rule is that the more a key is used, the bigger it is, yet the Enter is rather small, it’s the same size as the caps lock which serves almost no useful purpose in comparison.
The arrow keys are also minuscule, another probable example of Apple choosing form over function. The lack of delete key forces you to type both the fn + backspace keys instead, which is an unnecessary pain, it’s not like understanding the difference between delete and backspace is that confusing to people using a complex machine like a computer.

The lid

I love the way the hooks for the lid come out just when you close it. It makes for a neat screen without protruding bits of metal or plastic.
My main issue with the lid is with its limited opening angle: if you’re just a little tall and you place your laptop on your lap then there is no way to open the screen enough to have it properly face you.
This is also an issue if you place your Macbook Pro on a cooling pad or a riser that’ll put the laptop too vertical (for instance to free some space around it when using an external keyboard): you just can’t use these devices.

The sound

That one really makes me mad: the MacBook Pro has audio issues that you won’t even find in a US$15 MP3 player and it’s totally unacceptable.
When idle, I can hear hissing sounds that vary in power and frequency if I slide the volume control; when playing music, there is a lot of noise and “cutting” sounds between songs.
These are not noticeable when using the integrated speakers but,they become really annoying once you use earphones.
I am sorely disappointed with sound quality to say the least.

Electronic noises

On top of the annoying sounds from the sound chipset, the LCD inverter also makes a hissing sound that increases in volume when I lower the LCD brightness…
Coupled with that, the processor also makes a hissing noise when it gets into its C4 power saving state…
The noise is probably not high enough to get the laptop fixed but it might be great as a mosquito repellent.

Boot ‘song’

Apple knows best and they know that your only aim in life is to become a poster boy for the brand.
When booting/rebooting your mac, it likes to play its welcome song that says “hey, over here, I’m a mac and I’m telling the world I’m booting up. Everyone listen how awesome I am!”.
The perverse thing is that even if you have earphones plugged in (as in: you don’t want to disturb people around you) the boot song will be played on the speakers…
Of course, there is no option anywhere to disable it.
Apple knows best.
After much trials, I found that booting under OSX, lowering the volume to zero, then rebooting under windows and changing the volume there would be OK: no more booting song -at least for now- and I can still change the volume in OSX and Windows.

Conclusion

Would I buy a Mac again knowing what I know now?
Mmmm, probably not.
I find the annoyances a bit too much relative to my expectations and my usage scenario. To be fair, it’s not all bad and most of the time I’m happy with my mac but I find myself trying to avoid the things that infuriate me and it’s not really what I want from a laptop; it’s supposed to liberate me and give me the freedom I need to get things done, not get in the way.
Re-adapting to a strangely layout keyboard, having to deal with Apple’s brand awareness arrogance, battling with stuff that you just normally take for granted, all this is a bit too much of a price to pay for what is essentially for me a Windows development machine.
I prefer to waste my time actually getting things done rather than searching forums on how to keep Windows and OSX time in sync or bring back my machine for repair if a CD stays stuck in the CD Drive.

So, let’s say that our relationship is more ambivalent than needed to be.

References

Other links to pages of interest.

• Things that suck about Mac OS X
• Mac OS X annoyances: What’s up with the mouse acceleration?
• A manual setting to change the speed factor in OSX
• The Steermouse driver to unleash your mouse in OSX
• The iMouse utility to correct pointer acceleration in OSX.
• Another mouse utility called iMouse, for performing right-clicking in Windows this time.

I use VMware on my main desktop to keep multiple OS setups that match typical configurations of my customer’s machines.
This allows me to test my software before deployment and make sure everything works as expected. It saved me many times from strange bugs and I would consider these final tests to be a mandatory step before deployment.
My old trusty vaio would be hard pressed to run any of these without slowing down to a crawl.

I looked at some possible replacements. Initially I checked Lenovo’s offerings but they don’t seem to offer anything in large screen size (WUXGA 1920×1200) (Note, actually, they have, but not really for me).
Dito for Dell, not counting their humongous XPS M1730 luggable gaming machine that was wayyy over the top as a work computer, not to mention probably heavier than its volume in pure gold.

On a hint from a friend I checked out Apple’s online store and saw they had a nice Macbook Pro configuration. I went to check it out in the retail store close to my office and they had that exact specification in stock, so, in what must have been the highest rated expense/time-to-think ratio of any decision I ever took, well, I bought it…

The spec, some bragging rights:

• Macbook Pro 17″
• Core Duo T9500 2.6GHz processor
• nVidia 8600M GT 512MB graphics card
• 200GB 7200rpm drive
• Kingston 4GB DDR2 667MHz RAM
• Hi Resolution 17″ 1920×1200 glossy screen

It’s a very nice machine, Apple knows how to make nice hardware, there is no question there.
OSX has some cool features, some of them still a bit foreign to me and some minor annoyances are creeping up, like Thunderbird’s not picking up my system date and time settings and displaying the date in the wrong format (a pet peeve of me), probably not Apple’s fault but annoying nonetheless.
So far so good and while I don’t mind using OSX for my browsing, email and creative stuff, that machine is meant to be running Windows Server 2008 x64 as a development platform.

Why Windows Server 2008 x64?

Well, it has some excellent features, a smaller footprint than Vista, all the aero eye candy, is apparently noticeably faster than Vista and has none of the nagging security prompt (you are considered administrator though, so keeping safe is entirely up to you).
The 64 bit version can also address the full 4GB of RAM without limitation and all server features are optionally installable.
By default, the installation is actually pretty minimal and you have to set services and options to get Windows configured as a proper workstation. It is after all, meant to be a server.
Oh, I almost forgot that there is also support for HyperV, although you must make sure you download the right version (if you list all available downloads in your MSDN subscription, you’ll see some that are explicitly without that technology).

Installing Windows Server 2008 x64 is remarkably easy.

• Get your hands on the ISO from your MSDN subscription or an install DVD from somewhere else (like a MS event, or even as a free 240 days download from Microsoft).
• You’ll need to repackage the ISO as it won’t work properly (something to do with non-standard file naming options).
  It’s fairly easy if you follow the instructions from Jowie’s website (cached version): you can get the ImgBurn software for free as well, which is a good find in itself. It should’t take more than 30 minutes to repackage the DVD.
• In OSX, go to Applications > Utilities > Boot camp and follow the instructions on screen.
  You will be able to resize the default partition by just moving the slider. I left 60GB for OSX and allocated the rest to Windows. The good thing is that OSX can read Windows partitions, so you can always store data there. Windows however, can’t read the HFS+ mac file system, although there are some third-party tools that can do it [1] [2] [3].
• Insert your repackaged DVD and Bootcamp will have rebooted the machine.
  After a few minutes of blank screen (and no HDD activity light to let you know something is happening), windows setup launches.
• You will be then prompted with the choice of partition to install to.
  Select the one named BOOTCAMP, then click the advanced options button and click format. From there one, windows will install everything, then reboot, then carry on installing, then reboot one last time.
• Now, insert your OSX recovery CD 1. It should automatically launch the driver installation.
  Once done, you’ll reboot to a nice, full-resolution windows prompt.
• All drivers will have been installed correctly except the one for Bluetooth. To easily solve that issue, just go to Spencer Harbar’s website and read how to install the Bluetooth drivers. Takes 5 minutes tops.

The final touches

A few notes to quickly get things running as expected.

• Get the most of your configuration by following the list of tweaks from Vijayshinva Karnure from Microsoft India.
• There are more tweaks, and even more tweaks available as well (don’t forget to enable Superfetch).
• Microsoft has a whole KB entry on enabling user experience.
• In the Control Panel > System > Advanced System Settings > Advanced > Settings > Advanced > Processor scheduling, set to Programs instead of Background services.
• Activate your copy of Windows using Control Panel > System.
  I was getting an error code 0x8007232B DNS name does not exist error. To force activation, just click on the Change Product Key button and re-enter the same key you used during install.
  Windows will activate straight away.
• When booting your Macbook, press the Option key and you will be presented a list of boot choices.
• You can check on Apple’s Bootcamp webpage other information about how to use the track pad, keyboard layouts etc,

References

• http://stuartd.blogspot.com/2008/04/windows-server-2008-boot-camp-and.html
• http://burnetts.homeserver.com/post/Windows-Server-2008-on-the-Mac.aspx
• http://csaborio.wordpress.com/2008/03/28/installing-windows-server-2008-on-a-mac-book-pro/
• http://exo-blog.blogspot.com/2008/03/windows-2008-vista-done-right.html
• http://www.sturmnet.org/blog/archives/2008/10/02/stuff-i-use-on-the-mac/ Lots of software recommendations for someone switching from Windows to the Mac.

The Linux server-based application would scan the vast (100,000s of CAD drawings) and, depending on their format, attempt to create thumbnails and extract textual information from them and populate a database so we could easily find related drawings through a simple web-interface.

The software would present thumbnails of the drawings; when the drawing was in DWG or DXF format, the thumbnail would be an SVG vector representation of the original drawing instead of an image. That made the preview much more useful as you could zoom in and still retain enough detail to ascertain if the drawings was actually what you were looking for.

Other file formats (TIFF, HPGL plots, etc) were transformed into PNG image previews that were saved into 2 size: a small thumbnail, good enough to be displayed in a list, and a larger one that would show more details.

The server software was written in Perl, with some of the converters in C. Every night, the server would go through the whole tree of drawings on the filesystem, looking for drawings it hadn’t seen before and it would pass them to the appropriate plug-in for extracting text -where possible- and create thumbnails.

Memories of fun projects…

Download links:

• cad2svg: http://etc.nkadesign.com/Download/Cad2svg
• cad2text: http://etc.nkadesign.com/Download/Cad2text

Checking for the existence of network paths

Checking for the existence of network paths (directories or files) is one of these issues that can freeze an application for 30 seconds or more if the folder is not accessible.

This is a type of problem that benefits greatly from running in a separate thread: it can take such a long time that the best way to check for these remote paths is to launch the verification for their existence outside of Access and somehow get the result back and cache it for the current session so we don’t have to suffer these delays again every time we check for that path’s existence.

One easy way to do achieve that goal is to create plain DOS batch files that execute hidden from view, create a result file when they complete their task and delete themselves automatically when they are finished.

How to use it

Download the sample database below then just add the FileUtilities, HashTable and MD5 modules to your project and you can use the code as such:

Dim status As AsyncDirectoryStatus status = FileUtilities.PathExistAsync("\\123.45.67.89\shared folder") The status variable will return either of the following values:

• AsyncDirectoryStatus.OK if the path was found.
• AsyncDirectoryStatus.NotFound if the path was not found (either because it doesn’t exist or you don’t have the rights to access it).
• AsyncDirectoryStatus.Checking if the verification is in progress and we haven’t received a definite answer yet.
  It’s up to you to decide how you want to handle that case. You could periodically check it, like I did in the example database, or you could disable the controls until you’re getting a confirmed result (by checking every time the user performs some action, like moving from record to record in a datasheet for instance).

You can call PathExistAsync as often as you want to check the status: it will not slow down your application (read the optional arguments section below though).
The result of the verification is cached, so querying the existence of the path is actually only done once; the result of subsequent queries for the same path is just instantly retrieved from memory.

Optional arguments

If you want to force the actual re-checking of a path without using the cached value, you can simply pass the ForceCheck optional parameter: Dim status As AsyncDirectoryStatus status = FileUtilities.PathExistAsync("\\123.45.67.89\shared folder", ForceCheck:=true) The first time you query for a path (or force it to be rechecked) there will be a short 150ms delay to give a chance to the function to return its result straight away (in case the path can be resolved quickly).
This may not be desirable if you’re checking a bunch of directories at a time. For instance, this is what I do when my application launches: ' Check a bunch of paths in parallel PathExistAsync strPathToQualityDocuments, NoDelay:=true PathExistAsync strPathToFinancialDocuments, NoDelay:=true PathExistAsync strPathToShippingDocuments, NoDelay:=true PathExistAsync strPathToPurchasingDocuments, NoDelay:=true By querying the existence of all these paths as soon as my application launches, I am starting the verification process without introducing delays in the application itself: each verification will start in its own process, in parallel to the main application.
Later in the application, when I need to actually use these paths, their result is likely to be known.

How it works

The FileUtilities module contains the main code.
In it, the PathExistAsync function works in slightly different ways depending on whether it’s the first time it is being called for a particular path or not.

The first time
The first time the function is called for a given path, we create in the user’s temporary folder the small batch file whose name is simply a MD5 hash (see below) of the path with .bat appended to it.
This batch file simply checks for the existence of the path and will create a small file (whose name is the MD5 hash of the path) with either 0 or 1 in it depending on the result of the verification.
We initially cache the status of the verification for the Path into the AsyncDirectories hashtable (see below) as Checking.

Example of batch file automatically created to verify a path: IF NOT EXIST "\\123.56.78.9\going nowhere" GOTO NOTEXIST echo 1 > "C:\DOCUME~1\Renaud\LOCALS~1\Temp\463C7367D8329BD6209A65A70A7DA08C" GOTO END :NOTEXIST echo 0 > "C:\DOCUME~1\Renaud\LOCALS~1\Temp\463C7367D8329BD6209A65A70A7DA08C" :END DEL %0 The Batch file name is 463C7367D8329BD6209A65A70A7DA08C.bat where the long number is actually the MD5 hash of the path we’re checking \\123.56.78.9\going nowhere.

Getting back the result
Whenever the PathExistAsync function is called, we check the currently cached result from the AsyncDirectories hastable.
If it is still Checking then we try to verify if we the result file has been created from the running batch. If not, we just return the same status, if yes, we read the result from the file, save it in the hashtable and delete the result file.

Useful libraries

The code makes use of 2 extremely useful libraries that I end up using quite often:

• a HashTable implementation.
  It makes it easy to create hashtable objects (otherwise known as Associative Arrays) to store and retrieve key/value pairs quickly.
  Hashtables are often used to cache data and can be thought of arrays where the index is a string value instead of an number.
  Here I use a hashtable to keep track of the paths we’ve checked and their result.

• a MD5 hash implementation.
  MD5 is a way to get a somewhat unique fixed-length value from a chunk of data.
  It’s a mathematical function that guarantees that a small change in input (say a single bit in the input data) has a large effect on the output value (a totally different number will be generated) and that you can’t reverse the function (you can’t obtain the input just by looking at the output).
  It is often used in security applications to transform sensitive data like passwords into unique values that can be (somewhat) safely stored because you can’t easily reverse a md5.
  Well, MD5 are not that secure any longer but here we just use their ability to transform our path into a unique number that we can easily use as a filename and a key for our hash to retrieve the current status of the path being checked.

Sample database

Download the PathExistAsync01.zip (67KB) containing the Access 2007 ACCDB database.

Download the PathExistAsync02b.zip (121KB) containing the MDB database¹ (untested as I only have Access 2007).

License

Please refer to the source code in the database for the exact licensing terms.
Note that the license only refers to code by me. When code from other sources is used, you will have to conform to their own licensing terms.

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

References

• Hashtable implementation in VB
• Cryptography Software Code (for the MD5 hash implementation in VB)

Here is the absolute proof:

He calls himself Erik Meijer now and apparently solved his skin issue but you only need to listen to them to know it’s the same person.

Yes, they are both Dutch, which in itself is already proof enough of their evilness and further confirms they are one and the same.

There is some talk that Microsoft has been the refuge of all that is Bad, and some even say that Steve Ballmer may be none other than Dr Evil himself…

That being said, you should check out videos on Channel 9 where Erik Meijer appears. Despite his obvious evilness, he is actually one of the very bright and interesting characters at Microsoft and always has fascinating things to say about programming languages:

• Expert to Expert: Erik Meijer and Bertrand Meyer -- Objects, Contracts, Concurrency, Sleeping Barbers
• Tier Splitting Volta: Who. What. How. Why -- Part 1 and Part 2
• Erik Meijer: Functional Programming
• Erik Meijer, Gilad Bracha, Mads Torgersen: Perspectives on Programming Language Design and Evolution
• Paul Vick and Erik Meijer -- Dynamic Programming in Visual Basic
• Erik Meijer: Democratizing the Cloud
• Anders Hejlsberg, Herb Sutter, Erik Meijer, Brian Beckman: Software Composability and the Future of the CLR
• JAOO 2007: Joe Armstrong -- On Erlang, OO, Concurrency, Shared State and the Future, Part 1 and Part 2
• CLR Team Tour -- The Future of Languages (PDC panel preview) -- Part 1 and Part 2

RAID configurations explained

Doing this, I was able to remotely connect to the working server and access the disk, mount it and access the data.

Salvaging the data

Once one of the RAID1 drives was placed into the USB enclosure and connected to the other available Linux box it was easy to just remount the drives:

fdisk will tell us which partitions are interesting, assuming that /dev/sdc is our usb harddrive:

[root@fax ~]# fdisk -l /dev/sdc

Disk /dev/sdc: 81.9 GB, 81964302336 bytes
16 heads, 63 sectors/track, 158816 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes  

Device      Boot    Start         End      Blocks   Id  System
/dev/sdc1   *           1         207      104296+  fd  Linux raid autodetect
/dev/sdc2             208       20526    10240776   fd  Linux raid autodetect
/dev/sdc3           20527       22615     1052856   fd  Linux raid autodetect
/dev/sdc4           22616      158816    68645304    f  W95 Ext'd (LBA)
/dev/sdc5           22616      158816    68645272+  fd  Linux raid autodetect

We can’t simply mount the partitions, they need to be assembled into a RAID partition first:

[root@fax ~]# mdadm --assemble /dev/md6 /dev/sdc1 --run
mdadm: /dev/md6 has been started with 1 drive (out of 2).

The --run argument forces the RAID partition to be assembled, otherwise, mdadm will complain that there is only a single drive available instead of the 2 -or more- it would expect.

Now simply mount the assembled partition to make it accessible in /mnt for instance:

[root@fax ~]# mount /dev/md6 /mnt

We can now salvage our data by repeating this process for every partition.
Using RAID1 means you have at least 2 disks to choose from, so if one is damaged beyond repair, you may be lucky and the mirror one on the other drive should work.

If the drives are not physically damaged but they won’t boot, you can always use a pair (or more) of USB HDD enclosures and reconstruct the RAID arrays manually from another Linux box.

Planning for disasters

The lesson here is about planning: you can’t foresee every possible event and have contingencies for each one of them, either because of complexity or cost, but you can easily make your life much easier by planning ahead a little bit.

Most small businesses cannot afford dedicated IT staff, so they will usually end-up having the least IT-phobic person become their ‘system administrator’.
It’s your job as a consultant/technical support to ensure that they have the minimum tools at hand to perform emergency recovery, especially if you cannot intervene yourself quickly.

On-Site emergency tools

In every small business spare parts closet there should be at least:

• Whenever possible, a spare Linux box, even if it’s just using older salvaged components (like a decommissioned PC). Just have a generic Linux install on it and make sure it is configured so it can be plugged in and accessed from the network.
• a spare US$50 router, preferably pre-configured to be a temporary drop-in replacement for the existing router/firewall. Ideally, configure it to forward port 22 (SSH) to the spare Linux box so you can easily access the spare box from outside.
• USB external hard-drive enclosure.
• a spare PC power supply.
• some network cables, a couple of screwdrivers.

There are many more tools, such as rescue-CDs (like bootable Linux distributions), spare HDD, etc that can be kept but you have to remember that your point of contact need to be able to be your eyes and hands, so the amount of tools you provide should match their technical abilities.
Don’t forget to clearly label confusing things like network ports (LAN, WAN) on routers, cables and PCs.

The point is that if you can’t be on site within a short period of time, then having these cheap tools and accessories already on site mean that your customers can quickly recover just by following your instructions on the phone.
Once everything is plugged-in, you should be able to remotely carry-out most repairs.

Resources

• Linux software RAID1 setup
• Fedora System Recovery Week

What’s wrong with the default MsgBox

The default message box in Access is sometimes useful to warn, inform or ask confirmation from the user.

It has, however, a few drawbacks:

• It is bland: the standard message box does not even follow the currently selected Office 2007 scheme.
• The amount of text it can display is limited: if you try to display too much text it will be truncated.
• You can’t copy or save the content of the message.
• Because popup boxes are viewed as intrusive, people tend not to read them and end-up closing message boxes before they realize they may have contained useful information.
• They only displays plain text: you cannot format the message to draw attention to the key points.
• They are blocking, meaning that nothing can happen in the main application while the box is displayed (it can’t even shut down).

Sometimes you need to display an important message or require users to make take a decision.
Message boxes are not to be abused but they serve a useful purpose.

An enhanced message box

Rather than using the bland standard message box you can now have something a bit more customized.

Plain Text version of the enhanced custom message box under the Office Blue Colour Scheme:

RichText version of the enhanced custom message box under the Office Black Colour Scheme:

Here are the features of the enhanced message box:

• It is entirely compatible with the standard one: just change MsgBox to Box using find and replace should be enough (see tip below to avoid getting strange errors).
• It allows the user to simply click on a button to copy the content of the message to the clipboard or save it to a text file to a configurable default location.
• It looks and feels like it belongs to the main application, following its colour scheme.
• It attempts to prevent users from blindly closing the modal box reading the message: buttons will first be inactive for a configurable amount of time. It’s not a perfect solution, but it is quite effective.
• There is a RichBox version that can display rich HTML content, not just plain text, so important parts of the message can be formatted in a useful way.
• It is able to display large amount of data. While it’s not something you usually want, it may be useful for the message box to display more text in some situations (log or tracing information, legal documentation, etc).
• Rather than sprinkling your code with “& vbCrLf & _” uglies, you can embed newlines in the text itself by using C-style “\n” escape sequences that will automatically be transformed into the appropriate newlines. Makes for clearer code and less typing.
• Because you get the source, you can easily customise the message box with new icons and colours to better match your overall application’s personality.
• It is non-blocking: if your application forces users to log-off after a certain amount of inactivity, the enhanced message box will just close rather than prevent Access from shutting down like the standard MsgBox does. Of course, it’s up to you to decide how to handle that gracefully, if at all.
• It properly displays the expected button captions based on the language of the operating system, so it behaves very much like the default MsgBox (for instance, it will properly display “Cancel” on English systems and “Annuler” on French ones).
• It also properly plays the system sounds associated with the type of message. You can also enable or disable the sound effect as needed.

How to use it

Download the demo below and copy (drag & drop) the following into your application:

• the FormDialog form,
• the API_GetTextMetrics module,
• the Dialog module.

If you rename the FormDialog, make sure you replace any occurrence to it in the code, in particular in the Dialog module.

Since the enhanced message box is just a replacement for the standard one, you just use it like you would use the MsgBox.
'----------------------------------------------------------------------------- ' Simple use of the Plaintext box ' Note the use of n that will be converted into a newline Dialog.Box "This is a plaintext message.\nClick OK to dismiss", vbOKOnly+vbinformation, _ "Message Title" '----------------------------------------------------------------------------- ' Getting the result back Dim dr as vbMsgBoxresult dr = Dialog.Box("Are you sure you want to delete?", _ vbYesNoCancel+vbQuestion, "Confirm action") if (dr = vbYes) then DeleteRecords '----------------------------------------------------------------------------- ' Using named parameters Dialog.Box Prompt:="All your bases are belong to us", _ Buttons:=(vbOkOnly+vbCritical), _ Title:="Bad error" '----------------------------------------------------------------------------- ' Using the RichBox to display simple HTML ' The first line will be bold, then the word 'button' will be printed in red ' Here the n will be escaped to '<br/>' tags to simulate newlines. Dialog.RichBox "<strong>This is a bold message</strong>.\n" & _ "Click the <font color=""#FF0000"">button</font> to dismiss.", vbOKOnly+vbinformation, _ "RichText Message Title" There are a few additional settings that can be used to change the behaviour of the enhanced message boxes.
One is that you can adjust the delay before the buttons become activated.
'----------------------------------------------------------------------------- ' Use the ButtonDelay to specify the time in seconds before the buttons become activated ' The default is 2s. Use 0 to activate the buttons immediately. Dialog.Box Prompt:="All your bases are belong to us", _ Buttons:=(vbOkOnly+vbCritical), _ Title:="Bad error", ButtonDelay:=1 '----------------------------------------------------------------------------- ' Change the default delay value. ' To disable the activation delay Dialog.DefaultButtonDelay = 0 ' To make the user wait 3 seconds before they can press any button Dialog.DefaultButtonDelay = 3 Another one is that you can enable or disable whether beeps should be played or not.
'----------------------------------------------------------------------------- ' Use AllowBeep to specify whether beeps should be played when the message box opens ' By default, they are. Dialog.Box Prompt:="All your bases are belong to us", _ Buttons:=(vbOkOnly+vbCritical), _ Title:="Bad error", AllowBeep:=false '----------------------------------------------------------------------------- ' Change the default behaviour. This is true by default. Dialog.DefaultAllowBeep = false The last settings is the folder where we should save the content of the message when the user clicks the Save button on the message box.
'----------------------------------------------------------------------------- ' Change the save folder. ' By default, the text messages will be saved in the same directory as the database. ' Here we want them to be saved to a temp directory Dialog.DefaultSavedTextFileFolder= "C:temp" These few settings make the enhanced message box more customizable.

Large text

The standard MsgBox cannot display much text. On the other hand, there is no real limitation to the amount of text the Box and RichBox can display.
When the amount of information is too much to fit the maximum allowed size for the message box the text will overflow and can be scrolled up/down as necessary.

Limitations of the RichBox

The RichBox version relies on the normal TextBox control’s ability under Access 2007 to display RichText wich is nothing more than lightweight HTML.
Because font size may be varying a lot in the message, it becomes very difficult to accurately predict the size of the box needed to display the whole message.
Short of implementing a complete HTML engine, we have to rely on some assumptions to display HTML.
The risk is that sometimes the content may not properly fit the TextBox control in some circumstances.
If you use the RichBox, thoroughly try displaying your messages and tweak the HTML as necessary to include additional lines or non-breaking spaces to ensure that the result looks good.
If you don’t overuse font size and don’t display in multiple fonts the RichBox should do the right thing most of the time.
Don’t overuse the RichBox to display colourful messages. There is a fine line between being informative and tasteless. Keep colours and formatting where it is useful.
I think that in most cases, the plain text version Box is more than enough.

Replacing MsgBox in existing code

As I said above, replacing the standard MsgBox is easy but you need to make sure your search and replace parameters are configured correctly:

If you’re getting strange compile errors, it may be because you forgot to tick the Find Whole Word Only and some of the strings containing the letter sequence “msgbox” were replaced in the process.

If that’s the case, you can revert the damage by simply doing a search and replace across the whole project on:
- VbboxStyle or VbDialog.BoxStyle to be replaced with VbMsgBoxStyle
- VbboxResult or VbDialog.BoxResultto be replaced with VbMsgBoxResult

How it works

The code makes extensive use of Win32 API calls.
Most of the hard work is done in the FomDialog class form. There is too much there to really go into the details but you are welcome to have a look at the commented code.
The code relies also on a utility function from Stephen Lebans used to calculate the size of of text. I have made some minor modification to that code so I would refer you to his original implementation if you are interested in calculating TextBox sizes for forms or reports.

In the code for the FormDialog, I re-implement some of the expected functionalities of the MsgBox: proper arrangement of the buttons, displaying of the appropriate icon, etc.
Once this is done, we calculate the size of the textbox needed to display the whole of the message.
In the case of RichText, we first use Application.PlainText() to convert the HTML into properly formatted plain text. We then calculate the Textbox size using a slightly larger font than needed as a way to ensure that the content of the RichText message will fit the box in most cases.
Once we know the size of the TextBox, we can easily resize the form to properly display the TextBox.
If there is too much text, we resize the form to its maximum permissible (70% or screen width and 90% of screen height) and change some of the visual cues to let the user know the text is overflowing.

One thing of note is the way the form is kept modal.
Rather than using DoCmd.OpenForm and DoCmd.Close I use the form as a class and create an instance manually (see the code in Dialog.Box and Dialog.Richbox). I keep this instance alive until I got the form’s result back.
If you are interested in knowing how the form is made modal, this is the code in FormDialog.ShowModal() what keeps the form open until the user clicks a button:
Public Function ShowModal() As VbMsgBoxResult ... ' Here we reset the result for the clicked button such as vbOK, vbYes, etc ' This is set in each Button's Click event m_Result = -1 ' Wait for the user to click a button Do While (m_Result = -1) DoEvents Sleep 50 Loop ShowModal = m_Result End Function The Sleep() function is a Win32 API that stops the current process for the given number of milliseconds. This in effects hands back the control to the Operating System for a short time. That way the system is still responsive and does not consume resources when it’s just waiting for user input.

Sample database

You can download a sample database containing all the necessary code as well as a number of tests.
This version only contains the database in Microsoft Access 2007 accdb format.

Download the EnhancedMsgBox01.zip (116KB), v1.3 containing the ACCDB database.

Code Updates

v1.3: 17MAR2009
Thanks to Henry of Access-Pro.de for proposing a correction to the default buttons behaviour.

• Updated behaviour for the default buttons. They are now focused in a way that matches that of the standard msgbox.
• Reversed the naming of the buttons on the form to make it a bit more consistent with the standard box.

v1.2: 07SEP2008
Thanks to Andy Colonna (http://www.pcesoft.com) for uncovering the following bugs (check out his free Spell Checker with source code!):

• Corrected bug in Form_FormDialog.FilenameSanitize() function that would fail to remove all invalid characters for a file name.
• File name for the saved text message will be truncated to first 32 characters of message box title in Form_FormDialog.MakeFriendlyFileName().
• Changed the use of FollowHyperlink to ShellExecute to avoid security warning in some instances in Form_FormDialog.btCopyToFile_Click()
• Corrected twips to pixel conversion bug in API_GetTextMetrics.fTextWidthOrHeight() that would result in an improperly sized dialog box when the text message was too wide.

v1.1: 08AUG2008

• Corrected code for DefaultButtonDelay (thanks to Geoffrey) (was referencing wrong variable, causing self-referencing code).
• Corrected code for Box and RichBox to take the DefaultSavedTextFileFolder into account (the path was previously not passed onto the dialog boxes and the text file would always be created in the application folder instead of the one specified by DefaultSavedTextFileFolder)
• Added license notice at top of source code.

v1.0: 20MAY2008

• Original version

Resources

• Dissecting the MessageBox article on CodeProject
• XMessageBox -- A reverse-engineered MessageBox() article on CodeProject
• TextWidth-Height code demo from Stephen Lebans (great resource, check it out!).
• Pixel to Twips conversion from MSDN.
• Copy Text to Clipboard from the excellent site The Access Web.
• Getting Resource Strings and more from DLLs.

This work is licensed under a Creative Commons Attribution 3.0 Unported License.
Free for re-use in any application or tutorial providing clear credit is made about the origin of the code and a link to this site is prominently displayed where end-users can easily access it.

Being able to cleanly restart and compact the application is also useful in other instances:

• Changes made to the environment
• Recovering from errors (for instance after a network disconnection)
• Forcing the user to re-log cleanly into the application
• Automatically restarting a long-running application (for instance so that it may automatically compact on close and restart afresh with or without user intervention).

The problem is that you cannot -to the best of my knowledge- close and open again the same database from within MS Access itself.
Most executables cannot do that and the way to solve the issue is usually to pass the control to another boostrap programme, close the main application and let the bootstrap programme re-open the main application again.
I wanted a simple and clean way of using it. One that would not require shipping external dependencies.

How to use it

Download the sample database below, copy the Utilities module or just the Restart sub defined in it into your own application.

To use it, just call the Restart sub and the application will close and re-open.
If you supply the optional Compact:=true parameter, the database will also be compacted during the restart process.
This will work for normal databases (mdb/accdb) and also compiled (mde/accde) and runtime (accdr) databases as well.

Important note

If you want to use this code do not enable the Compact on Close option in Access for your database as the code doesn’t pick that up yet.
Instead, you can either simply call restart Compact:=true on user action (for instance from a menu) or on other triggers, for instance when the database is being open and hasn’t been compacted for more than a week.

How it works

If you’re curious about the technical details, here is how it was put together.
The main idea is that the MS Access database application has to be self-sufficient and restart itself by performing these steps:

• create a small batch file
• run the batch file, passing the path and extension of our database
• close the main application
• the running batch file would wait for the MS Access lock file to be removed
• once the lock file disappears, we open the database after compacting it if required.

The key point here is that the batch file cannot just reopen the database right away: if the application is big or if it’s compacting on close for instance, it may take several seconds to actually close.
The only moment we can be pretty sure that the database is effectively closed is when the lock file is deleted by MS Access.

The batch file is hard-wired in the Restart sub that does all the work:
SETLOCAL ENABLEDELAYEDEXPANSION SET /a counter=0 :CHECKLOCKFILE ping 0.0.0.255 -n 1 -w 100 > nul SET /a counter+=1 IF "!counter!"=="60" GOTO CLEANUP IF EXIST "%~f2.%4" GOTO CHECKLOCKFILE "%~f1" "%~f2.%3" /compact start " " "%~f2.%3" :CLEANUP del %0 When the application runs the batch file, it passes 4 arguments:

• the full path to the MSAccess.exe executable (used for compacting the database)
• the full path to the database without the extension
• the database file extension without the leading “.”
• the appropriate database lock file extension (laccdb or ldb).

This allows us to easily construct the path to either the database or the lock file at line 07 and 09.
Line 08 is actually only inserted if we need to compact the database: it simply launches MSAccess.exe with the /compact command line switch.

The funny use of PING is actually a simple way to wait for some time before we check if the lock file is still there or not. There is no SLEEP or WAIT function provided by default in Windows so we have to be a bit creative and use the time-out option of the PING command trying to ping an nonexistent, but valid, IP address.
Once the lock file has disappeared, we open the database at line 09 and then delete the batch file itself so we leave no leftovers.

The other thing of note is that we now use a counter to keep track of the number of times we checked the existence of the lock file.
Once this counter reaches a pre-determined amount (60 by default, ~ 45 seconds) we consider that there is a problem and the database application didn’t close, so we just exit and delete the batch file.

Download the DatabaseRestart.zip (48KB) containing both an Access 2007 ACCDB and Access 2000 MDB test databases.

Other implementations

• Roger’s Access Library (MVP) has a different implementation on offer.

Code Updates

v1.2: 13FEB2009

• Added optional parameter to compact the database during restart.

v1.1: 09AUG2008

• Now a separate test database (used to be bundled with the Colour Scheme sample).
• Added support for older Access versions (an Access2000 MDB is now included).
• Corrected wrong lock file extension for accd* files.
• Added a time-out feature after which the batch file will delete itself after a while if the Access lock file wasn’t released (for instance following a crash).
• Added checks to delete the batch file if it has not deleted itself for some reason (for instance after a reboot).
• The batch file now has a unique name based on the name of the database, allowing multiple databases to be restarted from the same directory.
• Added license notice at top of source code.
• Updated the article to reflect the changes.

v1.0: 06MAY2008

• Original version

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Luckily for us, Office 2007 stores the global colour scheme setting in the registry under:
HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Common\Theme
and Office 2010 in:
HKEY_CURRENT_USER\Software\Microsoft\Office\14.0\Common\Theme

The values being stored under that key are:

1. Blue scheme
2. Silver scheme
3. Black scheme

With this information, we can easily both read and set the colour scheme. The only caveat is that I could not find a way to notify Access to reload the setting automatically once it is changed, so users will have to restart the application before the change becomes active. A small price to pay but if anyone has a better idea, please let me know.

To write the new value to the registry I use a set of WIN 32 APIs that are more flexible than the default ones provided in VBA.

You can download the sample database as it contains all necessary files, including the definition for the Win32 API functions.

Download the ColorSchemeV1.3.zip (31KB) containing the ACCDB database.

The sample also contains some code to restart the database. This is the subject of another post: Restarting and compacting the database programmatically.

Improvements/uses:

• Find a way for Access to reload the settings without having to restart the application.
• Use the knowledge about the current scheme to change the other colour settings in the application (or even adapt the form’s theme).

Updates:

• 31MAY2011: added support for Office 2010.

References:

• Setting the BackColor to match the Office 2007 color scheme
• API Get / Set Registry functions from http://www.arcatapet.net/.

The trick is to use some WIN 32 API calls to modify the transparency of a standard MS Access form made to cover the screen.

The effect is quite neat and I thought I would try it and make a sample database for others to tinker with it.
My version allows you to chose between covering the whole screen or just the main Access window and it will test if it’s running under a Remote Desktop Terminal and disable the layer in that case.

Update 07MAY2008

Following Rob’s improvements I made another sample database that incorporates his code with a few improvements:

• I added the LightBoxForm.LayerToFullScreen property so users can choose explicitly how they want the layer to be shown.
• I moved the code to hide the layer into a Hide() sub so you can just show/hide the layer using LightboxForm.Show and LightboxForm.Hide.
• I changed the Form’s Resize event code in the LightBoxForm class to avoid flickering: resizing the form within its Resize event actually trigger the Resize event again a second time which causes flickering.
  I simply modified the code to make the form totally transparent (opacity of 0) the first time the event is fired and assign it the expected opacity when the event handler in re-entered.

'----------------------------------------------------------------------------- ' Handle the Layer Form Resize event '----------------------------------------------------------------------------- Private Sub m_objForm_Resize() Static busyResizing As Boolean Dim lngStyle As Long Dim r As RECT ' disable screen updates m_objForm.Painting = False ' When the form opens initially, we make it totally transparent to avoid flickering lngStyle = GetWindowLong(m_objForm.hWnd, GWL_EXSTYLE) SetWindowLong m_objForm.hWnd, GWL_EXSTYLE, lngStyle Or WS_EX_LAYERED SetLayeredWindowAttributes m_objForm.hWnd, 0, 0, LWA_ALPHA ' If the Access window is maximized, then maximize the lightbox form. ' If the Access window is not maximized, then ' position the lightbox form so that it covers the Access window If IsZoomed(hWndAccessApp()) Or m_bLayerToFullScreen Then DoCmd.Maximize Else GetWindowRect Application.hWndAccessApp(), r MoveWindow m_objForm.hWnd, r.x1, r.y1, (r.x2 - r.x1), (r.y2 - r.y1), True End If If busyResizing Then ' Get the current window style, then set transparency lngStyle = GetWindowLong(m_objForm.hWnd, GWL_EXSTYLE) SetWindowLong m_objForm.hWnd, GWL_EXSTYLE, lngStyle Or WS_EX_LAYERED SetLayeredWindowAttributes m_objForm.hWnd, 0, (m_sngOpacity * 255), LWA_ALPHA ' enable screen updates m_objForm.Painting = True ' Back to normal busyResizing = False Else busyResizing = True End If End Sub

Samples

There are now 2 sample databases. Ech zip contains a Microsoft Access 2007 ACCDB file and its conversion to Access 2000¹ and Access 2002-2003 MDB but please note that I have not been been able to test those in older version of Access and that form transparency doesn’t work in Operating Systems older than Windows 2000.

Download TransparentLayer02b.zip (138KB), recommended version
(improved, more flexible version, based on Rob’s updated article).

Download TransparentLayer01b.zip (122KB), original version
(simple code, based Rob’s original article).

Troubleshooting

• If you are getting security warnings: make sure that you open the database from a Trusted Location or you will receive a security prompt. If you don’t know how to do that, check these steps.
• If the layer appears on top of the login form instead of behind: make sure that the top-most form has ist Modal properties set to Yes and the frmLightBox form has its modal property set to No.
  If you improve on it, please let me know and I’ll post it here for all to find.

The basic premise of the article is that Apple will attack the corporate market through the back-door, using the iPhone and its forthcoming development platform.
The author contend that development to the iPhone will drive interest in the Mac and allow Apple to displace Windows by offering more business-related applications that would eventually not require Microsoft’s OS running side-by-side in your Mac, freeing you of the dominance of the evil empire.
Please read the article first. Done?
Hmmm…
Ok, let’s see…

First, let’s be nice: if Apple want to eat at Microsoft’s dominance of the OS market, then all the better: more competition will drive innovation and choice, which is always nice.
There is certainly some truth to the idea that Apple could offer some useful tools and technologies to the corporate world.

The problem is that author of the article was a bit too much of a Mc Fanboy(TM) to make his arguments compelling.

Let’s start direct quotes from the article.

Speaking of the ability of the Mac to quickly switch from a Windows application to a Mac by using a keyboard shortcut:

“Windows users, in the very near future, will be free to switch to Apple computers and mobile devices, drawn by a widening array of Mac software, without suffering the pain of giving up critical Windows-based applications right away.”

Useful as it may be, it’s forgetting one thing: if you need to be able to run your windows application under a Mac, you will need to give Microsoft money for the Windows license, making the newly found convenience of running the two OS together seamlessly a fairly expensive one, both in terms of computer resources and money.
While it will interest a lot of people and they may use their windows apps on Apple hardware is this really compelling enough to make it a viable migration path?

About the new Mac OS kernel:

“That kernel has proved easily adaptable across the entire Apple product line, from highly complex servers all the way down to the relatively simple iPod Touch. Such modularity allows Apple to add whatever functions are necessary for each product environment—all while maintaining cross-product compatibility.
By contrast, Microsoft has held on to an OS tethered to the 1980s, piling additions upon additions with each upgrade to Windows. With last year’s arrival of Vista, Windows has swollen to 1 billion bytes (a gigabyte) or more of software code. The “Mach” kernel of the Mac OS X, however, requires less than 1 million bytes (a megabyte) of data in its smallest configuration, expanding modestly with the sophistication of the application.”

All this is so silly and skewed that I wonder where to start.
First, kernel size is irrelevant: a basic kernel functionality does not make an operating system. Additionally, I doubt that we’re comparing the same functionality and for desktop or server applications this is utterly irrelevant.
For small devices Windows has its own flavours and they can quickly be deployed to almost any hardware with minimal effort.
Applications built for Windows built for mobile devices can be ported to Windows without too much trouble, and vice versa. Whatever platform you’re developing on, applications for small hardware are always an exercise in compromise: there is no way that you can just recompile an app for a different target and have it just work beautifully in any device. A mobile app must take care of power requirements, limited screen estate, memory and CPU limitations and specific usability issues like the presence of a touch-screen instead of a keyboard and mouse.
The point is that you have to take the platform into consideration when crafting software. Besides, using .Net makes it fairly easy to develop applications for any flavour of Windows. Additionally, whether the OS under the XBox is different or not from the desktop version of Windows is also irrelevant: you can develop games under the XNA platform that compile and work just the same under Windows as they do on the Xbox.

Now, for the argument regarding the “additions upon additions” made upon Windows that contribute to its bloat, it’s an easy shot to make and an unfair one: Apple has had no regards for its legacy applications and since its primary market was consumers, it could get away with making each of its major OS release incompatible with the previous one.
We can argue whether it was a good thing for Microsoft to keep all the quirks from previous releases of its operating system alive but we must not forget that in the corporate world, Windows has close to 100% market share on the desktop.
What that means is that Microsoft could not afford not supporting business applications across versions. it would have been suicidal to even envisage dropping compatibility.
If Apple ever get into that corporate market above a few percent it will have to guarantee compatibility between versions of its OS and programming tools if it wants to have any chance at being taken seriously at all.
When companies spend money developing an application that is critical to their business, they don’t want to hear about its supporting OS becoming incompatible every couple of years.
Apple would have to support and maintain compatibility for at least 10 years for every major OS version. So far Apple hasn’t shown that it was capable of that type of commitment.

A small digression.
A couple of weeks ago I found some old files of mine on a floppy.
They were Ami Pro files, made at a time when Ami Pro 3.1 was the best word processor around.
Problem was that I could not safely open these files any longer: filters for various word processors would mangle the complex layout of the pages, making them un-usable. Out of curiosity, I found a full version of the original Amipro 3.1.
I had no expectations of being able to run that setup package. After all, Ami Pro 3.1 came out in 1994, before Windows 95 was even released!
Well, I clicked on that setup.exe file and watched the installation process go through… Everything went fine.
Surely, I thought, there is no way this is going to run under Windows XP. It’s going to crash for sure.
I double-clicked on the 16 colour icon and lo and behold, the whole thing actually ran! Flawlessly!
I was able to open my old files without any issue, save them under another format. Everything worked, miraculously.
That application was written at a time the Internet didn’t even exist yet I was able to install and run it without problem on a current operating system.
Back to our regular schedule.

“Despite Apple’s relative scarcity on corporate desktops, Mac laptops are already well accepted within the enterprise, with a market share of more than 20% and growing. For business travellers, the new MacBook Air, some three pounds lighter than comparable Windows-based laptops, already offers one huge advantage.”

First, Apple has a almost 20% market share on overall laptops sales only, not corporate sales.
Second, that figure is for the US only. Apple’s laptop market share in the world is certainly not bad, but it’s a quarter of that figure, making adoption of Apple laptops outside the US very small indeed.
Third, while Apple laptops are certainly sexy, they bring their own issues to businesses: lack of in-house serviceable parts, issues with making them fit into a complex infrastructure, hardware and software compatibility problems.
Bringing Macs into your organisation can be painful. Of course, it’s never impossible, but beyond simple setups, it takes time, energy and money to make stuff work seamlessly.
So I contend that, unless you have a serious commitment to Apple, most of the Macs getting into companies are for simple usage: fetching emails, browsing, making presentations and maybe Photoshop, although it seems that the application that was once the sole reason for some to use macs is now going to switch side, at least for a little while.
For Macs to become a first-class corporate platform would require that Apple makes a very serious commitment to cross-platform development to make applications work on Windows as well.
I seriously don’t think that the ability of Macs to run Windows side-by-side will do much: as mentioned above, the expense incurred offers no real benefit to businesses and having to support both platforms would increase the cost of ownership because of the upfront cost of buying and installing multiple OS, 15% higher average cost of Apple laptops compared to other similarly specified laptops and the cost of maintaining and supporting multiple hardware and OS stacks.
Slickness can only get you so far I suppose.

“While Mac desktops offer a growing number of superior features over Windows desktops”

Seriously dude, what superior features?

“Apple’s recently introduced Leopard servers compete in a market of unhappy Vista server buyers where Microsoft’s market share is only 40%. Leopard has a decent chance to expand from its small beachhead.”

Huh? Vista servers?
WTF is that?
You can’t be talking about Windows 2003 because it’s actually a rock-solid platform and there isn’t much to complain about.
Windows 2008 is just coming out and it looks just as promising.
Apple has some seriously nice server hardware, it’s so beautiful to look at all this engineering that it gets me all hot and bothered.
Seriously though, on the server side, Leopard would not be in competition with Windows but with *nix.
Server-side applications like ASP, Exchange, Sharepoint, SQL Server and the myriad other Microsoft-only server software will only work on Windows.
Apple is not going to compete as a platform for hosting these which means it will host services traditionally found on Linux/Unix/BSD systems.
In that *nix camp Apple would certainly be a good contender, although it would have to prove that it can compete with the low license and hardware cost of its competitors and offer more.

One more thing to remember: Apple is in the hardware business: it uses software to lock people into its hardware business.
I’m all for competition but at least using Linux or Microsoft products doesn’t lock me into a single hardware/software platform pair: I have orders of magnitude more choices when it comes to my servers, desktops and laptops than what Apple can offer me.
Apple products may be slick and beautifully engineered but they certainly don’t offer more freedom: chose an iPhone and get stuck with a single network provider, buy an iPod and get stuck with iTune, buy a mac and get stuck with Apple, limited software offerings, no serious games and limited hardware support.

One other thing to keep in mind is that Microsoft has acquired a huge weight in terms of software development power: new technologies have been pouring out of Redmond at a pace that is impossible to follow.
The number of developers and companies able to develop software for Windows platforms is wayyy above what Apple can dream of at the moment (we’re talking about millions of developer making a living off Windows).
Apple will need to make serious efforts to woo the huge amounts of developers it needs to bring enough business and non-business applications to its platforms before it can become a serious competitor to Windows in the corporate world.
Whether Apple has the capacity and will to do that instead of remaining a consumer-oriented company remains to be seen.

That being said I wouldn’t mind developing software for Apple platforms. I would love to be able to use a generic “surface laptop”, a sort of larger iPhone that could be used by sales people for taking orders and showing off new products.

Anyway, the point of all this was simply to offer a modest reality-check to an article that should have been a bit more measured and balanced in its fanboy-ish enthusiasm.

Good graphic designers who work on User Interfaces are too few and far between, and large organisations will take time to get staff with a sufficient level of expertise to be useful in developing meaningful WPF/Silverlight applications.
Until now, the bulk of the development effort was done by developers. Most ISV only have developers onboard.

As far as I can tell, it seems that main reason behind this slow adoption is that WPF/Silverlight has only been noticed by developers and they don’t know what to make with it.
WPF/Silverlight are not really developer-friendly: the development infrastructure is there but we’re left with a blank canvas and nothing to drag and drop onto it.

There will be a great gap, probably lasting a few years, before there are enough experiments in these new user interfaces that some useful lowest common denominator can be exploited by non-designers.
We’ll have to wait for tool vendors to provide the bulk of user interface blocks and for a few new patterns to appear before all the designer-challenged developers like me can make decent interfaces without requiring a professional graphic designer in their midst.
In the mean time, we’ll probably have useless, but pretty, applications, some useful, but ugly ones and a lot of people scratching their head trying to couple the designers and the developers in a working relationship.

Here are some reasons why SQL Server Express may be a good choice:

You’re upsizing an Access database

SQL Server is the natural extension of upsizing an existing Access database. It work automagically with minimum effort providing that you followed some simple good-design rules from the start.

You’re future-proofing your needs

Because SQL Server comes in many flavours, you know you -or your customers- can upgrade to a more capable (albeit more expensive) version in the future if needed.

Very flexible

As usual with a lot of Microsoft development tools, SQL Server will happily let you shoot yourself in the foot by providing you with a fairly easy way to treat your database as a complete development platform.
It’s good in the sense that you have interesting tools and capabilities included in the server, and it’s bad for the exact same reasons.
I tend to prefer database servers to be just that: data repositories, and I’m not too fond of relying on specific, non-standard features of a particular database system, but what do I know.

Excellent out-of-the-box development support

Deep integration with .Net and Visual Studio, without any effort, Microsoft saw to that of course.
In some cases, such as LINQ to SQL, it’s almost the only real choice, although the other database vendors are working hard at the necessary providers, so that lead should be short-lived.
There is something to be said about developer productivity: you have to give credit to Microsoft for making their tools well integrated and usable from each-other. What it means is that for small developer shops there is much to gain in surrendering to this “ease of the default”.
Of course, it’s a double-edged sword, but having a complete development infrastructure work out of the box is certainly a big help, and if you don’t like it, you’re still free to chose something else.

Lots of tools

With SQL Server Advanced Services, you also get Server Management and Reporting Services. These are great tools made available for free.
The only missing one for SQL Server is the Reporting designer. While the reporting service means that you can use existing reports, only SQL Server Standard and Enterprise have it.
There is an option for developers though: the (nearly free) SQL Server Developer edition is in fact the same as SQL Server Enterprise, without the license to use in non-developer or tester environment. This means that as a developer, you can create and distribute your reports to be used by your customers who will be using SQL Server Express.

Did I mention it’s free?

All this is free, as in beer, not as in liberty though.
For commercial applications targeted at small businesses, SQL Server Express is a really good choice: you can distribute it without problem, the customer gets all the tools, can easily find outside support, and they can always migrate to a more beefy version if their needs grow, all that without having to depend on you.
So it sort of offers customers a kind of freedom that they wouldn’t have with other choices.

Of course you can get that with other database systems, although you have to be careful which Open Source one you choose: I recently decided not to use MySQL any longer for the simple reason that it’s too expensive and restrictive in a business environment, at least for the kind of work I do.

Why would you not want to use SQL Server Express?

You don’t want to depend on Microsoft

That can be a good reason enough sometimes. There is nothing preventing Microsoft from crippling SQL Server Express in the future to force users to move to a paying version early.
I suppose that whatever database system you use, even Open Source ones, there is always the possibility that the company supporting its development goes bankrupt, the Open Source projects goes dead or decides to go in a direction that doesn’t suit you..

It’s only supported on Microsoft OS

True, and that’s a good reason to chose something else.
There is a hidden cost in SQL Server Express: it needs to run on a Windows machine, and that’s not free, although SQL Server will work on older Windows 2000 machines and Windows XP which are arguably not expensive.

Your database needs will exceed SQL Server Express specifications

If you think any of your databases will grow beyond 4GB or that it will get busy and you need all the RAM and CPU you can get, then SQL Server Express is probably not for you as it will only use 1 CPU and 1GB of RAM at most.
If your needs go beyond that, then you’ll have to move to a paying version.

Upgrading can be expensive

It’s true that moving to the next cheapest upgrade of SQL Server Workgroup will cost you about US$700 for a 5 user license. The limits imposed on the database are much higher (2 processors, 3GB RAM and no size limit) but if you need more clients / or higher limits, then the expense will grow quite fast, and you’ll have to manage those hateful client access licenses.

Your needs are more modest

We haven’t talked here about single-file/single-user database systems. These databases don’t user resident services and are usually meant for more limited needs, sometimes allowing only a single user to be connected.
The footprint of these non-server databases is a lot smaller, typically only requiring a single dll or a handful of files to be installed.
They are extremely useful for desktop application that do not really require multi-user support or advanced security features.
Here again, Microsoft offers SQL Server Compact, which, despite the name, doesn’t have much to do with the other SQL Server editions. This one is also free, but has a limited feature set and only allow single-user access as it is meant to be a lightweight database and works well in limited memory environments such as those found on mobile devices.
Of course, here again there is a lot of competition: Thunderbird, SQLite, MS Access and VistaDB (for embedding into .Net applications, not free) to name a few.

These are pretty good times when it comes to databases: we get more choices now than we ever had.
As usual, choosing a database as a back-end for your products isn’t easy: you need to consider cost, licensing, support and the future.
There isn’t a single database system that will meet everyone’s needs for all types of use, so choose carefully.
SQL Server Express is a very good contender in that market. It should not be dismissed out of hand because it’s from Microsoft, in the same way that PostgreSQL shouldn’t be dismissed because it’s Open Source.
Just use the tool that best answers your needs for your particular circumstances.

References:

• SQL Server Express main page
  www.microsoft.com/sql/editions/express/
• Compare SQL Server editions
  www.microsoft.com/sql/prodinfo/features/compare-features.mspx
• System Requirements for SQL Server Express
  www.microsoft.com/sql/editions/express/sysreqs.mspx
• SQL Server Compact (single-user, small footprint database)
  www.microsoft.com/sql/editions/compact/
• The VistaDB database, commercial embeddable database for .Net applications
  www.vistadb.net/
• SQLite database, public domain, flexible file-database system
  www.sqlite.org/
• The Firebird DB (based on Interbase 6 from Borland)
  http://www.firebirdsql.org/
• PostgreSQL Open Source database server
  http://www.postgresql.org/

If you are from North America and a handful other countries 04/05/01 would mean 5th of April 2001.
If you’re in Asia or some eastern countries, if could be 1st of May 2004.
For the rest of the world, it would look like 4th of May 2001.

We cannot ignore any longer the importance of writing dates in a format that is understandable by everyone.
Still too often, web sites -even big companies- with a global audience state dates in a format that is probably obvious where they live but not consistently so to everyone else.

Developers are sometimes guilty of this sin, as can be seen in the format chosen to specify dates in Microsoft Access SQL, which requires that hard-coded dates be formatted in the US rather than the more neutral ISO8601 format.
I wonder how many bugs that little egocentric feature caused in business applications that where developed by foreign coders…
I know it also happened to me at least once…

The worst thing is that when you write 04/05/06, you may very well have thought about the rest of the world and meant DD/MM/YY; alas, unless the context grants us some clarification or the actual date is unequivocal, there is sometimes no way for a lot of us to know for sure what you really meant.

We should ban the use of the short date format in communications that may be read by more than 3 people, in today’s world, that’s pretty much all communication.

Instead a simple format such as 05MAY2006 is clear an unequivocal: it’s easy to read and understand, regardless of where you live.
You’re free to add spaces, dashes, dots, comas or slashes to it if you want, you can use 2006MAY05 if you prefer or even MAY05,2006, whatever pleases you: it still remains readable.

The rule is simple:write the year in full and never write the month as a two-digit number, that’s reserved for the day

The point is that you, me and the rest of the world can read it and understand exactly what was meant; it’s an easy problem to solve.
Surely that means something to you?

References

• http://en.wikipedia.org/wiki/Date_format
• Does you code pass the Turkey test?

If you don’t know him then he’s the mastermind being similar projects that centre around a simple premise: make humans do the work that computers can’t do.
One of his on-going projects is the ESP Game where 2 online players are trying to come up with a common description of a random picture. It’s apparently an addictive game and it helps solve a problem that computers are terrible at today: describing accurately the content of photographs.
Google is using his research to make image search more useful by returning content more relevant to your queries.

So, what’s the relevance with the CAPCTHA?
CAPTCHASs were invented to resolve a simple problem: stopping computers from automatically filling-in web forms to create accounts on popular free services that they would use to send spam from.
They are a visual version of the Turing Test, elaborated by the WWII genius cryptanalyst Alan Turing as a way to test how far machines could behave like humans: not knowing who she was interacting with, if a person could not tell the difference between a human and a machine, then the machine passes the test. It’s a measure of the success -or lack of- of artificial intelligence and the idea spawned many others, including CAPTCHAs.

CAPTCHAs simply require that small problem be solved before a web form can be submitted. Typical problems include blurry and distorted images of text or numbers that would be very hard for computers alone to decipher, but that our brain has no problem solving.
There are an estimated 60 millions of CAPTCHAs being solved by human beings every single day. That’s a huge amount of lost brain power as nothing really useful comes out of it (apart from preventing spam, of course).

reCAPTCHA‘s genius idea is to use that brain power to solve a problem that we would actually like computers to solve: digitizing books.
There are millions of books that were printed in the days before computers became ubiquitous, and there exist no electronic version of them except scanned images of their pages.
Optical Character Recognition software is getting very good, but when the scan is of poor quality or the book is old, many words cannot be automatically recognised.
Humans on the other hand are quite good at reading words, even if they are badly distorted and barely recognisable.
Instead of making up a distorted image that you would have to recognise, reCaptcha simply presents you with 2 words: one it knows and one it doesn’t and you’re asked to guess both.
Every unknown word is checked multiple times by different people and you thus end-up with a very accurate interpretation of the word that can be fed back into the electronic version of the book being scanned.

CAPTCHA do not entirely solve the problem of spamming, but they are an financial issue to spammers: automated electronic system cannot solve good CAPTCHAs, so some spammer rely on low-paid humans to do the dirty work for them.

It’s fine by me: poor people are getting paid to do something useful (help digitise books) and spammers are wasting their money doing so. In my case, they lose even more, because I use moderation to read comments before they are visible and Askimet to detect spam, which means that however hard they try, their spam never gets anywhere anyway.

In the fight against spammers, it makes me happy to know I’m costing them something for a change…

References:

• Breaking visual CAPTCHA
  http://www.cs.sfu.ca/~mori/research/gimpy/
• Vulnerabilities of some CAPTCHA implementations (reCAPTCHA isn’t)
  http://www.puremango.co.uk/cm_breaking_captcha_115.php
• Luis Von Ahn’s website
  http://www.cs.cmu.edu/~biglou/
• Lecture on Human computation by Luis von Ahn (technical, but very inspiring)
  http://video.google.com/videoplay?docid=-8246463980976635143
• The Internet Archive, benefiting from solving ReCAPTCHA, also a incommensurable source of free books
  http://www.archive.org/details/texts

Install the client-side highlighter

Download dp.SyntaxHighlighter. Uncompress its content under a new /skins/common/SyntaxHighlighter folder in your MediaWiki installation (don’t forget to make sure the files can be read by the web server; for instance, on Linux you may use chown apache.apache -R *).

In the skin template you are using for your MediaWiki site, insert the necessary code as required. In my example, I use the default /skins/MonoBook.php template into which I added the following:

Just before the closing </head> tag:
<!-- http://www.dreamprojections.com/syntaxhighlighter/ --> <link type="text/css" rel="stylesheet" href="/skins/common/SyntaxHighlighter/Styles/SyntaxHighlighter.css"></link>Just before the closing </body> tag:<!-- http://www.dreamprojections.com/syntaxhighlighter/ --> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shCore.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushCSharp.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushXml.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushSql.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushJScript.js"></script> <script language="javascript" src="/skins/common/SyntaxHighlighter/Scripts/shBrushPhp.js"></script> <script language="javascript"> dp.SyntaxHighlighter.HighlightAll('code'); </script>

Note that you must include a reference to each source file corresponding to the type of programming language you want to highlight.
Have a look under the /skins/common/SyntaxHighlighter/Scripts/ folder to see which languages you can highlight; there are a lot more than the few I use on my site.

Install the WikiMedia extension

I’ve created a small extension to WikiMedia to allow us to enclose any source code in a new <codesyntax> tag. Click on the View Plain option below and copy-paste the following code into a new file that you will save under /extensions/syntaxhighlighter.php (again, make sure this is readable by the webserver).
<?php /************************************************************ ** Simple WikiMedia extension to create a new <codesyntax> tag that renders ** syntax highlighted through the dp.SyntaxHighlighter javascript helper ** See http://www.dreamprojections.com/syntaxhighlighter/ ** See http://meta.wikimedia.org/wiki/Extending_wiki_markup ** (c) Renaud Bompuis, 2007 ** ------------------------------------------------------------------------------ ** v1.0 - 20EFB2007 : first version. Uses the <pre> tag to ensure that the content of ** the <textarea> containing the source code will not be messed up by WikiMedia. ** There are certainly better ways to do this, but they seem more complicated than this ** (registering hooks, recording content before it is expanded and then replacing ** messed up content with the original) ************************************************************/ $wgExtensionFunctions[] = "wfSyntaxHighlighterExtension"; function wfSyntaxHighlighterExtension() { global $wgParser; $wgParser->setHook( "codesyntax", "renderCodeSyntax" ); } // The callback function for converting the input text to HTML output function renderCodeSyntax( $input, $argv, &$parser ) { $output = "<pre style='border:0;padding:0;margin:0;'>" ."<textarea name='code' class='".$argv["lang"]."'>" .Xml::escapeTagsOnly($input) ."</textarea></pre>" ; return $output; } ?>
Add the following line to the end of your LocalSettings.php file, right before the closing ?> tag.
require_once 'extensions/syntaxhighlighter.php';

Usage

To highlight code in your MediaWiki pages, just enclose your source code with the new <codesyntax> tag. This tag takes a lang attribute to specify the options that normally would be listed in the class attribute in the dp.SyntaxHighlighter documentation.

For example:
<codesyntax lang="c#"> string url = "<a href=\"" + someObj.getUrl() + "\" target=\"_blank\">l;"; // single line comments // second single line override protected void OnLoad(EventArgs e) { if(Attributes["class"] != null) { //_year.CssClass = Attributes["class"]; } // base.OnLoad(e); } </codesyntax>
Will display as:
string url = "<a href=\"" + someObj.getUrl() + "\" target=\"_blank\">"; // single line comments // second single line override protected void OnLoad(EventArgs e) { if(Attributes["class"] != null) { //_year.CssClass = Attributes["class"]; } // base.OnLoad(e); }

For more information on using dp.SyntaxHighlighter see:
http://www.dreamprojections.com/syntaxhighlighter/Usage.aspx

When you start looking into all these platforms you can be blown away by the ease of implementation and elegance of some; your head spins at all the features and claims being made and it becomes hard to achieve a rational decision.

I’ve been faced with that very issue recently and it sucks. Actually, the time spent investigating all these possibilities is really what is being sucked away.
Of course, learning about new technologies is not just fun, it’s also essential: you have to keep on top of what’s new and decide if you want to join in on the fun or just stay comfortably where you are.

I also realized that you need to get away from the coolness factor of each these much touted technologies and actually try to be objective.
During my investigation, I tried to decide what were the really important factors that I should consider to make a rational choice.

I came up with the list of the following 10 questions that I think everyone should ask themselves when choosing a platform for an important project.

1- Is the platform wide enough?

New technologies that can show amazing productivity get all the buzz. After all, that’s what we developer want: enough with the nuts & bolts! Give us a toolbox with all new shinny power tools! The problem is that often these technologies are impressive, but looking beyond the sample projects into what actual users get confronted to, you start noticing discussion threads that should obviously signal that the platform is not wide enough.
Choosing a technology that hasn’t achieved its goals can be a huge problem down the line: you implement 80% of your project in record time, only to realise that the platform you chose doesn’t support proper cross-browser detection, or that its implementation of security is way too lax, or that it only support XML for data storage and now you actually need a real database for performance reasons…
In those cases, you can of course develop those areas yourself and contribute to the project, but doing so supposes that you planed it all along and you factored that in your schedule, both in time and in cost.

2- Is it popular?

In other words: are other professional people actually using that platform in real life? I’m not talking about Joe Developer using it for his local community church website. I’m talking about businesses or large organisations actually using that software platform successfully.
If all you need is a simple 3 page website, then professional popularity amongst the Top 500 business companies is probably not that important. On the other hand, if you plan to build a large project, or you know that your project could grow large in the future, then you probably want to ensure that others have taken it there before you.

3- Is it mature?

Has the platform existed for long enough that it went through enough abuse to withstand almost anything you can throw at it?
A technology that is only a couple year old may be too young to have evolved to the point where it actually can solve most common and not-so-common problems. Technology evolves through iterations: each one is a good long hard look at what didn’t work and tries to fix, improve and extend.
A young technology may be more cutting-edge and exciting, but when you need to commit to it for a big project, its age may quickly come as an issue; its shinny surface may blind you to the black holes waiting below…

4- Are local developers available?

Always think about the future. When you’re gone from the project, who is going to look after it? Getting a web project entirely in Rebol may be cool and fun, but how many people in your city actually use that thing anyway?
Aren’t you creating a liability if you’re dead-set on choosing a platform that cannot be maintained?

5- Is it scalable?

Most projects start small but dream of ending-up big.
It’s usually fine at the beginning when the number of users and visitors is manageable, but what happens when you start to be successful?
After all, that’s why we craft our projects: we want them to succeed, we want the world to see them. When the world starts coming though, will the terrific platform we’ve chosen break at the seams?
How much overhead does the platform create? It is flexible in its database support? Is there a way to implement clustering or persisting sessions accross multiple servers?
What’s the cost in memory and CPU resources? How many simultaneous connections can you get? What are the bottlenecks? What do you need to do to compensate for them? How much would it cost?

6- Do you have control?

Some frameworks work hard for you and can make you really productive. Sometime though, they try too hard and hide too much from you.
When a feature doesn’t exactly work the way you want you have to dig deep into the entrails of the beast to make sense of it and change its behaviour.
A platform that was all nice and pretty can get very dirty and complex inside, making it difficult to adapt to your needs.
A platform should do most of the hard work but still allow you to redefine its default behaviour without much hassle. It should be built with extensibility in mind, making easy thing easy and difficult things possible.
A case in point are 4G development platforms: they usually have a fairly narrow field of expertise and tend to use graphics and nice simple paradigms to quickly build complex tasks but when you are faced with a specific issue that hasn’t been solved by the platform, you often have to resort to ugly hacks to get around their limitations.

7- Does it satisfies the essential constraints for your project?

When being blinded by something cool, we often become skewed in our judgement and guilty of lowering the importance of constraints that are actually critical for our project.
In all honesty, I really wanted to use Ruby on Rail, and I liked the principle and elegance of it so much that I came to seriously consider it, that is, until I came to my senses and looked into something that is critical to my project: support for complex Asian languages.
Ruby is being developed in Japan, so you could think that it should be able to handle complex ideographic characters well.
Turns out that the Japanese are not terribly fond of Unicode and, unless I’m mistaken, prefer to use Code Pages instead. The result is that Ruby doesn’t have great support for Unicode, and that sucks.
That’s why Unicode was created: to attempt to simplify all this patchwork of implementations that are not compatible with each other.
Because I needed to be able to handle English, Mandarin, Cantonese and probably any other language out there, Unicode is the only viable option to abstract the issue of language enough that it becomes manageable.
So because of Ruby’s poor support for Unicode, it would not be rational to use Ruby on Rails for my particular project as it would probably greatly increase the complexity and hacks necessary to manage complex Asian languages in a unified and simple way.

8- Documentation?

All framework have a more or less steep learning curve: you need to think differently to adapt to the particular framework’s frame of mind, so to speak.
Make sure that the framework has a lively community, that it has more documentation than you can swallow and that documentation is well organised.
Make sure also that there are samples, tutorials, webcasts, videos or whatever that cover the principle aspects of it.
A beautiful framework no-one talks about means that no-one will answer when you have a question. A beautiful framework without documentation is useless. A beautiful framework with lots of disorganised docs means that you’re going to waste a lot of time experimenting instead of building your project.

9- Support?

Here I mean support in the wide sense of the word: who is behind the framework. Are they likely to stay in business long after your project has been completed?
Are they commercial or Open Source? In either case there are issues for and against: a commercial venture backed by a small company may falter and disappear overnight. Similarly, an exciting open source project managed by only 2 people can suffer the same fate when they decide to move on. Support also includes what help is available to you when you encounter issues. Is there any guarantee that you could have your important technical questions answered?
If you encounter a large issue, is there someone to help you?
I would contend that the best and most serious projects should have professional help available: you can always fall back on paying someone to help you through, whether it’s the original developers, a paid-for support hotline or a third-party specialist, it’s important that you know you can get your answers when you’re stuck.
Developing for an important project without safety net is dangerous: you can waste a lot of time and effort, endangering the very project you are undertaking, if you stumble on a problem you cannot solve yourself.
Large frameworks like .Net or big Open Source projects usually have enough users and support groups and specialists that it’s likely any question you may have has already been answered somewhere or that you can pay someone to help you out when you need it most.

10- What do you know best?

Every framework is based around a particular data-management model. Every framework is built with a particular programming language in mind. Every framework demands that you learn something new. How comfortable are you with the requirements? how long is it going to take you to start to get really productive?
Like their real-world counterpart, it can take very little time to learn a new language’s words and syntax but it takes years to become proficient in it. A framework adds another layer of abstraction that you will need to get experienced at.
If you need to both learn a new language and a new framework, chances are that you are at best months away from being able to churn code without having to look into the documentation every 5 minutes.
Learning new languages and concepts is necessary, but again, if you chose that route you must be sure that your project has that learning curve built-in, otherwise you’re going to move at the pace of a turtle when in fact you chose that particular framework because you though you would be more productive and faster than a sparrow.
If you chose a new framework, make sure that you already possess most of the knowledge it requires: in my particular instance, I chose .Net and C# because I have already worked in that framework and I know it can tackle the project without me having to waste too much time on learning the platform rather than implementing the project.

We all should seize any opportunity to learn something new. Learning is often more exciting than doing the same old thing over and over again and it’s a necessary part of staying in business. Curiosity is an excellent quality that must be nurtured and indulged.
The issue is when to do it.

Choosing a new development framework is not easy. The best way to tackle the problem is to pose it as a risk analysis study.

The framework you chose should simply be the one that poses the least risk to you succeeding your project.

A couple years ago I had this really depressing experience with a brand new system I was building. All the components were newly bought, but installing the OS (a Linux distro) used to fail almost at the end of the process.
No matter how many times I tried, I could never get to the end of it.
By clever subterfuge I managed to get it installed only to have it crash on a regular basis.
It was unstable, unreliable and after two days wasted banging my head against the walls, I gave up…

Well, no for too long. A flash of inspiration came to me and I popped in the install CD and ran the only utility that was available at the prompt: memtest86.
That simple tool is a godsend (if I may appropriate the word from the believers. I promise to give it back). After running its various memory tests for 10 minutes it reported errors in one of the RAM sticks installed on the motherboard.
All that aggravation for a puny bit that was not remembering its state…
I promptly returned the RAM and tested the new one for a few hours until I was confident there was no issue with its chips and went my merry way to install the OS I so desperately needed. All went without a hitch.

So my advice is: go to the memtest86 website. Burn the bootable ISO and test your PC from time to time, especially if you have strange intermittent issues that you can’t pin down to a simple software problem.
You’d be amazed at the number of times I had to ditch a stick of RAM… memtest86 saved by sanity countless times.
By the way, consider donating a little bit if it helped you too. That’s always cheaper than a session with a shrink…

The only thing I really dislike about Microsoft is its marketing, its pricing, its Genuine (Dis)Advantage that nags me every time I need to install something and its lack of openness when it comes to inter-operability with other competing implementations (here I’m particularly thinking about its network protocols that the Samba team tries to decipher and re-implement as an Open Source platform).
On the other hand, Microsoft is made of great programmers, great minds that you can watch on Channel 9 and read on their insightful blogs.

Microsoft is really (I mean REALLY) pro-developer: they understand that the Operating System alone is nothing without lots of applications sitting on top of it, and they offer developers a lot of goodies.
One such useful programme is +Microsoft Empower for ISV_. It’s a simple membership that allows a small software company (ISV meaning Independent Software Developer) to own a number of licenses for its internal use at a fraction of the price it would normally cost.
What you get is pretty wide for a small business: 5 licenses for Windows XP (whatever version), 5 licenses for Microsoft Office, 1 license for Windows 2003 Server and Exchange, SQL Server, SharePoint Portal, a MSDN Premium Subscription that covers almost anything else, including 5 licenses for Visual Studio 2005 Pro.
You also get access to MSDN downloads, beta software and tons of libraries, SDK, etc. A MSDN subscription alone is about US$2000…
The Empower ISV programme is quite cheap and depend on the country you are in.
I paid mine HK$4,260 about US$530.

To get all this you need to register as a Microsoft Partner (that’s free), then apply for the Empower for ISV programme by making a promise to release a commercial software within 2 year at most. You need to give some details and pay your due. After a few days, you get confirmation if your application is accepted or not.
As far as I know, you need to be a company and have a company website, but that may not be mandatory in all regions.
The software you get is the normal US version plus whatever local version there are for your region. My bunch of DVD came in Japanese, Cantonese, Mandarin, some in Korea and Thai too.
At least, the English version is supplied. For some software, you also get the multilingual version that include European languages as well.
You manage your licenses by login under your MSDN account.

Really, it’s a nice touch from Microsoft to give us poor developers access to all this for such a reasonable price. I can only encourage other small software companies and independent developers to do the same.

Using PropertyInfo you get enough information about a particular property to decide if you want to display it or recurse into it if it’s a complex type that contain other properties.

The only big issue I faced was when dealing with indexers.

The issue with Indexers

Indexers are defined in code like that:
using System; using System.Collections; public class DayPlanner { // We store our indexer values in here Hashtable _meetings = new System.Collections.Hashtable(); // First indexer public string this[DateTime date] { get { return _meetings[date] as string; } set { _meetings[date] = value; } } // Second indexer, overloads the first public string this[string datestr] { get { return this[DateTime.Parse(datestr)] as string; } set { this[DateTime.Parse(datestr)] = value; } } } We’ve defined an overloaded indexer: one takes a DateTime, the other just a string representation of a date.

We could use them like that for instance:
DayPlanner myDay = new DayPlanner(); myDay[DateTime.Parse("2006/02/03")] = "Lunch"; ... string whatNow = myDay["2006/06/26"]; Now, let’s say we want to print all property names and their values for an arbitrary object.
Let’s try on our myDay intance:
Type type = myDay.GetType(); PropertyInfo[] picoll; picoll = type.GetProperties( BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static); foreach (PropertyInfo pi in picoll) { object val; val = pi.GetValue(myDay, null); Debug.WriteLine(pi.Name+"="+val) ); } So, that part is easy: we iterate through all properties of myDay and print their Name and they value.
Problem is, when we reach the indexer property, we get a TargetParameterCountException on line 8.

We need to detect the indexer before attempting to get its value.
To do that, we need to modify our foreach loop to:
foreach (PropertyInfo pi in picoll) { object val; ParameterInfo[] parcoll = pi.GetIndexParameters(); if (parcoll.Length == 0) { val = pi.GetValue(o, null); Debug.WriteLine(pi.Name+"="+val) ); } } Here we attempt to get the indexer parameters for every property. If we’re dealing with an indexer, the parcoll array will contain a list of ParameterInfo related to that particular indexer.

So what really is this parcoll and its ParameterInfos?

A ParameterInfo contain information about the parameter of an indexer property.
The parcoll array will contain one entry for each overloaded indexer.
In our concrete example this is what we get:

• pi.Name will be Items, this is standard for all indexer properties.
• pi.PropertyType is the type of the returned class of our indexer, in our case, a System.String.
• parcoll.Length will be equal to 2 because we have 2 overloaded indexers.
• parcoll[0].Name will be date.
• parcoll[1].Name will be datestr.
• parcoll[0].ParameterType will be System.DateTime.
• parcoll[1].ParameterType System.String.
• parcoll[0].Member and parcoll[1].Member are in fact references to pi itself

That’s a lot of nice information about our indexer, but what if we want to get a value from it? Consider the definition for GetValue:
PropertyInfo.GetValue(object o, object[]{} index). That second parameter is used to get a value from the indexer.

So, how do we use it?

If you know valid keys that you can feed to the indexer, then you just do that (assuming that pi is the PropertyInfo for the indexer).
string meeting; meeting = pi.GetValue(myDay, new string[]{"2006/02/25"}) as string; Or, if you prefer to use a DateTime: string meeting; meeting = pi.GetValue(myDay, new DateTime[]{DateTime.Now}) as string; But what if you don’t know anything about the values contained by the indexer?
Well, unless you use attributes to decorate the indexer or the underlying storage field with specific custom information, you’re basically screwed.

To understand why, just consider how the indexer is implemented:
When you pass it a DateTime or a string as a parameter, it just passes it on to the getter/setter as the value.
What is done with that is up to the implementer.
If you want to maintain a collection of some kind, then you need to use private fields to hold the keys and their values.
.Net doesn’t store the values passed to the indexer or the values returned by it anywhere.

Consider how .Net doesn’t give you any means of iterating through the keys and values of an indexer.
You can’t do a for(;;) or a foreach() loop, you can’t even know how many items are maintained through the indexer by directly questioning it.
So, it certainly is normal that you cannot get anything out of the indexer without knowing exactly how to access it.

The Observer Effect

In many fields of science, the act of measuring something, whether it’s an electronic circuit or a psychological experiment, can actually influence the observed phenomenon.
This is unsurprisingly called the Observer Effect.

In .Net, Reflecting on object can be very much the same.

Consider the use of Reflection when it comes to getting information about an object:
You use metadata describing the object to get more meta data information about its members You can use that meta data to direcly query the state of an object.

If you limit yourself to getting metada, then no harm is done: you’re not querying the object instance itself but the metadata about its class.

If you start querying an actual object instance for its values then you are actively interacting with it.
In most cases, if you’re just getting the value of Fields or Properties defined as follow, then it’s fine:
class GetMe { private string _meeting; public string Meeting { get { return _meeting; } set { _meeting = value; } } } Using reflection, we can get the values of both _meeting and Meeting without any side-effect to the object.

Now consider this:
class GetMeNot { private int _collection = null; public int Collection { get { if ( _collection == null ) { OpenDatabaseConnection(); _collection = FetchAllData(); } return _collection; } } } Now what we’ve got here is a case of lazy loading or lazy initialization.
The consequence is that by querying the Collection property we’re actually creating it, with a potential for a lot of side-effects, like the change of other properties, here maybe related to opening the database connection for instance.

So, when getting values of the properties of an object instance, we can actually completely change its state.
Whether this is a concern or not really depends on what you’re trying to achieve: you may not be able to achieve being a passive observer using Reflection unless you know about the construction of the particular class you’re dealing with.
Using Attributes, you could give hints about which Property not to query and which Field to get instead. In our case, we could add a custom attribute like [ImplementedBy("_collection")] to our Collection property for instance.

If you want to do something generic, then you’re out of luck it and have to deal with that fact.
One possible way to minimize the effect would be to not query Properties that only have a getter and no setter, but that’s not a guarantee that all properties that implement a getter/setter are going to play nice.
Then there is also the -probably rarer- case of the Property that only has a setter.
Then there is also the case of the property getter that has some in-build security and will return different things depending on who’s calling…

All this seems pretty obvious once you understand the whole picture but it’s easy to get so impressed and dazzled by the power of Reflection that you don’t realise its natural limits.

Its particularity is that is does its painting magic on the client side. This can be a drawback in some instances, where the client browser has JavaScript disabled for instance, but since the code to paint is located within <textarea> tags, it will still display no matter what.

Configuring _dp.SyntaxHighlighter_to work in WordPress is pretty easy: just add the relevant code in your WordPress templates for header and footer according to the instructions found on the website and you’re done.

The issue though is making it possible to insert code when writing a post.
First of all, I doubt that there is any elegant way to make one of the rich HTML editors work well, so you’ll have to go back to basics and edit in HTML (WordPress at least hides the <br> and <p> tags so its’ not too bad).
Then there is the issue of WP adding the <br> and <p> to replace linefeeds, and this is where it becomes annoying: if you insert <textarea> tags, everything in it will have the extraneous formatting code that will display verbatim on screen, sprinkling your nice source code with ugly html tags.

The solution I found was to use a small WordPress plug-in called codeautoescape and available from:
http://priyadi.net/archives/2005/09/27/wordpress-plugin-code-autoescape/

I then modified it a bit to take care of the <textarea> tag and the fact that we needed to take the attributes of the tag into account (dp.SyntaxHighlighter uses the class attribute to configure options for how to display the block of code).

You can download the modified version of the script: codeautoescape

To install it, just right-click and download from the link, rename it to codeautoescapte.php then install it in your /wp-content/plugins folder and activate it within the Plugins menu of WordPress.

Some examples of dp.SyntaxHighlighter in action:

C#

1 2 3 4 5 6 7 8 9 string path = ""; string safeReferrer = null; FileInfo fileInfo = new FileInfo(string.Format("{0}/{1}", path, image)); int index = 0;

// single line comments // second single line override protected void OnLoad(EventArgs e) { if(Attributes["class"] != null) { //_year.CssClass = _month.CssClass = _day.CssClass = Attributes["class"]; } // base.OnLoad(e); } /*********************************** ** Multiline block comments **********************************/ /// <summary> /// Gets or sets currently selected date of birth or <see cref="DateTime.MinValue"/> if selection is incomplete or invalid. /// </summary> public DateTime Date { String stringWithUrl = "http://blog.dreamprojections.com"; get { #region Hello world try { /* DateTime result = new DateTime( int.Parse(_year.SelectedItem.Value), int.Parse(_month.SelectedItem.Value), int.Parse(_day.SelectedItem.Value) ); i *= 2; */ return result; } catch { /* return _minDate; */ } #endregion } set { Day = value.Day; Month = value.Month; Year = value.Year; } }

JavaScript

/*********************************** ** Multiline block comments **********************************/ var stringWithUrl1 = "http://blog.dreamprojections.com"; var stringWithUrl2 = 'http://www.dreamprojections.com'; // callback for the match sorting dpSyntaxHighlighter.prototype.SortCallback = function(m1, m2) { // sort matches by index first if(m1.index < m2.index) return -1; else if(m1.index > m2.index) return 1; else { /* // if index is the same, sort by length if(m1.length < m2.length) return -1; else if(m1.length > m2.length) return 1; */ } alert('hello // world'); return 0; }

XML / HTML

<?xml version="1.0" encoding="utf-8" ?> <!-- comments --> <rootNode> <childNodes> <childNode attribute = "value" attribute='value' attribute=/> <childNode /> <childNode /> <childNode /> <childNode attribute="value"></childNode> <childNode> <![CDATA[ this is some CDATA content <!-- comments inside cdata --> <b alert='false'>tags inside cdata</b> ]]> </childNode> </childNodes> </rootNode> <!-- -- Multiline comments <b>tag</b> -->

PHP

/*********************************** ** Multiline block comments **********************************/ $stringWithUrl = "http://blog.dreamprojections.com"; $stringWithUrl = 'http://www.dreamprojections.com'; ob_start("parseOutputBuffer"); // Start Code Buffering session_start(); function parseOutputBuffer($buf) { global $portal_small_code, $portal_gzcompress; global $PHP_SELF, $HTTP_ACCEPT_ENCODING; // cleaning out the code. if($portal_small_code && !$portal_gzcompress) { $buf = str_replace(" ", "", $buf); $buf = str_replace("\n", "", $buf); $buf = str_replace(chr(13), "", $buf); } }

SQL

SELECT TOP 10 FirstName, LastName, Email, --SUBSTRING(Phone, 2, 3) AS CityCode, SUBSTRING(Phone, 7, 8) AS PhoneNumber, upper(LanguagePref) AS Language, Address1, UpdatedOn AS CreatedOn FROM profiles WHERE -- Exclude all test emails Email NOT LIKE '%test%' AND Email NOT LIKE '%asdf%' ORDER BY UpdatedOn DESC

There is an impedance mismatch between the usual programming technology and the database worlds. The mismatch occurs because most development is object-oriented and most database systems are query-based.
Getting data from the database into your objects and back is tricky and difficult to maintain: any modification in either camp needs to be reflected to the other.

One solution is to use an intermediate layer to take care of the impedance mismatch for us: that’s the job of he ORM.

When I initially took the decision to use an ORM I tried and reviewed a few. Each have a different way of doing their job and place emphasis on different things, depending on the school of thought they belong to.
Most still require a database-centric view and either abstract the whole and let you deal with meta-data that used to modify both code and database or they just let you modify the database and then update automatically the code.
Then there was XPO.
This one was a bit different: it really completely abstracted the database. It would create and update it automatically (whenever possible) without you having to ever deal with how the data is stored: it would work in the exact same way regardless of the underlying chosen database.
You could even use it on an already existing database and it would be happy to talk to it.

So I chose XPO.
It didn’t have fancy designers but it offers a simple, reliable and flexible way to persist my objects in the database of my choosing.
One thing to note too is that you can get it with full source code (not commented though).
I find it reassuring when a company doesn’t mind its code to be scrutinised by its customers. It’s also proven invaluable in finding out XPO’s inner working (not that I understand a lot though).

I’ll be blogging a few articles in the future on various aspects of XPO that I think may be of interest to those already working with it. I’m not going to re-state what’s in the documentation or on the Developer Express (DX) community website and official pages, there is a lot of information available out there, it’s just sometimes hard to get to know the fundamentals because it’s sometime buried deep. Whenever possible (depending on time and complexity) I’ll make a sample project. I’m even thinking about building a repository of samples showcasing various aspects of XPO.

Let’s see if I can hold the load for long…. I’ve also committed myself to DXCore recently… but that’s something for another post altogether…

References:

• Official XPO product page
  http://www.devexpress.com/Products/NET/XPO/
• XPO articles on the DX Community blog
  http://community.devexpress.com/blogs/xpo/
• DX Peer Support newsgroups
  http://www.devexpress.com/Support/Newsgroups.xml
• DX Knowledge base articles
  http://devexpress.com/Support/KnowledgeBase/
• DX Support center where additional information, tips and bug reports can be found.
  https://www.devexpress.com/support/center/
• An extensive list and description of available ORM for .Net
  http://www.howtoselectguides.com/dotnet/ormapping/

Geordi LaForge 70% Spock 52% Jean-Luc Picard 50% Mr. Scott 50% An Expendable Character (Redshirt) 45% Deanna Troi 45% Mr. Sulu 45% James T. Kirk (Captain) 45% Will Riker 40% Data 37% Chekov 35% Leonard McCoy (Bones) 35% Beverly Crusher 30% Worf 20% Uhura 20%

You work well with others and often fix problems quickly. Your romantic relationships are often bungled.

Click here to take the Star Trek Personality Test

I used to love/hate these. I would hate them when they were clearly biased in the favour of the customer (for these kind of large multi-million dollar contracts, the customer imposes his rules), and I would love them -or at least appreciate them- when they were fair and balanced as they would clarify every party’s duties and responsibilities.
After leaving that world and starting my own IT business I wanted the relationship with my customers to be as non-threatening and amicable as possible.

I decided -stupidly- not to have stringent legal documents defining the terms of the sales. What I would do instead was spend time making sure that the customer understood everything by explaining at length what they were getting and ensuring that they were no misunderstanding. In my written offers, I would painstakingly describe the technical part of their project and clearly mention what they were getting, when and at what cost.
My rationale was not to scare customers away and instil trust by being verbose and friendly.

Well, that worked for a little while until a particular customer ordered a server as part of a larger web-site project.
Once she confirmed her order to me I promptly ordered the hardware from Dell and paid for it in advance (their requirement for processing the order). A few days later, she phones and calls the whole project off, a project we’ve been spending time on for nearly 2 months.
She just happened to realise that she wouldn’t have time to build the content for it.

Well, that’s where not having stringent sales conditions bit me in the arse. When I told her I already ordered the hardware and couldn’t refund her until I got my own refund from Dell she started to spin her own fantasies and self-indulged delusions and started to treat me like a common criminal. She became a complete lunatic and I really didn’t know how to handle her any longer. I even gave her copies of my correspondence with Dell, bank account statements to show that the server was paid for…
In the end, I tried to explain where her assumptions were wrong and told her she would get her refund at the latest on a certain date, based on what Dell told me would be the longest waiting period. Turned out that it took 6 weeks to get my refund, just in time for hers.

The morale of this story is that:

• you shouldn’t be naively stupid and think you can base your relationship with a client on trust only.
• you should have conditions of sales accessible on your website and refer to them in all your correspondence with your clients.
• you shouldn’t work with people who can have a propensity for confabulation. That particular customer was a provider of dubious alternative medicines and was clearly a lost cause to reason. My first instinct and moral judgment of not working for her was right and I should have listened to it instead of just thinking about the much-needed cash it could get me…

…which bring me to the last lesson: don’t sell your ideals for money, however bad you need it.

One self-inflicting problem I’ve often been confronted to is over-engineering: I tend to over-think, over-design and over-anticipate.
Sometimes it’s good to just spend time trying to make your code more modular and as future-proof as you can, but often, it leads to code that has no real use right-now, and that ends up having to be modified because when you need it, it’s not in the form you’ve implemented it originally.
This is an Extreme Programming practice aptly named You Aren’t Gonna Need It.

A classic example of YagNi that I am paying dearly is a large application I’m working on at the moment (at least large from the perspective a single developer).
It’s a fairly complex database application with an equally demanding winform GUI layer in C# on top.
I built the GUI in a very modular fashion: each bit of information that could be grouped has its own User Control, sealing its particularities.
Then each of these controls is used in panels that embody another layer of functionality.
Each panel is then loaded as needed into a form that display the panels on demand.
The issue is that I now have to make major changes to the application and there are too many layers in these controls, making code hard to change.
I had built it this way for re-use, thinking that it could be useful to have independent specialized controls, say for entering addresses, or customer details, etc.

Well, each of these controls took me a lot of time to craft, trying to make them as independent upon my particular implementation as I could. It turns out though that I’ve used each control no more than once -maybe twice for a couple- in the whole of the application, and I now realize that it’s unlikely that I would get another job that would require the exact same user-interface broken down the exact same way.
So my uber-GUI controls are mostly a waste of time and complexity.

I should have simply created the forms directly, using partial classes (a new .Net 2 feature) to separate each logical part into something manageable.
My application would have been faster to develop and faster to run, with a lot less of unnecessary layers of abstraction.

I’m having a hard time with that, but I’m getting better, hopefully…

The main issue is that IE7 is now yet-another browser that web designers have to test and include support for.
It may not be a huge deal for most sites, but some do rely on specific browser versions. In my case the problem arose when I tried to connect to my company’s online bank account. For security reasons (?!) it only supports IE6, not even Firefox so it’s one of the few sites that requires Internet Explorer. Problem is that IE7 replaces IE6 entirely, with the unfortunate result that I now can’t connect to my bank accounts (company accounts only, the portal for personal accounts supports Firefox).
So I’ve had to remove that beta.

That is certainly a specific issue, but it will be interesting to see how fast websites that rely on specific versions of browsers, be it for security reasons (because they want to simplify their development to a manageable level and support well a single browser) or because they rely heavily on CSS and JavaScript, how fast these websites will adapt.
Early adopters of Vista will now doubt have a few -temporary I’m sure- surprises.

I think Microsoft made a big mistake when they decided to stop development of IE6 and not incorporate any more improvements into it.
Maybe the company was under too much legal spotlight at the time and leaving the scene for a while was a move to appease those who were concerned about its monopoly on internet browsing.
The unfortunate side-effect was that the browser millions relied-on had still some important flaws. Its User Interface and usability has now been overtaken by Firefox and Opera which benefited from the development interrruption of IE.

I am not sure if IE can become better than Firefox in terms of extensibility: Firefox sports thousands of extensions and themes to customize it. I’ve got quite a few useful ones that are certainly particular to my way of browsing the net. How is MS going to integrate that into IE7?
What made Firefox popular was its focus on carrying-on where IE6 had left and offering a better browsing environment that could be customized with all the widgets and utilities that you wanted.
If MS doesn’t incorporate the same level of extensibility, it will carry-on losing a big share of its browser market to Firefox, at least in the personal user world.

But Maybe Microsoft doesn’t care too much about browser wars any longer.
When you approach perfect standard compliance, which browser you use to render the content isn’t important any longer. Since MS failed in the past to tie revenues to the browser itself (remember the ill-fated channels that would bring commercial content to your browsers back in the days of IE4? No?, well I’m not surprised…) the incentive for building a better browser is elsewhere.

One of the reasons that may be motivating MS is that a lot of its other products are increasingly dependent on the Internet. In a corporate environment, almost all business-oriented services are accessed through a web interface of some kind.
Microsoft must ensure that it retains the monopoly on these implementations to keep control of the complete user experience.
If MS wasn’t pursuing its IE development, more companies would start to look at alternatives, not only to IE, but also to larger expensive products like Exchange: once you accept that IE isn’t needed to offer rich client-side user interfaces, and that you don’t need Exchange either, you’re endangering Microsoft hold on the corporate market somewhat.
So going further and ensuring tight integration of its products together is paramount.

IE7 is also terribly important to MS on the home user-front: making sure that users use the same browser both at home and at work ensures that they are more likely to stay with what they know rather than accept using a new browser, that may require retraining for some.
That reason may seem feeble, but it ensures that the vast majority of people is not exposed to other alternative technologies.
Without IE7, that risk becomes real as more and more savvy users and corporations are turning to Firefox for its increased security (or at least for the fact that it is a lesser target to malware).

That factor alone may obviously not be sufficient to force a company to move away from MS, but it may be sufficient to poison the mind by making people to consider alternatives they would otherwise be somewhat reluctant to contemplate by fear of the unknown; a fear that MS has not shied away from exacerbing through its FUD campaigns.

Of course, all this requires a few grains of salt, I may be completely wrong…

This has been a major pain for web designers trying to make their site look nice and a lot of hacks have been invented to get around the problem.

Note: Before implementing the solution described here, have a look at the references added on 31AUG2008 to the bottom of the article as they may offer more modern alternatives.
If they don’t work for you, then use the one described below.

A solution

The one I like best is using DHTM Behaviors, a Microsoft-only CSS extension that allows a programmer to extend the IE’s capabilities using JavaScript.

The behavior is loaded using a directive in your normal style sheet. Since it’s an IE-specific extension, it’s safely ignored by all other browsers. Similarly, you don’t need to change your HTML at all.
All that’s required is that you download a 1×1 pixel blank GIF image, the PNGbehavior script and add a couple of lines to the style sheet used in your site.

I’ve described a specific implementation for PmWiki before, this one is targeted to WordPress users.

• Get the blank.gif and pngbehavior.htc files and save them in your WordPress’ /wp-content/themes/ folder.
• Modify your theme’s stylesheet (for the default WordPress theme, it’s in /wp-content/themes/default/style.css) by adding the following anywhere in its file:

/* Fix for PNG alpha channel display in IE6 */
img {
  behavior: url("/wp-content/themes/pngbehavior.htc");
}

That’s all!

If you’re saving the behavior file or the blank GIF in another location, you will have to change the paths in the pngbehavior.htc file.

Make sure that the PNG images in your pages have their proper width and height set, otherwise if these attributes are missing from the <img> tag the images will appear as single pixel dots in Internet Explorer.

Note that I’ve modified the behavior file to make sure it can be used with the excellent LightBox JS 2.0 image display overlay script.

The original PNG hack comes from the impressive WebFX website.

Resources

• http://webfx.eae.net/dhtml/pngbehavior/pngbehavior.html
• http://www.huddletogether.com/projects/lightbox2/
• http://msdn.microsoft.com/workshop/author/behaviors/overview.asp

Modification on 30NOV2006

Corrected the pngbehavior script to only work in IE5.5 and IE6 as IE7 now correctly displays transparent PNG images.

Modification on 17FEB2007

Another reference to a slightly different implementation: http://homepage.ntlworld.com/bobosola/index.htm

Modification on 31AUG2008

A few of more comprehensive references:

• http://24ways.org/2007/supersleight-transparent-png-in-ie6
• http://www.alistapart.com/stories/pngopacity/
• http://labs.unitinteractive.com/unitpngfix.php
• http://code.google.com/p/ie7-js/

To help you type less, it has built-in autocompletion, meaning that usually only the first 2-3 letters of a command are enough.

Give it a shot, it’s free, but if you use and like it please consider a small offering to the developer so he gets the motivation to carry on working on it.

Get it from http://www.bayden.com/SlickRun/.

I must say that my first impression is that a lot of effort has been put in the interface. It’s simply magnificient, with subtle colours and a very pleasing responsiveness.

Now for all those Microsoft bashers who can’t see anything out of Redmond without scorn and disdain should look again. MS is often criticised for not being inovative, but they are taking a fairly big risk with Office 2007: the user interface of classic drop-down menus has now been changed for Ribbons, a large band of actions and functions that coherently group actions together, as buttons, menus and drop-downs.

While it’s easy to move around the interface, I can’t really comment yet on how quickly you get used to it or if it’s really a major improvement to usability. No doubt that legions of corporate users will have to be re-trained to familiarize themselves with the new interface, and that cost is going to make businesses think twice in upgrading the already over-the-top Office 2003 into something that does not really add any must-have features for the average user.

Sure, some large corporations will integrate reviewing and document exchange into their overall Microsoft-supplied architecture and no doubt that MS will find the marketing and commercial pressure points to “encourage” users into upgrading, but let’s face it: what do you need a word processor, a spreadsheet or a presentation package to do?

I’d say that close to 100% of the features you’ll ever need from an office suite have already been implemented years ago, so the pressure to keep you upgrading is now focused on improvements that most of us will never use anyway and, of course… eye-candy. Microsoft is taking a gamble but they have to: after Office 2003 there is not really anything significant to accomplish in the realm of office suites. Microsoft is trying to pull the rug under eveyone’s feet to make sure it still has the lead in some ways: between a demure-looking OpenOffice and the latest itration of MS Office, it’s going to be hard not to consider OO outdated, and I’m sure Microsoft will exploit that very feeling. Apple has demonstrated that User Interface is everything. I mean it! Apple’s hardware and software have polished, beautiful UIs. They have made of eye-candy and ease of use their driving marketing force. Everyone just assumes that an iPod is vastly superior in quality to anything else, but that’s not true. Everyone just want an iPod because they easy to use and are beautiful objects to own, like affordable jewelry that actually has a function. That has made them cool. Microsoft is now selling us improved software that we don’t really need, but that they need to shove down our throat to keep themselves in business. To achieve that and attract us like flies to dung, they have polished their products and made them very shinny.

I’m sure we won’t be able to resist…

They called me earlier this week to help them find a solution to a problem they had with getting their email off the company server, located in Hong Kong, from their office in Shanghai.
For months, every time more than a handful of connections were made, the server in Hong Kong would become unreachable, not just for email but for web access as well. The strange thing was that access to the rest of the Internet would be untouched.
Only that server would become impossible to access after a little while.

The office in Shanghai uses an ADSL connection provided by China Telecom. It’s a standard broadband service with a dynamic IP allocated every time they reconnect to the network. Nothing special there except that they block port 25 -used to send email-.
A lot of ISP do that in an effort to block spammers and worms infecting desktop PC: they require you to use their own SMTP servers to send email.
If it was only port 25, it would have been fine, but other ports to collect email would trigger the blocking behaviour as well: getting your email through IMAP or simple POP3 on ports 143 and 110 would result in everyone in the office being unable to connect to fetch their email.
When this occurred, the ADSL modem had to be physically disconnected. Everything would be fine again for a short while after reconnecting and getting a new IP.

So that was the situation when I started to think about ways to circumvent all that.

First thing I did was to open some randomly chosen ports on the mail server and redirect all traffic from those ports to their corresponding local ports: you would now access email on port 523 instead or the normal 143 for IMAP, etc.
Well, this didn’t work for long. After just a few hours it seem that they caught up with us again and we were back to square one.

Whatever was blocking the connection was more clever than just blindly blocking a few designated ports, it was also analysing the content of network packets to specially detect email activity.
The second solution was to again use 3 non-specific ports on the server for SMTP, IMAP and POP3, but this time use secure encryption (SSL) to ensure that the packets could not be analysed.

So far this seems to have worked. The only instance of blocking the office suffered so far was due to a laptop that was still using the original ports to send/receive email. Once its mail client was reconfigured, things went back to normal.

This spurred me add a couple of technical articles to my long and extensive email server how-to on : SecureAccess, AlternateAccess.