Posts filed under 'Software'

Sysadmin: file and folder synchronisation

Technology Over the years I’ve struggled to keep my folder data synchronised between my various desktop and laptops.

Here I present the tools I’ve tried and what I’ve finally settled on as possibly the ultimate answer to the problem of synchronising files and folders across multiple computers:

Sync Files

rsync

I’ve tried rsync, which is a great Open Source tool to securely synchronise data either one-way or both-ways.
It’s very efficient with bandwidth as it only transfer blocks of data that have actually changed in a file instead of the whole file. It can tunnel traffic across SSH and I’ve got a few cronjobs set up between various servers to back-up files daily.

It’s only weaknesses are that:

  • Every time it runs, it needs to inspect all files on both sides to determine the changes, which is quite an expensive operation.
  • Setting up synchronisation between multiple copies of the data can be tricky: you need to sync your computers in pairs multiple times, which quickly becomes expensive and risky if you have the same copy across multiple computers.
  • It doesn’t necessarily detect that files are in use at the time of the sync, which could corrupt them.

unison

It a folder synchronisation tool whose specific purpose is to address some of the shortcomings of rsync when synchronising folders between computers. It’s also a cross-platform Open Source tool that works on Linux, OS/X, Windows, etc.

Unison uses the efficient file transfer capabilities of rsync but it is better at detecting conflicts and it will give you a chance to decide which copy you want when a conflict is detected.

The issue though is that, like rsync, it needs to inspect all files to detect changes which prevents it from detecting and propagating updates as they happen.

The biggest issue with these synchronisation tools is that they tend to increase the risk of conflict because changes are only detected infrequently.

WinSCP

WinSCP Is an Open Source Windows GUI FTP utility that also allows you to synchonise folders between a local copy and a remote one on the FTP server.

It has conflict resolution and allows you to decide which copy to keep.

It’s great for what it does and allows you to keep a repository of your data in sync with your local copies but here again, WinSCP needs to go through each file to detect the differences and you need to sync manually each computer against the server, which is cumbersome and time consuming.

General Backup tools

There are lot more tools that fall into that category of backup utilities: they all keep a copy of your current data in an archive, on a separate disk or online. Some are great in that they allow you to access that data on the web (I use the excellent JungleDisk myself) but file synchronisation is not their purpose.

Now for some Captain Obvious recommendation: remember that file synchronisation is not a backup plan: you must have a separate process to keep read-only copies of your important data.
File synchronisation will update and delete files you modify across all your machines, clearly not what you want if you need to be able to recover them!

Revision Control Systems

Revision control software like cvs, subversion, git, etc are generally used to keep track of changes of source code files; however, they have also been used successfully to keep multiple copies of the same data in sync.
It’s actually exactly what I use for all my source code and associated files: I have a subversion server and I check-out copies of my software project folders on various computers.

After making changes on one computer, I commit the changes back to the server and update these changes on all other computers manually.

While great at keeping track of each version of your files and ideally suited to pure text documents like source code, using revision control systems have drawbacks that make them cumbersome for general data synchronisation:

  • you need to manually commit and update your local copies against the server.
  • not all of them are well suited to deal with binary files
  • when they work with binary files, they just copy the whole file when it changed, which is wasteful and inefficient.

Revision Control System are great for synchronising source code and configuration files but using them beyond that is rather cumbersome.

Complex setup

All of the above solutions also have a major drawback: getting them to work across the Internet requires complex setup involving firewall configurations, security logins, exchange of public encryption keys in some cases, etc.

All these are workable but don’t make for friendly and piece-of-mind setup.

What we want from data synchronisation

I don’t know about you but what I’m looking for in a synchronisation tool is pretty straightforward:

  • Being able to point to a folder on one computer and make it synchronise across one or multiple computers.
  • Detect and update the changed files transparently in the background without my intervention, as the changes happen.
  • Be smart about conflict detection and only ask me to make a decision if the case isn’t obvious to resolve.

Live Mesh folders

Enters Microsoft Live Mesh Folders, now in beta and available to the public. Live Mesh is meant to be Microsoft answer’s to synchronising information (note, I’m not saying data here) across computers, devices and the Internet.
While Live Mesh wants to be something a lot bigger than just synchronising folders, let’s just concentrate on that aspect of it.

Installing Live Mesh is pretty easy: you will need a Windows Live account to log-in but once this is done, it’s a small download and a short installation.

Once you’ve added your computer to your “Mesh” and are logged in you are ready to use Live Mesh:

  • You decide how the data is synchronised for each computer participating in your Mesh:
    you’re in charge of what gets copied where, so it’s easy to make large folders pair between say your laptop and work desktop and not your online Live Desktop (which has a 5GB limit) or your computer at home. You’re in control.
  • Files are automatically synchronised as they change across all computers that share the particular folder you’re working in.
    If the file is currently used, it won’t be synced before it is closed.
  • If the other computers are not available, the sync will automatically happen as they are up again.
  • There is no firewall setup: each computer knows how to contact the others and automatically -and uses- the appropriate network: transfers are local if the computers are on the same LAN or done across the Internet otherwise.
    All that without user intervention at all.
  • Whenever possible, data is exchanged in a P2P fashion where each device gets data from all the other devices it can see, making transfers quite efficient.
  • File transfers are encrypted so they should be pretty safe even when using unsafe public connections.
  • If you don’t want to allow sync, say you’re on a low-bandwidth dialup, you can work offline.
  • The Mesh Operating Environment (MOE) is pretty efficient at detecting changes to files. Unlike other systems, in most cases it doesn’t need to scan all files to find out which ones have been updated or deleted.

Some drawbacks

  • It’s not a final product, so there are some quirks and not all expected functionalities are there yet.
  • The Mesh Operating Environment (MOE) services can be pretty resource hungry, although, in fairness, it’s not too bad except that it slows down your computer’s responsiveness while it loads at boot time.
  • You can’t define patterns of files to exclude in your folder hierarchy.
    That can be a bit annoying if the software you use is often creating large backup files automatically (like CorelDraw does) or if there are sub folders you don’t need to take everywhere.
  • The initial sync process can take a long time if you have lots of files.
    A solution if you have large folders to sync is to copy them first manually on each computer and then force Live Mesh to use these specific folders: the folders will be merged together and the initial sync process will be a lot faster as very little data needs to be exchanged between computers.

Bear in mind that Live Mesh is currently early beta and that most of these drawback will surely be addressed in the next months.

Conclusion

I currently have more than 18GB representing about 20,000 files synchronised between 3 computers (work desktop, laptop and home desktop) using Live Mesh.

While not 100% there, Live Mesh Folder synchronisation is really close to the real thing: it’s transparent, efficient, easy to use and it just works as you would expect.

Now that Microsoft has released the Sync Framework to developers, I’m sure that other products will come on the market to further enhance data synchronisation in a more capable way.
In the meantime, Live Mesh has answered my needs so far.

References

3 comments January 19th, 2009

Sysadmin: Macbook Pro, after the honeymoon

security01.pngI’ve been using the MacBook Pro I introduced in my previous blog entry for a few weeks now.
Between love and frustration I hang…
Here is a review of our relationship so far.

The Great

Hardware delight

Whether running OS/X or Windows 2008 I’ve got no major complaint about the performance of the machine. It’s fast, stable (except sometimes it’s not waking up from sleep or it does but the screen remains black). The screen is nice and vibrant, I just love the magnetic power connector and the small size of the power adapter. I have a few complaints though, see below.

OSX battery Power usage

For such a large and powerful laptop I’m pleasantly surprised with the duration of the battery under OSX: I’ve been able to watch videos for 3h, full screen, without trouble and overheating (although I would lower the screen brightness to reduce consumption).
I haven’t had such luck under Windows 2008 where I’ve been struggling to find the right power settings balance, but remember that’s a server OS and it’s not really meant to be run on a laptop.

The Ugly

The mouse

You wonder why Apple, in all its hardware expertise could design the mighty-mouse with a single big button that can still do right-clicking but can’t give us the same thing with the enormous single-button of the trackpad.
Now the new models -just released this week- have done away with the button entirely, which may be just as well although I’m curious about how well the drivers will work under Windows.

Mouse acceleration in OSX is pretty frustrating to me. When you’ve got a large screen, you’re endlessly shuffling the mouse to get that pointer in the right place. It feels slow, inaccurate and is extremely irritating after a while. The problem is even worse when you’re working in OSX under VMware Fusion: while it might still be usable under OSX, the difference is really severe and unnatural in Windows.
This does not happen under bootcamp though where mouse acceleration behaves as you would expect (for windows).
I’ve tried a number of utilities (iMouse , SteerMouse and others) but none gave me what I needed.

The keyboard

The keyboard feels great to type on, with a nice spring and softness. There are a few issues though:
Why is the Return key so small compared to the right Shift?
The rule is that the more a key is used, the bigger it is, yet the Enter is rather small, it’s the same size as the caps lock which serves almost no useful purpose in comparison.
The arrow keys are also minuscule, another probable example of Apple choosing form over function. The lack of delete key forces you to type both the fn + backspace keys instead, which is an unnecessary pain, it’s not like understanding the difference between delete and backspace is that confusing to people using a complex machine like a computer.

The lid

I love the way the hooks for the lid come out just when you close it. It makes for a neat screen without protruding bits of metal or plastic.
My main issue with the lid is with its limited opening angle: if you’re just a little tall and you place your laptop on your lap then there is no way to open the screen enough to have it properly face you.
This is also an issue if you place your Macbook Pro on a cooling pad or a riser that’ll put the laptop too vertical (for instance to free some space around it when using an external keyboard): you just can’t use these devices.

The sound

That one really makes me mad: the MacBook Pro has audio issues that you won’t even find in a US$15 MP3 player and it’s totally unacceptable.
When idle, I can hear hissing sounds that vary in power and frequency if I slide the volume control; when playing music, there is a lot of noise and “cutting” sounds between songs.
These are not noticeable when using the integrated speakers but,they become really annoying once you use earphones.
I am sorely disappointed with sound quality to say the least.

Electronic noises

On top of the annoying sounds from the sound chipset, the LCD inverter also makes a hissing sound that increases in volume when I lower the LCD brightness…
Coupled with that, the processor also makes a hissing noise when it gets into its C4 power saving state…
The noise is probably not high enough to get the laptop fixed but it might be great as a mosquito repellent.

Boot ‘song’

Apple knows best and they know that your only aim in life is to become a poster boy for the brand.
When booting/rebooting your mac, it likes to play its welcome song that says “hey, over here, I’m a mac and I’m telling the world I’m booting up. Everyone listen how awesome I am!”.
The perverse thing is that even if you have earphones plugged in (as in: you don’t want to disturb people around you) the boot song will be played on the speakers…
Of course, there is no option anywhere to disable it.
Apple knows best.
After much trials, I found that booting under OSX, lowering the volume to zero, then rebooting under windows and changing the volume there would be OK: no more booting song -at least for now- and I can still change the volume in OSX and Windows.

Conclusion

Would I buy a Mac again knowing what I know now?
Mmmm, probably not.
I find the annoyances a bit too much relative to my expectations and my usage scenario. To be fair, it’s not all bad and most of the time I’m happy with my mac but I find myself trying to avoid the things that infuriate me and it’s not really what I want from a laptop; it’s supposed to liberate me and give me the freedom I need to get things done, not get in the way.
Re-adapting to a strangely layout keyboard, having to deal with Apple’s brand awareness arrogance, battling with stuff that you just normally take for granted, all this is a bit too much of a price to pay for what is essentially for me a Windows development machine.
I prefer to waste my time actually getting things done rather than searching forums on how to keep Windows and OSX time in sync or bring back my machine for repair if a CD stays stuck in the CD Drive.

So, let’s say that our relationship is more ambivalent than needed to be.

References

Other links to pages of interest.

2 comments October 19th, 2008

SysAdmin: Installing Windows Server 2008 x64 on a Macbook Pro

security01.pngMy trusty old gigantic Sony Vaio is about 4 years old. It served me well and still works but it’s about to become my main development machine for the next couple of months and I can’t afford to have it die on me during that time.
It was time to get something as gigantic and more up-to-date in terms of technology.

I use VMware on my main desktop to keep multiple OS setups that match typical configurations of my customer’s machines.
This allows me to test my software before deployment and make sure everything works as expected. It saved me many times from strange bugs and I would consider these final tests to be a mandatory step before deployment.
My old trusty vaio would be hard pressed to run any of these without slowing down to a crawl.

I looked at some possible replacements. Initially I checked Lenovo’s offerings but they don’t seem to offer anything in large screen size (WUXGA 1920×1200) (Note, actually, they have, but not really for me).
Dito for Dell, not counting their humongous XPS M1730 luggable gaming machine that was wayyy over the top as a work computer, not to mention probably heavier than its volume in pure gold.

On a hint from a friend I checked out Apple’s online store and saw they had a nice Macbook Pro configuration. I went to check it out in the retail store close to my office and they had that exact specification in stock, so, in what must have been the highest rated expense/time-to-think ratio of any decision I ever took, well, I bought it…

The spec, some bragging rights:

  • Macbook Pro 17″
  • Core Duo T9500 2.6GHz processor
  • nVidia 8600M GT 512MB graphics card
  • 200GB 7200rpm drive
  • Kingston 4GB DDR2 667MHz RAM
  • Hi Resolution 17″ 1920×1200 glossy screen

It’s a very nice machine, Apple knows how to make nice hardware, there is no question there.
OSX has some cool features, some of them still a bit foreign to me and some minor annoyances are creeping up, like Thunderbird’s not picking up my system date and time settings and displaying the date in the wrong format (a pet peeve of me), probably not Apple’s fault but annoying nonetheless.
So far so good and while I don’t mind using OSX for my browsing, email and creative stuff, that machine is meant to be running Windows Server 2008 x64 as a development platform.

Why Windows Server 2008 x64?

Well, it has some excellent features, a smaller footprint than Vista, all the aero eye candy, is apparently noticeably faster than Vista and has none of the nagging security prompt (you are considered administrator though, so keeping safe is entirely up to you).
The 64 bit version can also address the full 4GB of RAM without limitation and all server features are optionally installable.
By default, the installation is actually pretty minimal and you have to set services and options to get Windows configured as a proper workstation. It is after all, meant to be a server.
Oh, I almost forgot that there is also support for HyperV, although you must make sure you download the right version (if you list all available downloads in your MSDN subscription, you’ll see some that are explicitly without that technology).

Installing Windows Server 2008 x64 is remarkably easy.

  • Get your hands on the ISO from your MSDN subscription or an install DVD from somewhere else (like a MS event, or even as a free 240 days download from Microsoft).
  • You’ll need to repackage the ISO as it won’t work properly (something to do with non-standard file naming options).
    It’s fairly easy if you follow the instructions from Jowie’s website (cached version): you can get the ImgBurn software for free as well, which is a good find in itself. It should’t take more than 30 minutes to repackage the DVD.
  • In OSX, go to Applications > Utilities > Boot camp and follow the instructions on screen.
    You will be able to resize the default partition by just moving the slider. I left 60GB for OSX and allocated the rest to Windows. The good thing is that OSX can read Windows partitions, so you can always store data there. Windows however, can’t read the HFS+ mac file system, although there are some third-party tools that can do it [1] [2] [3].
  • Insert your repackaged DVD and Bootcamp will have rebooted the machine.
    After a few minutes of blank screen (and no HDD activity light to let you know something is happening), windows setup launches.
  • You will be then prompted with the choice of partition to install to.
    Select the one named BOOTCAMP, then click the advanced options button and click format. From there one, windows will install everything, then reboot, then carry on installing, then reboot one last time.
  • Now, insert your OSX recovery CD 1. It should automatically launch the driver installation.
    Once done, you’ll reboot to a nice, full-resolution windows prompt.
  • All drivers will have been installed correctly except the one for Bluetooth. To easily solve that issue, just go to Spencer Harbar’s website and read how to install the Bluetooth drivers. Takes 5 minutes tops.

The final touches

A few notes to quickly get things running as expected.

  • Get the most of your configuration by following the list of tweaks from Vijayshinva Karnure from Microsoft India.
  • There are more tweaks, and even more tweaks available as well (don’t forget to enable Superfetch).
  • Microsoft has a whole KB entry on enabling user experience.
  • In the Control Panel > System > Advanced System Settings > Advanced > Settings > Advanced > Processor scheduling, set to Programs instead of Background services.
  • Activate your copy of Windows using Control Panel > System.
    I was getting an error code 0x8007232B DNS name does not exist error. To force activation, just click on the Change Product Key button and re-enter the same key you used during install.
    Windows will activate straight away.
  • When booting your Macbook, press the Option key and you will be presented a list of boot choices.
  • You can check on Apple’s Bootcamp webpage other information about how to use the track pad, keyboard layouts etc,

References

22 comments August 31st, 2008

SysAdmin: When your computer becomes forgetful

BusinessSometimes your computer crashes without reason. It happens at any time, for no particular reason.
Other times you’re trying to install a new OS on a brand new PC and at some point, it fails, reboot itself or just hangs.

A couple years ago I had this really depressing experience with a brand new system I was building. All the components were newly bought, but installing the OS (a Linux distro) used to fail almost at the end of the process.
No matter how many times I tried, I could never get to the end of it.
By clever subterfuge I managed to get it installed only to have it crash on a regular basis.
It was unstable, unreliable and after two days wasted banging my head against the walls, I gave up…

Memtest86 Well, no for too long. A flash of inspiration came to me and I popped in the install CD and ran the only utility that was available at the prompt: memtest86.
That simple tool is a godsend (if I may appropriate the word from the believers. I promise to give it back). After running its various memory tests for 10 minutes it reported errors in one of the RAM sticks installed on the motherboard.
All that aggravation for a puny bit that was not remembering its state…
I promptly returned the RAM and tested the new one for a few hours until I was confident there was no issue with its chips and went my merry way to install the OS I so desperately needed. All went without a hitch.

So my advice is: go to the memtest86 website. Burn the bootable ISO and test your PC from time to time, especially if you have strange intermittent issues that you can’t pin down to a simple software problem.
You’d be amazed at the number of times I had to ditch a stick of RAM… memtest86 saved by sanity countless times.
By the way, consider donating a little bit if it helped you too. That’s always cheaper than a session with a shrink…

Add comment September 20th, 2006

Software: Cheap Microsoft Licenses

BusinessI’m not particularly pro -Microsoft but I’m not against it either.
I love Linux, got my RHCE (Red Hat Certified Engineer) a bit more than a year ago and I love Open Source, Linux and all things GNU.

The only thing I really dislike about Microsoft is its marketing, its pricing, its Genuine (Dis)Advantage that nags me every time I need to install something and its lack of openness when it comes to inter-operability with other competing implementations (here I’m particularly thinking about its network protocols that the Samba team tries to decipher and re-implement as an Open Source platform).
On the other hand, Microsoft is made of great programmers, great minds that you can watch on Channel 9 and read on their insightful blogs.

Microsoft is really (I mean REALLY) pro-developer: they understand that the Operating System alone is nothing without lots of applications sitting on top of it, and they offer developers a lot of goodies.
One such useful programme is +Microsoft Empower for ISV_. It’s a simple membership that allows a small software company (ISV meaning Independent Software Developer) to own a number of licenses for its internal use at a fraction of the price it would normally cost.
What you get is pretty wide for a small business: 5 licenses for Windows XP (whatever version), 5 licenses for Microsoft Office, 1 license for Windows 2003 Server and Exchange, SQL Server, SharePoint Portal, a MSDN Premium Subscription that covers almost anything else, including 5 licenses for Visual Studio 2005 Pro.
You also get access to MSDN downloads, beta software and tons of libraries, SDK, etc. A MSDN subscription alone is about US$2000…
The Empower ISV programme is quite cheap and depend on the country you are in.
I paid mine HK$4,260 about US$530.

To get all this you need to register as a Microsoft Partner (that’s free), then apply for the Empower for ISV programme by making a promise to release a commercial software within 2 year at most. You need to give some details and pay your due. After a few days, you get confirmation if your application is accepted or not.
As far as I know, you need to be a company and have a company website, but that may not be mandatory in all regions.
The software you get is the normal US version plus whatever local version there are for your region. My bunch of DVD came in Japanese, Cantonese, Mandarin, some in Korea and Thai too.
At least, the English version is supplied. For some software, you also get the multilingual version that include European languages as well.
You manage your licenses by login under your MSDN account.

Really, it’s a nice touch from Microsoft to give us poor developers access to all this for such a reasonable price. I can only encourage other small software companies and independent developers to do the same.

Add comment September 18th, 2006

All sites won’t work with IE7

technology01.pngI installed a beta of IE7 last week and was quite pleased at the new interface and tabbed browsing, which has been available to Opera and Mozilla used and many other browsers for years. However, looking good and being slick and more standard-compliant hasn’t been enough to keep it on my machines.

Continue Reading Add comment May 31st, 2006

SlickRun: Desktop command-line launcher

technology02.pngIf you feel you often have to interrupt your typing to move your mouse around and click on it to open applications or websites, the SlickRun is the perfect power toy for you.

It’s just a small application that can either stay in your tray until you call for it or that can display a small semi-transparent prompt where you can type in configurable commands that it will launch.

Continue Reading Add comment May 28th, 2006

Microsoft Office 2007 beta 2

technology02.pngI received an email from Microsoft updates this morning and there awas a link to download the latest beta of Office 2007.
After registering, I got a few files including the Microsoft Office Professional Plus package at about 440MB and proceeded to installing the beast.

Continue Reading 1 comment May 27th, 2006


Most Recent Posts

Categories

Links

Posts by Month