I recently published a blog post about my open source file sync project Syncany. I explained the main idea of the project and also went into some of the details about where the development is headed. The post was the first of a series I am planning to write — showing what the project is about from different angles.

While the first post had a few technical elements, it mostly discussed the project’s process and its high level goals and ideas. In this second article, I’d like to go beyond the high level concepts and go a lot deeper into the different packages and modules of the software. Why, you ask? Because I think it might be interesting of others and because I believe that supporters and other developers will benefit from it.

Continue Reading

Many many moons ago, I started Syncany, a small open source file synchronization project that allows users to backup and share certain folders of their workstations using any kind of storage, e.g. FTP, Amazon S3 or Google Storage.

At the time of the initial annoucement of the project (May 2011), there was a big hype around it. I received many e-mails and lots of support from people around the world. People were excited because the features Syncany offers are great: File synchronization à la Dropbox, paired with storage flexibility (use-your-own), client-side encryption (sorry about that, NSA!), and intelligent versioning.

At the time, I didn’t actually release a runnable version of Syncany. The sole purpose of the announcement (on WebUpd8 and on the Ubuntu Podcast) was to get developers excited about the project in order to get help for the last steps of creating a somewhat stable release. Unfortunately, I was further away from this “stable release” than I could have imagined.

In this blog post, I’d like to recap the idea behind Syncany, what went wrong with the development, and how I brought the project back on track (or so I believe). I’ll also talk about what I plan to do with Syncany and how people can help (if they still want).

Continue Reading

I use rsnapshot to backup all of my data to my HTPC and home server (the home partition, office documents and the root file system). While rsnapshot is not as shiny as other backup tools, it is very flexible and effective: rsnapshot is based on rsync and makes hardlink-based backups (like cp -al), i.e. backups that point to the same inode on the disk if a file in consecutive backups is identical (much like SIS in deduplication).

However, rsnapshot is meant to be triggered by cronjobs and is built for always-on server machines rather than for lid-open-lid-close-type machines like laptops: That means that rsnapshot must be scheduled to run at a certain time (no retries!) and is not prone sudden system shutdowns. Furthermore, it does not detect failures and simply leaves unfinished backups as if they were complete. That in turn leads to more disk space being used for the backups, because the last complete backup is not really complete.

I wrote a little helper script to fix exactly this behavior: rsnapshot-once makes sure that (1) rsnapshot is only called if a backup is necessary (once every 24h for ‘daily’, once ever 7 days for ‘weekly’, …) even if rsnapshot-once is called multiple times, and (2) that crashed/interrupted backup runs are rolled-back before starting a new backup run.

Continue Reading

This post introduces my Master’s thesis “Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example”. I submitted the thesis in January 2012, and now found a little time to post it here.

The key goal of this thesis was to determine the suitability of deduplication for end-user applications — particularly for my synchronization application Syncany. As part of this work, the thesis introduces Syncany, a file synchronizer designed with security and provider independence as a core part of its architecture.

Continue Reading

As some of you might know, Unison is this great tool that allows bidirectional synchronisation of two hosts, – no matter which operating system they’re running… Well, at least the well known ones are supported.

Since Unison can also be used to sychronise more than two hosts, it’s perfect for big amounts of data that has to be shared in a team.

A scenario like this is possible and works for me: UserA <-> Server <-> UserB.
But of course, also other users could sync with the server. Unison rocks!

Today, after reinstalling his OS, my friend got the following error message:

Continue Reading

The good thing about the file sync tool Unison is, that it’s available for several operating systems. This is great for groups working on different systems (Mac, Linux and Windows) but want to share and synchronize files on a remote server.

Well, the bad thing about Unison on the other hand is, that its backwards compatibility is anything but great, so that you have to make sure that everybody in the team uses the same version. And this can be tricky depending on what system you are using.

My home system is Ubuntu Hardy, the remote server system is Debian Etch. Both come with Unison 2.16.13 which would be great if not Apple’s new Leopard brings the newest version 2.27.57. Long story short, I needed the newest version on Hardy and Etch.

Continue Reading