Everything and the kitchen rsync

THIS POST HAS BEEN UPDATED
Please read my new post here: http://technofart.blogspot.com/2012/02/rsync-controlled-by-python.html
There's still some relevant information, e.g. that about Online Storage Solution, so I'll leave this one up.
*****************************************

We recently realized our growing mass of critical data wouldn't stick around forever. We've just been trusting that our hard drives will continue to truck on. This is how people lose important data.

When I say "critical", I mean important to us. Music recordings, video productions, photo collections, gamesaves... So I started looking into different online storage hosts.

Web hosts who offer unlimited space are out due to their terms and conditions. The big names in the business, Mozy and Carbonite, look great as far as being quite reputable. I'd feel trusted in their hands, but they don't provide as much flexibility as I was looking for (performing custom rsync-based backups, having to update through a client, little or no Linux support, etc.). This was the case throughout the industry.

Among my digging and delving, I chanced upon Online Storage Solution. These guys are seemingly the poop of the trade. They have awful reviews where people complain about everything from lost files to receiving requests from the company to limit their storage to 500GB (despite their advertisement of unlimited storage space). One reviewer felt it necessary to purchase the domain http://onlinestoragesolutionsucks.com/ which was subsequently bought by the actual company who put it to use as a mirror of their homepage (although that mirror is not in sync and is now outdated). The most common complaint was that the service was often inaccessible where users could not download from or upload to their servers.

I graciously signed up for two years with these folks. Why? Besides $38 for two years of service, they offer the most flexibility outside of having your own server set up remotely. More flexibility if you consider that they provide a few more tools out of the box and take care of their own maintenance and network security. They run an rsync daemon, allow FTP and FTPES, WebDAV, and CIFS. They will host files publicly at your discretion. Oh, and did I mention they provide at least a limited level of support? They advertise it as being more robust, but people have trouble getting responses out of them. So I'm prepared to support myself, but there's at least someone out there.

Why would I rely on these guys if the reviews are so bad? Well, I won't. At no point will I keep data with them I don't keep myself, and certainly nothing sensitive to the end that it'd be unsafe to leave it with them. Then why pay money? Because it's another safety net at a very low price. Also, I wanted to try my hand at designing and setting up automatic backups.

Rsync = awesome. Prior to backing up files, it will scan the source and destination to determine if anything in the file structure has changed since the last backup. Then it will upload only what's needed. It also has the ability to make incremental archives, keeping copies of items that have changed and timestamping them so you can go back and find what you want from when you want.

I've been dabbling in Python for the past couple months, so I decided to look for a pre-written Python rsync backup script. This is what I found. It manages the parsing of the rsync command, handles archive rotation (deletion) after specifiable intervals, determines if it's been at least 24 hours since the last reboot (great for systems that are online sporadically like laptops), logs the transactions and alerts me by email if something went wrong. Let's go with that!

I had to make an alteration. Because it's not a local transfer, the rsync process would require a password. The script provided no way to invoke the --password-file rsync switch. So I wrote into the Python module and re-installed it. You can find my altered module here (hosting courtesy of Online Storage Solution). Just slip it into the lib/ directory of the RSyncBackup stuff before you run "python setup.py install" to integrate the module into your library thingy.

Open up examples/backup.py for a working example. It's fairly straightforward, even if you don't know Python. Look for the section where you specify the source and destination and all that. In my case, my source is local and my destination is remote. Here's what my backup.backup line looked like:

exclude = ['iPod/']
backup.backup (source="/home/", destination="rsync://username@ols10.com/username/current", archive="/archives/", excludeList=exclude, passwordfile="/root/.ols10pwd"

Oh, and just a hint on the password file, create it anywhere you like under any name. Its contents will be a single line and will only be your password (unless your password includes newlines, which I'm not sure is accepted). After created it, run

sudo chmod 600 /root/.ols10pwd

or whatever the name of your file is.

I'm currently in the process of uploading my initial batch, so I haven't automated the process yet. I'm on day 2 and so far I've uploaded ~72GB. It would've been more except for sporadic connection issues and some internal issues. But it's nice; both my ISP and Online Storage Solution (owned by iicinternet.com) are just off the Level3 backbone. My pipe to them is satisfactorily wide.

So if a crappy online storage provider can help my redundancy a little bit, I'm all for it. Good day all.