Sunday, March 25, 2007

Breaking up large files for transport to S3

I decided to try and place a copy of my windows XP virtual image on S3 for backup. I quickly realized that JungleDisk has a file size limitation and so does S3. Probably not a bad idea since transferring a large file from home or office will most likely be limited to 384k or 768k up. For a 30G file, this would take a while. If we break the file up and something happens to the connection, it will be much easier to start over the next time.

I looked for an easy way to split the files up and stumbled across SPLIT. This will take an input file and break it into pieces based on info provided. See the MAN page for details.
Here's the command I used:
split -b 102400k winxp.hdd.aa winxp.hdd.part

This broke my file up into 100M chunks. Now I can load them to S3.

If you ever need to re-construct the files, the cat command will work well. You'll need to CAT them together in the appropriate order but that shouldn't be difficult given the way we split them up. For example, to create the original image, you would simple type
cat winxp.hdd.part.aa winxp.hdd.part.ab ...

Could this be any easier?

No comments: