Keeping Your Photos Safe and Secure

Introduction

No photographer, either amateur or pro, wants to suddenly lose his works. Just because a "simple" HDD failure hasn't happened already, it's not a guarantee that it will never happen. In order to minimize the impact of such an unfortunate event, one has to consider backing up photos, and also storing them in safer ways.
Before proceeding to enumerating backup/storage solutions, here are a few general starting pointers.

Important Note: Too many people do not make a distinction between backup and storage. They are not related. Storage is something where you store your pictures/data, whereas backup is for "In case something really bad happens...". Any reasonable storage system has two independent systems connected by a backup policy, one is storage, and other is backup. The ability to recover data is strictly depends on the backup policy.


Types of Defence Against Data Loss
1. Backup: A backup is an additional copy of the data in question. For it to be useful, it should be resistant to whatever might cause the original copy of the data to be unusable, such as; fire, theft, system failure, terrorism, etc.. A lot of people seem to think that burning files to CD's and then removing them from their hard disk is a backup. It isn't.
2. Redundancy/fault-tolerance: Redundancy refers to the ability of your storage system to withstand common failures without data loss, and preferrably without downtime. This could be a live mirror/array, or it could be some other copying mechanism.
3. Archival/snapshots: Historical copies of the data, allowing you to recover items that might have been deleted or modified inadvertently at some point in the past.


Common Failures In Storage-Backup Process
1. Backups aren't actually running, or aren't being tested to ensure that they are working. Or... No testing is being done to make sure that the data can be restored(!)
2. Backups are being stored too close to the original data, so that fire, theft, acts of God, etc. can destroy the original and the backup(s) in the event.
3. Backup doesn't have enough history to be useful when information is needed that was lost less recently. (i.e. having only a single backup that is overwritten nightly as opposed to backups for each night of the week, each week of the month, each month of the year, to the degree necessary for each situation). If you delete/harm something and don't notice for a week or more, can you get it back?
4. "The backup fairy", No one knows what is going on with the backup, what's being restored, etc... They just assume that the backup fairy is coming at night and making sure everything is OK.
5. Systems are configured so that a simple common failure like a dead hard disk, or power surge are able to destroy the data. Having a RAID-1 system without having an electrically and moderately geographically disconnected backup is an example of this. RAID is great, but its part of a bigger strategy and isn't a substitute for a proper backup.


Storage and Backup Solutions
1. Optical Disks
CDs and DVDs are probably the most known means of data backup. However, they are losing ground to other storage options, mainly because of the increasing number of pixels in cameras. Nowadays, a single raw file can have up to 16-18MB and post-processed files go even higher. In these conditions, a collection of 700MB CDs or 4GB DVDs can easily accumulate a few dozens (if not hundreds) of discs in time.
Altough the BluRay DVDs offer a higher capacity, they are still quite expensive. People are more inclined to buy a larger HDD, rather than a couple of these discs.

In general, optical discs certainly aren't the safest way of storing information for a very long period, as they degrade over time, causing data loss. However, writable discs are cheap and easily portable. Used in combination with another storage method, they can be a very useful backup option, especially for files of smaller size (like JPGs).

2. External Hard Disk Drives
External HDD drives are becoming more and more popular for data storage and backup, due to their large capacities and their portability. A 250GB drive can roughly store the equivalent of 360 CDs or 80 DVDs, but has only a fraction of their physical volume. Another big advantage is the ability of keeping the HDD drive separately from the computer that stores the main copies of the files. So, in case of a disaster that may hit either the computer or it's internal drives (consider here anything from HDD failure, to theft or a big fire and flood), files can be recovered from the external drive that was kept at a different location.

3. RAID Arrays
RAID (Redundant Arrays of Independent Disks) is probably the best way of protecting your data from hard disk drive failures. As the name suggests, RAID's principle is using extra disks to store redundant data which is not needed for normal operation, but in case a failure should occur, this redundant data can be used to construct the remaining data. As HDD prices are decreasing constantly, RAID configurations are becoming more affordable to everyone. There are several RAID types, of which RAID 1 and 5 are probably the most used (for detailed info on RAID types, see RAID on Wikipedia).

~ For photo storage purposes, RAID1 should be the best way to go. This type of RAID is also called "mirroring". It requires an even number of drives (typically 2 or 4), and mirrors one half of the drives to the other, so half of the total capacity is lost. However, the array can survive the loss of either half of the drives. A convenience is that it is usually possible to update an existing drive to RAID 1 by adding another disk of at least same size.

~ Some external storage boxes use RAID 5 to provide fault tolerance. This mode of RAID uses 4 disks, but only 3 disk-full of space is available to the user. Remaining space is distributed among 4 drives and used for redundant parity data. RAID5 systems can survive the loss of any single drive in the system. If one should fail, the data in it can be stored on a replacement disk and the array can regain it's initial state.

An often encountered computer setup comprises a single, fast HDD used for the Operating System, together with a RAID1 system of two or more disks, used for storing imporant data (photos in our case).
However, it must be said that RAID is only a storage system. It does ensure local data safety, but IT IS NOT a backup unless multiple RAID systems are used for storing the same data. Also, while a RAID1 provides safety against a single drive failure, it does not help when the whole computer is fried. It won't prevent a virus or misbehaving application from wiping out your photos or otherwise corrupting the file system. And RAID is still prone to human failure, i.e. if you delete your files yourself it will offer no protection.
On the other hand, if a RAID setup is used in combination with CD/DVD or external HDD backups, the risks of losing photos are greatly decreased.

A thing to consider with RAID is the RAID controller itself, if such is used. For a proprietary solution it may be difficult to obtain a compatible controller in the future (e.g. if the card breaks). Although the meaning of RAID levels is consistent across manufacturers, the implementation (how the data blocks are laid out on the disks themselves) are not. In other words, don't expect to be able to take your RAID disks from one controller to another. This can have some serious consequences if you need to recover data from your disks after your controller dies. In this regard, software RAID is a safer solution. Many of the so-called RAID controllers do it in software anyhow, with perhaps a small BIOS implementation to allow booting from a RAID array other than a mirrored one.

4. Dedicated Storage Devices
Demand for RAID systems initiated the launch of dedicated external storage solutions. Although they can be more expensive than user-built internal RAID setups, they are valuable because of their portability (as opposed to internal storage). Usually, RAID 5 is the type of array used in building these devices. It's main advantage is that it uses parity across disks. Thus the array can survive loss of any single drive (at the cost of one drive's worth of storage space) making it probably the most secure RAID system. Read speed will be extremely fast, however write times are prolonged because of the calculations that have to be made.

Here are a few links to such dedicated systems:
~ Buffalo Terrastation
~ Drobo Storage Robot
~ Intel Storage NAS
~ Synology products

5. Online Storage
Another good option for storage/backup is online services that offers storage on their servers. You have can have your files stored online either by using a photo-sharing service (like SmugMug or Zenfolio) or an online file storage service (either free offers like 5GBfree or paid services like ElephantDrive).

A photo-sharing service is good because it also offers the possibility of showcasing your photographs. For a rather small yearly fee they offer unlimited storage space. The downside is that not all of them accept RAW files.
The online file storage services usually come at slightly higher prices than photo-sharing offers, but they are much more versatile, as you can store any types of files, and can even design your own photography website.


Storage and Backup Tips
~ Keep your files live. Do not remove them from your hard disk system. Expand your storage as needed to keep everything live. You want your photos accessible at any time, right?
~ Test your backups to make sure that you can restore them. What good are they if you can't? Don't wait to find out.
~ Burned CD's and DVD's have a nasty habit of not working when you need them desperately. It's a good idea to burn a copy for each CD/DVD, to have a spare in case the original disc becomes unusable. Refresh the entire sets at least yearly. Take them offsite ASAP.
~ Make sure that your backup can't be killed in the same event that kills original files. For example an external hard disk can be killed in an electrical event, fire, theft, etc. along with the PC. So can be the RAID devices.
~ External drives connected via USB have a weak spot: one might accidentally interrupt cable connection while write operations are working.. this WILL cause immediate data corruption for sure. So be careful with that USB cable.
~ An external HDD is best used as a backup option, rather than simple storage. These drives can fail as easy as an internal HDD. If the photos are not backed up somewhere else, they will be gone forever (if data recovery is not possible). If at all possible, they should be connected to a computer only when they are used to copy data to/from them. An electrical powersurge that can fry your PC will most likely fry the external drive connected to it as well.
~ If you are going to use RAID (well, you should, its a good idea) please make sure you have good voltage regulation and uninterrupted power on the line. Shortly, get a UPS. If a voltage jump can fry a HDD, it can fry two HDDs too.
~ For extra safety, photos can be stored with recovery data, which can be generated using par2 for example.
~ A supplementary boost in protection levels can be gained by using multiple computers, situated in different locations, synchronised on a regular basis. A common tool for synchronizing data between separate computers is rsync (primarily a Linux application, but it can be used on Windows systems as well, together with Cygwin)


Planning Your Backup Policy
~ The hardest decision of them all is the Backup Policy. It is easily the place where most errors are made. With such errors, you won't like the results in hindsight. Please follow the steps:
1. Decide what to backup.
2. How much data are you willing to lose?
3. How quick the recovery is desired?
4. How do you find what to recover?
5. Should the recovery be destructive?
6. Should backup be absolute? (Everything on the storage as it is now)
7. or should it be incremental? (only changes from today).

Know The Answers Of These Questions For Your Backup System
1. Does it work? (Test backup, and recovery)
2. What failures can it handle?
3. How you can recover after a failure?
4. If disk(s) fail and you can recover or continue to use your system, how do you get your system back to its initial state? (for instance: replace damaged disk, and auto build the disk, etc.)


Final Note: One needs to keep in mind that at a certain point of effort for backups (eg triple, quadruple redundancy), the risk of getting hit by a car on the way to the post office for picking up the next HDD is bigger than losing any data due to a certain type of failure. In this situation, any efforts to increase your protection against these certain type of failures are in vain.

(continued in next post)

Edited by Turerkan - 31 December 2007 at 17:09

Portable Storage Devices
The constantly increasing number of MPs in our dSLRs do not affect only home-storage devices of smaller sizes (like CD/DVDs) but also the CF/MS cards in the cameras themselves. Although cards nowdays come in extended capacities (8/16GB CF cards aren't "exotic" anymore), when a photographer goes on a longer holiday, he is often forced to bring along his laptop, if not for editing shots, then just for storing them until he gets back home.
However, lugging a laptop around is not always practical (for example on hiking trips), so one must consider other types of photo storage. Namely, portable storage devices. Most of them come with mp3/video playback support as well, while others are only used for storing files. Here is a list of the most popular devices, with links to the manufacturer pages:

Epson - Digital Photography Products
Creative ZEN
Archos - products
ImageTank - products
Wolverine - Digital Camera Backup Solutions
Vosonic - products
Next - products
Cowon - products (hard-disk type)
JOBO products - Mobile HardDrives
DigiMate III - reseller page

And in closing, a couple of user opinions on such portable devices:
- Zero's X-Drive Image Tank review

- setting up a DigiMate III with a Seagate 60GB HDD:

Michael USA wrote: I really decided that the best item to make a long term investment for my photography hobby is a GOOD storage device for trips and such. I now know that a 60 GB storage device with allow me to take as many pictures as I desire without ever having to worry about my storage needs. Now mind you.... I have four 2 GB CF cards I use. Yes four. So that is allot of storage already. But now, I can shoot as many shots as I like to get "the perfect shot"! We all know that we look at several similar shots and wish we had that one detail "just right" to make the photo perfect. Well now I can shoot the subject from every view point in a matter of minutes and not worry about storage! I will not have to spend time reviewing and deleting on the fly [walking,.."ouch", "oops", "excuse me", sorry!..&$%(*@)^$] as I would have before. Review them at home on your computer! I purchased the digitmate III shell and the Seagate 60GB drive from Meritline.com at a discount of $114.00 USD [including shipping in USA]. The shell was only $29 USD and the 60 GB harddrive was the most expensive item and the rest of the cost. I built the unit in about 5 minutes and it took another 45 minutes getting the 60GB hard drive configured correctly! Tricky, but not impossible if you are knowledgeable in computers and file formats. BUILD: All the items arrived as promised by Meritline.com. Digital Camera Mate III and 2.5" laptop Seagate 60 GB hard drive. The actual build of the Digitmate III and the hard drive was very simple and only took four screws to assemble. The instructions that are included are more than enough for anyone to follow. The 60GB hard drive fit very well into the Digitmate III. I turned on the unit and it lit up right away and the hard drive 'spun and hummed' perfectly. Whoa, good so far. FORMAT THE NEW DRIVE: Now for the interesting part...you have to 'format' the drive. You do this by hooking the Digitmate III up to your Computer via the USB cable supplied. READ this to understand how to format a new drive. It is easy to do! Don't be scared. "How to Use the Fdisk Tool and the Format Tool to Partition or Repartition a Hard Disk by Microsoft": http://support.microsoft.com/kb/255867/EN-US/ The Digitmate III runs on an OLD Microsoft file system known as FAT32. It will not operate with the newer NTFS by Microsoft. And in here lies the problem! Yes, a problem. "While the FAT32 file system can support drives up to a standard theoretical size of 2 terabytes, Windows 2000 Professional and XP Professional cannot FORMAT a volume larger than 32 GB in size using their native FAT32 file system. If you attempt to format a FAT32 partition larger than 32 GB, the format fails near the end of the process with the following error message: Logical Disk Manager: Volume size too big." Reference: Windows XP Professional File Systems Overview; http://support.microsoft.com/kb/314463 << a must read for this issue! Soooooooo, I had to make TWO partitions on the drive - since it was larger than 32 GB. I chose to make one parition the 32GB limit and the another at 23.88 GB [the remaining minus the operating system for the drive itself.] I called the first drive "P:" and the second drive "T:". "P" for pictures and "T" for transport. "P:" is the primary drive that the CF cards download onto when using the device. Now I have 32 GB for immediate storage of pictures/files on the fly and another 23.88 GB for backup. I can plug the Digitmate III into a computer via a USB port and use any part of the drive [both "T" and "P"] as a storage device or move files around as I desire. If I fill the 32 GBs, I can 'move' [on a PC via USB] some of the files over to the "T" drive to make room for more downloading. CONCLUSION: A very easy build and cheap way to have an great storage device! However, I would chose a 30 GB hard drive to save money and the difficulty in formatting if I was to do it again. That is why most of the 'simple storage devices' you see on the market are 30 GB or less!

Turerkan: Thanks to our fellow member cezarL for preparing this excellent article.


Edited by Turerkan - 31 December 2007 at 17:08

The Lime-Technology stuff sorta hurts my head. It sounds easy on the outside, but then I looked at the docs and it seems a bit complex. Just use a Drobo - same benefits, WAY less complexity.

My 2 Cents.

Having been in the Point of Sales systems/Computer industry for many years I have learned that the only way to save files is redundancy, redundancy, redundancy. My experience with popular backup softwares including those that backup registry files has been nightmarish. Backing up full sets only to find out that they do not restore, and this from a backup software manufacturer who has tens of thousands of clients, who could not figure out why their software refused to restore.   It was around the changeover time when Vista came out and I am sure that they were well aware of the problem.

RAID arrays are good, but one must consider where one lives. Never buy a house on a flood plain is perhaps the best backup advice. I recently found out that most of those writable disks that people purchased in New Orleans did not survive the Katrina flooding. One might at first think that the plastic coated disk is like a zipper bag protecting its contents, however water and oil mixed together with a toxic soup will destroy these disks instantly. A flooded basement etc. will wipe out all of them.

I have heard of 100 year disks that are apparently 'archive' grade, surviving such conditions, even my venerable batch of DVD +R (the better format to back up on) disks were destroyed on track one, by, of all things, hand moisturizer. Yes, you got it, Moisturizer! The oils in some of these penetrate the plastic layer and deposit themselves right inside the disks, clogging the reflective properties of them. If you ever see a secretary with a big pump bottle of moisturizer on her desk in the middle of December, do not let her back anything up to disk unless she wears surgical gloves!   One hand application and just touching the sides of the disk will cause the oils in the hand lotion to migrate down the outer layer of the disk just on the edges, enough to prevent the disk from ever being read again. I have yet to try this with the archival grade disks which include an extra layer of gold which apparently supports the written layer. Consider the cheapo disks self destructive within three years or less. I have seen one spin up and split in half while loading on the drive simply because of the low quality plastic that was used.

What I do now is quite simple. I purchase a hard drive, whenever I feel like it, perhaps one is on sale. I then copy everything onto it, and then put it in a zipper bag and take it somewhere else. Make sure that it is in a 'anti-static' bag first because plastic bags will shock the circuit boards, and while these can be replaced they are hard to find.

My experience with flash drives has also been one of horror, if one of those drives gets even the tiniest amount of moisture on them you can forget about whatever is on them. The moment a drive that was in a sweaty pocket or hanging on a neck strap gets a drop of rain on it, it is TOAST!

So make sure you have many, and put them in different places, like a safety deposit box.   BEWARE of the small Gorilla pods with the magnetic feet, one can easily watch someone put it on top of an external drive and wipe it out.

It looks like 'The Cloud' (as that is what they call renting servers now according to Larry Ellison) is perhaps the second alternative, but then they too would be prone to attacks by hackers who could, potentially wipe out all of their data. When banks lose millions of dollars in a few minutes, from modern day hackers, if one of these peoples kids gets on your service providers platform to experiment, he or she could quite possibly wipe out all your photos.   They cannot, however get inside the safety deposit box.

The rule in the computer industry has always been 'Grandfather, Father, Son' three disks. The Grandfather disk is kept in a safety deposit box and is updated every quarter. The 'Father is kept at another location, like a sisters house far from the kids and is updated monthly and the 'Son' disk is kept at home, and used to backup the daily events.

This redundancy ensures that even a long term virus (under three months) does not get to the Grandfather.

Newer Blue Ray DVD's are apparently coated with the same material as the 100 year Gold DVD's but whether or not one scratch on them can ruin the equivalent of 4 regular DVD's, I have yet to find out, because so far every time I look into a Blue Ray burner, I hear that it blew up after a months usage. They are, apparently, not that mainstream enough yet.

Perhaps someone will go back to tape drives. VHS like tapes have been modified for backing up large systems, however their cost is in the tens of thousands, and quite recently Paul McCartney had to redo all his masters because the tapes are prematurely wearing out (beware of this if you have Ampex or Sony SLH tapes, they must be baked for 12 hours at around 120 degrees before they can be used or the heads will scrape all the material off of them) so tape backup is also quite risky.

Automation helps me on this chore...

*on-site network server automatically backs up the processing PC's image disks daily.

*Windows-based image processing PC backs up automatically to on site external eSATA disks.

*Manual weekly backups stored off site in storage rental.

Backup, backup, backup.

As a newbie to digital photography it might sound like a daft question
but aren't memory sticks any good for storage as you can know get 32Gb for a reasonable price?

I have a 3 fold back-up plan for my photos.

1. A Drobo at home, which is basically a large external hard drive bay which holds four hard drives and stores the data in the companies proprietary version of RAID

2. My nephew is even a bigger computer geek than I am, and he built and maintains a server in his house two states away, where I also backup my files.

3 I also have an account with Smugmug to store all my photos.

So basically, short of nuclear war, I should be able to recover my photos in case of disaster