Taking pictures until a disaster strikes

by Wim Hultink

 

 

Have you ever lost pictures and felt this as a personal loss? For your sake I hope not.
Many of us will take thousands of pictures in a lifetime, of all the highlights, what we cherish and what we are proud of. Or maybe just recording the beauty we are seeing. We want to capture that one space in time and keep it as a fond memory. We often feel the loss of a picture is personal. We will not have a second change for taking that one image again. It is a feeling like something that is irreplaceable is permanently gone, a blank piece of memory.

This happened to me last summer while on a visit to our children and grandchildren. They are living over the ocean. Unfortunately, due to health problems of my wife, this will be our last visit together. Being an important visit, I purchased a new digital camera to record the cherished moments. I made many beautiful photos and saved them all on my laptop.

In a moment, I accidentally erased a folder that was filled with over 300 photos! They were not in the recycle bin, and nowhere to be found. However, I still had the CF card the pictures where taken on. On this card were different files, such as; jpg’s, jpe’s, RAW files and thm’s (thumbnails). Also, some files from second digital camera.This CF card was formatted, but I knew the pictures are still there on the card. Therefore, it is possible for recovery.

Fortunately, the card came with a program that claims it would be able to retrieve lost files. However I was not satisfied with this program in particular.  It took 30 minutes to retrieve 200 pictures, but still I was missing 100. The back up did not work either. The program told me there was not enough room 1 GB is needed and I have 20GB free space.
Searching on the Internet, I found dozens of programs promising to retrieve everything and your money back guaranteed if it fails. However, the small print says it is only applicable for the standard jpg and tiff files. That is a typical recovery program.  Another program promised to be the fastest, but unfortunately, it only recovers less than 100 photos. The third program wanted more money for retrieving RAW files. I was disappointed after spending money and having less then 100 useable photos recovered. A free program promised professional recovery; just to give erroneous errors and it took more than 5 hours before all possibilities were scanned for.  Many programs do not have a back up facility and back up is convenient when you are in a hurry and want to use the card again as quickly as possible.
I have tried a dozen programs before I gave up. After all, the first program seemed to be one of the best with 200 pictures. Most programs only recovered less then 100 pictures, only the standard jpg’s. But I was in need of recovering my RAW files. 
The best results were 175 to 190 usable pictures.

In general, the most common file types as jpg are recognized rather well. Recognizing the raw type seems to be more difficult, not to mention the thm’s and jpe’s. On average scanning the 1 GB card takes 6 minutes. Performing several scans before getting the best results was taking too much time. Moreover, I do have a very fast card with a very fast connection. Normally copying a full card will take a little more than 1 minute.
Only one program excelled in recovering, retrieving 284 useable pictures out of 302. However, it took 3 hours time and the program froze at the end. It was not the only program that got hung while scanning.

Most programs are recognizing the standard file types as jpg and are able to retrieve deleted pictures rather well with undelete. With a formatted card, distinguishing good and less good programs suddenly becomes clearer.
Good programs for recovering lost files are for technical investigation, the forensic research. If available at all, these programs are too expensive and too technical for common use. Even with the best results of all programs together, I did not recover all the pictures I had on the card.

In 1985, I wrote my first commercial program for data recovery and now decided to write my own program for photo recovery. I decided it should be possible to recover all the photos in less time. I succeeded and got all of my pictures back in less than 70 seconds. Although, my programming took some more time! With this program, I helped some people who were having the same problems in recovering pictures. They asked me to sell them a copy of my new program.

The program I named RecoverPlus: a more accurate and faster program. My goal is to stay on the top for accuracy and speed.  RecoverPlus has more features and is unique for a recovering program, which has a built in learning engine TP Adaptive. TP Adaptive enables to add new file types and to refine the predefined types like for example jpg for recognizing thm’s or jpe’s as well. The possible number of these sub types is unlimited. Having Canon thm differs from a Minolta thm and RecoverPlus recognizes these as well after using TP Adaptive for these types. With the back up facility, a back up of the card is possible even after reformatting. The Scan Backup is also able to scan back ups made by other programs.

Lessons learned.
Be prepared. When a disaster strikes, you probably will not have the time for research and search for the best solution. Doing the home work in advance will save time and you will know the do’s and don’ts. Having a good program for recovering lost photos and files, you are able to make a back up of the card for recovering later. After making the back up, your card is useable for taking pictures again. Therefore, having a good recovery program with back up facilities at hand and knowing what to do, you will be able to recover the most without loosing too much time for taking pictures.

 

 

Tips

1. Search for a good program in advance. In distress, you do not have the time to make a good choice.

2. A good program is able to recover the pictures not only with undelete but also even after (re)formatting the card/stick etc.

3. A good program has back up facilities: making and scanning. Make a back up and the card is useable again for taking new pictures.

4. A good program gives you the choice what types of files to recover. Disable an erroneous recovered type.

5. A good program is able to recover more than you need now. Another camera can have other types of files.

6. Practice before you really need the program. Only then, you know what to do and what you have to avoid.

7. Do not use the card you want to recover. Taking pictures or otherwise saving to the card will severely limit recovery of the former content.

8. Investigate whether or not the files are recoverable after a (re)format: format with the camera and with Windows

9. Avoid formatting if files are not recoverable after a format, but use ‘delete’ to erase files from the card.

10. Do not use SMPrep or other specific card programs if you do not really need them. They perform a full format: nothing is recoverable.

 

Technical context
Sine there are many programs designed for photo recovery, one might have the expectation that it is easy to recover files. However, it is more difficult than it looks.
Pictures are just files. The graphic programs show pictures because they know how to handle these files by the filename (the last part of the filename, the extension). The filenames are stored on disk in the directories and in the File Allocation Table (FAT) where you can find where the file physically is stored on the disk. A file can be stored fragmented, e.g. a part in sector 32-37 and the rest in sector 409-418.
Fortunately, cards and sticks are often empty before taking new pictures again. Files are not stored fragmented then. Therefore, if a new file is found, the former must be ended then.
When erasing a file, the file is not actually erased from the disk, but in the directory the first letter of the filename is changed in a token. The physical place of the file is freed in the FAT and available for a new storage. Therefore, after erasing a file, the name is partly available and the only the first sector of the file is still known. Saving a new file to the disk, the former content will be overwritten. The old filename is still there, but the content is (partly) the new file. When erasing this new file also it becomes a bit more complicated.
Deleted files are often successfully retrievable with ‘undelete’. In our case, the first file is not retrievable because the second file overwrites it. But this second file is recoverable. The kind of file is derived from what is left of the name and the first sector is known. With the information about the other files in the directory, the end of the file becomes an educated guess.

Much more complicated recovering becomes after formatting the medium. All directories are erased now and there is no information left.
With a full format, the whole disk is written with numbers and there is nothing recoverable. However, a full format takes much time. Therefore, most cameras and most versions of Windows are performing a quick format, only erasing the directories and the FAT. The files remain on disk. However, how much and where, the beginning and the end and what the kind of file is unknown. All this information is wiped.
There are no storage conventions for the kind of file, nor the beginning or the end of a file. In fact, they are not stored in the file itself so that is not a possible solution for recovery. However, often each kind of file has his own rules or habits and each camera manufacturer has his own logic for the raw formats. For example, most jpg files begin with the hexadecimal FFD8 bytes, but not all of them! There are over a dozen different types of jpg. Moreover, FFD8 can also occur somewhere in another type of file. Then less accurate programs will show more files than there are, useless and degrading the number of possible recoverable files too.
And how are Nikon, Pentax and Kodak raw files distinguishable when they all have the first six bytes in common? Explainable that a manual stated that in these cases the user has to change the extension in the proper one when recognition is erroneous.
There has to be more for accurate recovering and that is finding more specifics in the file itself, a fixed sequence. The place of this sequence can differ for each file. The fixed first bytes and the found sequence together are signature for that kind of file. TP Adaptive is able to analyze files and to determine the signature for the kind of file. It is a powerful tool for adding new file types or for refining existing types with sub types. The jpg type can now be refined to a thm and jpe for example. The benefit is increased accuracy and avoiding necessary renaming of the recovered files. The more files TP Adaptive can analyze the more accurate it becomes.

Finding the beginning of a file is one thing, the end is another matter. Except for the bmp type, the end of a file is not stored in the file itself, nor marked at the end of the file. When files are stored sequentially and not fragmented, the end of a file is found by the beginning of a next file. Therefore the bottleneck stays the accuracy in recognizing all kind of files and the proper beginning of the file. It is here where most programs show failures. Some programs do not recognize some types of jpg or the more specific propriety formats like thm, jpe and the raw types. When a program does not recognize a kind of file, it will show bigger files of the recognized types than expected. Except for not finding all possible, this is not severe. Most graphic programs simply ignore the added bytes. If errors occur, try to load the file with another graphics program. If you succeed, save a copy with this program and this copy will be useable for any program.