FAQ FAQ  Forum Search   Events   Register Register  Login Login

Avoiding (or easily spotting) RAW copy errors

Author
eldonito View Drop Down
Senior Member
Senior Member

Joined: 11 November 2006
Country: Sweden
Location: Norrköping
Status: Offline
Posts: 708
Post Options Post Options   Quote eldonito Quote  Post ReplyReply Direct Link To This Post Topic: Avoiding (or easily spotting) RAW copy errors
    Posted: 24 July 2018 at 08:03
Dear Dyxum community,

Sometimes my RAW files get corrupted when copying from the SD card to the hard drive, and I don't spot them until much later -- I was wondering if you had any suggestions for a simple, error-proof photo file copy in Windows 10.

My current workflow:
1) Shoot many GB of RAW files (α77 + ~5-year-old SanDisk Extreme Pro 32GB card)
2) Make a desirable folder/directory in Windows 10. I locally keep my photos are in a striped volume of four old HDD's for speed. [I also backup all files to an external NAS array afterwards]
3) Copy files from SD card to said local folder (i.e., a simple copy/paste or drag'n'drop operation)
4) Launch Adobe LR and import local files into catalog

Sometimes the copying isn't done right. I.e., the RAW file in the local folder is corrupted, but the file on the SD card is fine. It's a simple matter of re-copying the file over.

However, the RAW thumbnail is fine, so I can't tell whether any file is corrupt until I've actually opened it. And since it takes 3-4 seconds to fully open a RAW file in my PC, making sure all files are OK can be very tedious. Sometimes I don't edit the photos I've taken until way later, so by that time the SD card has been formatted and the intact file is irretrievable.

I have to say, it's a rare-ish occurrence, like, one file every 1,000 or so. But I do shoot a lot (to compensate for my camera's temperamental AF) and I will come across one, maybe two bad files per set. The last one is a shot bang in the middle of a Seattle skyline panorama.

Sssso, here's what I'm wondering:
1) Does this happen to you, at this sort of frequency? The SD card is probably fine (files get written OK), but maybe it's worth trying different card readers, HDD configurations, etc.?
2) Am I doing something glaringly wrong about handling my files?
3) Is there an "error-checking" way to copy files in Windows (ideally free?)
4) Is it possible to rebuild the thumbnails in Windows? I don't mind if it takes time as long as it gets done in the background...
5) Any other suggestions?

Thank you for reading. As always, I am really grateful for any advice.

Happy shooting!

Edited by eldonito - 24 July 2018 at 08:11
 



Back to Top
owenn01 View Drop Down
Alpha Eyes group
Alpha Eyes group

Joined: 20 May 2008
Country: United Kingdom
Location: Kent
Status: Offline
Posts: 9825
Post Options Post Options   Quote owenn01 Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 09:26
Hi eldonito,

To be honest, I've never suffered this sort of issue and my workflow isn't too dissimilar to yours in copying files over from the SD card to a specific drive on a separate HD via a Mac.

However, one line that did make me wonder was the "~5-year-old SanDisk Extreme Pro 32GB card" - is it possible that it's the SD card that is at fault and corrupting during the transfer process? Have you tried using a new card and seeing whether the fault persists?

Otherwise I can't see anything else to help you as I don't have the problem!

Good luck and it will be interesting to see others experiences on here.

Best regards, Neil.

My Mantra: "Comment on other's work as you would wish to have yours commented upon". Go on - it's fun!
Back to Top
addy landzaat View Drop Down
Senior Member
Senior Member

Joined: 22 April 2006
Country: Netherlands
Location: Netherlands
Status: Offline
Posts: 9181
Post Options Post Options   Quote addy landzaat Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 09:54
Do you only have one card? Try another one.
Do you only have one camera? Try another one if you have another camera.
Do you use a cardreader or do you use the camera as a cardreader? Try another cardreader.

I never noticed this, I use different kind of cards, usually (maybe always, but I am not that organised) with an USB3 cardreader.

Why do you think the card is okay? Just because it seems to write okay? It could be the combination of card and reader.

Like Neill I would put my money on the SD-card.

Re. thumbnails, AFAIK they are baked into the RAW-file and Windows just shows those.
Why not follow me on Instagram? @Addy_101
Back to Top
stiuskr View Drop Down
Moderator Group
Moderator Group

Joined: 01 September 2006
Country: United States
Location: West Virginia
Status: Offline
Posts: 11125
Post Options Post Options   Quote stiuskr Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 12:27
If you only use that one card then yes, you should get another just because that's the way it's done. One camera, two cards and two batteries.

You say you shoot a lot so I assume that you typically fill the card until it refuses to take another? That in itself can cause corrupted files, always pull the card when you get down to a dozen or so shots left or have another card in the second slot to carry on when slot 1 is at capacity.

The only thing I'd change in your workflow would be to upload directly to Lr or Bridge. I doubt it'd make a difference with your issue, it's just not necessary to create the folder beforehand. Start up Lr, load the card reader and in Lr click Import and then create the folder within Lr and it will open Windows Explorer for folder creation.
Rob Suits Jr.
a99M2 a99 a77 a700 KM7D|Min24/2.8 Min35/2 So50/1.4 So50/2.8 Min85/1.4G Tam90/2.8 Tam180/3.5|Tam17-50 CZ24-70G2 KM28-75D So70-200G1 So70-300G So70-400G1| SonyF60 AD200R2
Back to Top
addy landzaat View Drop Down
Senior Member
Senior Member

Joined: 22 April 2006
Country: Netherlands
Location: Netherlands
Status: Offline
Posts: 9181
Post Options Post Options   Quote addy landzaat Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 15:32
Originally posted by stiuskr stiuskr wrote:

The only thing I'd change in your workflow would be to upload directly to Lr or Bridge. I doubt it'd make a difference with your issue, it's just not necessary to create the folder beforehand. Start up Lr, load the card reader and in Lr click Import and then create the folder within Lr and it will open Windows Explorer for folder creation.
In the past Lr would place the files in the location it decided on - I am talking Lightroom 3 as I quickly stopped using it. You could set the standard folder name and location and if you imported it, it went to that location. I did not like that, as I want my folders to have meaningful names and I do not want to go into all kind of steps to change the name. Simply importing into a folder named I like, is much easier. Is this better these days?

I stopped using Lr partly because of the annoying habit of it trying to control my workflow. I found it frustrating to use, but that is probably me as I seem to be in a minority.
Why not follow me on Instagram? @Addy_101
Back to Top
pegelli View Drop Down
Admin Group
Admin Group
Dyxum Administrator

Joined: 02 June 2007
Country: Belgium
Location: Schilde
Status: Offline
Posts: 25688
Post Options Post Options   Quote pegelli Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 16:29
Addy, from the very beginning (LR 1 or 2) I created my own folder structure and set Lightroom to import into this structure as I wanted it (I determined the folder names and the "tree" and could also rename the files upon import into my own system).

So my conclusion is that it's not better these days than in the past, but what you describe and want could have been done in the past quite easily imho.


Back to the original question: I only delete the files from my memory card after:
- having opened the raw in Lightroom
- the files are in my automated cloud backup (runs daily)
- the files are on an external (USB) hard drive (runs manually before a big trip/shoot or when my card gets about 75% full)

So I only empty/format my memory card(s) periodically (every 3-5 weeks)
Mind the bandwidth of others, don't link pictures larger then 1024 wide or 960 pix high, see here
 



Back to Top
RvdM View Drop Down
Groupie
Groupie

Joined: 24 September 2014
Country: Netherlands
Location: Oud-Beijerland
Status: Offline
Posts: 79
Post Options Post Options   Quote RvdM Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 16:29
I've seen this before on a laptop pc with build in card reader which was working fine in 98% of the time. The other 2% it was giving corrupted files. The file corruption was kicking in randomly. Some times 2000 images without problems, some times errors all over. This seemed not dependent on the size or make of the card.

OTOH, I have a Sony action cam too. When I use the cable and software supplied by Sony to copy the files, I see some files which are corrupted. When I just put the card into an SD adapter and then into the card reader to copy the files manually, no problem at all.


Originally posted by addy landzaat addy landzaat wrote:

...I did not like that, as I want my folders to have meaningful names and I do not want to go into all kind of steps to change the name. Simply importing into a folder named I like, is much easier. Is this better these days?

I stopped using Lr partly because of the annoying habit of it trying to control my workflow. I found it frustrating to use, but that is probably me as I seem to be in a minority.


Luckily it is yet much better than that. On the copy/import process you can choose whatever drive or folder you want. Or just create a new folder with a name of your like.

Edit: I see that Pegelli has a different experience. Above is my experience on my Mac computer, maybe on windows it is different.
Back to Top
pegelli View Drop Down
Admin Group
Admin Group
Dyxum Administrator

Joined: 02 June 2007
Country: Belgium
Location: Schilde
Status: Offline
Posts: 25688
Post Options Post Options   Quote pegelli Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 16:46
Originally posted by RvdM RvdM wrote:


Edit: I see that Pegelli has a different experience. Above is my experience on my Mac computer, maybe on windows it is different.


I think we have a very similar experience, you can create the new folders (with your own chosen name) both in Windows Explorer, but also you can start this process within Lightroom and create the folders from there (LR opens some Windows to do it in the same manner) with the advantage that LR immediately knows where to import and you don't have to search through your tree to point it in the right direction.

So I don't think there's a huge difference between MAC and Windows in this regard.
Mind the bandwidth of others, don't link pictures larger then 1024 wide or 960 pix high, see here
Back to Top
luke View Drop Down
Senior Member
Senior Member

Joined: 09 April 2013
Status: Offline
Posts: 877
Post Options Post Options   Quote luke Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 17:34
Originally posted by addy landzaat addy landzaat wrote:

Do you only have one card? Try another one.
Do you only have one camera? Try another one if you have another camera.
Do you use a cardreader or do you use the camera as a cardreader? Try another cardreader.

I never noticed this, I use different kind of cards, usually (maybe always, but I am not that organised) with an USB3 cardreader.

Why do you think the card is okay? Just because it seems to write okay? It could be the combination of card and reader.

Like Neill I would put my money on the SD-card.

Re. thumbnails, AFAIK they are baked into the RAW-file and Windows just shows those.


I personally would put my money on the striped HDD array. Did you check that?

Good luck
L
Back to Top
luke View Drop Down
Senior Member
Senior Member

Joined: 09 April 2013
Status: Offline
Posts: 877
Post Options Post Options   Quote luke Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 17:36
I also wanted to add (but forgot) that there are copy handlers out there (like teracopy) which allow one to check by a checksum after copying. Did you try that?
Back to Top
addy landzaat View Drop Down
Senior Member
Senior Member

Joined: 22 April 2006
Country: Netherlands
Location: Netherlands
Status: Offline
Posts: 9181
Post Options Post Options   Quote addy landzaat Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 18:27
Maybe Lr is too complicated for me - I found it I only could use a pre-installed type of folder structure, I never saw a Windows explorer screen with Lr 3. Sounds like my fault.

@eldonito: it seems there are several places where this could have gone wrong. I guess you need to experiment a bit more to find the cause.
Why not follow me on Instagram? @Addy_101
Back to Top
eldonito View Drop Down
Senior Member
Senior Member

Joined: 11 November 2006
Country: Sweden
Location: Norrköping
Status: Offline
Posts: 708
Post Options Post Options   Quote eldonito Quote  Post ReplyReply Direct Link To This Post Posted: 24 July 2018 at 19:18
Originally posted by addy landzaat addy landzaat wrote:

@eldonito: it seems there are several places where this could have gone wrong. I guess you need to experiment a bit more to find the cause.


Yes, that's a good way to sum up! Thank you all very much for your suggestions.

I guess the consensus is that yes, there is definitely something wrong with one (or more) of the components in my workflow: there shouldn't be corrupted RAW files, unless something's wrong (as in RvdM's experience), and there are plenty of areas that could be problematic: old SD card, cheap card reader, striped HDD array, etc.

Since I now have an SD card almost full of photos, it's a great opportunity to troubleshoot. I'll make sure to make a backup of uncorrupted files and then try out different experiments... like pulling photos straight from the camera, trying out direct LR import, copying to another HDD, etc.

I didn't know about Teracopy -- will definitely give it a shot. EDIT: I'm thinking it would be a useful tool in flagging copy errors, so I can hopefully identify the source of the errors without having to check each RAW file every time.

Again, thank you all very much!

Edited by eldonito - 24 July 2018 at 19:22
Back to Top
Dyxum main page >  Forum Home > Equipment forums > Other photographic topics

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.094 seconds.

Monitor calibration strip

Dyxum.com - Home of the alpha system photographer

In memory of Cameron Hill - brettania

Feel free to contact us if needed.