How to verify the integrity of thousands of CR2 files

[FTb-n]

Anyone know of software that can scan a directory (and sub-directories) of CR2 files to verify file integrity?

After downloading images to my computer, I back them up to two external drives then spot check the folders to make sure all is well. Then, I run through the photos in Photo Mechanic to mark those worth processing in Lightroom. After at least one viewing, I reformat the card(s) in camera for the next shoot.

On occasion, I've found a corrupted file in Lightroom that looks fine in Windows File Explorer or PM. As I understand it, each CR2 includes an embedded JPG which is displayed on the camera's LCD and used by Windows for thumbnails. On occasion, I've found corrupted images in Lightroom that were not apparent in PM or File Explorer.

Rather than importing every image into Lightroom from every backup drive, I'd like a utility that can scan a large collection of CR2 files to single out any corrupted ones. Anyone know of such a utility?

 

[Mt Spokane Photography]

Take a look at this. I am not a user, just the first hit on a google search.

Of course, any file can be damaged at almost any time. It usually happens during file transfer from the camera.

http://basepath.com/site/detail-ImageVerifier.php

Adobe DNG converter will find them as well.

 

[DFM]

Rather than search for corruption after the fact, it's far better to use a copy utility that CRC-verifies every operation. For Windows the go-to application is TeraCopy, a free replacement for the internal copy handler.

In Lightroom there is an option (Library menu > Validate DNG Files) to scan for damage within the active catalog selection, but at this time it doesn't support native raw files.

 

[Mt Spokane Photography]

Rather than search for corruption after the fact, it's far better to use a copy utility that CRC-verifies every operation. For Windows the go-to application is TeraCopy, a free replacement for the internal copy handler.

In Lightroom there is an option (Library menu > Validate DNG Files) to scan for damage within the active catalog selection, but at this time it doesn't support native raw files.

A raw file can be damaged, and still have its CRC correct. I'm not sure what type of damage he is looking for. The structure if the file can be damaged and still be written as a file with a Valid CRC, but when you open it, the image has streaks or other visual damage. DNG's do not have CRC's, but store hashes internally.

Its not so simple as just a CRC.

 

[FTb-n]

The corruption that I've found is with streaks over a good portion of the image.

I may shoot in excess of 30,000 images annually and have only encountered a handful of corrupted files. But, when I find them, it's long after reformatting and reusing the card. So, I never know if the corruption occurred in camera, during the download, during a file copy, or from some other process.

I actually have two objectives.

1. Identify corrupt files during or immediately after the initial download from the card without needing to inspect each image via Lightroom.

2. Verify files copied to a backup drive. Going forward, it should be easy to verify files during the copy process. But, this only verifies that the target is identical to the source -- which could be corrupted. I anticipate making backups of older external drives and want a way to determine the integrity of existing CR2 files before the copy.

 

[Mt Spokane Photography]

The corruption that I've found is with streaks over a good portion of the image.

I may shoot in excess of 30,000 images annually and have only encountered a handful of corrupted files. But, when I find them, it's long after reformatting and reusing the card. So, I never know if the corruption occurred in camera, during the download, during a file copy, or from some other process.

I actually have two objectives.

1. Identify corrupt files during or immediately after the initial download from the card without needing to inspect each image via Lightroom.

2. Verify files copied to a backup drive. Going forward, it should be easy to verify files during the copy process. But, this only verifies that the target is identical to the source -- which could be corrupted. I anticipate making backups of older external drives and want a way to determine the integrity of existing CR2 files before the copy.

I'd try the software on some known corrupted files and see what it does. I know it adds hash information to a file, so it can detect one that has changed, but just how good it is at finding defects in the file structure of a raw file, I don't know. Out of 100;s of thousands of files, I don't recall ever having a damaged one. I think I'm lucky that way.

If there was a error copying files from camera to computer, then a CRC check should find it. If the camera corrupted the file before the crc was added, its a total loss in any event.

 

[DFM]

A raw file can be damaged, and still have its CRC correct.

That is a vanishingly-small probability (2^-128) given how CRC hashes are calculated, and the discrepancies usually introduced by drive read/write errors.

 

[Mt Spokane Photography]

A raw file can be damaged, and still have its CRC correct.

That is a vanishingly-small probability (2^-128) given how CRC hashes are calculated, and the discrepancies usually introduced by drive read/write errors.

Believe what you want.