DNG vs. original RAW in the long term

[gargamel]

How do you plan to archive the photos, whichever format you select? Just curious.

Well, I am just in the researching stage... Which file format, what sort of media, cloud storage vs. local hard disk, using a database (for metadata and thumbs) or just the file system etc. I am not a pro, and not even a real enthusiast, still I have acquired several thousand images over the years. They are currently distributed over too many places (read: disks), and I never find this one image that I am just looking for at this very moment. Therefore I am preparing for a more or less big task of consolidation and data cleansing...

Most of the points I mentioned are undecided, yet. What I know already is this:

IF I go the dabase way, I'll use Digikam for that.
In any case, for the actual RAW files stored in the file system I'll use exiftool to sort and rename all files by data and time, plus a sequence number. This is a simple scheme, but I think that's an advantage, and after trying it out for a couple of hundreds of files it really seems to do what I expect it to do.
And now: File format. I'll continue to listen to the very useful advice I get here (thanks again to all!), but so far it's a clear cut: CR2, not DNG; not alone, at least.

Interesting: the automatic ad (or whatever it is) that pops up on this thread between the posts, at least for me, is for a file converter. Is everyone else getting that? I presume that whatever software Canon Rumors uses to place those ads reads the content of the thread and selects an ad that is pertinent (I also get a lot of "Canon vs. Nikon" ads).

Big Red is watching us? Reading DR related threads one would hope that they would also listen to us...

 

[Spiros Zaharakis]

My workflow keeps my RAW files for up to 6 months then I convert to DNG for Archiving purposes. I do so using Lossy compression. Doing this gives me a savings of anywhere between 50% and 80% depending on the photo. For example... for a full day wedding I typically shoot between 50GB - 75GB worth of Raw files... by time time I am done editing this could turn into over 100GB for a wedding. Converting to DNG can get my full wedding in around 25GB...

The biggest drawback here is that once you convert... you cannot convert back. And if you wish to use DPP in your workflow... your out as DPP only supports CR2 (from my understanding). If you are using ACR... I don't see to much difference as you can do everything with a DNG that you can do with a CR2. And as long as you are not creating new DNG'S from DNG's over and over... I really don't see a loss of quality issue.

Right to the point.
I use DPP as I really like the skin tone rendering but when I am done I am happy to convert RAWs to Lossy DNG for archives.

 

[nc0b]

On the other side of the coin, go back to your high school. Look at class photos on the wall from the 20s or 30s through the 60s. Likely in B&W and still fine. Then in the 70s everything likely is in color, or what is left of the color images. Faded green ghost images at my high school.

Are there negatives around to reprint these archive pictures in B&W? Not likely that anyone can find. A tin type print is still viewable today, as is most anything in silver rendered monochrome. I dry mounted B&W photos I printed in the early 70s, but was the paper acid free? I should dig those portfolios out and see what shape they are in.

As far as archival data, CD and DVD "rot" is a problem. 10 to 30 years from now the data is likely gone. Data on tape backups may be there but likely no transport to read them, let alone software that can read the tape. Besides that, tapes are supposed to be retentioned every few years. Good luck on that happening.

Hard drives that have sat around for a long time may have a head striction problem and never spin up again. Try to access data on an MFM, RLL or ESDI drive today. You likely cannot find a controller, let alone the driver you will need.

I doubt anyone really knows how long data lasts on a flash drive. Try to find someone who can still open a Wordstar file today, or Ventura Publisher. The same problem will be faced by our grandchildren, if not our children when they want to look at our digital images we cherish today.

 

[nc0b]

I meant hard drive stiction problem. (Darn auto correct). Anyway, I have seen hard drives that haven't spun up in two years have the heads stuck to the media. Even if we can plug a USB 2 drive into a USB X slot on your computer, that is no guarantee the data (images) will be readable. I keep my images on a file server, and maybe I can keep it running for 10 years, but that isn't very long compard to looking at prints from the civil war, let alone WWI.

 

[RLPhoto]

What are the advantages and disadvantages?

I've converted everything to dng lately as I don't intend to switch away from LR/ACR, it's just too good - and multiple other apps start supporting dng.

Advantages of dng:
* metadata information inside the file, no xmp sidecar
* faster rendering vs. cr2 (if you "embed fast load data")
* you can update the jpeg preview to the post-processing state
* saves 20% disk space right away (if you drop the full res thumbnail that's in the cr2)
* lossy & downsized options while retaining dr and lossless wb adjustment

Disadvantages:
* some post-processing software doesn't support it, namely dpp & dxo
* xmp sidecar of cr2 gets written faster than the whole dng when updating metadata

However, the option to convert cr2 to raw won't go away, if you're not sure you can do it anytime in the future, even with the free Adobe DNG converter.

They are always adding stuff to the DNG converters so converting them now could restrict what future developments would be possible later one. IE: Lens correction tech.

I don't understand that - you can just update an older dng version to the new one.

Unless you're using lossy dng, the "converter" just *wraps* the raw data from the cr2 into a new dng container and adds a new thumbnail and fast load data (if you request it). Absolutely no file information is lost on cr2->dng, you could even write an app to convert it back.

I checked my older 7D files that I sampled in .DNG two years ago(experimenting at the time) and they were lossy compressed. Your right, and I tested a new file w/o lossy compression and I cannot see a difference between the .cr2 and the .DNG.

But I can't help but cringe knowing that the file is smaller somehow and that the converter has to move data around from the original sensor data in the .Cr2. Then again, if we're talking long term, .DNG wasn't really accepted fully by the top camera manufacturers and has no better chance of lasting 20+ years than a .NEF or .CR2. Its still not a ISO standard yet and if I had to choose between the two Multi-national billion dollar companies like Adobe or Canon, I'd chose Canon way before Adobe. The way Adobe has been using their software as leverage to me means that the yet adopted ISO standard .DNG is not free and clear from their control. If anything, its giving more control to adobe because to get the best performance, adobe software will be the best, thus sell more product, and if you have all your catalog in .dng, have fun trying to move to another software workflow. In a way, it could turn into an AdobeRaw format per say.

Another note that some photo competitions require the original raw and a converted .dng could be brought into question. Even though not very likely to occur but another reason to stray away from .dng
http://photo.net/casual-conversations-forum/00WQjJ

Long term means trust, and Nikon/Canon has been around alot longer than Adobe.(which still makes the best .dng converter around)

 

[Aglet]

hard drives are cheap; 6TB for <$200
I store everything in native raw formats + some editing intermediaries for things I want to re-process.
If I like the finished results, then a low-compression jpg at full rez will do just fine for archiving the finished image. I save the sidecar with the raw and toss the intermediaries.
DNG probably loses too much OEM-specific meta data and I don't like that it's develop data is embedded; it's convenient, but risks corruption.
I don't trust optical media for long term, I keep migrating everything to newer HD tech as it comes along, fortunately, that's not very often. last big move was from SCSI and IDE drives to SATA.

for long term, I also archive some software AND the machines required to run it. Been doing that since the Mac Plus, and still have one that runs.

 

[hgraf]

what happens 20 years from now? the software that processes these files will not run on anything available then.

False. Completely false.

Let's look at history here. Consider a computer that's >30 years old: the Commodore C64. You can STILL run pretty much any software that ran on that platform using emulators. And the C64 was WAY WAY less common and popular then windows and x86.

Even if a computer platform 20 years from now won't be able to run windows and x86 software, it will have an emulator that will. Even android has DOS emulators today.

The conundrum gets even worse if you think 100 years from now. Will there be any devices that can read our USB drives that far into the future? We have family images created 150 years ago and while not necessarily in greatest of conditions, we can at least view them. How do we archive for our grand kid's grand kids?

Easy, do what I do: live backups.

While I have offline backups using HDs at the moment, everything I have is also in live backup form, meaning a platform that is current for the day. At the moment it's 2.5" hard drives, which is a migration from DVDs. IN the future? I'll simply migrate it to whatever format comes next.

Is it work? Yup. But what in life that's worth doing not worth a little effort?

 

[hgraf]

Software doesn't "disappear", it may no longer be updated, but you'll always be able to find it on the web.

How about this, if you're REALLY worried about it, archive your RAW files WITH a piece of software that does the conversion.

Yes, but that alone won't help. As has been said before, you need an OS and hardware to run such software on. I have a low motivation to put a 32-bits Windows XP machine, that takes ages to boot, in a safe, with a copy of DPP, just to make sure...

Yes, it will. Just because your current machine might not natively be able to run the software, doesn't mean it can't run at all. Emulators for pretty much every platform in existance can run on current machines, often with more performance then the original hardware! Getting a win32 XP emulator 50 years from now is pretty much as guaranteed as you can get.

 

[AcutancePhotography]

I am lucky. I don't imagine any photograph I take being interesting even 20 years from now, no less 100.

If by chance I do take "the" photograph, I would have it professionally printed on archival material.

 

[gargamel]

Software doesn't "disappear", it may no longer be updated, but you'll always be able to find it on the web.

How about this, if you're REALLY worried about it, archive your RAW files WITH a piece of software that does the conversion.

Yes, but that alone won't help. As has been said before, you need an OS and hardware to run such software on. I have a low motivation to put a 32-bits Windows XP machine, that takes ages to boot, in a safe, with a copy of DPP, just to make sure...

Yes, it will. Just because your current machine might not natively be able to run the software, doesn't mean it can't run at all. Emulators for pretty much every platform in existance can run on current machines, often with more performance then the original hardware! Getting a win32 XP emulator 50 years from now is pretty much as guaranteed as you can get.

Well... I am not so sure.

(1) I am not aware of any Win95 emulators (not that I was in need of one, and not that I have enough interested in this to search the web for one... But to my knowledge, the software that came with my USB 2.0 DVB-T receiver would not run on any system of today. Luckily, this doesn't worry me, as this software was crap, anyway, and I've never used the stick with Windows. I mention it just as an example. When cameras of today are replaced by something completely new, such as the technology that Lytro uses, formats and software of today will become obsolete. How quickly that will happen, and when it starts... only time will tell.
(With Linux the DVB-T stick runs as fine as ever, BTW).

(2) Even if there are emulators that are able to run the system and the software, you use now: Can you be sure you can connect all your storage devices? What this means is, that every once in a while, when connector standards evolve, new drives need to be purchased and huge amounts of data have to migrated to them. Noone expects to see USB 2 compatible devices in 20 years, I guess...

gargamel

 

[gargamel]

RLPhoto +1

I have been re processing some 3 year old 5DmarkII landscapes with DXO v10 and the improvements are quite marked. if they have been dinged then this wouldn't be possible. I also agree with the metadata being lost comments, made earlier, which may be where DXO are able to get the better results. I find it unexpected that while i really question the DXO Mark ratings of camera's, they do such a great job with Canon .CR2's

As someone who has experienced the improvements in raw convertors I totally agree with the advice.

This is really interesting. Could it be, that in fact, sensors are still the same, and just the software, and, in particular, the RAW converterst have been improved, in recent years, and that this is the reason, why new cameras get higher scores? Could software improvements influence test lab results, this way, and are older cameras better than we are used to think? And does this mean that you cannot really compare test results from, say, three years back with test results of now, even if the test lab claims the results to be "normalised"?
Finally: Would this explain, why images taken by a skilled photographer with a 40D look so much greater than images taken with a 6D taken by me?

gargamel

 

[hgraf]

Well... I am not so sure.

(1) I am not aware of any Win95 emulators (not that I was in need of one, and not that I have enough interested in this to search the web for one... But to my knowledge, the software that came with my USB 2.0 DVB-T receiver would not run on any system of today. Luckily, this doesn't worry me, as this software was crap, anyway, and I've never used the stick with Windows. I mention it just as an example. When cameras of today are replaced by something completely new, such as the technology that Lytro uses, formats and software of today will become obsolete. How quickly that will happen, and when it starts... only time will tell.
(With Linux the DVB-T stick runs as fine as ever, BTW).

The reason you don't see a win95 emulator is two fold:
1) most applications that run on win95 will run on the latest version of windows
2) for those that don't, install win95 on a spare machine or in a VM

As for your example, yes, there are certainly pieces of hardware and associated software that may be more difficult to get working.

That said, let's not loose sight of what we're discussing here. We're NOT talking about a day-to-day machine here. We're talking about the theoretical situation where ALL current software has lost the ability to read a CR2 file, and the only solution is to run some ancient software. In that case, all we are interested in is "resurrecting" this lost format and converting it to something "current". In that case, creating a VM or running an emulator is a small amount of work to get what we want.

Our concern is never getting the images back, not editing on an ancient platform.

(2) Even if there are emulators that are able to run the system and the software, you use now: Can you be sure you can connect all your storage devices? What this means is, that every once in a while, when connector standards evolve, new drives need to be purchased and huge amounts of data have to migrated to them. Noone expects to see USB 2 compatible devices in 20 years, I guess...

gargamel

Prior history says yes. 20 years ago is 1994. The predominant storage formats were 3.5" floppies and CDROMs. Both are easily read by modern machines with either a preinstalled DVD-ROM drive, or a USB-floppy drive (have one in my desk).

Go back 30 years and things may get a little more dicey. 5.25" floppy drives are much harder to find.

All that said, yes, if you leave a bunch of backups in a box for 50 years you MIGHT be in trouble. That's why I recommend people do what I do: live backups. Keep your data up on modern media. As the media progresses bring your data with you. If it is REALLY that important to you then it's a small price to pay.

My backups started on ZIP disks and CD-ROMs, moved to DVD-ROMs, and now live on hard drives, drives that all get updated in various degrees. My live backups are current as of an hour (cron job running every hour). My offsite (away from my home) backups are at most a month out of date. My last tier is at most 6 months out of date, and is secure in a vault.

Am I crazy to put this much effort into it? Perhaps, but my photos and other data are that important to me.

TTYL

 

[gargamel]

Well... I am not so sure.

(1) I am not aware of any Win95 emulators [...]

The reason you don't see a win95 emulator is two fold:
1) most applications that run on win95 will run on the latest version of windows
2) for those that don't, install win95 on a spare machine or in a VM

As for your example, yes, there are certainly pieces of hardware and associated software that may be more difficult to get working.

1) is only safe to say for software that is still maintained. Just like Win95 programs would not run on Windows NT, they won't run on Win7/8/10, because these systems are much closer relatives to NT than to 9x. There are major differences regarding security and user privileges, that a running program gets. On Win95 all programs were allowed to do and access virtually everything. Because programs don't get adminstrator privileges by default in the much more secure incarnations of Windows of today, and because some DLLs are no longer available, old programs don't run that well, on them... Of course, is mostly, but not only, a problem for drivers.

2) Yes, that seems to be a reasonable strategy for disaster recovery.
Apart from that, I think, there are now Win9x emulators, just because it would be illegal to publish one, without Microsoft's permission, and I guess they would be reluctant to approve such a request...

That said, let's not loose sight of what we're discussing here. We're NOT talking about a day-to-day machine here. We're talking about the theoretical situation where ALL current software has lost the ability to read a CR2 file, and the only solution is to run some ancient software. In that case, all we are interested in is "resurrecting" this lost format and converting it to something "current". In that case, creating a VM or running an emulator is a small amount of work to get what we want.

Our concern is never getting the images back, not editing on an ancient platform.

Yep, that's the point. I agree.

(2) Even if there are emulators that are able to run the system and the software, you use now: Can you be sure you can connect all your storage devices? What this means is, that every once in a while, when connector standards evolve, new drives need to be purchased and huge amounts of data have to migrated to them. Noone expects to see USB 2 compatible devices in 20 years, I guess...

gargamel

Prior history says yes. 20 years ago is 1994. The predominant storage formats were 3.5" floppies and CDROMs. Both are easily read by modern machines with either a preinstalled DVD-ROM drive, or a USB-floppy drive (have one in my desk).

Go back 30 years and things may get a little more dicey. 5.25" floppy drives are much harder to find.

I also fully agree with you, here.

All that said, yes, if you leave a bunch of backups in a box for 50 years you MIGHT be in trouble. That's why I recommend people do what I do: live backups. Keep your data up on modern media. As the media progresses bring your data with you. If it is REALLY that important to you then it's a small price to pay.

My backups started on ZIP disks and CD-ROMs, moved to DVD-ROMs, and now live on hard drives, drives that all get updated in various degrees. My live backups are current as of an hour (cron job running every hour). My offsite (away from my home) backups are at most a month out of date. My last tier is at most 6 months out of date, and is secure in a vault.

Am I crazy to put this much effort into it? Perhaps, but my photos and other data are that important to me.

TTYL

Makes completely sense to me, and will become a relevant part in my own backup and archiving strategy!
Thanks for your patience. This is all very useful for me, and I really appreciate it!

 

[gargamel]

Again, I want to thank you all for sharing your knowledge and experience with me. I've learned quite a bit from this thread, much more than I could have hoped for, when I started it. Much appreciated!

In fact, my original question has been answered in depth and I have come to a conclusion: I'll use native RAW, and I'll make sure, that I have, at least, two copies of the original RAW files on separate disks stored at different locations. I'll also make sure, that every once in while (after ~3 or 4 years?) I'll transfer the files to newer media.

Still some more aspects I have to do a little research for, but for now:

Enjoy the holidays!