Best Methods For Long Term File Storage ??

neuroanatomist

I post too Much on Here!!
Jul 21, 2010
24,610
2,072
Drizzt321 said:
...I know it's not forever.
But what is? I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy? ...dating myself again, I know). The data may be intact, but I have no way to read them...
 

ahab1372

EOS RP
Nov 8, 2012
326
0
neuroanatomist said:
Drizzt321 said:
...I know it's not forever.
But what is? I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy? ...dating myself again, I know). The data may be intact, but I have no way to read them...
Common problem, and a huge one for archives, libraries etc.
Every time we upgrade or dispose of hardware and/or software, we should be checking if there is data somewhere that we should be converting or copying. I'm not obliged to archive everything, and most files that I haven't looked at in a year or two, or was created with software that I cannot run anymore, are probably not worth keeping. But for pictures I might judge differently.
 

cliffwang

EOS 7D MK II
Feb 7, 2012
501
0
48
Morgan Hill, CA
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
 

Drizzt321

EOR R
Nov 23, 2011
1,667
0
Lala land
www.aaronbaff.com
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
 

cliffwang

EOS 7D MK II
Feb 7, 2012
501
0
48
Morgan Hill, CA
Drizzt321 said:
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
I used to have FreeNAS as my NAS. However, my wife complains I had too many systems. Thus, I have to reduce my toys. I have already phased out two systems. We still have few systems at home. My target is to have only one desktop(server) and two Windows 8 PRO tablets. If I build another server, I will get trouble with my wife. ;D
 

Drizzt321

EOR R
Nov 23, 2011
1,667
0
Lala land
www.aaronbaff.com
cliffwang said:
Drizzt321 said:
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
I used to have FreeNAS as my NAS. However, my wife complains I had too many systems. Thus, I have to reduce my toys. I have already phased out two systems. We still have few systems at home. My target is to have only one desktop(server) and two Windows 8 PRO tablets. If I build another server, I will get trouble with my wife. ;D
Hmm...maybe have one big honkin' machine that you run virtualized Windows on for your Windows needs? I've recently had the thought that maybe I'll do that for my Lightroom/PS needs...still haven't made up my mind on investigating that yet, but it's crossed my mind.
 

kdw75

EOS M50
Feb 7, 2012
32
0
Because of changing media and technology I tend to favor cloud storage. I keep a copy on my computer and when my drive fills up I put in a larger one and put the old one in a drawer. This way I can always revert back to older drives if necessary. Then I also keep my stuff stored with a backup service since keeping it in that form doesn't rely on me having a particular computer or medium to read it. It is accessible any where I go with an internet connection. I probably should backup off site, but I haven't gone that far. I plan to get a RAID system at some point.
 

gmrza

EOS 7D MK II
Jan 21, 2011
521
1
neuroanatomist said:
Drizzt321 said:
neuroanatomist said:
sb said:
I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.
How many is 'a few'? I recently listened to The Cars Greatest Hits, a CD that I bought in 1985. :eek:
That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.
Makes sense, thanks!
A bit of reading I did into DVD/BluRay media recently basically indicated that there are no real standards for archival optical media, and quite honestly, your mileage may vary. - It is rather uncharted territory, and there seems to be no consensus as to how long a burned DVD or Bluray disk could last. (I have some CD-Rs which are around 20 years old, and still readable. - Is that pure luck? I have no idea.)

The bottom line is that for data backups you need to have a policy of copying to new media every few years. That however tends to happen by default as existing drives become too small and larger, cheaper drives hit the market.

To be heretical: the most reliable long-term storage medium for a photo still seems to be paper. Provided a print is stored under controlled conditions, you may achieve the rated life of around 200 years for archival paper. - I don't think any of us would be around to confirm. Of course, that does not answer the OP's question!
 

CharlieB

EOS RP
Jul 29, 2012
301
0
I dump to primary HD on desktop. It automatically copies to secondary on my Windows server. The Windows server cross-pollinates with Linux server nightly. Done. Three copies, three machines, three hard drives.

CD... dont last, unless you do a slow burn on them. I've got two and three year old CD's... lots of problems with those.
 

dirtcastle

EOS RP
Dec 10, 2011
390
0
eric-nord.com
If money is not an issue, I would go with a "cloud" solution. The main two problems with cloud storage is moving the files and host reliability. The issue of upload/download speed can be solved with a Fios or a fast cable connection. The reliability issue can be solved by having two cloud accounts and also have a local backup. Or you could do the double backup on your end and use a single cloud host.

While it's true that there will be some "bit rot" on large collections being stored over decades of time... unless you are the Smithsonian, I don't see why a little loss would be a major problem.

I think the main problem with long term storage is organization. By using a reliable, long term, cloud host, you can have a single storage solution with easy access and no barriers to organization. All existing "local" storage methods will have limits of both size and ability to organize.

If money is an issue... forget long term storage: carefully pick your favorites and buy cheap arrays. Reliable long term storage for large collections costs money; there's no way around it (at least, not that I know of).
 

Hillsilly

EOS 6D MK II
Oct 16, 2010
1,096
0
I'm hedging my bets. I print my favourite photos at 5x7 or 8x12. I upload photos to an online site. I keep backups on multiple hard drives. I also have photos stored on the hard drives of a couple of computers. With film, I keep negatives in archival plastic sleeves.

But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?
 

papa-razzi

EOS 80D
Jul 21, 2010
159
0
Hillsilly said:
But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?
There will be data mining software that will recognize faces and other image content to pull up specific photos of people, events, etc.
 

cayenne

EOR R
Mar 28, 2012
1,967
125
papa-razzi said:
Hillsilly said:
But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish. Is anyone ever going to look at that many photos or care for such a collection?
There will be data mining software that will recognize faces and other image content to pull up specific photos of people, events, etc.
You might wanna all read the TOS for the Cloud service you are using......THEY just might be using said data mining software using your images already for their own purposes.

I'm fairly sure three letter agencies are likely doing it quite prevalently...too large of a free data store for them to pass by.
 

Mt Spokane Photography

I post too Much on Here!!
Mar 25, 2011
15,523
752
What do you mean by "Long Term" Do you mean hands off as compared to periodic maintenance and renewal?

5yr?, 10yr?, 50 Yr?, forever?

There really are no 50 or 100 year hands off storage solution, too many issues can and will happen over that time, and you will likely have to move image files to different media every 10 years or so.
Solid State Memory - forget it
Hard Drives - better, but no
Online storage - no
Prints - possibly

Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all. In about 2000, my hard drive crashed and everythinng was lost. I had a one year old backup on CD of my images, and recent backups on floppy disks.

The CD's had deteriorated in only one year and were unrecoverable, but I did recover most from the floppy drives. I then started making copies to hard drives as well as CD's and DVD's. I also switched to a raid system. Since then, I've never had a crash that lost data, but its a matter of when, not if.

I have a box full of 100 year old B&W prints and another box of negatives going back to 1940 of my ancestors. The color prints starting in the 1950's are faded away, and I have some color negatives from the 1940's that have faded as well. Some of the prints have been copied and sent to me by relatives, so the process of distributing images to relatives who might want to keep them seemed to work. I have scanned them and backed up the scans as well.

Current printing technology has been said to produce 100+ year life prints, so that might be the best backup method, prints and a fireproof box. Send copies to relatives as well. Many may still be arounnd in 100 years when the files have long been lost.
 

revup67

Memories in the Making
Dec 20, 2010
641
4
Southern California
www.flickr.com
With respect to all of these ideas I wish to comment in that how are you backing up? Is it a straight copy without verification? Possibly a bad move. After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50. There are other tools in the suite as well for partitioning, etc. You should be doing a data verification at the very least if using a copy method. Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons. Paragon will actually tell you if a drive is faulty as it checks the drive before backup. I do IT for a living and this actually happened at a client's. Ultimately I replaced her drive. In addition, I use Diskeeper. When files are created fragmentation takes place. In NTFS you have 4096 bytes in each cluster size. Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order. When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive. Solid state drives are not excluded from fragmentation of course. Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more). It runs real time, small footprint and no performance impact. All HD drives on a single PC are kept at 0% fragmentation. You can run all kinds of reports on the drive. There are various indicators that will tell you the drives health as well. Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance. I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's. I've got a link to disk keeper on my site if anyone is interested drop me a note. I can get you a demo. PS they are now called Condusiv Technologies. Sorry if this sounded like an ad but its my personal experience and using these products daily. Been using Diskeeper for approx. 9 years and know it well. It's on every server I manage.
 

Drizzt321

EOR R
Nov 23, 2011
1,667
0
Lala land
www.aaronbaff.com
revup67 said:
With respect to all of these ideas I wish to comment in that how are you backing up? Is it a straight copy without verification? Possibly a bad move. After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50. There are other tools in the suite as well for partitioning, etc. You should be doing a data verification at the very least if using a copy method. Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons. Paragon will actually tell you if a drive is faulty as it checks the drive before backup. I do IT for a living and this actually happened at a client's. Ultimately I replaced her drive. In addition, I use Diskeeper. When files are created fragmentation takes place. In NTFS you have 4096 bytes in each cluster size. Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order. When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive. Solid state drives are not excluded from fragmentation of course. Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more). It runs real time, small footprint and no performance impact. All HD drives on a single PC are kept at 0% fragmentation. You can run all kinds of reports on the drive. There are various indicators that will tell you the drives health as well. Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance. I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's. I've got a link to disk keeper on my site if anyone is interested drop me a note. I can get you a demo. PS they are now called Condusiv Technologies. Sorry if this sounded like an ad but its my personal experience and using these products daily. Been using Diskeeper for approx. 9 years and know it well. It's on every server I manage.
I've used diskeeper in the past, worked fine. Generally if it's a single large disk that you simply dump photos onto without deleting, chances are you won't get much, if any, fragmentation. Paragon seems like simply a quality backup solution, but there's nothing inherently unique about it. Doing basic S.M.A.R.T. checks, there are a bunch of tools out there, and you can always use something like rsync and/or par2 parity blocks and checks which are F/OSS. However, Paragon is likely to do all that in a much easier to use, nicer to use, prettier package that will probably be well worth the cost since it seems fairly inexpensive.

That said, data at rest on a HDD is, over time, likely to experience bit-rot, which is where a proper backup solution will include some form of parity generation which can at worst detect a problem, and most of the time actually fix/recover the missing data. This you get automatically with a filesystem such as ZFS, but not NTFS, Fat32, Ext2/3 (not sure about Ext4, I don't think so). You can generate that information from my previously mentioned par2 generated parity blocks, or I'm sure there are other tools out there. Not sure if Paragon would do that or not, something to look for in the feature list anyway.
 

dirtcastle

EOS RP
Dec 10, 2011
390
0
eric-nord.com
Mt Spokane Photography said:
f

Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all. In about 2000, my hard drive crashed and everythinng was lost. I had a one year old backup on CD of my images, and recent backups on floppy disks.
While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive. We've all had personal hard drives fail. I've also had hosted drives fail too. But the difference was that my online host had a backup system and restored everything without me paying any extra money or doing any extra work.

Ultimately, the best strategy is redundancy. For a home solution, it means arrayed drives (and a fireproof safe). But the level of cost and maintenance on such a system is formidable and you will be doing all of the work of maintenance, organizing, and syncing. And then, presumably, you will need to completely rebuild the system every 10 years or so. And without an associated online solution, you have only local redundancy.

For me, the big difference between a home solution and "hosted" solution is that the host will upgrade their equipment over the years. They will transfer your data and maintain backups. I would never suggest relying exclusively on a cloud host. Never. But let's recognize that storing data is what these companies do; and setting up an equivalent system at home is unrealistic for most photographers.

That said, as someone who has had hosted drives go down, I can understand people's wariness of using an online solution. And it's true that if the host goes down, those crusty backups will be the last line of defense. That was why, for super critical collections, I suggested having two online hosts.

It's all about redundancy and multiple points of failure. No single system will give a high level of reliability. All of these systems involve risk.
 

cayenne

EOR R
Mar 28, 2012
1,967
125
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
Well, you know there are other OSes out there besides Windows....some are VERY robust, etc.

You've heard the saying...the right tool for the right job.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

A little DIY and learning a new thing...is never a bad thing.

I don't use windows much, unless a tool I need to use requires it...but I figure every OS has its uses....best not to get too dependent on any one, especially one owned by a company that can change its mind, direction or go out of business as your primary OS.

You might get to start familiarizing yourself with Linux or other variants of Unix type OSes....they can really be your friend. Sure there is a learning curve.....but there is always a LC when learning any new, valuable skill, no?

;D

cayenne
 

cliffwang

EOS 7D MK II
Feb 7, 2012
501
0
48
Morgan Hill, CA
cayenne said:
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.
Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.
Well, you know there are other OSes out there besides Windows....some are VERY robust, etc.

You've heard the saying...the right tool for the right job.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

A little DIY and learning a new thing...is never a bad thing.

I don't use windows much, unless a tool I need to use requires it...but I figure every OS has its uses....best not to get too dependent on any one, especially one owned by a company that can change its mind, direction or go out of business as your primary OS.

You might get to start familiarizing yourself with Linux or other variants of Unix type OSes....they can really be your friend. Sure there is a learning curve.....but there is always a LC when learning any new, valuable skill, no?

;D

cayenne
I used to use Linux system and built FreeNAS years ago. However, Windows systems are very stable nowadays. Now what I am looking for is a compromised solution between performance, simple, and few machines. What I might going to do later is building a Windows Server 2012 with ReFS system and run some VMs on it for different jobs. VHD will let me manage storage easily. I will play around and to see if I need to have an extra file server or not. Since my current file server is already 4 years old, I might build a FreeNAS next year if necessary.