March 03, 2015, 11:58:16 AM

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - epsiloneri

Pages: 1 [2] 3 4 ... 26
Post Processing / Re: Backup to Blu-ray
« on: January 04, 2015, 05:05:20 AM »
Wow, that is deep, I nodded off a third of the way down, dropped my blooming ipad.
:) I hope your ipad wasn't harmed in any way. An interesting thing about the internet is that no matter how fringe your query, there is bound to be a small group somewhere that makes it their life's mission to delve into that topic with excruciating detail. Archiving is such a topic.

I won't go into the endless details of RAID here but I will say that unless you are ready to face a steep learning curve and a lot of stress and expense, don't implement a RAID array for yourself any more complicated than a RAID 1 mirror.  It's just not worth it IMHO.  Not only do you need to understand the technology, you need to understand the hardware and how to operate it.  And once you venture past a simple RAID 1 mirror array, the hardware is critical for performance, acceptable reliability and even the possibility of recovery.
I agree with your points about RAID 5 being inadequate for the big disks of today, and that RAID 6 has slow performance. But RAID 10 is very expensive if you don't need the performance (for 5 or more disks, say) and RAID 6 should be good enough for on-line back-up solutions. Any non-enterprise user should stay with software RAID, so understanding hardware quirks shouldn't be necessary. I thus think that RAID 6 is more often the better option for the usage pattern expected for those readers of this thread with massive backup needs. Better still would be RAID Z2, using the ZFS filesystem that protects the data against corruption through bit rot, but that is more for technically inclined.

It seems like as drive capacity/density has increased over the last few years, so has the failure rate (or at least the likelihood of data loss).
I know there is the perception that this is true, but is it indeed factually true? Hard drive failures have always been an issue, with some notable examples, e.g. the IBM deskstar 75GXP (aka the "deathstar"), and more recently the Seagate 1.5TB Barracuda.

In the end, I think this about the drive quality is a bit of a red herring. Yes, you should avoid obvious duds (such as the drives mentioned above), and yes, enterprise disks have more stringent quality control. But in the long run, all drives, without exception, will fail. The trick is to plan for it, and take appropriate measures.

BUT - we are digressing.  This thread is about BACKUP.
While backup is indeed in the title of the thread, I think this thread is actually about data archiving:
The key difference between backup vs. archiving is that data backups are designed for the rapid recovery of operational data, while data archiving stores data that's no longer in day-to-day use but must still be retained.

I stated my thoughts above and I'll repeat that if you put your faith in writable dye based media, you better test it every year or two because there is a definite history of this type of media failing after a few years.
Yes, that is why regular non-LTH blu ray discs are a much better option. Or the (more expensive) m-disc DVDs. They are not based on organic dye, so do not decay on short time scales. LTH BDs use organic dye like regular writable CDs/DVDs, so don't use them for archiving.

Post Processing / Re: Backup to Blu-ray
« on: January 02, 2015, 05:30:26 PM »
There appears to be some confusion in this thread. Since long-term archiving of digital data is essential for me (and no doubt for many of you), I have investigated this in some detail:

Hard drives. Great for short-term storage, backups, but not for long-term archives. Biggest problem is not de-magnetisation of the disks (half-life is typically ~70 yr) but aging of lubricants etc, resulting in mechanical failure after 3-5 yr. The problem is actually exacerbated by storing the drives powered off, off-line. Best strategy is to store multiple copies and re-write the data every 2 yr or so. On-line discs are ok for backup, but too volatile for archive.

SSD/flash drives lose their charge over time and typically last 5-10 yr unless re-written.

Cloud storage requires high bandwidth connection, useful for smaller amounts of shorter-term backup, but not for archives. Do you trust the cloud storage provider to still keep your data safe 20 yr from now? Will the company still be around?

Optical media - considered to be the ultimate digital archival solution, adopted by libraries etc. Note that there is a huge difference between pressed media and write-able discs. Write-able CDs and DVDs tend to use organic dye that deteriorate in time - could be as bad as in a few years only. There are some exceptions, m-disc being a notable one which uses inorganic dye (in the form of "rock powder"!), but requires specially designed writers with extra powerful lasers. Regular blue ray discs also use inorganic dyes, so should be fine. There is also an m-disc BD version, but it is not clear if there is any longevity advantage yet (although they are marketed as 1000-yr solutions). Stay clear from the BD LTH discs, however, which are developed to use the same manufacturing processes as CDs/DVDs with organic dye, to significantly reduce the manufacturing costs. They have similar lifetimes as the regular write-able CDs/DVDs. Note that pressed media are different and should have lifetimes on the order of 100's of years. They are typically not useful for archival purposes, though, since you need a very big series before it becomes economically feasible to press discs.

Magnetic tape - the most economical archival solution for huge data sets is to use magnetic tapes, such as LTO. There are archival WORM tapes ("write once, read many") that are certified for 40 yr storage, and you can get a 2.5 TB tape for less than $60. These tapes are typically used by huge data centers. A problem is of course that the tape drives are typically > $1500. Perhaps they are rentable.

A good practice to protect from bit errors is to save redundancy information together with the data, in the form of error correcting codes (ECC). Popular software to generate these are ICE ECC, MultiPar, and dvdisaster (the last optimised for ECC on optical media).

Some general, disorganised rants about archiving that I nevertheless found useful.

In summary, hard drives are good for short term backups, (non-LTH) BDs for long-term storage of limited data generation rates (less than ~ TB/yr), magnetic tapes for large data rates (several TB/yr).

Technical Support / Re: Grand Canyon panoramas
« on: December 30, 2014, 01:17:46 PM »
Longer focal lengths are generally easier to stitch due to reduced projection effects.
Whilst that comment is just plain silly
Why do you think it's silly? Have you compared stitching photos from ultra-wide lenses to photos from long tele-photo lenses? Rectilinear lenses approximate the field of view by gnomonic projection on a flat surface, thereby converting a solid angle into a flat area. It is the same problem experienced when producing flat maps of the spheroidal Earth - the smaller the area covered, the smaller the distortions in the projection. That's why flat city maps look OK, while maps of the whole globe tend to look distorted.

Now, for a photo using a rectilinear lens, this distortion is a 1/cos(angle) function of the angle to the direction you are pointing the camera. With a long focal length, the difference between the centre and the edge will be much smaller than for a wide angle lens, so the image scale will be more uniform and the distortion smaller. That means that to stitch photos from wider-angle lenses, more geometric correction has to be applied in order to get adjacent images to match seamlessly. More required post-processing correction = more difficult to match well and loss of IQ.

Technical Support / Re: Grand Canyon panoramas
« on: December 29, 2014, 05:43:52 AM »
It doesn't matter what lens you use.
Longer focal lengths are generally easier to stitch due to reduced projection effects. Depending on the field of view intended to be captured, it might not be practical to have lenses of very long focal lengths as the number of required individual shots for the stitch may become too many. E.g., on full frame a 100mm lens needs 2*pi/(0.7*36/100) = 25 shots for full 360 degree panorama in landscape mode (with 30% overlap), 38 in portrait mode. If you also want more vertical coverage, the number of shots grow accordingly.

EOS Bodies / Re: High Megapixel Camera Coming in 2015 [CR3]
« on: December 17, 2014, 12:52:34 PM »
The 5DIII will be 3 years old in March next year. A move to 50MP will pretty much follow Moor´s law.
Camera resolution has historically increased way slower than Moore's law. Canon 10D had 6.3 MP in 2003. Following Moore's law, we should have GP-cameras by now. This is the reason I find complaints about too many MPs due to limited computer resources exaggerated. Relatively speaking, we have more than  50x more computer performance (memory, storage, GFLOPS...) per MP than we had in 2003.

Canon General / Re: Canon USA Addresses the Gray Market
« on: December 17, 2014, 02:57:36 AM »
And from another perspective, Canon USA products are popular gray market here in Europe. Obvious caveats are a shorter one-year US-only warranty, American-style power cords, and noticeable import duty. Documentation in our native languages is available online from Canon, so that's not a problem. For lenses, focus distance units are given in both feet and meters, so that's no problem either  :P

Canon General / Re: Canon USA Addresses the Gray Market
« on: December 17, 2014, 02:44:23 AM »
That power cord must be made of pure gold to be worth £1199
Perhaps in the same class as this one, for $1999.99?

Lenses / Re: I *HATE* UPS.
« on: November 29, 2014, 08:31:24 AM »

Bah, we all know the quality of those slow mirror tele lenses, I'm waiting for the big white refractor version. f/15 is way too slow for AF. Good with IS though.

Meaning, I care more about me liking my photographs than what others think.
Probably you're not a pro photog, then :-p ...
You're quite correct!
... however one valid point is: If you don't like the shots yourself, at least at some level, can you produce good results? If you think "well, the framing/cropping is off, but who cares?", can you continue to do good postprocessing on it?
What you think and what others think is often very different things, "good" is subjective. If you're a pro and you fulfill your client's standards, your own may not be relevant.

Technical Support / Re: Dynamic Range questions..
« on: September 27, 2014, 02:18:31 AM »
   One think I don't understand is that how a camera can have more than 14 stops of DR when the camera only record in 14-bits...
What camera does that? Detectors sometimes become non-linear near the saturation limit, but the A/D conversion is usually linear, AFAIK.

Many people obsess over things that no-one else sees.
It's nice if others like my photographs, but first I want to like them myself. Meaning, I care more about me liking my photographs than what others think. Maybe a tad egocentric :) With a more positive spin it could be called artistic integrity.

Photography Technique / Re: Why 3:2 aspect ratio?
« on: September 27, 2014, 02:00:17 AM »
I often crop to 4:3, in particular for portrait. Landscapes generally get much wider crops.

Canon General / Re: Square or cross shaped sensors
« on: September 27, 2014, 01:56:36 AM »
The obvious answer would be to use a circular raw detector, and then crop to whatever format you like (either in camera or post). This would make ideal use of the optics, which inherently provide a circular image. I think we will eventually get sensors like this, once the sensors become significantly less expensive than the optics. The pixels may then be hexagonal as well, perfectly sampling an f/1.0 diffraction-limited image and interpolated to produce the desired orientation of square pixels.

Lenses / Re: Is there a need for a 50mm?
« on: September 07, 2014, 04:10:38 PM »
I love walking around with my 70-200 with the 40 in my pocket.
You really need big pockets to put in the 70-200 should you want to switch to the 40, though.

Pages: 1 [2] 3 4 ... 26