October 24, 2014, 08:32:36 AM

Author Topic: Bottleneck when opening images of CF card?  (Read 2693 times)

V8Beast

  • 1D X
  • *******
  • Posts: 1022
    • View Profile
    • Stephen Kim Automotive Photography
Bottleneck when opening images of CF card?
« on: November 04, 2013, 05:07:46 PM »
I'm planning out my next custom PC build, and want to narrow down potential bottlenecks. More specifically, when reading images directly off the CF card using a card reader, I'd like them to open up as quickly as possible. I'd imagine that this is determined by the read speed of the CF card itself, and the transfer rate of the USB interface. Opening up 200 or so RAW files at once seriously slows down my current system, so I'd like to cut down the wait time as much as possible.

Dumb question #1: Does the amount of RAM factor into this equation at all?

Dumb question #2: Since the images aren't being copied into the hard drive in this scenario, does the HDD write speed matter for this specific issue? My guess would be no.

I'm debating whether or not it's worth it to go with solid state drives for storage, or go with a standard HDD in a RAID 0 configuration to boost read/write speed. A 1 TB solid state drive still costs $600, and while they're blazing fast, that's still a lot of money.   

canon rumors FORUM

Bottleneck when opening images of CF card?
« on: November 04, 2013, 05:07:46 PM »

vlad

  • PowerShot G1 X II
  • ***
  • Posts: 48
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #1 on: November 04, 2013, 05:37:08 PM »
What do you mean by "opening" 200 RAWs at once?  Do you mean importing them to your hard drive, or actually opening that many in Photoshop at once (seems extremely unlikely)?

To import them faster, using a fast card and a fast (usb 3) card reader is your best bet.  Amount of RAM won't make a noticeable difference, although with RAM it's always good to have enough to not worry about it.

SSDs versus HDDs for faster photo editing?  I have experimented with keeping photos on an SSD, or keeping just my Lightroom catalog and previews on an SSD, and in my workflow (mostly Lightroom with a bit of Photoshop), I have not seen any performance boosts that would warrant major investment into SSD.  The bottleneck with importing or exporting is usually with Lightroom itself: the disk and the CPU don't even come close to 100% utilization for these time-intensive tasks.

Also, I would strongly recommend against raid0, unless you have a great and reliable backup system that you can't wait to try out. 

pwp

  • 1D X
  • *******
  • Posts: 1606
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #2 on: November 04, 2013, 07:02:33 PM »
I get very fast downloads from CF card to PC using either a USB3 Sandisk card reader or a blindingly fast internal SATA card reader.  Obviously fast cards will help.

I wouldn't be storing on an SSD... far too expensive and unnecessary. On my two current home-built PC's I have the OS (Win7) and programs on a modest sized 180Gb Intel SSD, and local storage on a 2Tb HDD, then archive network storage on a five-HDD Synology NAS. BTW, I wouldn't go with Win 8. It's a dog in the tradition of Vista and the dreaded ME. Win 7 is the new XP.

If you were really needing maximum speed, you could keep a small folder on your SSD named "Current Project" and import your current job, do your work and then shunt it out again once you've done your post-pro. Photoshop scratch would also be on the SSD.

Here's a good tip, buy your RAM from http://www.crucial.com/ If you give them the specs of your EXACT mobo, they identify & send you the most compatible memory possible for that board.

-pw

timmy_650

  • Rebel SL1
  • ***
  • Posts: 93
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #3 on: November 04, 2013, 07:19:58 PM »
What a few of my computer friends do is they put a smaller SSD drive in to put the operation system on it so that will run faster and they put a normal hard drive (2tb range).

So what I am guessing what you are asking; When you are transferring you Pictures to you computer, you are worried about transfer speed. So if you card reader is usb 3 you should be fine or if you build it in your computer (sata) you should be fine.

fugu82

  • EOS M2
  • ****
  • Posts: 167
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #4 on: November 04, 2013, 07:26:34 PM »
+ 1 on the above comments. Buy a very fast USB3 reader, and don't go through a bus.

I use a Lexar LRW300U and it's been great.

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Bottleneck when opening images of CF card?
« Reply #5 on: November 04, 2013, 07:26:45 PM »
A good quality USB3 reader (personally I love my Lexar Pro one) along with a quality spinning disk to hold all of the files, and then I store the Lightroom Catalog + Previews on an SSD. My OS and stuff lives there, and those don't take all that much more space. Plus with the LR5 Lossy Previews, I can keep the files elsewhere if I need to and just export later when I get home.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

V8Beast

  • 1D X
  • *******
  • Posts: 1022
    • View Profile
    • Stephen Kim Automotive Photography
Re: Bottleneck when opening images of CF card?
« Reply #6 on: November 04, 2013, 08:37:45 PM »
Thanks for the great advice, fellas.

What do you mean by "opening" 200 RAWs at once?


Yeah, that's what I meant :)

Quote
SSDs versus HDDs for faster photo editing?  I have experimented with keeping photos on an SSD, or keeping just my Lightroom catalog and previews on an SSD, and in my workflow (mostly Lightroom with a bit of Photoshop), I have not seen any performance boosts that would warrant major investment into SSD.  The bottleneck with importing or exporting is usually with Lightroom itself: the disk and the CPU don't even come close to 100% utilization for these time-intensive tasks.

Good to know!

Quote
Also, I would strongly recommend against raid0, unless you have a great and reliable backup system that you can't wait to try out.

My plan, as others have suggested, was loading the OS and major software programs onto a small SSD. Then, I planned on utilizing a RAID 0 on a standard HDD for storage, and having a separate RAID 1 HDD to back up the two RAID 0 HDDs. If there is no real benefit in system performance from the RAID 0 setup, I will gladly forgo the added complexity of the RAID 0+1 arrangement. Instead, I'd probably just opt for a RAID 1 setup, and back that up onto another external HDD so I'd have triple redundancy of my images. 
« Last Edit: November 04, 2013, 08:41:10 PM by V8Beast »

canon rumors FORUM

Re: Bottleneck when opening images of CF card?
« Reply #6 on: November 04, 2013, 08:37:45 PM »

V8Beast

  • 1D X
  • *******
  • Posts: 1022
    • View Profile
    • Stephen Kim Automotive Photography
Re: Bottleneck when opening images of CF card?
« Reply #7 on: November 04, 2013, 08:47:15 PM »

I wouldn't be storing on an SSD... far too expensive and unnecessary. On my two current home-built PC's I have the OS (Win7) and programs on a modest sized 180Gb Intel SSD, and local storage on a 2Tb HDD, then archive network storage on a five-HDD Synology NAS. BTW, I wouldn't go with Win 8. It's a dog in the tradition of Vista and the dreaded ME. Win 7 is the new XP.

-pw

That's a slick setup, even more so since it can be setup as a cloud.

And yes, I will gladly pass on Windows 8. My wife's new laptop has it, and I hate it. I thought Microsoft got its act together after the Vista debacle, and Windows 7 was a refreshing improvement, and then it comes out with a rubbish OS like Windows 8. No thanks.

Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • *********
  • Posts: 8880
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #8 on: November 04, 2013, 09:16:06 PM »
If you are ope3ning files, they will load into RAM, and opening 200 at once takes a lot of ram.  A 30mb CR2 file uncompresses to more like 50mb, and 200 of them is 10GB.  With overhead for the OS and Lightroom, you are going to gobble up 16GB of ram, I'd get 24 or 32.
Your lightroom catalog needs to be on a SSD if you want fast opening, its a gigantic file.  I use a 512GB SSD for my OS, Lightroom Catalog, and all of my programs (256GB is plenty).  My data files are on a 3TB SSD and a 12GB NAS.
 
SSD's and memory are cheap now , and a Samsung 256 GB SSD for a boot disk to hold programs and Lightroom Catalog as well as scratch disk for photoshop is usually plenty large, and costs less than $200.
You will get more speed by spending $400 on memory and a SSD over a upgrade to a $1000 processor.

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Bottleneck when opening images of CF card?
« Reply #9 on: November 04, 2013, 09:53:29 PM »

My plan, as others have suggested, was loading the OS and major software programs onto a small SSD. Then, I planned on utilizing a RAID 0 on a standard HDD for storage, and having a separate RAID 1 HDD to back up the two RAID 0 HDDs. If there is no real benefit in system performance from the RAID 0 setup, I will gladly forgo the added complexity of the RAID 0+1 arrangement. Instead, I'd probably just opt for a RAID 1 setup, and back that up onto another external HDD so I'd have triple redundancy of my images.

Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.

Plus, RAID is not a backup, regardless of if it's 1, 5, 6, 10, 50, etc. It's a means to higher availability and sometimes better performance.

If you're after performance, invest in a decent SSD. Otherwise, if you want storage space/reliability, go for RAID1 or RAID6 (RAID5 is relatively fragile). 0+1/1+0 is best if you want a better balance between performance & reliability.

I agree, backup to an external source (and then to another geographically distributed place) is the correct idea.

If I were you, I'd invest in a better CPU, and then do a RAID1/6 on your storage drive if you want enhanced reliability with a proper backup strategy. And either invest a significant amount of money into a proper RAID controller...or go for Windows software RAID. It'll be much more reliable and theoretically portable as opposed to whatever motherboard half-assed software RAID that they do.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

pwp

  • 1D X
  • *******
  • Posts: 1606
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #10 on: November 04, 2013, 10:05:32 PM »
My data files are on a 3TB SSD and a 12GB NAS.
Awesome! You have a 3Tb SSD? Is this a typo or did you mean HDD?
The biggest SSD on the planet is 4Tb and costs an eye-watering $29,000
http://semiaccurate.com/2013/06/23/lsi-puts-out-a-4tb-pcie3-ssd/

-pw
« Last Edit: November 04, 2013, 10:26:13 PM by pwp »

V8Beast

  • 1D X
  • *******
  • Posts: 1022
    • View Profile
    • Stephen Kim Automotive Photography
Re: Bottleneck when opening images of CF card?
« Reply #11 on: November 05, 2013, 12:29:40 AM »
Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.

You're right. I read up some more about RAID configurations, and I have lots more research to do. RAID6 is completely over my head. I think I'll just stick with a standard HDD for storage and back up to an external drive :)

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Bottleneck when opening images of CF card?
« Reply #12 on: November 05, 2013, 12:34:02 AM »
Ok, so, you seem to have a mis-understanding of what RAID 0+1 (or 1+0, aka RAID10) is. It's doing a strip (RAID0) across mirrors (RAID1), or mirrors across strips. As opposed to having a separate RAID0 then a separate RAID1.

You're right. I read up some more about RAID configurations, and I have lots more research to do. RAID6 is completely over my head. I think I'll just stick with a standard HDD for storage and back up to an external drive :)

Or RAID1 (mirror), that's pretty basic and easy and will help ensure high availability and avoid having to restore from a backup in the event of a single disk failure. Of course, still backup properly, but RAID1 is a viable part of that.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

canon rumors FORUM

Re: Bottleneck when opening images of CF card?
« Reply #12 on: November 05, 2013, 12:34:02 AM »

Halfrack

  • Canon 7D MK II
  • *****
  • Posts: 480
    • View Profile
Re: Bottleneck when opening images of CF card?
« Reply #13 on: November 05, 2013, 09:54:02 AM »
OP needs to look at Photo Mechanic - http://www.camerabits.com/products/ - if you're going to open up lots of files directly, it renders faster than Lightroom, but it isn't a full on photo editor.

If you're going to open that many files at once, you are spending more money on getting top end CF cards - or import them onto a SSD and open from there.  It adds the benefit of not deleting the only copy of an image.
"Me owning a lens shop is kind of like having an alcoholic bar tender." - Roger Cicala

RustyTheGeek

  • Buy and Sell
  • 1D Mark IV
  • ********
  • Posts: 917
    • View Profile
    • Images I've Shot...
Re: Bottleneck when opening images of CF card?
« Reply #14 on: November 05, 2013, 11:31:49 AM »
OK, IT guy here.  Sorry for the long post but I hope this advice helps.

I've read everything up to now and it all makes sense and I agree.  So far, the info, comments, opinions, corrections, etc all add up and make sense.  No need to repeat it all again.

My primary advice is that you take your time.  Build it, install Win7Pro x64 OS (great OS choice at this time) to a decent SSD like maybe an Intel 520 series or Samsung 840 Pro series and then systematically install all the updates and find the most up to date drivers.  Get a solid motherboard from ASUS, Intel or several other reputable companies.  Don't buy cheapo motherboard+CPU deals.  Buying the motherboard is usually my hardest decision.  Make sure the BIOS is fully updated.  Make sure the USB3 firmware is fully updated.  Put your catalog and/or photoshop cache files on the SSD.  Store the actual image files on a secondary large RAID protected volume.  Don't buy cheapo RAM.  Stay with Crucial, Micron, Kingston, PNY, G.Skill, Patriot, etc.  The reputable motherboard manufacturers usually have lists of RAM they certify to work that they tested.

Once all that is done, do an image backup and start benchmark/stress testing the system with file transfers, benchmark utilities, etc.  Make sure you are getting the performance of the CPU, RAM, Drives, Motherboard, Video, etc that you paid for.

Oh, and get a high quality power supply.  Get a large tower case.  (Antec 300 is a good choice.)  You'll be adding drives, believe me.  Get something that is at least 750W.  Real watts, not cheapo watts.  Something like a good Corsair unit.  If you paid less than $100 for the power supply, it probably isn't good enough.  Newegg has great informative IT geek reviews and will likely help you get a good unit for your needs.

Put this system and the display on a UPS with AVR.  My Costco regularly carries the CyberPower 1350AVR for around $100.  It's a great unit and it's on Newegg for around $145.  Just get one with AVR.  Direct power from the wall (the grid) isn't nice to your sensitive electronics.  I have a good UPS on everything in my home that is electronic and worth more than $100.  You know, like that $3500 plasma tv!  It's not a guarantee against calamity but it's better than nothing.

Don't go overboard on the video card for a photo pc.  Most of the performance benefit will be gained from the SSD, CPU, RAM, USB3, then video, etc.  Spend the video money on a nice high end IPS display and calibration tool.  Maybe get dual displays but one large high end IPS panel is better than two cheapo LCD panels.  Put in plenty of storage in RAID 1 configurations for working data storage and individual internal and external drives for backups.

The hard part isn't just the hardware, it's setting up the software, getting an effective sync/backup system in place and then keeping up with it.  You can't have too much backup.  Consider an offsite solution as well, online or physically carrying external drives to another location or both.  Online is good for data but not for recovering the system so you need a local image backup of at least the system drive.

Finally, don't use this nice fast photo pc to do everything.  If you install a bunch of stuff like QuickBooks,  Office, HP Printer software, huge antivirus security packages, etc, you will kill the performance.  Just use it for photography.  If you don't use it to surf porn, play online games or do your online banking, don't install antivirus or just install the perfectly fine Microsoft Antivirus.  Seriously, you'll thank me later.
Yes, but what would  surapon  say ??  :D

canon rumors FORUM

Re: Bottleneck when opening images of CF card?
« Reply #14 on: November 05, 2013, 11:31:49 AM »