canon rumors FORUM

Gear Talk => Software & Accessories => Topic started by: cayenne on April 24, 2013, 02:38:10 PM

Title: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 24, 2013, 02:38:10 PM
Hi All,

My set up:
macbook pro (late 2011 model, 16GB ram), Dell U2711 monitor.

Now, as you can imagine, shooting all RAW, making multiple versions of some images in Post...and doing lots of video, editing it...versions after color grading, etc.

Well, I'm eating LOTS of disk space...that's not even keeping in mind, the software I've installed on said computer (aperture, fcpx, plugins, VMware, GIMP...etc, etc).

I'm having a terrible time keeping disk space open. As soon as I finish a project, I just move all the files off to an external single drive I plug into a USB connected drive docking station.

I'm thinking that I need some sort of fairly speedy, maybe dual disk NAS type thing....?  Maybe do it RAID 1 (mirrored) there.

I'm trying to fix up a separate computer to build a freeNAS system for large backups, haven't gotten that up and running yet, but I think for right now...I just need a large, fast disk storage system for holding and working on my current few projects.

Does anyone have a suggestion for that?

My macbookpro, has one thunderbolt connector on it...I'd think that would be the way to go, however, I'm using that with a thunderbolt to displayport cable/adapter...to run the Dell U2711 monitor, and I don't think there is a way to chain off that...?

So, likely firewire or sadly, USB2 is about my only option for connecting....

Anyway, does anyone have a suggestion on what I might look towards?

Thanks in advance!!

cayenne
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: CR Backup Admin on April 24, 2013, 02:56:05 PM
When you say "scratch disk", I think of the setting for scratch disk in photoshop.  It needs to be very fast and if you use a external disk, USB and Firewire are painfully slow.

You might be better off getting a large fast SSD, a 960GB Crucial SSD costs $600.

http://www.crucial.com/store/listmodule/SSD/~983040~~M500~/list.html (http://www.crucial.com/store/listmodule/SSD/~983040~~M500~/list.html)
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 24, 2013, 04:42:01 PM
When you say "scratch disk", I think of the setting for scratch disk in photoshop.  It needs to be very fast and if you use a external disk, USB and Firewire are painfully slow.

You might be better off getting a large fast SSD, a 960GB Crucial SSD costs $600.

http://www.crucial.com/store/listmodule/SSD/~983040~~M500~/list.html (http://www.crucial.com/store/listmodule/SSD/~983040~~M500~/list.html)

Perhaps I'm using the wrong terminology...

Basically the situation is, the hardisk in my macbookpro (750GB?) is getting full. I only have about 35-60GB free at any given time here lately.

I want to have my RAW photo projects and Video projects I'm working on...on some sort of external drive setup, that would be fast enough to keep up with working on them....like with FCPX editing/rendering...with Davinci Resolve Lite...etc.

But..sounds like you're saying that since I likely only have USB and firewire on my computer, they won't be fast enough?

Hmm...wonder if there is any possible way I could hook something externally on the one thunderbolt connector, and then, chain off that storage unit...to a displayport adapter for my monitor?

I just gotta do something, as that my internal harddrive is just too full and most of it seems to be from my installed applications and iTunes maybe. I think it is frankly mostly installed apps.

I need to set up somehow externally for photo and video work...

C
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Mt Spokane Photography on April 24, 2013, 06:16:28 PM
I wonder if something like this would work??  Its a thunderbolt hub with a extra port for daisy chaining and two esata ports for external hard drives.  The question is if it will work with a display.
 
http://www.newegg.com/Product/Product.aspx?Item=N82E16822154646&Tpk=thunderbolt%20hub&IsVirtualParent=1 (http://www.newegg.com/Product/Product.aspx?Item=N82E16822154646&Tpk=thunderbolt%20hub&IsVirtualParent=1)
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Grumbaki on April 24, 2013, 10:58:49 PM
If you are really blowing through disk space, I d recommand a full on NAS that can run raid 5 or 10 depending on your budget and your space needs.

I have an older version of that
http://www.synology.com/products/product.php?product_name=DS412%2B&lang=us (http://www.synology.com/products/product.php?product_name=DS412%2B&lang=us)

Not cheap but excellent investment all around. mine runs 3Tb of useable space in raid 5 (3/4rth of your phisical space if all disks are of equal size), one disk can go to hell you don't lose anything. Ethernet is used so speed isn't an issue. I just see slow down when my router gets prompts from too many devices (3 or 4 out of the 6 we have at home). I'm stuck with a crappy one due to my internet provider (yay china) using weird ass protocols and the absence of competent english speaking techie so right now I'd say router might be the weak link of the set up.

The day it's affordable i get the NAS to run on SSD as the HDD are the next weak point.

(and the day i own my flat and not rent it I get in walls fiber :D )

Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: scottkinfw on April 24, 2013, 11:30:46 PM
I have a great suggestion for you to look into.  I wouldn't suggest RAID 1 either because what you gain in speed, you lose in protection.  Why back up only to lose it all- right?

Look at Drobo.  a tad expensive but fast, protected, hot swappable, gives you notice when a drive will fail, if a drive fails, all data is saved on other disks.  When you fill up the drives, replace with bigger ones.  If you max out Drobo (possible, but with capacious drives, hard to imagine at this point) you can daisy chain them.  There are a good number to choose from. 

I have been using mine a couple of years and love it.  It is easy to set up.  Additionally, you can start with two drives and add more when you get more $$.  It is a lot easier than the setup you describe (I have tried NAS before and they can be a pain to set up and keep up).  I am on Mac's and no problem.

I would finally suggest that you keep your Macbook clean and simple.  Just keep your programs on there and keep you pics etc. on the Drobo.  With fast drives in the Drobo, you shouldn't have a problem doing intensive editing.

Hope this helps.

sek

Hi All,

My set up:
macbook pro (late 2011 model, 16GB ram), Dell U2711 monitor.

Now, as you can imagine, shooting all RAW, making multiple versions of some images in Post...and doing lots of video, editing it...versions after color grading, etc.

Well, I'm eating LOTS of disk space...that's not even keeping in mind, the software I've installed on said computer (aperture, fcpx, plugins, VMware, GIMP...etc, etc).

I'm having a terrible time keeping disk space open. As soon as I finish a project, I just move all the files off to an external single drive I plug into a USB connected drive docking station.

I'm thinking that I need some sort of fairly speedy, maybe dual disk NAS type thing....?  Maybe do it RAID 1 (mirrored) there.

I'm trying to fix up a separate computer to build a freeNAS system for large backups, haven't gotten that up and running yet, but I think for right now...I just need a large, fast disk storage system for holding and working on my current few projects.

Does anyone have a suggestion for that?

My macbookpro, has one thunderbolt connector on it...I'd think that would be the way to go, however, I'm using that with a thunderbolt to displayport cable/adapter...to run the Dell U2711 monitor, and I don't think there is a way to chain off that...?

So, likely firewire or sadly, USB2 is about my only option for connecting....

Anyway, does anyone have a suggestion on what I might look towards?

Thanks in advance!!

cayenne
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Niterider on April 24, 2013, 11:48:30 PM
If you dont use the cd/dvd drive too often, I have heard that you can remove it and put a ssd in its place. Because that will be a sata connection, it will be the fastest.

I have a server with windows server 2012 on it, but I wouldn't be caught dead using one of it's disks as a scratch disk. Way too slow. NAS would be even slower.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: FOB2009 on April 25, 2013, 12:09:49 AM
Check out http://macperformanceguide.com/index_topics.html (http://macperformanceguide.com/index_topics.html) and then http://eshop.macsales.com/shop/SSD/OWC/ (http://eshop.macsales.com/shop/SSD/OWC/)
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: dirtcastle on April 25, 2013, 01:59:51 AM
My macbookpro, has one thunderbolt connector on it...I'd think that would be the way to go, however, I'm using that with a thunderbolt to displayport cable/adapter...to run the Dell U2711 monitor, and I don't think there is a way to chain off that...?

New Apple displays have an extra T-bolt on the back. But obviously that doesn't help in your case. :-(

I wonder if you could connect your monitor through the FireWire (?), which would open up the Thunderbolt port.

Or you could get a FireWire 7200rpm drive and settle for that. I've got a 7200rpm G-Drive and it seems reasonably fast with After Effects, although I have no idea how it really stacks up against thunderbolt and/or SSD.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: privatebydesign on April 25, 2013, 08:26:16 AM
Any decent TB device can be used as a daisychain, just plug a storage device into the computer TB socket then plug your display into the storage devices second TB socket.

I am running an older MBP 17" with an eSATA card. I have a 256GB SSD in the original HDD space, this is partitioned into two, a 200GB operating system and apps drive, 50+ as a scratch disc and temporary storage place. I then have a 1TB HDD in the place of the CD/DVD tray, this works really well, so well in fact that I got a Mac Mini and did the same thing to it as a "home" desktop.

I have six external HDD's attached to the MBP, two via eSATA, the other four via daisychained Firewire 800. I have no RAID setups. I use Carbon Copy Cloner and Time Machine to cover my backup tasks.

Use a dedicated disc for Time Machine, it is a great program that will save your butt. I use CCC for everything else, including fully bootable backups of both internal HDD's and all external drives copied, apart from the disc I use for TM.

The real problem with DROBOS (which has a very mixed review history) and most other one cable RAID solutions, they use proprietary software to write your info to disc, this means if you have a serious gear failure the only way your info can be read is by sending your HDD's off to the manufacturer and paying them to get all that stuff back off your discs that they encoded!

That is why I use CCC to make simple copies, at a block level, of my HDD's as backups. Just plug any copy into your computer and it works, everything is there, including working app's, preferences, everything.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 25, 2013, 11:04:40 AM
Dang, wish I'd seen that Lacie option earlier....

I found a sale and went with this:

http://www.newegg.com/Product/Product.aspx?Item=N82E16822108095 (http://www.newegg.com/Product/Product.aspx?Item=N82E16822108095)

Just to have something to play with and give me room for now.

I'll try it on the network as a NAS, but if that isn't speedy enough hooked to my router, I was thinking of trying to maybe get a crossover cable, and plug it DIRECTLY into my macbook pro's ethernet port. If both are gigabit...I'd think that would be fast enough to work with?

I'd just use wireless on the mac for network connectivity in place of the wired ethernet port....wondering if that might work?

Anyway, this is a stopgap measure, and I'll be able to use this unit repurposed later....the deal ended today with the code, I got it for $184 which was about as cheap as I'd seen around anywhere.

I should get it in on Sat and will play with it. Again, if it doesn't work fast enough, I can use it on the network for other things.

Thanks for the insight so far....I'm still researching a better end solution...

cayenne
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 25, 2013, 11:06:57 AM
I have a great suggestion for you to look into.  I wouldn't suggest RAID 1 either because what you gain in speed, you lose in protection.  Why back up only to lose it all- right?



sek



I'm a little confused.....maybe I need to review my knowledge of RAID levels.

I thought RAID 1 was mirroring.....so, if I lose one drive, I still have the mirrored one there to rebuild with?

cayenne
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Mt Spokane Photography on April 25, 2013, 11:07:08 AM
I have a 12TB NAS but its not the way to go for editing directly from it.  Even with two 1 GB network cables in parallel, its slow for editing.   I won't even edit still directly from it unless its urgent.
Its great for storing files and backups, which is what I use it for.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: ahab1372 on April 25, 2013, 11:11:44 AM
NAS is great to store files. I can also recommend Synology.

For the you are currently working on (especially video projects), I'd consider an external drive attached with Firewire 800, USB 3 or Thunderbolt (not for the 2011 Macbook though). The network connection of a NAS, even over a Gigabit network, could still be a bottleneck, depending on the networking performance of the NAS and/or your computer.
Use the internal drive as the Scratch disk, and keep the video files on the external drive until your done. Use the NAS for long term storage and backup.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 25, 2013, 01:41:49 PM
I have a 12TB NAS but its not the way to go for editing directly from it.  Even with two 1 GB network cables in parallel, its slow for editing.   I won't even edit still directly from it unless its urgent.
Its great for storing files and backups, which is what I use it for.
What do you recommend for working from them, that is external to the computer (in my case a laptop)?

TIA,

cayenne
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Mt Spokane Photography on April 25, 2013, 02:04:19 PM
I have a 12TB NAS but its not the way to go for editing directly from it.  Even with two 1 GB network cables in parallel, its slow for editing.   I won't even edit still directly from it unless its urgent.
Its great for storing files and backups, which is what I use it for.
What do you recommend for working from them, that is external to the computer (in my case a laptop)?

TIA,

cayenne

It sounds like using a thunderbolt drive and daisy chaining it to your monitor is preferred.  Thunderbolt can make any disk faster, but with a SSD, it will be really fast.  USB3 is fast as well, but I believe that you don't have it.
Firewire 800 might be a choice, but its kind of a dead end with Thunderbolt taking over.
The limiting factor is USB2.  GB Network is faster, I can occasionally get 110MB/sec pc to pc over mine with SSD's in both computers, ther issue is thats its not always that fast dependinng on file type.  Installing a SSD in my computer gives me close to 500MB/sec with a SATA 3 port.  You should get that speed with a Thunderbolt external SSD.
 
I use a Qnap NAS with six 2TB disks and Raid 5.  Speed depends greatly on file type, large uncompressed files are very fast, while compressed files are much slower. 
 
If you can afford it, get a Thunderbolt Drive and Daisy Chain.  Make sure you can return it if it doesn't work with your laptop.
 
http://www.amazon.com/Seagate-Thunderbolt-Desktop-External-STCB3000400/dp/B009HPGBNY/ref=sr_1_4?ie=UTF8&qid=1366912959&sr=8-4&keywords=Thunderbolt+hard+drive (http://www.amazon.com/Seagate-Thunderbolt-Desktop-External-STCB3000400/dp/B009HPGBNY/ref=sr_1_4?ie=UTF8&qid=1366912959&sr=8-4&keywords=Thunderbolt+hard+drive)
 
 
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: dolina on April 25, 2013, 02:51:11 PM
SSD may make a better external scratch disk.
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: viggen61 on April 25, 2013, 03:07:57 PM
For your MBP, Thunderbolt is the way to go. No, you can't daisy-chain from your monitor, but if you select the right bits, you can have the monitor at the end of the chain.

There are several Thunderbolt Hubs (plug that into Google, and look at this:http://www.macrumors.com/2013/04/09/caldigit-thunderbolt-station-adds-to-growing-ranks-of-thunderbolt-docks/) out there these days. Belkin, Matrox, CalDigit and a few others. Some have many different ports, others are one-trick ponies.

I have a Retina MBP, so I have two TB ports, and USB3. For external storage, I went with LaCie Thunderbolt to ESATA hubs ($180-$200 plus cables. I got my last as a refurb from LaCie for $99.) These connect to your TB bus, and you can connect two external drives via ESATA. I chose this mainly because I already had drives with ESATA (I was using the FW800 ports on them previously). They also have two TB ports, so I now have one TB-ESATA daisy-chained off another TB-ESATA. I could still plug in my external monitor with a MDP-DVI adapter, or another TB device.

Now I didn't really need extra USB3 ports (I use a Belkin hub for those), but many of the TB hubs will have some combination of FW800, Ethernet, USB3, USB2, MDP, ESATA, etc. You'll need to shop around for the one that you like best. One added bonus, is if you take the MBP places, you only have to disconect one TB cable!

I have to say that in addition to a much faster MBP (quad i7/16GB/512GB), the drive I/O speed has dramatically improved by putting everything on either USB3 or ESATA. Dumping 1000+ RAWs, with rename and back up, from a 32GB card is very quick with the Hoodman USB3 reader and UDMA7 CF cards. Oh, and my only solid-state storage is in the MBP with just the system and apps. Everything else is on spinning platters...
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: cayenne on April 29, 2013, 02:19:40 PM
For your MBP, Thunderbolt is the way to go. No, you can't daisy-chain from your monitor, but if you select the right bits, you can have the monitor at the end of the chain.

There are several Thunderbolt Hubs (plug that into Google, and look at this:http://www.macrumors.com/2013/04/09/caldigit-thunderbolt-station-adds-to-growing-ranks-of-thunderbolt-docks/) out there these days. Belkin, Matrox, CalDigit and a few others. Some have many different ports, others are one-trick ponies.

I have a Retina MBP, so I have two TB ports, and USB3. For external storage, I went with LaCie Thunderbolt to ESATA hubs ($180-$200 plus cables. I got my last as a refurb from LaCie for $99.) These connect to your TB bus, and you can connect two external drives via ESATA. I chose this mainly because I already had drives with ESATA (I was using the FW800 ports on them previously). They also have two TB ports, so I now have one TB-ESATA daisy-chained off another TB-ESATA. I could still plug in my external monitor with a MDP-DVI adapter, or another TB device.

Now I didn't really need extra USB3 ports (I use a Belkin hub for those), but many of the TB hubs will have some combination of FW800, Ethernet, USB3, USB2, MDP, ESATA, etc. You'll need to shop around for the one that you like best. One added bonus, is if you take the MBP places, you only have to disconect one TB cable!

I have to say that in addition to a much faster MBP (quad i7/16GB/512GB), the drive I/O speed has dramatically improved by putting everything on either USB3 or ESATA. Dumping 1000+ RAWs, with rename and back up, from a 32GB card is very quick with the Hoodman USB3 reader and UDMA7 CF cards. Oh, and my only solid-state storage is in the MBP with just the system and apps. Everything else is on spinning platters...
Hmm...very interesting. A question bout your drives...are these single, external drives in enclosures you're talking about? Just trying to get a picture of what exactly you're plugging into the thunderbolt hub.

Thanks!

C
Title: Re: What external 'scratch' disk setup do you use/recommend?
Post by: Mt Spokane Photography on April 29, 2013, 02:30:29 PM
For your MBP, Thunderbolt is the way to go. No, you can't daisy-chain from your monitor, but if you select the right bits, you can have the monitor at the end of the chain.

There are several Thunderbolt Hubs (plug that into Google, and look at this:http://www.macrumors.com/2013/04/09/caldigit-thunderbolt-station-adds-to-growing-ranks-of-thunderbolt-docks/) out there these days. Belkin, Matrox, CalDigit and a few others. Some have many different ports, others are one-trick ponies.

I have a Retina MBP, so I have two TB ports, and USB3. For external storage, I went with LaCie Thunderbolt to ESATA hubs ($180-$200 plus cables. I got my last as a refurb from LaCie for $99.) These connect to your TB bus, and you can connect two external drives via ESATA. I chose this mainly because I already had drives with ESATA (I was using the FW800 ports on them previously). They also have two TB ports, so I now have one TB-ESATA daisy-chained off another TB-ESATA. I could still plug in my external monitor with a MDP-DVI adapter, or another TB device.

Now I didn't really need extra USB3 ports (I use a Belkin hub for those), but many of the TB hubs will have some combination of FW800, Ethernet, USB3, USB2, MDP, ESATA, etc. You'll need to shop around for the one that you like best. One added bonus, is if you take the MBP places, you only have to disconect one TB cable!

I have to say that in addition to a much faster MBP (quad i7/16GB/512GB), the drive I/O speed has dramatically improved by putting everything on either USB3 or ESATA. Dumping 1000+ RAWs, with rename and back up, from a 32GB card is very quick with the Hoodman USB3 reader and UDMA7 CF cards. Oh, and my only solid-state storage is in the MBP with just the system and apps. Everything else is on spinning platters...
Hmm...very interesting. A question bout your drives...are these single, external drives in enclosures you're talking about? Just trying to get a picture of what exactly you're plugging into the thunderbolt hub.

Thanks!

C

Although I looked at the LaCie Hub at first, careful reading reveals that its e-SATA ports are SATA2, not SATA3.  That makes a huge difference in speed with a modern hard drive.
 
If you get a hub, make sure it supports TB Drives, or SATA3.  I would only purchase SATA 3 drives as well.  I just finished upgrading all of mine to SATA 3 and they are up to twice as fast with my SSD reading 559 and writing at 492 MB/sec.