Jump to content
Division-M Community
Sign in to follow this  
esac

Caching Hard Drive

Recommended Posts

I have a spare 30 GB SSD lying around. Not enough to really put any data on, but would love it if DriveBender can use it as a cache. Keep the most recently used files on it for reading, and all writes go to the SSD, and then get moved on periods of low disk activity to the larger pool.

Share this post


Link to post
Share on other sites

You are still bound by the max you can get from a single Gigabit connection so this is pointless.

 

Also lets say this was in place and you had to copy 50GB of files onto the WHS it will fail with a not enough free storage as it dumps it on the cached disk first then writes over time to the pooled drives.

 

If you want my advice take that ssd and stick it in your desktop or laptop instead don't waste it on whs.

Share this post


Link to post
Share on other sites

You are still bound by the max you can get from a single Gigabit connection so this is pointless.

 

Also lets say this was in place and you had to copy 50GB of files onto the WHS it will fail with a not enough free storage as it dumps it on the cached disk first then writes over time to the pooled drives.

 

If you want my advice take that ssd and stick it in your desktop or laptop instead don't waste it on whs.

 

Wow, so much misinformation here.

 

1. Who said I was copying across the network.

2. Even if it was across the network, disk transfer rates for a 7200 RPM fall within the 100 megaBYTE range for sequential reads and writes (800 megabit) with no other disk activity occuring, so still not enough to saturate the gigabit (admittedly it does get close, but again this is with no other disk activity). Random reads and writes do not even come CLOSE to saturation.

3. If you were copying 50GB of files onto your server, DriveBender is already smart enough to know where to place a file if it is too big for the destination pool drive, so it might write the first 20GB to the SSD, and then future writes would go to the larger drive.

4. Not everybody here runs WHS (which is a poor assumption on your part). I run Windows 7 Ultimate with Media Center, streaming to 3 XBOX 360's throughout the day, backing up 4 laptops worth of data via Windows Live Mesh (bunch of small file reads and writes), and uploading that all to Crashplan. It also acts as a file and print server, although not as much.

5. All of my other computers do have SSD's, but even if they didn't, 30GB is not really enough to even put an operating system on nowadays, so unless the system had SRT, I wouldn't even consider it (and even then as i've had issues with SRT).

Share this post


Link to post
Share on other sites

Windows does a caching. in the OS its self, Why re invent the wheel. But as someone already said if you did not have enough drive space on the drive it writes to after the file is no longer is use would render you system redundant. Swap files are incorporated in the OS why change it. I believe that you can in large the swap file space or in the new windows OS they are call it " Paging File Size" which can be changed.

Share this post


Link to post
Share on other sites

Windows does a caching. in the OS its self, Why re invent the wheel. But as someone already said if you did not have enough drive space on the drive it writes to after the file is no longer is use would render you system redundant. Swap files are incorporated in the OS why change it. I believe that you can in large the swap file space or in the new windows OS they are call it " Paging File Size" which can be changed.

 

Hate to say it, but a ton of misinformation here as well.

 

Windows does have a file system cache. Typically this is enough so that if you are accessing the file within a short time frame (temporal locality), then it will utilize that cache. It is an in memory cache, and is actually rather small.

 

It has nothing to do with 'swap files' or as windows calls it 'paging files'. In fact, paging files are used as a backing store for code and data that resides in memory, so that when there is a demand for memory that is higher than what the system has, it can satisfy that from disk instead. Paging is actually BAD in most sense of the word (better than running out of memory, but still you would rather get that code/data from memory rather than disk). In fact, what you are saying makes  no sense, that instead of going to disk, it will ... uhm ... go to disk???

 

Anyway, even if that all made sense (it doesn't), then that solves nothing for writes. Windows can not cache data it doesn't know about which is what happens when you write a file.

 

Finally, as i've pointed out before, Drive Bender is already architected to take disk space into account. What do you think happens when you have a 1TB and a 2TB drive in the same pool? Eventually, 1TB will be used up, and all subsequent files will go to the 2TB drive. This scenario makes no difference.

 

Writing to an SSD drive makes a HUGE performance improvement and I think this functionality will set DriveBender apart from competing products.

Share this post


Link to post
Share on other sites

I'd like to add another scenario where this is a good idea.  I ended up stumbling across this happening and was incredibly happy.

 

Please bear with me.  This may be a long story/explanation.

 

I have a media server that has a number of internal drives and a LOT of external USB 2.0 drives (27 megs/sec max).  The external drives are split as such: port from computer to hub, hub to 5 drive enclosures, each drive enclosure has 2 drives.  Currently there are 20 external 2tb drives in 10 enclosures utilizing 2 USB ports on the system.

 

The external drives are used to store my video files and have always ran perfectly.  The only glitch I ever have is when I'm watching something on XBMC and then my system tries to copy a file to one of the drives in the chain.  During the copy, the read speed drops so low that I have to pause the video, allow the copy to finish and then resume the video.  Sometimes this could take minutes.

 

I purchased Drive Bender a few weeks back and decided I was going to try and create some pools to make my life a little easier.  The internal drives became one set and the externals will eventually be 2 sets.  Currently I'm only done with one external pool (reason is next).

 

During my creation of pools I realized that Drive Bender was never made to create an empty pool and merge a drive into it without it already having a drive in the pool.  For instance, I tried to create a "personal" pool and merge a drive with data on it to that pool.  Because the personal pool didn't already have a drive mount, I was unable to do this.  I ended up using an internal "temp" 500g drive as my seed drive for the pool.  I added the temp drive to it, then merged the new drive in and then removed the temp drive from the pool.  Worked great.

 

So, I had to do the same pattern for my first external pool.  I had 6 drives all ready with data.  Created a pool called "tv" that had the internal 500g drive (which by the way, is on a 6gb link and can transfer at 130 megs/sec easy) and then merged, one by one, the external drives.  This, once again, worked perfectly.  I then felt like watching tv after the last drive was installed (had to test it, of course).

 

While I was watching XBMC on the tv pool, my system ripped another file to put in the pool.  Since I didn't remove the initial seed drive (which was completely empty) from the tv pool, the system used it to store this copy.  During the saving, I never had a single hiccup in the streaming of what I was watching.  I didn't have to pause anything.  This went on for hours.  Never once did I have to pause.  This was amazing...

 

So, in all of that, what I would like to ask for is: Utilizing a designated drive as a "cache drive" and then move files from it (to get it empty) on a schedule.  That way I could have a very fast internal drive receiving my copies while I use the very slow external drives to watch shows.  Then at 2am, when I'm not watching, the system would move all of the files from the internal drives to the others.  It'd be the best of both worlds.

 

Thanks for reading if you got this far.

 

James

 

TL;DR: Use designated drive as a cache then move files off of it on a schedule to speed up overall performance.

 

[EDIT] Another user requested this same scenario in another message titled "Landing zones".

 

Share this post


Link to post
Share on other sites

+1 to this idea, especially jamesbc's explanation above regarding slow external drives!

 

Also, add an option to manually set the schedule for migrating data off the cache drive (maybe using the Windows Task Scheduler for plenty of flexibility?).  One example for how a user-controllable schedule would be useful:

 

I'm planning on setting up my new server with a combination of Drive Bender and FlexRAID's Snapshot mode.  The main idea is that it'll allow me to have duplicate+parity protection on my important files and parity-only protection on less-important ones like TV recordings (as opposed to Drive Bender alone, which would give me dups for important/no protection for non-important if I don't want to waste space by dup-ing everything).  The problem with this system is that as my data grows, FlexRAID will start taking a long time to do parity updates and would be much happier if my data wasn't being changed along the way.  If I could use a DB cache drive, I could have it migrate the data on a schedule every few days and schedule parity updates a few hours later so that the data would remain much more static during the update.

 

If my understanding of how DB works is correct (creating virtual "mounts" to intercept all data writes and re-direct them to the appropriate disk), it probably wouldn't be that hard to implement this idea.  All new writes would first be sent to the cache drive and put in a folder structure organizing where they belong (Pool1/Mount2/Folder/subfolder/).  Theoretically, a single disk could even serve as the cache for multiple Pools!  Then, using a schedule or triggers like disk activity level or cache drive % full, the data could be migrated from the cache to the appropriate Pool disk and duped if necessary (I'm guessing schedules would be easiest to implement, but the other triggers would be ideal for some users).

 

On second thought, this idea only works for new files.  I'm not sure how a cache drive would work for modifying existing files in the Pool.  I suppose the simplest option would be that any modifications/changes would be written straight to the Pool as they are now and the cache drive would be left out of the process.  I'm guessing most file modifications aren't that large anyway, so the main speed benefits would be for new files anyway.

Share this post


Link to post
Share on other sites

Wouldn't a RAMDisk improve performance over an SSD if attached as a cache drive?

 

Theoretically, a RAMDisk would offer a performance increase, but I don't think the practical difference would be that great and the downsides would, at least in my mind, far outweigh the benefits.

 

Assuming we're using the model I proposed earlier (new data goes on the cache drive and migrated later, changed data written directly to the Pool), here's how I imagine that working:

 

On the plus side:

-A RAMDisk would eliminate the bottleneck of writing data to the cache drive, so then we'd only be limited by how fast we can give it data to cache (either in generating new files, transferring files from non-DB Pool drives, or transferring over the network from other machines).  In any of those cases, I'm struggling to imagine a scenario in which we're able to send data to the cache drive fast enough to take any meaningful advantage of the RAMDisk over a decent mid-range SSD.

-That being said, I think a RAMDisk would offer a more meaningful performance difference when writing many small files to the Pool, which is an area that DB currently struggles with.  But again, you're still limited by how fast you can give it those small files to write.

 

On the negative side:

-RAMDisk capacity limits.  Even if you're running 32GB+ RAM in your DB Server, it's going to be tough setting aside enough memory for the RAMDisk to use without filling up right away.  Especially for those of us with only 4-8GB of RAM, the cache disk would fill up so quickly when transferring larger files that we'd have to start writing directly to the array again and lose all benefits.  Especially when you consider transferring large files like backup images, blu-ray rips, or HDTV recordings, a cache drive only starts becoming useful if you can fit multiple 10+GB files on it before it files up.

-Power outages (on systems with no or undersized-UPSs).  If there's a blackout while files are sitting on an SSD cache drive waiting to be migrated, everything that's already on the SSD is ok.  If the same thing happens with a RAMDisk cache, the entire cache is lost and unrecoverable.  Obviously, this is only for files that have already been written to the cache drive: anything still being transferred is going to get lost either way.

Share this post


Link to post
Share on other sites

I for one would love to see Drive Bender implement a Cache Drive feature, simply to improve network transfer speeds of commonly used files. After all, we tend to use Drive Bender for effortless and flexible mass storage and therefore our systems become servers for other NETWORKED PCs and it's the platter HDD that is now the bottleneck for your average gigabit LAN setup.

 

To me, an SSD Cache Drive is the perfect addition to a storage pool as its fast enough to easily saturate a gigabit LAN connection(even multi LAN Link Aggregtion), and also robust in that it's not volatile like RAM.

 

Drive Bender is half way there in that a landing zone has been added to increase write speeds (SSD preferred for this - guessing it has to be pooled though ?), now its time to finish it off by utilizing that same fast drive as a Cache Drive for reads.

 

Yes we could add an additional Cache Drive to the system (outside the pool) running off third party software (I think so anyway, anyone test this with Drive Bender ?) but it would be brilliant if Drive Bender went to the effort at some point to cover off full read/write performance as well as its great storage performance.

 

P.s. Please add it to the To Do list Anthony.

Share this post


Link to post
Share on other sites

another +1 vote

 

for me this is the only feature why i'm preferring storage spaces over drive bender.

 

it would be truly awesome if we could enable a write-back-cache and storage tiering like in windows storage spaces.

it definitly gives a huge performance boost.

 

using storage spaces the gain is depeding on your configuration at about 10times faster performance.

see this test for example: http://www.aidanfinn.com/?p=15430

or just google it, there are many similar tests getting the same effect.

 

so it's like files getting moved automatically to the "landing zone" if system recognizes high I/O from these files.

 

i hope you get this precious feature into final v2.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...