ZFS L2ARC

revodrive

OCZ RevoDrive 80GB PCIe

Nice hardware but sadly quite slow by modern standards: 75 000 IOPS. Modern SSD can achieve similar performance. [Edit: as mentioned later on, the figures SSD manufacturers provide are misleading so the question remains: can this device provide constant 75 000 IOPS in which case it is much much better at that]

See Wikipedia

But it gave me an idea.

sonet

This one can house two SSD. It would leave me two on-board SATA completely free and server could still have SSD.

That would be the absolute best solution. It would mean I can have two Western Digital Re 3TB with two SSD which would give me double the IOPS vs. one and hence extreme performance on my pool.

While doing research on what would be the best SSD IOPS-wise, I found out the manufacturers exaggerate their IOPS figures. According to this the figures they give can only be achieved on a new drive for very short periords of time.

Some manufacturers seem to provide Sustained figures which use some sort of standard-type agreed-upon way of measuring the real IOPS figures, while others don’t.

OCZ gladly gives these figures for their Vector 150 drives but sadly 12 000 steady random writes isn’t that good at all.

And while this is quite an entry-level SSD and there are better it is of little use until they start to provide these measurement figures.

And one cannot trust the tests done by testers unless they understand this and run their tests for extended periors of time! Because otherwise we may get skewed results.

But the best plan still would be to get as much RAM as possible, then of course get as much disk as possible, and finally have fast FLASH storage for anything that spills over the RAM.

And with ZFS and FLASH cache I could perhaps even use consumer grade SATA to save money and still have reliability and performance.

Let us calculate for the fun of it.

Western Digital Red 3 TB, 2 pieces for 250 €
Sonnet Tempo SSD Pro, 250 € delivered

And it seems by comparing OCZ Vertex 460 120GB and OCZ Vector 150 120GB that they use similar technology on both of these since steady-state random write figures are exactly the same.

Looking at other parameters we go with the Vector 150.

OCZ Vector 150 240GB, 150 €

Which would give 6TB of storage with 240GB of L2ARC for 650 €.

Compared to original plan which was to buy 3TB enterprise quality SATA and combine it with 250GB SSD which would cost 180 € for Re4 3TB and 107 € for Samsung 840 EVO 250GB for a total of 287 €.

Now that’s a difference!

Good, bad? That certainly is a very good question. But the 650 euro one would perhaps provide much greater performance.

2.3 times the price for 2.0 times the capacity. The 0.3 should then be covered by the fact that there would then be empty slot for another SSD and by the performance increase.

So I think the 650 euro deal is the better one.

And with 4TB WD Desktop Mainstream one would get 8TB of storage for an additional 100 euro.

So 750 euros for 8TB pool with 240GB cache. That’s a shitload of money.

But still, that is only 9,3 euro cents a gigabyte of high-performance and reliable storage. 9,3 cents for gigabyte 10 years ago was considered cheap, and it was for hard drive only.

Raw enterprise storage would with Re4 3TB be 6 cents. So there is definite margin there.

But comparing these (raw vs. real setup) is quite useless. One can buy storage and that’s it but one can’t then just get the performance out of it.

Upgrade the disks to 5TB, add in another 240GB cache for a total of 480GB (or even 1TB) and you have 10TB pool with 1TB cache.

What would that cost?

WD Red 5TB, 2 pieces for 410 €
OCZ Vector 150 480GB, 2 pieces for 530 €
Sonnet Tempo SSD Pro, 250 € delivered

That would be 1190 € for 10TB high-performance, high-reliability storage pool.

Talking about enterprise or business money that is peanuts and nothing.

Edit: I made a mistake where I sacrificed reliability for money saving since I figured I won’t need faster disks because I have SSD to cache. But I still need reliable disks to achieve that reliability.

So the prices will go up by perhaps 20% since reliability still requires Re4 enterprise level disks for raw storage. Bonus from this accident is increase in performance.

So, now we would have 8TB pool with 1TB cache for 480 € for 2 pieces of Re4 4TB, 530 € for 2 pieces of OCZ Vector 150 480 GB and 250 € for Sonnet Tempo SSD Pro for a total of 1260 €.

Which of course is some 70 euros more for 2 TB less and for 20% larger MTBF, the performance increase and two years longer warranty time.

WD Red

[pdf]http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-771442.pdf[/pdf]

WD Re4

[pdf]http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701338.pdf[/pdf]

Seems to also have an order of magnitude lower rate of Non-recoverable read errors so go figure.

Disks for surveillance use I would never touch because those are made for video stream which isn’t too sensitive about bit flipping and other errors which can in the context of video be considered minor errors.

Finally memory

$_57

I don’t have money for this but this deal is so good I had to take it away.

Apparently IBM certified Samsung memory, 4 GB modules for 16 GB, $10 shipping for total cost of $70.

There may be some $10 import charge but it is still cheaper than anything on eBay I’ve seen.

Sells for 20 € a piece in German eBay without postage so big savings. And if there will ever be need for more than 16 GB it is great to know there are same modules available.

8x2GB would have been cheaper by perhaps $30 but then if I need more than 16 GB I end up with useless modules. And these 4 GB have better value.

And I get the Quad gigabit NIC next week so that too. Different server though; this one already has 12 GB. Maximum for DL380 G3.

But it seems FB-DIMM is a bit slower than what you might call traditional memory.

This means workloads that use many writes (such as high-performance computing) will be significantly slowed. However, this slowdown is nowhere near as bad as not having enough memory capacity to avoid using significant amounts of virtual memory, so workloads that use extreme amounts of memory in irregular patterns might be helped by using fully buffered DIMMs. –Wikipedia

So it is a technically inferior solution in this sense. But I doubt if my possibly 8-core setup would count as high-performance computing; I guess it could; that’s what we are aiming here for.

Samsung 840 EVO vs. WD VelociRaptor

So, I have been reluctant to go with the SSD because I have felt they are too consumer-ish for me.

But looking at the reviews it seems that is foolish because even these consumer grade SSD seem to outperform VelociRaptor in every possible scenario.

But there is one thing to keep in mind at least with this particular TLC one, which is apparently number of NAND chips.

The number of chips affects the speed as seen on this very illustrative graph:

turbowritesm

AnandTech

The big drops are where this drive’s “turbo” runs out, which essentially is faster SLC type of memory used for writes. All writing is first done to this fast SLC type memory from where it is “slowly” copied to TLC memory.

But that is not the point.

The point is as you can see 120 GB model’s performance which is absolutely poor. That one would loose in throughput to VelociRaptor whereas the 250 gibibyte model would overpower it.

And not only throughput but latency-wise SSD will easily outperform VelociRaptor. From what I have seen VR has latency of some 5-6 milliseconds where these have latency in range of microseconds.

Erase latency is higher at 2-3ms but still lower than any HDD I have seen. Maybe some 15K SAS excluded. Don’t know.

Also IO operations far outperform mechanical disks. So in retrospect I would have been stupid and naive to think VelociRaptor would be better choice for, what practically would be operating system type workload (virtual machines); lots of small IO operations.

And even if the throughput was lower on this SSD (it is not) it would still be better, performance wise, to have more IO vs. more throughput.

So it seems I will go with Western Digital Re4 3TB for big storage and backup, and have 250GB for virtual machine images.

In addition to this, this SSD will provide L2ARC cache for my ZFS. So it is absolutely win-win-win situation. And more L2ARC should mean less RAM, or at least I can afford to use more RAM for VM without serious sacrifice on underlying ZFS performance.

Goes to show the importance of re-evaluating your own opinions and held beliefs, since I lived under the impression that my server would be better because it would have had 10K state-of-the-art HDD when, in fact, relatively consumer-grade SSD is still going to be better choice.

TLC of course will suffer from wearing since it will loose its ability to distinguish between three different voltage levels faster than it would if there were two (MLC) or one (SLC) voltage level. But even with this it would still last for years and years and years EVEN with hundred or so gigabyte daily write rate.

SWAP on small portion of TLC SSD could prove to be bad idea, but then again if one’s server is swapping that much that it will eat away an SSD, then that probably isn’t the biggest of one’s problems.

And here’s another nail on the coffin of WD VelociRaptor:

untitled-3

 

Guru3D

I do not even care what this PCMark tests but the language of this speaks for itself no matter what it measures basically.

And when you look at the table here you can see even the MTBF is bigger than that of VelociRaptor’s. Given of course these are calculated in a somewhat hazy manner usually as far as I have heard.

blockdia

 

And another:

untitled-5

Which clearly shows it is an order of magnitude improvement over old technology and it would be shame to not take full advantage of this.