Revisiting the home data center architecture

If all goes well I will be adding one or two extremely powerful and new servers in the coming months.

Those servers use 2.5″ disks so the only question is how to implement large scale storage system. I have an old E6600 based server which would be perfectly fine if two 1Gbit connections were trunked together to get 2Gbit iSCSI connection.

2TB in 2.5″ form factor seems to be most cost effective, and prices for 3TB are beyond economical. So if one server could take 4 disks that would in mirrored configuration give 2TB of storage with some faster storage in form of SSD; left over from L2ARC and SLOG.

The old DL360 G3 would be dedicated to only work as firewall and traffic shaper and routing and switching would be moved to dedicated managed gigabit switches.

Also now all servers boot from NFS which has proven to be good, but problematic in case of failure in that NFS server, which has potential to either lock or bring down all the other servers. So NFS would be removed in favor SSD based mirrored ZFS root.

One question mark is my current networking setup which relies heavily on Linux, and which would need to be ported to managed switches. It shouldn’t be a problem, though, since it is technically all VLAN based with some bridges with more specific rules; so those would need to addressed somehow.

Also something like pfSense could be considered. But with firewall and router, if such system is used, I would like to move from i386 to 64bit architecture because currently there have been problems with not enough memory. HP ProLiant DL380 G5 might suit the purpose perfectly as a low cost server.

Quad gigabit PCIe network cards seem to be quite cheap so with three slots it would act as 12-port gigabit router. That would enable either the current Linux-based routing scheme or transition to something like BSD based pfSense. BSD has a reputation of being network oriented system and some studies have demonstrated that it performs extremely well as a router.

But one thing to remember with Linux/BSD based routers is to make absolutely certain that the driver support for network cards is perfect. Otherwise the stack will fall apart. Dedicated routing hardware works perfectly because it has been built to match perfectly with what it was built to be — router and nothing more.

So if the new QEMU/KVM hypervisor would set me back 400 €, disks perhaps 500 €, router 300 € and one or two additional small switches yet another 200 € and 1400VA UPS 250 € then the price tag woud be 1 550 € which isn’t too bad.

That cost would hopefully give me room for another 3 years at least and 2TB of storage and possibility to expand that storage to 14TB by using the router as FC based storage node by dropping 4 gigabit ports to accomodate for the FC card.

Novel idea for making use of spare memory slots

Look what I found while browsing DRAM based memory devices:

satadimm_ssd_illustration

More here: http://www.vikingtechnology.com/satadimm-ssd

It isn’t using the memory bus to transmit the data but it is using it for its space! That’s one problem I am facing because 1U servers are so compact that there isn’t much to spare. So that’s a very novel idea but for very limited use cases.

But perhaps if you need to do tricks like this it is an indication that you need new hardware. I do not know what the margins here are or if the product is otherwise superior, but my feeling is that once you need to go for these sort of things the better alternative might be to get new server.

But perhaps this would work fine for some extremely urgent situation where you absolutely need the space but cannot find any other way to do it. Or some other odd situation like that. Perhaps you have server which was designed to be used as RAM server but you aren’t using it for that so you can equally well get a SATA controller and use those slots for disk space. It might save you the price of new server.

I quite like the idea.

ZFS: Deduplication on SSD pool

Works just fine unlike the HDD tests previously. I have no ZIL nor L2ARC for that pool but because DDT is on SSD and is therefore fast, the problem of evicted DDT from ARC doesn’t become such an issue.

DDT-sha256-zap-duplicate: 130595 entries, size 286 on disk, 141 in core
DDT-sha256-zap-unique: 841291 entries, size 301 on disk, 157 in core

dedup = 1.14, compress = 2.36, copies = 1.00, dedup * compress / copies = 2.68

Small 120GB SSD so that additional 14% saving there becomes handy.

Saving old, practically identical CentOS images seem to deduplicate quite nicely.

Edit: and after all the images were copied, the deduplication went up quite a bit, along wit the compression:

# zdb -D vm
DDT-sha256-zap-duplicate: 277827 entries, size 288 on disk, 141 in core
DDT-sha256-zap-unique: 1251538 entries, size 303 on disk, 158 in core

dedup = 1.36, compress = 2.49, copies = 1.00, dedup * compress / copies = 3.39

So it’s storing much more data than the whole drive is in size. Giving me essentially 170GB SSD for the price of 120GB. The server and setup used isn’t high-end and there is no need for superior performance so the hit from deduplication combined with heavy compression doesn’t affect me much.

The additional things that I can do with that extra 50GB is warmly welcomed.

Upgrade to local virtualization supervisor

Went and acquired cheap (65 €) Kingston HyperX Fury 120GB SSD just to test out how the performance improves.

Using it as L2ARC for ZFS but not for ZIL because it isn’t good enough for that. The rest of it which is left is dedicated to virtual host disk images and preliminary test gives quite impressive results.

snap1203

The disk itself is nowhere near capable of these sort of figures to the ZFS ARC must be interfering with these tests somehow. Or something else is going on. But the disk nor the 3Gb/s SATA are capable of 923MiB/s read performance. So that must come from cache of some sort. But they still should give somewhat accurate performance results for at least any ordinary work.

If I want to get ZILs online I am going to have to spend pretty penny because UPS will cost 250 € and two data center grade SSDs will probably cost an additional 250 € minimum. And they probably still won’t be SLC.

And these numbers are with GZIP-9 compression enabled. Theoretically that can add quite a bit to these numbers, depending on sort of data the application is writing. If it compresses well then these can certainly be real world figures that are coming from the disk.

I did some simple dd testing with Linux without any encryption (these are AES 256 bit results) and got read speed of about 280MB/s and write speed of about 230MB/s.

Ideal server

If I am willing to pay 150 € for the server, then the additional upgrades would come in as follows:

  • HP Smart Array P800
    • 50 €
    • The included E200i does not support JBOD
  • 5 pieces of 10K RPM SAS 300GB
    • 270 €
    • 600GB of fast SAS storage plus one spare
  • CPU upgrade to Dual Quad-Core X5470
    • 100 €
  • Intel 320 120GB
    • 40 €
    • To be used as sLOG

So the price of this new server which began as 75 € idea suddenly would cost over 600 €.

But that would then be quite a beast.

And then my home data center would also require UPS which would add another 200 €. It is good to have dreams.

Updated calculations

Still looking for this server but with modified specs:

  • HP Smart Array P800
    • 50 €
    • The included E200i does not support JBOD
  • 3 pieces of 10K RPM SAS 300GB
    • 170 €
    • 300GB of fast SAS storage plus one spare
  • CPU upgrade to Dual Quad-Core E5440
    • 55 €
  • Intel DC S3500 120GB
    • 117 €
    • To be used as sLOG

And the upper limit to pay for locally available pick-up only server — after which it becomes cheaper to get this from eBay — is 135 € and not 150 €.

Intel 320 does not seem to be very available any more so S3500 should replace it. Also some suggest that one could define all the disks in the original E200i controller as RAID0 and use them like that. It would save that 50 €.

So that would be about 520 € server for 22.64GHz of Core 2 Hapertown architecture with 32GB of memory and 300GB of mirrored 10K SAS with SSD sLOG.

Lack of Vt-d support

This is the big question. Am I willing to spend this much money when I would really like to get Vt-d. http://ark.intel.com/search/advanced?VTD=true

The next generation (G6) has additionally DDR3 and probably otherwise quite a bit more advanced as well. Also supports 12 cores. So I might put this G5 on hold because it would regardless cost close to 400 €. Perhaps if I can put in 200 € more I can get the G6.

DL360 G6 setup

The following DL360 G6 would cost about 650 € so only 130 € more than the G5 so the G5 is out of the question.

  • Base server with 8GB of memory
    • 210 €
  • Heatsink for second CPU
    • 38 €
  • CPU upgrade to Dual Quad-Core X5570
    • 50 €
  • Memory upgrade to 32GB
    • 72 €
  • 3 pieces of 10K 300GB SAS
    • 170 €
  • Intel DC S3500 120GB
    • 117 €

Surprisingly cheap. The disk is the most expensive upgrade here and could perhaps be postponed. I need to be on the look for good base on top of which buy the upgrades. That seems to be the cheapest route to go.

Little bit better setup

Replacing the three spinning SAS disks with two OCZ Vector 180 240GB would reduce the cost 6 € but more importantly guarantee superior performance under all conditions. 60GB less space and no spare but also no used disks and all brand new and extremely fast.

There of course was a catch behind that price:

For cost reasons, OCZ didn’t go with full power loss protection similar to enterprise SSDs and hence PFM+ is limited to offering protection for data-at-rest. In other words, PFM+ will protect data that has already been written to the NAND, but any and all user data that still sits in the DRAM buffer waiting to be written will be lost in case of a sudden power loss. (http://www.anandtech.com/show/9009/ocz-vector-180-240gb-480gb-960gb-ssd-review)

So we are back to square one. The alternate solution would be to use the Intel DC S3500 but that would limit the space even further, to 120GB. At which point we come close to being so tight that not all capability of the server can be realized. If you want to test something you must have disk space available to you. 300GB in addition to super fast SSD is for that.

What about aes-ni?

I am glad you asked, because it turns out that the three six-core Xeons supported by the G6 have aes-ni. The downside is that they are 6 cores and 12 threads (and cost $100+ each) and with that there is nothing for me to burn those cycles with. Currently if I do disk IO, all my traffic goes through AES encryption and if that is speeded up, there is really nothing to spend the CPU capacity on.

So that’s a positive problem. At this point it would seem that the best way would be to go and purchase that Intel DC S3500 and use it to greatly increase the performance of my current DL140 G3.

I love this planning but I also hate it because there are so many ways to go.

Eyeing on DL360 G5 server

Looking if I can get one of these for cheap price.

c04284194

It would be without hard drives but with 32GB of memory and both CPU sockets occupied with something. One other bidder with 25 € and 8 days to go. Not sure if there’s a minimum price, which I suspect there might be, but I would be willing to bid 100 € easily.

And according to this video it is also extremely silent one.

So would make great desktop machine to replace my old one or to replace my virtual host which is X5350, while this could support X5470.

And maximum memory is 192GB so that would be perfect for even commercial system. Apparently supports only 64GB but that would still be double my current maximum of 32GB.

Four SAS/SATA ports would also allow raidz1 + one SSD for caching and would deliver extraordinary performance with 15K disks.

One sad thing is that if my memory serves correctly, 5400 series processors do not support PCI-passthrough so it would not be possible to virtualize Windows on top of Linux and get native graphics support. Same for network but these are minor issues.

Calculating the cost

I am beginning to warm for the idea of paying upto 150 € for this machine as it has 32GB of memory in it which is worth 50-70 €.

The rest of the upgrades to get either desktop of server would cost something like this:

  • CPU upgrade from 4 low-end cores to 8 high-end cores 120 € (X5470)
  • SSD upgrade for desktop/server use 70 € (half decent Corsair)
  • 15K RPM SAS disk upgrade for server 140 € (3x300GB)

So fully eguipped with 32GB for high-performance server use the total cost would be close to 500 €. It would be more cost beneficial to buy two SSDs, put them in mirroring mode and then use cheap iSCSI storage.

That would lower the cost from 480 € to 340 € which is quite a good deal for two SSDs and safety and high performing machine. So I don’t know. If it goes for under 150 € I am quite certain I will take it to myself.

Attachments

The Best Solution

I have clearly been thinking too much but I think I have finally reached some sort of conclusion with the best possible performance and it comes with low price and large window for expansion.

My worry was that I have PCIe v1.x which would mean that those cheap x1 cards would run only at 250MB/s which would become bottleneck for an SSD.

And now that I actually looked at it that would have in fact been the case.

Intel 5000X chipset which I have is PCIe v1.1 which is 250MB/s just like the original PCIe is.

So, that rules out any x1 card. And quite a number of other cards as well. In fact, x8 or x16 are the only cards that would provide wide enough bandwidth, say, if I would want to run more than one SSD.

So the only viable option for me are either the expensive NAND FLASH PCIe devices or what I think is much better option, the professional battery backed SAS/SATA RAID controllers.

Exactly like I initially figured.

These usually come in as x8 so that would give me 2GB/s full duplex which would be enough for 4 SSD fully saturated. Situation very unlikely to happen.

I may even sacrifice the x16 for this since I can’t think of any other use for it.

adaptec-12port

ASR-31205 ADAPTEC 12 PORT SAS SATA PCIe RAID CONTROLLER

Which would give me 12 SAS/SATA ports. More than enough for anything I can think of.

The 2U rack mount case can’t even accomodate that much extra disks even if SSDs and how small they are so that’s the end of that problem.

But still, could easily fit in at least 6 if I needed. And split the 3-5 watts of power for those off of the two SATA power present.

And it has support in Linux.

Or ASR-2405 which at times are available for less than $30. Lacking battery but could live without one.

2405

RAID-Controller Adaptec 2405 PCI-Express 4x SAS/SATA

I could have either of these. And 4 ports would be more than enough, and would save some money which would buy me more SSD.

But this would be best because it has 6Gb/s per port and 8-lane PCIe so it could handle the SSD speed by the port speed and have it transferred forward at great speed.

There is 4-lane model but it would be just enough for two fast SSDs and might just become the bottleneck.

6805

Adaptec RAID ASR-6805 2271200-R 512MB 6Gb/s SATA/SAS 8i-Port Controller Card

Sadly this costs about 150 €. So the 8-lane 3Gb for 40 € with 375 MBps per SSD or 150 € for full potential of any SSD.

Adaptec product page.

Pure SATA vs PCIe SSD/FLASH

That is another question.

Many a company seem to produce PCIe devices that host similar FLASH memory that SSD drives are made of.

Need to look into those as well.

The best solution I believe would be to buy fast PCIe FLASH in quantities of hundreds of gigabytes and then buy as large HDDs as possible (enterprise).

But since this would cost thousands of euros the next best thing is to buy single HDD largest one can afford and stick next to it the next best SSD one can afford.

Then when money comes in, buy another HDD, simple PCIe SSD card, stick the SSD to it and if it works then fine, and if it doesn’t then get the Sonnet and get another SSD to double the space, speed and IOPS.

The cheap ones are like this:

pcie-ssd

And there is very little hardware in them. But it might just do the job, barely. Or perhaps poorly, but one would only loose 30 € and best case save 250 € for not needing to buy real hardware.

Looking at hardware differencies there isn’t really that much to talk about:

sonnet

Because it almost looks like there is exact same chip as there is on this cheap one with fifth the cost of Sonnet:

pcie-ssd2

Perhaps just little less filtering and noise reduction or something. Sonnet has the possibility to add another SSD with included expansion module.

Then there apparently also is another Sonnet which has eSata for yet another additional drives.

But I believe I will get the cheap one first to test it out and if it works then that will be fine since it is only for caching and if it doesn’t work then I will sell it away and buy a real one.

And when everything fails I get this one:

ocz-1600gb

1600 GB just over $7000.

OCZ Z-Drive R4 C Series PCI-Express SSD CM88 1.6TB ZD4CM88-FH-1.6T

OCZ_Z-Drive_R4_88