Fourth revision of personal DC

Testing new equipment:

RAID6 of 8 disks of 450GB capacity each and of 10,000 RPM. One of which is left as spare.

This project has been in development for little under a year. The thing holding it back has been disks: fast disks, be it [preferably] SSD or SAS, are prohibitively expensive. One might argue EqualLogic with 8 10,000 RPM disks isn’t any cheaper, but as we old [enough] people know: there are certain advantages working in the industry.

Setup procedure was extremely simple. Only problem came when I tried to use point-to-point for the two iSCSI connections; Linux [multipath] wanted to connect to both of the targets via both of the connections, which obviously wasn’t possible with point-to-point. But after a switch was introduced, there is now working 2Gbit multipath connection. Raw performance is about 160MB/s for both read and write.

Single modern SSD can easily outperform that but when combined with ZFS, both SLOG and cache; and compression; the performance exceeds all requirements.

But it will probably be a long time before this setup sees the light of day.

Testing

Speed inside a virtualized CentOS 7 is pretty good:

# dd if=/dev/sda bs=1M >/dev/null
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 13.9013 s, 2.5 GB/s

Of which none actually hit the disks. I have no idea where it came from, why it was in the memory, but not complaining.

Firewall hardening

Currently using the following lists to block non-wanted IP addresses:

Chain FIREHOL (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_apache src LOG flags 0 level 4 prefix "blocklist_de_apache: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_apache src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_bots src LOG flags 0 level 4 prefix "blocklist_de_bots: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_bots src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_bruteforce src LOG flags 0 level 4 prefix "blocklist_de_bruteforce: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_bruteforce src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_imap src LOG flags 0 level 4 prefix "blocklist_de_imap: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_imap src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de src LOG flags 0 level 4 prefix "blocklist_de: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_mail src LOG flags 0 level 4 prefix "blocklist_de_mail: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_mail src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_ssh src LOG flags 0 level 4 prefix "blocklist_de_ssh: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_ssh src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_strongips src LOG flags 0 level 4 prefix "blocklist_de_strongips: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_de_strongips src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_net_ua src LOG flags 0 level 4 prefix "blocklist_net_ua: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist_net_ua src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set bruteforceblocker src LOG flags 0 level 4 prefix "bruteforceblocker: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set bruteforceblocker src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set cruzit_web_attacks src LOG flags 0 level 4 prefix "cruzit_web_attacks: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set cruzit_web_attacks src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set cybercrime src LOG flags 0 level 4 prefix "cybercrime: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set cybercrime src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set darklist_de src LOG flags 0 level 4 prefix "darklist_de: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set darklist_de src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set dshield_top_1000 src LOG flags 0 level 4 prefix "dshield_top_1000: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set dshield_top_1000 src
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set et_compromised src LOG flags 0 level 4 prefix "et_compromised: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set et_compromised src

This is for inbound traffic. For outbound I have different set of lists.

Libvirt/virsh modify live network

Add new VLAN to libvirt network:

# virsh net-update ovsbr2 add portgroup "<portgroup name='VLAN-27'><vlan><tag id='27'/></vlan></portgroup>"

Attach new device to QEMU/KVM Instance:


# virsh attach-device 1-centos7-rev0.2-vm3 /tmp/device.xml 

<interface type='network'>
    <mac address='xx:xx:xx:xx:xx:xx'/>
    <source network='ovsbr2' portgroup='VLAN-27'/>
    <model type='virtio'/>
</interface>

 

And modify OpenvSwitch trunk to carry the VLAN 27

# ovs-vsctl set port vnet1 trunks=10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27

libvirt UEFI boot

<os>
  <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
  <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/3-windows-10-rev0-vm1_VARS.fd</nvram>
  <boot dev='hd'/>
  <bootmenu enable='yes' timeout='3000'/>
</os>

LAG with LACP 3xGbE

After failing with this 8 months ago, I finally pushed it through. I have no idea why it was so difficult, because it really wasn’t. Must have made too many mistakes.

# ovs-appctl bond/show bond0
---- bond0 ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 6
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 7733 ms
lacp_status: negotiated
active slave mac: xx:xx:xx:xx:xx:xx(ens1f0)

slave ens1f0: enabled
        active slave
        may_enable: true
        hash 64: 1 kB load
        hash 116: 1 kB load
        hash 140: 13 kB load
        hash 141: 1 kB load
        hash 164: 1 kB load
        hash 204: 1 kB load
        hash 214: 1 kB load

slave ens1f1: enabled
        may_enable: true
        hash 28: 10 kB load
        hash 142: 22 kB load
        hash 221: 1 kB load

This is because I have gigabit link to my desktop machine and single gigabit as a trunk from switch to router isn’t enough because GbE isn’t big enough to handle that and all the other network traffic. And because of which I have had two trunks, which isn’t very elegant solution and causes unnecessary configuration and other weird situations to the network topology.

I have only moved the internet to this new trunk and so far it is working very well. Still need to move all the other networks and perhaps add another gigabit link.

ens1f0 and ens1f1 are the interfaces, and I should use two different interface cards to get redundancy if one goes wrong.

I probably had something else to say but I seem to have forgotten what it was.

I am using OpenvSwitch and here is basically all that was needed:

ovs-vsctl add-br ovsbr5
ovs-vsctl add-bond ovsbr5 bond0 ens1f0 ens1f1 lacp=active 
bond_mode=balance-tcp other_config:lacp-time=fast vlan_mode=trunk
ovs-vsctl set port bond0 vlan_mode=trunk

The last command may not even be necessary as the bond may default on trunk. But try it out.

Here’s ovs-ovsctl show:

Bridge "ovsbr5"
    Port "bond0"
        trunks: [3, 4]
        Interface "ens1f0"
        Interface "ens1f1"
    Port "vnet17"
        trunks: [3, 4]
        Interface "vnet17"
    Port "ovsbr5"
        Interface "ovsbr5"
            type: internal

Upgrade to 3Gbit

Could not find out how I could have upgraded an existing OpenvSwitch bond from 2 links to 3 links so I had to recreate it, but that went without any problems.

bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
/         iface                   Rx                   Tx                Total
==============================================================================
         ens1f0:         269.81 KB/s          260.87 KB/s          530.68 KB/s
       enp3s0f0:          36.12 KB/s           58.99 KB/s           95.10 KB/s
         ens1f1:         455.73 KB/s        33333.58 KB/s        33789.31 KB/s
------------------------------------------------------------------------------
          total:         761.66 KB/s        33653.44 KB/s        34415.10 KB/s

 All three links are now utilized.

Major network change but everything went without any major problems. OpenvSwitch seems robust and pfSense only had bunch of minor clitches.

Does it work

Testing shows I cannot pull more than 1Gbit through the three links. I have no idea why. Should it not be using all the available bandwidth? I do not understand.

References

ovs-vswitchd.conf.db.5

OPENVSWITCH – Playing with Bonding on Openvswitch

https://fedoraproject.org/wiki/QA:Testcase_Virtualization_NIC_Hotplug_virsh_attach_detach

https://linux.die.net/man/1/virsh

https://support.citrix.com/article/CTX217646

Attachments

Linux and SAS3081E-R

DSASW00232822

These are good devices. With these I can hot swap all of my disks and do all maintenance without any hardware RAID configuration, which with SmartArray required booting multiple times! HP might provide some configuration tool for their SmartArray but with this I need none.

Once I get the other controller I think I have to get couple more disks to store the data temporarily to and then move all the drives in each mirror to different controller. That way I will have no problems even if a controller dies. The operating system disks will have to use hardware RAID 1 because any other solution on Linux isn’t going to be without problems.

Attachments

New disk bays online + future changes

The new 8 disk bays are now online with LSI 1068E and it seems to just pass the disks right to the operating system so I might have to get another one to replace the HP SmartArray P410 which requires me to use RAID0 which in turn causes problems because now there is that extra HW RAID getting involved with everything such as disk failures and so on.

I just hooked some old 32GB disk to try it out and it seems to work perfectly:

dd_rescue: (info): read /dev/sdg (35566480.0kiB): EOF
dd_rescue: (info): Summary for /dev/sdg -> /dev/null
dd_rescue: (info): ipos:  35566480.0k, opos:  35566480.0k, xferd:  35566480.0k
                   errs:      0, errxfer:         0.0k, succxfer:  35566480.0k
             +curr.rate:    39202kB/s, avg.rate:    52107kB/s, avg.load:  5.6%
             >-----------------------------------------< 100%  TOT:  0:11:23

I think I will just order the card right away. 14.39 € so practically free. After which the server is just about as stacked as possible. Probably still worth less than the price of electricity it will save in 18 months.

Disk failure

Did not take long for one of the cheap 300GB disks to fail.

  pool: pool2
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Dec 25 13:17:30 2016
    183G scanned out of 612G at 18.3M/s, 6h39m to go
    45.9G resilvered, 29.98% done
config:

        NAME                                             STATE     READ WRITE CKSUM
        pool2                                            DEGRADED     0     0     0
          raidz1-0                                       DEGRADED     0     0     0
            replacing-0                                  OFFLINE      0     0     0
              luks-80ec16d0-3d1f-4ed5-a4f0-0ee564547280  OFFLINE     52   191     0
              luks-fd4c98cc-d1be-4812-ae33-3650fdc984c0  ONLINE       0     0     0  (resilvering)
            luks-e7868ceb-fe11-4e40-a5e3-b6e56129b380    ONLINE       0     0     0
            luks-31f81d2e-36eb-4213-b101-e17d13907df5    ONLINE       0     0     0
            luks-c265d6da-5c40-435b-9f95-c2512c5a8bc5    ONLINE       0     0     0

errors: No known data errors

Had to take the 300GB from another pool until I get a replacement.

zpool offline pool2 luks-80ec16d0-3d1f-4ed5-a4f0-0ee564547280
zpool replace -o ashift=9 pool2 luks-80ec16d0-3d1f-4ed5-a4f0-0ee564547280 luks-fd4c98cc-d1be-4812-ae33-3650fdc984c0

I think I’ll wait for the missing hardware for the additional 8 bays and then migrate from 4x300GB RAIDZ1 to something else. Maybe 6-disk RAIDZ2 with online spare. Better to use those disk that I have. Or maybe get a bunch of small SSD disks from work.

Virsh: mount new cdrom live

virsh # qemu-monitor-command mirage --hmp --cmd "info block"
drive-virtio-disk0: type=hd removable=0 file=/home/daoist/mirage/mirage.qcow2 ro=0 drv=raw encrypted=0
drive-ide0-0-0: type=cdrom removable=1 locked=0 file=/home/daoist/iso/en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso ro=1 drv=raw encrypted=0
virsh # qemu-monitor-command mirage --hmp --cmd "eject drive-ide0-0-0"
virsh # qemu-monitor-command mirage --hmp --cmd "change drive-ide0-0-0 /path/to/new.iso"

QEMU-KVM Installing Windows 10 Client

https://en.wikibooks.org/wiki/QEMU/Monitor

Pretty powerful tool

Postfix and other email server problems

I couldn’t figure out why spam was getting through but then after looking deeper into how postgrey works, it turned out that it had learned my tcp-proxy’s IP address as trustworthy and passed everything through!

And it did that because the tcp-proxy relayed all tcp/25 traffic to my actual mail server, and the mail server saw the private IP address of the proxy, and not the public address of the actual email originator.

But now I made changes and moved the postgrey to the proxy machine, ditched the proxy for smtp, and use postfix smtp relay instead, which is the right way to do it, by the way.

The way it now works is that the first smtp (#1) takes the messages, applies postgrey and spamhaus lists to it, and if messages pass those checks, then it will relay it to the private smtp (#2) which will do ClamAV virus check first and then spamassassin at the very end.

This is practically the setup that I have had extremely good success with figting the spam and it is almost an occasion if anything gets through, and this should be even a little bit better than that.

Looking at the smtp #1 logs, some 40% of incoming messages are dropped because spammers didn’t read the RFC:

Nov 11 12:28:09 sensored postfix/smtpd[11682]: NOQUEUE: reject: RCPT from unknown[1.30.194.205]: 450 4.7.1 <ilpsv>: Helo command rejected: Host not found; from=<lmmmm@realisaatio.fi> to=<326122335@qq.com> proto=ESMTP helo=<ilpsv>

Only if they said HELO correctly they would have much more success. Hopefully few spammers will ever read this wink