Libvirt/virsh modify live network

Add new VLAN to libvirt network:

# virsh net-update ovsbr2 add portgroup "<portgroup name='VLAN-27'><vlan><tag id='27'/></vlan></portgroup>"

Attach new device to QEMU/KVM Instance:


# virsh attach-device 1-centos7-rev0.2-vm3 /tmp/device.xml 

<interface type='network'>
    <mac address='xx:xx:xx:xx:xx:xx'/>
    <source network='ovsbr2' portgroup='VLAN-27'/>
    <model type='virtio'/>
</interface>

 

And modify OpenvSwitch trunk to carry the VLAN 27

# ovs-vsctl set port vnet1 trunks=10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27

libvirt UEFI boot

<os>
  <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
  <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/3-windows-10-rev0-vm1_VARS.fd</nvram>
  <boot dev='hd'/>
  <bootmenu enable='yes' timeout='3000'/>
</os>

Virsh: mount new cdrom live

virsh # qemu-monitor-command mirage --hmp --cmd "info block"
drive-virtio-disk0: type=hd removable=0 file=/home/daoist/mirage/mirage.qcow2 ro=0 drv=raw encrypted=0
drive-ide0-0-0: type=cdrom removable=1 locked=0 file=/home/daoist/iso/en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso ro=1 drv=raw encrypted=0
virsh # qemu-monitor-command mirage --hmp --cmd "eject drive-ide0-0-0"
virsh # qemu-monitor-command mirage --hmp --cmd "change drive-ide0-0-0 /path/to/new.iso"

QEMU-KVM Installing Windows 10 Client

https://en.wikibooks.org/wiki/QEMU/Monitor

Pretty powerful tool

LXC configuration

I put this here so that I can use it later. Feel free to use it yourself.

# Distribution configuration
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.mount.entry = /etc/nginx/ssl etc/nginx/ssl none bind.ro 0.0

# Container specific configuration
lxc.rootfs = /var/lib/lxc/server1/rootfs
lxc.utsname = server1

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4.gateway = 192.168.10.1
lxc.network.ipv4 = 192.168.10.2/28

lxc.cgroup.cpuset.cpus = 4,5,6,7
lxc.cgroup.cpu.shares = 1024
lxc.cgroup.memory.limit_in_bytes = 4096M
lxc.cgroup.memory.memsw.limit_in_bytes = 4096M

lxc.start.auto = 1
lxc.start.delay = 5
lxc.start.order = 20

lxc.hook.clone = /usr/share/lxc/hooks/clonehostname

New base image strategy

-r-----r--. 1 root root  26G Jul 31 15:21 2-windows2012r2
-r-----r--. 1 qemu qemu 194K Jul 31 15:37 2-windows2012r2-rev0
-rw-r-----. 1 qemu qemu  26M Jul 31 15:43 2-windows2012r2-rev0-vm1

The idea is that the 26GB file is the absolute base image on top of which multiple revisions can be used on top of which there can be multiple virtual machines.

This is because the base image will get old over time and new revisions may need to be created. But because these new revisions most likely will not take much space, it is cost effective to have that additional layer, versus always creating new complete 26GB base file.

26MB of difference was created after first boot.

Also learned from the history, the sources\sxs is not included within the base image so that all features are easy to install.

There is also performance increase because now every server references the same blocks on the file system and the underlying intelligent filesystem can better manage the blocks and keep them in cache.

Also theoretically the base image could be stored on highly compressed volume while the differential would be stored on non or slightly compressed volume. It would be balanced act between least amount of storage space used vs. fast access time to instance specific data.

This change will save me about 30GiB which is a lot considering the limited space, and the price of larger SAS disks.

Building more backups

So I have quite a good plan for Linux server and Linux desktop backups but virtual machines and especially VMware has been neglected and I even thought maybe I don’t need to do a backup but because that’s fundamentally a bad idea and because the VMware I have is the core of the infrastructure it is required to have a backup.

So Veeam was something that I had heard and decided to give it ago.

snap1470-2

Not fast with current setup but it doesn’t matter much because it will still finish in couple of hours. With compression it will probably take less than 20GiB for good amount of guests.

Also my Hetzner backup rent server was delivered today and looking at upload speeds it seems that I have approximately 1000GiB of daily upload capacity so network will not become a bottleneck for quite a while. I am expecting to be using somewhere around 20-35GiB daily.

The server had so much power that I can easily compress everything with gzip-9. Also storage isn’t going to be a problem because I expect to use roughly 500GiB for daily backups which will leave me with over 2500GiB unused space.

And I must say that the other systems I have built are pretty damn good as well. This is a clear step forward in terms of versatile backup plan.

Notes

So that I remember later.

# lvcreate -Z y -n thin1 -l100%FREE -T -r 0 vg0_sda5
# lvcreate -Z y -n thin1 -l100%FREE -T -r 0 vg0_sdb5
# lvcreate -n pool -r 0 -T -V 500G vg0_sda5/thin0
# lvcreate -n pool -r 0 -T -V 500G vg0_sdb5/thin0

And that is how thin pool is created. I use this method to create the filesystem on top of these thinly provisioned volumes to enable couple tricks and make the setup more versatile.

Ran out of money and progress with servers

Ran out of money so disks and caddies must wait for a month or so but I have made some progress.

The overview

So there are two servers now, the DL365 G1, which has been given management designation and the DL385 G6 which is still sort of work in progress, but will most likely be sort of general virtualization machine plus the router.

The idea with this is that the management will run the VMware vSphere Server and Client and will be the absolute fallback machine with which to boot the systems in case of DL385 would ever fail one way or the other. And the idea is that this management server will be mostly static and no unnecessary experimentation will be done with it.

The main role of that server, though, is to be the virtualized Windows management server for the main DL385 server.

The management is pretty much fully equipped and has 8 cores and 32GiB of memory + 2 disks in hardware RAID1 and 4 disks in RAIDZ1, so there is good amount of both disk and fault tolerance. Also this server has redundant power supplies which is why I sort of made it the management server; it is less likely to fail.

Also the infrastructure is such that I don’t want any dependencies between any of the servers; no unnecessary dependencies, because if one server fails it should not mean failure for other servers. Management server can fail because ESXi does not require active managing, and vice versa, ESXi can fail but it won’t affect the management server.

Internet will also be L2 routed with switch directly to this management server but it won’t be physically plugged in unless it is needed; in case the virtualized router fails; then internet may be necessary so it would then be routed via secondary virtualized router.

The idea and compromise

My idea is to build the most efficient, most professional and most compact setup with least amount of hardware. And what I consider as professionalk is that a) encryption is a must b) virtualize as much as possible c) use professional systems and tools d) make everything as simple as possible and e) redundancy and no single point of catastrophic failure.

Because during the years I have learned that once systems get complicated, one single failure can cripple the whole school of systems. And to boot that system up again can then be extremely difficult. Currently what I am building is exactly like that but only for the build phase; once finished it should be very simple to maintain.

Because disks break, power is lost, networks go down, and people make mistakes.

Final thoughts on CloudStack

It fucking sucks and it simply doesn’t work. It has been project on-going for who knows how long and still they haven’t been able to make it work.

It is not possible to install it without reading thousands of pages of useless documentation or taking some sort of retaredly prices courses or some shit, so my patience have ran out with such a retarded system.

If you got time to fiddle around and risk loosing everything because nothing ever seems to work then please go ahead and write ahead a good explanation to your boss why the fuck everything just suddenly died the fuck off.

I will be using raw QEMU because quite frankly that’s shit load easier than playing with this sort of abstraction which simply doesn’t work. And I cannot understand how could anyone trust a system like this.

It looks nice and it might be cool but quite frankly IT SUCKS

serveimage (110)

CloudStack as a management server

So after cursing around hopelessly, somehow this happened unexpectedly:

snap1276

I was having problems with adding the host but then it suddenly worked.

And after hours and hours of small incremental steps towards working first instance I now have this:

snap1278

But there are so many things I know nothing about. Networking is one of those things.

https://github.com/apache/cloudstack-docs-install/blob/master/source/hypervisor/kvm.rst

https://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.7/qig.html

And here we have Windows Server 2008 R2 being installed:

snap1279

Performance-wise the DL365 G1 with RAID1 10KRPM HP SAS drives is excellent as I get 160MB/s write – with encryption. So the Opteron 2356 is quite heavy performed in that sense too.

I am beginning to like CloudStack:

snap1280

It has quite clean firewall built in. Obviously not very configurable but for basic usage.

And on Hypervisor I immediately started saving memory with ksm:

Shared  Sharing Unshared        Volatile        Sharing:Shared  Unshared:Sharing        Saved
32,467  1,029,836       438,788         293,152         31.710000:1     0.420000:1              4,022M

And since swap is compressed I get some benefits there as well:

# zfs get compressratio dpool/swap
NAME        PROPERTY       VALUE  SOURCE
dpool/swap  compressratio  3.06x  -

Current problems

So the problem now is that because CloudStack will not allow over-subscription of thinly provisioned primary storage (more or less) the 68 available gigabytes won’t be enough for everything I want. So two 300GB SAS disks must be purchased ASAP to rectify the situation.

Also I am doubting my use of two servers, because to have one just for ESXi management is stupid. It can be used for other things but to have to have one to get ESXi souns like bad design. But I am not certain whether CloudStack is stable and quite frankly, interesting enough to only use that. ESXi is, still, a proper enterprise system.

One alternative to the 300GB disks is to use 4 73GB disks on RAIDZ1 which would be sub-optimal solution but might work. This would give me little over 200GiB of storage which would be enough.

Disk change

So I made the changes I discussed above and I am getting excellent write performance:

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
dpool       25.1G   247G      0    807    357  15.2M
dpool       25.1G   247G      0  1.24K      0   157M
dpool       25.1G   247G      0  1.29K      0   164M
dpool       25.1G   247G      0  1.61K    511   204M
dpool       25.1G   247G      0  1.30K    511   165M
dpool       25.1G   247G      0  1.80K      0   229M
dpool       25.2G   247G      0  3.27K      0   117M
dpool       25.2G   247G      0  12.3K      0  50.6M
dpool       25.2G   247G      0  12.2K      0  66.8M
dpool       25.2G   247G      0  9.29K      0  33.3M
dpool       25.2G   247G      1  9.59K   1022  30.2M

There is some “thing” with ZFS and QEMU or QCOW2, that makes it impossible to use RAW ZFS with some settings of QEMU, or what not, and so because I didn’t want to go there, I just planted ext4 on top of thinly provisoned ZFS block device. Should not affect performance at all and could even improve it.

And these speeds are with gzip-9 compression and individual LUKS encryption on each of the four disks. And I am just about CPU bound. So I am extremely impressed with this AMD Opteron 2356 and it isn’t the fastest Opteron I got available.

snap1281

A lot of free space now.