Building a SmartOS home data center

For the main bulk of my short career as a Systems Administrator I have been squarely sat in the Redhat Enterprise Linux (and friends) camp, however recently I became disheartend with a QNAP TS-410 hosting all of my production guests and data at home given a number of problems with the device such as:

  • Weak hardware leading to sad NFS and iSCSI response times
  • Administration that is relatively painful via its GUI interface
  • Generally suffering from security issues in the firmware
  • Uses no battery backed cache for the RAID10 (write cache disabled furthering the performance pain)
  • I am paranoid about loosing things so the appeal of # zfs scrub is very high!

Finally the primary reason for looking into a solution to replace my existing/old kit was:

  1. Wife Agro - My existing kit makes far too much heat in our small office during a typical Queensland summers day. It had to get consolidated!

So I needed to consolidate the following hardware:
2x Intel E8200 hosts running as KVM hosts on CentOS
1x Intel E4200 host running as a dedicated Sophos UTM 9 gateway
1x QNAP TS-410 with 4x 1TB SATA disks with a further 2x external 1TB standalone disks

Specs

POC Host
CPU: Intel Core i5 3570K
Motherboard: Gigabyte GA-Z77-D3H
RAM: 8GB G-Skill 1600Mhz DDR3 (2x 4GB)
Storage:
* Western Digital WD3202ABYS 320GB SATA2 HDD
* Seagate ST3320613AS 320GB SATA2 HDD
* Toshiba 4GB USB2 Thumbdrive
NIC:
* Onboard Atheros 1Gbit NIC
* HP NC380T Dual Port 1Gbit NIC ( Broadcom )

Production Host(s)
Parts are slowly arriving this week!

Smart OS
CPU: Intel Xeon E3-1230V2
Motherboard: Supermicro X9SCL-F
RAM: 32GB Kingston 1333Mhz DDR3 ECC Unbuffered (4x 8GB)
Storage:
* 4x Hitachi 7K3000 Ultrastar 2TB SATA3 HDD
NIC:
* 1x Intel 82579LM 1Gbit onboard NIC
* 1x Intel 82574L 1Gbit onboard NIC
* 1x Dedicated IPMI

PXE Bootstrap
I have a FIT-PC2 lurking to run a PXE boot service and manage the UPS.

Proof of Concept
I had some basic requirements to meet before I would let myself be drawn into the rabbit hole that is SmartOS and the greater Illumos community in general.

Requirement 1 - Firewall and Routing
I had to be able to get the firewall virtualised and configured to work with my existing VLAN segregated network.

Building the firewall guest
I wont really enter into aboslute detail since the SmartOS docs cover it quite well however these are the steps I followed to create my virutalised firewall.
Also note before you use this JSON you will need to have a NIC tagged as '''external''' in your /usbkey/config like follows.

  external_nic=0:18:71:79:15:00

It is important to note here a point the SmartOS docs brush over is that Crossbow will activley enforce declarations made about VNIC's in this configuration. Thus if you want to virtualise a router like I have here you need to allow the extra traffic it generates with the following in you '''nics''' section:

      "allow_restricted_traffic": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_dhcp_spoofing": true,
      "ip": "dhcp"

Also note that we set the guest to '''dhcp'''. Initially to me this seemed wrong however the result is effectively telling crossbow that the guest will handle its own network configuration.

For the full JSON reference see https://github.com/joyent/smartos-live/blob/master/src/vm/man/vmadm.1m.md
1) Create the guest JSON spec using vi. You need a minimum of two NIC's for the Sophos UTM 9 installer. Also note that VLAN 1000 is actually my PPPoE session to a bridged modem. This is needed since the Sophos UTM will not allow a PPPoE session on anything but a dedicated NIC. By tagging the traffic with SmartOS the guest is none the wiser!

/opt/gateway.json
{
  "hostname": "gateway.example.com",
  "alias": "gateway",
  "autoboot": true,
  "brand": "kvm",
  "ram": "1024",
  "vcpus": "1",
  "resolvers": ["192.231.203.132", "192.231.203.3"],
  "disks": [
    {
      "boot": true,
      "model": "virtio",
      "size": 40960
    }
  ],
  "nics": [
    {
      "nic_tag": "external",
      "model": "virtio",
      "vlan_id": 1000,
      "allow_restricted_traffic": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_dhcp_spoofing": true,
      "ip": "dhcp"
    },
    {
      "nic_tag": "external",
      "model": "virtio",
      "allow_restricted_traffic": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_dhcp_spoofing": true,
      "ip": "dhcp"
    }
  ]
}

2) Now we create the guest and its zone

# vmadm create -f /opt/gateway.json

3) Once created you will see your guest is running, we will stop it and download the ISO into its zone root.

  # vmadm list | grep gateway
    36a44254-7e27-11e2-b650-cf463dcdfa40  KVM   1024     running           gateway
  # vmadm stop 36a44254-7e27-11e2-b650-cf463dcdfa40

4) Now wget the ISO into the zone

  # cd /zones/36a44254-7e27-11e2-b650-cf463dcdfa40/root/
  # wget http://download.astaro.com/UTM/v9/software_appliance/iso/latest_asg_v9_s...

5) We can now boot the guest using the ISO and attach to the VNC console

  # vmadm boot 36a44254-7e27-11e2-b650-cf463dcdfa40 order=cd,once=d cdrom=/latest_asg_v9_software.iso,ide
  # vmadm info 36a44254-7e27-11e2-b650-cf463dcdfa40 vnc
    {
      "vnc": {
        "host": "192.168.1.1",
        "port": 57379,
        "display": 51479
      }
    }

6) During the installer you will need to select the Admin interface to use eth1 as that is your current untagged network for clients. Of course the finer grained details on how you configure your home network is up to you!
7) Attach your PC to the physical NIC you tagged as '''external''' and you should be able to connect to the WebUI on https://192.168.2.1:4444 or what ever you set during the installer
8) Optionally, if your in my situation you can then upload a copy of your config backup and be underway. In version 9 the OS will attempt to match the NIC's as closely as possible so your old eth1 network will appear on the new virutal eth1 when it comes up

Requirement 2 - Host and Guest Security
In the RHEL world where I normaly live we implement guest and host separation via the SELinux policies integrated into libvirt. SmartOS zones are a very similar implementation to dynamic SELinux domains (if not a little bit stronger more like SELinux Domains + cgroups) so I feel SmartOS ticks the boxes here.

PS: If there are readers who have more of a clue than I (which is not hard!) please let me know if I missed the mark here at kahn [at] this domain.

Requirement 3 - Backups
I had to be able to backup the guests to the QNAP across the network (Yes it will live on in a diminshed role).

My initial testing involves just a simple snapshot and dump to a file. In the long run this process will involve either a remotely mounted volume (not sure we can use iSCSI or NFS from within the SmartOS Global Zone).

Backup KVM guest to file
Note: The doco at https://github.com/joyent/smartos-live/blob/master/src/vm/README.migration describes this process in detail
Warning:
Using vmadm send first stops the guest then performs the send. After the send the guest remains powered down.

vmadm send <UUID> | gzip > example.backup.tgz

Restore KVM guest from file

  gunzip -c example.backup.tgz | vmadm recieve

You can also use this process to migrate guests between hosts. See the doco I mentioned above.

Requirement 4 - Migrate existing KVM guests from Linux
I had to be able to migrate the existing CentOS guests over to the SmartOS host and ensure that the existing platforms (Windows 7, CentOS 5 & 6) continue to operate normaly.

Migrate disk images from Linux KVM to SmartOS KVM
I had a feeling that I could hack this together using my trusty old migration tool aka dd since this is my go to method for V2V.
1) Firstly on your Linux host use the following to confirm your images are in RAW format (sparse files should still be OK).

[[email protected] images]# qemu-img info mon1.img
image: mon1.img
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 3.0G
cluster_size: 65536

2) I am going to guess that qcow2 will fail miserably so we'll convert the guest to RAW first.

   qemu-img convert -O raw mon1.img mon1.raw.img

3) Now we SCP the image over to /opt on our SmartOS host
4) Using DD we copy the image over to the ZFS disk

dd if=/opt/mon1.raw.img of=/dev/zvol/dsk/zones/19a7af4f-5441-4b3e-bedf-fd3a2c6223df-disk0

NOTE: A trap for Windows guests is using the virtio disk model without the guest drivers being present. You will instead need to use IDE mode until the drivers are installed.

 "disks": [
    {
      "boot": true,
      "model": "ide",
      "size": 40960,
    }

5) Once DD finishes you can startup the guest!

Signing Off
So I am now commited to running my home datacenter needs on SmartOS for the foreseeable few years! And I have to admit I am excited to be a part of it. If you would like to get in touch flick me an email at kahn [ at ] this domain or jump in our IRC channel.
PS: I appologize for the terrible formatting. This started off in mediawiki format and now is stuck in broken Drupal CSS!
Cheers,
Kahn