Testing CoreOS on homelab servers - part 1

Testing CoreOS on homelab servers - part 1

I've been working on some scripts to build and deploy cloud-config files in my homelab.

core01: 192.168.61
core02: 192.168.62
core03: 192.168.63
core04: 192.168.64

Create and deploy cluster configuration

 From my Mac, build and deploy new cloud-config with:

$ ./create.and.deploy.sh  


For each core server in the homelab, the scripts build and deploy configuration files for my coreos test cluster.

E.g for server core01

#cloud-config
#version: 20161210_203041
hostname: "core01"
ssh_authorized_keys:
  - ssh-rsa ...
coreos:
  etcd2:
    # Static cluster
    name: core01
    advertise-client-urls: http://192.168.1.61:2379
    initial-advertise-peer-urls: http://192.168.1.61:2380
    initial-cluster: "core01=http://192.168.1.61:2380,core02=http://192.168.1.62:2380,core03=http://192.168.1.63:2380,core04=http://192.168.1.64:2380"
    initial-cluster-state: new
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://0.0.0.0:2380,http://0.0.0.0:7001
  fleet:
    public-ip: $public_ipv4
    metadata: "role=services"
  flannel:
    interface: $public_ipv4
  update:
      reboot-strategy: "etcd-lock"
  units:
    - name: 00-eth0.network
      runtime: true
      content: |
        [Match]
        Name=eno1

        [Network]
        DNS=192.168.1.1
        Address=192.168.1.61/24
        Gateway=192.168.1.1
        Domains=home.lab

    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: flanneld.service
      drop-ins:
        - name: 50-network-config.conf
          content: |
            [Service]
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
      command: start
    - name: docker-tcp.socket
      command: start
      enable: true
      content: |
        [Unit]
        Description=Docker Socket for the API
        [Socket]
        ListenStream=2375
        Service=docker.service
        BindIPv6Only=both
        [Install]
        WantedBy=sockets.target

write_files:
  - path: "/etc/motd"
    permissions: "0644"
    owner: "root"
    content: |
      --- My CoreOS Cluster (core01) ---

Checking cluster status

core@core01 ~ $ etcdctl member list
4374c5ef9f2370d6: name=core03 peerURLs=http://192.168.1.63:2380 clientURLs=http://192.168.1.63:2379 isLeader=true
45337feea7d7a60f: name=core01 peerURLs=http://192.168.1.61:2380 clientURLs=http://192.168.1.61:2379 isLeader=false
6688d9448380b482: name=core02 peerURLs=http://192.168.1.62:2380 clientURLs=http://192.168.1.62:2379 isLeader=false
c9a76f89ee66e035: name=core04 peerURLs=http://192.168.1.64:2380 clientURLs=http://192.168.1.64:2379 isLeader=false

core@core01 ~ $ fleetctl list-machines
MACHINE        IP        METADATA
2d0e73b6...    192.168.1.64    role=services
497f6384...    192.168.1.61    role=services
9f8f9d8a...    192.168.1.62    role=services
c6d410a0...    192.168.1.63    role=services

Launching containers and testing failover

Container configuration

vi myapp.service

--- snip begin ---
[Unit]
Description=MyApp
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill busybox1
ExecStartPre=-/usr/bin/docker rm busybox1
ExecStartPre=/usr/bin/docker pull busybox
ExecStart=/usr/bin/docker run --name busybox1 busybox /bin/sh -c "trap 'exit 0' INT TERM; while true; do echo Hello World; sleep 1; done"
ExecStop=/usr/bin/docker stop busybox1

--- snip end ---
 

vi apache@.service

--- snip begin ---
[Unit]
Description=My Apache Frontend
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill apache1
ExecStartPre=-/usr/bin/docker rm apache1
ExecStartPre=/usr/bin/docker pull coreos/apache
ExecStart=/usr/bin/docker run --rm --name apache1 -p 80:80 coreos/apache /usr/sbin/apache2ctl -D FOREGROUND
ExecStop=/usr/bin/docker stop apache1

[X-Fleet]
Conflicts=apache@*.service
--- snip end ---

Launch the containers

core@core01 ~ $ fleetctl start myapp.service
core@core01 ~ $ fleetctl start apache@1
core@core01 ~ $ fleetctl start apache@2


core@core01 ~ $ fleetctl list-units
UNIT            MACHINE                ACTIVE    SUB
apache@1.service    2d0e73b6.../192.168.1.64    active    running
apache@2.service    c6d410a0.../192.168.1.63    active    running
myapp.service        2d0e73b6.../192.168.1.64    active    running




 

Testing failover

core@core04 ~ $ sudo reboot

core@core01 ~ $ fleetctl list-units
UNIT            MACHINE                ACTIVE    SUB
apache@1.service    497f6384.../192.168.1.61    active    running
apache@2.service    c6d410a0.../192.168.1.63    active    running
myapp.service        9f8f9d8a.../192.168.1.62    active    running

 

Setting up CoreOS on homelab servers

Setting up CoreOS on homelab servers with static IPs
 
Work Notes

core01: 192.168.61
core02: 192.168.62
core03: 192.168.63
core04: 192.168.64

For each node core01,core02,core03,core04
core@coreXX ~ $ sudo vi /var/lib/coreos-install/user_data
core@coreXX ~ $ sudo rm -rf /var/lib/etcd2/*; sudo rm -f /etc/systemd/system/etcd*
core@coreXX ~ $ sudo reboot

FILE: /var/lib/coreos-install/user_data  (core01)
--- snip --
#cloud-config
hostname: core01
ssh_authorized_keys:
  - ssh-rsa ...
coreos:
  etcd2:
    # Static cluster
    name: core01
    advertise-client-urls: http://192.168.61:2379
    initial-advertise-peer-urls: http://192.168.61:2380
    initial-cluster: "core01=http://192.168.61:2380,core02=http://192.168.1.62:2380,core03=http://192.168.1.63:2380"
    initial-cluster-state: new
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://0.0.0.0:2380,http://0.0.0.0:7001
  fleet:
    public-ip: $public_ipv4
    metadata: "role=services"
  flannel:
    interface: $public_ipv4
  update:
      reboot-strategy: "etcd-lock"
  units:
    - name: 00-eth0.network
      runtime: true
      content: |
        [Match]
        Name=eno1

        [Network]
        DNS=192.168.1.1
        Address=192.168.61/24
        Gateway=192.168.1.1

    # To use etcd2, comment out the above service and uncomment these
    # Note: this requires a release that contains etcd2
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: flanneld.service
      drop-ins:
        - name: 50-network-config.conf
          content: |
            [Service]
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
      command: start
    - name: docker-tcp.socket
      command: start
      enable: true
      content: |
        [Unit]
        Description=Docker Socket for the API
        [Socket]
        ListenStream=2375
        Service=docker.service
        BindIPv6Only=both
        [Install]
        WantedBy=sockets.target

write_files:
  - path: "/etc/motd"
    permissions: "0644"
    owner: "root"
    content: |
      --- My CoreOS Cluster ---

---snip--


core@core01 ~ $ etcdctl cluster-health
member 4374c5ef9f2370d6 is healthy: got healthy result from http://192.168.1.63:2379
member 45337feea7d7a60f is healthy: got healthy result from http://192.168.1.61:2379
member 6688d9448380b482 is healthy: got healthy result from http://192.168.1.62:2379

core@core02 ~ $ etcdctl member list
4374c5ef9f2370d6: name=core03 peerURLs=http://192.168.1.63:2380 clientURLs=http://192.168.1.63:2379 isLeader=true
45337feea7d7a60f: name=core01 peerURLs=http://192.168.1.61:2380 clientURLs=http://192.168.1.61:2379 isLeader=false
6688d9448380b482: name=core02 peerURLs=http://192.168.1.62:2380 clientURLs=http://192.168.1.62:2379 isLeader=false

core@core02 ~ $ fleetctl list-machines
MACHINE   IP    METADATA
497f6384... 192.168.1.61  role=services
9f8f9d8a... 192.168.1.62  role=services
c6d410a0... 192.168.1.63  role=services

Ubuntu 16.04 boot failure

One of my homelab servers (ts01) running Ubuntu 16.04 refused to boot today.

It's running kernel 4.4.0-36-generic.

Looking at the console I see this

Multiple /scripts/local-block ... done lines. The USB keyboard I have attached did not work, so I rebooted and at the grub boot menu I choose another kernel:

  4.4.0-34-generic instead of the default  4.4.0-36-generic which fails to boot.

Kernel  4.4.0-34-generic boots ok.

I tried to re-install the packages for the kernel, but still got the same failure.

So, I compared the files kernel files with another identical homelab server (ts02).

This server works:
root@ts02:/boot/grub# ls -l /boot/
total 87640
-rw-r--r-- 1 root root 32061904 Aug 29 21:55 initrd.img-4.4.0-36-generic


This server fails during boot.
root@ts01:/boot/grub# ls -l /boot/
total 87555
-rw-r--r-- 1 root root 31977897 Sep  4 20:42 initrd.img-4.4.0-36-generic

So I copied the initrd.img-4.4.0-36-generic from ts02 to ts01. This fixed the problem and ts01 booted OK.

DC/OS installation on Ubuntu 16.04

Work In Progress ...

My work notes so far trying to install DC/OS 1.8.1 on Ubuntu 16.04

Source: https://dcos.io/docs/1.8/administration/installing/custom/advanced/

My DC/OS homelab servers: 192.168.1.71, 192.168.1.72, 192.168.1.73
My bootstrap server: 192.168.1.73

WORKDIR: /sw/dcos

FILE: genconf/config.yaml (bootstrap server)
---
bootstrap_url: http://192.168.1.73:8877
cluster_name: 'soulfunk'
exhibitor_storage_backend: static
ip_detect_filename: /genconf/ip-detect
master_list:
- 192.168.1.73
- 192.168.1.72
- 192.168.1.71
resolvers:
- 8.8.4.4
- 8.8.8.8

FILE: genconf/ip-detect (bootstrap server)
#!/usr/bin/env bash
set -o nounset -o errexit
export PATH=/usr/sbin:/usr/bin:$PATH
echo $(ip addr show | grep inet | grep 192.168.1 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -1)


FILE: get_boostrap_install.sh (bootstrap server)
#!/bin/bash
curl -O https://downloads.dcos.io/dcos/EarlyAccess/dcos_generate_config.sh

FILE: create_build_file.sh (bootstrap server)
#!/bin/bash
bash dcos_generate_config.sh

FILE: run_nginx.sh (bootstrap server)
#!/bin/bash
docker run -d -p 8877:80 -v $PWD/genconf/serve:/usr/share/nginx/html:ro nginx

FILE: symlink_binaries.sh (all servers)
#!/bin/bash
cd /usr/bin || exit -1
FILES="ln mkdir tar"
for f in $FILES ; do
  test -e /usr/bin/$f || ln -s /bin/$f $f
done

FILE: get_dcos_installer_from_bootstrap_server.sh (all servers)
#!/bin/bash
curl -O http://192.168.1.73:8877/dcos_install.sh

1) On boostrap server, download installer and create build file:


# cd /sw/dcos
# mkdir genconf
# <create genconf/config.yaml> 
# <create genconf/ip-detect>  
# ./get_bootstrap_install.sh
# ./create_build_file.sh

2) On bootstrap server, start nginx:
# ./run_nginx.sh

3) On each server:
# ./get_dcos_installer_from_bootstrap_server.sh
# ./symlink_binaries.sh
# bash dcos_install.sh master
# bash dcos_install.sh slave


Monitor Exhibitor

http://192.168.1.73:8181/exhibitor/v1/ui/index.html

DC/OS Web Interface:

http://192.168.1.73/

Upgrading from Ubuntu 14.04 to 16.04

Upgraded from Ubuntu 14.04 to 16.04 on one of my homelab test-servers today.

After the upgrade, running 'apt-get update' resulted in this error:

# apt-get update
apt-get: relocation error: /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0: symbol _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE7compareERKS4_, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference



To fix this, I tried to manually install libstdc++6:

Download:

wget http://security.ubuntu.com/ubuntu/pool/main/g/gcc-5/libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb


Install:

# dpkg -i libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb

 But this failed with:

dpkg: warning: downgrading libstdc++6:amd64 from 6.1.1-3ubuntu11~14.04.1 to 5.4.0-6ubuntu1~16.04.2
dpkg: regarding libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb containing libstdc++6:amd64:
 libstdc++6:amd64 breaks libboost-date-time1.55.0
  libboost-date-time1.55.0:amd64 (version 1.55.0-1) is present and installed.

dpkg: error processing archive libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb (--install):
 installing libstdc++6:amd64 would break libboost-date-time1.55.0:amd64, and
 deconfiguration is not permitted (--auto-deconfigure might help)
Errors were encountered while processing:
 libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb


So, ran this a few times, to remove libboost libraries:

# for p in $(dpkg -l |grep libboost|awk '{print $2}'); do dpkg --purge "$p" ; done

To do a  dry-run (ie check what would be removed but not actually do it), add an echo infront of dpkg:

Dry-run (test without removing)

# for p in $(dpkg -l |grep libboost|awk '{print $2}'); do echo dpkg --purge "$p" ; done



And then could finally run:

# dpkg -i libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb
dpkg: warning: downgrading libstdc++6:amd64 from 6.1.1-3ubuntu11~14.04.1 to 5.4.0-6ubuntu1~16.04.2
(Reading database ... 147174 files and directories currently installed.)
Preparing to unpack libstdc++6_5.4.0-6ubuntu1~16.04.2_amd64.deb ...
Unpacking libstdc++6:amd64 (5.4.0-6ubuntu1~16.04.2) over (6.1.1-3ubuntu11~14.04.1) ...
Setting up libstdc++6:amd64 (5.4.0-6ubuntu1~16.04.2) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...

And apt-get was working again

GitLab CE web-interface "broken" after upgrade on Ubuntu 14.04

I've installed GitLab CE in my homelab as detailed here.

After an apt upgrade, the web interface was not loading the JavaScript and CSS assets:

The fix that worked for me:

mygitlab$ sudo gitlab-rake cache:clear 
mygitlab$ sudo gitlab-ctl restart

@NorSoulx (twitter)

Soulful Goodness

Pc power on problems

Suddenly got this problem with my Komplett Gamer PC, after working for 18 months.

Update: 2016.08.28 - I've had to replace two faulty Corsair AX1200i PSUs in the past 5 months.

GitLab CE Quick Installation Guide on Ubuntu 14.04

Here's a short write up of how I installed GitLab CE on one of my homelab servers

Install GitLab

First add gitlab repository, key and install it with 'apt-get' thru the script 'gitlab_ubuntu1404_setup.sh'

File: gitlab_ubuntu1404_setup.sh

#!/bin/bash
#
# GitLab CE Quick Installation Guide on Ubuntu 14.04
#
# @NorSoulx 2016
#

if [ "$EUID" != 0 ] ; then echo "Please run as root"; exit ; fi

#####################
# 1) Add GitLab Repos
#
APTSRC=/etc/apt/sources.list.d/gitlab_gitlab-ce.list
BACKUPFILE=/tmp/gitlab.apt.source.bak.$(date '+%Y%m%d-%H%M%S').$$
if [ -f $APTSRC ]; then cp $APTSRC $BACKUPFILE ; fi

cat << APTEND > $APTSRC
#GitLab
deb https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ trusty main
deb-src https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/ trusty main
APTEND


#####################
# 2) Add GitLab key
#
wget -qO - https://packages.gitlab.com/gpg.key | apt-key add -

#####################
# 3) Install packages
#
PKGS="openssh-server ca-certificates postfix gitlab-ce"

apt-get update
for PKG in $PKGS; do
  apt-get install $PKG
done

#####################
# 4) Configure
#
gitlab-ctl reconfigure

# 5) Summary
#
cat << DIREND
GITLAB DIRS
-----------
/opt/gitlab             application code
/var/opt/gitlab         application data and configuration files (gitlab-ctl)   DO NOT EDIT
/etc/gitlab             configuration files (omnibus-gitlab).                   EDIT MANUALLY
/var/log/gitlab         log data (omnibus-gitlab)
DIREND



Installed by running:

mygitlab$ sudo ./gitlab_ubuntu1404_setup.sh

Configure GitLab

I have multiple bonding interfaces and subnets configured on this homelab server, so I  put GitLab on a new ip alias.

Add alias IP

To manually add the alias ip, I did (I've also added this IP to my boot-startup-script)

mygitlab$ sudo ip addr add 10.11.12.13/32 dev bond0


Added new alias IP to local DNS or /etc/hosts.

mygitlab$ grep mygitlab /etc/hosts
10.11.12.13 mygitlab

Create certificate

Created local certificate for https:

mygitlab$ sudo mkdir -p /etc/gitlab/ssl
mygitlab$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/gitlab/ssl/mygitlab.key -out /etc/gitlab/ssl/mygitlab.crt

Edit /etc/gitlab/gitlab.rb

Made changes to /etc/gitlab/gitlab.rb.
Change url and ssh host, move backups and datadir, configure ssl.

Set:

external_url 'https://mygitlab'
gitlab_rails['gitlab_ssh_host'] = 'mygitlab'
gitlab_rails['backup_path'] = "/data02/gitlab/backups"
git_data_dir "/data01/gitlab/git-data"
nginx['ssl_certificate'] = "/etc/gitlab/ssl/mygitlab.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mygitlab.key"
nginx['listen_addresses'] = ['mygitlab']

Reconfigure

Ran gitlab-ctl reconfigure for changes to take effect:

mygitlab$ sudo gitlab-ctl reconfigure

Login and change default root password

Login with web-browser: https://mygitlab

Default:
Username: root
Password: 5iveL!fe (change to something else)


Next add users and create projets via web-interface.

Git repositories are located under directory:

/var/opt/gitlab/git-data/repositories

@NorSoulx (twitter)

Soulful Goodness

Windows 8.1 - "Preparing Automatic Repair" Loop

Today, when I rebooted my Windows 8.1 instance (running under VMware Fusion 7 Pro), I was greeted by the "Preparing Automatic Repair" during boot. I keep backups of my VMware instances so I could easily restore the instance from backup - and all my source code is also kept in remote git repositories. But, I wanted to test if I could salvage this instance, and maybe learning something in the process.

Say hello to: Preparing Automatic Repair

 

I tried the various options presented but could not get out of "Preparing Automatic Repair"; 

on every reboot it would enter this state.

In addition, when I tried to do a repair, I was notified that I had to little space left (3.4 GB left). 

First, I tried to remove some unnecessary user produced files, by dropping into the command console and deleting them, but even 6GB free space was not enough. So, instead I decided to increase the VMware disk size for this instance by about 20GB. This allowed repair to start.

However, when I tried to complete a "fresh repair", it failed right after 1%.

  

So, I rebooted, and then through the Advanced options dropped into the command console, and ran these commands:

I also disabled "Automatic Repair", 

When I next rebooted, I was greeted by this message during boot:

 

     Boot critical file is corrupt: C:\Windows\system32\drivers\intelide.sys

I choose once again to run Repair, 

and this time it completed. 

All installed applications, such as Visual Studio, were gone, but my user data/source code was still there.

Next up, restore my VMware instance backup.

@NorSoulx (twitter)

Soulful Goodness

VMware Fusion 7 Pro

I use VMware Fusion on my MacBook Pro to do both testing in Linux and Visual Studio development in Windows.

I keep my instances on external SSD disk:

• Buffalo Ministation 2.5" 256gb SSD Thunderbolt USB 3.0

  

Today I upgraded from VMware Fusion 6 Pro to 7 Pro, and one cool feature with the recently 
released VMware Fusion 7 Pro is the ability to:

• Access virtual machines hosted on VMware vSphere, ESXi or Workstation

At home, I also run an ESXi home-lab server 

So thru the new VMware Fusion 7 Pro I can now access the instances running on this ESXi home-lab server:

 CPU, DISK and MEMORY usage on ESXi server:

And  now I can also access the consoles of my instances on my ESXi home-lab server directly from Fusion 7 Pro.

VMware Fusion 7 Pro also adds the ability to:

• Upload or download virtual machines to vSphere, ESXi or Workstation

@NorSoulx (twitter)

Soulful Goodness

VMware vSphere, ESXi or Workstation

VMware vSphere, ESXi or Workstation

Benchmarking 8 x Intel 520 SSDs on ESXi home-lab server with IOZONE

Introduction

For a long time I've used VMware instances to prototype and test systems and network topologies I've used in my work. I also use them to run my own home projects.

The list includes:

• Redundant load-balancers and firewalls (iptables, Pacemaker, Corosync, ldirectord, iproute2)
• NoSQL solutions, such as Riak and Tokyo Cabinet.
• Oracle, PostgreSQL, MySQL databases
• IBM GPFS clustered filesystem with clustered NFS on top using ISCSI nodes.
• Firewall and NSM solution to DMZ each of my kids' sub-networks at home.

Recently I upgraded my ESXi 5 home-lab server with 8 x Intel 520 240GB SSDs to bring some more IO speed to my testing facilities, but also enough room to test NoSQL and distributed storage solutions with close to 2 TB of data. Sidenote: These 8 SSD drives cost in total about the same amount as one Seagate 2GB SCSI disk I bought in 1994.

The SSDs are connected to an LSI 9211-8i card as JBODs, and this card is configured in ESXi passthrough-mode. To get a feel for what kind of performance (IOPS) I can expect from these 8 SSDs when run in parallel, my first test uses IOZONE. This server has 16GB RAM, and 1 Xeon E3-1220 @ 3.10GHz CPU (4 cores).

The ESXi server also contains an Areca raid card with 256MB cache (w/BBU) connected to an external enclosure with 8 SATA disks, so these 8 SSDs should deliver a nice speedup when testing distributed storage solutions.

For this test I created an Ubuntu 12.04 server instance with 12GB RAM and 4 cores.
The other instances running on this ESXi host were shutdown during the IOZONE testing.

I'm using two ICYDOCK hot-swap cabinets for the 8 SSDs.

Partitioning: gdisk

All 8 SSDs were partitioned using gdisk.

$ sudo gdisk -l /dev/sdb

GPT fdisk (gdisk) version 0.8.1

Found valid GPT with protective MBR; using GPT.

Disk /dev/sdb: 468862128 sectors, 223.6 GiB

Logical sector size: 512 bytes

Partition table holds up to 128 entries

First usable sector is 34, last usable sector is 468862094

Partitions will be aligned on 2048-sector boundaries

Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name

   1            2048       468862094   223.6 GiB   8300  Linux filesystem

Filesystem: XFS

I've been using XFS on Linux servers since around 2001, and have been very satisfied with it.

#!/bin/bash

DEVICES="sdb sdc sdd sde sdf sdg sdh sdi"

devno=1

for dev in $DEVICES; do

  echo /dev/${dev}1 

  mkfs.xfs -L ssd0$devno /dev/${dev}1

  let devno=$devno+1 

done

I used nobarrier for this benchmark; I've always used enterprise hardware with BBU cache when using this XFS mount option, but this test system will not contain any production-critical data so I've opted for the extra performance boost.

$ mount|grep xfs|sort 

/dev/sdb1 on /ssd01 type xfs (rw,noatime,nobarrier)

/dev/sdc1 on /ssd02 type xfs (rw,noatime,nobarrier)

/dev/sdd1 on /ssd03 type xfs (rw,noatime,nobarrier)

/dev/sde1 on /ssd04 type xfs (rw,noatime,nobarrier)

/dev/sdf1 on /ssd05 type xfs (rw,noatime,nobarrier)

/dev/sdg1 on /ssd06 type xfs (rw,noatime,nobarrier)

/dev/sdh1 on /ssd07 type xfs (rw,noatime,nobarrier)

/dev/sdi1 on /ssd08 type xfs (rw,noatime,nobarrier)

IOZONE

For this test I used IOZONE version 3.420 (compiled from source).

Setup configuration:

• Ubuntu RAM size:      12 GB
• Total file-size used: 24 GB
• Record-size used:      4 KB

Linux IO scheduler is set to noop for all SSD devices during testing.

Below are the results from two of the test-runs I did.

Benchmark IOPS with 1 SSD drive

Ran 1 IOZONE process with 24 threads.
Each thread writes a 1048576 Kbyte file in 4 Kbyte records.

iozone -l24 -T -k 1000 -s1g -r4k -i0 -i1 -i2 -O -+u -R -b res.xls

Benchmark IOPS with 8 SSD drives

Ran 8 IOZONE processes in the parallel (one process on each SSD), each running 12 threads.
Each thread writes a 262144 Kbyte file in 4 Kbyte records.

iozone -l12 -T -k 1000 -s256m -r4k -i0 -i1 -i2 -O -+u -R -b res.xls

All 8 IOZONE processes started at the same second and ended within 1 second a part.

This is by no means the absolute result, but an indication of the performance the server hardware is capable of delivering with the configuration and setup chosen. YMMV.

Twitter: @NorSoulX

Soulful Goodness