Monday, December 20, 2010

Installing Mediatomb on Solaris 11 Express

My home NAS is now running Solaris 11 for the 'express' purpose (sorry) of being able to use ZFS, and all the light, joy and happiness it brings to the world of storage. I'm using three 1TB SATA disks in a RAIDZ configuration, and using ZFS's built-in filesystem compression, CIFS and NFS sharing capabilities. Since I use a PS3 as my media center front-end I need a DLNA/UPnP media server and having used Mediatomb previously on Linux, that's what I'll walk through installing from source here today.

Before starting, make sure you have GNU C compiler (gcc) and make installed. This is easily achieved using the new package management tools in Solaris 11. Try man pkg to get started.

For Mediatomb to correctly identify certain files' MIME types (like image/jpeg, for example) you need to build the GPL'd version of the file utility for a library called libmagic.
tar zxvf file-5.04.tar.gz
cd file-5.04
./configure --prefix=/usr/local
make
make install
Check for libmagic library and headers:
ls /usr/local/include
ls /usr/local/lib
Next, download the source for Mediatomb and extract it.
tar zxvf mediatomb-0.12.1.tar.gz
cd mediatomb-0.12.1
The source code contains some apparently outdated prelink logic which breaks the build on newer versions of the OS, so we need to comment it out. You can either edit src/main.cc by hand or just use the following patch - save it to main.cc.patch0.
*** src/main.cc.orig    2010-12-20 13:13:11.080796210 +1000
--- src/main.cc 2010-12-20 13:13:30.478542756 +1000
***************
*** 141,146 ****
--- 141,147 ----

      Ref<Array<StringBase> > addFile(new Array<StringBase>());

+ /*
  #ifdef SOLARIS
      String ld_preload;
      char *preload = getenv("LD_PRELOAD");
***************
*** 155,160 ****
--- 156,162 ----
          exit(EXIT_FAILURE);
      }
  #endif
+ */

  #ifdef HAVE_GETOPT_LONG
      while (1)
And apply the patch:
patch src/main.cc < main.cc.patch0
Now configure, build and install:
./configure --enable-libmagic --with-magic-h=/usr/local/include --with-magic-libs=/usr/local/lib
make
make install
At this stage, you could add your config files and fire up the daemon, however I'm going to create service manifests so we can manage startup and shutdown of Mediatomb via the Solaris Service Management Framework (SMF). Save this file as /lib/svc/method/svc-mediatomb.
#!/bin/sh
. /etc/mediatomb.conf
LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/mediatomb -d \
    -u $MT_USER \
    -g $MT_GROUP \
    -P $MT_PIDFILE \
    -l $MT_LOGFILE \
    -m $MT_HOME \
    -f $MT_CFGDIR \
    -p $MT_PORT \
    -e $MT_INTERFACE \
    $MT_OPTIONS
Create /etc/mediatomb.conf, setting appropriate values for variables above. Something like the following would suffice:
MT_INTERFACE="rge0"
MT_OPTIONS=""
MT_PORT="50500"
MT_USER="media"
MT_GROUP="media"
MT_PIDFILE="/var/run/mediatomb.pid"
MT_LOGFILE="/var/log/mediatomb"
MT_HOME="/etc"
MT_CFGDIR="mediatomb"
Now for the service manifest - create this file as /var/svc/manifest/application/mediatomb.xml.
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type="manifest" name="mediatomb">
    <service name="application/mediatomb" type="service" version="1">
        <create_default_instance enabled="false"/>
        <single_instance/>
        <dependency name="network" grouping="require_all" restart_on="error" type="service">
            <service_fmri value="svc:/milestone/network:default"/>
        </dependency>
        <dependency name="filesystem" grouping="require_all" restart_on="error" type="service">
            <service_fmri value="svc:/system/filesystem/local"/>
        </dependency>
        <exec_method type="method" name="start" exec="/lib/svc/method/svc-mediatomb" timeout_seconds="60">
        </exec_method>
        <exec_method type="method" name="stop" exec=":kill" timeout_seconds="5">
        </exec_method>
        <property_group name="startd" type="framework">
            <propval name="ignore_error" type="astring" value="core,signal"/>
        </property_group>
        <stability value="Evolving"/>
        <template>
            <common_name>
                <loctext xml:lang="C">
                    UPnP Media Server
                </loctext>
            </common_name>
            <documentation>
                <manpage title="mediatomb" section="1" manpath="/opt/local/share/man"/>
                <doc_link name="mediatomb.cc" uri="http://mediatomb.cc"/>
            </documentation>
        </template>
    </service>
</service_bundle>
Then validate, import and run the service:
svccfg validate /var/svc/manifest/application/mediatomb.xml
svccfg import /var/svc/manifest/application/mediatomb.xml
svcs -a mediatomb
svcadm enable mediatomb
If all went well you should now be able to connect to the Mediatomb web interface on port 50500.

Saturday, November 6, 2010

Scan for new devices in Solaris

This is how to tell Solaris to look for new devices when booting - append a '-r' to the kernel boot arguments:

Select the Solaris entry GRUB menu that you want to boot, to edit, enter e
Select the "kernel /platform" line, to edit that again enter e
Add to the end of the 'kernel' line a space followed by -r
Press enter key to accept the change
Press b to boot

Via: http://blogs.sun.com/harcey/entry/solaris_x86_vmware_adding_a

Wednesday, August 4, 2010

Setting up a highly available NFS cluster

The following is a proof-of-concept HA NFS cluster running on RHEL 5, using DRBD for block-level replication and heartbeat for failover.

In this example, I'll be setting up 2 nodes: nfs01 and nfs02. Ensure that ALL commands and setup is run against both nodes EXCEPT where stated otherwise.

Following is the partition table for a test install - the metadata partition /dev/sda7 and the data partition /dev/sda8 are unformatted and unmounted.
[root@nfs01 ~]# fdisk -l /dev/sda

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         274     2096482+  82  Linux swap / Solaris
/dev/sda3             275         535     2096482+  83  Linux
/dev/sda4             536        2610    16667437+   5  Extended
/dev/sda5             536         796     2096451   83  Linux
/dev/sda6             797        1057     2096451   83  Linux
/dev/sda7            1058        1074      136521   83  Linux
/dev/sda8            1075        2610    12337888+  83  Linux
Instructions for building the DRBD RPMs, including the kernel module can be found on the DRBD website.

The kernel in use is the latest update for RHEL 5.5 at the time of writing, version kernel-2.6.18-194.3.1.el5.x86_64.rpm - DRBD must be compiled against your specific version of kernel-headers.

Ensure at least the following RPMs are installed.
rpm-build-4.4.2.3-18.el5    
elfutils-0.137-3.el5        
elfutils-libs-0.137-3.el5   
flex-2.5.4a-41.fc6          
glibc-devel-2.5-49.el5_5.2  
glibc-headers-2.5-49.el5_5.2
kernel-2.6.18-194.3.1.el5   
kernel-devel-2.6.18-194.3.1.el5  
kernel-headers-2.6.18-194.3.1.el5
gcc-c++-4.1.2-48.el5
gcc-4.1.2-48.el5
make-3.81-3.el5
Use DRBD's built-in functionality to build both the userland and kernel module RPMs.
[root@nfs01 build]# tar zxf drbd-8.3.8.tar.gz 
[root@nfs01 build]# cd drbd-8.3.8
[root@nfs01 drbd-8.3.8]# ./configure --with-km=/usr/src/kernels/2.6.18-194.3.1.el5-x86_64/
[root@nfs01 drbd-8.3.8]# make km-rpm rpm
...
You now have:
/usr/src/redhat/RPMS/x86_64/drbd-bash-completion-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-pacemaker-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-km-2.6.18_194.3.1.el5-8.3.8-12.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-udev-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-utils-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-xen-8.3.8-1.x86_64.rpm
/usr/src/redhat/RPMS/x86_64/drbd-heartbeat-8.3.8-1.x86_64.rpm
Install DRBD and modprobe the new module - a reboot is given in the examples here to check the module loads on a system restart.
[root@nfs01 drbd-8.3.8]# rpm -ivh /usr/src/redhat/RPMS/x86_64/drbd-*
[root@nfs01 drbd-8.3.8]# modprobe drbd # or...
[root@nfs01 drbd-8.3.8]# reboot
...
[root@nfs01 drbd-8.3.8]# lsmod | grep drbd
drbd                  277784  0 
Install the sample /etc/drbd.conf file.
# A comprehensively commented example of this file exists at:
# /usr/share/doc/drbd-utils-8.3.8/drbd.conf.example

resource r0 {

    protocol C;
    
    # Parse error here:
    # incon-degr-cmd "halt -f";
    
    startup {
        degr-wfc-timeout 120;    # 2 minutes.
    }

    disk {
        on-io-error   detach;
    }

    net {
    }

    syncer {
        rate 10M;
        # group 1;
        al-extents 257;
    }

    on nfs01 {                   
        device    /dev/drbd0;          
        disk      /dev/sda8;           # NFS data partition
        meta-disk /dev/sda7[0];        # 1024MB partition for DRBD metadata
        address   192.168.1.101:7788;
    }

    on nfs02 {                   
       device    /dev/drbd0;           
       disk      /dev/sda8;            # NFS data partition
       meta-disk /dev/sda7[0];         # 1024MB partition for DRBD metadata
       address   192.168.1.102:7788;  
    }

}
Install the sample /etc/exports file, and ensure the NFS service is stopped and is not set to launch on system startup.
/export/ 192.168.1.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync)
[root@nfs01 drbd-8.3.8]# /etc/init.d/nfs stop
[root@nfs01 drbd-8.3.8]# chkconfig nfs off
Configure the NFS statd daemon to use the hostname for the floating IP.
[root@nfs01 ~]# echo >> /etc/sysconfig/nfs 'STATDARG="-n nfs.localdomain"'
Ensure there's no existing filesystem on your metadata partition, /dev/sda7 in this case.
[root@nfs01 ~]# dd if=/dev/zero of=/dev/sda7 bs=1M count=256
[root@nfs01 ~]# drbdadm create-md r0
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.

[root@nfs01 ~]# mkfs.ext3 /dev/sda8
[root@nfs01 ~]# drbdadm up all

[root@nfs01 ~]# cat /proc/drbd 
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by root@nfs01, 2010-06-18 16:52:07
 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12217404
Do this only on node 1:
[root@nfs01 ~]# drbdadm adjust all 
[root@nfs01 ~]# drbdadm -- --force primary r0

[root@nfs01 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by root@nfs01, 2010-06-18 16:52:07
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
    ns:537600 nr:0 dw:0 dr:537600 al:0 bm:32 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:11679804
 [>....................] sync'ed:  4.5% (11404/11928)M delay_probe: 102
 finish: 0:15:41 speed: 12,272 (10,336) K/sec

# Completed now:
[root@nfs01 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by root@nfs01, 2010-06-18 16:52:07
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    ns:12217401 nr:0 dw:0 dr:12217401 al:0 bm:746 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Do this only on node 1:
[root@nfs01 ~]# mount /dev/drbd0 /export
[root@nfs01 ~]# mv /var/lib/nfs /export/
[root@nfs01 ~]# cd /var/lib/
[root@nfs01 ~]# mv nfs nfs.old
[root@nfs01 ~]# ln -s /export/nfs
Do this only on node 2:
[root@nfs02 ~]# cd /var/lib/
[root@nfs02 lib]# mv nfs nfs.old
[root@nfs02 lib]# ln -s /export/nfs
Ensure the EPEL repository is enabled and install heartbeat.
[root@nfs01 ~]# yum install heartbeat.x86_64
Edit the /etc/ha.d/ha.cf file.
logfacility local0
keepalive 2
deadtime 10
bcast eth0
node nfs01.localdomain nfs02.localdomain
auto_failback on
Edit /etc/ha.d/haresources - the IP address is the floating one.
nfs01.localdomain IPaddr::192.168.1.103/24/eth0 \
    drbddisk::r0 Filesystem::/dev/drbd0::/export::ext3 nfs
Add a secret to /etc/ha.d/authkeys file.
auth 3
3 md5 pa55word
chmod 600 /etc/ha.d/authkeys
Make sure the drbd and heartbeat servces are set to start on boot, and start them up:
/etc/init.d/drbd start
/etc/init.d/heartbeat start

chkconfig drbd on
chkconfig heartbeat on
That's a very brief and tersely documented example - you can now test failover by pulling a network cable :-)

Monday, April 12, 2010

Automount filesystems by label

Here's a handy (and from what's in the autofs the man pages, undocumented) trick for auto-mounting ext3 filesystems by label. Say I had a removable USB drive formatted as ext3 with the label WD-250 I could add this line to my auto.misc file:
WD-250 -fstype=ext3 :LABEL=WD-250
The filesystem UUID also seems to work using this method.

Tuesday, March 30, 2010

Init script for Redmine

With my new role at work as an Infrastructure Engineer in our Projects team, I've recently started using Redmine, a flexible project management web application written using Ruby on Rails framework, to keep myself and my projects organised.

Redmine is distributed as source and doesn't include any management scripts other than the Ruby ones used to manage Rails. In order to start and stop it automatically on boot on my Fedora 12 workstation, I'm using the following init script.
#!/bin/bash
#
# chkconfig: - 16 84
# description: Start up Redmine
#
# processname: redmine
# config: /etc/sysconfig/redmine

# source function library
. /etc/rc.d/init.d/functions

# Get network config
. /etc/sysconfig/network

[ "${NETWORKING}" = "no" ] && exit 0

# Defaults
DAEMON_HOME=/usr/local/redmine
DAEMON_ARGS="-e production -d"

# Daemon
NAME=redmine
RUBY=$(which ruby)

APP_PIDFILE=$DAEMON_HOME/tmp/pids/server.pid
DAEMON_PIDFILE=/var/run/$NAME.pid
DAEMON_LOCKFILE=/var/lock/subsys/$NAME

start() {
    echo -n $"Starting ${NAME}: "
        
    cd $DAEMON_HOME
    $RUBY script/server webrick $DAEMON_ARGS

    sleep 2

    status -p $APP_PIDFILE &> /dev/null && echo_success || echo_failure
    RETVAL=$?

    if [ $RETVAL -eq 0 ]; then
        touch $DAEMON_LOCKFILE
        cat $APP_PIDFILE > $DAEMON_PIDFILE
    fi

    echo
}

stop() {
    echo -n $"Shutting down ${NAME}: "
    
    killproc -p $DAEMON_PIDFILE
    RETVAL=$?

    [ $RETVAL -eq 0 ] && /bin/rm -f $DAEMON_LOCKFILE $DAEMON_PIDFILE

    echo
}

case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        stop
        start
    ;;
    status)
        status -p $DAEMON_PIDFILE $NAME
    ;;

    *)
        echo "Usage: $SCRIPTNAME {start|stop|restart|status}" >&2
        exit 3
    ;;
esac
Once you have it in place in /etc/init.d, run the following commands to install it, enable Redmine startup on boot and fire it up.
chmod +x /etc/init.d/redmine
chkconfig --add redmine
chkconfig redmine on
/etc/init.d/redmine start

Sunday, March 21, 2010

Automatically update the Terminal window title in Mac OS X

It seems Mac OS X doesn't provide the nice Bash functionality found in RHEL/CentOS/Fedora Linux that updates the terminal title after each command. For example, using Linux, if I do the following:
cd ~/Developer
the window title is updated to:
jason@rocksteady:~/Developer
While the Terminal app in OS X does allow you to customize the title to some extent by adding the TTY name, name of running process etc. I prefer the 'user@host:pwd' format.

After digging around in /etc/bashrc on CentOS 5, I found you can set the PROMPT_COMMAND Bash variable and have your title updated after every command.

Add the following to your .bash_profile in Mac OS X to get the same behaviour:
PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/~}"; echo -ne "\007"'

Saturday, February 27, 2010

Dstat plugin for squid proxy

I've been using Dag Wieers' excellent system utility Dstat for a little while now and have recently starting exploring the possibility of extending it with my own plugins. Dstat is written in Python and so are its plugins, so that made the possibility all the more likely for me.

In case you don't know what Dstat is:
Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
Check the Dstat homepage for more details.

I was mostly interested in obtaining some useful metrics on a squid proxy and plugging them into Dstat. Using squidclient, you can get all sorts of interesting info using a command line interface and that's what I'm using to get these stats.

You'll need to ensure you have manager access enabled in your squid.conf like so:
http_access allow manager localhost  
http_access deny manager
The metrics I've chosen to extract are just a proof-of-concept really - I'll eventually tweak them to be somewhat more useful, and this O'Reilly article Eleven Metrics to Monitor for a Happy and Healthy Squid has always proven to be a good reference.

You can check out the dstat_squid.py plugin from my dstat-plugins GitHub repository, or copy it below.
# Dstat plugin for measuring various squid statistics.
# Author: Jason Friedland <thesuperjason@gmail.com>
#
# This plugin has been tested with:
# - Dstat 0.6.7
# - CentOS release 5.4 (Final)
# - Python 2.4.3
# - Squid 2.6 and 2.7
 
global squidclient_options
squidclient_options = os.getenv('DSTAT_SQUID_OPTS') # -p 8080
 
class dstat_squid(dstat):
    def __init__(self):
        self.name = 'squid status'
        self.format = ('s', 8, 100)
        self.vars = ('Number of file desc currently in use',
            'CPU Usage, 5 minute avg',
            'Total accounted',
            'Number of clients accessing cache',
            'Mean Object Size')
        self.nick = ('fdesc',
            'cpu5',
            'mem',
            'clients',
            'objsize')
        self.init(self.vars, 1)
        
    def check(self):
        if not os.access('/usr/sbin/squidclient', os.X_OK):
            raise Exception, 'Needs squidclient binary'
        return True
 
    def extract(self):
        try:
            for line in os.popen('/usr/sbin/squidclient %s mgr:info' 
                    % (squidclient_options), 'r').readlines():
                l = line.split(':')
                for item in self.vars:
                    if l[0].strip() == item:
                        self.val[item] = l[1].strip()
                        break
        except Exception, e:
            if op.debug > 1: print '%s: exception' (self.filename, e)
            for name in self.vars: self.val[name] = -1