Zones, Crossbow, and a virtual DMZ

One of the aspects of OpenSolaris that I've been most excited about has been Zones. Unfortunately, due to some complications in my home setup, I couldn't make the networking come out right. Since I first saw the description of what the Crossbow project has brought to the table, I had high hopes that it could do what I needed it to do.

My OpenSolaris server at home wears a number of different hats. Firstly, it's my firewall and performs NAT and filtering. It has two interfaces: ipbr0 connected to the outside, and bge0 connected to the inside. Secondly it's my private file server. ZFS was the biggest reason I migrated this machine from Linux to OpenSolaris, and it's worked wonderfully. Thirdly, I do some web development on the side so it serves as my test web server.

As you can see, my configuration is a little involved. Before Crossbow, I tried to build a zone for the web server, but I couldn't make it talk to both the inside and outside networks. I could make it talk to the inside network fine, but that keeps it from being publicly accessible. I could probably have made it talk to the outside if I paid my cable provider for a second IP, but then I couldn't use it inside. There might have been a way to make it work, but I never found it.

Before I go any further, I want to thank Ben Rockwood. His post about Crossbow at http://www.cuddletech.com/blog/pivot/entry.php?id=1001 was extremely helpful in figuring out how to get started with Crossbow.

The first step was to update my system to build 105. To do this, I needed to point my system to the dev repository by executing "pkg set-authority -O http://pkg.opensolaris.org/dev/ opensolaris.org". This took a minute or two to complete, but it ran without errors. After that, I did a "pkg image-update" to upgrade to the latest packages and rebooted. That took care of getting the bits required for Crossbow.

The next step was to create the virtual switch (etherstub) and network interfaces (vnic). Below are the commands that I used to create an etherstub named dmz1 and two virtual nics named dmz1_host1 and dmz1_web1. It appears that the last character of the vnic name must be a digit.

# dladm create-etherstub dmz1
# dladm create-vnic -l dmz1 dmz1_host1
# dladm create-vnic -l dmz1 dmz1_web1
# dladm show-link
LINK        CLASS    MTU    STATE    OVER
iprb0       phys     1500   up       --
bge0        phys     1500   up       --
dmz1        etherstub 9000  unknown  --
dmz1_host1  vnic     9000   up       dmz1
dmz1_web1   vnic     9000   up       dmz1

Once you've created the etherstub and the vnics, you need to configure them. First you need to plumb the and assign an IP to the host interface.

# ifconfig dmz1_host1 plumb
# ifconfig dmz1_host1 192.168.0.1 netmask 255.255.255.0
# ifconfig dmz1_host1 up

The next step is to create and install the zone. This is very similar to all the examples of zone configuration that I've found, but the key difference is setting the ip-type to exclusive. This is marked in bold below. Before beginning, I created a zfs filesytem at /data/zones. I've configured the zone so that it will share the /www filesystem from the host. If you don't need this, simply skip the lines from "add fs" through the next "end" statement.

# zonecfg -z web1
web1: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:web1> create
zonecfg:web1> set zonepath=/data/zones/web1
zonecfg:web1> set ip-type=exclusive
zonecfg:web1> add net
zonecfg:web1:net> set physical=dmz1_web1
zonecfg:web1:net> end
zonecfg:web1> add fs
zonecfg:web1:fs> set dir=/www
zonecfg:web1:fs> set special=/www
zonecfg:web1:fs> set type=lofs
zonecfg:web1:fs> end
zonecfg:web1> info
zonename: web1
zonepath: /data/zones/web1
brand: ipkg
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
net:
address not specified
physical: dmz1_web1
defrouter not specified
zonecfg:web1> verify
zonecfg:web1> commit
zonecfg:web1> exit
# zoneadm -z web1 install
A ZFS file system has been created for this zone.
Authority: Using http://pkg.opensolaris.org/dev/.
Image: Preparing at /data/zones/web1/root ...  done.
Cache: Using /var/pkg/download.
Installing: (output follows)
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  53/53   7972/7972   77.15/77.15

PHASE                                        ACTIONS
Install Phase                            12074/12074
PHASE                                          ITEMS
Reading Existing Index                           9/9
Indexing Packages                              53/53

Note: Man pages can be obtained by installing SUNWman
Postinstall: Copying SMF seed repository ... done.
Postinstall: Working around http://defect.opensolaris.org/bz/show_bug.cgi?id=741
Done: Installation completed in 90.608 seconds.

Next Steps: Boot the zone, then log into the zone console
     (zlogin -C) to complete the configuration process
# zoneadm -z web1 boot
# zlogin -C web1

Once you've connected to the console of the new zone using zlogin, you will be walked through some configuration options. This should be the same as a standard sysidconfig process. When asked for an IP address for the interface dmz1_web1, enter an IP the falls within the same subnet as the dmz1_host1 interface. Use the same subnet mask and use the IP address of dmz1_host1 as the gateway.

Since my server is also functioning as a firewall, I need to configure access rules. To start with, I'm simply allowing free access between my internal network and the dmz. Later, I'll configure specific access rules. For now, I added "pass in quick on dmz1_host1 keep state" and "pass in quick on dmz1_host1 keep state" to /etc/ipf/ipf.conf. To activate these rules, I ran "ipf -Fa -f /etc/ipf/ipf.conf". You may also need to adjust your NAT rules to allow access to the outside world.

If everything has gone according to plan, you should be able to log into your new zone and access both the Internet and your internal network. Next, install and enable apache.

# pkg install SUNWapch22
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    4/4   1342/1342     6.07/6.07

PHASE                                        ACTIONS
Install Phase                              1656/1656
PHASE                                          ITEMS
Reading Existing Index                           9/9
Indexing Packages                                1/4
Indexing Packages                                4/4
# svcadm enable http:apache22
#

Now try to access the new zone from a system on your internal network by pointing your browser to http://192.168.0.2.

I hope that this post is able to help people get started with Zones and Crossbow. These are very flexible technologies that work very nicely together.

Reference:
http://www.cuddletech.com/blog/pivot/entry.php?id=1001
http://www.sun.com/software/solaris/howtoguides/containersLowRes.jsp
http://blog.thilelli.net/post/2006/10/05/Zones-and-ZFS-Integration-and-New-Features-in-OpenSolaris

HOWTO: Get tun/tap compile for x64 OpenSolaris

I've been trying to figure out how to get tun/tap working on OpenSolaris for a while so that I can use OpenVPN. I finally put together all the different pieces to make it work. There may be some things here that aren't strictly required, but this is what finally got it working for me. I've included the main references that helped get it working below.

Firstly, I used the Sun Studio C compiler so you'll need to install that to use this process. On OpenSolaris, you can do this by executing "pkg install sunstudioexpress". The download is over 500MB, so it will take some time.

The next step is to download and extract the tun/tap driver sources. These are freely available from http://vtun.sourceforge.net/tun/. At this time, the current version is 1.1. They will extract into a directory named tun-1.1. You'll also need to download a new version of tun.c from http://openvpn.net/solaris/tun.c and replace the tun-1.1/solaris/tun.c file. The version that is distributed with the tun/tap sources compiles but doesn't work correctly.

After downloading and updating the sources, now we need to configure the build environment. This is where the majority of the work was in getting tun/tap to work correctly. Firstly, make sure that the Sun Studio compilers are in your path by executing "export PATH=/opt/sunstudioexpress/bin/:$PATH". Now cd into tun-1.1/solaris and execute "./configure". Unfortunately, the Makefile that is generated doesn't work correctly. I've included the entire contents of my working Makefile below. The changes I've made are to use the Sun Studio C compiler, use a different set of CFLAGS, add some LDFLAGS, and install the driver files into some additional locations.

References:
Fixing the "relocation error: R_AMD64_32" error
http://opensolaris.org/jive/thread.jspa?messageID=185654

Fixing the failed to attach error:
http://openvpn.net/archive/openvpn-users/2005-08/msg00002.html

# Generated automatically from Makefile.in by configure.
#
#  Universal TUN/TAP device driver.
#
#  Multithreaded STREAMS tun pseudo device driver.
#
#  Copyright (C) 1999-2000 Maxim Krasnyansky 
#
#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
#  GNU General Public License for more details.
#
#  $Id: Makefile.in,v 1.7 2000/06/20 03:14:17 maxk Exp $
#
CONFIGURE_FILES = Makefile config.status config.cache config.h config.log

CC = cc
LD = ld

DEFS =  -DTUN_VER=\"\"
CFLAGS = $(DEFS) -O2 -m64 -D_KERNEL -I. -xmodel=kernel -KPIC
LDFLAGS = -64
ADD_DRV = /usr/sbin/add_drv
REM_DRV = /usr/sbin/rem_drv
DRV_DIR = /kernel/drv/
DRV_DIR2 = /kernel/drv/amd64
DRV_DIR3 = /platform/i86pc/kernel/drv

INSTALL = /usr/bin/ginstall -c

all: module

module: tun.o
       $(LD) $(LDFLAGS) -r -o tun tun.o

tun.o: tun.c if_tun.h
       $(CC) $(CFLAGS) -c tun.c

inst: module
       $(INSTALL) -m 644 -o root -g root if_tun.h /usr/include/net
       $(INSTALL) -m 644 -o root -g root tun $(DRV_DIR)
       $(INSTALL) -m 644 -o root -g root tun.conf $(DRV_DIR)
       $(INSTALL) -m 644 -o root -g root tun $(DRV_DIR2)
       $(INSTALL) -m 644 -o root -g root tun.conf $(DRV_DIR2)
       $(INSTALL) -m 644 -o root -g root tun $(DRV_DIR3)
       $(INSTALL) -m 644 -o root -g root tun.conf $(DRV_DIR3)
       -$(REM_DRV) tun >/dev/null 2>&1
       $(ADD_DRV) tun 

clean:
       rm -f tun *.o *~

distclean:
       rm -f $(CONFIGURE_FILES)

Is Sun Solaris on its deathbed?

As a long-time Linux user and a new Solaris user, I found the article “Is Sun Solaris on it's deathbed” to be misguided. Firstly, I think that there definitely are some people who are taking a fresh look at Solaris and especially OpenSolaris. In my case, my exposure to an extremely diverse Linux environment has shown me the appeal of the consistency and compatibility that Solaris offers. Even the differences between Solaris 10 and OpenSolaris, which seem to bother many Solaris admins, seem minor compared to the differences between Linux distributions.

I think there are a few factors related to finding experienced Linux administrators versus Solaris administrators. Most people agree that the best way to learn something is to use it in production. Linux has really only been accepted in the enterprise for about 5 years, which is pretty short in comparison to Solaris. An admin who has worked with Linux for 5 years can be considered experienced. On the other hand, an admin who has worked with Solaris for 5 years is a relative newcomer.

Solaris is known for reliability, and the deployments where it is used have tended to reflect that. This has led the companies who deploy it to demand experienced admins. As stated above, being an experienced Solaris admin is a much higher bar than with Linux. This leads those companies compete for the best of a limited supply of experienced administrators. It also makes it a very difficult specialty to break into.

To address these issues, I think Sun needs to do more to showcase the benefits of Solaris as well as to encourage more people to learn the system. Either one of these alone isn't going to address the problem. By doing both, you create both more demand for the system as well as more people qualified to administrate it. This is one area where Linux has been remarkably effective.

I think the demand for DTrace and other advanced tools relates to the kind of environments where a system is used. It used to be that applications scaled vertically. It was expensive to add capacity because at some point the whole system needed to be replaced. Administrators were required to learn how to ensure maximum reliability and to wring every last drop of performance out of the system due to the expense of an upgrade. This demanded advanced tools like DTrace.

Today, more and more applications are scaled horizontally. Rather than spend time pouring over the internals of the system followed by extensive tuning to improve performance by 10%, it's easier to just add another server and ignore the problem. Because of this mindset, Linux hasn't required the kind of tools that Solaris brings to the table. I think that efficiency efforts may be changing this mindset, because people are realizing that the cost of a server over it's lifetime is much more than the initial purchase price.

What can be done about this? I think that Sun needs to do a better job of showing what these tools bring to the table. They should also show that most applications deployed on Linux can be deployed on Solaris with minimal effort and that the tools Solaris offers will let companies get more out of their hardware investment.

I also think that Sun should clarify the relationship between OpenSolaris and Solaris. If I were to start a business, in the beginning I'm more likely to use OpenSolaris on commodity hardware and find my own solutions to any problems that arise. As the business grew, I'd be more likely to be interested in hardware optimized for Solaris and long-term support. I think clarifying the road map could make companies considering OpenSolaris more comfortable in that decision.

The final thing that I think Sun should do is to calm people's fears about using Solaris. Many startups have minimal money to spend, and so they don't want to start with one platform only to run into a wall that requires them to shell out a ton of money. People know that Linux is Free, and that if they want to support themselves they can use it without paying for it and that nobody will come in and take it away. I think more small businesses would use Solaris and OpenSolaris if Sun can show that they are safe choices.

A work re-work: a rough draft

So what am I doing with this? So far, I've purchased an electronic organizer. I've opted for an Apple iPod Touch, which is very slim but has a nice calendar and supports third-party applications. The calendar application supports multiple calendars, so I can have both my work and personal calendars available, but not mixed together.

I've also purchased Things through the iTunes App Store from Cultured Code. It is a GTD-oriented task management application, and so far it seems to be working well. I've started to input all the projects that I have in progress at the moment. I'm also recording any task that I think of as soon as I'm able. The list is depressingly long and growing fast, but at the same time it's also a relief to have one place to look for work to get done. I haven't been using it long, but already it's a relief to know that I have a record of stuff that I need to get done.

I'm also going to try an experiment at the office this week. I'm going to start closing Outlook. I have my calendar available on my iPod Touch, so I don't need to worry about missing meetings. I'll try to check my mail every hour or two, and process everything then. I plan to notify the people who I deal with most often, but everybody else will just have to wait. If it's urgent, they'll be calling anyway.

A work re-work: time management

I've tried various organization techniques, such as blocking out time for functional areas of my job, trying to pre-schedule my weekly activities, etc. Unfortunately, those haven't really helped. For example, I find that when my calendar tells me that it's time to work on storage I don't know what I should be doing. I know there are things that need to be done, I don't know what I can do at the current time to actually move things forward.

That leads me to today. In an attempt to turn things around, I'm attempting to become better at managing my time. If I don't succeed, I think I'm going to suffer from major burnout. I've already had weeks where I just don't want to do anything at all.

My company has various online training courses, and I started with an online time management course. Unfortunately, it was targeted for people in a more traditional office job where they may have 5-10 major tasks to accomplish in a week, with maybe 20% change based on shifting requirements. Unfortunately, I've found IT to be quite different, with 50-80 smaller tasks to accomplish in a week, with 50-80% change based on shifting requirements.

I'm currently working my way through Time Management for System Administrators by Thomas Limoncelli. While I'm only about half way through the book, there are some valuable tips here. One of the most useful, I expect, is the Mutual Interruption Shield. If you work with other people, set aside times when you are not to be interrupted except in an emergency. The other person fields all questions and requests, which allows you to focus completely on the task at hand. Then you switch places. This gives both of you real focus time. He also advocates establishing routines. This allows you to get the routine work done without having to think about it. The author also provides a simple system for keeping track of tasks that need to get done, and when they're due.

I've also started reading Getting Things Done by David Allen. I'm not very far in, but this book seems a bit more philosophical. The organizational system that the author advocates is definitely more complex, but it may also be more robust. One of the core tenants of this system, as well as Time Management for System Administrators, is to get the lists of things to do out of your head and into some kind of trusted organizer. This can be paper or digital, just so long as it can be counted on.

A work re-work: background

I've been in my current job for nearly three years, and I've been very productive from early on. When I first started, I was able to work almost entirely on technical projects. This was largely because I was new and there were very few other demands placed on me. I established some good habits such as logging my work daily in OneNote, tracking tasks that needed to be done, etc. I became “the organized one,” a title I never thought would be applied to me.

For the past six months or eight months, I've felt like my productivity has been slipping. Some weeks I look back on Friday and wonder what I've accomplished. I'm spending more and more time communicating about work, and less time doing it. What good habits I developed have started to slip. I will miss days or weeks of work logs, and my task list is getting stale.

Unfortunately, at the same time the amount of work that I'm expected to complete has expanded during this same period. So now I have more to do, but I'm getting less done. I would guess that these are related. As I've become more busy, I've let my productivity tools slide. That has put me into a vicious cycle of getting more and more busy with less and less support from my tools. It hasn't reached crisis stage yet, but I know that there is a problem coming if I don't change something.

This cycle has left me feeling thoroughly unfulfilled at work. I've read both “Now, Discover Your Strengths” by Marcus Buckingham and Donald O. Clifton, and “Go Put Your Strengths to Work” by Marcus Buckingham. These have helped me to get a better idea of what I'm good at and where I need to be. Unfortunately, I'm still scrambling to try to get my current work done. This leaves me precious little time to try to leverage myself into a position where I play more to my strengths.

OpenSolaris power management

One aspect of OpenSolaris that I've taken an interest in is power management. In my research, I've come across several useful things.

One of the first things I tried to find was the current CPU speed to see if power management was doing anything. I finally stumbled across this post by Mark Haywood. In that post he shows these commands:

$ kstat -m cpu_info -s supported_frequencies_Hz
module: cpu_info                        instance: 0
name:   cpu_info0                       class:    misc
       supported_frequencies_Hz        2800000000:3200000000

$ kstat -m cpu_info -s current_clock_Hz
module: cpu_info                        instance: 0
name:   cpu_info0                       class:    misc
       current_clock_Hz                2800000000

These report the supported frequencies and the current frequency of the CPU. If the supported frequencies field shows a range as indicated above, OpenSolaris supports power management for your CPU.

Another very handy tool is PowerTop. It captures statistics about how much CPU time is idle, how much time it spends running at different speeds, and what is causing the CPU to wake up from idle most frequently.