Fear not! Help is on the way. Read on to learn how to use gdbserver and ssh tunnelling to debug remote processes.

Click to enlarge

In embedded systems development making do with less is the name of the game – less CPU power, less physical RAM and less peristant storage (if any!) to name a few. Debugging a misbehaving process in this environment can be challenging, but a little ingenuity coupled with plenty of free software eases the problem.

In this article I will explain how to use GDB for debugging remote processes running on embedded systems from our desktop workstation.

For the purposes of this article the embedded system is a RPX_Lite from EmbeddedPlanet, which I discussed in a previous article. This board has a blindingly fast (not) 80MHz 823e PowerPC processor with 16MB of RAM. It's pretty cute, a 3 inch square that includes ethernet, USB, serial and a PCMCIA slot. Just right for experimenting.

Major Differences

While many differences exist between desktops and embedded systems running GNU/Linux, the biggest difference is size and power. Embedded systems have very specific design goals limiting the overall power consumption and physical dimensions. This leads to the use of low power CPUs, limited physical RAM and little or no persistant storage devices.

The next major difference is the CPU architecture – embedded systems often use low power CPUs that are not x86 based. To compile programs from your x86 based desktop you need "cross-compilation tools", which run on your local desktop, but generate excutable code for the target architecture. In this article I will be cross-compiling for the PowerPC architecture.

The last major difference is perspective. You will be debugging a process that is executing on a remote CPU not your local workstation. This requires a slightly different mindset then traditional debugging.

ELF and Binutil Background

Before getting to the nuts and bolts here's a quick review about how executable code and debugging information is stored in an ELF binary. Most modern *NIX systems use the the ELF format for executables and shared libraries.

On a GNU/Linux system a family of utilities called binutils exists for examining and manipulating ELF objects. In a cross-compiling development environment the usual tool names like gcc, gdb and all the binutils will have a prefix that describes the target architecture.

In this article I'm targeting the PowerPC 823e and the tool prefix is "ppc_8xx-". I am using the wonderful Embedded Linux Development Kit (ELDK), which has complete toolchains for the PowerPC.

Let's play around with some of the binutils using a simple "Hello World" application:

  #include 

  int main(void) {
    
    printf("Hello World\n");

    return 0;

  }

First let's compile the program with debugging symbols using the -g option.

  ppc_8xx-gcc -g -o hello hello.c

Note I used the cross compiler, ppc_8xx-gcc, to compile the program for the PowerPC 823e target.

On my system the resulting binary size is 20632 bytes.

To see what symbols the binary contains use the nm binary utility. Remember I'm using the PowerPC version of nm, ppc_8xx-nm.

   coz:~/articles/rgdb$ ppc_8xx-nm hello
   10010868 D _DYNAMIC
   10010948 T _GLOBAL_OFFSET_TABLE_
   [... stuff deleted]
   1001085c W data_start
   10000408 t frame_dummy
   1000048c T main
   100109d4 b object.2
   10010860 d p.0
            U printf@@GLIBC_2.0
   
The interesting lines for us are near the bottom:
   1000048c T main
            U printf@@GLIBC_2.0

The first line shows the address of the main() function is 1000048c. "T" means the text section, which is an old term for the section where the code resides. The next line shows that printf() is an unresolved symbol and will be loaded from a shared library at run time.

Interesting.

The ELF format defines sections where various information about the executable is stored. The most interesting sections are:

• text -- where the executable code lives
• data -- where global variables live
• rodata -- where global "read only" constants live

In addition several other sections also are present, including sections containing the debugging information. To see all the sections and their sizes use the objdump binutil with the -h option to display section headers.

  coz:~/articles/rgdb$ ppc_8xx-objdump -h hello
  
  hello:     file format elf32-powerpc
  
  Sections:
  Idx Name          Size      VMA       LMA       File off  Algn
    0 .interp       0000000d  10000114  10000114  00000114  2**0
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    1 .note.ABI-tag 00000020  10000124  10000124  00000124  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    2 .hash         00000030  10000144  10000144  00000144  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    3 .dynsym       00000070  10000174  10000174  00000174  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    4 .dynstr       0000007a  100001e4  100001e4  000001e4  2**0
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    5 .gnu.version  0000000e  1000025e  1000025e  0000025e  2**1
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    6 .gnu.version_r 00000020  1000026c  1000026c  0000026c  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    7 .rela.dyn     0000000c  1000028c  1000028c  0000028c  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    8 .rela.plt     00000030  10000298  10000298  00000298  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
    9 .init         00000028  100002c8  100002c8  000002c8  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, CODE
   10 .text         00000528  100002f0  100002f0  000002f0  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, CODE
   11 .fini         00000020  10000818  10000818  00000818  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, CODE
   12 .rodata       00000024  10000838  10000838  00000838  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
   13 .sdata2       00000000  1000085c  1000085c  0000085c  2**2
                    CONTENTS, ALLOC, LOAD, READONLY, DATA
   14 .data         00000008  1001085c  1001085c  0000085c  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   15 .eh_frame     00000004  10010864  10010864  00000864  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   16 .dynamic      000000c8  10010868  10010868  00000868  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   17 .ctors        00000008  10010930  10010930  00000930  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   18 .dtors        00000008  10010938  10010938  00000938  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   19 .jcr          00000004  10010940  10010940  00000940  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   20 .got          00000014  10010944  10010944  00000944  2**2
                    CONTENTS, ALLOC, LOAD, CODE
   21 .sdata        00000000  10010958  10010958  00000958  2**2
                    CONTENTS, ALLOC, LOAD, DATA
   22 .sbss         00000000  10010958  10010958  00000958  2**0
                    ALLOC
   23 .plt          00000078  10010958  10010958  00000958  2**2
                    ALLOC, CODE
   24 .bss          0000001c  100109d0  100109d0  00000958  2**2
                    ALLOC
   25 .comment      0000016e  00000000  00000000  00000958  2**0
                    CONTENTS, READONLY
   26 .debug_aranges 00000020  00000000  00000000  00000ac6  2**0
                    CONTENTS, READONLY, DEBUGGING
   27 .debug_pubnames 00000040  00000000  00000000  00000ae6  2**0
                    CONTENTS, READONLY, DEBUGGING
   28 .debug_info   00001f81  00000000  00000000  00000b26  2**0
                    CONTENTS, READONLY, DEBUGGING
   29 .debug_abbrev 00000271  00000000  00000000  00002aa7  2**0
                    CONTENTS, READONLY, DEBUGGING
   30 .debug_line   00000232  00000000  00000000  00002d18  2**0
                    CONTENTS, READONLY, DEBUGGING
   31 .debug_frame  0000003c  00000000  00000000  00002f4c  2**2
                    CONTENTS, READONLY, DEBUGGING
   32 .debug_str    00000041  00000000  00000000  00002f88  2**0
                    CONTENTS, READONLY, DEBUGGING

WOW! That's a lot of sections and many of them contain debug information. These sections can add quite a bit of size to an executable and none of it is essential to running the program. The information is only useful when trying to debug.

Aside: the ctors and dtors sections are for "constructors" and "destructors", like those used for static C++ objects.

In order to save as much space as possible on an embedded system we often strip off all the non-essential information from the ELF sections. The binutil strip does just this.

   coz:~/articles/rgdb$ ppc_8xx-strip hello

Now the size of my executable is 4092 bytes, a reduction of 16540 bytes. That is over an 80% reduction – awesome! But it comes at a price. Without the debug sections debugging will be impossible...

Sort of. More on that later.

Remote Debugging With GDB

So now we have a stripped application that we can execute on our embedded system. But what if it is crashing and we want to debug it? A couple of obsticles sit in our way.

First, the target system has limited storage so we did not bother to put a cross-compiled version of GDB on it. On my development workstation the GDB executable is 9973975 bytes, nearly 10 megabytes. Clearly that won't leave much room for anything else if I only have 16MB total on the embedded system.

What to do?

The answer is to use gdbserver, a small footprint server that implements the low level features of GDB.

Consider the following diagram – using TCP the feature-rich GDB on my workstation connects to the light-weight gdbserver running on the embedded system. Most of the heavy lifting is done by the GDB on my workstation, while gdbserver deals with the low level interactions.

Click to enlarge

On my system gdbserver is only 59303 bytes, a considerable improvement over the size of the full GDB program.

In the following examples my workstation, coz, has the IP address of 10.0.0.10 and the embedded system, gw, has an IP address of 10.0.0.20 as shown in the above diagram.

The first step is to attach the gdbserver to a process on the target system. You can have gdbserver start a program and attach to it immediately or you can attach to an already running process using the process ID (pid). The last argument you need to specify is the TCP port that gdbserver will listen on. Here's the syntax:

   gdbserver host:2222 PROGRAM [ARGS...]
   gdbserver host:2222 --attach PID

In the above examples the gdbserver would listen on port 2222. Using gdbserver to launch our hello world application on the embedded system looks like this:

   gdbserver host:2222 hello
   Process hello created; pid = 23125
   Listening on port 2222

The prompt does not comeback as the gdbserver is now blocking, waiting for connections on port 2222.

The next step is to start the main GDB program on your workstation and connect to the gdbserver process. In order for the main GDB debug my program it needs to examine the "unstripped" version of executable that contains all of the debugging symbols. The simplest thing to is to chdir to the directory containing the unstripped executable and start the cross compiled version of the gdb, like this:

   coz:~$ cd ~/articles/rgdb
   coz:~/articles/rgdb$ ppc_8xx-gdb
   GNU gdb Yellow Dog Linux (5.2.1-4b_4)
   Copyright 2002 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public License, and you are
   welcome to change it and/or distribute copies of it under certain conditions.
   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show warranty" for details.
   This GDB was configured as "--host=i386-redhat-linux --target=ppc-linux".
   (gdb)

To "tell" GDB to read the symbols from the unstripped executable use the GDB file command like this:

   (gdb) file hello
   Reading symbols from /home/curt/articles/rgdb/hello...done.

If your application uses shared libraries and most real world applications do, then you also need to tell GDB where to locate these libraries. These need to be the unstripped versions of these libraries so that GDB can tell you more info.

Set the GDB "solib-search-search-path" variable so that GDB can find the shared libraries used by your application, like this:

   (gdb) set solib-search-path  [path to libraries]

If your application has a lot of shared libraries spread all over your source tree (and most real world ones do) then here's a little trick for the solib-search-path variable. Create one directory and populate it with symlinks to all of your shared libraries. Then you need only specify this one directory when setting the solib-search-path variable. Comes in handy.

Now we are ready to connect to the gdbserver running on the embedded system. We use the target remote command from the main GDB command prompt, like this:

   (gdb) target remote 10.0.0.20:2222
   Remote debugging using 10.0.0.20:2222
   0x10000120 in ?? ()

On the embedded system console you should see this output:

   Remote debugging from host 10.0.0.10

Now we are connected and the program being debugged is currently paused. Now would be a good time to set some break points and then continue running the program. Here's an example:

   (gdb) b main
   Breakpoint 1 at 0x1000048c: file hello.c, line 4.
   (gdb) continue
   Continuing.
   
   Breakpoint 1, main () at hello.c:4
   4		printf("Hello World\n");

And there we are! We are remotely debugging the stripped executable. Pretty cool, huh? I love it! You can now use all your favourite GDB commands and techniques to debug.

Personally I like running GDB from within Emacs, but your tastes may vary. You can use any GDB front-end you want for remote debugging. Very nice.

SSH Tunneling and GDB

Suppose you have the network topology shown in the following diagram and you want to debug a process running on the host wonkel. In this topology the host gw is dual homed with one interface on the 10.0.0.0/24 network and one on the 192.168.1.0/24 network. The host wonkel is also on the 192.168.1.0 network, while your workstation is on the 10.0.0.0 network.

Click to enlarge

The diagram also depicts a serial console connection from coz to wonkel – serial UART connections are very common on embedded systems for debug purposes, a back door for when the network connection is not working. Even though they are a bit slow, UARTs are cheap, reliable and easy to configure. A serial console is usually the first bit of hardware tested out when bringing up a new board.

The problem here is that no routable network path exists from coz to wonkel – gw cannot act as a gateway and forward packets in the normal manner.

What to do? I'll be honest – a lot solutions present themselves. You could configure iptables on gw to forward packets from coz to wonkel. That works fine if you have root access to gw.

Another method is to use the port forwarding capabilities of your old friend, ssh. The trick here is to forward connections on a local coz port to a port on wonkel – as a side benefit the traffic on the tunneled port is also encrypted by the SSH protocol.

Via the serial console you can login to wonkel and perform basic commands. Using the serial console start the gdbserver on wonkel, just like the previous example:

   gdbserver host:2222 hello
   Process hello created; pid = 23125
   Listening on port 2222

Now we need to create an SSH tunnel from coz to wonkel via gw. Here's the command to do that:

   ssh -L 4000:wonkel:2222 curt@gw

This opens a listening TCP socket on the local host, coz:4000. Whenever a connection is made to this socket it is forwarded to the sshd process on gw, which then opens a connection to wonkel:2222. This is shown in the following diagram.

Click to enlarge

Now we can start GDB on coz like before, but this time the "target host" command use localhost:4000, like this:

   (gdb) target remote localhost:4000
   Remote debugging using localhost:4000
   0x10000120 in ?? ()

This connection is tunneled via gw to wonkel:2222. On the wonkel serial console you should see:

   Remote debugging from host 192.168.1.20

Note wonkel thinks the debug request is coming from gw, not coz.

Now you can proceed to debug as before.

I hope you have found this intro to remote debugging and ssh tunnelling useful. All comments are welcome.

Happy hacking!

-Curt

---------------------
No commercial products were harmed in the writing of this article

The questions worth asking have not answers worth knowing, but rather they engender journeys worth taking.
– Curt Brune

This article describes the motivations behind and the inner workings of a "IP address to Google Map" mash-up application I wrote recently called Geo IP Route. Read on!.

What should I do?
Where should I go?
Am I sure this is the right direction?
What if I had chosen a different path?

Most of us have asked ourselves these types of questions at one time or another. It's only natural to ponder the meaning of our existence and wonder aloud at the complexity of the universe.

Like all good questions these questions apply to many different levels of our existance, from superficial daily decisions to subdermal compromises of family and career to the innermost vexing decisions regarding ethics and morality.

Off hand the questions arising from routine daily decision making are of little interest to me. "Don't sweat the small stuff," someone once said. Good advice as life is too short. Make decisions, take action and keep moving forward.

No, I find the subtle, difficult questions much more intriguing. As these decisions lie closer to our core being, the effects of minor course corrections dramatically influence the route we travel. While it's true destinations help focus your energy, upon arrival the journey gives more insight to where you are then the destination itself.

I would like to share with you one of my recent existential programming journeys. Along the way I ponder questions, fiddle with traceroute and geocoding. As I've written previously the best way to get going to is to have a project to work on. With the project in hand we can sharpen the saw and get to work.

Here's a snapshot of the final application:

Read on for more ...

Charting Your Route

I wanted to do something new, something new to me and to you. Also I wanted to do something different from my day job (embedded system design and programming). For the something different I wanted to learn how to make web pages with interactive maps using the Google Maps API.

If you have been following along you know I have a history of working with GIS and digital map applications. I had never played with online map applications though.

Now the something new to me and you was going to be difficult. Mapping APIs are great, but what are you going to put on the map? To put items on a map you first need their latitude and longitude, otherwise known as geocoding. Where could I get geocodes of something interesting? For free of course.

After much head scratching I decided it would be cool to map the Internet routers used for various transactions. If you are familiar with traceroute you know exactly what I was thinking.

I wanted to make sure what I was doing didn't already exist. I did a Google search for "traceroute geocoding" – much to my dismay several sites came up. Only one seemed relevant, in fact it was pretty much exactly what I was thinking of doing. Somewhat long faced I clicked on the link to check out the application. Much to my delight the application was non-existant, the link provided didn't work. I was still in business.

This project had several challenges that had to be surmounted in order to succeed.

The first trick was to convert the Internet Protocol (IP) addresses into latitude and longitude coordinates. I had a couple of ideas on how to do this.

The second trick was learning the Google Maps API and various bits of javascript to glue it all together. While this seemed straightforward at first plenty of surprises were lurking to insure an eventful journey.

With destination in hand, the navigation charts arrayed before me I was ready to embark on the journey.

IPv4 to Latitude and Longitude

How to do this? Companies sell databases that map IP addresses to latitude and longitude, along with a subscription service to keep the database up to date. That would likely be the mostly accurate way to go, but that would take all the fun out of it. Plus I don't have money to pay for it.

I harkened back to my experience which traditional geocoding, where the latitude/longitude (lat/long) is derived from a street address. Maybe a I could transform IP addresses to street addresses and then to lat/long. How could I get street addresses from IP addresses?

Given that this was a for-fun-not-for-profit project 100% accurracy was not required, so I decided to use the whois database to look up the administrative contact for given IP addresses. This was not the final solution I used for the finished application, but it was a start. Starting is key to any journey!

Clearly using whois was not going to tell me the exact location of the server in question, but it would tell me the location of the organization that ran the server. Good enough.

The following is the abbreviated whois output for LinuxDevices.com. First I looked up the IP address for www.linuxdevices.com and then fed that IP address into the whois command, using the whois.arin.net whois sever.

linux:$ whois -h whois.arin.net + 216.218.185.154
...
OrgName:    Hurricane Electric 
OrgID:      HURC
Address:    760 Mission Court
City:       Fremont
StateProv:  CA
PostalCode: 94539
Country:    US
...

Hurricane Electric is a co-lo in the San Jose, CA area. This is likely where the servers for LinuxDevices.com reside. Considering all the places the server could have been, this narrows it down considerably.

OK, great so I have the street address given an IP address. Now what?

I need a geo-coder. Free geocoders exist as web services, which are convienent, but also introduce latency as the query is serviced by a third-party. I didn't want to introduce more latency – I was already querying the whois database.

I wanted to do the geo-coding locally on my own server. Luckly the U.S. Census Bureau makes its Tiger Line data freely available for download. A nifty perl module, Geo::Coder::US, provides an interface to this database.

I was now in business – I could turn an IP address into a location on the map. While not optimal it got me going so I could move on to the next challenge, using the mapping APIs and making something worth looking at.

Google Maps and Javascript

Dear reader if you know next to nothing about Javascript then I knew less than you when I embarked on this adventure. Sure I had programmed a bunch in various language, but never javascript. I needed a good reference to get up to speed.

I found the W3 Schools Javascript Tutorial to be an excellent on-line reference. It is really, really helpful with nice working examples. Go there and read up.

The Google Maps API is well documented and pretty straight forward to use. My applicatin required two basic mapping features

• Draw a map, centered on a lat/long at a particular zoom level.
• Put icons on the map at particular locations.

Most mapping applications do little more than this. The Google APIs made this a relatively easy task.

In my application the user sends a series of IP addresses to my server, where I turn those IP address in to lat/long locations and send them back to the user's browser. The highlight for me was the using XML to send the data back to the browser.

The Google API provides a really nice XML parser making it simple to marshall arguments between web applications. What a time saver! Totally cool.

Another cool thing for me was using anonymous Javascript functions for asynchronous HTTP query/response handling. First an HTTP request is made and an anonymous function is registered as the response handling function. Later the response is handled asynchronously by the registered function. It was cool using an anonymous function for this – reminded me of the lambda functions of LISP.

Iteration and Closure

We are what we repeatedly do. Excellence then is not an act but a habit.
– Aristotle

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
– Antoine de Saint Exupery (1900 - 1944)

Just like writing an essay, creating good computer programs usually benefit from one or more development iterations – quickly putting down the first draft captures the fleeting flashes of insight bouncing around your ideas. Subsequent drafts smooth out the rough spots, refine the algorithms, shorten the code paths and delete the cruft.

Earlier I alluded that the final version of my application did not stick with the whois/geocoding method for turning IP addresses into lat/long coordinates. During development I found a more satisfying method.

The first enhancement was to realize that someone else was already offering a free service for U.S. and U.K IP addresses. Go check out hostip.info. This solved the problem for two large countries, but what about the rest of the world?

Well that was going to be tricky. In the end I knew the application did not need to be perfect, but only provide an estimate for the path that packets took across the Internet.

The hostip.info query fails whenever it does not have a location for an IP address – it can sometimes fail even for U.S. and U.K. addresses that it does not know about. When it fails, however, it does tell you the country that the IP address is in.

If the country is the U.S. then I fall back on the my original whois/geocode method, which works OK. For other countries I use the lat/long of the capital city of the country. So the resuls are not perfect, but the spirit of where the packets are going is preserved. I used the list of world capital lat/longs found here.

Until Next Time

Well I hope you have enjoyed this journey – good travels!

Oh, I almost forgot – please check out my Geo IP Route application.

A boot loader, sometimes referred to as a boot monitor, is a small piece of software that executes soon after powering up a computer. On your desktop Linux PC you may be familiar with lilo or grub, which resides on the master boot record (MBR) of your hard drive. After the PC BIOS performs various system initializations it executes the boot loader located in the MBR. The boot loader then passes system information to the kernel and then executes the kernel. For instance the boot loader tells the kernel which hard drive partition to mount as root.

In an embedded system the role of the boot loader is more complicated since these systems do not have a BIOS to perform the initial system configuration. The low level initialization of microprocessors, memory controllers and other board specific hardware varies from board to board and CPU to CPU. These initializations must be performed before a Linux kernel image can execute.

At a minimum an embedded loader provides the following features:

• Initializing the hardware, especially the memory controller.
• Providing boot parameters for the Linux kernel.
• Starting the Linux kernel.

Additionally, most boot loaders also provide "convenience" features that simplify development:

• Reading and writing arbitrary memory locations.
• Uploading new binary images to the board's RAM via a serial line or Ethernet
• Copying binary images from RAM to FLASH memory.

localhost:$ cd u-boot
localhost:$ make mrproper
localhost:$ make <board>_config

/* Serial port configuration */
#define CONFIG_BAUDRATE         19200
#define CFG_BAUDRATE_TABLE      { 9600, 19200 }

/* Network configuration */
#define CONFIG_IPADDR           10.0.0.11
#define CONFIG_NETMASK          255.255.255.0
#define CONFIG_SERVERIP         10.0.0.1 

cd u-boot
make

#!/usr/bin/kermit
echo connecting /dev/ttyS0 .....
set line /dev/ttyS0
set FLOW AUTO
set speed 19200
set serial 8n1
SET CARRIER-WATCH OFF
connect 

U-Boot 1.1.2 (Aug  3 2004 - 17:31:20)
RAM Configuration:
Bank #0: 00000000  8 MB
Flash:  2 MB
In:    serial
Out:   serial
Err:   serial
u-boot # 

Inside are the gems I discovered at this year's Embedded Systems Conference in San Francisco, including current directions in embedded Linux development, progressive silicon art, robots for science and entertainment, public transportation and the relentless drive of the economy.

The Approach The day began on an inauspicious note when I decided to use public transportation to commute from my apartment in the South Bay to the conference in San Francisco aboard the CalTrain. It seemed like a good idea – eco-friendly, no need to deal with traffic or parking. A quick shuttle from the CalTrain station to the Moscone Convention Center and the trip would be complete, or so I thought. Sometimes even the simple things take focused effort. While waiting for the train I gazed around, taking in the sounds and sights of a bustling train stop – after about 15 minutes it dawned on me that at least two trains should have arrived and departed, but none had done either. What was up? I pulled out one of my favorite and most useful portable devices – a tiny, portable radio that runs on one AAA battery. After two minutes with the headphones and KQED, the traffic report came around explaining that north bound CalTrain service was disrupted indefinitely due to a gas leak. Super! Sculpture I could take this news one of two ways. I could either get upset at CalTrain for delaying my trip, cursing technology up and down for the failure or I could be grateful that some embedded gadget somewhere had detected a gas leak and had notified living, breathing humans of the problem. So with my glass both half-empty and half-full I head home and drive myself to the conference. Once in San Francisco I park in the parking lot of SBC Park (formerly known as PacBell Park) and took a shuttle bus (public transportation at last) to the Moscone Center. Stay tuned – the return bus trip is worth remembering. The day was a typical, cool gray, drizzly Spring morning in San Francisco. Outside the Moscone Center a lustrous public sculpture dispelled the dreariness and I knew I was in San Francisco. This is going to be fun.

Keynote Address Despite my issues with public transportation I arrived with plenty of time to spare before the keynote address. Staking out my spot in the auditorium I began to take in the atmosphere and measure the mood of the crowd. The demographics are pretty pedestrian – mostly male, wearing blue jeans and toting their conference backpacks. The lights dimmed and it was time for the keynote address. Keynote Auditorium Dr. Robert Ballard, an oceanographer, geophysicist and marine geologist from the University of Rhode Island, gave the keynote address at this year's conference. He is probably most famous for leading the team that discovered the Titanic in 1985 – read his bio for more about this fascinating character. Dr. Ballard is not your typical speaker at an embedded systems conference, but he was one of the best I have ever heard. His main technical thrusts concerned bandwidth from the un-manned deep sea vehicles to the support vessels on the surface from which video would be up-linked via satellite to colleagues and middle school classrooms throughout the world. His encouraging philosophy of bringing remote exploration to the masses is embodied in The JASON Project, a multi-disciplinary program that sparks the imagination of students and enhances the classroom experience. Truly inspirational. But he did not have a lot to say about GNU/Linux, embedded processors, PCI Express or much else to do with electronics. Funny guy though, highly recommended. Hopefully his deep sea adventure tales can help steady the sails of the listing embedded systems economy.

The Exhibit Hall I won't bore you with a run down of all the exhibits, you can read the website for that, nor will I bore you with the obvious – embedded Linux is here today, growing fast and a force to be reckoned with. Conference Action For those you who have never been to an embedded systems conference let fill you on a few details. This conference deals with the fundamental components for making consumer and industrial electronic devices. You don't see a lot of polished, marketable consumer products like cellphones or game consoles – the exciting things you see are the next generation silicon, processor cores, software tools, development tools and serial interconnect products. With that said all the big hardware players had a GNU/Linux offering, some partnering with the likes of MontaVista, Red Hat, LynuxWorks and TimeSys. Everybody has development boards and board support packages (BSP) for GNU/Linux. At the same time, however, most of the big players also supported old school RTOS vendors like Green Hill Software, QNX and VxWorks. Embedded system teams are a conservative lot. Looking at the various embedded GNU/Linux vendors it was difficult for me to see a clear winner – all of them offer an IDE (many using eclipse), with integrated source level debuggers and kernel debuggers. Is there really that much difference between one vendor's IDE and another? I may be slightly biased, as I'm an emacs user, which does everything these IDE's do and then some. Call me a Luddite, I won't cringe. Less mouse, more coding. It seems pretty clear to me that we haven't had the final shake out in the embedded GNU/Linux distribution arena – too many vendors without enough differentiators. They all look the same and some will have to go. Only time will tell which ones. I promised I wouldn't bore you with the obvious so I will continue on with the less obvious.

ChipX art exhibit

ChipX, a structured ASIC company, had a fascinating gallery of silicon-inspired art from various artists. These artworks incorporated bits of silicon wafers, packaged ICs and other hi-tech left overs to create visually stunning pieces. A sampling from the gallery is shown here:

Other interesting things GNU/Linux Silicon Motion OKI, a company focusing on two-way communication products, offered several ARM7 based development boards using uClinux, a flavor of linux for CPUs with no MMU. For the parents out there OKI makes the CPUs inside the popular LeapFrog music toys for children – my 15-month old son loves his LeapStart Learning Table. A lot of the music toys for kids offend my ears, but the Learning Table uses digitized instruments with a high bit rate. Parents know, anything that teaches kids music and keeps them entertained without annoying the hell out of you is a wonderful piece of technology. Silicon Motion makes a wide range of low-power multimedia chips that are DirectX and OpenGL compliant (think nVidia/ATI without a honking heatsink and fan on the GPU). Here's a shot of their development board running MontaVista Linux – notice the GPU without a fan. Toshiba 64-bit Home Gateway Toshiba offered a wide variety of TV set-top boxes, DVRs and residential gateways running GNU/Linux. All of these products are powered by Toshiba's MIPS based 64-bit processors – that's a lot of horsepower for a set-top box. Check out their residential gateway.

Robots Spectrum Mars Rover Battle-Bot The conference had two noteworthy robots in attendance at opposite ends of the spectrum. One robot represented peaceful, scientific exploration and the other representing high testosterone, heart pumping entertainment. The first was a replica of the Mars Rover at the WindRiver booth – keep track of your flash memory. Seriously though, sending a robot to Mars and controlling it remotely with a round-trip light-speed delay of 40+ minutes is very impressive. But why run the thing with Java on top of VxWorks? Why not include some lisp and python while you are at it? The other really cool robots were the SozBots at the Tensilica booth. These mini battle-bots duked it out in a small arena, complete with safety Plexiglas cage. These little bots were a huge hit at the show. Here's a photo of a full sized saw blade battle-bot.

I strongly believe that we learn best by doing, that what you know is a synthesis between what you have studied and what you have accomplished. While there is no substitue for reading, attending classes and good mentorship, you will never know a field until you undertake a project and grapple with the details.

I had a physics professor, Dr. Reid, who taught "Mathematical Methods for Physics", which was a great class. My favourite part was "complex functional analysis", where we dealt with contour(path) integrals, complex residues and the like (see details). Rather abstract stuff, but it had practical applications.

Dr. Reid had a saying:

The process of learning physics is an active process. The information does not enter the brain through the eyes and ears alone, but rather it takes a slow journey through your fingers and hands, up your arm and into your brain.

In other words you actually need to work the homework problems to understand the material.

Indeed.

You cannot create experience. You must undergo it.
– Albert Camus

All of this makes me think of programming as an existential process. I cannot, however, claim to have invented the term existential programming (damn!). I don't think anyone agrees on what it really is, but the University of Kansas offers this course. My own interpretation is somewhat broader than this.

The type of programmer you are depends on the types of programs you have written and the programs you are writing now. Choose carefully and seize your destiny.

It Takes a Project

With philosophy and education as my guides it is time to embark on a project to truly learn the ins and outs of embedded programming. It would have been nice if my day job coincided exactly with my programming goals, but it did not. Not quite.

I am not down playing the useful experience I gain from my day job, but the goals here were a little different. The day job is all about embedded programming for the VxWorks operating system – in my heart of hearts I wanted to do GNU/linux embedded programming. Sitting around wishing and hoping I would do GNU/linux programming would have been foolish.

So once again I made some investments in my future. I decided to teach myself embedded GNU/linux programming by reading books, mailing-lists and by purchasing a small embedded processor board.

My first goal was to boot linux on the board. This proved to be a wonderful journey.

At first this was a little daunting – where to begin? What board to buy, what GNU/linux distribution to use, what books to read... What to buy and what to build myself? I went to the neighborhood technical bookstore and browsed the shelves looking for interesting books as a starting point. I found two that I highly recommend for different reasons:

• Embedded Linux: Hardware, Software and Interfacing
  Craig Hollabaugh, 2002. ISBN 0-672-32226-9
• Linux Device Drivers, 2nd Edition
  Alessandro Rubini & Jonathan Corbet, 2001. ISBN 0-596-00008-1

Let me tell you a little bit about these books:

Embedded Linux is a great hands on book. What I really like about it is that the author leads you through the design and development of a complete system (not just a single computer) that spans three different processor architectures – PowerPC, ARM and x86. This is great because the different architectures expose different parts of the linux kernel.

I will not bore you with a complete review of the book as a good review is already written here. Craig Hollabaugh also has a nice website that serves as a companion to his book.

It is a really great book. The author covers topics like portability as he shows how to write the same device driver for the three different target architectures. It really helps to see a problem solved in three different ways. Highly recommended.

Also Craig was very responsive to questions I emailed him as I went through booting my board for the first time. Very nice.

Linux Device Drivers, 2nd Edition is also a really great book. First off it is released under the GNU Free Documentation License and is freey available online from O'Reilly. I recommend buying the book as paper is so much easier to read.

It covers writing device drivers for standard 2.4 kernels targeted for the Intel x86 architecture. It does not cover embedded GNU/linux, however it does cover the kernel and device drivers in great detail. A very valuable resource.

Information's pretty thin stuff unless mixed with experience.
– Clarence Day

After reading most of Craig's book I was ready to purchase a board and get busy. To make my life a little easier I decied to stick to the same processor architecture I used at my day job, the Motorola PowerPC.

I purchased a RPX_Lite from EmbeddedPlanet, which has a blindingly fast (not) 80MHz 823e PowerPC processor with 16MB of RAM. It's pretty cute, a 3 inch square that includes ethernet, USB, serial and a PCMCIA slot. Just right for experimenting. This board is similar to the RPX_Classic that Craig uses in his book.

Next I needed a GNU/linux distrubtion to use with my board. This is not as easy as it sounds and Craig's book leads the way. The first thing you need is a cross-compiler for your board, so that you can build all the executables that your board will need. This includes not only the kernel, but all the supporting GNU software (cp, rm, ls, etc.) .

It is a lot of work to build a linux distribution from source. It is much easier to use a pre-built distribution for your board. This is what companies like Lineo, MontaVista and LynuxWorks are selling. See here for a good overview of embedded linux distributions.

Also I hit my first bump in the road when I realized my processor did not have a Floating Point Unit (FPU). Most available GNU/linux distributions for the PowerPC assumed you were running GNU/linux on a MacIntosh, whose processor do have an FPU. Those binaries would not work for me.

It looked like I was going to build everything myself...

Experience is not what happens to you; it is what you do with what happens to you.
– Aldous Huxley

Until I found the perfect distro for my needs called the "Embedded Linux Development Kit" (ELDK) from DENX Software Engineering in Germany.

The ELDK comes with pre-built binaries for a range of PowerPC processors including my version. This includes the cross-compiling GCC toolchain and hundreds of pre-compiled programs and utilities that come with GNU/linux.

Oh and the ELDK is GPL and available for free download. The price is right.

I must give special thanks to Wolfgang Denk (owner of DENX), who helped me numerous times via email and patiently answered my neophyte questions. Companies selling a commercial product should envy the customer support I recieved from Wolfgang – above and beyond my expectations. Truly awesome.

Now with the ELDK in hand I was quickly able to configure and build my kernel. I was getting tantalizingly close to my goal of "booting the board". All I wanted to see was the first login prompt from my little board coming to me over a serial connection. I desperately wanted to see the results of "uname -a" as proof that the board was up and running GNU/linux.

Since the board has a limited amount of memory I decided to mount the root filesystem over NFS – this allowed me to have access to all the programs of the ELDK (100s of megabytes). I had to setup an NFS server and configure the kernel to mount root via NFS.

Once that was completed the moment of truth was at hand:

Linux/PPC load: 
Uncompressing Linux...done.
Now booting the kernel
Linux version 2.4.4 (curt@hongkong) (gcc version 2.95.4 20010319
(prerelease/franzo/20011204)) #10 Sat Jan 11 14:40:07 PST 2003
Calibrating delay loop... 63.89 BogoMIPS
Memory: 14752k available (904k kernel code, 364k data, 48k init, 0k highmem)
eth0: CPM ENET Version 0.2 on SCC2, 00:10:ec:00:32:d6
VFS: Mounted root (nfs filesystem) readonly.
INIT: version 2.78 booting
Remounting root filesystem in read-write mode: 
Mounting proc filesystem:  [  OK  ]
Configuring kernel parameters:  [  OK  ]
Activating swap partitions:  [  OK  ]
Checking filesystems    [  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling swap space:  [  OK  ]
INIT: Entering runlevel: 3

hoho login: root
Last login: Thu Jan  1 00:00:36 on console

hoho:~# uname -a
Linux hoho 2.4.4 #10 Sat Jan 11 14:40:07 PST 2003 ppc unknown

hoho:~# ping www.ucdavis.edu
PING www.ucdavis.edu (169.237.104.199) from 10.0.0.9 : 56(84) bytes of data.
64 bytes from 169.237.104.199: icmp_seq=0 ttl=51 time=79.617 msec
64 bytes from 169.237.104.199: icmp_seq=1 ttl=51 time=69.767 msec

hoho:~# cat /proc/cpuinfo 
processor	: 0
cpu		: 8xx
clock		: 64MHz
bus clock	: 32MHz
revision	: 0.0
bogomips	: 64.39
zero pages	: total: 0 (0Kb) current: 0 (0Kb) hits: 0/0 (0%)

Sweet! It was working. My first project was complete – the board booted.

Life must be understood backwards. But it must be lived forward.
– Soren Kierkegaard

Now that the board booted I need a new goal, something embedded. Fun fun fun.

Cheers, Curt

Recommended Books

• Embedded Linux: Hardware, Software and Interfacing
  Craig Hollabaugh, 2002. ISBN 0-672-32226-9
• Linux Device Drivers, 2nd Edition
  Alessandro Rubini & Jonathan Corbet, 2001. ISBN 0-596-00008-1

Recommended URLs

• http://www.brainyquote.com/quotes/
• http://www.denx.de/
• http://penguinppc.org/
• http://www.linuxdevices.com/
• http://alllinuxdevices.com/

SRAM, DRAM, SDRAM -- Do you know the difference? I didn't and everybody else did. Processor data cache lines, memory subsystems (what is a single beat read anyway?), LVDS drivers, pull-up resistor, bias voltage, tri-state logic, BSP, VHDL. It was like hearing a foreign language in a bad dream -- you recognized the words, but you have no idea what they mean and everyone around you nods knowingly.

On top of all the electronics terms were all the physics, mathematics and statistics terms. What's a Poisson distribution? A chi-square? Probalistic histogram? Hmmm, college calculus class was over 10 years ago ....

So I began re-educating myself. I found a couple of handy technical dictionaries on the web:

• Math Thesaurus
  
• Technical Encyclopedia
  
• Electronics Terms
  
• Law Dictionary
  

How did that last one sneak in there? It's a good one to have anyway.

These links helped hide some of my ignorance, but honestly when the conversation was 30,000 ft over my head I just asked questions. The answers were always forthcoming -- no funny looks.

Another area were I thought I knew something, but realized quickly that I didn't, is C compilers. I had programmed a lot and written dozens of C programs, but I never gave much thought to the compiler (or the linker for that matter). One piece of advice for life in the embedded world:

Know Thy Compiler and Toolchain

Never before had the destinctin between operating system and architecture mattered. Now I understood that you can have differenet operating systems on the same hardware architecture (e.g. linux and Windoze running on i386 hardware) and you can have the same operating system running on different architectures (like linux running on i386 and on PowerPC).

In our project the data acquisition system software runs the VxWorks operating system for the PowerPC architecture (our computers are 3U CompactPCI single board computers based on the Motorola PowerPC microprocessor). I in no way endorse VxWorks as a hard real time operating system (RTOS). The company behind VxWorks, WindRiver, is the Microsoft of the embedded OS world. I will not say another word about VxWorks -- it is the hand I was delt.

If you are just starting out look seriously into running linux on your embedded system. If you have "hard real time" requirements consider using the real time extentions to linux. Linux on the small devices is here to stay.

Motorola, on the other hand, is great. They give away (as in free) tons of documentation about their processors. They will even mail you printed manuals for their processors, for free. Very useful. Try their document search.

In our work we cross compile the programs for the embedded system. Briefly cross compiling is when you run the compile/link stage on one OS (e.g. Solaris), but the resulting binary is for a different OS/Architecture (VxWorks on PowerPC). For a more detailed treatment of cross-compiling for the PowerPC using the GNU Compiler Collection see here.

Also read up on GCC in general, particularly the sections about GNU extentions to C and C++. Some of these are quite useful.

In our testing we try to exercise our code as much as possible before downloading it to the embedded processors. To this end we cross-compile our code for three different platforms:

• VxWorks on PowerPC -- the real system
  
• Solaris on UltraSPARC -- development system
  
• Linux on i386 -- development system
  

While we cannot debug everything on the development system (some tests must run on the real embedded system), we can flush out many compile time bugs by pushing our code through three seperate compile/link stages. Sometimes different compilers for different architectures catch different bugs and warnings.

I could write an entire article about the compile/link process we use -- but I won't do that here, unless people are interested. I find it interesting.

The first part of my re-education is well underway. I was getting a good handle on my tools. It is very important that a craftsman understand the benefits and limitations of his tools.

Now I needed a project to work on ...

I took a 20% pay cut to work at SLAC, a DOE National Laboratory operated by Stanford University. What I recieved in exchange for the 20% has been absolutely invaluable -- I learned how to think, I learned all about how the computer/interfacing world operates. I learned and understood the importance of system engineering. More on that later.

The project I am working on is called GLAST, a large gamma-ray observatory satelite that will measure gamma-ray sources throughout our universe as the satelite orbits the Earth. I work on the embedded system software that runs the data acquisition system for the detector.

I'm right where the rubber mets the road -- where an electrical engineer's piece of super custom hardware meets a Motorola PowerPC single board computer. I write device drivers for one-of-a-kind hardware devices. Oh yah, and the whole system will be stuffed on the tip of an explosive rocket and launched into orbit.

Deliciously geeky.

Up front though I want to give props to the physicists, researchers and engineers who toil away building the best possible machines for measuring and understanding the universe in which we live -- they dedicated their lives to scientific pursuits.

SLAC is a very interesting place to work -- it is one of the few DOE labs that does not do defense research, i.e. it is not a bomb factory. Also all of the projects and results are open for public scrutiny -- no so called black projects that occupy so many defense and aerospace companies. The research is funded with our tax dollars and is open to you, me and everyone else.

That is refreshing.

It is also refreshing to work with people who have decided to spend their entire life at SLAC for that very reason. These are sharp people who believe in what they are doing, but with humility, graciousness and a sense of how things could be much, much, darker. More about the dark side in a future post.

So the employment was a fair trade I'd say. They picked up a seasoned computer programmer who also had a physics background. It's just my computer programming background was writing applications -- it was now time to dive into and beneath the operating system and write device drivers. This is the area I wanted to explore, the embedded system. While I knew nothing about the low level world or embedded systems it seems fascinating and valuable as computers become more pervasive as they shrink smaller and smaller.

For their part they signed up to teach me embedded system programming. Or so I thought. So far it has turned out to be much, much more. They taught me about system engineering. How to architect a complete, complex system of custom electronics and software. How to think way, way outside of the box when designing (and debugging) an ambitious compu-electronic data acquisition system.

I feel the pain of the hiring managers at SLAC -- everything is a trade off. They have ridiculously small budgets, but they need really experienced engineers to build the devices they design. However all the really good, experienced engineers all work in industry where the pay is about 20% higher. So they make do with the people they can afford, knowing they will have to do a bit of teaching to make it all work.

I'm sure they don't realize it, but I have been absorbing as much knowledge as possible at every opportunity.

It is not always fun, however.

I came from running my own company, where I was in control and I decided what needed doing when. This was a change -- someone else was in control for the direction. Even more than that I had no experience in the field at all. It's quite an unsettling feeling after being in charge for so long. I did not know enough about embedded systems to know if their ideas were right or wrong.

I felt small.

And dumb.

To be continued ...

So I hopped off the train.

I quit.

Resigned, from the startup -- poof went all my "equity". A little less than two years after the madness had begun it was suddenly over. All I had left were 500 now-useless business cards.

But I had a plan, of course.

"Hey, I can do some consulting work... Work for myself... yeah. I just might do that... Yeah.. That's what I'm going to do... YEAH."

Being unemployed did not worry me at all... At least it didn't until I actually started looking for work.

Then it worried me a bit. No work for consultants -- economic downturn -- the big boys were having layoffs and weren't renewing contracts with the people they had. No consulting jobs for you, Mr. SmartGuy.com .

Luckily for me I had a very understanding fianc�e (we're married now) at the time. "I can support us while you look for work", she said. She is the BEST, without whom I would be a complete mess. It felt weird to be supported by someone else, like being out of control -- assuming you are in control is one of the biggest mistakes you can make.

While being unemployed is spooky it has its perks as well.

I spent a lot of time reading: technical articles, well known "Art of Programming Vol. 1", newsgroups, mailing lists and some fiction too (read LOTR in preparation for it's release in 12/01). I cannot over emphasize how important reading for pleasure is -- it's absolutely amazing how fun it is to read a good book. Turn off your televisions and read something you enjoy ( BTW never, ever torture yourself by finishing a book you don't enjoy).

Being between jobs also forces you to ponder the troublesome questions, "What do I want to do?" More of the same? Something different? Both? Neither? Move? Stay?"

At the startup I co-founded I was the VP of Engineering, managing a team of about 20 people including development, ops and QA -- and managing not to grieviously maime some of my fellow founders and executives. Effectively managing people is an art requiring vision, organization, patience and flexibility -- I've had good manager/mentors and I've had horrible schedule-centric dolts with no personality. I tried to "do my best".

Managing people reminded me of teaching physics to bio-sci majors while studying particle physics in graduate school -- sure the students wanted to learn, but the current subject was of little interest. At UCSB ( U Can Study Buzzed ) most students would rather not be in an electronics lab on the 4th floor of the Physics building on a Friday afternoon.

As a manager I tried to challenge people, to push them slightly beyond their current comfort zone and in the direction where I needed work done. I found providing people with a method/process as an approach to problem solving is very valuable, especially when I did not know exactly what to -- I relied on the employee's expertise and let them reason it out.

employee: But I don't see how I'm going to balance the load of
spawning user logins across the server farm. How do we
figure out where to spawn the next login session?

me: Well, what are our precious resources consumed by logins?
What can we monitor about the farm?

emp: Memory usage, CPU usage, open file handles, number of processes,
uptime, current sessions.

me: Derive a load function using those variables and minimize it.

emp: uh, uh. (unsure)

me: The machine is "loaded" whenever those variables are high. Low
values mean the machine is not so busy, right?

emp: OK. (nodding)

me: Consider a load function as a linear combination of those system
variables. Query all the machines in the farm and spawn
the new login on the machine with the minimum load
function.

emp: Linear what? I'm just going to add them together with different
scaling factors for each term. (Beaming as comprehension
washes over)

me: That's an even better idea.

emp: It also seems if the load function is too high for every
machine we should post a message saying all machines are
busy.

me: That sounds like an excellent idea to me.

An so it goes... Managing people had its rewarding moments. It also
has it tough times too: firing a problematic employee and laying off a
good employee and friend.

But I digress, this article is about programming. I wanted to get
back to programming, real programming. I wished to be a linux kernel
hacker, but I knew little about it. There was something appealing
about getting closer to the iron, but I could not quite put my finger
on it.

Whenever I wanted to learn a new programming language (like python) it
always took me a project to really get an understanding. I could read
the book, but I didn't know what it was like until I actually did
something -- typing in the examples didn't not count. It takes some
problem, some "itch" to stratch to really get into something.

I knew my programming quest and getting closer to the iron would be
like that too.

__Job_Hunting__

I applied to various jobs and head hunters. The head hunters did an
excellent job of finding me jobs doing what I had done previously --
they said I'd do well in executive engineering positions, which did
not interest me in the slightest. A recurring theme in my life --
being good at something, but not really wanting to do it for a living.

I wanted to code.

I wanted to code for real. I wanted the end result to have some
significance, some kind of purpose. I wanted to care -- I was tired
of not giving a shit.

I went to one job interview for a wireless handheld company that was
making sync-stations in the car. The sync-stations wirelessly
communicated with their backend servers delivering "content" to the
handhelds (that is if you consider stock quotes and sport scores
content -- why do wireless companies always advertise stock quotes? ).

Their "technology" seemed like YAWS to me -- Yet Another Web Server.
I was getting bored with web serveres, connected to app servers,
connected to database servers. After a while they all look the same
with no real innovation happening.

The kiss of death for my employment at said company came when I asked
them my favourite interview question "How do you keep Micro$oft from
eating your lunch?" The answer was "Mumble, mumble, we have partners
... broad support for our platform.... mumble, mumble, partners..."

Yeah right and I'm the Easter Bunny. Be afraid, be very afraid when a
company leads with its "partnerships" as a selling point. It's
double-speak for "These companies agreed to let us tell others they're
our partners if we give them our software for free. We have no real
revenue, but hope to seed the market with our partners." Run -- don't
walk -- away.

The answers you want to hear involve the word "customer", not
"partner". Having customers implies someone, somewhere is willing to
exchange ca$h for something the company has. While that seems simple,
it was an excruciating lesson I will never, ever forget.

__Job_Fair__

I went to a job fair in San Jose and was suprised to see some
companys' booths with signs reading "No Recent Graduates", "Minimum 2
years experience" -- tough times, glad I had experience. Too bad what
I had experience in didn't interest me that much any more. I headed
straight for the hardware companies.

company: Do you have any experience writing device drivers?

me: Sort of. Does writing a linux kernel module count?

company: That depends. What did the module do?

me: Printed "Hello World" in /var/log/syslog using printk(). (grin)

compnay: I don't think so.

After trolling the isles of the job fair for a while I went to the
booth of a company that makes energy effecient microprocessors and
also employs the creator of a well known open source UNIX-like kernel.

company: What kind of experience do you have?

me: I have two degrees in physics and I've been programming for
over 10 years.

company: What can you tell me about pipe lining in microprocessors?

me: Well, an instruction enters the pipeline and starts to be decoded,
as it moves through part of the instruction is fetched from
memory, while other units prepare for branching and and
and.... Something like that.

company: good luck.

The crossover was going to be harder that I thought.

__In_Between_Jobs__

My unemployment was going on three months when an old friend of mine
named Jim called asking what I was up to. Jim is an interesting guy,
a seasoned programmer and a quick thinking smooth talker -- just the
kind of guy to have an interesting business proposition.

Just what I needed.

As usual Jim had lots of ideas and "prospects", but little money.
Would I help him pull together YAWS(see above) for a suite of projects
he had brewing? I had nothing going on either -- "What do I have to
lose?", I thought. At best I earn a little spending money, at the
worst I keep my brain busy while I keep looking for something
permanent.

Plus it was Jim. It was like helping family. I was not expecting
compensation at $100/hr for my time. The point was to stay active,
stay in the race.

Later I realized I cared more about that job I did for Jim than I had
about anything during the past six months. It's weird, but the less
money was involved the more pride I took in the work. I was doing it
because I wanted to.

That and $950,000.00 can buy you a house in Palo Alto, CA. I
obviously could not work for free, but my salary expectations
definitely changed for the betterment of my state of mind.

__Ultimate_Frisbee__

Since my youth I have lived an active lifestyle -- basketball,
running, softball, hiking and ultimate frisbee. For those uninitiated
ultimate frisbee is "sort of like football only with a frisbee",
played on a grass field slightly larger than a soccer field. With
seven people on a team the idea is to catch the "disc" in the end zone
for a point. It has some similarities with basketball regarding
defense and getting open -- it's an incredible cardiovascular workout
do to the size of the playing field and amount of ground you have to
cover.

The fundamental rule in ultimate is "no bodily contact", none. This
is very unlike basketball (no boxing out) and soccer (no slide
tackles). No contact, means no contact. It's an incredibly simple
rule and easy to "call" -- at the "lunch time pickup" level very few
arguments ever occur regarding fouls. Almost none.

This keeps the games very civil for the most part (note: the national
and world tournments are very competitive and the contestants are not
always so nice, I'm talking about noon-time pickup games in the park).
This is in stark constrast to noon-time basketball and soccer games
I've been in where people think the world championship is on the line
and argue all the time.

People play ultimate because they enjoy running and jumping around
outside on a grassy field in the sunshine. The people who play
ultimate tend to be genuinely nice people, interesting people from all
walks of life: musicians, teachers, bankers, engineers, acocuntants,
students, translators, business people, doctors, scientists, mothers
and fathers.

Deals of many natures (business or otherwise) are known to occur on
the fields of Ultimate.

One fortuitous sunny day in May I was tossing some warm-up throws with
Ron, a guy who works for the major university in Palo Alto, California
whose mascot is a shade of red -- at the high energy physics
accelerator center.

Lacing up my cleats I querried, "What are you working on these days?
I studied physics for a while in grad school."

Whizzz went the disc.

"We're doing something new in high energy physics. We're launching a
satelite based gamma-ray observatory. We know about building atom
smashers, but the space environment is something altogether
different." as Ron tossed a high arcing outside-in forehand throw.

Catching the disc I thought to myself, "Why the hell not?". "Hey, are
you guys hiring people?", I asked as I threw the disc with a quick
snap of my wirst.

"I'm not, but the guy two doors down the hall from me is. He's in
charge of the embedded system software that runs on the instrument
when it's in orbit. I don't know too much about it, as my job in the
experiment is dealing with the data once it gets down to earth -- his
job is to capture the data and get it to the ground. If you're
interested I can get your resume in front of him."

When you least expect it history comes up and bites you on the ass. I
thought I was done with Physics for good.

It turned out the position was for a programmer to be part of the
"flight software" team -- software which would fly on aboard the
satelite. It was all about embedded programming: hard real-time,
multi-processor programming on Motorola PowerPCs driving custom
particle detector flight electronics.

Cool, I could get into this. Instead of programming up from the
operating system and to applications, it would be programming down
from the OS through device drivers -- interrupts, semaphores, ring
buffers, spin locks, all manner of techniques I knew very little
about.

I was concerned it was going to be the same old story again -- no
experience, no job. To my suprise after the interview I had a job
offer two weeks later. The fact that I knew computers and some
physics (as opposed to knowing physics and some computers like all the
physicists) was just what they were looking for. That I was not a
embedded guru, but wanted to get into it was fine with them.

It also did not hurt that I was willing to take a 20% pay cut going
from industry to a scientific research lab. It was like when I worked
for my friend Jim -- it's not completely about the money, I want to do
something I care about. I wanted to do something I was proud of.

Besides it sounded totally cool -- it's every boy's dream to have
something to do with a gigantic rocket's firey blast off.

to be continued...

The next installment

"Back to School"

email me and I'll send you next month's installment -- curt@acm.org

In these heady daze(sic) of .NET, mono, SOAP, java, xml, object oriented software design and "network as computer" metaphors we (professional programmers) seem to have lost sight of the "computer as machine" truism. Despite all the power of abstractions a fundamental understanding of how the machine really works remains invaluable.

Like many modern engineers today I began life with a computer in the house -- OK technically not since birth, but since the time I was 8 years old and Pops brought home that Apple ][+ things just haven't been the same (with a little math YOU can date me). By no means do I consider myself a great programmer, or even a good programmer -- in polite company I stifle a cough when saying I'm a "Software Engineer", quietly knowing I'm a little more than a quick hack. I know people who can code circles around me in various problem domains, whose understanding of the underpinnings of the problem far exceed my own.

I am an accidental programmer.

That first Apple that my father bought did little to teach me the "Art of Computer Programming". What it did teach me is that playing games like Ultima, The Bard's Tale and Wizardy was wickedly fun -- not a very practical thing to know.

As a teenager I spelunked into 6502 programming enough to implement a bubble sort algorithm simply because I could, not out of necessity. I also played with "Apple Pascal" -- it's compile and link stages fascinated the BASIC programmer I was. I thought it cute I had to shuffle four 5.25" floppy disks in and out of the drive to compile and link a simple "Hello World" application. The LOGO-like "TurleGraphics" was also very cool, with simple statements like
move(30);
turn(120);
move(30);
turn(120);
move(30);

used to make isoscelese triangles.

All things being equal computers rated about a 6.5 on a scale of one to ten, somewhat behind playing basketball during the day and drinking cold beer during cool summer nights in the Mojave Desert where I grew up. After all you couldn't get a computer drunk under a stary desert sky as you unhooked her bra all the while explaining that the twinkling starlight was billions of years old. (BTW the desert is an excellent place to be a teenager, where one can get into all manner of red neck rural mischief -- guns, dirt bikes, beer, explosives ... all the stories you've ever heard are true.)

Computers did rank above going to church on Sunday's and watching TV strangely enough. Computers were fun, but my sport-o, jock-o, red neck friends could care less -- the sun is shining (the sun is always shining) outside in the desert.

Being mathematically inclined and curious about the universe I enrolled as a Physics major at a University of California. Physics seemed like the ideal major for one whom mathematics came easily, but for whom actually **doing** math was tedious. The beauty of partial differential equations, wave mechanics and complex numbers intrigued me -- using these concepts to model and understand the sights, sounds and experiences of the physical world around me was extremely compelling.

The further I went in physics the more intriguing it became, going from simple torque equations that explained why I could ride my bicycle with no hands to the more esoteric quantum mechanics, general relativity, high energy particle physics, astrophysics and cosmology. Very heady concepts for a young man set about understanding the universe and securing his place in it.

But I digress, this article is about programming.

As a physics major I received my first mainframe computer account unlocking a Pandora's Box that has not closed since. Gopher, FTP, email, who-is -- all these people connected to machines, which in turn are connected to more machines with people connected to them.

Fascinating.

People to machines. Machines to people.

Fascinating, but not physics.

Physics was all about FORTRAN, long doubles, complex variables, crunch, crunch, crunch. While in graduate school in the early 1990s two realizations would lead me to discover that Physics was not for me, sort of.

First, while questioning the universe I thought physics held some answers to the unexplainable. So I attended a seminar on "Early Universe Cosmology" given by visiting theoretical "bad-asses" from around the globe. What a disappointment to find these "experts" arguing over the fundamentals of their science. In truth they knew just as little as I did, perhaps less.

Second, while writing pattern recognition software for a high energy physics detector I decided that FORTRAN was crap and would instead write my filter routines in C and figure out how to link to the monolithic FORTRAN libraries (we were running Ultrix at the time). A fellow grad student questioned this approach to which I dryly replied "If physics does not pan out I hope to get a real job some day." (BTW Graduate school is a great place to see what you do not want to become -- either a grumpy 7th year student, or worse an even grumpier post-doc making slightly less than the campus grounds keeper on four hours of sleep per night.)

It was also clear I was no Einstein, that I was not destined to be a theoretical physicist ( my hat's off to those that are!) . At 23 years old I was hopelessly washed up as a theoretical physicist and the thought of spending all my time indoors in a basement laboratory sounded, well ... horrible.

That the summer the kiss of death came: I discovered this little program called mosaic. The rest, as they say, is history. I dropped out of grad school, packed my bags and landed a job in the heart of silicon valley -- Menlo Park, CA.

__Close_but_no_cigar__

I landed in Silicon Valley, but I did not land on the Internet. When the dust cleared I found myself at a digital map company, writing software that integrated text tour guide information with digital map data. All of this was for GPS car navigation systems and had little to do with the Internet (though a couple of people left said digital map company to go and found MapQuest about the time I joined).

Here I really cut my teeth with UNIX on SunOS 4.1 . Here I had to keep up with people who were "real" computer scientists, people with Ph.D's in mathematics and topology (calculating routes on a topologically correct road network map is non-trivial). These people were good to me and taught me many programming principles, most of which were gleaned by trying to grok their source code.

I also learned to leverage the power of an OS with virtual memory -- I really could allocate a gigabytes worth of unsigned ints, even if my Sparc5 workstation only had 128MB of RAM. Really, how cool is that? What a beautiful abstraction! That discovery also marked the beginning of my decline -- I started to become a really lazy programmer.

From there I moved to C/C++ Win32 programming and sloppy GUI design. I copied and pasted, cursed, used the debugger, swore and generated the most horrible looking makefiles. At one point I even generated afx DLLs that worked only during certain phases of the moon. Nasty bad habits.

Then things took a turn for the worse....

I was assigned to a Visual Basic project -- now I really could sit back and forget all about how the underlying OS machinery worked (except when it came time to write the InstallShield, which unfailing fucked any system we installed to -- uninstall, my butt). VB made life so simple, so easy, so ... dangerous.

About this time the Internet madness was really picking up steam, when it seemed like any hack (me) with confidence could get funding for even the most hair brained idea. It was time to found a start-up Internet company. What could be more en vogue?

Luckily for me and my partners funding a bad idea is a two-way street -- first it takes a hare brained idea and then it takes someone zany enough to invest in it. Coupled together you really understand the phrase "burn rate". The story about the formation, function and dis-function of that start-up I will leave for a future article. Remember I promised this story, I mean article, would be about programming.

Our start-up was based on a well known GNU operating system including Apache, PERL and postgresql. Perl is great for implementing whatever you want in the shortest amount of time possible ... a hacker's language.

Among other things our company was into the nuevo chic business of wireless e-commerce. We were right in the middle of everything that was happening: web meets phone meets bank account. Who has the most efficient method of moving money from one wallet into one's own?

We quickly adopted SOAP before most application server vendor's supported it -- it seemed like the perfect way to get disparate machines talking the same language quickly and easily. Our colleagues on the other end of the pipe balked at first, but ultimately gave into our design more out of fear of being left behind in the information arms race than anything else.

__Kaboom__

Then the bubble burst... wireless wasn't hot, e-commerce wasn't hot, nothing was hot. Our fun little start-up whirlwind settled down to little more than a gentle breeze on a Sunday afternoon.

It was about this time I started going to a little Lutheran church around the corner from my apartment -- I was raised Lutheran and I could draw some comfort from ceremonies repeated throughout my childhood. Something stable.

The first Sunday my fianc� and I attended Church we stood around after the service talking with various people. One tall older gentleman came up, shook our hands and asked if we were new to the area. Did we work at Stanford?

"No," I replied. "I work at a small Internet startup," I said sheepishly while stifling a cough.

"Oh, well don't tell me anything I need to sign an NDA for," he said.

"Don't worry about that," I laughed. "All of our stuff is open source and public domain anyway."

The old man cocked an eyebrow at me and said, "I've written a few open source programs in my day. Perhaps I didn't introduce myself. My name is Don ... Don Knuth."

Holy shit. Exposed. In church no less. In front of a true computer scientist, if not THE defining computer scientist.

"I have your books on my bookshelf," I gushed like a school girl.

"A lot of people have my books on their bookshelves. That's the problem, they need to read them," he said.

I don't remember much of what happened next. Later my fianc� told me I looked sort of pitiful with my mouth hanging open. She had never seen my fast talking self stopped dead in my tracks. It was terribly humbling. Terribly.

It was then I decided that my next gig would be to uncover the lost art of programming. Time to quit screwing around and actually learn the trade.

to be continued...

The next installment

"Crossover -- WSDL to Gamma Ray Telescopes"

email me and I'll send you next month's installment -- curt@acm.org