January 11, 2003
Existential Programming
A man who carries a cat by the tail learns something he can learn in no other way.
– Mark Twain
I strongly believe that we learn best by doing, that what you know is a synthesis between what you have studied and what you have accomplished. While there is no substitue for reading, attending classes and good mentorship, you will never know a field until you undertake a project and grapple with the details.
I had a physics professor, Dr. Reid, who taught "Mathematical Methods for Physics", which was a great class. My favourite part was "complex functional analysis", where we dealt with contour(path) integrals, complex residues and the like (see details). Rather abstract stuff, but it had practical applications.
Dr. Reid had a saying:
The process of learning physics is an active process. The information does not enter the brain through the eyes and ears alone, but rather it takes a slow journey through your fingers and hands, up your arm and into your brain.
In other words you actually need to work the homework problems to understand the material.
Indeed.
You cannot create experience. You must undergo it.
– Albert Camus
All of this makes me think of programming as an existential process. I cannot, however, claim to have invented the term existential programming (damn!). I don't think anyone agrees on what it really is, but the University of Kansas offers this course. My own interpretation is somewhat broader than this.
The type of programmer you are depends on the types of programs you have written and the programs you are writing now. Choose carefully and seize your destiny.
It Takes a Project
With philosophy and education as my guides it is time to embark on a project to truly learn the ins and outs of embedded programming. It would have been nice if my day job coincided exactly with my programming goals, but it did not. Not quite.
I am not down playing the useful experience I gain from my day job, but the goals here were a little different. The day job is all about embedded programming for the VxWorks operating system – in my heart of hearts I wanted to do GNU/linux embedded programming. Sitting around wishing and hoping I would do GNU/linux programming would have been foolish.
So once again I made some investments in my future. I decided to teach myself embedded GNU/linux programming by reading books, mailing-lists and by purchasing a small embedded processor board.
My first goal was to boot linux on the board. This proved to be a wonderful journey.
At first this was a little daunting – where to begin? What board to buy, what GNU/linux distribution to use, what books to read... What to buy and what to build myself? I went to the neighborhood technical bookstore and browsed the shelves looking for interesting books as a starting point. I found two that I highly recommend for different reasons:
- Embedded Linux: Hardware, Software and Interfacing
Craig Hollabaugh, 2002. ISBN 0-672-32226-9 - Linux Device Drivers, 2nd Edition
Alessandro Rubini & Jonathan Corbet, 2001. ISBN 0-596-00008-1
Let me tell you a little bit about these books:
Embedded Linux is a great hands on book. What I really like about it is that the author leads you through the design and development of a complete system (not just a single computer) that spans three different processor architectures – PowerPC, ARM and x86. This is great because the different architectures expose different parts of the linux kernel.
I will not bore you with a complete review of the book as a good review is already written here. Craig Hollabaugh also has a nice website that serves as a companion to his book.
It is a really great book. The author covers topics like portability as he shows how to write the same device driver for the three different target architectures. It really helps to see a problem solved in three different ways. Highly recommended.
Also Craig was very responsive to questions I emailed him as I went through booting my board for the first time. Very nice.
Linux Device Drivers, 2nd Edition is also a really great book. First off it is released under the GNU Free Documentation License and is freey available online from O'Reilly. I recommend buying the book as paper is so much easier to read.
It covers writing device drivers for standard 2.4 kernels targeted for the Intel x86 architecture. It does not cover embedded GNU/linux, however it does cover the kernel and device drivers in great detail. A very valuable resource.
Information's pretty thin stuff unless mixed with experience.
– Clarence Day
After reading most of Craig's book I was ready to purchase a board and get busy. To make my life a little easier I decied to stick to the same processor architecture I used at my day job, the Motorola PowerPC.
I purchased a RPX_Lite from EmbeddedPlanet, which has a blindingly fast (not) 80MHz 823e PowerPC processor with 16MB of RAM. It's pretty cute, a 3 inch square that includes ethernet, USB, serial and a PCMCIA slot. Just right for experimenting. This board is similar to the RPX_Classic that Craig uses in his book.
Next I needed a GNU/linux distrubtion to use with my board. This is not as easy as it sounds and Craig's book leads the way. The first thing you need is a cross-compiler for your board, so that you can build all the executables that your board will need. This includes not only the kernel, but all the supporting GNU software (cp, rm, ls, etc.) .
It is a lot of work to build a linux distribution from source. It is much easier to use a pre-built distribution for your board. This is what companies like Lineo, MontaVista and LynuxWorks are selling. See here for a good overview of embedded linux distributions.
Also I hit my first bump in the road when I realized my processor did not have a Floating Point Unit (FPU). Most available GNU/linux distributions for the PowerPC assumed you were running GNU/linux on a MacIntosh, whose processor do have an FPU. Those binaries would not work for me.
It looked like I was going to build everything myself...
Experience is not what happens to you; it is what you do with what happens to you.
– Aldous Huxley
Until I found the perfect distro for my needs called the "Embedded Linux Development Kit" (ELDK) from DENX Software Engineering in Germany.
The ELDK comes with pre-built binaries for a range of PowerPC processors including my version. This includes the cross-compiling GCC toolchain and hundreds of pre-compiled programs and utilities that come with GNU/linux.
Oh and the ELDK is GPL and available for free download. The price is right.
I must give special thanks to Wolfgang Denk (owner of DENX), who helped me numerous times via email and patiently answered my neophyte questions. Companies selling a commercial product should envy the customer support I recieved from Wolfgang – above and beyond my expectations. Truly awesome.
Now with the ELDK in hand I was quickly able to configure and build my kernel. I was getting tantalizingly close to my goal of "booting the board". All I wanted to see was the first login prompt from my little board coming to me over a serial connection. I desperately wanted to see the results of "uname -a" as proof that the board was up and running GNU/linux.
Since the board has a limited amount of memory I decided to mount the root filesystem over NFS – this allowed me to have access to all the programs of the ELDK (100s of megabytes). I had to setup an NFS server and configure the kernel to mount root via NFS.
Once that was completed the moment of truth was at hand:
Linux/PPC load: Uncompressing Linux...done. Now booting the kernel Linux version 2.4.4 (curt@hongkong) (gcc version 2.95.4 20010319 (prerelease/franzo/20011204)) #10 Sat Jan 11 14:40:07 PST 2003 Calibrating delay loop... 63.89 BogoMIPS Memory: 14752k available (904k kernel code, 364k data, 48k init, 0k highmem) eth0: CPM ENET Version 0.2 on SCC2, 00:10:ec:00:32:d6 VFS: Mounted root (nfs filesystem) readonly. INIT: version 2.78 booting Remounting root filesystem in read-write mode: Mounting proc filesystem: [ OK ] Configuring kernel parameters: [ OK ] Activating swap partitions: [ OK ] Checking filesystems [ OK ] Mounting local filesystems: [ OK ] Enabling swap space: [ OK ] INIT: Entering runlevel: 3 hoho login: root Last login: Thu Jan 1 00:00:36 on console hoho:~# uname -a Linux hoho 2.4.4 #10 Sat Jan 11 14:40:07 PST 2003 ppc unknown hoho:~# ping www.ucdavis.edu PING www.ucdavis.edu (169.237.104.199) from 10.0.0.9 : 56(84) bytes of data. 64 bytes from 169.237.104.199: icmp_seq=0 ttl=51 time=79.617 msec 64 bytes from 169.237.104.199: icmp_seq=1 ttl=51 time=69.767 msec hoho:~# cat /proc/cpuinfo processor : 0 cpu : 8xx clock : 64MHz bus clock : 32MHz revision : 0.0 bogomips : 64.39 zero pages : total: 0 (0Kb) current: 0 (0Kb) hits: 0/0 (0%)
Sweet! It was working. My first project was complete – the board booted.
Life must be understood backwards. But it must be lived forward.
– Soren Kierkegaard
Now that the board booted I need a new goal, something embedded. Fun fun fun.
Cheers, Curt
Recommended Books
- Embedded Linux: Hardware, Software and Interfacing
Craig Hollabaugh, 2002. ISBN 0-672-32226-9 - Linux Device Drivers, 2nd Edition
Alessandro Rubini & Jonathan Corbet, 2001. ISBN 0-596-00008-1
Recommended URLs
January 04, 2003
Back To School – Part II
The first six months were rough -- so much new information. While most of the computing terms were familar so many electronics terms left me in the dark. I had heard of programming concepts like spin-lock, semaphore, ISR, inter-locked linked list ... but I had never used them, therefore I really did not know them.
SRAM, DRAM, SDRAM -- Do you know the difference? I didn't and everybody else did. Processor data cache lines, memory subsystems (what is a single beat read anyway?), LVDS drivers, pull-up resistor, bias voltage, tri-state logic, BSP, VHDL. It was like hearing a foreign language in a bad dream -- you recognized the words, but you have no idea what they mean and everyone around you nods knowingly.
On top of all the electronics terms were all the physics, mathematics and statistics terms. What's a Poisson distribution? A chi-square? Probalistic histogram? Hmmm, college calculus class was over 10 years ago ....
So I began re-educating myself. I found a couple of handy technical dictionaries on the web:
How did that last one sneak in there? It's a good one to have anyway.
These links helped hide some of my ignorance, but honestly when the conversation was 30,000 ft over my head I just asked questions. The answers were always forthcoming -- no funny looks.
Another area were I thought I knew something, but realized quickly that I didn't, is C compilers. I had programmed a lot and written dozens of C programs, but I never gave much thought to the compiler (or the linker for that matter). One piece of advice for life in the embedded world:
Know Thy Compiler and Toolchain
Never before had the destinctin between operating system and architecture mattered. Now I understood that you can have differenet operating systems on the same hardware architecture (e.g. linux and Windoze running on i386 hardware) and you can have the same operating system running on different architectures (like linux running on i386 and on PowerPC).
In our project the data acquisition system software runs the VxWorks operating system for the PowerPC architecture (our computers are 3U CompactPCI single board computers based on the Motorola PowerPC microprocessor). I in no way endorse VxWorks as a hard real time operating system (RTOS). The company behind VxWorks, WindRiver, is the Microsoft of the embedded OS world. I will not say another word about VxWorks -- it is the hand I was delt.
If you are just starting out look seriously into running linux on your embedded system. If you have "hard real time" requirements consider using the real time extentions to linux. Linux on the small devices is here to stay.
Motorola, on the other hand, is great. They give away (as in free) tons of documentation about their processors. They will even mail you printed manuals for their processors, for free. Very useful. Try their document search.
In our work we cross compile the programs for the embedded system. Briefly cross compiling is when you run the compile/link stage on one OS (e.g. Solaris), but the resulting binary is for a different OS/Architecture (VxWorks on PowerPC). For a more detailed treatment of cross-compiling for the PowerPC using the GNU Compiler Collection see here.
Also read up on GCC in general, particularly the sections about GNU extentions to C and C++. Some of these are quite useful.
In our testing we try to exercise our code as much as possible before downloading it to the embedded processors. To this end we cross-compile our code for three different platforms:
- VxWorks on PowerPC -- the real system
- Solaris on UltraSPARC -- development system
- Linux on i386 -- development system
While we cannot debug everything on the development system (some tests must run on the real embedded system), we can flush out many compile time bugs by pushing our code through three seperate compile/link stages. Sometimes different compilers for different architectures catch different bugs and warnings.
I could write an entire article about the compile/link process we use -- but I won't do that here, unless people are interested. I find it interesting.
The first part of my re-education is well underway. I was getting a good handle on my tools. It is very important that a craftsman understand the benefits and limitations of his tools.
Now I needed a project to work on ...
January 02, 2003
Back To School – Part I
When last we met back in January 2002 (almost one year a go), the accidental programmer decided it was time to get serious and really learn the trade of computer programming. To that end I wanted to program embedded systems where a thorough knowledge of the underlying hardware is absolutely necessary.
I took a 20% pay cut to work at SLAC, a DOE National Laboratory operated by Stanford University. What I recieved in exchange for the 20% has been absolutely invaluable -- I learned how to think, I learned all about how the computer/interfacing world operates. I learned and understood the importance of system engineering. More on that later.
The project I am working on is called GLAST, a large gamma-ray observatory satelite that will measure gamma-ray sources throughout our universe as the satelite orbits the Earth. I work on the embedded system software that runs the data acquisition system for the detector.
I'm right where the rubber mets the road -- where an electrical engineer's piece of super custom hardware meets a Motorola PowerPC single board computer. I write device drivers for one-of-a-kind hardware devices. Oh yah, and the whole system will be stuffed on the tip of an explosive rocket and launched into orbit.
Deliciously geeky.
Up front though I want to give props to the physicists, researchers and engineers who toil away building the best possible machines for measuring and understanding the universe in which we live -- they dedicated their lives to scientific pursuits.
SLAC is a very interesting place to work -- it is one of the few DOE labs that does not do defense research, i.e. it is not a bomb factory. Also all of the projects and results are open for public scrutiny -- no so called black projects that occupy so many defense and aerospace companies. The research is funded with our tax dollars and is open to you, me and everyone else.
That is refreshing.
It is also refreshing to work with people who have decided to spend their entire life at SLAC for that very reason. These are sharp people who believe in what they are doing, but with humility, graciousness and a sense of how things could be much, much, darker. More about the dark side in a future post.
So the employment was a fair trade I'd say. They picked up a seasoned computer programmer who also had a physics background. It's just my computer programming background was writing applications -- it was now time to dive into and beneath the operating system and write device drivers. This is the area I wanted to explore, the embedded system. While I knew nothing about the low level world or embedded systems it seems fascinating and valuable as computers become more pervasive as they shrink smaller and smaller.
For their part they signed up to teach me embedded system programming. Or so I thought. So far it has turned out to be much, much more. They taught me about system engineering. How to architect a complete, complex system of custom electronics and software. How to think way, way outside of the box when designing (and debugging) an ambitious compu-electronic data acquisition system.
I feel the pain of the hiring managers at SLAC -- everything is a trade off. They have ridiculously small budgets, but they need really experienced engineers to build the devices they design. However all the really good, experienced engineers all work in industry where the pay is about 20% higher. So they make do with the people they can afford, knowing they will have to do a bit of teaching to make it all work.
I'm sure they don't realize it, but I have been absorbing as much knowledge as possible at every opportunity.
It is not always fun, however.
I came from running my own company, where I was in control and I decided what needed doing when. This was a change -- someone else was in control for the direction. Even more than that I had no experience in the field at all. It's quite an unsettling feeling after being in charge for so long. I did not know enough about embedded systems to know if their ideas were right or wrong.
I felt small.
And dumb.
To be continued ...
|
This work is licensed under a Creative Commons License. Copyright © 2000-2006 by Curt Brune |