December 19, 2001

Accidental Programming

In this month's article I'll examine "The Lost Art of Computer Programming" giving credit to the original author of "The Art of Computer Programming". Along the way you'll find a status report on my continuing journey towards heightened awareness and a reminder of the dirty tricks of history.

In these heady daze(sic) of .NET, mono, SOAP, java, xml, object oriented software design and "network as computer" metaphors we (professional programmers) seem to have lost sight of the "computer as machine" truism. Despite all the power of abstractions a fundamental understanding of how the machine really works remains invaluable.

Like many modern engineers today I began life with a computer in the house -- OK technically not since birth, but since the time I was 8 years old and Pops brought home that Apple ][+ things just haven't been the same (with a little math YOU can date me). By no means do I consider myself a great programmer, or even a good programmer -- in polite company I stifle a cough when saying I'm a "Software Engineer", quietly knowing I'm a little more than a quick hack. I know people who can code circles around me in various problem domains, whose understanding of the underpinnings of the problem far exceed my own.

I am an accidental programmer.

That first Apple that my father bought did little to teach me the "Art of Computer Programming". What it did teach me is that playing games like Ultima, The Bard's Tale and Wizardy was wickedly fun -- not a very practical thing to know.

As a teenager I spelunked into 6502 programming enough to implement a bubble sort algorithm simply because I could, not out of necessity. I also played with "Apple Pascal" -- it's compile and link stages fascinated the BASIC programmer I was. I thought it cute I had to shuffle four 5.25" floppy disks in and out of the drive to compile and link a simple "Hello World" application. The LOGO-like "TurleGraphics" was also very cool, with simple statements like
move(30);
turn(120);
move(30);
turn(120);
move(30);

used to make isoscelese triangles.

All things being equal computers rated about a 6.5 on a scale of one to ten, somewhat behind playing basketball during the day and drinking cold beer during cool summer nights in the Mojave Desert where I grew up. After all you couldn't get a computer drunk under a stary desert sky as you unhooked her bra all the while explaining that the twinkling starlight was billions of years old. (BTW the desert is an excellent place to be a teenager, where one can get into all manner of red neck rural mischief -- guns, dirt bikes, beer, explosives ... all the stories you've ever heard are true.)

Computers did rank above going to church on Sunday's and watching TV strangely enough. Computers were fun, but my sport-o, jock-o, red neck friends could care less -- the sun is shining (the sun is always shining) outside in the desert.

Being mathematically inclined and curious about the universe I enrolled as a Physics major at a University of California. Physics seemed like the ideal major for one whom mathematics came easily, but for whom actually **doing** math was tedious. The beauty of partial differential equations, wave mechanics and complex numbers intrigued me -- using these concepts to model and understand the sights, sounds and experiences of the physical world around me was extremely compelling.

The further I went in physics the more intriguing it became, going from simple torque equations that explained why I could ride my bicycle with no hands to the more esoteric quantum mechanics, general relativity, high energy particle physics, astrophysics and cosmology. Very heady concepts for a young man set about understanding the universe and securing his place in it.

But I digress, this article is about programming.

As a physics major I received my first mainframe computer account unlocking a Pandora's Box that has not closed since. Gopher, FTP, email, who-is -- all these people connected to machines, which in turn are connected to more machines with people connected to them.

Fascinating.

People to machines. Machines to people.

Fascinating, but not physics.

Physics was all about FORTRAN, long doubles, complex variables, crunch, crunch, crunch. While in graduate school in the early 1990s two realizations would lead me to discover that Physics was not for me, sort of.

First, while questioning the universe I thought physics held some answers to the unexplainable. So I attended a seminar on "Early Universe Cosmology" given by visiting theoretical "bad-asses" from around the globe. What a disappointment to find these "experts" arguing over the fundamentals of their science. In truth they knew just as little as I did, perhaps less.

Second, while writing pattern recognition software for a high energy physics detector I decided that FORTRAN was crap and would instead write my filter routines in C and figure out how to link to the monolithic FORTRAN libraries (we were running Ultrix at the time). A fellow grad student questioned this approach to which I dryly replied "If physics does not pan out I hope to get a real job some day." (BTW Graduate school is a great place to see what you do not want to become -- either a grumpy 7th year student, or worse an even grumpier post-doc making slightly less than the campus grounds keeper on four hours of sleep per night.)

It was also clear I was no Einstein, that I was not destined to be a theoretical physicist ( my hat's off to those that are!) . At 23 years old I was hopelessly washed up as a theoretical physicist and the thought of spending all my time indoors in a basement laboratory sounded, well ... horrible.

That the summer the kiss of death came: I discovered this little program called mosaic. The rest, as they say, is history. I dropped out of grad school, packed my bags and landed a job in the heart of silicon valley -- Menlo Park, CA.

__Close_but_no_cigar__

I landed in Silicon Valley, but I did not land on the Internet. When the dust cleared I found myself at a digital map company, writing software that integrated text tour guide information with digital map data. All of this was for GPS car navigation systems and had little to do with the Internet (though a couple of people left said digital map company to go and found MapQuest about the time I joined).

Here I really cut my teeth with UNIX on SunOS 4.1 . Here I had to keep up with people who were "real" computer scientists, people with Ph.D's in mathematics and topology (calculating routes on a topologically correct road network map is non-trivial). These people were good to me and taught me many programming principles, most of which were gleaned by trying to grok their source code.

I also learned to leverage the power of an OS with virtual memory -- I really could allocate a gigabytes worth of unsigned ints, even if my Sparc5 workstation only had 128MB of RAM. Really, how cool is that? What a beautiful abstraction! That discovery also marked the beginning of my decline -- I started to become a really lazy programmer.

From there I moved to C/C++ Win32 programming and sloppy GUI design. I copied and pasted, cursed, used the debugger, swore and generated the most horrible looking makefiles. At one point I even generated afx DLLs that worked only during certain phases of the moon. Nasty bad habits.

Then things took a turn for the worse....

I was assigned to a Visual Basic project -- now I really could sit back and forget all about how the underlying OS machinery worked (except when it came time to write the InstallShield, which unfailing fucked any system we installed to -- uninstall, my butt). VB made life so simple, so easy, so ... dangerous.

About this time the Internet madness was really picking up steam, when it seemed like any hack (me) with confidence could get funding for even the most hair brained idea. It was time to found a start-up Internet company. What could be more en vogue?

Luckily for me and my partners funding a bad idea is a two-way street -- first it takes a hare brained idea and then it takes someone zany enough to invest in it. Coupled together you really understand the phrase "burn rate". The story about the formation, function and dis-function of that start-up I will leave for a future article. Remember I promised this story, I mean article, would be about programming.

Our start-up was based on a well known GNU operating system including Apache, PERL and postgresql. Perl is great for implementing whatever you want in the shortest amount of time possible ... a hacker's language.

Among other things our company was into the nuevo chic business of wireless e-commerce. We were right in the middle of everything that was happening: web meets phone meets bank account. Who has the most efficient method of moving money from one wallet into one's own?

We quickly adopted SOAP before most application server vendor's supported it -- it seemed like the perfect way to get disparate machines talking the same language quickly and easily. Our colleagues on the other end of the pipe balked at first, but ultimately gave into our design more out of fear of being left behind in the information arms race than anything else.

__Kaboom__

Then the bubble burst... wireless wasn't hot, e-commerce wasn't hot, nothing was hot. Our fun little start-up whirlwind settled down to little more than a gentle breeze on a Sunday afternoon.

It was about this time I started going to a little Lutheran church around the corner from my apartment -- I was raised Lutheran and I could draw some comfort from ceremonies repeated throughout my childhood. Something stable.

The first Sunday my fianc� and I attended Church we stood around after the service talking with various people. One tall older gentleman came up, shook our hands and asked if we were new to the area. Did we work at Stanford?

"No," I replied. "I work at a small Internet startup," I said sheepishly while stifling a cough.

"Oh, well don't tell me anything I need to sign an NDA for," he said.

"Don't worry about that," I laughed. "All of our stuff is open source and public domain anyway."

The old man cocked an eyebrow at me and said, "I've written a few open source programs in my day. Perhaps I didn't introduce myself. My name is Don ... Don Knuth."

Holy shit. Exposed. In church no less. In front of a true computer scientist, if not THE defining computer scientist.

"I have your books on my bookshelf," I gushed like a school girl.

"A lot of people have my books on their bookshelves. That's the problem, they need to read them," he said.

I don't remember much of what happened next. Later my fianc� told me I looked sort of pitiful with my mouth hanging open. She had never seen my fast talking self stopped dead in my tracks. It was terribly humbling. Terribly.

It was then I decided that my next gig would be to uncover the lost art of programming. Time to quit screwing around and actually learn the trade.

to be continued...

The next installment

"Crossover -- WSDL to Gamma Ray Telescopes"

email me and I'll send you next month's installment -- curt@acm.org

Posted by curt at December 19, 2001 01:17 PM | TrackBack
Comments

Reader response to "Lost Programming"[1]

I did not expect hundreds of people to write me back asking for the
next installment -- quite a shock. Beyond that responses took my breath away
for their insight, other replies made me laugh out loud and still more
made me realize I had struck a chord with many of you.

Well I decided I could not keep all these gems to myself so I am
listing the best ones below (anonymously of course). The topics raised
in the article and the replies below is fertile ground for discussion
-- as such I am also posting this message on Usenet[2]
so that we can all respond to one another.

Finally I will not request that you email me for the next installment
-- I recieved a lot of email, plus some of your are rightly concerned
about handing out your email address to strange accidental programmers.
Every month drop you can visit [3] and see if a new article
is waiting. I promise to do my best to keep new material up on the
site every month.

However, if you do want email notification of when new articles are
ready you can email me (curt@acm.org) and indicate it's OK for me to
spam you with articles (just kidding). It's not an auto-responding email-bot
(it's me) so please write something interesting :) .

Cheers,
Curt

References

1. http://www.cucy.net/lp/120901.txt
2. http://groups.google.com/groups?selm=b07d5cf5.0112111314.4f01fb99%40posting.google.com,
http://groups.google.com/groups?group=comp.software-eng
3. http://www.cucy.net/lp/

----------- Response to "Lost Programming" ------------

[This first one is long, but boy is it worth it -- we need to clone
this gentleman]

I come from a rather different background, being also of
an earlier generation. I was lucky enough to be taught
about computers and computer programming at an early age
by some of the creators of the first practical digital
computers. We programmed on the bare metal, and the
translation of the program into the necessary sequences
of bits was a mindless exercise in frustration. What was
a miraculous journey was designing the program as a pure
theoretical exercise, discussing the algorithms to death,
refining the whole abstract structure again and again, to
produce the most effective programmed solution.

Programming was all about designing the abstract program,
and studying it in theory to ensure that it would really
do what was needed, as effectively as possible, and without
side effects. Actually implementing the program was a very
tedious exercise, where all errors were entirely as a result
of miscoding. By the time we came to implement, we knew the
program would work - we knew it backwards, forwards, sideways,
and inside out - we just had to translate the design into the
instruction set of the computer.

I shared some of your FORTRAN experiences studying engineering
in various forms at university, although I had a very hard
time matching the university use of FORTRAN as a tool to the
elegance of programming as I'd had the good fortune to learn
it at school in earlier years.

I ended up entirely by accident in the computer industry,
soon gravitated to the software side, and have been a software
worker ever since. There is no such discipline as software
engineering: if there were, there would be certifications,
regulations, regulators, inspections, and law suits. There
would also be standards with respect to such concepts as
software busses, method attachment and execution, and other
such unlikely practicalities of engineering with global
reach.

One of my roles within my current organisation is as mentor
to the engineering group. Spread as it is across ten time
zones, and speaking a dozen human languages on a daily basis,
this is quite a challenge. Your experience describes that of
a depressingly large percentage of our engineering staff, and
overall they are an excellent bunch. However, they are lazy
without knowing that they are lazy because they use tools
that hide reality from them to the point where it becomes
impossible to touch reality directly, and they lack any
education in the electro-mechanical realities (and their
associated underlying theories) which actually are their
environment.

I look at a mono-culture labelled Microsoft Windows and weep
for the ignorance that it engenders, not to mention the sad
state of the tool itself. How can one have a thread-safe
library that isn't, or a mathematical library written by
students that doesn't handle limit cases?

Our teams are required to achieve warning-free compilation
for their code on five separate compilers on five separate
platforms (hardware plus operating system). Even such a
simple step as this is beyond most of our less than middle-
aged recruits, and they have to be educated in what to them
is a startling range of techniques for writing good code.

Sorry, I'm rambling, but it is interesting to have a first
hand account from someone taking the time to look inwards.
Now, how do we persuade the educators to start producing
courses in the reality of what would be software engineering
if there was such a thing?

---------------------------------------------------------------------------

I can identify with it as I am nearly dangerous in perl myself.

---------------------------------------------------------------------------

I'm *so* glad that someone recognizes that there is in fact a
difference between those who understand and are passionate about the
machine (to which programming comes more naturally) and those who
treat programming like a trade that anyone can pick up (as we saw in
recent years). It is a lost art, and it's good that you're bringing
it to people's attention before it is forgotten that it ever existed.

---------------------------------------------------------------------------

The naked truth of ASCII

---------------------------------------------------------------------------

I'm an English Ph.D. who has also become an accidental programmer; in
fact, my whole career consists of the the struggle to be legit -- a
real, hardcore software developer -- in a sea of mathematicians and CS
profs.

---------------------------------------------------------------------------

If you changed some of the names and places in your piece, you could
have been speaking about me. The irony is rich:

Teenager Years: Apple II+, Wizardry, Pascal, Math, & Beer (lots of
it)...

Early College: Physics, Mainframes, more beer...

Graduate School: More Physics, High Energy Neutrino Experiments,
Cashed in Fortran for C, then C++, and finally Java (you should have
seen their eyeballs when the committee realize we performed event
reconstruction in java and not fortran). Oh, and more beer of
course...

Post Doctoral: Never happened. Got into real estate and then a
startup.

Today: Still kicking. Still beer drinking. Startup is alive and
well...

Looking forward to your next installment on the Lost Art of
Programming, which as far as I am able to really ascertain is only
'lost' on the poor souls actually trained in programming (spoken like
a true physicist...)

---------------------------------------------------------------------------

I keep thinking I'm going to settle down and study and become a real
programmer some day, but meanwhile I muddle along with a bit of perl,
a bit of C, and a lot of debugging....

---------------------------------------------------------------------------

From my younger days: Cobol chapter heading: Resist the urge to code!
Best advice ever given to a programmer.

---------------------------------------------------------------------------

I know of the feeling you hint at. Just about everyone you meet mistakes
you for a real software engineer/hacker(traditional definition)/expert/god/
guru.... only you know that in actuality whom the real gods are, and you
aren't on that list.

---------------------------------------------------------------------------

frightenningly enough, you sound very much like me. I also went into
physics only to become very disenchanted by it and resort to
"professional hacking" - i.e., not a real programmer, but I play one
on TV ;-)

---------------------------------------------------------------------------

I identify with this narrative so strongly it's scaring the hell out
of me.

---------------------------------------------------------------------------

They describe our subculture in a manner more approachable than, say,
The Hacker's Dictionary, at which we all laugh uncontrollably while
bystanders react like we are diseased and coughing on them.

---------------------------------------------------------------------------

you know dude, you're going to be an old man and your going to tell
your grandkids, "did I ever tell you the story of how I met Donald
Knuth?", and they're going to say "Only a thousand times already,
grandpa curt", and the little ones will say "Who the hell is Donald
Knuth, anyway, and why do I care?" and the ones that were baby-sat by
uncle **** won't even look up from their mega-gameboy consoles to
answer you...

---------------------------------------------------------------------------

I was there with the apple (though mine was a //e)

I was there with Wizardry etc... (did you ever play Swashbuckler?.

I was there with pascal -- much cooler than basic.

Then I went into English Lit, at least as much wasted time.

And now I'm learning to program.

---------------------------------------------------------------------------

This might be hopeless nostalgia but do you think there was something
about the simple graphics, like your body becoming attuned to a
strobe. The higher graphic format today brings too much noise and data
all at once, those cga graphics, or perhaps there novelty, had a
really mesmeric quality to them. I remember feeling for the time to
strike in Swashbuckler, I dont think I feel that quiet ease with games
today.

---------------------------------------------------------------------------

I also have a degree in Physics. I also am an accidental programmer
who started programming in basic (ZX Spectrum at the time) and moved
to Pascal and finally to C and whatever. Right now my main source of
income is typesetting technical books using TeX (on any machine, win,
macos, macosx, but most of the time on a linux).

---------------------------------------------------------------------------

I just read the first installment of your article. It was a very
interesting start. I would love to finish it, but fear in a month
from now I will not remember where to find it. (Perl can make you
forgetful as well as a bad programmer.)

---------------------------------------------------------------------------

Great article -- loved the ASCII format

---------------------------------------------------------------------------

As a side note, I'm making an effort of my own to steer toward the
core of what we do as programmers, rather than be stuck in
"productized" development. So, I'm working through Lisp hacking for
fun, and trying to avoid mere customizing on top of apps which are
rotten at the core. It's very much against the flow of what's going on
in this telco-subsidiary Internet startup, so it's relegated to lunch
hours and at home. Still, waaaay more satisfying than superficially
coding to APIs which the black-box app itself doesn't do
right. So... if you have any suggestions on how to find that great
employer that does interesting, complex things instead of spewing
buzzwords, I'm all ears!

---------------------------------------------------------------------------

I learned FORTRAN on an IBM 1620 in 1969. My higher education started
out in pursuit of a physics degree, but was sidetracked by an SDS
Sigma-7 (it was a virtual memory machine, designed right into the
hardware). I appreciate the usefulness of mathematics, but hate using
it. I also have Knuth's books on the shelf and would most likely
quake in my shoes if I encountered him. When a professor walked up to
me one day to extoll the virtues of Java, html, and how the network
WAS the machine, I quietly pointed out to him that the network itself
never produced a single programmable CPU cycle for me, but was simply
a glorified bus (I dislike regurgitated hyperbole). I've learned
several languages in my life. I still have a nostalgic fondness for
Snobol, one of the more useless languages I ever learned.

---------------------------------------------------------------------------

Loved the article. I'm in the same boat. I've had the first volume
for something like 18 years now, but I can't say I've actually sat
down and *read* it. I've read maybe a quarter of Searching and
Sorting. I don't think I've ever had a serious look at Seminumerical
Algorithms at all.

I think it's the assembler code that puts me off. Unfortunately, I
don't enjoy assembler. I've never been in a situation where I was
forced to use assembler seriously and I see no reason to force myself
to do so, although maybe I'll feel a little embarrassed now and give
it a go :-).

---------------------------------------------------------------------------

At twenty-eight I came to computers through unix after an attempt at
art school. Art school was the fulfillment of a life long dream. Now
I found something I'd rather do with my life.

---------------------------------------------------------------------------

I'm lucky enough to get paid as a consultant, and in the process I get
to learn(=play) with some cool (but practical) stuff. But, more
importantly, I get to consult with people I consider far brighter than
myself. Like the Red Queen says, you need to run as fast as you can
just to stay where you are ... you can't learn it all, but I still
want to learn enough that I don't feel so small!

---------------------------------------------------------------------------
I find myself in the situation pretty much like yours. I work in a
.com, but I am seriously considering returning to the 'lost art of
programming' because I feel that every day of working with VB, SOAP,
XML and other similar stuff (especially VB!) takes me further from
what I once loved in programming.

---------------------------------------------------------------------------

I feel identified with you in many ways, although I still haven't been
to college, much less trying to major in physics ;) But I had an Apple
too...and I learnt basic, logo etc etc... man I almost cried when I
was able to run Aztec in the apple 2 emulator a few years ago. Things
have changed a lot, and I find myself wondering where to go or what to
do next to keep things interesting... we're in exciting times, no
doubt about that, but I think I miss some of the old days too...maybe
we're just getting older hehe ;)

---------------------------------------------------------------------------


Posted by: Anonymous on January 2, 2003 11:08 AM

It took me a long time to find a job where I could do some programming, although I have been interested in it since high school and I'm pushing 60. I studied symbolic logic on my own in high school. Learned a little fortran in college as a physic major. Wrote an undergraduate thesis about computer music as a music major. Learned to progam computer controlled machinery as a toolmaker's apprentice. Did you know you can program melodies for a stepping motor?. I machined volutes in real time using an Apple II and created a 2D program for creating tapes for numerical controlled machines. My first home computer was a Radio Shack Color Computer. I sold a program , done in 6809 assembler, called Music Box which generated 4 part polyphonic music. Made almost enough money to pay for the COCO. Finally I landed a job as a project engineer where I do database programming, PLC programming and a little electronic design. Ain't it fun to get paid for doing something you like to do?

Posted by: Don on May 13, 2003 08:59 AM

excellent topic

Posted by: Zero on October 21, 2003 10:51 AM

I'm no LOGO guru, but wouldn't that triangle be equilateral?
I mean yeah, it's isosceles too, but come on.

Posted by: indrax on June 13, 2006 05:41 PM

This is a great topic and a great site.

Posted by: Sam on June 14, 2006 05:46 AM

great topic! really - I also identify a bit although I'm kind of new. I started with a Z80 programming Basic(not much of it - just few algorithms), then went to Pascal, a bit of Asm, played with VB, had 2yr in web programming (PHP, Perl, Mysql) and now I'm going to C..just simple, plain C. I really like this simple programming..is almost like when I was making GUIs in Pascal for Dos..:)
Hey..where's the next episode?

Posted by: cristian on June 15, 2006 12:24 AM
Post a comment