June 1, 2006Talking Dirty with GDB and SSH Tunneling
Ever debugged a program remotely and felt like telling your computer where to go and how to get there? Hopelessly adding calls to Fear not! Help is on the way. Read on to learn how to use gdbserver and ssh tunnelling to debug remote processes.
 Click to enlarge In embedded systems development making do with less is the name of the game – less CPU power, less physical RAM and less peristant storage (if any!) to name a few. Debugging a misbehaving process in this environment can be challenging, but a little ingenuity coupled with plenty of free software eases the problem. In this article I will explain how to use GDB for debugging remote processes running on embedded systems from our desktop workstation. For the purposes of this article the embedded system is a RPX_Lite from EmbeddedPlanet, which I discussed in a previous article. This board has a blindingly fast (not) 80MHz 823e PowerPC processor with 16MB of RAM. It's pretty cute, a 3 inch square that includes ethernet, USB, serial and a PCMCIA slot. Just right for experimenting.
Major DifferencesWhile many differences exist between desktops and embedded systems running GNU/Linux, the biggest difference is size and power. Embedded systems have very specific design goals limiting the overall power consumption and physical dimensions. This leads to the use of low power CPUs, limited physical RAM and little or no persistant storage devices. The next major difference is the CPU architecture – embedded systems often use low power CPUs that are not x86 based. To compile programs from your x86 based desktop you need "cross-compilation tools", which run on your local desktop, but generate excutable code for the target architecture. In this article I will be cross-compiling for the PowerPC architecture. The last major difference is perspective. You will be debugging a process that is executing on a remote CPU not your local workstation. This requires a slightly different mindset then traditional debugging.
ELF and Binutil BackgroundBefore getting to the nuts and bolts here's a quick review about how executable code and debugging information is stored in an ELF binary. Most modern *NIX systems use the the ELF format for executables and shared libraries.
On a GNU/Linux system a family of utilities called binutils exists for
examining and manipulating ELF objects. In a cross-compiling
development environment the usual tool names like In this article I'm targeting the PowerPC 823e and the tool prefix is "ppc_8xx-". I am using the wonderful Embedded Linux Development Kit (ELDK), which has complete toolchains for the PowerPC. Let's play around with some of the binutils using a simple "Hello World" application: #include
First let's compile the program with debugging symbols using the ppc_8xx-gcc -g -o hello hello.c
Note I used the cross compiler, On my system the resulting binary size is 20632 bytes.
To see what symbols the binary contains use the
coz:~/articles/rgdb$ ppc_8xx-nm hello
10010868 D _DYNAMIC
10010948 T _GLOBAL_OFFSET_TABLE_
[... stuff deleted]
1001085c W data_start
10000408 t frame_dummy
1000048c T main
100109d4 b object.2
10010860 d p.0
U printf@@GLIBC_2.0
The interesting lines for us are near the bottom:
1000048c T main
U printf@@GLIBC_2.0
The first line shows the address of the main() function is 1000048c. "T" means the text section, which is an old term for the section where the code resides. The next line shows that printf() is an unresolved symbol and will be loaded from a shared library at run time. Interesting. The ELF format defines sections where various information about the executable is stored. The most interesting sections are:
In addition several other sections also are present,
including sections containing the debugging information. To see all
the sections and their sizes use the
coz:~/articles/rgdb$ ppc_8xx-objdump -h hello
hello: file format elf32-powerpc
Sections:
Idx Name Size VMA LMA File off Algn
0 .interp 0000000d 10000114 10000114 00000114 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
1 .note.ABI-tag 00000020 10000124 10000124 00000124 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .hash 00000030 10000144 10000144 00000144 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
3 .dynsym 00000070 10000174 10000174 00000174 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
4 .dynstr 0000007a 100001e4 100001e4 000001e4 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
5 .gnu.version 0000000e 1000025e 1000025e 0000025e 2**1
CONTENTS, ALLOC, LOAD, READONLY, DATA
6 .gnu.version_r 00000020 1000026c 1000026c 0000026c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
7 .rela.dyn 0000000c 1000028c 1000028c 0000028c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
8 .rela.plt 00000030 10000298 10000298 00000298 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
9 .init 00000028 100002c8 100002c8 000002c8 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
10 .text 00000528 100002f0 100002f0 000002f0 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
11 .fini 00000020 10000818 10000818 00000818 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
12 .rodata 00000024 10000838 10000838 00000838 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
13 .sdata2 00000000 1000085c 1000085c 0000085c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
14 .data 00000008 1001085c 1001085c 0000085c 2**2
CONTENTS, ALLOC, LOAD, DATA
15 .eh_frame 00000004 10010864 10010864 00000864 2**2
CONTENTS, ALLOC, LOAD, DATA
16 .dynamic 000000c8 10010868 10010868 00000868 2**2
CONTENTS, ALLOC, LOAD, DATA
17 .ctors 00000008 10010930 10010930 00000930 2**2
CONTENTS, ALLOC, LOAD, DATA
18 .dtors 00000008 10010938 10010938 00000938 2**2
CONTENTS, ALLOC, LOAD, DATA
19 .jcr 00000004 10010940 10010940 00000940 2**2
CONTENTS, ALLOC, LOAD, DATA
20 .got 00000014 10010944 10010944 00000944 2**2
CONTENTS, ALLOC, LOAD, CODE
21 .sdata 00000000 10010958 10010958 00000958 2**2
CONTENTS, ALLOC, LOAD, DATA
22 .sbss 00000000 10010958 10010958 00000958 2**0
ALLOC
23 .plt 00000078 10010958 10010958 00000958 2**2
ALLOC, CODE
24 .bss 0000001c 100109d0 100109d0 00000958 2**2
ALLOC
25 .comment 0000016e 00000000 00000000 00000958 2**0
CONTENTS, READONLY
26 .debug_aranges 00000020 00000000 00000000 00000ac6 2**0
CONTENTS, READONLY, DEBUGGING
27 .debug_pubnames 00000040 00000000 00000000 00000ae6 2**0
CONTENTS, READONLY, DEBUGGING
28 .debug_info 00001f81 00000000 00000000 00000b26 2**0
CONTENTS, READONLY, DEBUGGING
29 .debug_abbrev 00000271 00000000 00000000 00002aa7 2**0
CONTENTS, READONLY, DEBUGGING
30 .debug_line 00000232 00000000 00000000 00002d18 2**0
CONTENTS, READONLY, DEBUGGING
31 .debug_frame 0000003c 00000000 00000000 00002f4c 2**2
CONTENTS, READONLY, DEBUGGING
32 .debug_str 00000041 00000000 00000000 00002f88 2**0
CONTENTS, READONLY, DEBUGGING
WOW! That's a lot of sections and many of them contain debug information. These sections can add quite a bit of size to an executable and none of it is essential to running the program. The information is only useful when trying to debug.
Aside: the
In order to save as much space as possible on an embedded system we
often strip off all the non-essential information from the ELF
sections. The binutil coz:~/articles/rgdb$ ppc_8xx-strip hello Now the size of my executable is 4092 bytes, a reduction of 16540 bytes. That is over an 80% reduction – awesome! But it comes at a price. Without the debug sections debugging will be impossible... Sort of. More on that later.
Remote Debugging With GDBSo now we have a stripped application that we can execute on our embedded system. But what if it is crashing and we want to debug it? A couple of obsticles sit in our way. First, the target system has limited storage so we did not bother to put a cross-compiled version of GDB on it. On my development workstation the GDB executable is 9973975 bytes, nearly 10 megabytes. Clearly that won't leave much room for anything else if I only have 16MB total on the embedded system. What to do? The answer is to use gdbserver, a small footprint server that implements the low level features of GDB. Consider the following diagram – using TCP the feature-rich GDB on my workstation connects to the light-weight gdbserver running on the embedded system. Most of the heavy lifting is done by the GDB on my workstation, while gdbserver deals with the low level interactions.
 Click to enlarge On my system gdbserver is only 59303 bytes, a considerable improvement over the size of the full GDB program.
In the following examples my workstation,
The first step is to attach the gdbserver to a process on the target
system. You can have gdbserver start a program and attach to it
immediately or you can attach to an already running process using the
process ID ( gdbserver host:2222 PROGRAM [ARGS...] gdbserver host:2222 --attach PID In the above examples the gdbserver would listen on port 2222. Using gdbserver to launch our hello world application on the embedded system looks like this: gdbserver host:2222 hello Process hello created; pid = 23125 Listening on port 2222 The prompt does not comeback as the gdbserver is now blocking, waiting for connections on port 2222.
The next step is to start the main GDB program on your workstation and
connect to the gdbserver process. In order for the main GDB debug my
program it needs to examine the "unstripped" version of executable
that contains all of the debugging symbols. The simplest thing to is
to coz:~$ cd ~/articles/rgdb coz:~/articles/rgdb$ ppc_8xx-gdb GNU gdb Yellow Dog Linux (5.2.1-4b_4) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i386-redhat-linux --target=ppc-linux". (gdb)
To "tell" GDB to read the symbols from the unstripped executable use
the GDB (gdb) file hello Reading symbols from /home/curt/articles/rgdb/hello...done. If your application uses shared libraries and most real world applications do, then you also need to tell GDB where to locate these libraries. These need to be the unstripped versions of these libraries so that GDB can tell you more info. Set the GDB "solib-search-search-path" variable so that GDB can find the shared libraries used by your application, like this: (gdb) set solib-search-path [path to libraries] If your application has a lot of shared libraries spread all over your source tree (and most real world ones do) then here's a little trick for the solib-search-path variable. Create one directory and populate it with symlinks to all of your shared libraries. Then you need only specify this one directory when setting the solib-search-path variable. Comes in handy.
Now we are ready to connect to the gdbserver running on the embedded
system. We use the (gdb) target remote 10.0.0.20:2222 Remote debugging using 10.0.0.20:2222 0x10000120 in ?? () On the embedded system console you should see this output: Remote debugging from host 10.0.0.10 Now we are connected and the program being debugged is currently paused. Now would be a good time to set some break points and then continue running the program. Here's an example:
(gdb) b main
Breakpoint 1 at 0x1000048c: file hello.c, line 4.
(gdb) continue
Continuing.
Breakpoint 1, main () at hello.c:4
4 printf("Hello World\n");
And there we are! We are remotely debugging the stripped executable. Pretty cool, huh? I love it! You can now use all your favourite GDB commands and techniques to debug. Personally I like running GDB from within Emacs, but your tastes may vary. You can use any GDB front-end you want for remote debugging. Very nice.
SSH Tunneling and GDB
Suppose you have the network topology shown in the following diagram
and you want to debug a process running on the host
 Click to enlarge
The diagram also depicts a serial console connection from
The problem here is that no routable network path exists from
What to do? I'll be honest – a lot solutions present themselves.
You could configure iptables on
Another method is to use the port forwarding capabilities of your old
friend, ssh. The trick here is
to forward connections on a local
Via the serial console you can login to gdbserver host:2222 hello Process hello created; pid = 23125 Listening on port 2222
Now we need to create an SSH tunnel from ssh -L 4000:wonkel:2222 curt@gw
This opens a listening TCP socket on the local host,
Click to enlarge
Now we can start GDB on (gdb) target remote localhost:4000 Remote debugging using localhost:4000 0x10000120 in ?? ()
This connection is tunneled via Remote debugging from host 192.168.1.20
Note Now you can proceed to debug as before. I hope you have found this intro to remote debugging and ssh tunnelling useful. All comments are welcome. Happy hacking! -Curt
---------------------
Posted by curt at June 1, 2006 8:24 AM | TrackBack Comments
This was a nicely written introduction to remote debugging. Thank you for sharing! Posted by: William on June 13, 2006 10:34 AMPost a comment
|
|
|
Copyright © 2000-2012 by Curt Brune Permission granted to reproduce this work in any medium provide this notice remains intact. |
 This work is licensed under a Creative Commons Attribution 3.0 Unported License. |