Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Jason Machacek]

Hello everyone,

I recently ran into an issue where my embedded application became too
large to run natively under GDB. Unfortunately, my SBC provider doesn't
supply a precompiled version of GDB, so I've been trying to build a
GDB/gdbserver combination myself. My application is a C++ app with
several threads running.

I can successfully build both GDB (native x86 with --target=arm-linux
specified) and gdbserver (ARM-native). However, when I connect to
gdbserver on my target with gdb, it doesn't understand that it's a
multithreaded application--it breaks on each context switch (SIG32) and
can't figure out what instruction it's looking at.

I found a 2005 thread from this mailing list
(http://www.cygwin.com/ml/gdb/2005-11/msg00364.html) and several other
messages from other places that set me on the right track, but I'm still
not quite there.

Here are the commands I used to build GDB (x86):

# cd gdb-6.7.1/
# PATH=$PATH:/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/
./configure --target=arm-linux --disable-sim --disable-tcl
--enable-threads --enable-shared
# make

Here are the commands I used to build gdbserver (ARM):

# cd gdb/gdbserver/
# PATH=$PATH:/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin
CC=/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-gcc
./configure --host=arm-linux --target=arm-linux --enable-threads
--enable-shared
# make

I execute the gdbserver binary on the target:

$ ./gdbserver :12345 app
Process app created; pid = 270
Listening on port 12345


I then connect to the target from the host like so:

# ../gdb-6.7.1/gdb/gdb app
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-linux"...
(gdb) set solib-absolute-prefix
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/
(gdb) file app
Load new symbol table from "/home/jmachacek/juice/juice/src/app"? (y or
n) y
Reading symbols from /home/jmachacek/juice/juice/src/app...done.
(gdb) target remote 10.0.150.55:12345
Remote debugging using 10.0.150.55:12345
0x2aaaba70 in _start () from
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb)


So far so good. "info threads" and "info shared" look okay:

(gdb) info threads
* 1 Thread 270 0x2aaaba70 in _start ()
from
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb) info shared
From To Syms Read Shared Object Library
0x2aaaba70 0x2aabd888 Yes
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb)


I can debug to main() with no problem:

(gdb) b main
Breakpoint 1 at 0x20808: file Main/Main.cpp, line 41.
(gdb) c
Continuing.

Breakpoint 1, main (argc=1, argv=0x7ffffec4) at Main/Main.cpp:41
41 BSP_init(argc, argv);
(gdb) bt
#0 main (argc=1, argv=0x7ffffec4) at Main/Main.cpp:41
(gdb)


At this point, I use 'c' to start running and GDB stops when it receives
SIG32 (which appears to be generated when the CPU switches between
threads):

(gdb) c
Continuing.

Program received signal SIG32, Real-time event 32.
0x2abd49e8 in ?? ()
(gdb) bt
#0 0x2abd49e8 in ?? ()
(gdb) info threads
* 1 Thread 270 0x2abd49e8 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0x2abd49e8 in ?? ()
(gdb) info shared
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libm.so.6"
is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libgcc_s.s
o.1" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libc.so.6"
is not at the expected address (wrong library or version mismatch?)
From To Syms Read Shared Object Library
0x2aaaba70 0x2aabd888 Yes
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
0x2aacc050 0x2aad5e1c No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libpthread.
so.0
0x2ab254c4 0x2ab5d5d0 No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libm.so.6
0x2ab9ab84 0x2aba12d8 No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libgcc_s.so
.1
0x2abc0f80 0x2acbd404 No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libc.so.6
(gdb)


The warnings in "info shared" are due to libc being statically linked in
my application. Can anyone help me figure out why I'm unable to get
gdb/gdbserver to work properly with my target?


Thanks very much,
Jason


Jason Machacek | Stratos Product Development | Software Engineer |
[phone]

[Paul Pluzhnikov]

 Can anyone help me figure out why I'm unable to get
gdb/gdbserver to work properly with my target?

A couple of questions:
- Does gdbserver config.log mention libthread_db.so, and if so, which one?
- Does ldd gdbserver on target show libthread_db.so, and if so, is the
right one? (Should come from the same compilation from which libc.so.6
and libpthread.so.0 came.)
- Is libpthread.so.0 on target stripped?

Most of the "SIG32" problems come from stripped libpthread.so.0 or
libpthread/libthread_db mismatch.

Good luck!

[Jason Machacek]

Hi Paul,

Thanks for the reply.

Post by Paul Pluzhnikov
- Does gdbserver config.log mention libthread_db.so, and if so, which one?

$ cat config.log | grep libthread_db -A8
configure:3888: checking for libthread_db
configure:3919:
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-gcc -o
conftest -g -O2 conftest.c -lthread_db >&5
configure:3925: $? = 0
configure:3929: test -z
|| test ! -s conftest.err
configure:3932: $? = 0
configure:3935: test -s conftest
configure:3938: $? = 0
configure:4014: result: -lthread_db

It appears gdbserver with the correct libthread_db--my toolchain is
mentioned, so I'm assuming it's linking the correct library. It's the
only ARM-compatible libthread_db.so on my system.

Post by Paul Pluzhnikov
- Does ldd gdbserver on target show libthread_db.so, and if so, is the
right one? (Should come from the same compilation from which libc.so.6
and libpthread.so.0 came.)

I unfortunately don't have ldd on my target, but gdbserver's maps file
shows that it's grabbing libthread_db.so. I installed that version of
libthread_db.so myself from the toolchain along with libpthread.so, and
libc.so.6 is statically linked into my application.

$ cat /proc/212/maps
00008000-00016000 r-xp 00000000 1f:01 549 /root/gdbserver
0001e000-0001f000 rw-p 0000e000 1f:01 549 /root/gdbserver
0001f000-00042000 rwxp 00000000 00:00 0
2aaab000-2aac0000 r-xp 00000000 1f:01 607 /lib/ld-2.3.2.so
2aac0000-2aac1000 rw-p 00000000 00:00 0
2aac7000-2aac8000 rw-p 00014000 1f:01 607 /lib/ld-2.3.2.so
2aac8000-2aacc000 r-xp 00000000 1f:01 799
/lib/libthread_db-1.0.so
2aacc000-2aad0000 ---p 00004000 1f:01 799
/lib/libthread_db-1.0.so
2aad0000-2aad4000 rw-p 00000000 1f:01 799
/lib/libthread_db-1.0.so
2aad4000-2abec000 r-xp 00000000 1f:01 687 /lib/libc-2.3.2.so
2abec000-2abf6000 rw-p 00110000 1f:01 687 /lib/libc-2.3.2.so
2abf6000-2abf9000 rw-p 00000000 00:00 0
7fffd000-80000000 rwxp ffffe000 00:00 0

Post by Paul Pluzhnikov
- Is libpthread.so.0 on target stripped?

It doesn't appear to be. I made a copy of libthread_db-1.0.so and ran
'strip' on it, and the file size dropped. Is this the correct approach?

$ cp
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libthread_d
b-1.0.so .
$ ls -l libthread_db-1.0.so
-rwxr-xr-x 1 jason jason 29579 2009-12-17 09:49 libthread_db-1.0.so
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-strip
libthread_db-1.0.so
$ ls -l libthread_db-1.0.so
-rwxr-xr-x 1 jason jason 18296 2009-12-17 09:50 libthread_db-1.0.so

I tried just stripping debugging information as well, and that also
decreased the file size:

$ ls -l libthread_db-1.0.so
-rwxr-xr-x 1 jason jason 29579 2009-12-17 09:53 libthread_db-1.0.so
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-strip
--strip-debug libthread_db-1.0.so
$ ls -l libthread_db-1.0.so
-rwxr-xr-x 1 jason jason 23781 2009-12-17 09:54 libthread_db-1.0.so


Does this indicate that I've got a library version mismatch somewhere in
my system? I'm using a toolchain from my SBC supplier, but perhaps that
wasn't the toolchain they used to build all the libraries included on
the target.

Is there any problem with setting up a second set of libraries on my
system? E.g. /lib2 that contains the libraries copied from my
toolchain. Would I have to make changes to my toolchain's directory
structure to ensure the directories matched up with those on the target
(for GDB that is)?


Thanks,
Jason

[Paul Pluzhnikov]

Post by Paul Pluzhnikov
- Is libpthread.so.0 on target stripped?

It doesn't appear to be.  I made a copy of libthread_db-1.0.so and ran
'strip' on it, and the file size dropped.  Is this the correct approach?

It's not libthread_db, it's the libpthread that matters.

You want to copy /lib/libpthread.so.0 (which should be a symlink to
/lib/libpthread-2.3.2.so) from target to host, and then:

arm-linux-nm libpthread.so.0 | egrep 'version|threads_events'

You should see one of: nptl_version, __linuxthreads_version or
__pthread_threads_events.

If you see 'no symbols' instead, then your libpthread is stripped, and
that explains your problems.

and libc.so.6 is statically linked into my application.

It is impossible to statically link libc.so.6 into an application. Perhaps
you mean that your app is linked with libc.a ?

In general GDB may have harder time debugging statically-linked threaded
executables, and you should always prefer to link against system libraries
dynamically.

Cheers,

[Jason Machacek]

Post by Paul Pluzhnikov
You want to copy /lib/libpthread.so.0 (which should be a symlink to
arm-linux-nm libpthread.so.0 | egrep 'version|threads_events'
You should see one of: nptl_version, __linuxthreads_version or
__pthread_threads_events.

Sorry about that, I misread your last message. libpthread.so isn't
stripped:

$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
libpthread-0.10.so | egrep 'version|threads_events'
0000de40 r __linuxthreads_version
00018e68 b __pthread_threads_events

Post by Paul Pluzhnikov
It is impossible to statically link libc.so.6 into an application. Perhaps
you mean that your app is linked with libc.a ?
In general GDB may have harder time debugging statically-linked threaded
executables, and you should always prefer to link against system libraries
dynamically.

Yes, I did mean that I statically linked my application with libc.a.
Perhaps I'll reexamine my build environment as this may be the root of
my problems with GDB. When running GDB natively on my target, I had no
problems debugging my application despite it being statically linked.
I'll do some tests to check whether addressing this allows gdb to
recognize threads in my remote application.

Thanks again,
Jason

[Paul Pluzhnikov]

Sorry about that, I misread your last message.  libpthread.so isn't
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
libpthread-0.10.so | egrep 'version|threads_events'
0000de40 r __linuxthreads_version
00018e68 b __pthread_threads_events

AFAIU, this is the host copy of libpthread.

One common mistake is stripping it *on target*. Hence I asked to copy it
back and run nm on what is actually on target.

Of course you could just 'ls -l' or md5sum to verify that host and target
copies are identical.

Yes, I did mean that I statically linked my application with libc.a.
Perhaps I'll reexamine my build environment as this may be the root of
my problems with GDB.  When running GDB natively on my target, I had no
problems debugging my application despite it being statically linked.

Was it on the same target you are using now? (That would rule out stripped
libpthread, or libpthread/libthread_db mismatch, as gdb and gdbserver
should be similarly broken.)

Cheers,

[Jason Machacek]

Post by Paul Pluzhnikov

Post by Jason Machacek
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
libpthread-0.10.so | egrep 'version|threads_events'
0000de40 r __linuxthreads_version
00018e68 b __pthread_threads_events

AFAIU, this is the host copy of libpthread.

That was the target's copy of libpthread.so--I did copy it back to the
host before running arm-linux-nm on it as you suggested.

Post by Paul Pluzhnikov
Was it on the same target you are using now? (That would rule out stripped
libpthread, or libpthread/libthread_db mismatch, as gdb and gdbserver
should be similarly broken.)

Yes, it was the same target. Running GDB natively works perfectly with
multiple threads, but when running GDB from my host and connecting to
gdbserver on my target GDB gets confused when the program reaches the
first pthread_create() call. Unfortunately, my application has recently
become too large to fit in the target's memory while GDB is running, so
I can no longer run GDB natively on my target.

Best regards,
Jason