gdb doesn't show all threads, but gdbserver does

[N. van Bolhuis]

I built a gdb (v6.8) for powerpc to be executed on powerpc target
(This GDB was configured as "powerpc-e300c3-linux"...).
If I use this target gdb to debug my powerpc program and hit a
breakpoint "info threads" shows only one thread.
This is incorrect since there are 7 threads (cat /proc/<pid>/status
shows 7 threads).

If I debug the same program using gdbserver on target and gdb on a
host=i686 (This GDB was configured as "--host=i686-linux
--target=powerpc-e300c3-linux"...) "info threads" shows the expected 7
threads.

How can this be ?

(I noticed the target gdbserver needs libthread_db.so.1, the target gdb
does not. Maybe this is what's causing this).

[teawater]

Both gdbserver and gdb use libthread_db.

gdbserver link together with the lib.
gdb dlopen the lib when it need it.

So I think maybe gdb didn't find the lib or something else. Do you gdb
have some error message?

Post by N. van Bolhuis
I built a gdb (v6.8) for powerpc to be executed on powerpc target
(This GDB was configured as "powerpc-e300c3-linux"...).
If I use this target gdb to debug my powerpc program and hit a
breakpoint "info threads" shows only one thread.
This is incorrect since there are 7 threads (cat /proc/<pid>/status
shows 7 threads).
If I debug the same program using gdbserver on target and gdb on a host=i686 (This GDB was configured as "--host=i686-linux
--target=powerpc-e300c3-linux"...) "info threads" shows the expected 7
threads.
How can this be ?
(I noticed the target gdbserver needs libthread_db.so.1, the target gdb
does not. Maybe this is what's causing this).

[Norbert van Bolhuis]

thanks for your reply teawater.

Btw. I'm using linux-2.6.28, gcc-4.2.4, glibc-2.7 and gdb-6.8.

There is no error message from gdb, see below:

# ./gdb /mnt/norbert/mainctrl.exe
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-e300c3-linux"...
(gdb) br PisIdInterpreter.C:265
Breakpoint 1 at 0x10046644: file /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C, line 265.
(gdb) run
Starting program: /mnt/norbert/mainctrl.exe
========
Build information:
Date: 09-01-15 Time: 13:14:32
CVS tag: (none)
User: nvbolhuis

Ali Model:
Date: 09-01-13 Time: 10:01:26
========


InitProcesses: applications initialized
Username:root
[New LWP 928]
[Switching to LWP 928]

Breakpoint 1, PisIdInterpreter::interpreter (this=0x10365d68)
at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
265 if (!userInput) {
(gdb) info thr
* 1 LWP 928 PisIdInterpreter::interpreter (this=0x10365d68)
at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
(gdb) [1] + Stopped ./gdb /mnt/norbert/mainctrl.exe
# ps
PID USER VSZ STAT COMMAND
1 root 3184 S init
2 root 0 SW< [kthreadd]
3 root 0 SW< [ksoftirqd/0]
4 root 0 SW< [watchdog/0]
5 root 0 SW< [events/0]
6 root 0 SW< [khelper]
82 root 0 SW< [kblockd/0]
95 root 0 SW< [khubd]
141 root 0 SW [pdflush]
142 root 0 SW [pdflush]
143 root 0 SW< [kswapd0]
191 root 0 SW< [aio/0]
198 root 0 SW< [nfsiod]
799 root 0 SW< [mtdblockd]
874 root 0 SW< [hid_compat]
879 root 0 SW< [rpciod/0]
886 root 3188 R -sh
892 root 0 SWN [jffs2_gcd_mtd6]
897 root 3184 S telnetd
919 root 19588 T ./gdb /mnt/norbert/mainctrl.exe
920 root 13352 T N /mnt/norbert/mainctrl.exe
930 root 3188 R ps
# cat /proc/920/status
Name: mainctrl.exe
State: T (tracing stop)
Tgid: 920
Pid: 920
PPid: 919
TracerPid: 919
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 32
Groups: 0 1 2 3 4 6 10
VmPeak: 13352 kB
VmSize: 11428 kB
VmLck: 0 kB
VmHWM: 3404 kB
VmRSS: 3404 kB
VmData: 6696 kB
VmStk: 84 kB
VmExe: 528 kB
VmLib: 3592 kB
VmPTE: 28 kB
Threads: 7
SigQ: 1/1024
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000001000
SigIgn: 0000000000080004
SigCgt: 0000000188102002
CapInh: 0000000000000000
CapPrm: fffffffffffffeff
CapEff: fffffffffffffeff
CapBnd: fffffffffffffeff
voluntary_ctxt_switches: 11
nonvoluntary_ctxt_switches: 2
# fg
./gdb /mnt/norbert/mainctrl.exe

* 1 LWP 928 PisIdInterpreter::interpreter (this=0x10365d68)
at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
(gdb)

[Paul Pluzhnikov]

On Mon, Jan 19, 2009 at 11:28 PM, Norbert van Bolhuis

Post by Norbert van Bolhuis
# cat /proc/920/status

What did you hope to find from status?

What you really want is /proc/919/maps: this will show whether
libthread_db.so.1 was loaded into gdb, and if so, which one.

It should be the same as the one which shows up in
'ldd gdbserver' output.

Cheers,

[N. van Bolhuis]

Post by Paul Pluzhnikov
On Mon, Jan 19, 2009 at 11:28 PM, Norbert van Bolhuis

Post by Norbert van Bolhuis
# cat /proc/920/status

What did you hope to find from status?

proof that 7 threads are really there.

Post by Paul Pluzhnikov
What you really want is /proc/919/maps: this will show whether
libthread_db.so.1 was loaded into gdb, and if so, which one.
It should be the same as the one which shows up in
'ldd gdbserver' output.

In fact I did just that. I didn't mention it since it's there
(libthread_db.so.1 is loaded into gdb) so this is not the issue.

[Paul Pluzhnikov]

Post by N. van Bolhuis
In fact I did just that. I didn't mention it since it's there
(libthread_db.so.1 is loaded into gdb) so this is not the issue.

Hmm, I wonder if the "stripped libpthread.so.0" struck again.

Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
note which libpthread.so.0 it is using, and then

nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'

If this shows 'no symbols', you have stripped libpthread.so.0 on
target (but not on host). Don't do that. Do "strip --strip-debug"
instead.

Cheers,

[N. van Bolhuis]

Post by Paul Pluzhnikov

Post by N. van Bolhuis
In fact I did just that. I didn't mention it since it's there
(libthread_db.so.1 is loaded into gdb) so this is not the issue.

Hmm, I wonder if the "stripped libpthread.so.0" struck again.
Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
note which libpthread.so.0 it is using, and then
nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'
If this shows 'no symbols', you have stripped libpthread.so.0 on
target (but not on host). Don't do that. Do "strip --strip-debug"
instead.

aha, that must be it. I did (fully) strip all libraries on target.

yes if I do "strip --strip-debug" it works.

thanks a lot!

I did not realize gdb needs libpthread debug symbols to "discover"
threads.
Hmm, if I think about it I still not fully understand why.
Anyway it work now and I'm happy.

[Andreas Schwab]

Post by N. van Bolhuis
I did not realize gdb needs libpthread debug symbols to "discover"
threads.

It's actually libthread_db that needs a non-global symbol from
libpthread.

Andreas.

[Paul Pluzhnikov]

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)

Stripping libpthread appears to be a very common mistake:

- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.

I wonder if we should warn glibc users? Something like:

Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.

[teawater]

Agree. Most of multi-thread bug that I meet are stripped libpthread.

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.
--
Paul Pluzhnikov

[Norbert van Bolhuis]

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.

I consulted the GDB manual first, then google.

http://sourceware.org/gdb/wiki is an interesting source of information.
I expect to see this common mistake being mentioned there, but I can't
find it.

Anyway, thanks to everybody for all answers.

[teawater]

What about FAQ?

On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis

Post by Norbert van Bolhuis

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.

I consulted the GDB manual first, then google.
http://sourceware.org/gdb/wiki is an interesting source of information.
I expect to see this common mistake being mentioned there, but I can't
find it.
Anyway, thanks to everybody for all answers.

[teawater]

My mean is maybe you can update this question and answer to FAQ in wiki. :)

Post by teawater
What about FAQ?
On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis

Post by Norbert van Bolhuis

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.

I consulted the GDB manual first, then google.
http://sourceware.org/gdb/wiki is an interesting source of information.
I expect to see this common mistake being mentioned there, but I can't
find it.
Anyway, thanks to everybody for all answers.

[Paul Pluzhnikov]

I just updated the wiki ...

Post by teawater
My mean is maybe you can update this question and answer to FAQ in wiki. :)

Post by teawater
What about FAQ?
On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis

Post by Norbert van Bolhuis

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.

I consulted the GDB manual first, then google.
http://sourceware.org/gdb/wiki is an interesting source of information.
I expect to see this common mistake being mentioned there, but I can't
find it.
Anyway, thanks to everybody for all answers.

[teawater]

I make a patch to output a warning.
http://sourceware.org/ml/gdb-patches/2009-02/msg00234.html

Thanks,
Hui

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.
--
Paul Pluzhnikov

[teawater]

http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html

Post by teawater
I make a patch to output a warning.
http://sourceware.org/ml/gdb-patches/2009-02/msg00234.html
Thanks,
Hui

Post by Paul Pluzhnikov

Post by Andreas Schwab
It's actually libthread_db that needs a non-global symbol from
libpthread.

IOW, it's a glibc feature :)
- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
must not be stripped.
Warning: your /lib64/libpthread.so.0 has been stripped.
GDB will not see other threads, and multithreaded debugging will
be unreliable.
See http://sourceware.org/gdb/wiki/... for further info.
--
Paul Pluzhnikov

[teawater]

Really?
Why your gdbserver can be use? Is it static link?

Post by N. van Bolhuis

Post by Paul Pluzhnikov

Post by N. van Bolhuis
In fact I did just that. I didn't mention it since it's there
(libthread_db.so.1 is loaded into gdb) so this is not the issue.

Hmm, I wonder if the "stripped libpthread.so.0" struck again.
Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
note which libpthread.so.0 it is using, and then
nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'
If this shows 'no symbols', you have stripped libpthread.so.0 on
target (but not on host). Don't do that. Do "strip --strip-debug"
instead.

aha, that must be it. I did (fully) strip all libraries on target.
yes if I do "strip --strip-debug" it works.
thanks a lot!
I did not realize gdb needs libpthread debug symbols to "discover"
threads.
Hmm, if I think about it I still not fully understand why.
Anyway it work now and I'm happy.

[Paul Pluzhnikov]

Post by teawater
Really?
Why your gdbserver can be use? Is it static link?

I think it works for OP because with gdbserver, the host GDB is
doing symbol lookups, and the copy of target libpthread which host
GDB uses has not been stripped.

Cheers,

[teawater]

I think gdbserver need non-strip libpthread too. I met this issue in before.
So it make me confused. :)

Post by Paul Pluzhnikov

Post by teawater
Really?
Why your gdbserver can be use? Is it static link?

I think it works for OP because with gdbserver, the host GDB is
doing symbol lookups, and the copy of target libpthread which host
GDB uses has not been stripped.
Cheers,
--
Paul Pluzhnikov

[teawater]

This part of code in gdb/gdbserver/thread-db.c:thread_db_init.

Post by teawater
I think gdbserver need non-strip libpthread too. I met this issue in before.
So it make me confused. :)

Post by Paul Pluzhnikov

Post by teawater
Really?
Why your gdbserver can be use? Is it static link?

I think it works for OP because with gdbserver, the host GDB is
doing symbol lookups, and the copy of target libpthread which host
GDB uses has not been stripped.
Cheers,
--
Paul Pluzhnikov

[Daniel Jacobowitz]

Post by teawater
This part of code in gdb/gdbserver/thread-db.c:thread_db_init.

No, what Paul said is correct. The symbol lookup is performed by the
GDB client, not by gdbserver - so it uses the copy of libpthread.so on
the host.

[teawater]

Thank you for help me make it clean, Daniel. :)

Hui

Post by Daniel Jacobowitz

Post by teawater
This part of code in gdb/gdbserver/thread-db.c:thread_db_init.

No, what Paul said is correct. The symbol lookup is performed by the
GDB client, not by gdbserver - so it uses the copy of libpthread.so on
the host.
--
Daniel Jacobowitz
CodeSourcery