Discussion:
debug of multithreaded program turns to be impossible during execution
(too old to reply)
Avi Gozlan
2014-07-31 09:27:27 UTC
Permalink
Raw Message
Hello,

I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:

(gdb) c
Continuing.
[New Thread 0x41401940 (LWP 13370)]
[Thread 0x40a00940 (LWP 13333) exited]
[New Thread 0x41e02940 (LWP 13371)]
...
[New Thread 0x45a08940 (LWP 13377)]
[New Thread 0x46409940 (LWP 13378)]
[Thread 0x41e02940 (LWP 13371) exited]
[Switching to Thread 0x45a08940 (LWP 13377)]
Cannot remove breakpoints because program is no longer writable.
Further execution is probably impossible.
0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
ptrace: No such process.

Surprisingly, although debugging is no more possible, the program is not dead:

(gdb) q
A debugging session is active.

Inferior 1 [process 13296] will be killed.

Quit anyway? (y or n) y
[Thread 0x46409940 (LWP 13378) exited]
[Thread 0x45a08940 (LWP 13377) exited]
...
[Thread 0x2aaaaccbe580 (LWP 13296) exited]

Some details:
GDB version: 7.6
OS: RHEL 5.1 based
GLIBC: glibc-2.5-18
/lib64/libthread-1.0.so is not stripped.

Your assistance will be appreciated.
Thanks,
Avi
Pedro Alves
2014-07-31 10:17:25 UTC
Permalink
Raw Message
On 07/31/2014 10:27 AM, Avi Gozlan wrote:
> Hello,
>
> I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:
>
> (gdb) c
> Continuing.
> [New Thread 0x41401940 (LWP 13370)]
> [Thread 0x40a00940 (LWP 13333) exited]
> [New Thread 0x41e02940 (LWP 13371)]
> ...
> [New Thread 0x45a08940 (LWP 13377)]
> [New Thread 0x46409940 (LWP 13378)]
> [Thread 0x41e02940 (LWP 13371) exited]
> [Switching to Thread 0x45a08940 (LWP 13377)]
> Cannot remove breakpoints because program is no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
>
> Surprisingly, although debugging is no more possible, the program is not dead:

I have a vague recollection that this was fixed a while ago. Like,
GDB was picking the wrong thread to access memory, and it ended up
picking a running or exited thread, and that fails (in either
case ptrace complains the same say).

> GDB version: 7.6

Is there a chance you could try a more recent GDB version? Ideally
7.8, which we just released.

Thanks,
Pedro Alves
Avi Gozlan
2014-08-14 15:52:40 UTC
Permalink
Raw Message
GDB 7.8 yields a similar behavior:

(gdb) c
Continuing.
...
[New Thread 0x43c05940 (LWP 3311)]
[New Thread 0x44606940 (LWP 3312)]
...
[Thread 0x44606940 (LWP 3312) exited]
[Switching to Thread 0x43c05940 (LWP 3311)]
Cannot remove breakpoints because program is no longer writable.
Further execution is probably impossible.
0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
ptrace: No such process.
(gdb) [Thread 0x46409940 (LWP 3315) exited]
[Thread 0x45a08940 (LWP 3314) exited]
[Thread 0x43c05940 (LWP 3311) exited]
[Inferior 1 (process 3268) exited normally]

The only difference is that in 7.8 case, GDB continues automatically after giving the ptrace message.

Avi

-----Original Message-----
From: Pedro Alves [mailto:***@redhat.com]
Sent: Thursday, July 31, 2014 1:17 PM
To: Avi Gozlan; ***@sourceware.org
Subject: Re: debug of multithreaded program turns to be impossible during execution

On 07/31/2014 10:27 AM, Avi Gozlan wrote:
> Hello,
>
> I'm debugging a multithreaded program using GDB. Various threads are created and exit during execution. GDB copes successfully with the multiple threads (e.g. 'thread apply 3 bt' reports successfully backtrace of thread 3), but then at one point loses control of the program:
>
> (gdb) c
> Continuing.
> [New Thread 0x41401940 (LWP 13370)]
> [Thread 0x40a00940 (LWP 13333) exited] [New Thread 0x41e02940 (LWP
> 13371)] ...
> [New Thread 0x45a08940 (LWP 13377)]
> [New Thread 0x46409940 (LWP 13378)]
> [Thread 0x41e02940 (LWP 13371) exited] [Switching to Thread
> 0x45a08940 (LWP 13377)] Cannot remove breakpoints because program is
> no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
>
> Surprisingly, although debugging is no more possible, the program is not dead:

I have a vague recollection that this was fixed a while ago. Like, GDB was picking the wrong thread to access memory, and it ended up picking a running or exited thread, and that fails (in either case ptrace complains the same say).

> GDB version: 7.6

Is there a chance you could try a more recent GDB version? Ideally 7.8, which we just released.

Thanks,
Pedro Alves


Email secured by Check Point.
Pedro Alves
2014-08-19 16:05:51 UTC
Permalink
Raw Message
On 08/14/2014 04:52 PM, Avi Gozlan wrote:
> GDB 7.8 yields a similar behavior:
>
> (gdb) c
> Continuing.
> ...
> [New Thread 0x43c05940 (LWP 3311)]
> [New Thread 0x44606940 (LWP 3312)]
> ...
> [Thread 0x44606940 (LWP 3312) exited]
> [Switching to Thread 0x43c05940 (LWP 3311)]
> Cannot remove breakpoints because program is no longer writable.
> Further execution is probably impossible.
> 0x00002aaaab98348f in start_thread () from /lib64/libpthread.so.0
> ptrace: No such process.
> (gdb) [Thread 0x46409940 (LWP 3315) exited]
> [Thread 0x45a08940 (LWP 3314) exited]
> [Thread 0x43c05940 (LWP 3311) exited]
> [Inferior 1 (process 3268) exited normally]
>
> The only difference is that in 7.8 case, GDB continues automatically after giving the ptrace message.

Any chance you could try your reproducer with a more recent
system? Mainly, I'm wondering whether kernel/glibc are at play here.

Also, a run with:

set debug infrun 1
set debug target 1
set debug lin-lwp 1

may help diagnose the issue.

Thanks,
Pedro Alves
Loading...