第 41 / 187 篇

3.6. 用 GDB 调试多线程程序

说明

本文是个人学习用的非官方中英对照译文,原文:https://diveintosystems.org/book/C3-C_debug/gdb_pthreads.html。
原书采用 CC BY-NC-ND 4.0 许可(禁止演绎),请勿公开传播本译文。
标为「译注」「现代更新」的内容是译者补充,不代表原书作者观点;代码与原文逐字一致,未作改动。

3.6. Debugging Multithreaded Programs with GDB

由于存在多条执行流,以及并发执行的线程之间的相互作用,调试多线程程序可能很棘手。一般来说,下面这些做法能让调试多线程程序稍微容易一些:

Debugging multithreaded programs can be tricky due to the multiple streams of execution and due to interactions between the concurrently executing threads. In general, here are some things to make debugging multithreaded programs a bit easier:

  • 尽可能调试线程数尽量少的程序版本。
  • When possible, try to debug a version of the program with as few threads as possible.
  • 在代码中加入调试用的 printf 语句时,打印出正在执行的线程的 ID,以便识别是哪个线程在打印,并以 \n 结束这一行。
  • When adding debugging printf statements to the code, print out the executing thread’s ID to identify which thread is printing and end the line with a \n.
  • 只让其中一个线程打印它自己的信息和公共信息,以限制调试输出的数量。例如,如果每个线程都把自己的逻辑 ID 存放在名为 my_tid 的局部变量中,就可以对 my_tid 的值加一个条件判断,把调试输出限制在一个线程上,如下例所示:
  • Limit the amount of debug output by having only one of the threads print its information and common information. For example, if each thread stores its logical ID in a local variable named my_tid, a conditional statement on the value of my_tid can be used to limit printing debug output to one thread, as illustrated in the following example:
1
2
3
4
if (my_tid == 1) {
printf("Tid:%d: value of count is %d and my i is %d\n", my_tid, count, i);
fflush(stdout);
}

3.6.1. GDB 与 Pthreads

3.6.1. GDB and Pthreads

GDB 调试器对调试多线程程序有专门的支持,包括为单个线程设置断点、检查单个线程的栈。在 GDB 中调试 Pthreads 程序时要注意,每个线程至少有三种标识符:

The GDB debugger has specific support for debugging threaded programs, including setting breakpoints for individual threads and examining the stacks of individual threads. One thing to note when debugging Pthreads programs in GDB is that there are at least three identifiers for each thread:

  • Pthreads 库为线程分配的 ID(它的 pthread_t 值)。
  • The Pthreads library’s ID for the thread (its pthread_t value).
  • 操作系统为线程分配的轻量级进程(lightweight process,LWP)ID。操作系统部分地用这个 ID 来跟踪该线程,以便进行调度。
  • The operating system’s lightweight process (LWP) ID value for the thread. This ID is used in part for the OS to keep track of this thread for scheduling purposes.
  • GDB 为线程分配的 ID。在 GDB 命令中指定某个线程时用的就是这个 ID。
  • The GDB ID for the thread. This is the ID to use when specifying a specific thread in GDB commands.

这几种线程 ID 之间的具体关系,因操作系统和 Pthreads 库实现的不同而有所差异,但在大多数系统上,Pthreads ID、LWP ID 和 GDB 线程 ID 之间是一一对应的。

The specific relationship between thread IDs can differ from one OS and Pthreads library implementation to another, but on most systems there is a one-to-one-to-one correspondence between a Pthreads ID, an LWP ID, and a GDB thread ID.

我们介绍在 GDB 中调试多线程程序的一些基础知识。关于在 GDB 中调试多线程程序的更多信息,请参见该链接。

We present a few GDB basics for debugging threaded programs in GDB. See the following for more information about debugging threaded programs in GDB.

3.6.2. GDB 中与线程相关的命令:

3.6.2. GDB Thread-Specific Commands:

  • 开启线程启动和退出事件的打印:

    1
    set print thread-events
  • Enable printing thread start and exit events:

    1
    set print thread-events
  • 列出程序中现有的所有线程(列出的第一个值是 GDB 线程编号,遇到断点的那个线程用 * 标出):

    1
    info threads
  • List all existing threads in the program (the GDB thread number is the first value listed and the thread that hit the breakpoint is denoted with an *):

    1
    info threads
  • 切换到某个特定线程的执行上下文(例如,以便执行 where 查看它的栈),用线程 ID 指定线程:

    1
    2
    3
    4
    thread <threadno>

    thread 12 # Switch to thread 12's execution context
    where # Thread 12's stack trace
  • Switch to a specific thread’s execution context (for example, to examine its stack when executing where), specify the thread by its thread ID:

    1
    2
    3
    4
    thread <threadno>

    thread 12 # Switch to thread 12's execution context
    where # Thread 12's stack trace
  • 只为某个特定线程设置断点。在设置断点的代码位置执行的其他线程,不会触发这个断点使程序暂停并打印 GDB 提示符:

    1
    2
    3
    break <where> thread <threadno>

    break foo thread 12 # Break when thread 12 executes function foo
  • Set a breakpoint for just a particular thread. Other threads executing at the point in the code where the breakpoint is set will not trigger the breakpoint to pause the program and print the GDB prompt:

    1
    2
    3
    break <where> thread <threadno>

    break foo thread 12 # Break when thread 12 executes function foo
  • 要把某条 GDB 命令应用到所有线程或部分线程,可以在 GDB 命令前加上前缀 thread apply <threadno | all>,其中 threadno 指 GDB 线程 ID:

    1
    thread apply <threadno|all> command

    这并不适用于所有 GDB 命令,尤其是设置断点,所以设置线程专属的断点时要改用下面的语法:

    1
    break <where> thread <threadno>
  • To apply a specific GDB command to all or to a subset of threads, by adding the prefix thread apply <threadno | all> to a GDB command, where threadno refers to the GDB thread ID:

    1
    thread apply <threadno|all> command

    This doesn’t work for every GDB command, setting breakpoints in particular, so use this syntax instead for setting thread-specific breakpoints:

    1
    break <where> thread <threadno>

默认情况下,到达断点时,GDB 会暂停所有线程,直到用户输入 cont。用户可以改变这一行为,要求 GDB 只暂停遇到断点的那些线程,让其他线程继续执行。

Upon reaching a breakpoint, by default, GDB pauses all threads until the user types cont. The user can change the behavior to request that GDB only pause the threads that hit a breakpoint, allowing other threads to continue executing.

译注

这种模式在 GDB 中叫不停止模式(non-stop mode),用 set non-stop on 开启(需要在程序开始运行之前设置);默认的「一停全停」叫 all-stop 模式。GDB 手册:Non-Stop Mode

3.6.3. 示例:

3.6.3. Examples:

下面展示一些 GDB 命令,以及在由 racecond.c 文件编译出的多线程可执行文件上运行 GDB 时的输出。

We show some GDB commands and output from a GDB run on a multithreaded executable compiled from the file racecond.c.

这个有缺陷的程序在访问共享变量 count 时缺少同步。因此,程序的不同次运行会得出不同的 count 最终值,这表明存在竞态条件。例如,下面是用五个线程运行程序两次、得出不同结果的情形:

This errant program lacks synchronization around accesses to the shared variable count. As a result, different runs of the program produce different final values for count, indicating a race condition. For example, here are two runs of the program with five threads that produce different results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
./a.out 5
hello I'm thread 0 with pthread_id 139673141077760
hello I'm thread 3 with pthread_id 139673115899648
hello I'm thread 4 with pthread_id 139673107506944
hello I'm thread 1 with pthread_id 139673132685056
hello I'm thread 2 with pthread_id 139673124292352
count = 159276966

./a.out 5
hello I'm thread 0 with pthread_id 140580986918656
hello I'm thread 1 with pthread_id 140580978525952
hello I'm thread 3 with pthread_id 140580961740544
hello I'm thread 2 with pthread_id 140580970133248
hello I'm thread 4 with pthread_id 140580953347840
count = 132356636

修复方法是用一个 pthread_mutex_t 变量,把对 count 的访问放进临界区。如果用户仅凭查看 C 代码看不出这个修复方法,那么在 GDB 中运行并在访问 count 变量的位置附近设置断点,也许能帮助程序员发现问题。

The fix is to put accesses to count inside a critical section, using a pthread_mutex_t variable. If the user was not able to see this fix by examining the C code alone, running in GDB and putting breakpoints around accesses to the count variable may help the programmer discover the problem.

下面是在 GDB 中运行这个程序时使用的一些示例命令:

Here are some example commands from a GDB run of this program:

1
2
3
4
5
6
7
(gdb) break worker_loop   # Set a breakpoint for all spawned threads
(gdb) break 77 thread 4 # Set a breakpoint just for thread 4
(gdb) info threads # List information about all threads
(gdb) where # List stack of thread that hit the breakpoint
(gdb) print i # List values of its local variable i
(gdb) thread 2 # Switch to different thread's (2) context
(gdb) print i # List thread 2's local variables i

下面的例子展示了用 3 个线程(run 3)在 GDB 中运行 racecond 程序的部分输出,演示了在 GDB 调试会话中使用 GDB 线程命令的情形。主线程总是 GDB 线程编号 1,创建出来的三个线程是 GDB 线程 2 到 4。

Shown in the example that follows is partial output of a GDB run of the racecond program with 3 threads (run 3), showing examples of GDB thread commands in the context of a GDB debugging session. The main thread is always GDB thread number 1, and the three spawned threads are GDB threads 2 to 4.

调试多线程程序时,GDB 用户在发出命令时必须清楚当前存在哪些线程。例如,当 main 中的断点被触发时,只存在线程 1(主线程)。因此,GDB 用户必须等到线程被创建之后,才能设置只针对某个特定线程的断点(这个例子展示的是只为线程 4 在程序第 77 行设置断点)。查看这段输出时,注意断点是何时设置和删除的,也注意用 GDB 的 thread 命令切换线程上下文时,每个线程的局部变量 i 的值:

When debugging multithreaded programs, the GDB user must keep track of which threads exist when issuing commands. For example, when the breakpoint in main is hit, only thread 1 (the main thread) exists. As a result, the GDB user must wait until threads are created before setting a breakpoint for only a specific thread (this example shows setting a breakpoint for thread 4 only at line 77 in the program). In viewing this output, note when breakpoints are set and deleted, and note the value of each thread’s local variable i when thread contexts are switched with GDB’s thread command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
$ gcc -g racecond.c -pthread

$ gdb ./a.out
(gdb) break main
Breakpoint 1 at 0x919: file racecond.c, line 28.
(gdb) run 3
Starting program: ...
[Thread debugging using libthread_db enabled] ...

Breakpoint 1, main (argc=2, argv=0x7fffffffe388) at racecond.c:28
28 if (argc != 2) {
(gdb) list 76
71 myid = *((int *)arg);
72
73 printf("hello I'm thread %d with pthread_id %lu\n",
74 myid, pthread_self());
75
76 for (i = 0; i < 10000; i++) {
77 count += i;
78 }
79
80 return (void *)0;

(gdb) break 76
Breakpoint 2 at 0x555555554b06: file racecond.c, line 76.
(gdb) cont
Continuing.

[New Thread 0x7ffff77c4700 (LWP 5833)]
hello I'm thread 0 with pthread_id 140737345505024
[New Thread 0x7ffff6fc3700 (LWP 5834)]
hello I'm thread 1 with pthread_id 140737337112320
[New Thread 0x7ffff67c2700 (LWP 5835)]
[Switching to Thread 0x7ffff77c4700 (LWP 5833)]

Thread 2 "a.out" hit Breakpoint 2, worker_loop (arg=0x555555757280)
at racecond.c:76
76 for (i = 0; i < 10000; i++) {
(gdb) delete 2

(gdb) break 77 thread 4
Breakpoint 3 at 0x555555554b0f: file racecond.c, line 77.
(gdb) cont
Continuing.

hello I'm thread 2 with pthread_id 140737328719616
[Switching to Thread 0x7ffff67c2700 (LWP 5835)]

Thread 4 "a.out" hit Breakpoint 3, worker_loop (arg=0x555555757288)
at racecond.c:77
77 count += i;
(gdb) print i
$2 = 0
(gdb) cont
Continuing.
[Switching to Thread 0x7ffff67c2700 (LWP 5835)]

Thread 4 "a.out" hit Breakpoint 3, worker_loop (arg=0x555555757288)
at racecond.c:77
77 count += i;
(gdb) print i
$4 = 1

(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff6fc3700 (LWP 5834))]
#0 0x0000555555554b12 in worker_loop (arg=0x555555757284) at racecond.c:77
77 count += i;
(gdb) print i
$5 = 0

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff77c4700 (LWP 5833))]
#0 worker_loop (arg=0x555555757280) at racecond.c:77
77 count += i;
(gdb) print i
$6 = 1