Debugging Multithreaded Applications
TracNav menu
-
User Guide
- Copyright Information
- Acknowledgements
- Terminology
- Introduction
-
PetaLinux Overview...
-
Working with PetaLinux - The Basics...
-
Customising PetaLinux...
-
Advanced Topics...
-
Debugging
- Debugging User Applications
- Debugging Multithreaded Applications
-
PetaLinux Bootloader Solutions...
-
Supported Reference Designs...
- PetaLinux Tools Reference
- Getting Help
Overview
This document describes how to debug multithreaded user applications under the uClinux operating system for Xilinx MicroBlaze. The GDB debugger (and Insight graphical interface) is available for debugging applications on the MicroBlaze Linux target. To do so, it is first necessary to build the gdbserver helper application. gdbserver runs on the MicroBlaze, and acts as an interface between GDB running on the host, and the target application.
The debug scenario is illustrated in Figure 1.
Figure 1. uClinux application debugging overview
Assumptions
The following assumptions were made in the preparation of this tutorial:
- Basic familiarity with the PetaLinux or MicroBlaze uClinux development methodology
- Existence of a working MicroBlaze uClinux system, including TCP/IP networking support
- Familiarity with single-threaded application debugging.
Preliminaries
A number of configuration settings must be made prior to debugging threaded applications. The steps to select these options are detailed below.
Enable debuggable applications and libraries
To ensure that the appropriate debugging symbols are inserted in the application and library object code, and to limit aggressive compiler optimisations, it is necessary to enable debug options on user applications and library code.
Start the Kernel config tool.
$ cd $PETALINUX/software/petalinux-dist $ make menuconfig
Select the Customize Vendor/User Settings menu
┌──────────────────── Kernel/Library/Defaults Selection ────────────────────┐ │ Arrow keys navigate the menu. <Enter> selects submenus --->. │ │ Highlighted letters are hotkeys. Pressing <Y> includes, <N> excludes, │ │ <M> modularizes features. Press <Esc><Esc> to exit, <?> for Help. │ │ Legend: [*] built-in [ ] excluded <M> module < > module capable │ │ ┌───────────────────────────────────────────────────────────────────────┐ │ │ │ --- Kernel is linux-2.4.x │ │ │ │ (None) Libc Version │ │ │ │ [ ] Default all settings (lose changes) │ │ │ │ [ ] Customize Kernel Settings │ │ │ │ [*] Customize Vendor/User Settings │ │ │ │ [ ] Update Default Vendor Settings │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────────────────┤ │ <Select> < Exit > < Help > │ └───────────────────────────────────────────────────────────────────────────┘
Save and Exit from the menu.
Under the Debug Builds menu, select the options as per the example given below.
┌────────────────────────────── Debug Builds ───────────────────────────────┐ │ Arrow keys navigate the menu. <Enter> selects submenus --->. │ │ Highlighted letters are hotkeys. Pressing <Y> includes, <N> excludes, │ │ <M> modularizes features. Press <Esc><Esc> to exit, <?> for Help. │ │ Legend: [*] built-in [ ] excluded <M> module < > module capable │ │ ┌───────────────────────────────────────────────────────────────────────┐ │ │ │ [*] build debugable libraries │ │ │ │ [*] build debugable applications │ │ │ │ --- Debug tools │ │ │ │ [ ] tpt │ │ │ │ --- Debug libraries │ │ │ │ [ ] ccmalloc │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────────────────┤ │ <Select> < Exit > < Help > │ └───────────────────────────────────────────────────────────────────────────┘
Save and Exit from the menu.
!
When debugging applications under uClinux, you should always make sure that they are built with the “debuggable libraries” and “debuggable applications” options enabled. Failure to do so can make using the debugger difficult, if not impossible.
For production software releases, make sure these options are disabled, to improve performance and reduce code size.
Building gdbserver
gdbserver is the helper application used to control the program that is under debug.
Start the Kernel config tool.
$ cd $PETALINUX/software/petalinux-dist $ make menuconfig
Select the Customize Vendor/User Settings menu
┌──────────────────── Kernel/Library/Defaults Selection ──────────────────── │ Arrow keys navigate the menu. <Enter> selects submenus --->. │ │ Highlighted letters are hotkeys. Pressing <Y> includes, <N> excludes, │ │ <M> modularizes features. Press <Esc><Esc> to exit, <?> for Help. │ │ Legend: [*] built-in [ ] excluded <M> module < > module capable │ │ ┌───────────────────────────────────────────────────────────────────────┐ │ │ │ --- Kernel is linux-2.4.x │ │ │ │ (None) Libc Version │ │ │ │ [ ] Default all settings (lose changes) │ │ │ │ [ ] Customize Kernel Settings │ │ │ │ [*] Customize Vendor/User Settings │ │ │ │ [ ] Update Default Vendor Settings │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────────────────┤ │ <Select> < Exit > < Help > │ └───────────────────────────────────────────────────────────────────────────┘
Save and Exit from the menu.
Under the Miscellaneous Configuration menu, select the gdbserver option.
┌─────────────────────── Miscellaneous Applications ──────────────────────── │ Arrow keys navigate the menu. <Enter> selects submenus --->. │ │ Highlighted letters are hotkeys. Pressing <Y> includes, <N> excludes, │ │ <M> modularizes features. Press <Esc><Esc> to exit, <?> for Help. │ │ Legend: [*] built-in [ ] excluded <M> module < > module capable │ │ ┌───────────────^(-)────────────────────────────────────────────────────┐ │ │ │ [ ] dhrystone │ │ │ │ [ ] de2ts-cal │ │ │ │ [ ] expat-examples │ │ │ │ [ ] frob-led │ │ │ │ [ ] gdbreplay │ │ │ │ [*] gdbserver │ │ │ │ [ ] gdbreplay (old) │ │ │ │ [ ] gdbserver (old) │ │ │ │ [ ] grep │ │ │ │ [*] hd │ │ │ │ [ ] lcd │ │ │ └───────────────┴(+)────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────────────────┤ │ <Select> < Exit > < Help > │ └───────────────────────────────────────────────────────────────────────────┘
Do not exit from this menu yet.
Building the thread test application
For this tutorial, we will debug the standard thread demo applications that are part of the uClinux distribution (thdm).
In the same Miscellaneous Applications menu, enable the pThreads threaddemos option.
┌─────────────────────── Miscellaneous Applications ──────────────────────── │ Arrow keys navigate the menu. <Enter> selects submenus --->. │ │ Highlighted letters are hotkeys. Pressing <Y> includes, <N> excludes, │ │ <M> modularizes features. Press <Esc><Esc> to exit, <?> for Help. │ │ Legend: [*] built-in [ ] excluded <M> module < > module capable │ │ ┌───────────────^(-)────────────────────────────────────────────────────┐ │ │ │ [ ] strace │ │ │ │ [ ] stty │ │ │ │ [ ] tcsh │ │ │ │ [ ] tinytcl │ │ │ │ [*] pThreads threaddemos │ │ │ │ [ ] tip │ │ │ │ [ ] tripwire │ │ │ │ [ ] unrar │ │ │ │ [ ] unzip │ │ │ │ [ ] unzoo │ │ │ │ [ ] zmodem utils │ │ │ └───────────────┴(+)────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────────────────┤ │ <Select> < Exit > < Help > │ └───────────────────────────────────────────────────────────────────────────┘
Rebuilding the PetaLinux image
After making the various configuration changes detailed above, it is required to perform a “make clean” and complete system rebuild. This ensures that the C libraries and applications are rebuilt with debug enabled, and that the applications are linked against the debuggable libraries.
$ cd $PETALINUX/software/petalinux-dist $ make clean dep all
Once the rebuild is complete, boot the target MicroBlaze system with your new image.
Debugging a multithreaded application
On the MicroBlaze system, use the gdbserver application to launch your application. In this tutorial, we will debug the thdm thread demo application.
# gdbserver host:1234 /bin/thdm Process /bin/thdm created; pid = 50 Listening on port 1234
The command line tells gdbserver to listen on TCP/IP port 1234 for a connection from the remote debugger.
On your development workstation, change into the directory where the application was built, e.g.
[host]$ cd $PETALINUX/software/petalinux-dist/user/threaddemos
Among the files in this directory, will be a file thdm.gdb – this is a special version of your application that helps the gdb debugger to work.
Launch the uClinux application debugger:
[host]$ microblaze-uclinux-gdb thdm.gdb
!
When debugging uClinux applications, you must use the uClinux-specific version of the gdb debugger – microblaze-uclinux-gdb. Do not confuse this with the standalone !MicroBlaze debugger, mb-gdb
Once the debugger window opens, select File → Target Settings from the main menu. This will open the Target Settings Dialog box. The settings must match those shown in Figure 2 below, except for the IP address field, which must correspond to the IP address of your MicroBlaze target board (192.168.0.10 in this example). Finally, press OK to close the Target Settings dialog box.
Figure 2. GDB Target Settings dialog box
Now, press the “run” icon. GDB will connect with gdbserver on the MicroBlaze, and run your application until the start of the main() function. You will see the following output on your MicroBlaze console, showing that the connection has been made (192.168.0.1 is the IP address of the host computer in this example):
Remote debugging from host 192.168.0.1
Figure 3 shows the debugger with the application stopped on the first line of the main() function.
Figure 3. GDB ready for debugging
In addition to the standard debug controls, some extra options are now available for multithreaded applications. Select the View->Threads option from the main menu. At the moment, there is only a single thread, as shown in Figure 4.
Figure 4. The thread info window at the start of the main() function
Next, set a breakpoint a little lower down in the main function, on line 33, after the loop that creates the threads (pthread_create() function call). Press the Continue icon to resume the program. The application will stop with the green highlight on line 33 (Figure 5).
Figure 5. The application stopped after creation of the worker threads
Take a look at the thread window, it now shows that a number of threads have been created (Figure 6).
Figure 6. The thread window after four worker threads are created
?
If there is one main thread, and four worker threads (THREADS is #defined as 4 in this example), why are there six threads?
The answer is that the pthreads library creates a manager thread to handle thread creation and synchronisation events. In this example it’s thread number 2 in the thread list.
You can select the various threads by clicking on them in the thread window. Clicking on any of the worker threads (thread IDs 3 → 6) will show that they are all stopped at the sched_yield() library call (line 16 of the test program).
Next set a breakpoint in a thread. Place a breakpoint on line 15 of the program, inside the thread_code() function. Press the Continue icon again, gdb should return almost immediately, but now stopped in one of the threads, at line 15. The thread window will identify which of the threads has stopped.
!
By default, a breakpoint in a thread will stop all threads at that same point. Setting breakpoints in individual threads can be done from the gdb console, with the command “break {line-num} thread {thread-id}”
Each thread has its own context, with stack, local variables and so on. You can view the context of a thread by selecting the View->Stack option from the main menu. Figure 7 shows the stack trace of one of the threads.
Figure 7. Stack trace of a stopped thread
Finally, remove all breakpoints, and press the Continue icon once more. The program will run to completion, its output displayed on the uClinux console.
Going Further
For more information on the gdb tool, consult the GNU manual, available online at
http://sources.redhat.com/gdb/current/onlinedocs/gdb_toc.html
In particular, see Section 4.9 on debugging threaded applications and Section 5.4 on stopping and starting threaded applications.
Attachments
- after_4_worker.png (9.4 kB) - added by bennyc on 10/20/06 07:31:57.
- app_debug_overview.png (9.7 kB) - added by bennyc on 10/20/06 07:32:09.
- at_start_of_main.png (6.5 kB) - added by bennyc on 10/20/06 07:32:20.
- stack_trace.png (5.1 kB) - added by bennyc on 10/20/06 07:32:34.
- stop_after_creation.png (26.6 kB) - added by bennyc on 10/20/06 07:32:47.
- target_settings_diag_box.png (15.0 kB) - added by bennyc on 10/20/06 07:32:58.
- ready_for_debug.png (26.6 kB) - added by bennyc on 10/20/06 11:42:21.
