I keep taking detours! This week, I built a Linux debug image, and looked at kernel source and symbols using SourcePoint, our JTAG-based hardware-assisted debugger.
In Episode 41, Hacking the Linux Kernel, Part 2, I hacked the Ethernet driver for VirtualBox (function e1000_probe()) and for the native Ubuntu partition on my home PC (function igb_probe()). My next step was to do the same for the Ethernet driver on the MinnowBoard; I got so far as to identify that it’s a Realtek Ethernet Controller on the board. Although the command “lspci -v” does not work within the core-image-minimal that I built, there are a lot of references to the rtl8169 Realtek network controller procedures within the system.map file – more on that later.
Since I’m using the Yocto build framework for creating my MinnowBoard image, I had to use a different approach to patching than what was in Episode 41. Luckily, this is pretty well documented within the Yocto Kernel Reference Manual. There are two options for hacking the kernel:
- Using Traditional Kernel Development to Patch the Kernel
- Using devtool to Patch the Kernel
I spent a fair bit of time with #1, but could never successfully build the image as directed in the applicable section in the Yocto Kernel Reference Manual. I’m not sure what I am doing wrong; I’m following the directions exactly, but I just keep getting error messages out of the build:
Oh well, I’ll check this out later, and also try the #2 option, using devtool. The latter is probably the better approach, anyway.
But in the meantime, I decided I just wanted to explore the kernel source code and its operation, using our SourcePoint tool.
By using some of the source-level and symbolic debug capabilities of SourcePoint, combined with the trace facilities within the Bay Trail processor on the MinnowBoard, it should be easy to watch the Linux kernel in action, and see how programs flow and interact with each other.
The first step was to build a “debug image” that supports symbolic debugging. I looked all over the web, and found it very difficult to find a definitive resource. The Yocto documentation contains a lot of information, but it was hard to piece together this specific use case. I Googled all over the place, and after a lot of searching, I found that I needed to include these three lines in my poky local.conf configuration file:
EXTRA_IMAGE_FEATURES ?= “debug-tweaks dbg-pkgs”
DEBUG_BUILD = “1”
SELECTED_OPTIMIZATION = “${DEBUG_OPTIMIZATION}”
Wow. That was hard to find.
I then ran the Yocto build, and it succeeded nicely.
Now, the next step was to find out where the Yocto build deposited the symbol information for SourcePoint to use. Initially, I thought that this was easy: I found the file:
System.map-4.9.126-intel-pk-standard
in the directory:
/poky/build/tmp/work-shared/intel-corei7-64/kernel-build-artifacts
This is a human-readable file, and here’s some random lines within that file:
0000000000018c80 d call_single_queue
0000000000018cc0 d cfd_data
0000000000018d00 D softnet_data
0000000000018ec0 d rt_uncached_list
0000000000018f00 d rt6_uncached_list
0000000000018f18 D __per_cpu_end
0000000001000000 A phys_startup_64
ffffffff81000000 T _text
ffffffff81000000 T startup_64
ffffffff81000110 T secondary_startup_64
ffffffff810001a9 t verify_cpu
ffffffff810002a0 T start_cpu0
Checking within Wikipedia gives a good explanation. On the left is the address, in the middle is the symbol type (“d” or “D” is an initialized data section; “A” is Absolute; “T” is for a text or code section; and so on). Just looking at this system.map file is informative; you can see the addresses of all the symbols in the build, for example my rtl8169 Ethernet driver code symbols, and so on. Fascinating.
But, that’s not what SourcePoint wants for symbol information. Instead, it wants the vmlinux file. vmlinux is a statically linked executable file that contains the Linux kernel in one of the object file formats supported by Linux, which includes ELF, COFF and a.out. The vmlinux file is required for kernel debugging, symbol table generation or other operations, and is made bootable before being used as an operating system kernel by adding a multiboot header, bootsector and setup routines. Note that vmlinux is different from vmlinuz; the latter is compressed. SourcePoint needs the vmlinux file for extracting the symbols.
After some digging, I found the vmlinux file in this directory:
/poky/tmp/work/corei7-64-intel-common-poky/linux/linux-intel/4.9.126+gitAUTOINC+f9e90afd4e_38d41ee6fd-r0/linux-corei7-64-intel-common-standard-build
After that, it was a simple matter to identify the symbol file to SourcePoint:
And then, voila:
You can see at the top of the Code window that we are at the beginning of the procedure rtl8169_init_phy, and there are calls to __x86_indirect_thunk_rax, msleep, rtl_apply_firmware, and rtl_eri_write scattered throughout the window. But, it seems that not all of the symbols are decoded, and the source code is not displayed. Why not? That’s for the next episode!
