After last week’s distraction, I do a deep dive into the debug & trace tools that are built into the Linux SDK.
In Episode 33, I built a Yocto image for the MinnowBoard with source and symbols, and started working with the “perf” command, something of a superset of debug capabilities for Linux.
I took a side trip in Episode 34 to load a Yocto image into VirtualBox. This was a lot of fun, but there were a number of incompatibilities running the Yocto build inside the VM, as I had somewhat expected.
So, I decided to return to working my way through the Yocto Mega-Manual, and looking at Linux debug and trace on the MinnowBoard. I used ssh to log into the Minnow from my PC’s Ubuntu VM. Last week, I had left off using the “perf record” and “perf report” commands to watch what was happening when I did a “wget” of a remote file onto the Minnow; here’s a sample of the terminal session on the right of the perf call tree results of a wget command:
The functions prefaced by a [k] represent kernel functions, and those with [.] are userspace. But, I see from the Yocto Mega-Manual that wget is actually implemented as an applet in busybox (which is itself a single executable that contains a bunch of Linux tools), so while the process name is “wget”, the executable we're actually interested in is busybox. Thanks to Wikipedia for explaining that to me! And you can see from the call tree above that there is just a hex address, and not symbols, in the busybox.nosuid call.
If I want to see the symbols for the code within busybox, I learned that I need to take a few extra steps. Specifically, I need to install the debuginfo for the busybox package into my live Linux build.
The first step is to generate that debug info for the packages in the image, by adding “dbg-pkgs” to EXTRA_IMAGE_FEATURES in local.conf. This had to be added: EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs".
Additionally, in order to generate the type of debuginfo that perf understands, we also need to add the following to local.conf: PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'.
Then, it was time to recreate the build, using the bitbake core-image-sato-sdk command as usual.
Unfortunately, the Yocto build errored out this time, and restarted Ubuntu. I’ve noticed that this happens intermittently but rarely, and usually when I re-run the bitbake it just completes successfully. Unfortunately, that was not true this time:
So, now what?
I did notice that the rpm archive was in fact in the right place: ~/poky/build/tmp/deploy/rpm/corei7_64. So I thought, what the heck, let’s go ahead and install the debuginfo for busybox. What’s the worst that can happen? In this case I had to transfer the rpm file by typing from my build machine:
scp ~/poky/build/tmp/deploy/rpm/corei7_64/busybox-dbg-1.24.1-r0.corei7_64.rpm root@192.168.254.27:
followed by typing on the target via the ssh session, the command necessary to install the debug rpm:
rpm -i busybox-dbg-1.24.1-r0.corei7_64.rpm
The latter took about ten seconds to execute. I was beginning to worry.
And then, success! Running “perf report” again (note that you don’t have to run “perf record” again, because the data is preserved in a “perf.data” file) shows:
Pretty cool, huh? That busybox call actually went to the function “progress_meter”.
But then, expanding that still leaves a display of hex (0xebb39) using up 3.78% of the CPU time.
What’s that about?
I’ll have to dig into that for the next episode.
I still haven’t seen anything as graphically intuitive as SourcePoint’s call graphs. But, time will tell.
