Debate rages in the embedded software development community about the cost/benefit of using printf statements to debug code. With today’s silicon embedded intellectual property (IP), there is no reason why instrumentation can’t be used in conjunction with trace to provide immediate real-time insight into defect-laced code.
So before I start weighing in on the topic, let’s start with some pretty reasonable definitions to establish a baseline.
From Wikipedia, we have the following definition for printf.
“Printf format string (of which "printf" stands for "print formatted") refers to a control parameter used by a class of functions in the string-processing libraries of various programming languages. The format string is written in a simple template language, and specifies a method for rendering an arbitrary number of varied data type parameters into a string. This string is then by default printed on the standard output stream, but variants exist that perform other tasks with the result, such as returning it as the value of the function. Characters in the format string are usually copied literally into the function's output, as is usual for templates, with the other parameters being rendered into the resulting text in place of certain placeholders – points marked by format specifiers, which are typically introduced by a % character, though syntax varies. The format string itself is very often a ‘string_literal’, which allows static analysis of the function call. However, it can also be the value of a variable, which allows for dynamic formatting but also a security vulnerability known as an ‘uncontrolled_format_string’ exploit.”
That is quite a definition! Within that definition we can see that handling different output types and formatting of the data is part of what is expected of the library. Also, the definition indicates that the result of a printf is an output to a hardware device. So, the printf function requires target memory to hold the library and a device driver to control the serial stream to the hardware. Depending upon the complexity of the embedded system, this may be trivial or a lot of work. Before starting this blog, I went to several sources to see what is being discussed about printf and found several threads dealing with debugging a printf function. Some examples involved the printf throwing exceptions and others that the data emitted was not in the expected format. All this was the result of an attempt to get the printf function to work on a particular embedded target, essentially creating a debug tool specific to the target platform.
Looking again at Wikipedia: “Debugging is a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware, thus making it behave as expected. Debugging tends to be harder when various subsystems are tightly coupled, as changes in one may cause bugs to emerge in another.”
This methodical process often breaks down rather rapidly if you rely solely on printf. It becomes more of a shotgun approach to debugging. Add a print statement here and there, and then: “Oh, what about that ‘if statement’! I need one in the code loop.” And so it goes. Printf debugging on multi-core systems provides for a more difficult scenario as the context of the problem could be core-specific and the output of the printf without any means within the printf to determine what core.
Dealing with printf statements has some additional baggage. I have already mention memory consumption and hardware driver development. Printf(s) are not much value in dealing with memory allocations or interrupts. When dealing with real-time sensitive code, adding a printf is almost certainly not an option. Other baggage is also the development time lost in the compile/link workflow as the printf statements are included.
In an article by Andrew Binstock in Dr. Dobbs titled “Embedded print Statements != Debugging”, Andrew makes some very relevant points about location, time cost, complexity, Heisenberg effect and clean up. The comments about location and time costs were spot on. With the complexity of embedded systems today, the long pole in the development cycle is clearly software development. Spending time debugging a function whose sole purpose is debugging might not be the best use of our limited time resource. And the rub is: you need to know where in your code to insert the printf to begin debugging and by the time you have backed up far enough in the code to get a picture of the real execution of the code, you have printf statements littering your code and this can alter the execution performance of the system you’re trying to deliver. There has to be a better way! And there is.
By combining the new trace IP in the silicon with tool-hosted printf, a developer can have the best of both worlds. Using a debugger to trace code execution is like a scalpel in the hands of an experienced surgeon. By tracing the code execution and then examining both code and data trace output, the developer can quickly dissect the problem, expose the bug and remove it. The problem with trace is that often you only have one event that is visible from the system viewpoint to begin tracing. Rarely is that observable event the actual root cause. If the trace depth were unlimited, you could have the trace stop at that event and just walk backwards through the trace and find the problem. The problem with this methodology is a single loop will often consume the entire trace buffer even if it were unlimited. But who has the time or patience to examine gigabytes of data? So, you have to get creative about where to trace. This is where tool-hosted printf comes in.
The section titled “STM: A new programming paradigm” in Larry Traylor’s eBook, “Debug and Trace using ARM System Trace Macrocell (STM)” describes how the silicon provides the means by which tool-hosted printf can be implemented. Tool-hosted printf does not rely on printf functionality that is totally CPU execution-dependent the way that traditional printf functionality does. Instead, tool-hosted printf also uses trace IP to provide real-time debug messaging within the trace stream.
With tool-hosted printf we can instrument only a few statements in the code and use the output to trigger a trace. Then use the trace capture to examine the larger context of the event. Due to the nature of how tool-hosted printf is implemented and the data stream containing the string, we also get a timestamp, which can correlate when a print occurred and which processor in a multi-core system initiated it. Stay tuned for the next article as we delve deeper into tool-hosted printf and learn how, with trace, you can master the dissecting of defect-laden code.
In the meantime, if you’d like to learn a bit more about the value of trace, check out a recent addition to our eResources. It’s a new eBook called, coincidentally, “Hardware-Assisted Debug and Trace within the Silicon.”
