High-end, mission-critical systems often use a plethora of diagnostic routines for power-on self test (POST), data logging, operational measurements, Built-In Self Test (BIST), and platform audits. Embedded x86 JTAG run-control adds out-of-band services to this mix. This article describes one such use case for Power-On Self Test (POST) of PCI Express ports.
Run-control technology operates at the very lowest level of silicon instrumentation, thereby bypassing any requirements for boot loader, kernel, driver or OS services. This method of access is often termed “out-of-band”, because it has no dependency upon higher level functions. This makes run-control a very powerful utility for performing certain operations very quickly and efficiently. Within Intel devices, run-control is known as In-Target Probe, or ITP for short. ITP is often used in a benchtop environment to access a platform externally via its XDP connector, or, on some more modern platforms, over a USB cable. ITP is used for platform debug, test and hardware validation functions: Intel has written powerful applications, such as the CScripts, just for these purposes.
Consider an arbitrary test routine that is meant to be run prior to a system booting fully to the operating system; perhaps in a manufacturing environment. This could be done, for example, by writing a UEFI application in ‘C’ and then booting to the UEFI shell and running it. But this involves a lot of complexity and system dependencies (for example, the SPI being fully operational on the board) and time (booting to the UEFI shell will add time and slow down the beat rate of production testing).
For POST, this becomes even more critical. When a system boots up, it is optimal to have very low-level functions executed quickly to validate the hardware sanity of the platform before any firmware or software gets loaded. One example of this is in the hardware validation of PCI Express (PCIe) channels on the board. Serial protocols like PCIe provide very high speed and throughput, and this results in a high level of complexity within the physical layer. With this complexity comes a high probability of bugs in the upstream and downstream devices, which might only expose themselves intermittently over extended periods of testing time. There might also be marginalities in the board design that affect link quality. Ferreting these bugs and issues out is often done as part of the hardware qualification process, and repeated link retrains is one mechanism often used.
Physical layer link initialization and training is a very complex process. In PCIe devices, this process undertakes many important tasks, such as link width negotiation, link data rate negotiation, bit lock per lane, symbol lock/block alignment per lane, decision-feedback equalization (DFE), etc. etc. All of these functions are accomplished by the Link Training & Status State Machine (LTSSM), which observes the stimulus from the remote link partner as well as the current state of the link, and responds accordingly:
One approach to validating the sanity of the PCIe channels on the platform is simply to have the system boot up and go through the training process, and look for errors. This is a good “quick and dirty” approach, but may miss some intermittent bit errors. And, depending on the frequency and nature of bit errors, the system may later display any of the following symptoms:
- Lowered throughput/performance
- Errors are logged
- Drivers do not load properly
- Speed degrades
- Link width degrades
- Intermittent dropouts
- Surprise Link Down (“SLD”)
- Device Not Found (“DNF”)
- System crash/hang (“blue screen”)
The higher the bit error rate, the more severe the symptoms on a design. A drop in system performance may be just an annoyance or even invisible to an end user; link width degradations may cause a level of customer dissatisfaction; but SLDs, DNFs and crashes are partial or total system outages, which can have a major negative impact.
Engineers often check on the sanity of the PCIe network as part of the board bring-up and hardware validation process. Benchtop ITP exercises the CScripts needed to perform some of the following functions:
- Basic Link Retrains
- Link Enables/Disables
- TxEQ Retrains
- Speed Changes (i.e. Gen1 -> Gen2 -> Gen3)
- L1 ASPM (Active State Power Management)
Even better, run-control for PCIe validation can be used in POST if the board BMC has the requisite connections to the CPU JTAG ports, as in the Microsoft Project Olympus servers:
If the run-control library is resident down within the BMC, as in ASSET’s SED solution, accessing PCI config registers can uncover problems at the lowest level of the silicon (at the RTL level), and also identify problems with reference clock quality, signal voltage level, platform margins, firmware and driver issues, and so on. This can be done as part of POST to quickly identify what ports or devices may be marginal.
Implementing this using ASSET’s SED API is fairly straightforward, with assistance for example from an Intel External Design Specification, by looking at the related register definitions. One register relevant to this discussion is the LNKCON PCI Express Link Control register. Writing to a bit within this register initiates link retraining on a given port. A code snippet needed to run retrains “m_loops” times is below:
void lt_loop(int mHandle, uint32_t bus, uint32_t dev, uint32_t fun)
{
//major test; loop on link training.
int i;
int j;
uint32_t lnksts;
uint32_t lnkcon;
uint32_t startWidth;
uint32_t startSpeed;
uint32_t currentWidth;
uint32_t currentSpeed;
//read LNKSTS and get link active, speed, width
lnksts = readRegister(mHandle, bus, dev, fun, 0xA2, AI_bw16);
//Check for all ones; indicates we are not reading anything valid.
if (lnksts == 0xFFFF)
{
printf("Link status is all ones; aborting test.\n");
return;
}
//check to make sure the link is active before we begin. May be inactive due to not connected on target.
if ((lnksts & 0x2000) == 0)
{
printf("Link is not active on startup, aborting test.\n");
return;
}
startWidth = (lnksts >> 4) & 0x3F;
startSpeed = lnksts & 0xF;
for (i=0; i<m_loops; i++)
{
//Write (read first) the LNKCON register to cause the link to retrain
lnkcon = readRegister(mHandle, bus, dev, fun, 0xA0, AI_bw16);
lnkcon = lnkcon | 0x20; //set the retrain_link bit
writeRegister(mHandle, bus, dev, fun, 0xA0, lnkcon, AI_bw16);
//loop checking the link status until the link has finished retraining (or we give up)
for (j=0; j<100; j++) //max number of times to try
{
lnksts = readRegister(mHandle, bus, dev, fun, 0xA2, AI_bw16);
if ((lnksts & 0x800) == 0)
{
printf("Exit link retrain status loop after: %d tries.\n", j);
break; //Normal exit for this loop after 1 or 2 tries
}
}
if ((lnksts & 0x800) != 0)
{
printf("Link failed to finish re-training.\n");
break;
}
if ((lnksts & 0x2000) == 0)
{
printf("Link is no longer active after retrain.\n");
break;
}
currentWidth = (lnksts >> 4) & 0x3F;
if (currentWidth != startWidth)
{
printf("Width: %d after retrain does not equal start width: %d\n", currentWidth, startWidth);
}
currentSpeed = lnksts & 0xF;
if (currentSpeed != startSpeed)
{
printf("Speed: %d after retrain does not equal start speed: %d\n", currentSpeed, startSpeed);
}
}
}
This is pretty simple. The main work is done in the writeRegister and readRegister() functions. The latter simply reads the contents at the PCI config register specified at the given bus, device, function and offset. Since this uses ITP, it is super fast – much faster than using a BIOS or OS-based utility.
Having this capability within POST means that the solution can deploy on as many systems as needed, and be invoked anywhere and anytime. There is no need to have a remote workstation initiate the routine – although that is always an option:
Having JTAG and run-control libraries available within the BMC opens up a new world of out-of-band capabilities. One can conceive of any number of the CScripts being ported to run down on the BMC. POST is just one example of an application that is best controlled out-of-band by a service processor. Having the BMC provide system monitoring, logging, and operational measurement reporting are yet some examples. System audits that run periodically (like at 2:00am, when the system is under light load) is another. And, finally, BIST, built perhaps using boundary scan (with JTAG as a foundation), can be used to identify root-cause of intermittent failures that could not be triaged in any other way. I’ll write more on this topic later.
In the meantime, to learn more about SED, check out our eBook SED Technical Overview (note: requires registration).
