In the last article on this topic, we did a dive into the main routine of the lt_loop JTAG-based On-Target Diagnostic, seeing the overall flow of the program. In this article, we’ll look at the routine that does the heavy lifting for retraining the PCI Express link and checking for errors.
For background, please read Coding to the SED API: Part 1 and Part 2. The source code and initial code walkthroughs are available there. The example program we’re working on exercises the Link Training & Status State Machine (LTSSM) on PCI Express ports on x86 machines. For more information on the theory, see Built-In Self Test (BIST) for PCI Express using Embedded Run-Control.
Looking at the source code of the entire program as found in Part 1, we can see that much of the “heavy lifting” happens in the lt_loop(int mHandle, uint32_t bus, uint32_t device, uint32_t function) routine. Note that it makes use of some of the key register definitions as defined early in the program:
#define REG_LNKSTS 0x52 #define REG_DMI_LNKSTS 0x1B2 #define REG_LNKCON 0x50 #define REG_DMI_LNKCON 0x1B0 #define REG_LNKCON2 0x70 #define REG_DMI_LNKCON2 0x1C0 #define REG_LNKCAP 0x4C #define REG_LTRCON 0x11C #define REG_CORERRSTS 0x110 //"ERRCORSTS" correctable error status #define REG_UNCERRSTS 0x104 //"ERRUNCSTS" uncorrectable error status #define REG_SECBUS 0x19 #define REG_AERCAPHDR 0x100 //Advanced Error Reporting Extended Capability Header #define REG_CPUBUSNO 0x104 #define REG_CPUBUSNO1 0x108 #define REG_CPUBUSNO2 0x10A #define REG_CPUBUSNO_VALID 0x110 #define REG_SOCKET_BUS_RANGE 0x114
The PCIe register definitions such as REG_LNKSTS are found in the Intel EDS; but, in lieu of that, a lot of the underlying methodology is common to all x86 platforms and in the public domain, such as here: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/atom-processor-s1200-datasheet-vol-2.pdf. (Note to those who do have access to the EDS: LNKSTS has been renamed to LINKSTS, but we keep the old legacy LNKSTS here. Sorry for any confusion!). LNKSTS is at offset 0x52 and contains useful information such as whether link training is complete or in progress, what the link width is, whether it’s up or not, and the link speed. We see later that we read its contents with the call at line 621:
lnksts = readRegisterPeci(mHandle, bus, dev, fun, reg_lnksts, AI_bw16);
and then we do some compares, status checking, and place the link width and speed in our local variables:
if (lnksts == 0xFFFF)
{
printf("Link status is all ones; aborting test.\n");
return;
}
//check to make sure the link is active before we begin. May be inactive due to not connected on target.
if ((lnksts & 0x2000) == 0)
{
printf("Link is not active on startup, aborting test.\n");
return;
}
startWidth = (lnksts >> 4) & 0x3F;
startSpeed = lnksts & 0xF;
Note at the top that we retrieve lnksts using the function readRegisterPeci as opposed to a standard JTAG/MMIO read. I’ll explain further in a future article, but this routine, despite its name, does not use PECI as an access mechanism; rather, it uses the sideband IOSF access mechanism that I mentioned in the last article, as a means to not halt the target. So, despite the name, it’s still JTAG. The end result is the same, though: we retrieve all 16 bits of LNKSTS, do the “and” with bit 13 to see if the Data Link Control and Management State Machine is in the DL_Active state, and if so we get the Negotiated Link Width by shifting right four bits and ANDing with x3F, thereby isolating the bits 9:4 which contains the width. We also assign startSpeed to the last four bits (or first four bits, depending on how endian you are). 😊
Next we do the same thing with LNKCON (in the EDS, this has been renamed to LINKCTL):
lnkcon = readRegisterPeci(mHandle, bus, dev, fun, reg_lnkcon, AI_bw16); lnkcon = lnkcon | 0x20; //set the retrain_link bit
Note that we set bit 5 of LNKCON, via the OR with x20, to initiate the link retrain. We then later use run-control (not IOSF) to fire off the retrain:
//Write the LNKCON register to cause the link to retrain writeRegisterMem(mHandle, membus, dev, fun, reg_lnkcon, AI_bw16, lnkcon);
We then iteratively loop for up to 100 times (normally it only takes one or two loops), checking the link retrain bit, to see if the retrain succeeded:
//loop checking the link status until the link has finished retraining (or we give up)
for (j=0; j<100; j++) //max number of times to try
{
lnksts = readRegisterPeci(mHandle, bus, dev, fun, reg_lnksts, AI_bw16);
if ((lnksts & 0x800) == 0)
{
break; //Normal exit for this loop after 1 or 2 tries
}
}
if ((lnksts & 0x800) != 0)
{
printf("Link failed to finish re-training.\n");
noTrains++;
break;
}
if ((lnksts & 0x2000) == 0)
{
printf("Link is no longer active after retrain.\n");
noTrains++;
break;
}
And afterwards we check that the Link Training bit (bit 11 of LINKSTS, the result of the mask with 0x800) should be zero, and that we’re in the DL_Active state (via the mask with 0x2000, bit 13). Otherwise, we bail.
Finally, as an example, we collect the number of correctable and uncorrectable errors while we’re in the loop:
//Check for errors that occured during retrain
errs = readRegisterPeci(mHandle, bus, dev, fun, REG_UNCERRSTS, AI_bw32);
if (errs != 0)
{
uncorErrors++;
printf("Uncorrectable error: 0x%08X on port: %d loop: %d\n", errs, m_port, i);
}
//Check for errors that occured during retrain
errs = readRegisterPeci(mHandle, bus, dev, fun, REG_CORERRSTS, AI_bw32);
if (errs != 0)
{
corErrors++;
printf("Correctable error: 0x%08X on port: %d loop: %d\n", errs, m_port, i);
}
Pretty simple so far, right? If you’re still with me, you now understand the fundamentals of working with x86 architecture using the SED library.
There’ll be many more examples to come. Stay tuned!
