Mainstream Linux Kernel and PCIe Devices

Dear Mr Ziswiler,

At the beginning I would like to thank you for all your assistance and help with mainstream Linux kernel compilation and deployment. I have successfully deployed and run 4.6.2 Linux kernel. The good news is, that Linux is running and Ethernet is working. Whats worry me is poor PCIe devices list. I cannot see any mini-PCIe devices connected to the mini-PCIe slot. PCI devices list with Toradex Linux kernel 3.1.10:

    00:00.0 PCI bridge: NVIDIA Corporation Tegra3+ PCIe x2 Bridge (rev a1)
    01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
    02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
    02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
    02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
    04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
    06:00.0 PCI bridge: NVIDIA Corporation Tegra3+ PCIe x2 Bridge (rev a1)
    07:00.0 Ethernet controller: Intel Corporation Device 1531 (rev 03)

PCI devices list with Linux 4.6.2 kernel:

00:03.0 PCI bridge: NVIDIA Corporation Device 0e1d (rev a1)
01:00.0 Ethernet controller: Intel Corporation Device 1531 (rev 03)

Can you help me with enabling mini-PCIe slot on Apalis T30 and 4.6.2 kernel?

Dear Kamil

Due to an errata of the PCIe switch on our EvaluationBoard its reset line is connected to Apalis GPIO7 and needs to be toggled well in advance of releasing the regular RESET_MOCI signal. While this is taken care of by our regular BSP’s kernel here this is rather tricky to implement properly in mainline. The easiest would therefore probably be to do this in U-Boot as indicated here.

Depending on your device you might further have to limit it to GEN1 speed but let’s first see how you get along with it.

Cheers

Marcel Ziswiler

Platform Manager Embedded Linux

Dear Mr Ziswiler,

I have applied patch from above to the uboot, but still cant see any mini pci cards. U-boot cant load port 1, using 1 lanes:

U-Boot 2015.04-dirty (Jun 16 2016 - 10:05:34)

TEGRA30
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1, Tegra SD/MMC: 2
tegra-pcie: PCI regions:
tegra-pcie:   I/O: 0x2000000-0x2010000
tegra-pcie:   non-prefetchable memory: 0x20000000-0x30000000
tegra-pcie:   prefetchable memory: 0x30000000-0x40000000
tegra-pcie: 4x1, 1x2 configuration
tegra-pcie: probing port 1, using 1 lanes
tegra-pcie: link 1 down, retrying
tegra-pcie: link 1 down, retrying
tegra-pcie: link 1 down, retrying
tegra-pcie: link 1 down, ignoring
tegra-pcie: probing port 2, using 1 lanes
In:    serial
Out:   lcd
Err:   lcd
Model: Toradex Apalis T30 2GB V1.0E, Serial# 02650732
Net:   e1000: no NVM
e1000#0
Hit any key to stop autoboot:  0

Well, did you define APALIS_T30_PCIE_EVALBOARD_INIT in your configuration?

Dear Mr Ziswiler,

My bad. Didn`t notice, that in patch is +#undef APALIS_T30_PCIE_EVALBOARD_INIT, instead of #define. So, after update I can see more PCI bridges:

lspci
00:02.0 PCI bridge: NVIDIA Corporation Device 0e1d (rev a1)
00:03.0 PCI bridge: NVIDIA Corporation Device 0e1d (rev a1)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:02.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
02:03.0 PCI bridge: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch (rev ab)
06:00.0 Ethernet controller: Intel Corporation Device 1531 (rev 03)

but still cannot see card connected to mini pci slot: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

OK, well the uPD720202 probably does GEN2 which is not really supported by the T30 SoC plus our EvaluationBoard design probably is not GEN2 compliant neither. You would therefore have to make sure it exclusively uses GEN1 e.g. as follows.

BTW: Something else one could try is on a Ixora carrier board where its mini-PCIe slot is realised straight off the Apalis module without any PCIe switch or anything in between.

Dear Mr ziswiler,

Because I`m unexperienced in hacking kernels I would like to ask what is your suggestion for this modification? Is it better to do this on the kernel side like in your custom kernel by adding Macro DECLARE_PCI_FIXUP_EARLY and function quirk_apalis_plx_gen1 to the new arch/arm/mach-tegra/… file or is it better to do that in u-boot? I tried to do that in u-boot, by adding: pci_write_config_dword(0x8605, 0x98, 0x01);

diff --git a/board/toradex/apalis_t30/apalis_t30.c b/board/toradex/apalis_t30/apalis_t30.c
index 9f9731a..2ee612f 100644
--- a/board/toradex/apalis_t30/apalis_t30.c
+++ b/board/toradex/apalis_t30/apalis_t30.c
@@ -77,6 +77,7 @@ void pinmux_init(void)
                                   ARRAY_SIZE(apalis_t30_padctrl));
 }
 
+
 #ifdef CONFIG_PCI_TEGRA
 int tegra_pcie_board_init(void)
 {
@@ -120,6 +121,34 @@ int tegra_pcie_board_init(void)
                return err;
        }
 
+#ifdef APALIS_T30_PCIE_EVALBOARD_INIT
+#define PEX_PERST_N     GPIO_PS7 /* Apalis GPIO7 */
+#define RESET_MOCI_N    GPIO_PI4
+
+
+       /* Reset PLX PEX 8605 PCIe Switch plus PCIe devices on Apalis Evaluation
+          Board */
+       gpio_request(PEX_PERST_N, "PEX_PERST_N");
+       gpio_request(RESET_MOCI_N, "RESET_MOCI_N");
+       gpio_direction_output(PEX_PERST_N, 0);
+       gpio_direction_output(RESET_MOCI_N, 0);
+       /* Must be asserted for 100 ms after power and clocks are stable */
+       mdelay(100);
+       gpio_set_value(PEX_PERST_N, 1);
+       /* Err_5: PEX_REFCLK_OUTpx/nx Clock Outputs is not Guaranteed Until
+          900 us After PEX_PERST# De-assertion */
+       mdelay(1);
+       gpio_set_value(RESET_MOCI_N, 1);
+
+         /* The Apalis evaluation board needs to set the link speed to 2.5 GT/s (GEN1).
+        The default link speed setting is 5 GT/s (GEN2). 0x98 is the Link Control 2
+        PCIe Capability Register of the PEX8605 PCIe switch. The switch supports
+        link speed auto negotiation, but falsely sets the link speed to 5 GT/s. */
+        pci_write_config_dword(0x8605, 0x98, 0x01);
+        mdelay(50);
+
+#endif /* APALIS_T30_PCIE_EVALBOARD_INIT */
+
        return 0;
 }

but it looks, like is not working. There is an error during u-boot starting: pci_bus_to_hose() failed
Could you point me out on a proper way? Thank you in advance for the assistance.

I don’t know how to add any such to U-Boot plus our rather old U-Boot version has some issues with PCIe switch enumeration as far as I remember. The easiest would probably be to add it to the drivers/pci/quirks.c file similar as to the Tilera one as shown.

Dear mr Ziswiler,

Thank you for your help. I have added bellow code to the bottom of the file drivers/pci/quirks.c

static void quirk_apalis_plx_gen1(struct pci_dev *dev)
{
        dev_info(&dev->dev, "kamil: adding pci fixup for apalis eval board \n");
        pci_write_config_dword(dev, 0x98, 0x01);
        mdelay(50);
}
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PLX, 0x8605, quirk_apalis_plx_gen1);

and dmesg shows:

[    1.839561] pci 0000:01:00.0: kamil: adding pci fixup for apalis eval board 
[    1.907804] pci 0000:02:01.0: kamil: adding pci fixup for apalis eval board 
[    1.959105] pci 0000:02:02.0: kamil: adding pci fixup for apalis eval board 
[    2.010411] pci 0000:02:03.0: kamil: adding pci fixup for apalis eval board 

but still cannot see uPD720202 device. The question is if I did something wrong or problem lies elsewhere? In attachment full dmesg log and lspci -vv log link text. Any another ideas? Many thanks for your assistance.

Well, as mentioned before you could also try it on a Ixora carrier board.

Are you sure your mini-PCIe card is actually working? Have you tried it on any other board at all?

Dear mr Ziswiler,

Yes this card was perfeclty working on the same hardware with toradex kernel from Apalis_T30_LinuxImageV2.6. It`s just became invisible after kernel update to kernel 4.6.2.

Best regards,
Kamil