Mainline Kernel 4.20.6 Boot Error

Hello all , Mr @marcel.tx ,

I have been using mainline kernel 4.18.11 for both my Colibri and Aplais T30 and they were both fine.

I decided to upgrade the kernel to 4.20.6 for these two board and upgrade for Colibri T30 was done as expected and I could manage to boot the device.

However, for Apalis T30, I receive these error at U-Boot causing the device continuously restart…

U-Boot 2015.04 (Jun 28 2016 - 13:19:53)

TEGRA30
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1, Tegra SD/MMC: 2
tegra-pcie: PCI regions:
tegra-pcie:   I/O: 0x2000000-0x2010000
tegra-pcie:   non-prefetchable memory: 0x20000000-0x30000000
tegra-pcie:   prefetchable memory: 0x30000000-0x40000000
tegra-pcie: 4x1, 1x2 configuration
tegra-pcie: probing port 2, using 1 lanes
In:    serial
Out:   lcd
Err:   lcd
Model: Toradex Apalis T30 2GB V1.0E, Serial# 02680746
Net:   e1000: no NVM
e1000#0
Hit any key to stop autoboot:  0 
Loading kernel and rootfs from MMC/SD Card...
Reading DTB file...
reading tegra30-apalis-eval.dtb
36193 bytes read in 18 ms (1.9 MiB/s)
switch to partitions #0, OK
mmc2 is current device
reading /uImage
6694496 bytes read in 321 ms (19.9 MiB/s)
## Booting kernel from Legacy Image at 80408000 ...
   Image Name:   Linux-4.20.6-ELAR-Systems
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    6694432 Bytes = 6.4 MiB
   Load Address: 82008000
   Entry Point:  82008000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 82000000
   Booting using the fdt blob at 0x82000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 82000000, end 8200bd60

Starting kernel ...

undefined instruction
pc : [<8200800c>]          lr : [<ffa4ec10>]
reloc pc : [<026c800c>]    lr : [<8010ec10>]
sp : ff646820  ip : 0000000c     fp : ffa4ee00
r10: 00000000  r9 : ff64bee8     r8 : ff64f47c
r7 : 00000000  r6 : 82008000     r5 : ffaae4f8  r4 : 00000000
r3 : 00008d61  r2 : 82000000     r1 : 000011a1  r0 : 00000000
Flags: nZCv  IRQs off  FIQs off  Mode SVC_32
Resetting CPU ...

resetting ...

U-Boot SPL 2015.04 (Jun 28 2016 - 13:19:53)

Can you please advise if I am doing something wrong or there is some changes in mainline kernel causing this issue only on Apalis T30?!

Many thanks…

The module you are using also seems to be a rather ancient sample version. Have you tried any volume production version at all?

I really don’t think so. We run this stuff all the time but when using mainline I would really advice to use more recent boot loader and toolchain versions.

For me at least with BSP 2.7.4 as a base (e.g. with its U-Boot 2016.11) it works just fine. Log attached.

I have updated the u-boot to latest version. Same error…

Something is wrong… I don’t know why …

U-Boot 2016.11-ELAR-Systems (Feb 04 2019 - 19:54:49 +1100)

SoC: tegra30
Reset cause: POR
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1, Tegra SD/MMC: 2
In:    serial
Out:   vidconsole
Err:   vidconsole
Warning: Please change 'lcd' to 'vidconsole' in stdout/stderr environment vars
Model: Toradex Apalis T30 2GB V1.0E, Serial# 02680746
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
Loading kernel and rootfs from MMC/SD Card...
Reading DTB file...
reading tegra30-apalis-eval.dtb
36193 bytes read in 18 ms (1.9 MiB/s)
switch to partitions #0, OK
mmc2 is current device
reading /uImage
6471592 bytes read in 311 ms (19.8 MiB/s)
## Booting kernel from Legacy Image at 80408000 ...
   Image Name:   Linux-4.20.6-ELAR-Systems
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    6471528 Bytes = 6.2 MiB
   Load Address: 82008000
   Entry Point:  82008000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 82000000
   Booting using the fdt blob at 0x82000000
   Loading Kernel Image ... OK
   Using Device Tree in place at 82000000, end 8200bd60
   MAC address updated...

Starting kernel ...

undefined instruction
pc : [<8200801c>]          lr : [<ffb52c64>]
reloc pc : [<025c601c>]    lr : [<80110c64>]
sp : fd649700  ip : 000001a0     fp : 00000000
r10: 80408040  r9 : fd64fed8     r8 : ffb52e70
r7 : 00000000  r6 : 82008000     r5 : ffbc0c3c  r4 : 00000000
r3 : 00008d61  r2 : 82000000     r1 : 000011a1  r0 : 00000000
Flags: nZCv  IRQs off  FIQs off  Mode SVC_32
Resetting CPU ...

resetting ...

U-Boot SPL 2016.11-ELAR-Systems (Feb 04 2019 - 19:54:49)

Attached my uImage and device tree. A dump of the U-Boot environment may also be helpful.

Yours not resetting, however it stops as it tries to find rootfs…
But that should be another issue!!!

[    5.418946] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    5.436103] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    5.443284] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    5.449519] ALSA device list:
[    5.452148] cfg80211: failed to load regulatory.db
[    5.455200]   #0: tegra-hda at 0x70038000 irq 92
[    5.464872]   #1: Toradex Apalis T30
[    5.469357] Waiting for root device /dev/mmcblk0p2...
[  191.679439] random: crng init done

here is the bump of my u-boot environment:

    Apalis T30 # printenv
    arch=arm
    baudrate=115200
    board=apalis_t30
    board_name=apalis_t30
    boot_a_script=load ${devtype} ${devnum}:${bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
    boot_extlinux=sysboot ${devtype} ${devnum}:${bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf
    boot_file=uImage
    boot_prefixes=/ /boot/
    boot_script_dhcp=boot.scr.uimg
    boot_scripts=boot.scr.uimg boot.scr
    boot_targets=mmc1 mmc0 usb0 pxe dhcp 
    bootcmd=run sdboot; echo; echo sdboot faild; \
    run emmcboot; echo; echo emmcboot failed;\
    run usbboot; echo; echo usbboot failed;\
    run nfsboot; echo; echo nfsboot failed
    bootcmd_dhcp=usb start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi
    bootcmd_mmc0=setenv devnum 0; run mmc_boot
    bootcmd_mmc1=setenv devnum 1; run mmc_boot
    bootcmd_pxe=usb start; dhcp; if pxe get; then pxe boot; fi
    bootcmd_usb0=setenv devnum 0; run usb_boot
    bootdelay=1
    console=ttyS0
    cpu=armv7
    defargs=core_edp_mv=1300 usb_high_speed=1
    dfu_alt_info=apalis_t30.img raw 0x0 0x500 mmcpart 1; boot part 0 1 mmcpart 0; rootfs part 0 2 mmcpart 0; uImage fat 0 1 mmcpart 0; tegra30-apalis-eval.dtb fat 0 1 mmcpart 0
    distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
    dtbparam=0x82000000
    emmcargs=ip=off root=/dev/mmcblk2p2 rw,noatime rootfstype=ext3 rootwait
    emmcboot=run setup; setenv bootargs ${defargs} ${emmcargs} ${setupargs} ${vidargs}; echo Booting from internal eMMC chip...; run emmcdtbload; load mmc 0:1 ${kernel_addr_r} ${boot_file} && bootm ${kernel_addr_r} - ${dtbparam}
    emmcdtbload=setenv dtbparam; load mmc 0:1 ${fdt_addr_r} ${soc}-apalis-${fdt_board}.dtb && setenv dtbparam ${fdt_addr_r}
    ethact=e1000#0
    ethaddr=00:14:2d:28:e7:aa
    fdt_addr_r=0x82000000
    fdt_board=eval
    fdt_high=ffffffff
    fdtcontroladdr=fd649b58
    filesize=50dd78
    initrd_high=ffffffff
    ipaddr=192.168.10.2
    kernel_addr_r=0x81000000
    loadaddr=0x80408000
    memargs=vmalloc=256M mem=2035M@2048M fbmem=12M@4083M
    mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi
    netmask=255.255.255.0
    nfsargs=ip=:::::eth0:on root=/dev/nfs rw netdevwait
    nfsboot=run setup; setenv bootargs ${defargs} ${nfsargs} ${setupargs} ${vidargs}; echo Booting via DHCP/TFTP/NFS...; run nfsdtbload; dhcp ${kernel_addr_r} && bootm ${kernel_addr_r} - ${dtbparam}
    nfsdtbload=setenv dtbparam; tftp ${fdt_addr_r} ${soc}-apalis-${fdt_board}.dtb && setenv dtbparam ${fdt_addr_r}
    pxefile_addr_r=0x90100000
    ramdisk_addr_r=0x82100000
    scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done
    scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${bootpart} bootfstype; then run scan_dev_for_boot; fi; done
    scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
    scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
    scriptaddr=0x90000000
    sdargs=ip=off root=/dev/mmcblk0p2 rw,noatime rootfstype=ext3 rootwait
    sdboot=run setup; setenv bootargs ${defargs} ${sdargs} ${setupargs} ${vidargs}; echo Loading kernel and rootfs from MMC/SD Card...; run sddtbload; mmc dev 2 && load mmc 2:1 ${loadaddr} /uImage ; bootm ${loadaddr} - ${dtbparam}
    sddtbload=echo Reading DTB file...; setenv dtbparam; load mmc 2:1 ${fdt_addr_r} ${soc}-apalis-${fdt_board}.dtb && setenv dtbparam ${fdt_addr_r}
    serial#=02680746
    serverip=192.168.10.1
    setethupdate=if env exists ethaddr; then; else setenv ethaddr 00:14:2d:00:00:00; fi; tftpboot ${kernel_addr_r} flash_eth.img
    setsdupdate=setenv interface mmc; setenv drive 1; mmc rescan; load ${interface} ${drive}:1 ${kernel_addr_r} flash_blk.img || setenv drive 2; mmc rescan; load ${interface} ${drive}:1 ${kernel_addr_r} flash_blk.img
    setup=setenv setupargs igb_mac=${ethaddr} consoleblank=0 no_console_suspend=1 console=tty1 console=${console},${baudrate}n8 debug_uartport=lsport,0 ${memargs}
    setupargs=igb_mac=00:14:2d:28:e7:aa consoleblank=0 no_console_suspend=1 console=tty1 console=ttyS0,115200n8 debug_uartport=lsport,0 vmalloc=256M mem=2035M@2048M fbmem=12M@4083M
    setupdate=run setsdupdate || run setusbupdate || run setethupdate; source ${kernel_addr_r}
    setusbupdate=usb start && setenv interface usb; setenv drive 0; load ${interface} ${drive}:1 ${kernel_addr_r} flash_blk.img
    soc=tegra30
    stderr=serial,lcd
    stdin=serial
    stdout=serial,lcd
    usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi
    usbargs=ip=off root=/dev/sda2 rw,noatime rootfstype=ext3 rootwait
    usbboot=run setup; setenv bootargs ${defargs} ${setupargs} ${usbargs} ${vidargs}; echo Booting from USB stick...; usb start && run usbdtbload; load usb 0:1 ${kernel_addr_r} ${boot_file} && bootm ${kernel_addr_r} - ${dtbparam}
    usbdtbload=setenv dtbparam; load usb 0:1 ${fdt_addr_r} ${soc}-apalis-${fdt_board}.dtb && setenv dtbparam ${fdt_addr_r}
    vendor=toradex
    ver=U-Boot 2016.11-ELAR-Systems (Feb 04 2019 - 19:54:49 +1100)
    vidargs=mxc_hdmi.only_cea=1 video=mxcfb0:dev=hdmi,1920x1080M@60,if=RGB24 video=mxcfb1:off video=mxcfb2:off video=mxcfb3:off fbmem=32M

I am downloading the maniline kernel and compile it to see how it goes…

Yes, I am exactly using the same command . but base on your article, I am using different address:
Article You Explained

make -j4 uImage LOADADDR=0x82008000
make -j4 tegra30-apalis-eval.dtb
make -j4  modules

and of course modules…

Well, you may anyway want to migrate to using zImage rather than legacy uImage like we did in more recent BSP versions.

Concerning the root file system: Mainline just uses a different SDHCI controller instance ordering. Unless you patch that (e.g. as we did in the 4.14 used for the Toradex Easy Installer) you will need to use mmcblk2p2 for the on-module eMMC.

Hm, I don’t remember where the 0x82008000 there came from but usually for T30 it is really 0x80008000. At least our downstream BSPs always used that as long we still used legacy uImage (e.g. BSP 2.7 stable):

[user@host apalis-t30_bin]$ mkimage -l uImage--3.1.10+gitr0+7753763827-2.7.4-apalis-t30-20171005023218.bin
Image Name:   Linux-3.1.10-2.7.4+g7753763
Created:      Thu Oct  5 05:08:01 2017
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    4243776 Bytes = 4144.31 KiB = 4.05 MiB
Load Address: 80008000
Entry Point:  80008000

I’m not aware of any such issue. However, I would advice to use a more recent U-Boot version e.g. at least the downstream 2016.11 from our official BSPs and try again. Also, make sure to use a somewhat recent toolchain for mainline like e.g. the 8.2 from Arm I’m using.

@marcel.tx
No I do not have production version available.

By the way, comparing the “tegra30-apalis-eval.dts” in both 4.18.11 and 4.20.6 showing me too much changes!
I am not expert in device tree but some of device compatible are gone!
Do you think that might cause any issue?
So how come the Colibri T30 works fine and not Aplais T30.
I’m using the same toolchain and same environment to compile the kernel.
As you know only the kernel load address is different!!!

@marcel.tx
I used the compiler you advised (gcc-arm-8.2-2019.01-x86_64-arm-linux-gnueabihf.tar.xz). Same issue…
Haven’t tried the latest u-boot… I’ll try and see if that is because of U-Boot or not!

I have downloaded the kernel source again and compiled from scratch with just tegra_defconfig!

Same problem… I am getting confused.
Why mine does not work and yours does?

My build environment is “Ubuntu 18.04.1 LTS” on VMWare. As I mentioned , had no issue for Colibri T30 but Aplais T30 causes this issue!!!

Are you still using legacy uImage? What load address are you using upon generating? Usually, T30 uses 0x80008000 (e.g. by doing make uImage LOADADDR=0x80008000).

WOW!!!..
changing the load address from 0x8200800 to 0x80008000 fixed the issue!

Wondering why!!!

Thanks a lot dear @marcel.tx .
At least it loads but I still have issue finding the /dev/mmcblk0p2 when booting from SD…
Hope I can resolve this as well!

@marcel.tx

Is that easy to migrate to zImage?
I havn’t done that before . What changes do I need to apply? (kernel/u-boot)

can you give me a hint please?

Meanwhile, the size of my kernel with other options is about 9MB now and u-boot cannot handle it.It says I have to increase “CONFIG_SYS_BOOTM_LEN

But the file called “bootm.c” holding that value are available to many places. Do you know where is the right place to increase it?

changed it in ./common/bootm.c and seems that working!

Finlay I get it booted |O|

I appreciate if you advise how I can use zImage instead of uImage…

Just use e.g. our latest BSP 2.8b5 which will use zImage by default. If you haven’t used the Toradex Easy Installer just give that one a try to install.

@marcel.tx

Sure I will try that… but for zImage kernel, should I use the same command to compile it?

make -j4 zImage LOADADDR=0x80008000

Just one more thing.
I realized when booting from SD, it tries looking for rootfs on /dev/mmcblk0p2 but is not finding the rootfs and stuck.

It has to be /dev/mmcblk0p2 as I used /dev/mmcblk0p1 to load kernel and DTB file. If I tmporary tell u-boot to load from eMMC partition 2 it works but cannot find the rootfs on SD!

It was same on your sample uImage and DTB file which you had sent me.
It stuck finding the rootfs partition after loading the kernel and passed that part…

Any idea?
(I am not sure if its something about device tree!)

even when we boot from eMMC and insert SD card, it gives this error:

mmc0: error -110 whilst initialising SD card