eMMC formatting and/or not mounting after running SaveReg

JinShil · June 20, 2016, 6:37am

We are seeing some very strange behavior from the eMMC after making changes to the registry and running SaveReg.

This is what the debug console displays normally:

Toradex Bootloader 1.0 for Colibri Built Mar 23 2016
Board: Colibri iMX6 DualLite 512MB
CPU is running at 984000MHz.
Using eMMC boot partition (size: 4096 sectors).
Loading EBoot configuration...
16 sectors of configuration data loaded from boot.
Initializing L2 Cache.

Press [SPACE] to enter Bootloader Menu

Initiating image launch in 0 seconds.
System ready!
Preparing for download...
Loading compressed image...
Reading image from sector 40962.
Extracting 67554676 bytes from compressed image of 38630909 bytes
................................................................

Done.
Launching image at 10200000.
IMAGE(CE8): 1.0
Colibri iMX6 DualLite 512MB
SMP support enabled
Detected 2 CPUs
Waiting for 2 CPUs
 CPU0 is started
 CPU1 is started
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400003 TID:0048004E Port is used for OS debug!
PID:00400003 TID:0048004E Waiting for flash disk \FlashDisk for 60 seconds
PID:00400003 TID:0048004E Folder mounted
PID:00400003 TID:0048004E GALCORE 4.6.9(9754) (Sep 19 2014 12:04:16)
PID:00400002 TID:0048004E Major GPU: SysIntr=32 MemBases=0x130000 MMU Version=0
PID:00400002 TID:0048004E 2D GPU: SysIntr=33 MemBases=0x134000 MMU Version=0
PID:00400002 TID:0048004E VG GPU: SysIntr=34 MemBases=0x2204000
PID:00400002 TID:0048004E Video memory: BaseAddress=0x0 PhysBase=0x254b6000 size=0x3000000 physSize=0x0
PID:00400003 TID:054D0006 Maximum Allowed Error 5:
PID:00400003 TID:00560002 SoftRTC disabled
PID:04EE000A TID:04F8000A RTC Time restored (20.06.2016 05:59:29)
PID:04EE000B TID:04F8000A RTCSync: Completed.
PID:054F000B TID:0551000A Desktop Window QI, Unknown interfacePID:054F000B TID:0551000A Desktop Window QI, Unkno
wn interfacePID:054F000A TID:05E80032 Explorer(V2.0) taskbar thread started.

However, after a few reboots, we see the following:

Toradex Bootloader 1.0 for Colibri Built Mar 23 2016
Board: Colibri iMX6 DualLite 512MB
CPU is running at 984000MHz.
Using eMMC boot partition (size: 4096 sectors).
Loading EBoot configuration...
16 sectors of configuration data loaded from boot.
Initializing L2 Cache.

Press [SPACE] to enter Bootloader Menu

Initiating image launch in 0 seconds.
System ready!
Preparing for download...
Loading compressed image...
Reading image from sector 40962.
Extracting 67554676 bytes from compressed image of 38630909 bytes
................................................................

Done.
Launching image at 10200000.
IMAGE(CE8): 1.0
Colibri iMX6 DualLite 512MB
SMP support enabled
Detected 2 CPUs
Waiting for 2 CPUs
 CPU0 is started
 CPU1 is started
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400003 TID:00560002 Registry - loading version 1 from 16386.
PID:00400002 TID:0048004E Port is used for OS debug!
PID:00400002 TID:029E0002 FATFS!FormatVolume: Formatting volume. Flags = 0x16.
PID:00400002 TID:0048004E Waiting for flash disk \FlashDisk for 60 seconds
PID:00400002 TID:0048004E Folder mounted
PID:00400002 TID:0048004E GALCORE 4.6.9(9754) (Sep 19 2014 12:04:16)
PID:00400002 TID:0048004E Major GPU: SysIntr=32 MemBases=0x130000 MMU Version=0
PID:00400002 TID:0048004E 2D GPU: SysIntr=33 MemBases=0x134000 MMU Version=0
PID:00400002 TID:0048004E VG GPU: SysIntr=34 MemBases=0x2204000
PID:00400002 TID:0048004E Video memory: BaseAddress=0x0 PhysBase=0x258b0000 size=0x3000000 physSize=0x0
PID:00400003 TID:051A0006 Maximum Allowed Error 5:
PID:00400003 TID:00560002 SoftRTC disabled
PID:04BE000A TID:04C3000A RTC Time restored (20.06.2016 06:00:16)
PID:04BE000B TID:04C3000A RTCSync: Completed.
PID:04E8000B TID:04E7000A Desktop Window QI, Unknown interfacePID:04E8000B TID:04E7000A Desktop Window QI, Unkno
wn interfacePID:04E8000A TID:045E0032 Explorer(V2.0) taskbar thread started.

The troublesome part is this line:

 PID:00400002 TID:029E0002 FATFS!FormatVolume: Formatting volume. Flags = 0x16.

The data is sometimes wiped clean. We also sometimes see instances where the flash disk is not mounted resulting in an empty folder in Windows Explorer.

If you have any information that may be of help in troubleshooting or resolving this problem, please advise.

Thank you.

luka.tx · June 21, 2016, 7:37am

Hi,

We will look into this and get back to you.

valter.tx · June 28, 2016, 8:23am

We did not observe this behaviour during our tests, so we may need some more information to be able to reproduce it.
I see that registry is still loaded correctly, even when eMMC partion is re-formatted.
You are using eMMC boot partition, this will be used for bootloader, factory settings and config block. Registry, OS image and filesystem are on the user partition. The fact that the OS image boots and the registry is loaded means that the partition was not completely corrupted, only FS was someway damaged and caused a re-format. This happens when MBR of user partition is changed or when the first sector of FS partition does not contain a valid FAT.
Did you mount the flash partition via USB-mass storage?
Did you notice if some operations (running some applications, accessing some files on Flash etc.) seem to trigger this behaviour?

JinShil · June 29, 2016, 2:46am

We are not mounting the flash partition via USB-mass storage.

This past week, we have noticed file system corruption on both the eMMC and the SD Card with our aging test program. The program is quite simple: it repeatedly creates, writes, reads, and deletes 1MB files (with random data). After running for a while (30 minutes ~ 1 hours) both the eMMC and SD Card file systems become corrupt. The OS will no longer boot, and the SD Card can no longer be mounted.

Please allow me to articulate our bring up procedure in hope that it may provide additional clues:

We’re in the R&D stage, so we are only ordering low quantities. Our staff tells me we order from the Toradex office in China, and due to the low quantities we can’t select the OS. So, when we receive the modules, they contain Linux.
We then follow the procedure documented here (http://developer.toradex.com/knowledge-base/apalis-imx6-wince-bring-up) to change the OS to WEC 2013.
We enter recovery mode, and then flash the boot loader.
We then switch to the new partition method using usebootpartition true
We then have to enter recovery mode, and flash the boot loader again.
We then create a slightly modified OS in Platform Builder (just a few registry tweaks to support our TFT LCD), and flash the OS image.
When the OS boots, the user partition is too small (256MB). So we use the Storage Manager in Control Panel to resize the partition to utilize the full 4GB capacity.

Does that provide any additional leads? Should we be doing something differently?

valter.tx · June 30, 2016, 5:16am

On point 5 you can just use the flashloader command from the bootloader console and then reboot, but that shouldn’t make any difference in the process.
You should also use the preparefs command that will allow you to use the whole eMMC for filesystem storage, otherwise you will use only a 256MB partition. Resizing from control panel may have some side effects, we need to investigate this. We usually size the partition before we boot the OS for the first time.
We are seeing some issues accessing eMMC, so this issue may be related to the same issue, but at the moment we are still investigating the problem.

JinShil · July 1, 2016, 7:36am

I hope I’m not being premature here, but it appears that using the preparefs command instead of the Control Panel’s Storage Manager has solved this problem. I’ll give it a few more days of testing, and if all is well, I’ll mark this discussion answered. Thanks for the help!

JinShil · July 6, 2016, 6:16am

The problem that initially prompted this post appears to be solved by using the preparefs method. However, we are still seeing some weirdness like files disappearing between boots. I’ll mark this question answered and submit a separate question for the disappearing files problem after I collection some more information.

JinShil · July 6, 2016, 6:19am

See the comments under the original post. It appears using preparefs rather than the Control Panel’s Storage Manager has resolved the problem that originally prompted this question. We’re still seeing strange things like files disappearing between boots, but I’ll post a separate question for that after I collect some more information.