Vybrid vf50 freezes

Technical Support
, ,
1

Hi All,

I’m running linux on a vf50 based Colibri board which is mated to a custom build baseboard. The Colibri is replacing a cm-x300 Marvell based SoM.
I noticed that the uart stops working frequently when not disabling DMA, so I changed to pio mode as recommended. I also applied the TTY_OVERRUN patch to the uart driver.
Now the system seems more stable however when I leave the unit running overnight the next morning the unit does not respond any more, so no console output and my qt application UI is frozen.

I started with the Colibri_VF_LinuxConsoleImageV2.6.1 and then customized the kernel a bit to support boot logo and display basically.

Is this a known issue? Is there anything I can check?

Thank,
Vincent

2

Does it boot when your reset or re-power the module ?
If yes you can get the previous boot logs using journalctl at least to get some hints on what exactly went wrong.

3

As for the UART issue: With the V2.6.1 DMA should work fine again. There was one issue in the latest V2.6.1 release still, it should be solved with this commit:

http://git.toradex.com/cgit/linux-toradex.git/commit/?h=toradex_vf_4.4&id=a650cbfc7e981ba24313da0338f4482bef514d44

Update 2018-05-30: Make link point to a stable git hash

4

Hi,

It boots fine after a re-power.
This is the journal right after I left the unit for the night, so the “enterStandbyMode” is the app turning hardware off, the Colibri is not going into sleep mode.

Dec 06 17:49:16 colibri-vf sh[212]: ****INFO: confirmPowerOff: PowerOff confirmed
Dec 06 17:49:16 colibri-vf sh[212]: ****INFO: enterStandbyMode
Dec 06 17:57:56 colibri-vf login[321]: pam_unix(login:session): session closed for user root
Dec 06 17:57:56 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:57:56 colibri-vf systemd-logind[221]: Removed session c1.
Dec 06 17:57:57 colibri-vf systemd[1]: Stopped User Manager for UID 0.
Dec 06 17:57:57 colibri-vf systemd[1]: Removed slice user-0.slice.
Dec 06 17:57:57 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:57:57 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:57:58 colibri-vf agetty[706]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:58:08 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:08 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:08 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:58:10 colibri-vf agetty[710]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:58:20 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:20 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:20 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:58:21 colibri-vf agetty[714]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:58:31 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:31 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:31 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:58:32 colibri-vf agetty[718]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:58:41 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:41 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:41 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:58:42 colibri-vf agetty[722]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:58:52 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:53 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:53 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:58:57 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:58:57 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:58:57 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:59:02 colibri-vf login[730]: [[1;39mpam_unix(login:auth): check pass; user unknown[[0m
Dec 06 17:59:02 colibri-vf login[730]: [[1;39mpam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyLP0 ruser= rhost=[[0m
Dec 06 17:59:05 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:59:05 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:59:05 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:59:09 colibri-vf login[734]: [[1;39mpam_unix(login:auth): check pass; user unknown[[0m
Dec 06 17:59:09 colibri-vf login[734]: [[1;39mpam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyLP0 ruser= rhost=[[0m
Dec 06 17:59:12 colibri-vf login[734]: [[1;39mFAILED LOGIN (1) on ‘/dev/ttyLP0’ FOR ‘UNKNOWN’, Authentication failure[[0m
Dec 06 17:59:16 colibri-vf login[734]: [[1;39mpam_unix(login:auth): check pass; user unknown[[0m
Dec 06 17:59:16 colibri-vf login[734]: [[1;39mpam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyLP0 ruser= rhost=[[0m
Dec 06 17:59:19 colibri-vf login[734]: [[1;39mFAILED LOGIN (2) on ‘/dev/ttyLP0’ FOR ‘UNKNOWN’, Authentication failure[[0m
Dec 06 17:59:23 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:59:23 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:59:23 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:59:27 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:59:28 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:59:28 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:59:32 colibri-vf login[742]: [[1;39mpam_unix(login:auth): check pass; user unknown[[0m
Dec 06 17:59:32 colibri-vf login[742]: [[1;39mpam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyLP0 ruser= rhost=[[0m
Dec 06 17:59:36 colibri-vf login[742]: [[1;39mFAILED LOGIN (1) on ‘/dev/ttyLP0’ FOR ‘UNKNOWN’, Authentication failure[[0m
Dec 06 17:59:44 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:59:44 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:59:44 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 17:59:45 colibri-vf agetty[746]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 17:59:55 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 17:59:55 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 17:59:55 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 18:00:00 colibri-vf login[750]: [[1;39mpam_unix(login:auth): check pass; user unknown[[0m
Dec 06 18:00:00 colibri-vf login[750]: [[1;39mpam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyLP0 ruser= rhost=[[0m
Dec 06 18:00:02 colibri-vf login[750]: [[1;39mFAILED LOGIN (1) on ‘/dev/ttyLP0’ FOR ‘UNKNOWN’, Authentication failure[[0m
Dec 06 18:00:06 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 18:00:06 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 18:00:06 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 18:00:07 colibri-vf agetty[754]: [[1;31mcheckname failed: Operation not permitted[[0m
Dec 06 18:00:17 colibri-vf systemd[1]: serial-getty@ttyLP0.service: Service has no hold-off time, scheduling restart.
Dec 06 18:00:17 colibri-vf systemd[1]: Stopped Serial Getty on ttyLP0.
Dec 06 18:00:17 colibri-vf systemd[1]: Started Serial Getty on ttyLP0.
Dec 06 18:00:18 colibri-vf agetty[758]: [[1;31mcheckname failed: Operation not permitted[[0m
[[1;39m-- Reboot --[[0m
Dec 07 08:37:41 colibri-vf systemd-journal[76]: Runtime journal (/run/log/journal/) is currently using 4.0M.
Maximum allowed usage is set to 8.0M.
Leaving at least 8.8M free (of currently available 54.6M of space).
Enforced usage limit is thus 8.0M.
Dec 07 08:37:41 colibri-vf systemd-journal[76]: Runtime journal (/run/log/journal/) is currently using 4.0M.
Maximum allowed usage is set to 8.0M.
Leaving at least 8.8M free (of currently available 54.6M of space).
Enforced usage limit is thus 8.0M.
Dec 07 08:37:41 colibri-vf kernel: Booting Linux on physical CPU 0x0
Dec 07 08:37:41 colibri-vf kernel: [[1;39mLinux version 4.4.21-dirty (vincent@blofeld) (gcc version 5.2.1 20151005 (Linaro GCC 5.2-2015.11-2) ) #120 Mon Dec 5 17:43:17 CET 2016[[0
m
Dec 07 08:37:41 colibri-vf kernel: CPU: ARMv7 Processor [410fc051] revision 1 (ARMv7), cr=10c5387d
Dec 07 08:37:41 colibri-vf kernel: CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache

Next morning Dec 07 08:37:41 I boot the unit again.

So it seems the SoC is alive since the display is being refreshed however the console is completely dead and my app is not responding to user input any more.

Thanks,
Vincent

5

Hi,

Patch applied, I will re-enable the DMA again.

Thanks,
Vincent

6

If possible can you please brief about your application functionality ?

7

I use the app to control a Sharc DSP for controls like mute/gain/eq etc.

To me it seems there is something off with serial-getty@ttyLP0.service, as it constantly restarts.

Also the bash history shows allot of garbage after my last input.

265 root@colibri-vf:~# #
266 root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co
267 lihroot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co root@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@colibri-vf:~#oot@co h
268 C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^
269 C^
270
271
272 root@colibri-vf:~#
273 root@colibri-vf:~#
274 root@colibri-vf:~# r! ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H ^H^H
275
276 (arg: 1) root@colibri-vf:~# r! ^H^H ^H^H
277 root@colibri-vf:~# ^C
278 -sh: :s^C: substitution failed
279 -sh: :s^C: substitution failed
280 -sh: :s^C: substitution failed
281 (arg: 1) {.bash_history,4,8} z.bah-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failed-sh: :s^C: substitution failedhEEEE#
282 P(arg: 1) (arg: 1) {.bash_history,4,8} z.bah
283 h
284 P(arg: 1) actl h
285 (arg: 45) actl h
286 h
287 P(arg: 1) rooth
288 (arg: 45) rooth
289 Display all 727 possibilities? (y or n)
290 root@colibri-vf:~#
291 {.{.{.bashh
292 Bh
293 P(arg: 1) {.{.{.bashh
294 #(arg: 4) {.{.{.
295 #
296 root@colibri-vf:~# #
297 root@colibri-vf:~#
298 Display all 727 possibilities? (y or n)
299 root@colibri-vf:~#
300 root@
301 -sh: root@: command not found
302 -sh: root@: command not found
303 -sh: root@: command not foundot@colibri-vf:~# -
304 -sh: root@: command not found
305 -sh: root@: command not5
306 Ci
307 depmod fw_printenv poweroff sysctl.procps
308 @
309 P(arg: 45) @
310 root@colibri-vf:~# @
311 -sh: @: command not found

8

Which tty port you are using for control ?

9

I’m using ttyLP1 to communicate to external controller.

10

serial-getty@ttyLP0.service should not restart constantly.

I recommend to do small steps to isolate the issue. Is the problem reproduce-able without your application running?

11

Yes, so I left the board running again yesterday but with the uart patch and DMA enabled, so this morning the app was still running. However the console ttyLP0 was dead again.

I did not see the serial-getty@ttyLP0.service restarting messages in the journal this morning and also no random input in the shell history.

The problem was already present when I first ran the Toradex Console image V2.6.1 and I thought it was the uart which was hanging the system or at least the console.

12

I found something interesting…

My board is connected via a usb-serial adapter (powered from USB port)

[353941.549317] usb 1-1.2.4: Product: USB-Serial Controller
[353941.549319] usb 1-1.2.4: Manufacturer: Prolific Technology Inc.
[353941.549676] pl2303 1-1.2.4:1.0: pl2303 converter detected
[353941.551278] usb 1-1.2.4: pl2303 converter now attached to ttyUSB1

When I leave the system running for the night I put the host in sleep and the converter looses power, when power is restored the uart/console does not function properly any more. This also happens when I remove the adapter from the pc side. The board needs a reboot to get a working console again.