Apalis T30 freeze with java application

Dear all,

I’m using Java 8 application with GUI interface and MariaDB installed and working on Apalis T30 under Debian Linux and kernel provided by Toradex. I have eliminated a system by testing it under pure BSP from Toradex with only MariaDB and Java installed. The problem what I’m facing is that every few days or under a bigger stress with using this application within few minutes system freeze completely and becomes unresponsive. Ethernet light from evaluation board is still blinking, HDMI screen shows last image but I cannot do anything even under debug serial console connected. After a restart I cannot see anything in log etc /var/log/messages apart from the following:

Jan  1 01:08:10 ED4012 kernel: [  486.626107] tegra_uart tegra_uart.3: Rx DMA enabled

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
...
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
@r^@0.95-1^@2.5^@0.90.0^@2.14.0^@^@d70de009f48ea4b4f00e62329adb4f61^@3.84-2
^@0.5^@1.0.16^@1.12.4^@0.3.3^@2.7^@2.10^@1:4.4.0^@2.22.0^@2.12.0^@2.24.0^@1.9.5~dfsg-14
^@0.15.1b-3^@4.6^@1.1.2^@2.7.4^@1:1.1.4^@^@6effca71793975d2245f41b926117d10^@termit^@2.9.4-2
^@1.12.4^@2.13-28^@1.2.4^@2.9.0^@2.2.1^@2.22.0^@2.16.0^@2.16.0^@1.14.0^@1:0.28.0^@
^@914c279d9918558ebdb4531e8eee4eae^@termsaver^@0.1.1-1^@2.6.6-7~
^@^@ce1ee60f9643116da439bf0866623821^@terraintool^@1.12a-1
^@0.5^@^@cd855b8975a6676a18fc1906014652e4^@teseq^@1.0.0-2.1
^@2.7^@1.15.4^@^@9c9c0acdfa944027f875774e345ebddb^@slang-tess^@0.3.0-6
^@2.0^@0.99.17^@0.9.8.1pl1-4^@^@08a71f3deb1c57ba825b77c0a21e7ab1^@0.3.1-6+b1^@2.13-28
^@1:1.1.4^@2.6.6-7~^@2.8^@^@5a1938a48f0dbbb10be1521472eda9ed^@0.3.1-6+b1^@2.13-28
^@1:1.1.4

Jan  1 01:00:50 ED4012 kernel: imklog 5.8.11, log source = /proc/kmsg started.

I have enabled a crashdump in a kernel and installed a kexec-tools, added a crashkernel param to U-Boot params butt cannot see it in /proc/cmdline, and cat /sys/kernel/kexec_crash_loaded shows 0. Can someone help me with chasing and resolving this problem?

What exact Toradex hardware and software versions of things are you talking about? What exactly is your application doing (e.g. with MariaDB and Java)?

I have added hardware specification in environment position.
I would like to add, that we have around 50 apalis boards working with our custom boards and additionally I have tested this problem under apalis evaluation board and problem always occurs. About a software I have tested it under few releases of java like 1.8.0_121, Java SE Development Kit 8u151 downloaded from oracle website, pure Apalis_T30_LinuxImageV2.6_20160826.tar.bz2 and maria db installed with command opkg install mariadb, and under our custom debian wheezy created by my company with 10.0.29-MariaDB.

Our application shows few buttons, connects via RMI with another applications and exchanching a data, writes data to local maria db and shows and hides some JPanels, JLabels. Additionally, there are few threads showing and hidings panels. What I suspect there is some problem with gui, because system usually freeze when clicking buttons. I have been trying to freeze system by adding more or less 1 milion of data to the local database via RMI but in this case scenario application didn’t freeze. What I have notice system during hang writes usually a lot of "nul"l to most of open logs files log from java logback

tested our java app under Toradex Apalis T30 2GB V1.1A, Serial# 02715015, mainline kernel, Colibri-VF_LXDE-Image_2.7b4-20171005 rootfs and application works well, so it looks like some kernel issue

We finally found first and hopefully last case scenario, when linux freeze. It looks like there is some problem with sound driver and playing wave file from java app. When we start playing wave file and have to stop it during play, sytem quite often completely freeze. Below you can find testing code example:

import java.io.File;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.Clip;

public class ApalisMain {
       public static void main(String[] args) {
             Clip clip;
             try {
                    clip = AudioSystem.getClip();
                    clip.open(AudioSystem.getAudioInputStream(new File("s1.wav")));
                    clip.start();
                    try {
                          Thread.sleep(100);
                    } catch (Exception e) {
                          e.printStackTrace();
                    }            
                    //stopping audio clip kills apalis
                    clip.stop();

             } catch (Exception exc) {
                    exc.printStackTrace();
             }
       }
}

wave file, tested under jdk-8u151-linux-arm32-vfp-hflt.tar.gz, to run this program, first need to compile it with command: javac ApalisMain.java, and then run it with command: java ApalisMain

That now with downstream or mainline?

downstream

OK, do you have any debugging information (e.g. log files) of this?

There is no any debug. Apalis dies immediately without any debug info (ethernet stops working, nothing happens on the screen, no ping to host, debug console completely unresponsive, only hard reset helps). Just a lot of nulls or trash coming into all logs files (system logs as well) as you can see in posts above. I have tried to configure crash kernel dump, by adding it to kernel config and setting crashkernel param to boot command, but kernel didn’t start with crashkernel param added.

hi kamil

we are working on this issue and will come back to you during this week.

best regards, Jaski

Hi Kamil

What is exactly the Problem then? Is it solved?

Dear Jaski,

Do you have any update?

Best regards,
Kamil

Hi Kamil

I wrote a comment above to your post:

tested our java app under Toradex
Apalis T30 2GB V1.1A, Serial#
02715015, mainline kernel,
Colibri-VF_LXDE-Image_2.7b4-20171005
rootfs and application works well, so
it looks like some kernel issue.

What is exactly the Problem then? Is it solved?

Hi Jaski,
Problem is not solved. It looks like there is a bug in downstream kernel and sound driver provided by toradex. In newest mainline kernel we didn’t notice this problem but we are still using downstream kernel because of drivers issues. We have made a workaround for a moment with swaping wave to midi file, but at the end of the day we need to use wave files and we need properly working kernel with this driver.

hi kamil, thanks for your answer. I am trying to reproduce this issue. The java code you provided is neither working on my host (ubuntu 16.04) nor on the target. I have installed "openjdk version “1.8.0_151” on host and openjdk version "1.8.0_102 installed on target.

Which java version are you using exaclty?

Hi Jaski,

java -version
java version “1.8.0_121”
Java™ SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot™ Client VM (build 25.121-b13, mixed mode)

This code uses standard java library and after compilation should work on all subversions of java 8

to run this code:

  1. copy wave file from the attachment to the target machine
  2. create file ApalisMain.java in the same dir as the copied wave file
  3. put the content into this file from post: Answer by kamil · Jan 09 at 11:33 AM
  4. compile it with command: javac ApalisMain.java,
  5. And then run it with command: java ApalisMain

Thanks for the infos. I tried this already. The compilation was fine but the application was not running. I will try again on Monday and let you know the results.

Hi Kamil

I am having the following error:

java.lang.IllegalArgumentException: No line matching interface Clip supporting format PCM_SIGNED unknown sample rate, 16 bit, stereo, 4 bytes/frame, big-endian is supported.
        at javax.sound.sampled.AudioSystem.getLine(Unknown Source)
        at javax.sound.sampled.AudioSystem.getClip(Unknown Source)
        at ApalisMain.main(ApalisMain.java:12)

Dear Jaski,

In attachment you will find full log from original apalis image install, to kill apalis using java and jdk version downloaded from oracle website link text.

please note, that jdk installed with opkg install openjdk-8 is not able to play wave files.

@jaski.tx is currently at Embedded World 2018. He will continue to follow up early next week.