Torizon: no space left on device

probably, recently arm32v7-debian-qt5-wayland:2 updated

and now after

docker save register/application | ssh torizon@192.168.1.7 docker load

I get

Error processing tar file(exit status 1): write /opt/App_dir/bin/App_bin: no space left on device

or

Error processing tar file(exit status 1): Error setting up pivot dir: mkdir /var/lib/docker/overlay2/e2a0040189aba59f1b7aeac21efbe5fdf3f4917509b1246bf4e13b3a1ff779ae/diff/.pivot_root135283437: no space left on device

The size of the image is the same as before arm32v7-debian-qt5-wayland:2 was updated, 606 MB

There is free space on the board:

apalis-imx6:~$ df
Filesystem                 1K-blocks    Used Available Use% Mounted on
tmpfs                        1031352   22572   1008780   3% /run
devtmpfs                      961364       0    961364   0% /dev
/dev/disk/by-label/otaroot   3631504 2600500    826820  76% /sysroot
tmpfs                        1031352       0   1031352   0% /dev/shm
tmpfs                        1031352       0   1031352   0% /sys/fs/cgroup
tmpfs                        1031352      12   1031340   1% /tmp
tmpfs                        1031352       8   1031344   1% /var/volatile
tmpfs                         206268       0    206268   0% /run/user/1000

~$ docker images
register/App-client              latest              2781e9ae398b        31 minutes ago      606MB
register/App-weston              latest              346d3da9bd2f        2 months ago        288MB
qt5-target                       latest              0cdd441ae956        2 months ago        593MB
portainer/portainer              latest              9b438d60114e        2 months ago        62.4MB

apalis-imx6:~$ docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

To build the image I take the example for build with Qt and modify that

ARG BASE_NAME=arm32v7-debian-qt5-wayland:2
FROM torizon/$BASE_NAME

SHELL ["/bin/bash", "-c"]

RUN apt-get -y update && apt-get install -y --no-install-recommends \
    qml-module-qtquick-controls2 \
    qml-module-qt-websockets \
    qml-module-qt3d \
    qml-module-qtaudioengine \
    qml-module-qtav \
    qml-module-qtbluetooth \
    qml-module-qtcharts \
    qml-module-qtdatavisualization \
    qml-module-qtgraphicaleffects \
    qml-module-qtgstreamer \
    qml-module-qtlocation \
    qml-module-qtmultimedia \
    qml-module-qtnfc \
    qml-module-qtpositioning \
    qml-module-qtqml-statemachine \
    qml-module-qtquick-dialogs \
    qml-module-qtquick-controls \
    qml-module-qtquick-controls2 \
    qml-module-qtquick-extras \
    qml-module-qtquick-scene2d \
    qml-module-qtquick-scene3d \
    qml-module-qtquick-shapes \
    qml-module-qtquick-templates2 \
    qml-module-qtquick-virtualkeyboard \
    qml-module-qtsensors \
    qml-module-qtwayland-compositor \
    qml-module-qtwebchannel \
    qml-module-qtwebengine \
    qml-module-qtwebkit \
    qml-module-qtwebsockets \
    qml-module-qtwebview \
    qt5-qmltooling-plugins \
    openssh-server \
    gdbserver \
    rsync \
    && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*

# Allow empty password hack (only for developement)
# sed -i /etc/ssh/sshd_config -e 's/UsePAM yes/UsePAM no/g'
RUN sed -i 's/nullok_secure/nullok/g' /etc/pam.d/common-auth && \
    echo "PermitEmptyPasswords yes" >> /etc/ssh/sshd_config && \
    echo "PermitRootLogin yes" >> /etc/ssh/sshd_config && \
    sed -i /etc/shadow -e 's/root:.*/root::18254:0:99999:7:::/g'

RUN test $(arch) == "aarch64" && { cd /usr/lib/aarch64-linux-gnu/ && ln -s libGL.so.1.7.0 libGL.so.1.2; } || true

COPY entrypoint.sh /entrypoint.sh
COPY App_bin /opt/App_directory/bin/
COPY App_translation.qm /opt/App_bin/bin/

RUN echo "set -o allexport; . /etc/environment; set +o allexport;" >> /etc/profile

ENTRYPOINT ["/entrypoint.sh"]

App_bin is 12 MB
qml translation file is 22 kB
entrypoint is 403 B

Last week it worked.
Today repo updated and save fails.

How to find the bug?

Greetings @Juzujka,

It’s true that we updated our Debian containers recently this week. Though I don’t believe it should have caused this issue, since the updated image on our side only increased the size by a small amount.

Though I guess first let’s confirm whether it was the recent update that caused this for you. Please see the tags for torizon/arm32v7-debian-qt5-wayland here: Docker Hub

The updated tag that we pushed was for 2.3.2. The 2 tag that you use always points towards the latest tag, so in this case 2.3.2. and 2 should be the same image. Can you try rebuilding your container with an older tag like 2.3.0 or 2.2.0. Do you still get the same space issue?

Also regarding the other container images you have on your device. What base images are those using? If they’re also using torizon debian base images then those have all been updated as well. Perhaps it might help to get all your container images using the latest debian images from us. It might help cut down on some docker layer storage sizes since they should share common layers. If they’re using the same/similar base image that is.

Best Regards,
Jeremias

If you’ve been experimenting with containers it may help using:

docker system prune

to remove terminated containers and images no longer associated with a tag.
You may also want to use

docker images

to check if there are images you don’t use and then delete them using:

docker rmi

Thank you, @jeremias.tx

I try

ARG BASE_NAME=arm32v7-debian-qt5-wayland:2.3.0-20210519

and

ARG BASE_NAME=arm32v7-debian-qt5-wayland:2.2.2-20201117

the result is slightly different in the path to file which exceeds available space

then I push the image to the register and pull it via ssh from the board:

docker image pull register/App-client

loaded successfully

I.e. with the same image

docker save register/App-client | ssh torizon@IP docker load

fails with no space available and

docker image pull register/App-client

loads successfully.

About other images - I use docker-compose

version: "2.4"
services:
  weston:
    container_name: weston
    image: register/App-weston
    ...
  portainer:
    image: portainer/portainer:latest
    ...
  App-client:
    container_name: App-client
    image: register/App-client
    ...

App-weston built with Dockerfile with

ARG BASE_NAME=wayland-base
ARG WESTON=weston
ARG IMAGE_TAG=2
...

And this image was built at 2021.04.01

Perhaps the docker save/load method has some additional storage overhead that is pushing the storage over limit. I suggest for now then to stick with the docker pull method, at least until you can clear more space on the device that save/load works properly again.

Best Regards,
Jeremias