OV5640 camera module integration

I connected my camera to X28 MIPI-CSI port, followed this guide, and wanted to see camera preview. Initially in devices I used ‘/dev/video4’ and got an exception:

    wayland_1  | ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output
    wayland_1  | Additional debug info:
    wayland_1  | xvimagesink.c(1773): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
    wayland_1  | Could not open display (null)

After that I tried all devices returned by “dmesg | grep video”, but every time I got the same exception described above.
I don’t know the cause of this exception and have no idea how to solve it.
Help me please


That’s my Dockerfile for building an image which will use my camera:

FROM torizon/arm64v8-debian-wayland-base-vivante
 
RUN apt-get -y update && apt-get install -y \
	libgstreamer1.0-0 \
	gstreamer1.0-plugins-base \
	gstreamer1.0-plugins-good \
	gstreamer1.0-plugins-bad \
	gstreamer1.0-plugins-ugly \
	gstreamer1.0-libav \
	gstreamer1.0-doc \
	gstreamer1.0-tools \
	gstreamer1.0-x \
	gstreamer1.0-alsa \
	gstreamer1.0-gl \
	gstreamer1.0-gtk3 \
	gstreamer1.0-qt5 \
	gstreamer1.0-pulseaudio \
	&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
	
ENTRYPOINT ["gst-launch-1.0"] 

CMD ["-v", "v4l2src", "device=/dev/video10", "!", "capsfilter", "caps=\"video/x-raw, width=640, height=480, framerate=30/1\"", "!", "xvimagesink"]

That’s my docker-compose file:

version: "2.4"
services:
  weston:
    image: torizon/arm64v8-debian-weston-vivante:latest
    network_mode: host
    environment:
      - ACCEPT_FSL_EULA=1
    devices:
      - /dev/video10:/dev/video10
    volumes:
      - type: bind
        source: /tmp
        target: /tmp
      - type: bind  
        source: /run/udev
        target: /run/udev
      - type: bind
        source: /dev
        target: /dev
    cap_add:
      - CAP_SYS_TTY_CONFIG
    # Add device access rights through cgroup...
    device_cgroup_rules:
      # ... for tty0
      - 'c 4:0 rmw'
      # ... for tty7
      - 'c 4:7 rmw'
      # ... for /dev/input devices
      - 'c 13:* rmw'
      - 'c 199:* rmw'
      # ... for /dev/dri devices
      - 'c 226:* rmw'
      - 'c 81:* rmw'
  wayland:
    depends_on:
      - weston
    image: alexey777/camera
    devices:
      - /dev/video10:/dev/video10
    volumes:
       - /tmp:/tmp
       - /sys:/sys
       - /dev:/dev

That’s my output:

Creating torizon_weston_1 ... done
Creating torizon_wayland_1 ... done
Attaching to torizon_weston_1, torizon_wayland_1
weston_1   | NXP EULA has already been accepted.
weston_1   | Switching to VT 7
weston_1   | NXP EULA has already been accepted.
weston_1   | touch: cannot touch '/tmp/nxp-eula-accepted': Permission denied
weston_1   | Date: 2020-06-04 UTC
weston_1   | [11:29:11.699] weston 5.0.0
weston_1   |                https://wayland.freedesktop.org
weston_1   |                Bug reports to: https://gitlab.freedesktop.org/wayland/weston/issues/
weston_1   |                Build: unknown (not built from git or tarball)
weston_1   | [11:29:11.699] Command line: /usr/bin/weston --current-mode
weston_1   | [11:29:11.699] OS: Linux, 4.14.170-4.0.0-devel+git.fb6b17a22634, #1-TorizonCore SMP PREEMPT Mon May 11 14:38:24 UTC 2020, aarch64
weston_1   | [11:29:11.699] Using config file '/etc/xdg/weston/weston.ini'
weston_1   | [11:29:11.700] Output repaint window is 16 ms maximum.
weston_1   | [11:29:11.700] Loading module '/usr/lib/aarch64-linux-gnu/libweston-5/drm-backend.so'
weston_1   | [11:29:11.701] initializing drm backend
weston_1   | [11:29:11.702] using /dev/dri/card0
weston_1   | [11:29:11.702] DRM: supports universal planes
weston_1   | [11:29:11.702] DRM: supports atomic modesetting
weston_1   | [11:29:11.702] DRM: does not support picture aspect ratio
weston_1   | [11:29:11.703] Loading module '/usr/lib/aarch64-linux-gnu/libweston-5/gl-renderer.so'
weston_1   | [11:29:11.709] EGL client extensions: EGL_EXT_client_extensions
weston_1   |                EGL_EXT_platform_base EGL_KHR_platform_wayland
weston_1   |                EGL_EXT_platform_wayland EGL_KHR_platform_gbm
weston_1   | [11:29:11.713] warning: neither EGL_EXT_swap_buffers_with_damage or EGL_KHR_swap_buffers_with_damage is supported. Performance could be affected.
weston_1   | [11:29:11.713] EGL_KHR_surfaceless_context available
weston_1   | [11:29:11.718] EGL version: 1.5
weston_1   | [11:29:11.718] EGL vendor: Vivante Corporation
weston_1   | [11:29:11.718] EGL client APIs: OpenGL_ES
weston_1   | [11:29:11.718] EGL extensions: EGL_KHR_fence_sync EGL_KHR_reusable_sync
weston_1   |                EGL_KHR_wait_sync EGL_KHR_image EGL_KHR_image_base
weston_1   |                EGL_KHR_image_pixmap EGL_KHR_gl_texture_2D_image
weston_1   |                EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image
weston_1   |                EGL_EXT_image_dma_buf_import
weston_1   |                EGL_EXT_image_dma_buf_import_modifiers EGL_KHR_lock_surface
weston_1   |                EGL_KHR_create_context EGL_KHR_surfaceless_context
weston_1   |                EGL_EXT_buffer_age EGL_ANDROID_native_fence_sync
weston_1   |                EGL_WL_bind_wayland_display
weston_1   |                EGL_WL_create_wayland_buffer_from_image EGL_KHR_partial_update
weston_1   | [11:29:11.718] GL version: OpenGL ES 3.2 V6.2.4.p4.190076
weston_1   | [11:29:11.718] GLSL version: OpenGL ES GLSL ES 3.20
weston_1   | [11:29:11.718] GL vendor: Vivante Corporation
weston_1   | [11:29:11.718] GL renderer: Vivante GC7000XSVX
weston_1   | [11:29:11.718] GL extensions: GL_OES_vertex_type_10_10_10_2
weston_1   |                GL_OES_vertex_half_float GL_OES_element_index_uint
weston_1   |                GL_OES_mapbuffer GL_OES_vertex_array_object
weston_1   |                GL_OES_compressed_ETC1_RGB8_texture
weston_1   |                GL_OES_compressed_paletted_texture GL_OES_texture_npot
weston_1   |                GL_OES_rgb8_rgba8 GL_OES_depth_texture
weston_1   |                GL_OES_depth_texture_cube_map GL_OES_depth24 GL_OES_depth32
weston_1   |                GL_OES_packed_depth_stencil GL_OES_fbo_render_mipmap
weston_1   |                GL_OES_get_program_binary GL_OES_fragment_precision_high
weston_1   |                GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_sync
weston_1   |                GL_OES_texture_stencil8 GL_OES_shader_image_atomic
weston_1   |                GL_OES_sample_variables GL_OES_sample_shading
weston_1   |                GL_OES_texture_storage_multisample_2d_array
weston_1   |                GL_OES_shader_multisample_interpolation
weston_1   |                GL_OES_required_internalformat GL_OES_surfaceless_context
weston_1   |                GL_OES_copy_image GL_OES_draw_buffers_indexed
weston_1   |                GL_OES_geometry_shader GL_OES_geometry_point_size
weston_1   |                GL_OES_gpu_shader5 GL_OES_shader_io_blocks
weston_1   |                GL_OES_texture_border_clamp GL_OES_texture_buffer
weston_1   |                GL_OES_tessellation_shader GL_OES_tessellation_point_size
weston_1   |                GL_OES_texture_cube_map_array GL_OES_draw_elements_base_vertex
weston_1   |                GL_OES_texture_half_float GL_OES_texture_float
weston_1   |                GL_OES_primitive_bounding_box
weston_1   |                GL_KHR_texture_compression_astc_ldr
weston_1   |                GL_KHR_blend_equation_advanced GL_KHR_debug GL_KHR_robustness
weston_1   |                GL_EXT_texture_type_2_10_10_10_REV
weston_1   |                GL_EXT_texture_filter_anisotropic
weston_1   |                GL_EXT_texture_compression_dxt1 GL_EXT_texture_format_BGRA8888
weston_1   |                GL_EXT_texture_compression_s3tc GL_EXT_read_format_bgra
weston_1   |                GL_EXT_multi_draw_arrays GL_EXT_frag_depth
weston_1   |                GL_EXT_discard_framebuffer GL_EXT_blend_minmax
weston_1   |                GL_EXT_multisampled_render_to_texture
weston_1   |                GL_EXT_color_buffer_half_float GL_EXT_color_buffer_float
weston_1   |                GL_EXT_robustness GL_EXT_texture_sRGB_decode
weston_1   |                GL_EXT_draw_buffers_indexed GL_EXT_texture_border_clamp
weston_1   |                GL_EXT_texture_buffer GL_EXT_tessellation_shader
weston_1   |                GL_EXT_tessellation_point_size GL_EXT_geometry_shader
weston_1   |                GL_EXT_geometry_point_size GL_EXT_copy_image
weston_1   |                GL_EXT_texture_cube_map_array GL_EXT_gpu_shader5
weston_1   |                GL_EXT_shader_io_blocks GL_EXT_shader_implicit_conversions
weston_1   |                GL_EXT_multi_draw_indirect GL_EXT_draw_elements_base_vertex
weston_1   |                GL_EXT_texture_rg GL_EXT_primitive_bounding_box GL_EXT_sRGB
weston_1   |                GL_ANDROID_extension_pack_es31a GL_VIV_direct_texture
weston_1   | [11:29:11.718] GL ES 2 renderer features:
weston_1   |                read-back format: BGRA
weston_1   |                wl_shm sub-image to texture: yes
weston_1   |                EGL Wayland extension: yes
weston_1   | [11:29:11.729] event2  - LITEON Technology USB Multimedia Keyboard: is tagged by udev as: Keyboard
weston_1   | [11:29:11.729] event2  - LITEON Technology USB Multimedia Keyboard: device is a keyboard
weston_1   | [11:29:11.732] event1  - Logitech USB Optical Mouse: is tagged by udev as: Mouse
weston_1   | [11:29:11.732] event1  - Logitech USB Optical Mouse: device is a pointer
weston_1   | [11:29:11.734] event0  - sc-powerkey: is tagged by udev as: Keyboard
weston_1   | [11:29:11.734] event0  - sc-powerkey: device is a keyboard
weston_1   | [11:29:11.747] event3  - colibri-vf50-ts: is tagged by udev as: Touchscreen
weston_1   | [11:29:11.748] event3  - colibri-vf50-ts: device is a touch device
weston_1   | [11:29:11.767] Touchscreen - colibri-vf50-ts - /sys/devices/platform/vf50-touchscreen/input/input3/event3
weston_1   | [11:29:11.768] input device event3 has no enabled output associated (none named), skipping calibration for now.
weston_1   | [11:29:11.793] DRM: head 'HDMI-A-1' found, connector 146 is connected, EDID make 'PHL', model 'PHL 234E5', serial 'UHB142400574'
weston_1   | [11:29:11.793] Registered plugin API 'weston_drm_output_api_v1' of size 24
weston_1   | [11:29:11.803] Chosen EGL config details:
weston_1   |                RGBA bits: 8 8 8 0
weston_1   |                swap interval range: 1 - 60
weston_1   | [11:29:11.804] No backlight control for output 'HDMI-A-1'
weston_1   | [11:29:11.804] Output HDMI-A-1 (crtc 36) video modes:
weston_1   |                1920x1080@60.0, preferred, current, 148.5 MHz
weston_1   |                1920x1080@59.9, 148.4 MHz
weston_1   |                1920x1080@50.0, 148.5 MHz
weston_1   |                1680x1050@59.9, 119.0 MHz
weston_1   |                1280x1024@75.0, 135.0 MHz
weston_1   |                1280x1024@60.0, 108.0 MHz
weston_1   |                1440x900@75.0, 136.8 MHz
weston_1   |                1440x900@59.9, 88.8 MHz
weston_1   |                1280x720@60.0, 74.2 MHz
weston_1   |                1280x720@59.9, 74.2 MHz
weston_1   |                1280x720@50.0, 74.2 MHz
weston_1   |                1024x768@75.0, 78.8 MHz
weston_1   |                1024x768@60.0, 65.0 MHz
weston_1   |                800x600@75.0, 49.5 MHz
weston_1   |                800x600@60.3, 40.0 MHz
weston_1   |                720x576@50.0, 27.0 MHz
weston_1   |                720x480@60.0, 27.0 MHz
weston_1   |                720x480@59.9, 27.0 MHz
weston_1   |                640x480@75.0, 31.5 MHz
weston_1   |                640x480@72.8, 31.5 MHz
weston_1   |                640x480@66.7, 30.2 MHz
weston_1   |                640x480@60.0, 25.2 MHz
weston_1   |                640x480@59.9, 25.2 MHz
weston_1   |                720x400@70.1, 28.3 MHz
weston_1   | [11:29:11.804] associating input device event2 with output HDMI-A-1 (none by udev)
weston_1   | [11:29:11.804] associating input device event1 with output HDMI-A-1 (none by udev)
weston_1   | [11:29:11.804] associating input device event0 with output HDMI-A-1 (none by udev)
weston_1   | [11:29:11.804] associating input device event3 with output HDMI-A-1 (none by udev)
weston_1   | [11:29:11.806] Output 'HDMI-A-1' enabled with head(s) HDMI-A-1
weston_1   | [11:29:11.806] Compositor capabilities:
weston_1   |                arbitrary surface rotation: yes
weston_1   |                screen capture uses y-flip: yes
weston_1   |                presentation clock: CLOCK_MONOTONIC, id 1
weston_1   |                presentation clock resolution: 0.000000001 s
weston_1   | [11:29:11.806] Loading module '/usr/lib/aarch64-linux-gnu/weston/desktop-shell.so'
weston_1   | [11:29:11.808] launching '/usr/lib/aarch64-linux-gnu/weston-keyboard'
weston_1   | [11:29:11.816] Loading module '/usr/lib/aarch64-linux-gnu/libweston-5/xwayland.so'
weston_1   | [11:29:11.836] Registered plugin API 'weston_xwayland_v1' of size 32
weston_1   | [11:29:11.836] Registered plugin API 'weston_xwayland_surface_v1' of size 16
weston_1   | [11:29:11.836] xserver listening on display :0
weston_1   | [11:29:11.836] launching '/usr/lib/aarch64-linux-gnu/weston-desktop-shell'
weston_1   | could not load cursor 'dnd-move'
weston_1   | could not load cursor 'dnd-move'
weston_1   | could not load cursor 'dnd-copy'
weston_1   | could not load cursor 'dnd-copy'
weston_1   | could not load cursor 'dnd-none'
weston_1   | could not load cursor 'dnd-none'
wayland_1  | Setting pipeline to PAUSED ...
weston_1   | [11:29:16.328] Spawned Xwayland server, pid 26
weston_1   | glamor: 'wl_drm' not supported
weston_1   | Missing Wayland requirements for glamor GBM backend
weston_1   | Failed to initialize glamor, falling back to sw
weston_1   | [11:29:16.652] xfixes version: 5.0
weston_1   | [11:29:16.712] created wm, root 908
wayland_1  | No protocol specified
wayland_1  | ERROR: Pipeline doesn't want to pause.
wayland_1  | ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output
wayland_1  | Additional debug info:
wayland_1  | xvimagesink.c(1773): gst_xv_image_sink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
wayland_1  | Could not open display (null)
wayland_1  | Setting pipeline to NULL ...
wayland_1  | Freeing pipeline ...
torizon_wayland_1 exited with code 255

In the host OS execution of “dmesg | grep video” returns:

[    0.488475] Linux video capture interface: v2.00
[    3.520406] mx8-img-md: Registered mxc_isi.0.capture as /dev/video0
[    3.520502] mx8-img-md: Registered mxc_isi.1.capture as /dev/video1
[    3.520603] mx8-img-md: Registered mxc_isi.2.capture as /dev/video2
[    3.520699] mx8-img-md: Registered mxc_isi.3.capture as /dev/video3
[    3.520797] mx8-img-md: Registered mxc_isi.4.capture as /dev/video4
[    3.520894] mx8-img-md: Registered mxc_isi.5.capture as /dev/video5
[    3.521016] mx8-img-md: Registered mxc_isi.6.capture as /dev/video6
[    3.521116] mx8-img-md: Registered mxc_isi.7.capture as /dev/video7
[    3.523763] mxc-jpeg 58400000.jpegdec: decoder device registered as /dev/video9 (81,19)
[    3.524688] mxc-jpeg 58450000.jpegenc: encoder device registered as /dev/video10 (81,20)

In the host OS execution of “dmesg | grep camera” returns:

[    3.519099] ov5640_probe camera mipi ov5640, is found
[    3.521421] mxc-md camera: mxc_md_create_links

Greetings @AlexeyZ,

So it seems like here the main problem is that wayland for some reason can’t access the display.

Quick question as I already have a hunch of what might be wrong. Is there any other containers already running other than your containers? (i.e. what’s the output of docker ps?)

The reason I ask is that it might be the case that there is another graphical container running which might already have “control” of the display. This could lead to this error.

In any case make sure no other container is running then try to re-run your containers. If the issue persists please let me know.

Best Regards,
Jeremias

Hello @jeremias.tx

There were no any running containers except two described by me.
Initially I followed the guide without any changes and I was able to see a test video on my display, it was similar to this one. So it seems the problem is not directly related to my display.
Perhaps something wrong with my gstreamer settings.

Hi @AlexeyZ,

Thank you for mentioning your gstreamer settings as I didn’t notice it the first time. I believe the issue is your usage of xvimagesink. As I understand this renders videos to a draw-able XWindow. By default Torizon container images don’t use X-based graphics stacks.

The wayland equivalent would be ‘waylandsink’. On my setup your pipeline worked fine once I used waylandsink instead. However this was with a USB camera, though let me know if there is an issue specifically with the mipi-csi camera.

Best Regards,
Jeremias

Hello @jeremias.tx

I changed my gstreamer settings to these ones and after that I saw camera preview.
Thank you!

Best Regards,
Alexey

Glad to hear it worked out for you!