Using Cameras in Embedded Linux Systems

Thursday, July 28, 2016

Camera

Introduction

This blog post explores various ways to view and record videos using a Computer on Module (CoM) or System on Module (SoM). The CoM is optimized for cost-sensitive applications and has no hardware encoder/decoder for video.

Use of Cameras in Embedded Systems

There is a saying that goes like this: “A picture is worth a thousand words.” Maybe that’s why the use of cameras in embedded systems has grown tremendously in recent years. They are used in different situations, including:

Hardware Used

We will use our Colibri VF61 module, which is powered by NXP®/Freescale’s Vybrid processor, a heterogeneous dual core processor (ARM® Cortex-A5 + ARM Cortex-M4). In addition to the processor, the module has 256MB RAM and 512 MB Flash. In this article, we will be using only the Cortex™-A5. You can do various video-related tasks with these processors despite them not having specific hardware acceleration for it. Further information and instructions for the Colibri VF61 module can be found on the developers section of the Toradex website.

The cameras used were one Logitech HD 720p, a generic mjpeg USB camera module and an IP camera from D-Link model DCS-930L.

The Linux image used for this article has a desktop environment called LXDE, it is also our standard image. Our standard images can be found on our Developer's website here. The image version used in this article is V2.4. We will use a framework called GStreamer, which is widely used in the development of multimedia applications. GStreamer provides application’s multimedia service such as video editors, media streaming and media players. A series of plugins make GStreamer work with many different media libraries, such as MP3,ffmpeg, and others. Among these plugins are input and output elements, filters, codecs, and more.

Intalling Necessary Packages

This how-to has been written and tested with our V2.4 Linux image. At the time of its writing, the package feeds from V2.5 are missing one dependency for GStreamer. We are currently investigating this. As a workaround, you can download and install the missing dependencies manually from:
http://feeds.angstrom-distribution.org/feeds/v2015.06/ipk/glibc/armv7ahf-vfp-neon/base
It is necessary to install GStreamer, Video4Linux2 and a few additional packages. To do this, run the following commands in the module’s terminal:

opkg update
 
opkg install gst-plugins-base-meta gst-plugins-good-meta gst-ffmpeg

Now with the command gst-inspect you can see a list of the plugins and elements which have been previously installed. Shown below is an example of the list of plugins and elements that have been previously installed.

Plugins and elements installed
Image 1: Some of the plugins and elements installed.
A Quick Look into Pipelines and its Elements

Based on the GStreamer Application Development Manual in Chapter 3, an element is the most important object class of GStreamer. Usually one creates a chain of connected elements and the data flows through these chained elements. An element has a specific function which can be reading from a file, data decoding, and displaying this data on the screen. Elements put together in a chain are called a pipeline, which can be used for a specific task, for example, video playback or capture. By default GStreamer contains a vast collection of elements, making it possible to develop a variety of media applications. Throughout this article we will use a few pipelines to give you a brief explanation about a few elements.

Below is an illustration of a pipeline for a basic Ogg player, using a demuxer and two branches, one each for audio and video. We can see that a few elements just have a src pad while others just have a sink pad or both src and sink pads.

Gstreamer pipeline for a simple ogg player
Image 2: Gstreamer pipeline for a simple ogg player. Image taken from http://gstreamer.freedesktop.org

For putting together a pipeline, one needs to check using gst-inspect if the desired plugins are compatible with one another. Let us take an example of the ffmpegcolorspace plugin. Just run the following in the terminal:

gst-inspect ffmpegcolorspace

And, you may see the following description of the plugin:

Factory Details:
  Long name: FFMPEG Colorspace converter
  Class: Filter/Converter/Video
  Description: Converts video from one colorspace to another
  Author(s): GStreamer maintainers 

As well as the source and sink capabilities of the plugin:

SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw-yuv
      video/x-raw-rgb
      video/x-raw-gray
 
SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw-yuv
      video/x-raw-rgb
      video/x-raw-gray

Another example is v4l2src. It has just the source capabilities, so it can source a video stream to another element. And inspecting the ximagesink element, we see that it has the rgb format sink capabilities.

Reading the documentation found in http://gstreamer.freedesktop.org/documentation/plugins.html and together with gst-inspect, allows to understand the capabilities of the elements and understand their properties.

Displaying a Video Test Pattern

To view a video test pattern, use the following pipeline:

gst-launch videotestsrc ! autovideosink
Video test pattern
Image 3: Video test pattern.

The autovideosink \element automatically detects video output. The videotestsrc element is used to generate a test video in a variety of formats and can be controlled with the property "pattern".

gst-launch videotestsrc pattern=snow ! autovideosink
Vídeo test with snow pattern
Image 4: Vídeo test with snow pattern.
Displaying Video From a Webcam

A plugged in webcam should appear as a device under /dev/videox where x can be 0, 1, 2, and so on, depending on the number of cameras connected to the module.

Connected capture devices
Image 5: Connected capture devices.

To view the webcam video in full screen, use the following pipeline:

gst-launch v4l2src device=/dev/video0 ! ffmpegcolorspace ! ximagesink
Webcam in fullscreen mode
Image 6: Webcam in fullscreen mode.

The following link shows the webcam video in full screen mode: video_webcam_fullscr.mp4

Video4Linux is an API and driver framework for capture and playback that supports several types of USB cameras and other devices. v4l2src is an element of the plugin Video4Linux2 that reads frames of a Video4Linux2 device which in our case is a webcam.

The ffmpegcolorspace element is a filter which is used to convert video frames from a wide variety of color formats. Cameras are providing data in YUV color format whereas displays typically operate using the RGB color format.

The ximagesink is a standard videosink element based on desktop X.

We can see that by using the top command, the memory and CPU processing used by gstreamer when displaying the webcam video. In this first case we have 77.9% CPU load.

top - 16:32:37 up  4:36,  2 users,  load average: 0.83, 0.28, 0.13
Tasks:  72 total,   1 running,  70 sleeping,   0 stopped,   1 zombie
Cpu(s): 69.3%us, 7.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 23.6%si,0.0%st
Mem:  235188k total,   107688k used,   127500k free,       0k buffers
Swap:   0k total,       0k used,      0k free,     61232k cached
 
PID  USER  PR  NI VIRT  RES  SHR  S %CPU %MEM   TIME+   COMMAND 
1509 root  20  0  39560 8688 6400 S 77.9  3.7   0:16.83 gst-launch-0.10 
663  root  20  0  17852 8552 4360 S 19.7  3.6   4:12.32 X 
1484 root  20  0  2116  1352 1144 R 1.3   0.6   0:00.80 top

It is also possible to display the video with some properties such as size and other parameters. The parameters' width, height, and framerate can be adjusted according to the settings supported by the camera. Execute the pipelines mentioned below for cameras 1, 2 or 3 respectively (in this case three cameras are connected to the module) to view the webcam video in a 320x240 format or specify the desired format.

gst-launch v4l2src device=/dev/video0 ! 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! ximagesink 
gst-launch v4l2src device=/dev/video1 ! 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! ximagesink
gst-launch v4l2src device=/dev/video2 ! 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! ximagesink
Displaying the webcam video on a 320x240 screen
Image 7: Displaying the webcam video on a 320x240 screen.

In this case CPU load was 28.2%.

Displaying Two Cameras Simultaneously

You can also view two cameras simultaneously. The following pipeline was tested using a Logitech HD 720p camera and a generic MJPEG USB camera module.

gst-launch v4l2src device=/dev/video0 !
 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace 
 ! ximagesink v4l2src device=/dev/video1 
 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! 
 ximagesink
Two screens playing simultaneously
Image 8: Two screens playing simultaneously.

The following link contains a video showing the two webcams playing simultaneously: video_duas_webcams.mp4

In this case we had 64.8% CPU load.

Recording Video

To record a video in MP4 format we used the following pipeline:

gst-launch --eos-on-shutdown v4l2src device=/dev/video0 ! ffmux_mp4 ! filesink location=video.mp4

The --eos-on-shutdown command is used to close the file correctly. The ffenc_mjpeg element is an encoder for MJPEG format. ffmux_mp4 is a muxer for the MP4 format. The filesink element indicates that the data from the source v4l2 should be stored in a file and is not displayed in ximagesink element. You can also specify the file destination path.

To analyze the quality of the video recorded by the module, you can view the video at the following link: recording_video1.mp4

Recording a video with the above pipeline resulted in around 8% CPU load.

Video Playback

To view the previously recorded video, we use the following pipeline:

gst-launch filesrc location=video.mp4 ! qtdemux name=demux demux.video_00 ! queue ! ffdec_mjpeg ! ffmpegcolorspace ! ximagesink 

In this case, the video source comes from a filesrc element, i.e., the video is coming from a file and not from a video device such as a webcam. This video has been encoded previously using mjpeg so ffdec_mjpeg decoder was used to decode the video.

Considering that the video was recorded with maximum resolution provided by the webcam, the CPU load was about 95%.

Video Playback Using HTTP

To view a video of a given URL use the following pipeline:

gst-launch souphttpsrc location=http://upload.wikimedia.org/wikipedia/commons/4/4b/MS_Diana_genom_Bergs_slussar_16_maj_2014.webm ! matroskademux name=demux demux.video_00 ! queue ! ffdec_vp8 ! ffmpegcolorspace ! ximagesink 

souphttpsrc is an element that receives data as a client over the network via HTTP. In this case the location property receives a URL that contains a video file instead of a file on your computer. Note that the decoder ffdec_vp8 was used to decode the webm format.

In this case we had around 40% CPU load.

Video Streaming From Webcam Using TCP

It is also possible to stream video from VF61 modules to a computer running Ubuntu Linux.
         IP VF61 = 192.168.0.8
         IP Ubuntu = 192.168.0.7
Run the following pipeline in VF61 module terminal:

gst-launch v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! ffenc_mjpeg ! tcpserversink host=192.168.0.7 port=5000

And the following pipeline in the host computer to view the streaming:

gst-launch tcpclientsrc host=192.168.0.8 port=5000 ! jpegdec ! autovideosink

Note that the IP addresses should be changed according to your setup. To check your IP address, run the ifconfig command in the terminal.

The following link shows a video demonstration of the example above: video_streaming_webcam.mp4
Using a Logitech HD 720p camera, we had around 65% CPU load.

D-Link IP Camera and VF61 module
Displaying Video From IP Camera

In this example, we have used an IP camera from D-Link model DCS-930L. The camera was set to stream video encoded in JPEG format with a resolution of 320x240 at 15 frames per second and an average quality JPEG. The performance of the processor differs according to the streaming settings of the IP camera.

Camera D-Link DCS-930L
Image 9: IP Camera D-Link DCS-930L.
Setup screen for D-Link IP Camera
Image 10: Setup screen for D-Link IP Camera.

To view the IP camera’s video, use the following pipeline in the module's terminal:

gst-launch -v souphttpsrc location='http://192.168.0.200/video.cgi' is-live=true ! multipartdemux ! decodebin2 ! ffmpegcolorspace ! ximagesink 
IP Camera streaming on module’s screen
Image 11: IP Camera streaming on module’s screen.

The following link shows a video demonstration of the performance of the above mentioned example: video_streaming_ipcam.mp4

IP Camera Recording

You can also record the IP camera video with VF61 module using the following pipeline:

gst-launch --eos-on-shutdown –v souphttpsrc location='http://192.168.0.200/video.cgi' is-live=true ! multipartdemux ! decodebin2 ! ffmpegcolorspace ! ffenc_mjpeg ! ffmux_mp4 ! filesink location=stream.mp4

To analyze the quality of the video, you can assess it at the following link: video_gravacao_ipcam.mp4
With the camera resolution set at 320x240, 15 frames per second and average quality JPEG, we had around 40% CPU load.

TCP Streaming From One IP to Another IP

In this example we will be streaming video from the IP camera to the VF61 module where it will stream again to another IP address.

The following pipeline was executed in the VF61 module terminal:

gst-launch --eos-on-shutdown –v souphttpsrc location='http://192.168.0.200/video.cgi' is-live=true ! multipartdemux ! decodebin2 ! ffmpegcolorspace ! ffenc_mjpeg ! Tcpserversink host=192.168.0.12 port 5000

In this case the CPU load was around 95%.

To view the streaming data from the module on a Linux computer, use the following pipeline:

gst-launch tcpclientsrc host=192.168.0.8 port=5000 ! jpegdec ! autovideosink
Below is a comparative table of the CPU and RAM in each test performed.
Test performed
%CPU %MEM
Displaying Video From a Webcam 28.2 2.7
Displaying Two Cameras Simultaneously 64.8 3.1
Recording Video 8.0 4.9
Video Playback 95.0 3.9
Video Playback Using HTTP 41.9 4.1
Video Streaming Using TCP 66.0 3.6
Displaying Video from IP camera 25.2 3.3
IP camera recording 40.1 3.9
TCP streaming IP to IP 95.2 4.8
Conclusion

The duration of activities carried out in this article, including the theoretical research and study of various elements of GStreamer, can take a few days. Studying the plugins and using gst-inspect, one can check the properties of each plugin and then build a compatible pipeline between elements. However, there is a wide range of examples available on the Internet which will probably help in constructing the pipelines. It may be necessary for the reader to perform multiple researches, as was done to carry out this article. Installing and checking the plugins properties was necessary for the construction of pipelines and to find GStreamer elements compatible with the setup used in this article.

The NXP/Freescale Vybrid VF61 processors can often meet various needs concerning media and video processing without having specific hardware for such tasks. We were really surprised with the performance of this processor when executing video and image-related activities.

The VF61 Vybrid processor is a good option for cost-sensitive embedded designs that are not required to perform video processing tasks regularly. But when there is need for a high video processing with better performance, like seen in embedded vision systems, we indicate processors like iMX6, also from NXP/Freescale, which has specific hardware for video processing.

References

This blog post was originally featured on Embarcados.com in Portuguese. See here.

Author Giovanni Bauermeister, Toradex Brasil

1 comments

KIms - 1 year 3 months | Reply

It would be neat to see a sister-article to this, that looks at a front+rear facing car dash-cam with GPS and text overlay on the recordings, with a focus on file management and handling continuous recordings.

Leave a comment

Your email ID will be kept confidential. Required fields are marked *


Bitte geben Sie die angezeigten Buchstaben ein. Gross/Kleinschreibung ist unbedeutend.

* Your comment will be reviewed and then added. Thank you.