Android and Raspberry Pi Camera low/no latency via gstreamer

Article is inspired by video https://youtu.be/xmE99sHBgy0 .

I will not go into detail what gstreamer is and how gstreamer works, because I assume you already know it. I assume you also you have a basic knowledge and experience with video streaming and RPi devices.

It’s very hard to measure and achieve low latency or even zero latency, so we need to focus on simplification and removing bottle necks in our gstreamer pipeline to get best performance.

Hardware used in experiment:

  1. Raspberry Pi device, model B/B+/B2 (https://pl.wikipedia.org/wiki/Raspberry_Pi) with Raspbian Wheezy or Jessie installed
  2. Raspberry Pi camera module (https://www.raspberrypi.org/help/camera-module-setup/)
  3. Samsung S4 with Android 4.4.2 installed

Software used:

  1. gstreamer1.0 (1.4.3)
  2. RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2)

Preparation:

  1. Connect camera module and launch RPi. Please rememebr to enable your camera module (https://www.raspberrypi.org/documentation/usage/camera/README.md)
  2. Install gstreamer 1.x on RPi
    sudo apt-get install gstreamer1.0
    
  3. Install RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2) on your Android device

Now you can try to stream video from your camera via gstreamer and it’s time to focus on latency. If you have already tried experiment with raspivid, netcat and mplayer, then you see how effective such combination can be. Mplayer has lowest latency itself, but mplayer is not a media framework you may want to use and integrate with your other systems. However, here is an experiment you may try to see results:

 raspivid -t 0 -hf -n -h 512 -w 512 -fps 15 -o - | nc 192.168.0.12 5001
 c:\nc111nt_rodneybeede\nc.exe -L -p 5001 | c:\mplayer-svn-36251\mplayer.exe -vo direct3d -fps 24 -cache 512 -

Where 192.168.0.12 is your Laptop IP.

In my experiment with gstreamer I focused on achieving the same performance as I observed with mplayer. I started reducing bottle necks from network, then by pipeline simplification. Here, Netcat due to it’s simplicity has better performance than gstreamer’s tcpserversink and tcpclientsink. I was surprised, but UDP sinks/clients were even worst. I modified pipeline by adding small queue with size=1 to keep only one frame in the buffer and drop all the rest. Subsequent pipeline elements, like videorate and autovideosync is all we need.

In both examples below two IPs were used: 192.168.0.13 is an Android device, 192.168.0.14 is RPi.

Low latency solution – TCP version (where android device is a host):

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false
    

    android-low-latency-camera-raspberry-pi-client

  2. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc 192.168.0.13 5001
    
  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Low latency solution – TCP version (where RPi is a host):

  1. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc -l -p 5001
  2. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpclientsrc host=192.168.0.14 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    android-low-latency-camera-raspberry-pi-host

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

To get even better performance, you can also play with videorate, like:

tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! videorate ! video/x-raw,framerate=60/1 ! autovideosink sync=false

In theory, setting framerate bigger than input streams’ rate could help, but personally I did not see any difference. However, it’s worth to try and see results.

Network bandwidth is satisfied as well (nmon output):

nmon_gstreamer

Low latency solution – UDP version:

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    udpsrc port=5000 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    Screenshot_2016-06-25-21-22-19

  2. On Raspberry Pi (where 192.168.0.13 is IP of your Android device):
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | gst-launch-1.0 --gst-debug=3 fdsrc ! udpsink host=192.168.0.13 port=5000

    low latency udp

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Summary

I hope this article can help you with effective video streaming with minimal latency. I’m happy to discuss, so feel free to comment or contact with me.

Conversion VC++ library to GNU (MinGW, GCC) format

Sometimes you have to use other compiler than VC++ (ok, be honest, in most cases 🙂 ), but what to do when your libs are in VC++ format? Solution is very easy, and it will be shown below on MySQL library case.

I assume you have mingw.

Download mingw-utils:
https://sourceforge.net/projects/mingw/files/MinGW/Extension/mingw-utils/mingw-utils-0.3/mingw-utils-0.3.tar.gz/download

reimp.exe -d libmysql.lib
dlltool.exe -k -d libmysql.def -l libmysql.a

That’s all folks 🙂

How to enable remote access to Xserver on Ubuntu 11.04 and Ubuntu 11.10

First of all remove -nolisten tcp from /etc/X11/xinit/xserverrc file:

cat /etc/X11/xinit/xserverrc 

result:

#!/bin/sh
#exec /usr/bin/X -nolisten tcp "$@"
exec /usr/bin/X "$@"

Then:
* Ubuntu 11.04 – with gdm

sudo vim /etc/gdm/custom.conf

File contents:

[security]
DisallowTCP=false

* Ubuntu 11.10 – with lightdm

sudo vim /etc/lightdm/lightdm.conf

File contents:

[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
xserver-allow-tcp=true

Other useful params of lightdm.conf file:
http://hmontoliu.blogspot.com/2011_10_01_archive.html

Sample rdesktop launcher

#!/bin/bash
#*************************************************
#	2011-10-10 By Piotr Zuk
#*************************************************
# Needed 3rdparty: 
# - http://www.semicomplete.com/projects/xdotool/
#*************************************************

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
TITLE="MyWindowsRemoteDesktop"

#kill previous instance
WID=`xdotool search "$TITLE" | head -1`
for id in $WID; do
  xdotool windowkill $id
done

rdesktop -u piotr.zuk -g 1280x1024 -a 24 -D -T $TITLE 10.2.135.133 &
usleep 800000
WID=`xdotool search "$TITLE" | head -1`
for id in $WID; do
  echo $id
  #xdotool windowsize $id 500 500
  xdotool windowmove $id 0 0
done

to enable sound, just add -rsound:local param.

How to set 24bit colors and antialiasing on WindowsXP client using rdesktop on Linux

Add Keys to Windows registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp]
“AllowFontAntiAlias”=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations]
"AllowFontAntiAlias"=dword:00000001

Run Group Policy Editor (exec gpedit.msc)
In the GPE:
Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Terminal Services
Once you click on Terminal Services look in the right TAB and search for Limit Maximum Color Depth. It will probably have status Disabled by default. Right click on it and go Properties. The Properties window will open up. On the Setting TAB select Enable and at Color Depth select whatever you want (i.e. 24 bit)