Android and Raspberry Pi Camera low/no latency via gstreamer

Article is inspired by video https://youtu.be/xmE99sHBgy0 .

I will not go into detail what gstreamer is and how gstreamer works, because I assume you already know it. I assume you also you have a basic knowledge and experience with video streaming and RPi devices.

It’s very hard to measure and achieve low latency or even zero latency, so we need to focus on simplification and removing bottle necks in our gstreamer pipeline to get best performance.

Hardware used in experiment:

1. Raspberry Pi device, model B/B+/B2 (https://pl.wikipedia.org/wiki/Raspberry_Pi) with Raspbian Wheezy or Jessie installed
2. Raspberry Pi camera module (https://www.raspberrypi.org/help/camera-module-setup/)
3. Samsung S4 with Android 4.4.2 installed

Software used:

1. gstreamer1.0 (1.4.3)
2. RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2)

Preparation:

1. Connect camera module and launch RPi. Please rememebr to enable your camera module (https://www.raspberrypi.org/documentation/usage/camera/README.md)
2. Install gstreamer 1.x on RPi

   sudo apt-get install gstreamer1.0
   

3. Install RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2) on your Android device

Now you can try to stream video from your camera via gstreamer and it’s time to focus on latency. If you have already tried experiment with raspivid, netcat and mplayer, then you see how effective such combination can be. Mplayer has lowest latency itself, but mplayer is not a media framework you may want to use and integrate with your other systems. However, here is an experiment you may try to see results:

 raspivid -t 0 -hf -n -h 512 -w 512 -fps 15 -o - | nc 192.168.0.12 5001
 c:\nc111nt_rodneybeede\nc.exe -L -p 5001 | c:\mplayer-svn-36251\mplayer.exe -vo direct3d -fps 24 -cache 512 -

Where 192.168.0.12 is your Laptop IP.

In my experiment with gstreamer I focused on achieving the same performance as I observed with mplayer. I started reducing bottle necks from network, then by pipeline simplification. Here, Netcat due to it’s simplicity has better performance than gstreamer’s tcpserversink and tcpclientsink. I was surprised, but UDP sinks/clients were even worst. I modified pipeline by adding small queue with size=1 to keep only one frame in the buffer and drop all the rest. Subsequent pipeline elements, like videorate and autovideosync is all we need.

In both examples below two IPs were used: 192.168.0.13 is an Android device, 192.168.0.14 is RPi.

Low latency solution – TCP version (where android device is a host):

1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false
   

   

2. On Raspberry Pi:

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc 192.168.0.13 5001
   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

Low latency solution – TCP version (where RPi is a host):

1. On Raspberry Pi:

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc -l -p 5001

2. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   tcpclientsrc host=192.168.0.14 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

To get even better performance, you can also play with videorate, like:

tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! videorate ! video/x-raw,framerate=60/1 ! autovideosink sync=false

In theory, setting framerate bigger than input streams’ rate could help, but personally I did not see any difference. However, it’s worth to try and see results.

Network bandwidth is satisfied as well (nmon output):

Low latency solution – UDP version:

1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   udpsrc port=5000 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

   

2. On Raspberry Pi (where 192.168.0.13 is IP of your Android device):

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | gst-launch-1.0 --gst-debug=3 fdsrc ! udpsink host=192.168.0.13 port=5000

   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

Summary

I hope this article can help you with effective video streaming with minimal latency. I’m happy to discuss, so feel free to comment or contact with me.

How to compile boost for QT MSVC2012 x64

Download boost

open “Qt 5.2.1 64-bit for Desktop (MSVC 2012)” command prompt (or “VS2012 x64 Cross Tools Command Prompt”)

Execute .\bootstrap.bat

Execute b2 address-model=64 link=static toolset=msvc-11.0 --build-type=complete stage --with-filesystem --with-signals --with-system --with-thread --with-date_time --with-log

HowTo hibernate or suspend Linux from command line

pmi action suspend
sudo /etc/acpi/sleep.sh
echo 'pmi action suspend' | at now + 40 minutes
gnome-power-cmd suspend

pmi action hibernate
gnome-power-cmd hibernate

#to lock gnome session first
gnome-screensaver-command --lock

Conversion VC++ library to GNU (MinGW, GCC) format

Sometimes you have to use other compiler than VC++ (ok, be honest, in most cases 🙂 ), but what to do when your libs are in VC++ format? Solution is very easy, and it will be shown below on MySQL library case.

I assume you have mingw.

Download mingw-utils:
https://sourceforge.net/projects/mingw/files/MinGW/Extension/mingw-utils/mingw-utils-0.3/mingw-utils-0.3.tar.gz/download

reimp.exe -d libmysql.lib
dlltool.exe -k -d libmysql.def -l libmysql.a

That’s all folks 🙂

How to compile libjpeg using mingw

Downlaod latest MinGW from sourceforge: https://sourceforge.net/projects/mingw/files/ and install it to c:\MinGW

Download latest libjpeg sources from: http://www.ijg.org/files/ and untar it to c:\MinGW\msys\1.0\home\

Launch msys shell: c:\MinGW\msys\1.0\msys.bat
cd /home//libjpeg-
./configure
make
make install

now in .libs folder we will have files: libjpeg.a, libjpeg.dll.a and libjpeg-7.dll

How to make FreeTDS, unixODBC and QT working together

UnixODBC and FreeTDS driver compialtion and configuration has been taken from the http://it.toolbox.com/wiki/index.php/Ubuntu_Debian_FreeTDS_ODBC

Here are instructions for getting tsql and isql to play nice together.
Apt-getting doesn’t always get you all the tools you need to develop against SQLserver and Sybase.
Gather some important packages:

apt-get install libtool bison autotools-dev g++ build-essential tcsh unixodbc-dev tdsodbc

Download, extract and compile FreeTDS driver:

wget ftp://ftp.ibiblio.org/pub/Linux/ALPHA/freetds/stable/freetds-stable.tgz
tar xzvf freetds-stable.tgz
cd freetds-0.82/
./configure --prefix=/usr --sysconfdir=/etc --with-unixodbc=/usr --with-tdsver=8.0
 make && make install clean

Create a file named tds.driver.template with the following contents:

[FreeTDS]
Description     = v0.82 with protocol v8.0
Driver          = /usr/lib/libtdsodbc.so

Register the driver with ODBC

odbcinst -i -d -f tds.driver.template

On 64 bit Debian, I’ve run into the situation where freetds won’t compile against the distribution’s ODBC. To resolve this, it was necessary to compile and install a parallel ODBC and compile freetds against it.
If this is a server, you may not have a GUI, so don’t try to compile with it.

wget http://www.unixodbc.org/unixODBC-2.2.14.tar.gz
tar xvf unixODBC-2.2.14.tar
cd unixODBC-2.2.14
./configure --enable-gui=no
make
make install

Default installation dir is: /usr/local/bin/. Now, follow the above instructions replacing the configure line for freetds with this:

./configure --prefix=/usr --sysconfdir=/etc --with-unixodbc=/usr/local --with-tdsver=8.0

Possible errors:

• SQLInstallDriverEx failed with Invalid install path

. Probably /usr/local/etc/odbcinst.ini file is missing, or use root account to install driver template.

How to test unixODBC:

odbcinst -s -q	show available ODBC sources
odbcinst -d -q	show available ODBC drivers
odbcinst -j	show config
isql -v datasourcename login password	conenct using data source name

How to test freetds:

tsql -H 127.0.0.1 -p 1433 -U sa -P password	conenct using host name
tsql -S datasourcename -U sa -P password	conenct using data source name from the freetds.conf file

Configure QT to work with unixODBC

Note that QT application will use files:
/usr/local/etc/odbcinst.ini
/usr/local/etc/odbc.ini

, but isql uses:
/etc/odbcinst.ini
/etc/odbc.ini

The best solution is to create symbolic links and keep orginal files in one place.

cd $QT/src/plugins/sqldrivers/odbc
qmake "INCLUDEPATH+=/usr/local/unixODBC/include" "LIBS+=-L/usr/local/unixODBC/lib -lodbc"
make
make install

On Ubuntu x64 we need to modify following files manually:
sudo vim /usr/local/include/sqltypes.h

#ifdef ODBCINT64
//typedef ODBCINT64     SQLBIGINT;
typedef long    SQLBIGINT;
#endif
#ifdef UODBCINT64
//typedef UODBCINT64    SQLUBIGINT;
typedef unsigned long   SQLUBIGINT;
#endif

vim ../../../sql/drivers/odbc/qsql_odbc.cpp

when connection fails, then try to update add LD_LIBRARY_PATH in Qt project settings
LD_LIBRARY_PATH = /usr/lib:/usr/lib/odbc:/usr/local/lib:/usr/local/Trolltech/Qt-4.7.4/lib

cd $QT/src/plugins/sqldrivers/tds
qmake "INCLUDEPATH+=$QT/QtSources/4.8.1/include" "LIBS+=-L/usr/local/unixODBC/lib -lodbc"
make 
make install

qDebug() << QSqlDatabase::drivers();

result:
(“QSQLITE”, “QODBC3”, “QODBC”)

QSqlDatabase db = QSqlDatabase::addDatabase("QODBC3");
db.setConnectOptions("Driver={FreeTDSDriver};NetworkAddress=10.7.111.149,4101;Database=kplus;UID=kplus;PWD=kplus11;");
db.setUserName("kplus");
db.setPassword("kplus11");
if( !db.open() )
{
	qDebug() << "Connection errror:"  << db.lastError().text();
}
else
{
	QSqlQuery query("SELECT country FROM artist", db);
	while (query.next())
	{
		QString country = query.value(0).toString();
		qDebug() << country;
	}
}

/etc/odbc.ini
[blee]
#Driver = /usr/lib/libtdsodbc.so
Driver = FreeTDSDriver
Description = Northwind sample database
Trace = Yes
Server = 10.7.111.149
Port = 4101
Database = kplus
UID = kplus

cat /etc/odbcinst.ini
[FreeTDSDriver]
Description = v0.82 with protocol v8.0
Driver = /usr/lib/libtdsodbc.so
UsageCount = 2
CPTimeout =
CPReuse =

/etc/freetds.conf
[blee]
host = 10.7.111.149
port = 4101
tds version = 5.0
dump file = /var/log/freetds.log

References:
http://it.toolbox.com/wiki/index.php/Ubuntu_Debian_FreeTDS_ODBC
http://www.savelono.com/linux/how-to-configure-linux-odbc-connections-for-ms-sql.html
http://www.webaj.com/how-setup-mysql-dsn-datasbase-source-centos-myodbc-and-unixodbc-command-line.htm
http://www.clarksys.com/blog/2010/10/25/howto-configure-unix-odbc/

How to generate C++ .h file with current SVN revision number

@ECHO OFF
for /f "delims=: tokens=1,2" %%a in ('svn info') do (
  if "%%a"=="Revision" (
    set /a RELEASE_REVISION=%%b
  )
)

set SVNVERSIONFILE="svn_current_version.h"
echo #ifndef SVN_CURRENT_VERSION_INFO_H > %SVNVERSIONFILE%
echo #define SVN_CURRENT_VERSION_INFO_H >> %SVNVERSIONFILE%
echo #define SVN_CURRENT_VERSION %RELEASE_REVISION% >> %SVNVERSIONFILE%
echo #endif >> %SVNVERSIONFILE%

How to enable remote access to Xserver on Ubuntu 11.04 and Ubuntu 11.10

First of all remove -nolisten tcp from /etc/X11/xinit/xserverrc file:

cat /etc/X11/xinit/xserverrc 

result:

#!/bin/sh
#exec /usr/bin/X -nolisten tcp "$@"
exec /usr/bin/X "$@"

Then:
* Ubuntu 11.04 – with gdm

sudo vim /etc/gdm/custom.conf

File contents:

[security]
DisallowTCP=false

* Ubuntu 11.10 – with lightdm

sudo vim /etc/lightdm/lightdm.conf

File contents:

[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
xserver-allow-tcp=true

Other useful params of lightdm.conf file:
http://hmontoliu.blogspot.com/2011_10_01_archive.html

Sample rdesktop launcher

#!/bin/bash
#*************************************************
#	2011-10-10 By Piotr Zuk
#*************************************************
# Needed 3rdparty: 
# - http://www.semicomplete.com/projects/xdotool/
#*************************************************

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
TITLE="MyWindowsRemoteDesktop"

#kill previous instance
WID=`xdotool search "$TITLE" | head -1`
for id in $WID; do
  xdotool windowkill $id
done

rdesktop -u piotr.zuk -g 1280x1024 -a 24 -D -T $TITLE 10.2.135.133 &
usleep 800000
WID=`xdotool search "$TITLE" | head -1`
for id in $WID; do
  echo $id
  #xdotool windowsize $id 500 500
  xdotool windowmove $id 0 0
done

to enable sound, just add -rsound:local param.

How to set 24bit colors and antialiasing on WindowsXP client using rdesktop on Linux

Add Keys to Windows registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp]
“AllowFontAntiAlias”=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations]
"AllowFontAntiAlias"=dword:00000001


Run Group Policy Editor (exec gpedit.msc)
In the GPE:
Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Terminal Services
Once you click on Terminal Services look in the right TAB and search for Limit Maximum Color Depth. It will probably have status Disabled by default. Right click on it and go Properties. The Properties window will open up. On the Setting TAB select Enable and at Color Depth select whatever you want (i.e. 24 bit)