Android and Raspberry Pi Camera low/no latency via gstreamer

Article is inspired by video https://youtu.be/xmE99sHBgy0 .

I will not go into detail what gstreamer is and how gstreamer works, because I assume you already know it. I assume you also you have a basic knowledge and experience with video streaming and RPi devices.

It’s very hard to measure and achieve low latency or even zero latency, so we need to focus on simplification and removing bottle necks in our gstreamer pipeline to get best performance.

Hardware used in experiment:

  1. Raspberry Pi device, model B/B+/B2 (https://pl.wikipedia.org/wiki/Raspberry_Pi) with Raspbian Wheezy or Jessie installed
  2. Raspberry Pi camera module (https://www.raspberrypi.org/help/camera-module-setup/)
  3. Samsung S4 with Android 4.4.2 installed

Software used:

  1. gstreamer1.0 (1.4.3)
  2. RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2)

Preparation:

  1. Connect camera module and launch RPi. Please rememebr to enable your camera module (https://www.raspberrypi.org/documentation/usage/camera/README.md)
  2. Install gstreamer 1.x on RPi
    sudo apt-get install gstreamer1.0
    
  3. Install RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2) on your Android device

Now you can try to stream video from your camera via gstreamer and it’s time to focus on latency. If you have already tried experiment with raspivid, netcat and mplayer, then you see how effective such combination can be. Mplayer has lowest latency itself, but mplayer is not a media framework you may want to use and integrate with your other systems. However, here is an experiment you may try to see results:

 raspivid -t 0 -hf -n -h 512 -w 512 -fps 15 -o - | nc 192.168.0.12 5001
 c:\nc111nt_rodneybeede\nc.exe -L -p 5001 | c:\mplayer-svn-36251\mplayer.exe -vo direct3d -fps 24 -cache 512 -

Where 192.168.0.12 is your Laptop IP.

In my experiment with gstreamer I focused on achieving the same performance as I observed with mplayer. I started reducing bottle necks from network, then by pipeline simplification. Here, Netcat due to it’s simplicity has better performance than gstreamer’s tcpserversink and tcpclientsink. I was surprised, but UDP sinks/clients were even worst. I modified pipeline by adding small queue with size=1 to keep only one frame in the buffer and drop all the rest. Subsequent pipeline elements, like videorate and autovideosync is all we need.

In both examples below two IPs were used: 192.168.0.13 is an Android device, 192.168.0.14 is RPi.

Low latency solution – TCP version (where android device is a host):

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false
    

    android-low-latency-camera-raspberry-pi-client

  2. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc 192.168.0.13 5001
    
  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Low latency solution – TCP version (where RPi is a host):

  1. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc -l -p 5001
  2. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpclientsrc host=192.168.0.14 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    android-low-latency-camera-raspberry-pi-host

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

To get even better performance, you can also play with videorate, like:

tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! videorate ! video/x-raw,framerate=60/1 ! autovideosink sync=false

In theory, setting framerate bigger than input streams’ rate could help, but personally I did not see any difference. However, it’s worth to try and see results.

Network bandwidth is satisfied as well (nmon output):

nmon_gstreamer

Low latency solution – UDP version:

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    udpsrc port=5000 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    Screenshot_2016-06-25-21-22-19

  2. On Raspberry Pi (where 192.168.0.13 is IP of your Android device):
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | gst-launch-1.0 --gst-debug=3 fdsrc ! udpsink host=192.168.0.13 port=5000

    low latency udp

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Summary

I hope this article can help you with effective video streaming with minimal latency. I’m happy to discuss, so feel free to comment or contact with me.

Advertisements

How to compile boost for QT MSVC2012 x64

  • Download boost
  • open “Qt 5.2.1 64-bit for Desktop (MSVC 2012)” command prompt (or “VS2012 x64 Cross Tools Command Prompt”)
  • Execute .\bootstrap.bat
  • Execute b2 address-model=64 link=static toolset=msvc-11.0 --build-type=complete stage --with-filesystem --with-signals --with-system --with-thread --with-date_time --with-log
  • Conversion VC++ library to GNU (MinGW, GCC) format

    Sometimes you have to use other compiler than VC++ (ok, be honest, in most cases 🙂 ), but what to do when your libs are in VC++ format? Solution is very easy, and it will be shown below on MySQL library case.

    I assume you have mingw.

    Download mingw-utils:
    https://sourceforge.net/projects/mingw/files/MinGW/Extension/mingw-utils/mingw-utils-0.3/mingw-utils-0.3.tar.gz/download

    reimp.exe -d libmysql.lib
    dlltool.exe -k -d libmysql.def -l libmysql.a

    That’s all folks 🙂

    How to compile libjpeg using mingw

    Downlaod latest MinGW from sourceforge: https://sourceforge.net/projects/mingw/files/ and install it to c:\MinGW

    Download latest libjpeg sources from: http://www.ijg.org/files/ and untar it to c:\MinGW\msys\1.0\home\

    Launch msys shell: c:\MinGW\msys\1.0\msys.bat
    cd /home//libjpeg-
    ./configure
    make
    make install

    now in .libs folder we will have files: libjpeg.a, libjpeg.dll.a and libjpeg-7.dll

    How to make FreeTDS, unixODBC and QT working together

    UnixODBC and FreeTDS driver compialtion and configuration has been taken from the http://it.toolbox.com/wiki/index.php/Ubuntu_Debian_FreeTDS_ODBC

    Here are instructions for getting tsql and isql to play nice together.
    Apt-getting doesn’t always get you all the tools you need to develop against SQLserver and Sybase.
    Gather some important packages:

    apt-get install libtool bison autotools-dev g++ build-essential tcsh unixodbc-dev tdsodbc
    

    Download, extract and compile FreeTDS driver:

    wget ftp://ftp.ibiblio.org/pub/Linux/ALPHA/freetds/stable/freetds-stable.tgz
    tar xzvf freetds-stable.tgz
    cd freetds-0.82/
    ./configure --prefix=/usr --sysconfdir=/etc --with-unixodbc=/usr --with-tdsver=8.0
     make && make install clean
    

    Create a file named tds.driver.template with the following contents:

    [FreeTDS]
    Description     = v0.82 with protocol v8.0
    Driver          = /usr/lib/libtdsodbc.so
    

    Register the driver with ODBC

    odbcinst -i -d -f tds.driver.template
    

    On 64 bit Debian, I’ve run into the situation where freetds won’t compile against the distribution’s ODBC. To resolve this, it was necessary to compile and install a parallel ODBC and compile freetds against it.
    If this is a server, you may not have a GUI, so don’t try to compile with it.

    wget http://www.unixodbc.org/unixODBC-2.2.14.tar.gz
    tar xvf unixODBC-2.2.14.tar
    cd unixODBC-2.2.14
    ./configure --enable-gui=no
    make
    make install
    

    Default installation dir is: /usr/local/bin/. Now, follow the above instructions replacing the configure line for freetds with this:

    ./configure --prefix=/usr --sysconfdir=/etc --with-unixodbc=/usr/local --with-tdsver=8.0
    

    Possible errors:

    • SQLInstallDriverEx failed with Invalid install path
    • . Probably /usr/local/etc/odbcinst.ini file is missing, or use root account to install driver template.

    How to test unixODBC:

    odbcinst -s -q show available ODBC sources
    odbcinst -d -q show available ODBC drivers
    odbcinst -j show config
    isql -v datasourcename login password conenct using data source name

    How to test freetds:

    tsql -H 127.0.0.1 -p 1433 -U sa -P password conenct using host name
    tsql -S datasourcename -U sa -P password conenct using data source name from the freetds.conf file

    Configure QT to work with unixODBC

    Note that QT application will use files:
    /usr/local/etc/odbcinst.ini
    /usr/local/etc/odbc.ini

    , but isql uses:
    /etc/odbcinst.ini
    /etc/odbc.ini

    The best solution is to create symbolic links and keep orginal files in one place.

    cd $QT/src/plugins/sqldrivers/odbc
    qmake "INCLUDEPATH+=/usr/local/unixODBC/include" "LIBS+=-L/usr/local/unixODBC/lib -lodbc"
    make
    make install
    

    On Ubuntu x64 we need to modify following files manually:
    sudo vim /usr/local/include/sqltypes.h

    #ifdef ODBCINT64
    //typedef ODBCINT64     SQLBIGINT;
    typedef long    SQLBIGINT;
    #endif
    #ifdef UODBCINT64
    //typedef UODBCINT64    SQLUBIGINT;
    typedef unsigned long   SQLUBIGINT;
    #endif
    

    vim ../../../sql/drivers/odbc/qsql_odbc.cpp

    when connection fails, then try to update add LD_LIBRARY_PATH in Qt project settings
    LD_LIBRARY_PATH = /usr/lib:/usr/lib/odbc:/usr/local/lib:/usr/local/Trolltech/Qt-4.7.4/lib

    cd $QT/src/plugins/sqldrivers/tds
    qmake "INCLUDEPATH+=$QT/QtSources/4.8.1/include" "LIBS+=-L/usr/local/unixODBC/lib -lodbc"
    make 
    make install
    
    qDebug() << QSqlDatabase::drivers();
    

    result:
    (“QSQLITE”, “QODBC3”, “QODBC”)

    QSqlDatabase db = QSqlDatabase::addDatabase("QODBC3");
    db.setConnectOptions("Driver={FreeTDSDriver};NetworkAddress=10.7.111.149,4101;Database=kplus;UID=kplus;PWD=kplus11;");
    db.setUserName("kplus");
    db.setPassword("kplus11");
    if( !db.open() )
    {
    	qDebug() << "Connection errror:"  << db.lastError().text();
    }
    else
    {
    	QSqlQuery query("SELECT country FROM artist", db);
    	while (query.next())
    	{
    		QString country = query.value(0).toString();
    		qDebug() << country;
    	}
    }
    

    /etc/odbc.ini
    [blee]
    #Driver = /usr/lib/libtdsodbc.so
    Driver = FreeTDSDriver
    Description = Northwind sample database
    Trace = Yes
    Server = 10.7.111.149
    Port = 4101
    Database = kplus
    UID = kplus

    cat /etc/odbcinst.ini
    [FreeTDSDriver]
    Description = v0.82 with protocol v8.0
    Driver = /usr/lib/libtdsodbc.so
    UsageCount = 2
    CPTimeout =
    CPReuse =

    /etc/freetds.conf
    [blee]
    host = 10.7.111.149
    port = 4101
    tds version = 5.0
    dump file = /var/log/freetds.log

    References:
    http://it.toolbox.com/wiki/index.php/Ubuntu_Debian_FreeTDS_ODBC
    http://www.savelono.com/linux/how-to-configure-linux-odbc-connections-for-ms-sql.html
    http://www.webaj.com/how-setup-mysql-dsn-datasbase-source-centos-myodbc-and-unixodbc-command-line.htm
    http://www.clarksys.com/blog/2010/10/25/howto-configure-unix-odbc/

    How to generate C++ .h file with current SVN revision number

    @ECHO OFF
    for /f "delims=: tokens=1,2" %%a in ('svn info') do (
      if "%%a"=="Revision" (
        set /a RELEASE_REVISION=%%b
      )
    )
    
    set SVNVERSIONFILE="svn_current_version.h"
    echo #ifndef SVN_CURRENT_VERSION_INFO_H > %SVNVERSIONFILE%
    echo #define SVN_CURRENT_VERSION_INFO_H >> %SVNVERSIONFILE%
    echo #define SVN_CURRENT_VERSION %RELEASE_REVISION% >> %SVNVERSIONFILE%
    echo #endif >> %SVNVERSIONFILE%