Android and Raspberry Pi Camera low/no latency via gstreamer

Article is inspired by video https://youtu.be/xmE99sHBgy0 .

I will not go into detail what gstreamer is and how gstreamer works, because I assume you already know it. I assume you also you have a basic knowledge and experience with video streaming and RPi devices.

It’s very hard to measure and achieve low latency or even zero latency, so we need to focus on simplification and removing bottle necks in our gstreamer pipeline to get best performance.

Hardware used in experiment:

1. Raspberry Pi device, model B/B+/B2 (https://pl.wikipedia.org/wiki/Raspberry_Pi) with Raspbian Wheezy or Jessie installed
2. Raspberry Pi camera module (https://www.raspberrypi.org/help/camera-module-setup/)
3. Samsung S4 with Android 4.4.2 installed

Software used:

1. gstreamer1.0 (1.4.3)
2. RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2)

Preparation:

1. Connect camera module and launch RPi. Please rememebr to enable your camera module (https://www.raspberrypi.org/documentation/usage/camera/README.md)
2. Install gstreamer 1.x on RPi

   sudo apt-get install gstreamer1.0
   

3. Install RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2) on your Android device

Now you can try to stream video from your camera via gstreamer and it’s time to focus on latency. If you have already tried experiment with raspivid, netcat and mplayer, then you see how effective such combination can be. Mplayer has lowest latency itself, but mplayer is not a media framework you may want to use and integrate with your other systems. However, here is an experiment you may try to see results:

 raspivid -t 0 -hf -n -h 512 -w 512 -fps 15 -o - | nc 192.168.0.12 5001
 c:\nc111nt_rodneybeede\nc.exe -L -p 5001 | c:\mplayer-svn-36251\mplayer.exe -vo direct3d -fps 24 -cache 512 -

Where 192.168.0.12 is your Laptop IP.

In my experiment with gstreamer I focused on achieving the same performance as I observed with mplayer. I started reducing bottle necks from network, then by pipeline simplification. Here, Netcat due to it’s simplicity has better performance than gstreamer’s tcpserversink and tcpclientsink. I was surprised, but UDP sinks/clients were even worst. I modified pipeline by adding small queue with size=1 to keep only one frame in the buffer and drop all the rest. Subsequent pipeline elements, like videorate and autovideosync is all we need.

In both examples below two IPs were used: 192.168.0.13 is an Android device, 192.168.0.14 is RPi.

Low latency solution – TCP version (where android device is a host):

1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false
   

   

2. On Raspberry Pi:

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc 192.168.0.13 5001
   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

Low latency solution – TCP version (where RPi is a host):

1. On Raspberry Pi:

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc -l -p 5001

2. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   tcpclientsrc host=192.168.0.14 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

To get even better performance, you can also play with videorate, like:

tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! videorate ! video/x-raw,framerate=60/1 ! autovideosink sync=false

In theory, setting framerate bigger than input streams’ rate could help, but personally I did not see any difference. However, it’s worth to try and see results.

Network bandwidth is satisfied as well (nmon output):

Low latency solution – UDP version:

1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:

   udpsrc port=5000 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

   

2. On Raspberry Pi (where 192.168.0.13 is IP of your Android device):

   raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | gst-launch-1.0 --gst-debug=3 fdsrc ! udpsink host=192.168.0.13 port=5000

   

3. click play button on the center of preview window in RaspberryPi Camera Viewer
   

Summary

I hope this article can help you with effective video streaming with minimal latency. I’m happy to discuss, so feel free to comment or contact with me.