Android and Raspberry Pi Camera low/no latency via gstreamer

Article is inspired by video https://youtu.be/xmE99sHBgy0 .

I will not go into detail what gstreamer is and how gstreamer works, because I assume you already know it. I assume you also you have a basic knowledge and experience with video streaming and RPi devices.

It’s very hard to measure and achieve low latency or even zero latency, so we need to focus on simplification and removing bottle necks in our gstreamer pipeline to get best performance.

Hardware used in experiment:

  1. Raspberry Pi device, model B/B+/B2 (https://pl.wikipedia.org/wiki/Raspberry_Pi) with Raspbian Wheezy or Jessie installed
  2. Raspberry Pi camera module (https://www.raspberrypi.org/help/camera-module-setup/)
  3. Samsung S4 with Android 4.4.2 installed

Software used:

  1. gstreamer1.0 (1.4.3)
  2. RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2)

Preparation:

  1. Connect camera module and launch RPi. Please rememebr to enable your camera module (https://www.raspberrypi.org/documentation/usage/camera/README.md)
  2. Install gstreamer 1.x on RPi
    sudo apt-get install gstreamer1.0
    
  3. Install RaspberryPi Camera Viewer (https://play.google.com/store/apps/details?id=pl.effisoft.rpicamviewer2) on your Android device

Now you can try to stream video from your camera via gstreamer and it’s time to focus on latency. If you have already tried experiment with raspivid, netcat and mplayer, then you see how effective such combination can be. Mplayer has lowest latency itself, but mplayer is not a media framework you may want to use and integrate with your other systems. However, here is an experiment you may try to see results:

 raspivid -t 0 -hf -n -h 512 -w 512 -fps 15 -o - | nc 192.168.0.12 5001
 c:\nc111nt_rodneybeede\nc.exe -L -p 5001 | c:\mplayer-svn-36251\mplayer.exe -vo direct3d -fps 24 -cache 512 -

Where 192.168.0.12 is your Laptop IP.

In my experiment with gstreamer I focused on achieving the same performance as I observed with mplayer. I started reducing bottle necks from network, then by pipeline simplification. Here, Netcat due to it’s simplicity has better performance than gstreamer’s tcpserversink and tcpclientsink. I was surprised, but UDP sinks/clients were even worst. I modified pipeline by adding small queue with size=1 to keep only one frame in the buffer and drop all the rest. Subsequent pipeline elements, like videorate and autovideosync is all we need.

In both examples below two IPs were used: 192.168.0.13 is an Android device, 192.168.0.14 is RPi.

Low latency solution – TCP version (where android device is a host):

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false
    

    android-low-latency-camera-raspberry-pi-client

  2. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc 192.168.0.13 5001
    
  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Low latency solution – TCP version (where RPi is a host):

  1. On Raspberry Pi:
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | nc -l -p 5001
  2. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    tcpclientsrc host=192.168.0.14 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    android-low-latency-camera-raspberry-pi-host

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

To get even better performance, you can also play with videorate, like:

tcpserversrc host=192.168.0.13 port=5001 ! queue2 max-size-buffers=1 ! decodebin ! videorate ! video/x-raw,framerate=60/1 ! autovideosink sync=false

In theory, setting framerate bigger than input streams’ rate could help, but personally I did not see any difference. However, it’s worth to try and see results.

Network bandwidth is satisfied as well (nmon output):

nmon_gstreamer

Low latency solution – UDP version:

  1. On Android device, open RaspberryPi Camera Viewer and run following pipeline:
    udpsrc port=5000 ! queue2 max-size-buffers=1 ! decodebin ! autovideosink sync=false

    Screenshot_2016-06-25-21-22-19

  2. On Raspberry Pi (where 192.168.0.13 is IP of your Android device):
    raspivid -t 0 -hf -n -h 480 -w 640 -fps 15 -o - | gst-launch-1.0 --gst-debug=3 fdsrc ! udpsink host=192.168.0.13 port=5000

    low latency udp

  3. click play button on the center of preview window in RaspberryPi Camera Viewer
    android-play-button

Summary

I hope this article can help you with effective video streaming with minimal latency. I’m happy to discuss, so feel free to comment or contact with me.

Advertisements

8 thoughts on “Android and Raspberry Pi Camera low/no latency via gstreamer

  1. It should not be TCP!!! You should use UDP, that lowers again the latency as the receiving end doesn’t have to send confirmation packets to the sender.

    • Totally agree, TCP comes with big overhead. I updated post above with UDP case. Like I said above, I achieved very bad results with 1.3 UDP client. I’ll take a look deeply on gstreamer differences between 1.3 and 1.6 – maybe I find something interesting?

  2. Would be great if it is possible to use UDP streaming. For some applications like FPV latency is higher priority then quality.
    Tried many diferent ways to use UDP but simply couldnt…

  3. Did you manage to use Raspberry Pi Camera Viewer through UDP?
    I can perfectly use it through TCP but not thorough UDP….

    • When I wrote this article, I wasn’t able to achieve lower latency with UDP than TCP – what is strange, I know. Anyway, it was few months ago with version 1.3. Yesterday, thanks to cduvivier, I did new try with gstreamer 1.6 and I achieved very good results. I updated post above with steps I made.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s