Sensor Stream Server

High-level overview of what Sensor Stream Server does
1.
Sensor Stream Server is started with a path to a configuration yaml file as a command line argument
2.
The configuration file defines:
Where the data should be streamed to (destination)
Frame Source (where the data is coming from - frame list/video/kinect/iphone)
What frame types will be streamed (color/depth/ir/confidence)
Encoders for each of the frame types ( null/libav/nvenc/zdepth)
3.
IReader is constructed with parameters defined in the configuration file
The instantiated IReader's frame types are defined by a vector of integers returned by GetType() method
0 - if exists in vector then color is being sent (encoders available: null/libav/nvenc)
1 - if exists in vector then depth is being sent (encoders available: null/libav/nvenc/zdepth)
2 - if exists in vector then ir is being sent (encoders available: null/libav/nvenc)
3 - if exists in vector then confidence is being sent (encoders available: null)
IReader will use GetCurrentFrame() in Step #5 to grab data from the frame source and create a FrameStruct for each defined frame type
4.
Server creates a vector of IEncoders for each frame type returned by IReader's GetType()
The IEncoders will be used to encode FrameStructs in Step #5
NullEncoder - passes raw data through
LibAvEncoder - uses LibAv to encode video frames
NvEncoder - uses Nvidia Pipe to encode video frames
ZDepthEncoder - uses Zdepth to encode depth frames 
5.
Server creates a vector of encoded FrameStructs
Server is paced to transmit frames only as fast as the fps allows
IReader creates a vector of FrameStructs
One for each frame type returned by GetType()
IReader.GetCurrentFrame() - Fills FrameStruct
if (IReader.HasNextFrame()) - Used to iterate
IReader.NextFrame() - Used to iterate
Server iterates over vector of FrameStructs and  IEncoder which returns an encoded FrameStruct
IEncoder.AddFrameStruct() - Attempts encoding FrameStruct
IEncoder.HasNextPacket() - Checks if encoded frame available
IEncoder.CurrentFrameEncoded() - Returns encoded frame
6.
The vector of encoded FrameStructs is sent to destination zmq socket as a zmq message
Call CerealStructToString() on vector of encoded frames to create string message
Create zmq message out of the string message
Call socket.send() passing through the zmq message
Log timings
Currently the frame rate 
Detailed overview of Sensor Stream Server data flow
The Sensor Stream Server can stream from 4 frame (data) sources:
Images: frames from images stored on the disk.
Video: encoded frames that have been captured using a color/depth camera, such as the Kinect. All video types supported by FFmpeg/Libav can be processed.
Kinect: Live Kinect DK frame data.
iPhone: Live iPhone RGBD data (must have lidar)
Each of these data sources have their own implementation of IReaderdefined in image_reader.h which is able to create read data and write to FrameStructs.
ImageReader: Can be used to stream public datasets and can be composed into MultiImageReader
VideoFileReader: Reads .mkv files like those created by Azure Kinect DK recorder​
KinectReader: Reads Azure Kinect DK stream
IPhoneReader: Reads ARFrame data
For all four data sources, the data can be sent losslessly (very high bandwidth requirements), or compressed (20-50x lower bandwidth requirements), using Libav or NVCodec through the NVPipe.
As any compression will affect quality, we recommend first experimenting with the Sensor Stream Tester to figure out the optimal levels for your use case.
Server File: https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/servers/ssp_server.cc​
Building Sensor Stream Server
Building Sensor Stream Server also builds Sensor Stream Client and Sensor Stream Tester.
1
git clone [email protected]:moetsi/Sensor-Stream-Pipe.git
2
cd Sensor-Stream-Pipe
3
mkdir build
4
cd build
5
cmake .. -DSSP_WITH_KINECT_SUPPORT=ON -DSSP_WITH_K4A_BODYTRACK=ON -DSSP_WITH_NVPIPE_SUPPORT=ON
6
make ssp_server
Copied!
You can turn off Kinect, Bodytrack and NVPipe support by changing the following to the cmake .. options to off:
1
-DSSP_WITH_KINECT_SUPPORT=OFF
2
-DSSP_WITH_K4A_BODYTRACK=OFF
3
-DSSP_WITH_NVPIPE_SUPPORT=OFF
Copied!
Starting the Sensor Stream Server
1
./bin/ssp_server <configuration file>
Copied!
The Sensor Stream Server will start streaming frame data by default, but it will not keep any frames if it is not connected to a client. This is a zmq setting. From ssp_server.cc:
1
// Do not accumulate packets if no client is connected
2
socket.set(zmq::sockopt::immediate, true);
Copied!
When it connects to Sensor Stream Client, the packets in the buffer will be sent first. After the buffer has emptied, the Sensor Stream Server will resume reading frames from the selected input in order, ensuring that no frames are dropped.
The Sensor Stream Server configuration is stored in a YAML file. It includes Sensor Stream Client host and port, input data configuration and encoding configuration.
The format of the file (encoding Kinect DK frame data with the Nvidia encoder) is as follows:
1
general:
2
  host: "192.168.1.64"
3
  port: 9999
4
  log_level: "debug"
5
  log_file: "ssp_server.log"
6
  frame_source: 
7
    type: "kinect"
8
    parameters:
9
        stream_color_video: True
10
        stream_depth_video: True
11
        stream_ir_video: True
12
        streaming_color_format: "K4A_IMAGE_FORMAT_COLOR_BGRA32"
13
        streaming_color_resolution: "K4A_COLOR_RESOLUTION_720P"
14
        streaming_depth_mode: "K4A_DEPTH_MODE_NFOV_UNBINNED"
15
        wired_sync_mode: "K4A_WIRED_SYNC_MODE_STANDALONE"
16
        streaming_rate: "K4A_FRAMES_PER_SECOND_30"
17
        absoluteExposureValue: 0
18
video_encoder:
19
  0: #color
20
    type: "nvenc"
21
    codec_name: "NVPIPE_HEVC"
22
    input_format: "NVPIPE_RGBA32"
23
    bit_rate: 4000000
24
  1: #depth
25
    type: "zdepth"
26
  2: #ir
27
    type: "nvenc"
28
    codec_name: "NVPIPE_HEVC"
29
    input_format: "NVPIPE_UINT16"
30
    bit_rate: 15000000
31
​
Copied!
The config/ folder includes a set of examples for all types of data using multiple encoders, codecs and parameters.

Streaming iOS ARKit RGB-D Data

Next - Components Overview

Sensor Stream Client

Last modified 3mo ago

Copy link

Contents

Building Sensor Stream Server

Starting the Sensor Stream Server