IEncoders
IEncoders are used to encode data before serializing and sending. Once an IReader has pulled the FrameStructs, the IEncoders are called for each FrameStruct in Sensor Stream Server to encode the frames.
The encoder converts the raw frame into an encoded frame (represented as a packet in the code). Multiple raw frames may be needed to produce a packet. The code accounts for that and sends frames to the encoder until the encoder returns a frame.
The encoders know what type of frame they are encoding by
frame_type
from the FrameStruct.Frame Type | Encoders to Use | |
0 (color) | Null, LibAv, Nvenc all are good, need a good codec, explained below H264 and H265 are recommended | |
1 (depth) | Null, ZDepth | |
2 (ir) | Null (ZDepth is not tuned well to IR) | |
3 (confidence) | Null | |
In config.yaml:
type: "null"
There are no parameters other than “type”. The Null encoder does no processing and just returns the original uncompressed frame
Color/Depth/IR - Pixel/voxel are sent exactly as captured, at the cost of very high bandwidth. The codec is also very fast, unless the stream is being limited by network bandwidth.
LibAv is supported for hardware encoding on many platforms so check if there is support for libav on your desired platform and use a codec that is platform specific for best results. Example: iPhones have videotoolbox to accelerate encoding, see the ios.yaml
In config.yaml:
type: "libav"
codec_name: <string>
pix_fmt: <string> bit_rate: <int>
- codec_name - name of the codec to use. SSP supports all codes that LibAv supports. The simplest way to see these options is to run ffmpeg -encoders. The name to use is the abbreviated name of the codec (second column). Please check if the V(ideo) flag is supported.
- pix_fmt - pixel format to be used by the codec. Common values include yuv420 for color or gray12le for infrared data. You can check the list of supported pixel formats for each codec using ffmpeg -h encoder=libx265
- bit_rate - target (average) bit rate for the encoder to use in bits.
- options - codec specific parameters, see below for examples of common codecs
It is possible to run nvenc with libav, if it was built with the adequate options. We have not tested it in SSP. Check the options using ffmpeg -h encoder=nvenc_hevc (h265) or ffmpeg -h encoder=nvenc (h264).
In config.yaml:
type: “nvenc”
codec_name: “NVPIPE_HEVC”
input_format: “NVPIPE_RGBA32” bit_rate: 2000000
- codec_name - name of the codec to use (NVPIPE_HEVC or NVPIPE_H264).
- input_format - input format of the data to be streamed: NVPIPE_RGBA32 (for color data), NVPIPE_UINT4, NVPIPE_UINT8, NVPIPE_UINT16 (for IR data), NVPIPE_UINT32
- bit_rate - target (average) bit rate for the encoder to use in bits.
In config.yaml:
type: “zdepth”
send_I_frame_interval: “30”
- send_I_frame_interval - Omit to send only an I frame once. Check the discussion regarding these parameter here
ZDepth encoder can send both full (“I”) and partial (“P”) frames. In the best case scenario, you would only need to send the first frame of the stream as a full frame; all other frames can be compressed and sent as partial frames. This is the default behaviour, as ZeroMQ guarantees that all frames are delivered. If ZeroMQ is compiled to not block when waiting for a frame; the previous guarantee does not hold.
Thus, we’ve added the send_I_frame_interval parameter to the ZDepth YAML config. This value defines how often to send full frames. This is implemented by setting a counter in the FrameServer’s ZDepth encoder, and setting the encoder to send a full frame when the counter is a multiple of the interval.
This interaction is defined here https://github.com/moetsi/Sensor-Stream-Pipe/blob/ad751d07301da4d988a3295bcbc1b67c2187112e/encoders/zdepth_encoder.cc#L116