Add New Sensor Interface

Overview
Going to first make a new implementation of IReader to talk to new hardware. Checkout the iPhone IReader or Kinect IReader for inspiration on to get it done. You might then need to extend IFrame for possibly new frame types or data types. You can use parameters in the config.yaml to help define your IReader and make it flexible to things like fps, data types, resolution. Constructing your IReader will happen from those parameters. Make an example config.yaml to be used by ssp_server.cc. Then update ssp_server.cc to be able to read the config.yaml and create the IReader. If new data types are used and you want to use libav_encoder.cc you will need to update it to handle new data types. Finally, you will need to update image_converter.cc for any of the new data types in FrameStructToMat if you want to see it in sensor stream client. You made do some final rescaling to visualize in ssp_client_opencv.cc. If you need to add any libraries or deploy to new platforms will need to update cmake.
1. Create new IReader Implementation and example config file
Start by creating a new .yaml in /configs
Start with null encoders for each of the types
Good place to start is with Kinect example config because of of how many parameters it has
From the parameters the IReader will need to return at least GetTypes() and GetFps()
You can create "parameters" under "frame_source" to define what you need to correctly grab from your new frame source
Examples:
Kinect has stream rate and what streams
VideoReader has a path to the file
Then implement a new IReader
Requires the IReader interface​
IReader implementation can depend on the .yaml configuration parameters to alter function
Include error checking and graceful handling for hardware interface issues or configuration issues
Good example is Kinect IReader​
​
If implementing a frame type that requires local processing, like body detection, you will need to have a function that can be called that returns detected bodies. This will require opening a connection to a sensor feed that automatically detects bodies each frame, then pulling in the bodies into frame data in the FrameStruct.
2. Update FrameStruct (if necessary, most likely it is)
FrameStruct has fields
frame_type
frame_data_type
Likely that the new interface will possibly have a new frame_type (color, depth, ir, confidence)
object will be a new frame_type
Likely that the frame_data_type might be new
Maybe a new color space
Or confidence level is an int
3. Update encoder to handle frame_type (optional)
Can create a new encoder or can update an existing encoder
Most likely will be updating libav if it has to do with color frames
If you create a new encoder will need to also implement decoding encoding frames
The encoder converts the raw frame into an encoded frame (represented as a packet in the code). Multiple raw frames may be needed to produce a packet. The code accounts for that and sends frames to the encoder until the encoder returns a frame.
For each frame type, when you first sent a frame, it sends all the information required to decode those frames (codec type and all other coded information needed, remember the data and extra data fields).
The decoder of the client has a hashmap that keeps track of all these decoder information for all received frame streams
The decoder transforms this frame into a raw OpenCV image, with the same number of channels, width, height, .. as the original.
Can just use null encoder if do not want to implement a new IEncoder or update an existing IEncoder
4. Update FrameStructToMat in image_converter.cc and ssp_client_opencv
If planning to visualize a new frame_type or frame_data_type will need to update image_converter.cc
Follow the pattern for the different supported types currently in image_converter.cc​
If necessary can also update ssp_client_opencv to normalize or change color data
Can see example of this for confidence streamed from iPhone into black, grey, white
Can see example of this for depth streamed from iPhone
5. Update cmake with additional libraries needed to build updated SSP
Current approach is pre-built binaries for each platform
If can create dep.tar.gz for the platform moetsi will host and support the new platform dependency
ffmpeg 4.3 as shared libraries without the GPL option. Path in the dylib is changed to use @rpath for easier linking.
OpenCV 3.4.13 as a static library, only core, imgproc, imgcodecs and highgui modules are built.
Cereal 1.3.0, header only
spdlog 1.8.2, header only but built as static library for faster compile
Zdepth (commit 9b333d9aec520 which includes a patch to generate zdepthConfig.cmake)
yaml-cpp 0.6.3 as a static library
libzmq 4.3.4 as a static library
cppzmq 4.7.1, header only
Can require more libraries on the platform to interact with specific hardware possibly
6. Update ssp_server.cc to be able to read the config.yaml and create an IReader
Will need to add to the conditionals when reading the config.yaml and implement for the new frame source
Will need to feed the parameters to the IReader implementation constructor to create the correct IReader
This can require looking for the config.yaml file as is the case for iOS
In the case of plug-ins will also need to update to provide callable functionality as a plugin

Receiving Frames - Previous

Receiving Frames

Last modified 3mo ago

Copy link

Contents

Overview

1. Create new IReader Implementation and example config file

2. Update FrameStruct (if necessary, most likely it is)

3. Update encoder to handle frame_type (optional)

4. Update FrameStructToMat in image_converter.cc and ssp_client_opencv

5. Update cmake with additional libraries needed to build updated SSP

6. Update ssp_server.cc to be able to read the config.yaml and create an IReader