# What is Sensor Stream Pipe? Moetsi's Sensor Stream Pipe (SSP) is the first open-source C++ modular kit-of-parts that compresses, streams, and processes sensor data (RGB-D). It does this by efficiently compressing raw data streams, allowing developers to send multiple video types over the network in real time. Frame data can be sent in its raw form (JPG/PNG frames), or compressed using a myriad of codecs, leveraged on FFmpeg/LibAV and NV Codec to considerably reduce bandwidth strain. SSP is designed to help overcome the limitations of on-device sensor data processing. By taking data processing off device, you will be able to run far more powerful computations on your sensor data and make the most of the tools at your disposal. The Moetsi Sensor Stream Pipe is designed to overcome the limitations of on-device sensor data processing. It does this by encoding and compressing your device’s sensor output (like color or depth frames), and transmitting them to a remote server where they can be decoded and processed at scale. Currently, Moetsi’s Sensor Stream Pipe supports: * .mkv (matroska) RGB-D recordings (these are created using Azure Kinect's [recorder utility](https://docs.microsoft.com/en-us/azure/kinect-dk/azure-kinect-recorder)) * Live [Azure Kinect DK](https://azure.microsoft.com/en-us/services/kinect-dk/) RGB-D/IR sensor streams * Computer vision/spatial computing datasets (e.g. [BundleFusion](https://graphics.stanford.edu/projects/bundlefusion/), [MS RGB-D 7 scenes](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/) and [VSFS](http://graphics.stanford.edu/projects/vsfs/)) * iOS ARKit data (streams [ARFrame](https://developer.apple.com/documentation/arkit/arframe) data) * [OAK-D sensors](https://store.opencv.ai/products/oak-d) **Features include:** * Synchronized streaming of color, depth and IR frames * Support for Azure Kinect DK (live and recorded video streaming) and image datasets (e.g. [BundleFusion](https://graphics.stanford.edu/projects/bundlefusion/), [MS RGB-D 7 scenes](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/) and [VSFS](http://graphics.stanford.edu/projects/vsfs/)) and .mkv (matroska) files * Hardware-accelerated encoding (e.g. Nvidia codec), providing you with the lowest possible latency and bandwidth without compromising on quality * Interoperability with Libav and FFmpeg creates a hyperflexible framework * Access to the calibration data for each of the sensors on the Kinect, enabling you to build a point cloud from the color and depth images, perform body tracking, etc. ### But why though...? * If you have 4 sensor streams and want to do an environment reconstruction using their data feeds * If you have a couple of sensors and want to find where they are relative to each other * You want to run pose detection algorithms on a dozen sensors and synthesize the results into a single 3D model * Basically if you want to do any spatial computing/computer vision on multiple incoming data streams You can use Sensor Stream Server to send compressed sensor data to reduce bandwidth requirements and Sensor Stream Client to receive these streams as an ingestion step for a computer vision/spatial computing pipeline. If you want to synthesize RGB-D+ data from multiple feeds in real-time, you will probably need something like Sensor Stream Pipe. {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Component parts ### [Sensor Stream Server](/components-overview/sensor-stream-server) The ssp\_server is the frame encoder and sender. "Frames" are a sample of data from a frame source. For example, the Azure Kinect collects: RGB (color), depth, and IR data. If we want to stream RGB-D and IR, we sample our frame source (the Azure Kinect), and create 3 frames, one for each frame type: 1 for color data, 1 for depth data, and 1 for ir data. We then package these 3 frames as a zmq message and send through a zmq socket. Grabbing frames is done by the [IReader](/frame-interface/how-to-grab-a-frame) interface. More devices can be supported by adding a new implementation of the [IReader](/frame-interface/how-to-grab-a-frame) interface. Sensor Stream Server reads its configurations from a yaml file (examples in /configs). The config file provides Sensor Stream Server: a destination for its frames, the frame source (video, Azure Kinect, or dataset), and how each frame type should be encoded. ### [Sensor Stream Client](/components-overview/sensor-stream-client) The ssp\_clients are the frame receiver and decoder. They run on the remote processing server and receive the frames from the ssp\_server for further processing. There are a few templates for how you can use Sensor Stream Client in #### Sensor Stream Client with OpenCV processing If you run Sensor Stream Client with OpenCV visualization: ![ NVPipe example ](https://github.com/moetsi/Sensor-Stream-Pipe/raw/master/examples/example.png) You can see it’s receiving real-time data from a Kinect DK and rendering it for on-screen display. In this scenario we achieved a substantial 20x data compression, reducing the stream size from 400 Mbps to just 20 Mbps, along with a PSNR of \~39 dB and a processing overhead of \~10-15 ms 😱. Sensor Stream Client is built so it can be an ingestion step for a spatial computing/computer vision pipeline. ### Sensor Stream Tester A reproducible tester for measuring SSP compression and quality. You can use this to measure how different encodings and settings affect bandwidth/compression. ## Getting started We recommend going through {% content-ref url="/pages/-MTU7vLXw1X-5rvxOeUU" %} [Streaming a Video](/streaming-a-video) {% endcontent-ref %} to get up to speed quickly. You will stream using Sensor Stream Server and receive on Sensor Stream Client a pre-recorded RGB-D+ stream to get a quick feel of what Sensor Stream Pipe does. ### Moetsi’s Permissive License Moetsi’s Sensor Stream Pipe is licensed under the MIT license. That means that we don’t require attribution, but we’d really like to know what cool things you’re using our pipe for. Drop us a message on or post on our [forum](https://moetsi.com/pages/community) to tell us all about it! ### Future Work * Support for ARCore * Support for Quest 3 * Support for VisionPro * Updating so Zdepth can be used with iOS devices ### Authors * **André Mourão** - [amourao](https://github.com/amourao) * **Olenka Polak** - [olenkapolak](https://github.com/olenkapolak) * **Adam Polak** - [adammpolak](https://github.com/adammpolak) # Installation Linux We have 2 methods for installing on Linux. In one instance, we lean heavily on pre-built binaries being packaged with SSP (recommended). In the second method we manually install all necessary dependencies. These steps have been tested on **Ubuntu 20.04** 1. Pre-built binaries **(recommended)** * This will use the binaries we have already built for each platform and architecture * If you want to see how the binaries were built you can check out `3rdparty/build_linux_dep.sh` 2. Installing Manually * This will provide download instructions for each library {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Installing with Pre-Built Binaries (Recommended) ### 1.0 Install Pre-Requisites ``` sudo apt-get update sudo apt-get install build-essential sudo apt-get install cmake sudo apt-get install libgtk2.0-dev sudo apt-get install git sudo apt-get install ffmpeg sudo apt-get install libusb-1.0 ``` ### 2.0 Install Azure Kinect SDK/Azure Kinect Body Tracking SDK (optional) These steps are only required if you are interested in building with Azure Kinect and Body Tracking capabilities. From this comment: > These are the steps I followed to install k4a-tools, libk4a and libk4abt on Ubuntu 20.04. The general steps are as outlined in , with a couple of hacks to make things work on 20.04. - use of 18.04 repo, even though OS is 20.04 - installed lower versions of tools and libraries (as latest versions of sensor and body tracker don't seem to be compatible on 20.04) ``` curl -sSL https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add - sudo apt-add-repository https://packages.microsoft.com/ubuntu/18.04/prod curl -sSL https://packages.microsoft.com/config/ubuntu/18.04/prod.list | sudo tee /etc/apt/sources.list.d/microsoft-prod.list curl -sSL https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add - sudo apt-get update sudo apt install libk4a1.3-dev sudo apt install libk4abt1.0-dev sudo apt install k4a-tools=1.3.0= ``` Verify sensor library by launching camera viewer (require `sudo` for hardware access) ``` sudo k4aviewer ``` * Clone and build [Azure Kinect Samples](https://github.com/microsoft/Azure-Kinect-Samples). * Run `simple_3d_viewer` to verify body tracker works. ### 3.0 Install [OAK-D](https://store.opencv.ai/products/oak-d) Pre-Requisites (optional, if you want to stream an OAK-D using Xlink and run inference using OpenVINO) #### 3.1 Install python 3.8 if you do not have it (necessary for OAK-D) #### 3.2 Install OpenVINO (used to run inference on frames grabbed from OAK-D device) [ ]() ### 4.0 Build/Install Sensor Stream Pipe ``` git clone https://github.com/moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build && cd build cmake .. make -j4 ``` {% hint style="info" %} **NOTE:** **You must enable Kinect or Xlink functionality when building with the following flags:** `-DK4A_ENABLED=TRUE` (for Kinect) `-DXLINK_ENABLED=TRUE` (for OAK-D) `Example:` `cmake -DK4A_ENABLED=TRUE ..` or `cmake -DXLINK_ENABLED=TRUE ..` the flags are cached, so if you want to disable you will need to set them to false {% endhint %} ### 5.0 Use it! Now checkout [Streaming a Video](/streaming-a-video) or [Streaming a Dataset](/streaming-a-dataset) to see how you can stream local files with Sensor Stream Server to Sensor Stream Client. ## Installing Manually ### 1.0 Install Pre-Requisites To get our Sensor Stream Pipe up and running, you will require the following: The following steps were tested on Ubuntu 18.04. Installing on other recent Linux distributions should be pretty similar, but please check the installation instructions for OpenCV and Kinect DK on your respective platform first. Installation instructions for Windows should be ready soon. If you encounter any problems or have any suggestions, please let us know by emailing or post on our [forum](https://moetsi.com/pages/community). #### Dependencies To get our Sensor Stream Pipe up and running, you will require the following: * [OpenCV](https://opencv.org/) 3.2.0 (tested on version available on Ubuntu 18.04 repo) is used for image processing. * [libav](https://github.com/libav/libav/) 3.4.6 (tested on version available on Ubuntu 18.04 repo) encodes, decodes and processes image frames. * [Cereal](https://uscilab.github.io/cereal/) 1.2.2 (headers only) serializes data for network transmission. * [ZeroMQ](http://zeromq.org/) and [cppzmq](https://github.com/zeromq/cppzmq/) (libzmq3 4.3.1, cppzmq 4.3.0) perform network and low-level I/O operations. * [spdlog](https://github.com/gabime/spdlog/) 1.4.1 Logging library. * [yaml-cpp](https://github.com/jbeder/yaml-cpp/) 0.6.0 reads server configuration files. * [Zdepth](https://github.com/catid/Zdepth.git): compresses depth data. * [NvPipe](https://github.com/NVIDIA/NvPipe/) (*optional*, but **recommended if you have an NVidia GPU** ) encodes and decodes frames. This is optional, but recommended for users with Nvidia GPUs. * [Azure Kinect SDK](https://github.com/microsoft/Azure-Kinect-Sensor-SDK/) 1.3 (to support the Azure Kinect Body Tracking SDK), 1.4 otherwise (*optional*) accesses Kinect DK data. * [Azure Kinect Body Tracking SDK](https://docs.microsoft.com/bs-cyrl-ba/azure/Kinect-dk/body-sdk-download/) 1.0 (*optional*) SSP Body Tracking client. #### Download and install repo libraries **OpenCV 3.2.0** ``` sudo apt install libopencv-dev libopencv-core-dev uuid-dev ``` **Libav 3.4.6** ``` sudo apt install libavformat-dev libavutil-dev libavcodec-dev libavfilter-dev ``` #### Download and extract "out-of-repo" libraries First, create a folder where local libs are to be installed: ``` mkdir ~/libs mkdir ~/libs/srcOriginal ``` **Cereal 1.2.2** ``` cd ~/libs/srcOriginal wget https://codeload.github.com/USCiLab/cereal/tar.gz/v1.2.2 tar xf v1.2.2 cp -r cereal-1.2.2/include ~/libs ``` **ZeroMQ** If you want to take advantage of ZMQ pulling support to check for new frames, compile ZMQ and CPPZMQ with the [draft API suppport](https://github.com/zeromq/cppzmq/issues/175). Also, you must set SSP\_WITH\_ZMQ\_POLLING at SSP build time. Discussion available [here](https://github.com/moetsi/Sensor-Stream-Pipe/pull/7) **libzmq3 4.3.1** ``` cd ~/libs/srcOriginal wget https://github.com/zeromq/libzmq/releases/download/v4.3.1/zeromq-4.3.1.tar.gz tar xf zeromq-4.3.1.tar.gz cd zeromq-4.3.1 mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=~/libs make install -j4 ``` **cppzmq 4.3.0** ``` cd ~/libs/srcOriginal wget https://github.com/zeromq/cppzmq/archive/v4.3.0.tar.gz tar xf v4.3.0.tar.gz cd cppzmq-4.3.0 cp *.hpp ~/libs/include ``` **yaml-cpp 0.6.0** ``` cd ~/libs/srcOriginal wget https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.0.tar.gz tar xf yaml-cpp-0.6.0.tar.gz cd yaml-cpp-yaml-cpp-0.6.0 mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=~/libs make install ``` **Zdepth** ``` cd ~/libs/srcOriginal git clone https://github.com/catid/Zdepth.git cd Zdepth mkdir build cd build cmake .. -DCMAKE_INSTALL_PREFIX=~/libs make install cp libzdepth.a ~/libs/lib/ cp zstd/libzstd.a ~/libs/lib/ ``` **spdlog** ``` cd ~/libs/srcOriginal wget https://github.com/gabime/spdlog/archive/v1.4.1.tar.gz tar xf v1.4.1.tar.gz cd spdlog-1.4.1 && mkdir build && cd build cmake .. -DCMAKE_INSTALL_PREFIX=~/libs make -j make install ``` **NVPipe (optional, recommended for users with Nvidia GPU)** ``` cd ~/libs/srcOriginal git clone https://github.com/NVIDIA/NvPipe.git cd NvPipe/ mkdir build && cd build cmake .. -DCMAKE_INSTALL_PREFIX=~/libs make make install ``` **Azure Kinect SDK 1.3/4 (optional)** *Note: to avoid getting a password prompt, run any command as sudo before starting this section of the tutorial* 1\) Add the Linux Software Repository for Microsoft Products. ``` curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add - sudo apt-add-repository https://packages.microsoft.com/ubuntu/18.04/prod sudo apt-get update ``` 2\) Install Azure Kinect SDK 1.3 (change 1.3 to 1.4 if you do not need body tracking) ``` sudo apt install libk4a1.3 libk4a1.3-dev ``` 3\) To be able to use the Kinect as non-root, please run the following: ``` wget https://raw.githubusercontent.com/microsoft/Azure-Kinect-Sensor-SDK/develop/scripts/99-k4a.rules sudo cp 99-k4a.rules /etc/udev/rules.d/ ``` 4 a) If using 1.4, in the current package, the link to the canonical version of the depth lib is missing. You can create it by running the following command: ``` sudo ln -s /usr/lib/x86_64-linux-gnu/libk4a1.4/libdepthengine.so.2.0 /usr/lib/x86_64-linux-gnu/libdepthengine.so ``` 4 a) If using 1.3, the depth engine is missing from the package. Microsoft is aware of the problem, but it only corrected it in 1.4. You can get the libdepthengine.so.2.0 file from the package at . Open with Archive Manager (or equivalent), the file is in (data/./usr/lib/x86\_64-linux-gnu/libk4a1.4/), and extract it to `~/libs/lib`. You can then perform an equivalent command as above. ``` cd ~/libs/lib ln -s libdepthengine.so.2.0 libdepthengine.so ``` **Azure Kinect Body Tracking SDK (optional)** Check instructions above to add the Linux Software Repository for Microsoft Products and then do: ``` sudo apt install libk4abt1.0-dev ``` **Building Sensor Stream Pipe** Download and build the project (the ssp\_server, ssp\_client and ssp\_tester): ``` git clone git@github.com:moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build cd build cmake .. -DSSP_WITH_KINECT_SUPPORT=OFF -DSSP_WITH_K4A_BODYTRACK=OFF -DSSP_WITH_NVPIPE_SUPPORT=OFF make ``` Now checkout [Streaming a Video](/streaming-a-video) or [Streaming a Dataset](/streaming-a-dataset) to see how you can stream local files with Sensor Stream Server to Sensor Stream Client. # Installation Windows We have 2 methods for installing on Windows. In one instance, we lean heavily on pre-built binaries being packaged with SSP (recommended). In the second method we manually install all necessary dependencies. These steps have been tested on **Windows 10 Build 19041, Visual Studio 2019 Community Edition (VS) and Visual Studio Code 1.53** 1. Pre-built binaries **(recommended)** * Dependencies are prebuilt and stored on a server and downloaded by cmake using `FetchContent` feature * To rebuild dependencies: ``` cd Sensor-Stream-Pipe/3rdparty ./buid_win_dep.sh ``` * This will download/build/install dependencies in a `tmp` directory. The result is a `*_windep.tar.gz` file. * download prebuilt ffmpeg 4.3.2 * OpenCV 3.4.13 as a static library, only core, imgproc, imgcodecs and highgui modules are built * Cereal 1.3.0, header only * spdlog 1.8.2, header only but built as static library for faster compile * Zdepth (commit 9b333d9aec520 which includes a patch to generate `zdepthConfig.cmake`) * yaml-cpp 0.6.3 as a static library * libzmq 4.3.4 as a static library * cppzmq 4.7.1, header only 2. Installing Manually * This will provide download instructions for each library {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Installing with Pre-Built Binaries (Recommended) ### 1.0 Install Azure Kinect Pre-Requisites (optional, if you want to use Azure Kinect, or Azure Kinect Body Tracking SDK) #### 1.1 Manually install CUDA Toolkit 10.0 (needed for Body Tracking SDK) Download here: #### 1.2 Manually install cudNN64\_70 for CUDA 10.0 (needed for Body Tracking SDK) Download here: You will need to accept some NVIDIA agreements #### 1.3 Manually Install Azure Kinect 1.4.1 Follow the instructions here: #### 1.4 Manual Install Azure Body Tracking SDK 1.1 Info here: Download link: ### 2.0 Install [OAK-D](https://store.opencv.ai/products/oak-d) Pre-Requisites (optional, if you want to stream an OAK-D using Xlink and run inference using OpenVINO) #### 2.1 Install python3 if you do not have it (necessary for OAK-D) {% embed url="" %} Ensure that python is added as a path variable #### 2.2 Remove max path limit `git config --system core.longpaths true` #### 2.3 Install OpenVINO (used to run inference on frames grabbed from OAK-D device) [https://docs.openvino.ai/latest/openvino\_docs\_install\_guides\_installing\_openvino\_windows.html
]() By default, the Intel® Distribution of OpenVINO™ is installed to the following directory, referred to as \ elsewhere in the documentation: C:\Program Files (x86)\Intel\openvino\_\. For simplicity, a shortcut to the latest installation is also created: C:\Program Files (x86)\Intel\openvino\_2021 ### 3.0 Build/Install Sensor Stream Pipe Note: If planning on using Azure Kinect, Azure Kinect Body Tracking SDK, or Xlink and OpenVINO, they must be installed **before** SSP as SSP looks for the libraries when creating the Release ``` git clone https://github.com/moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build && cd build cmake -G "Visual Studio 15 2017 Win64" .. cmake --build . --config Release ``` {% hint style="info" %} **NOTE:** **You must enable Kinect or Xlink functionality when building with the following flags:** `-DK4A_ENABLED=TRUE` (for Kinect) `-DXLINK_ENABLED=TRUE` (for OAK-D) `Example:` `cmake -DK4A_ENABLED=TRUE -G "Visual Studio 15 2017 Win64" ..` or `cmake -DXLINK_ENABLED=TRUE -G "Visual Studio 15 2017 Win64" ..` and you will need to set the env variabels for OpenVINO so you will need to run "'C:/Program Files (x86)/Intel/openvino\_2021/bin/setupvars.bat'" or where ever setupvars.bat is installed the flags are cached, so if you want to disable you will need to set them to false {% endhint %} ### 3.0 Use it! #### 3.1 Add Body Tracking Components (optional) Move:\ \- dnn\_model\_2\_0.onnx (from body tracking sdk) and\ \- cudNN64\_7 (from step 1.2) to /Release (or where you run the executable) if you want to run Body Tracking on Sensor Stream Client #### 3.2 Give it a go Now checkout [Streaming a Video](/streaming-a-video) or [Streaming a Dataset](/streaming-a-dataset) to see how you can stream local files with Sensor Stream Server to Sensor Stream Client. ## Installing Manually ### 1.0 Make sure you have Build Tools, Git, and CUDA 1.1 Go to scroll down and download "[Build Tools for Visual Studio](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools\&rel=16)". 1.2 Run "Build Tools for Visual Studio", choose "Language packs" and add "English". Tested with "Visual Studio Build Tools 2019 16.8.5. 1.3 Download and install [Git for windows](https://git-scm.com/download/win). 1.4 Install CUDA 10 (if you would like to use Azure Kinect Body Tracking) 1.5 Install cuDNN 7.0 (if you would like to use Azure Kinect Body Tracking) ### 2.0 Install vcpkg 2.1 Follow vcpkg installation instructions available [here](https://docs.microsoft.com/en-us/cpp/build/install-vcpkg?view=msvc-160\&tabs=windows). 2.2 In Command window, change directories to the vcpkg root directory. Run `vcpkg integrate install` . This should respond with something similar to: ``` C:\Users\adamm\dev\vcpkg>vcpkg integrate install Applied user-wide integration for this vcpkg root. All MSBuild C++ projects can now #include any installed libraries. Linking will be handled automatically. Installing new libraries will make them instantly available. CMake projects should use: "-DCMAKE_TOOLCHAIN_FILE=C:/Users/adamm/dev/vcpkg/scripts/buildsystems/vcpkg.cmake" ``` ### 3.0 Install dependencies available on vcpkg 3.1 Install dependencies using vcpkg (in vcpkg root directory). ``` vcpkg install azure-kinect-sensor-sdk:x64-windows cereal:x64-windows cppzmq:x64-windows ffmpeg:x64-windows opencv3:x64-windows spdlog:x64-windows yaml-cpp:x64-windows zeromq:x64-windows ``` ### 4.0 Build and install remaining dependencies 4.1 Install [cmake](https://cmake.org/download/) #### Zdepth Zdepth is an encoder for depth frames. This library can be used to compress depth data. 4.2 Clone the Zdepth repo to your home directory (or where ever you usually save repos) ``` git clone https://github.com/catid/Zdepth.git ``` 4.3 Open CMakeLists.txt in Visual Studio Code 4.4 If you do not have them already installed, install [CMake Tools](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools) for VS Code 4.5 [Configure the project using CMake Tools](https://github.com/microsoft/vscode-cmake-tools/blob/main/docs/how-to.md#configure-a-project) (x64; Debug or Release) and [build](https://github.com/microsoft/vscode-cmake-tools/blob/main/docs/how-to.md#build-a-project) (this will create an output folder usually called /build) 4.6 Prepare a directory in your home directory (or where ever you usually save repos) to place the remaining dependencies, `/libs` (referred henceforth as `$LIBS`). Create subfolders `/include` and `/lib` within `/libs` ``` mkdir libs cd libs mkdir lib mkdir include ``` 4.7 Copy the contents of `Zdepth\include` into `$LIBS\include` and output lib folders (e.g. `ZDepth\out\*`) to `$LIBS\lib`. **Azure Kinect Body Tracking SDK (optional)** 4.9 Install Azure Body Tracker SDK from the instructions available [here](https://www.microsoft.com/en-us/download/details.aspx?id=100942). 4.10 Copy the SDK include and lib files from the SDK install list to `$LIBS`, or add the SDK path to SSP CMakeLists (see below) * Example * `C:\Program Files\Azure Kinect Body Tracking SDK\sdk\include` * Into `$LIBS\include` * `C:\Program Files\Azure Kinect Body Tracking SDK\sdk\netstandard2.0` and `C:\Program Files\Azure Kinect Body Tracking SDK\sdk\windows-desktop` * Into `$LIBS\lib` ### **5.0 Building Sensor Stream Pipe** 5.1 Clone the SSP repo to your home directory (or where ever you usually save repos) ``` git clone git@github.com:moetsi/Sensor-Stream-Pipe.git ``` Due to the differences in the build process, the Windows CMake file is named CMakeListsWindows.txt at the root of the SSP repo. Thus, you should: * 5.2 Delete CMakeLists.txt * 5.3 Rename CMakeListsWindows.txt to CMakeLists.txt. * 5.4 Open CMakeLists.txt in VS * 5.5 Update * the include ("C://Users//Andre//source//repos//vcpkg//installed//x64-windows//include") * and link paths ("C://Users//Andre//source//repos//vcpkg//installed//x64-windows//lib") * To follow the same path of where you cloned vcpkg in Step 1 5.6 Now we can build Sensor Stream Pipe. In /Sensor-Stream-Pipe run the following commands in terminal: ``` mkdir build cd build cmake .. -DCMAKE_TOOLCHAIN_FILE=/Users/adammac/dev/vcpkg-macos/vcpkg/scripts/buildsystems/vcpkg.cmake -DSSP_WITH_KINECT_SUPPORT=ON -DSSP_WITH_K4A_BODYTRACK=ON -DSSP_WITH_NVPIPE_SUPPORT=ON make ``` Now checkout [Streaming a Video](/streaming-a-video) or [Streaming a Dataset](/streaming-a-dataset) to see how you can stream local files with Sensor Stream Server to Sensor Stream Client. ## NvPipe Information `NvPipe` is deprecated so we will use an old version of the NVIDIA Video Codec SDK. An alternative will be to use the GPU integration in ffmpeg. Install [NVIDIA Video Codec SDK 9.1.23](https://developer.nvidia.com/video-codec-sdk-archive) in `c:/local`Bash ``` git clone cd NvPipe mkdir build && cd build cmake -G "Visual Studio 15 2017 Win64" \ -DNV_VIDEO_CODEC_SDK=c:/local/Video_Codec_SDK_9.1.23/ \ -DCMAKE_INSTALL_PREFIX=c:/local/nvpipe \ -DNVPIPE_BUILD_EXAMPLES=ON .. cmake --build . --config Release cmake --build . --config Release --target install ``` You will need to patch `CMakeLists.txt`Diff ``` index c0cf223..c0b85a9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -58,6 +58,7 @@ configure_file(src/NvPipe.h.in include/NvPipe.h @ONLY) include_directories(${CMAKE_CURRENT_BINARY_DIR}/include) # NvPipe shared library +include_directories(${NV_VIDEO_CODEC_SDK}/Interface) list(APPEND NVPIPE_SOURCES src/NvPipe.cu ${NV_VIDEO_CODEC_SDK}/Samples/Utils/ColorSpace.cu @@ -66,7 +67,7 @@ list(APPEND NVPIPE_LIBRARIES ${CMAKE_DL_LIBS} ${CUDA_LIBRARIES} ${CUDA_LIB} - nvidia-encode + nvencodeapi ) if (NVPIPE_WITH_ENCODER) ``` You could then compile `Sensor-Stream-Pipe` with NvPipe supportBash ``` cmake -G "Visual Studio 15 2017 Win64" \ -DNvPipe_DIR=c:/local/nvpipe/share/NvPipe/cmake \ .. ``` # Installation Mac If you are running on intel for mac you will download pre-built binaries. If you are running Apple Silicon the process will download dependency libraries from source. {% hint style="info" %} Note: You will not be able to stream a connected Kinect on macOS. Installation instructions do not include the Azure Kinect SDK as macOS is not yet supported by Azure Kinect SDK . {% endhint %} {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Pre-built binaries installation (recommended) ### 1.0 Pre-Installation Requirements 1.1 Install [Xcode](https://developer.apple.com/xcode/) 1.2 Install Xcode command line tools ``` sudo xcode-select --install ``` 1.3 Install cmake ``` brew install cmake ``` 1.4 Install pkg-config ``` brew install pkg-config ``` ### 2.0 Download and Install SSP 2.1 Download and make SSP ``` git clone https://github.com/moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build && cd build cmake .. make -j12 ``` h264\_videotoolbox codec is used for hardware codec on macOS The dependencies libraries are built with the 3rdparty/build\_mac\_dep.sh ### 3.0 Use it! Now checkout [Streaming a Video](/streaming-a-video) or [Streaming a Dataset](/streaming-a-dataset) to see how you can stream local files with Sensor Stream Server to Sensor Stream Client. # Installation iOS We have 2 different methods for building for iOS: 1. Installation for an iOS Application using XCode (no Unity)\ \- This will build an iOS app that only streams data 2. Installation for an iOS Deployed Unity App **(Unity Plugin)**\ \- This will build a Unity AR iOS app so you can stream data while running an AR app Both will use the binaries we have already built for each platform and architecture. If you want to see how the binaries were built you can check out `3rdparty/build_ios_dep.sh` To rebuild dependencies, in a git bash terminal ``` cd Sensor-Stream-Pipe/3rdparty ./buid_ios_dep.sh ``` This will download/build/install dependencies in a `tmp` directory. The result is a `*_ssp_iosdep.tar.gz` file. {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Building for an iOS App (no Unity) ### 1.0 Pre-Installation Requirements 1.1 Install [Xcode](https://developer.apple.com/xcode/) 1.2 Install Xcode command line tools ``` sudo xcode-select --install ``` 1.3 Install cmake ``` brew install cmake ``` 1.4 Install pkg-config ``` brew install pkg-config ``` ### 2.0 Download and Install SSP 2.1 FOR DEVICE - To download and make iOS ``` git clone https://github.com/moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build-ios && cd build-ios cmake -G Xcode \ -DCMAKE_TOOLCHAIN_FILE=../3rdparty/ios.toolchain.cmake \ -DPLATFORM=OS64 \ .. open ssp.xcodeproj ``` ### 3.0 Use it! Now checkout Streaming [iOS ARKit RGB-D Data](/streaming-ios-arkit-rgb-d-data) to see how you can stream ARFrame data with Sensor Stream Server to Sensor Stream Client. ## Building for an iOS deployed Unity App (Unity Plugin) ### 1.0 Pre-Installation Requirements 1.1 Install [Xcode](https://developer.apple.com/xcode/) 1.2 Install Xcode command line tools ``` sudo xcode-select --install ``` 1.3 Install cmake ``` brew install cmake ``` ### 2.0 Download and Install SSP 2.1 To download and make iOS Device ``` git clone https://github.com/moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build-ios && cd build-ios cmake -G Xcode \ -DCMAKE_TOOLCHAIN_FILE=../3rdparty/ios.toolchain.cmake \ -DPLATFORM=OS64 \ .. cmake --build . --target unity --config Release ``` 2.2 Moving the built libraries into the Unity Project * Move the library built in build-ios/ssp\_plugin\_unity to the Assets/Plugins/iOS folder of the Unity project cloned in [this step](https://sensor-stream-pipe.moetsi.com/pages/-Mbhzo95saHrTRQ5f9nL#id-3.0-setup-unity-project) {% hint style="info" %} If you get this issue Run this: `sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer` {% endhint %} ### 3.0 Use it! Now checkout Streaming [iOS ARKit RGB-D Data](/streaming-ios-arkit-rgb-d-data) to see how you can stream ARFrame data with Sensor Stream Server to Sensor Stream Client. # Streaming a Video ## Stream RGB-D/IR .mkv (matroska) Videos {% hint style="info" %} The current .mkv video format supported is the .mkv file type created by the Azure Kinect's [recorder utility](https://docs.microsoft.com/en-us/azure/kinect-dk/azure-kinect-recorder) We provide a sample .mkv video in the instructions below so you can check out how SSP works without needing to buy an Azure Kinect **Streaming .mkv is a good idea for Windows/Linux/Mac testing**. Although you can stream a local .mkv from an iOS device it isn't a "real" use case (why would that be needed?). {% endhint %} Sensor Stream Server currently supports streaming .mkv files. [Azure Kinect](https://www.microsoft.com/en-us/p/azure-kinect-dk/8pp5vxmd9nhq?activetab=pivot:overviewtab) comes with a [recorder app](https://docs.microsoft.com/en-us/azure/kinect-dk/azure-kinect-recorder) that allows you to create .mkv files from a live [Azure Kinect](https://www.microsoft.com/en-us/p/azure-kinect-dk/8pp5vxmd9nhq?activetab=pivot:overviewtab) feed. You can use Sensor Stream Server to stream these videos to Sensor Stream Client. We have provided an .mkv file to download to get you started if you want to quickly see what is going on with Sensor Stream Pipe. ### 1.0 Download an RGB-D Video Moetsi has made available a few RGB-D recordings taken with the Azure Kinect to be used for testing. 1.1 Install "wget" to download the file ``` brew install wget ``` 1.2 Use wget (or cURL) to download an example file Video file of a moving sensor (if using for SfM/SLAM pipelines): ``` wget https://moetsiblob.blob.core.windows.net/public-datasets/1080p_ShadesDown_Darker_Closer_Right_Moving_Right.mkv ``` Video file of a static sensor and a moving human body (for body tracking pipelines): ``` wget https://moetsiblob.blob.core.windows.net/public-datasets/Body_Tracking_Test_File.mkv ``` ### 2.0 Prepare Sensor Stream Pipe 2.1 Follow installation instructions for [Linux](/linux)/[Windows](/windows)/[MacOS](/installation-mac) to build Sensor Stream Pipe on your development platform. 2.2 Move the downloaded .mkv file to the same folder as the Sensor Stream Server executable (not required but this way you will not need to define the path to the video file in the config yaml in step 2.2) 2.2 Update line 10 (path) in Sensor-Stream-Pipe/configs/serve\_video.yaml to point at the downloaded .mkv file (or the path of any mkv file) If you are using the downloaded .mkv file and placed it in the same folder as the server executable then line 10 would be: ``` path: "1080p_ShadesDown_Darker_Closer_Right_Moving_Right.mkv" ``` ### 3.0 Run Sensor Stream Client and Sensor Stream Server 3.1 Run Sensor Stream Client with OpenCV (this will use OpenCV to visualize incoming streams) ``` ./ssp_client_opencv 9999 ``` Run with whatever port you have set in the config yaml (default is 9999) 3.2 Run Sensor Stream Server ``` ./ssp_server ../../configs/serve_video.yaml ``` You will need to provide an argument which has a path to the config yaml you want to use # Streaming with Azure Kinect DK ## Stream [Azure Kinect](https://azure.microsoft.com/en-us/services/kinect-dk/) on Linux and Windows Sensor Stream Server supports live streaming with a connected [Azure Kinect](https://azure.microsoft.com/en-us/services/kinect-dk/) on **Linux** and **Windows**. {% hint style="info" %} MacOS [vcpkg azure-kinect-sensor-sdk is currently broken](https://github.com/microsoft/vcpkg/issues/16329) so no streaming on MacOS until that is fixed (which is likely never). {% endhint %} [Azure Kinect](https://azure.microsoft.com/en-us/services/kinect-dk/) streams at high-bandwidth so if you want to send a sensor stream across the network it might be a good idea to compress the stream. ### 1.0 Prepare Sensor Stream Pipe 1.1 Follow installation instructions for [Linux](/linux)/[Windows](/windows) to build Sensor Stream Pipe on your development platform. ### 2.0 Plug-in your [Azure Kinect](https://azure.microsoft.com/en-us/services/kinect-dk/) 2.1 Plug in your [Azure Kinect](https://azure.microsoft.com/en-us/services/kinect-dk/) to the computer that you will be running Sensor Stream Server. ### 3.0 Run Sensor Stream Client and Sensor Stream Server 3.1 Run Sensor Stream Client with OpenCV (this will use OpenCV to visualize incoming streams) ``` ./ssp_client_opencv 9999 ``` Run with whatever port you have set in the config yaml (default is 9999) 3.2 Run Sensor Stream Server ``` ./ssp_server ../../configs/serve_kinect_raw.yaml ``` You will need to provide an argument which has a path to the config yaml you want to use. Above we are using a config yaml that sets us up to stream color, depth, and IR raw to Sensor Stream Client. You can update the config file to use different encoders to compress the feed and reduce bandwidth requirements. You may have to try to plug in your [Azure Kinect](https://www.microsoft.com/en-us/p/azure-kinect-dk/8pp5vxmd9nhq?activetab=pivot:overviewtab) to different USB ports if you get an error connecting with Sensor Stream Server. # Streaming a Dataset We have made available a few seminal computer vision/spatial computing datasets to be used with Sensor Stream Pipe. If you have a spatial computing/computer vision pipeline that you would like to compare to the pipelines of these published papers, you can stream the data to both pipelines with Sensor Stream Pipe. We have provided scripts that are able to take specific published datasets and generate a .txt file that can be ingested by Sensor Stream Server to stream those datasets to Sensor Stream Client. Our currently supported datasets: * [BundleFusion](http://graphics.stanford.edu/projects/bundlefusion/#data) - Real-time Globally Consistent 3D Reconstruction using Online Surface Re-integration * [Microsoft](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/#!publications) - Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images * [NYU Depth Dataset V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) - Indoor Segmentation and Support Inference from RGBD Images Follow the instructions below to stream a dataset and ingest the output with Sensor Stream Client, which can be used as a component of your own spatial computing pipeline. ### 1.0 Download a dataset and create the frame list file 1.1 Check out our [Sensor Stream Pipe Data Scripts Repo](https://github.com/moetsi/ssp-data-scripts) and follow the instructions to create a frame list file for the dataset of your choice. The frame list files are used by Sensor Stream Server's config file as a way to point as the frame source. The [`example/` folder](https://github.com/moetsi/Sensor-Stream-Pipe/tree/master/examples) includes an example frame list files. After extracting the paths of your created frame list files, change the paths on line 8 and 9 in `configs/serve_frames_color_depth.yaml` . The config file is currently pointed at example frame list files: `examples/stairs-seq-01-frames-color.txt` and `examples/stairs-seq-01-frames-depth.txt`. ### 2.0 Prepare Sensor Stream Pipe 2.1 Follow installation instructions for [Linux](/linux)/[Windows](/windows)/[MacOS](/installation-mac) to build Sensor Stream Pipe on your development platform. 2.2 Move the downloaded .mkv file to the same folder as the Sensor Stream Server executable (not required but this way you will not need to define the path to the video file in the config yaml in step 2.2) 2.2 Update line 10 (path) in Sensor-Stream-Pipe/configs/serve\_video.yaml to point at the downloaded .mkv file (or the path of any mkv file) If you are using the downloaded .mkv file and placed it in the same folder as the server executable then line 10 would be: ``` path: "1080p_ShadesDown_Darker_Closer_Right_Moving_Right.mkv" ``` ### 3.0 Run Sensor Stream Client and Sensor Stream Server 3.1 Run Sensor Stream Client with OpenCV (this will use OpenCV to visualize incoming streams) ``` ./ssp_client_opencv 9999 ``` Run with whatever port you have set in the config yaml (default is 9999) 3.2 Run Sensor Stream Server ``` ./ssp_server ../../configs/serve_frames_color_depth.yaml ``` You will need to provide an argument which has a path to the config yaml you want to use. In the example .yaml provided in the configs/ folder we assume you have created a frame list and put it in examples/. # Streaming iOS ARKit RGB-D Data We have 2 different methods for streaming RGB-D on iOS: 1. Streaming without running AR app (no Unity) 2. Streaming while running AR app **(Unity Plugin)** {% hint style="info" %} **Problems?! (shocker)** Reach out on [our discord](https://discord.gg/9r2HH4xH) and we will get you going! {% endhint %} ## Streaming without running AR app ### 1.0 Build Sensor Stream Pipe for iOS (No Unity) 1.1 Follow the instructions: [Installation iOS (Unity Plugin)](/installation-ios#building-for-an-ios-app-no-unity) ### **2.0 Build Sensor Stream Pipe for Mac** 2.1 Follow the instructions here: [Installation Mac](/installation-mac#pre-built-binaries-installation-recommended) ### 3.0 Run Sensor Stream Client on Mac 3.1 Navigate to `/Sensor-Stream-Pipe/build/bin` 3.2 Run Sensor Stream Client with OpenCV visualization ``` ./ssp_client_opencv 9999 ``` ### **4.0 Run and Build Sensor Stream Server** 4.1 Open ssp.xcodeproj in `Sensor-Stream-Pipe/build-ios` 4.2 Sign, in Signing & Capabilities ![](/files/-MevYMtnYr1x97ggABjz) 4.3 Build and run ssp\_server! If running into issues update host to the ip address of the computer running ssp client opencv in `serve_ios_raw.yaml`: ![](/files/-MevdllzBxMF-wHiQXbk) ## **Streaming while running AR app (Unity Plugin)** ### 1.0 Build Sensor Stream Pipe for iOS (No Unity) 1.1 Follow the instructions: [Installation iOS (Unity Plugin)](/installation-ios#building-for-an-ios-deployed-unity-app-unity-plugin) ### **2.0 Build Sensor Stream Pipe for Mac** 2.1 Follow the instructions here: [Installation Mac](/installation-mac#pre-built-binaries-installation-recommended) ### **3.0 Setup Unity Project** 3.1 git clone our sample project ```csharp git clone https://github.com/moetsi/SSP-Unity-Plugin-AR-App.git ``` 3.2 Update line 3 in Assets/StreamingAssets/serve\_ios\_raw\.yaml to the correct destination IP address
### **4.0 Deploy the Unity Project** 4.0 Go to Assets/Scenes and open the SampleScene 4.1 Move the plugins created in [this step](https://sensor-stream-pipe.moetsi.com/pages/-MYAjs1ldpderuy3UOBk#id-2.0-download-and-install-ssp-1) to Assets/Plugins/iOS 4.1 Open Unity (2022+ works well) Go to Build Settings, switch to iOS platform, and hit "Build and Run" (this will launch xcode) Set up your signing so you can deploy your application. 4.2 Run ssp\_client\_opencv on port 9999 on the device you are streaming frame data (check out [Streaming a Video](/streaming-a-video#3-0-run-sensor-stream-client-and-sensor-stream-server) if need instruction on how to do this) 4.3 Deploy the project onto an RGB-D (LiDAR) iPhone and check out the RGB-D data streaming as the iPhone runs an Unity AR Application! # Streaming an OAK-D using Xlink How to pull data from an OAK-D using Xlink and use OpenVINO to run inference ## Streaming from an [OAK-D](https://store.opencv.ai/products/oak-d) PoE using Xlink and Using OpenVINO for inference Currently the OAK-D Xlink build is dependent on OpenVINO library for functionality. Even if you decide to not use OpenVINO the library still requires OpenVINO to be installed to function properly. ### 1.0 Build Sensor Stream Pipe for OAK-D Xlink 1.1 Ensure that OpenVINO environment variables are set in the command terminal Linux: `source /opt/intel/openvino_2021/bin/setupvars.sh` Windows: `C:\Program Files (x86)\Intel\openvino_/bin/setupvars.sh` 1.1 Follow the instructions: [Installation Windows](/windows) or [Installation Linux](/linux) and use the `-DXLINK_ENABLED=TRUE` flag when building ### 2.0 Use Sensor Stream Pipe to Stream OAK-D frames using Xlink {% hint style="info" %} **You will likely need to edit** **oakd\_xlink\_reader.cc to grab the exact frames and run the model you want on those frames** Currently oakd\_xlink\_reader.cc does the following: * requests a 300x300 RGB image and streams it * runs inference on the 300x300 image [using the model from the OpenVINO HelloClassification example](https://docs.openvino.ai/latest/openvino_inference_engine_samples_hello_classification_README.html) * running inference on the model in /models **Why run inference and output the results in Sensor Stream Pipe?** We wanted to use an example that you can follow from OpenVINO documentation so you can upload different models to /models. You can extend SSP to support additional frames such as inference results from an OpenVINO model. {% endhint %} **2.1** Run SSP client to show the color frames `./ssp_client_opencv 9999` **2.2** Update serve\_xlink\_raw\.yaml to use the IP address of your OAK-D PoE (line 8) **2.3 (only if Linux)** Create USB rules for PoE `echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules` This creates that /etc/udev/rules.d/80-movidius.rules . It's required when using the DepthAI USB interface, for regular users (non root) **2.4 Trigger OAK-D to send SSP reader frames through Xlink** `./ssp_server ../../configs/serve_xlink_raw.yaml` # Sensor Stream Server Sensor Stream Server grabs data from the frame source, encodes it, and sends it out to a destination using zmq ![High-level overview of what Sensor Stream Server does](/files/-MTWivUTRrt8o0JZGAK7) 1. Sensor Stream Server is started with a path to a configuration yaml file as a command line argument 2. The configuration file defines: * Where the data should be streamed to (destination) * Frame Source (where the data is coming from - frame list/video/kinect/iphone) * What frame types will be streamed (color/depth/ir/confidence) * Encoders for each of the frame types ( null/libav/nvenc/zdepth) 3. `IReader` is constructed with parameters defined in the configuration file * The instantiated `IReader`'s frame types are defined by a vector of integers returned by `GetType()` method * 0 - if exists in vector then **color** is being sent (encoders available: null/libav/nvenc) * 1 - if exists in vector then **depth** is being sent (encoders available: null/libav/nvenc/zdepth) * 2 - if exists in vector then **ir** is being sent (encoders available: null/libav/nvenc) * 3 - if exists in vector then **confidence** is being sent (encoders available: null) * `IReader` will use `GetCurrentFrame()` in Step #5 to grab data from the frame source and create a `FrameStruct` for each defined frame type 4. Server creates a vector of `IEncoder`s for each frame type returned by `IReader`'s `GetType()` * The `IEncoder`s will be used to encode `FrameStruct`s in Step #5 * `NullEncoder` - passes raw data through * `LibAvEncoder` - uses [LibAv](https://libav.org/) to encode video frames * `NvEncoder` - uses [Nvidia Pipe](https://github.com/NVIDIA/NvPipe) to encode video frames * `ZDepthEncoder` - uses [Zdepth](https://github.com/catid/Zdepth) to encode depth frames 5. Server creates a vector of encoded `FrameStruct`s * Server is paced to transmit frames only as fast as the fps allows * `IReader` creates a vector of `FrameStruct`s * One for each frame type returned by `GetType()` * `IReader.GetCurrentFrame()` - Fills `FrameStruct` * if (`IReader.HasNextFrame()`) - Used to iterate * `IReader.NextFrame()` - Used to iterate * Server iterates over vector of `FrameStruct`s and `IEncoder` which returns an encoded `FrameStruct` * `IEncoder.AddFrameStruct()` - Attempts encoding `FrameStruct` * `IEncoder.HasNextPacket()` - Checks if encoded frame available * `IEncoder.CurrentFrameEncoded()` - Returns encoded frame 6. The vector of encoded `FrameStruct`s is sent to destination zmq socket as a zmq message * Call `CerealStructToString()` on vector of encoded frames to create string message * Create zmq message out of the string message * Call `socket.send()` passing through the zmq message * Log timings * Currently the frame rate ![Detailed overview of Sensor Stream Server data flow](/files/-Mevl9GV6qvNXYAC7yfA) The Sensor Stream Server can stream from 4 frame (data) sources: * **Images**: frames from images stored on the disk. * **Video**: encoded frames that have been captured using a color/depth camera, such as the Kinect. All video types supported by FFmpeg/Libav can be processed. * **Kinect**: Live Kinect DK frame data. * **iPhone**: Live iPhone RGBD data (must have lidar) Each of these data sources have their own implementation of `IReader`defined in `image_reader.h` which is able to create read data and write to `FrameStruct`s. * **ImageReader:** Can be used to stream public datasets and can be composed into MultiImageReader * **VideoFileReader:** Reads .mkv files like those created by [Azure Kinect DK recorder](https://docs.microsoft.com/en-us/azure/kinect-dk/azure-kinect-recorder) * **KinectReader:** Reads Azure Kinect DK stream * **IPhoneReader:** Reads ARFrame data For all four data sources, the data can be sent losslessly (very high bandwidth requirements), or compressed (20-50x lower bandwidth requirements), using Libav or NVCodec through the NVPipe. As any compression will affect quality, we recommend first experimenting with the Sensor Stream Tester to figure out the optimal levels for your use case. Server File: ## Building Sensor Stream Server Building Sensor Stream Server also builds Sensor Stream Client and Sensor Stream Tester. ``` git clone git@github.com:moetsi/Sensor-Stream-Pipe.git cd Sensor-Stream-Pipe mkdir build cd build cmake .. -DSSP_WITH_KINECT_SUPPORT=ON -DSSP_WITH_K4A_BODYTRACK=ON -DSSP_WITH_NVPIPE_SUPPORT=ON make ssp_server ``` You can turn off Kinect, Bodytrack and NVPipe support by changing the following to the `cmake ..` options to off: ``` -DSSP_WITH_KINECT_SUPPORT=OFF -DSSP_WITH_K4A_BODYTRACK=OFF -DSSP_WITH_NVPIPE_SUPPORT=OFF ``` ## Starting the Sensor Stream Server ``` ./bin/ssp_server ``` The Sensor Stream Server will start streaming frame data by default, but **it will not keep any frames if it is not connected to a client**. This is a zmq setting. From ssp\_server.cc: ``` // Do not accumulate packets if no client is connected socket.set(zmq::sockopt::immediate, true); ``` When it connects to Sensor Stream Client, the packets in the buffer will be sent first. After the buffer has emptied, the Sensor Stream Server will resume reading frames from the selected input in order, ensuring that no frames are dropped. The Sensor Stream Server configuration is stored in a YAML file. It includes Sensor Stream Client host and port, input data configuration and encoding configuration. The format of the file (encoding Kinect DK frame data with the Nvidia encoder) is as follows: ``` general: host: "192.168.1.64" port: 9999 log_level: "debug" log_file: "ssp_server.log" frame_source: type: "kinect" parameters: stream_color_video: True stream_depth_video: True stream_ir_video: True streaming_color_format: "K4A_IMAGE_FORMAT_COLOR_BGRA32" streaming_color_resolution: "K4A_COLOR_RESOLUTION_720P" streaming_depth_mode: "K4A_DEPTH_MODE_NFOV_UNBINNED" wired_sync_mode: "K4A_WIRED_SYNC_MODE_STANDALONE" streaming_rate: "K4A_FRAMES_PER_SECOND_30" absoluteExposureValue: 0 video_encoder: 0: #color type: "nvenc" codec_name: "NVPIPE_HEVC" input_format: "NVPIPE_RGBA32" bit_rate: 4000000 1: #depth type: "zdepth" 2: #ir type: "nvenc" codec_name: "NVPIPE_HEVC" input_format: "NVPIPE_UINT16" bit_rate: 15000000 ``` The `config/` folder includes a set of examples for all types of data using multiple encoders, codecs and parameters. # Sensor Stream Client Sensor Stream Client is the destination that receives the sent frames from Sensor Stream Server. The ssp\_client application receives network packets from the ssp\_server, de-serializes and decodes them, and makes them available for processing. For example, you could use the code with BundleFusion or another, comparable algorithm. This is beneficial if you’re ingesting multiple sensor streams, and want to run computer vision algorithms such as skeleton tracking. And also lends itself perfectly to projects involving sensors in multiple locations, i.e. if you want to combine streams from a variety of drones operating in different locations into one stream. You can have Sensor Stream Client receive multiple streams and ingest the data for a spatial computing/computer vision pipeline. ## Processing Examples To provide an example, [**ssp\_client\_opencv**](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_opencv.cc) converts the encoded frames into displayable OpenCV, whereas [**ssp\_client\_k4a**](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_k4a.cc) connects with remote Azure Kinect body tracking and [**ssp\_client\_pointcloud**](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_pointcloud.cc) generates a pointcloud from your Kinect DK data. These three different clients provide three examples of output format, but you can adapt this code for final processing however you like. Have a look at [ssp\_client\_template](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_template.cc) for a bare-bones template that you can play around with. ``` ./bin/ssp_client_opencv () () ./bin/ssp_client_k4a () () ./bin/ssp_client_pointcloud () () () ``` Due to the enqueuing process as described in the ssp\_server section, it is recommended that you start the ssp\_client application first. #### **Parallel processing** By default, the ssp\_client can receive frames from multiple servers in parallel, and will process the input using a [fair queuing method](http://zguide.zeromq.org/page:all#Divide-and-Conquer). # Sensor Stream Tester The Moetsi SSP has been designed to stream your sensor data in real-time. However, as streaming data in real-time is not reproducible, it is hard to test which parameters will work best for you. You need a way to "record results," so to speak. Thus, we built the Moetsi SSP Tester, which enables you to test how various settings and parameters affect latency, bandwidth and quality. The SSP Tester runs the full encoding process for an existing video or image dataset, and returns metrics for **MSE**, **PSNR**, **MSSIM**. It uses the same configuration file format as the Moetsi ssp\_server and supports the same input data and parameters to ensure that the results are comparable. ``` ./bin/ssp_tester () ``` Here is an example of the output for a Kinect DK video (color and depth data) with 20x compression: ``` ... [statistics];[0] [time];[0];32.4667;seconds [original_size];[0];240544293;bytes [compressed_size];[0];14909024;bytes [original_bandwidth];[0];59.2717;Mbps [compressed_bandwidth];[0];3.67368;Mbps [compression ratio];[0];16.1341;x [latency];[0];10.2854;ms [PSNR];[0];39.1411 [MSSIM];[0];0.938965;0.951397;0.93559;0.0 [statistics];[1] [time];[1];32.4667;seconds [original_size];[1];718110720;bytes [compressed_size];[1];51234739;bytes [original_bandwidth];[1];176.947;Mbps [compressed_bandwidth];[1];12.6246;Mbps [compression ratio];[1];14.0161;x [latency];[1];8.25051;ms [MSE];[1];2.0558 [MSE_4096];[1];2.05479 ... ``` The key results here are a **PSNR** of just **39 dB** and **latency** of only **8.25 ms**. # IReaders IReader's job is to be able to load data from a frame source into a FrameStruct for each frame type in the frame source `IReader`s interface with the frame source and create `FrameStruct`s. Based on the config yaml the Sensor Stream Server will create an `IReader` to pull data and create `FrameStructs`. If you want to [add a new sensor interface](/how-to-extend-sensor-stream-pipe/add-new-frame-source), you will need to make an implementation of the `IReader` interface that can pull data from the hardware and create a `FrameStruct`. ### [IReader Interface](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/ireader.h) ``` #pragma once #include "../structs/frame_struct.hpp" class IReader { public: virtual ~IReader() {} virtual std::vector> GetCurrentFrame() = 0; virtual std::vector GetType() = 0; virtual bool HasNextFrame() = 0; virtual void NextFrame() = 0; virtual void Reset() = 0; virtual void GoToFrame(unsigned int frame_id) = 0; virtual unsigned int GetCurrentFrameId() = 0; virtual unsigned int GetFps() = 0; }; ``` ## How IReader is interacted with by Sensor Stream Server * Sensor Stream Server is what uses the IReader * When Sensor Stream Server is started it reads the config.yaml to understand reader\_type * frames * which can then be MultiImageReader or ImageReader based on the parameter "path" * video * kinect * takes in parameters to build new ExtendedAzureConfig * iphone * **It then creates an IReader using the parameters** * **It then asks for "GetType()" which the IReader will respond with and array of types** * 0 - color * 1- depth * 2 - ir * 3 - confidence (iphone) * IReader knows how to respond based on the parameters it was passed when built * That will create IEncoders for each type * **It then asks for "GetFps()"** * IReader knows how to respond based on the parameters it was passed when built * **It then calls "GetCurrentFrame()" which gets provides a frame struct** * This is then encoded by the associated encoders * **It then calls HasNextFrame() and then NextFrame(), otherwise it calls Reset()** ### Available IReader Implementations * [ImageReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/image_reader.cc) * [MultiImageReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/multi_image_reader.cc) * [VideoFileReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/video_file_reader.cc) * [KinectReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/kinect_reader.cc) * [IPhoneReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/iphone_reader.mm) * OAK-D (coming soon!) # FrameStruct IReader pulls data from the frame source into a FrameStruct on GetCurrentFrame(). FrameStruct contains all the information necessary to be received and decoded by a Sensor Stream Client. `FrameStruct`s are a sample of sensor data of a certain data type. Sensors that collect different data can send different frame types. Azure Kinect can stream: RGB (color), depth, and IR data. This means that there are 3 frame types that Sensor Stream Server can send when streaming Azure Kinect data. Each frame type gets its own `FrameStruct` when sampling data. Sensor Stream Server reads the config yaml and depending on its configuration will create an `IReader` that pulls data from the frame source and generates `FrameStruct`s for each frame type. `FrameStruct` contains additional information to binary sensor data. It also contains information like the CameraCalibrationStruct, which provides the intrinsics of the sensor. This can be used in downstream computer vision/spatial computing pipelines. `FrameStruct` also contains information for how it has been encoded in `CodecParamsStruct`. This is so the receiving end, Sensor Stream Pipe, knows how to decode the frame struct. `FrameStruct`s are sent by the Sensor Stream Pipe by being converted to string through CerealStructToString method. The string is then packaged as a [zmq message](https://zeromq.org/messages/) and sent through a [zmq socket](https://zeromq.org/socket-api/). ### [FrameStruct Interface](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/structs/frame_struct.hpp) (frame\_struct.hpp) | Interfaces | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | **FrameStruct** | Actual frame data captured (one for each frame type) | | **CodecParamsStruct** | Codec information (how the frame was encoded by the IEncoder) | | **CameraCalibrationStruct** | Intrinsics of camera (information about the sensor that captured the `FrameStruct`- for computer vision/spatial computing pipelines) | | **CerealStructToString** | Converts FrameStruct to a string (which will be used by Server to send message) | | **ParseCerealStructFromString** | Creates FrameStruct from string (will be used by Client to read message from server) | | FrameStructToString | Same as CerealStructToString but takes in a pointer instead of a reference | ``` #pragma once #include #include #include #include #include #include #include #include "../utils/utils.h" struct CameraCalibrationStruct { // 0: Kinect parameters short type = -1; std::vector data; std::vector extra_data; CameraCalibrationStruct() {} CameraCalibrationStruct(unsigned int t, std::vector d, std::vector ed) : type(t), data(d), extra_data(ed) {} template void serialize(Archive &ar) { ar(type, data, extra_data); } }; struct CodecParamsStruct { // 0: av parameters, 1: nvPipe parameters, 2: zDepth parameters short type = -1; std::vector data; std::vector extra_data; CodecParamsStruct() {} CodecParamsStruct(unsigned int t, std::vector d, std::vector ed) : type(t), data(d), extra_data(ed) {} void SetData(std::vector &d) { data = d; } void SetExtraData(std::vector &ed) { extra_data = ed; } template void serialize(Archive &ar) { ar(type, data, extra_data); } }; struct FrameStruct { // message id, currenly set to 0 //This is to be used as "versioning", so if how messages are updated so that Sensor Stream Client //must interpret different "versions" of messages then this field will indicate the message version unsigned short message_type; // 0 for color, 1 for depth, 2 for ir, 3 for confidence unsigned short frame_type; // 0 for image frames, 1 for libav packets, 2 for raw RGBA data, 3 for raw // GRAY16LE data, 4 for NvPipe packets, 5 for raw 32FC1 data, 6 for YUV data // 7 for raw U8C1 data //This is used to select the decoder on the "receiving" side of the Pipe //Not all frame_type + frame_data_type combinations "make sense" or will be used unsigned short frame_data_type; // random 16 char string that uniquely ids the frame stream //Some decoders (like video) are stateful and so must keep track of streams //This is automatically generated std::string stream_id; // frame binary data //We use a vector to know the size, basically a vector of bytes to store binary data std::vector frame; // codec info for video frames, null for image frames //Video decoder needs to know about the last receive frame //Requires to know the codec as well as additional parameters CodecParamsStruct codec_data; // codec info for video frames, null for image frames CameraCalibrationStruct camera_calibration_data; // optional: scene description std::string scene_desc; // 0 for color, 1 for depth: currently redundant with frameType, but // distinction may be needed in the future unsigned int sensor_id; // integer device id: distingish between devices in the same scene //Can be set by user unsigned int device_id; // current frame number (increases over time) //Increases by 1 for each frame automatically when SSP server starts unsigned int frame_id; //Use for logging and timing to understand processing speeds std::vector timestamps; //Serialize method (not used by Server but is available) template void serialize(Archive &ar) { ar(message_type, frame_type, frame_data_type, stream_id, frame, codec_data, camera_calibration_data, scene_desc, sensor_id, device_id, frame_id, timestamps); } }; template static const std::string CerealStructToString(const T &t) { std::ostringstream os(std::ios::binary); { cereal::BinaryOutputArchive oarchive(os); oarchive(t); } return os.str(); } template static const std::string FrameStructToString(const T *t) { std::ostringstream os(std::ios::binary); { cereal::BinaryOutputArchive oarchive(os); oarchive(*t); } return os.str(); } template static T ParseCerealStructFromString(std::string &data) { T frame_in; std::istringstream is(data, std::ios::binary); { cereal::BinaryInputArchive iarchive(is); iarchive(frame_in); } return frame_in; } ``` # Config File Key sections of config file ### Destination Host and port will be used by the Sensor Stream Server to connect to this socket. ### Frame Source The frame source tells Sensor Stream Server where to get data from ### Frame Types This tells Sensor Stream Server what types of frames should be sent which is used by Sensor Stream Server to implement the right IReader interface. ### Encodings per Frame Types For each frame type listed an encoder is listed which will be used to encode the frames. # IEncoders ## Overview IEncoders are used to encode data before serializing and sending. Once an IReader has pulled the FrameStructs, the IEncoders are called for each FrameStruct in Sensor Stream Server to encode the frames. The encoder converts the raw frame into an encoded frame (represented as a packet in the code). Multiple raw frames may be needed to produce a packet. The code accounts for that and sends frames to the encoder until the encoder returns a frame. The encoders know what type of frame they are encoding by `frame_type` from the FrameStruct. ### Table of What Encoders to Use For a Frame Type | Frame Type | Encoders to Use | | | -------------- | ------------------------------------------------------------------------------------------------------------------------------- | - | | 0 (color) |

Null, LibAv, Nvenc all are good, need a good codec, explained below

H264 and H265 are recommended

| | | 1 (depth) | Null, **ZDepth** | | | 2 (ir) | **Null** (ZDepth is not tuned well to IR) | | | 3 (confidence) | **Null** | | ## NullEncoder In config.yaml: **type: "null"** There are no parameters other than “type”. The Null encoder does no processing and just returns the original uncompressed frame #### Performance on Frame Types Color/Depth/IR - Pixel/voxel are sent exactly as captured, at the cost of very high bandwidth. The codec is also very fast, unless the stream is being limited by network bandwidth.
## LibAvEncoder LibAv is supported for hardware encoding on many platforms so check if there is support for libav on your desired platform and use a codec that is platform specific for best results. Example: iPhones have videotoolbox to accelerate encoding, see [the ios.yaml](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/configs/serve_ios_raw.yaml.in) In config.yaml: **type: "libav"**\ **codec\_name: \**\ **pix\_fmt: \ bit\_rate: \** #### Descriptions of Codec Fields * **codec\_name** - name of the codec to use. SSP supports all codes that LibAv supports. The simplest way to see these options is to run ffmpeg -encoders. The name to use is the abbreviated name of the codec (second column). Please check if the V(ideo) flag is supported. * **pix\_fmt** - pixel format to be used by the codec. Common values include yuv420 for color or gray12le for infrared data. You can check the list of supported pixel formats for each codec using ffmpeg -h encoder=libx265 * **bit\_rate** - target (average) bit rate for the encoder to use in bits. * **options** - codec specific parameters, see below for examples of common codecs ## NvEncoder Nvidia encoder has been deprecated but there is a fork here: [https://github.com/udnaan/NvPipe ]() It is possible to run nvenc with libav, if it was built with the adequate options. We have not tested it in SSP. Check the options using ffmpeg -h encoder=nvenc\_hevc (h265) or ffmpeg -h encoder=nvenc (h264). In config.yaml: **type: “nvenc”**\ **codec\_name: “NVPIPE\_HEVC”**\ **input\_format: “NVPIPE\_RGBA32” bit\_rate: 2000000** #### **Descriptions of Codec Fields** * **codec\_name** - name of the codec to use (NVPIPE\_HEVC or NVPIPE\_H264). * **input\_format** - input format of the data to be streamed: NVPIPE\_RGBA32 (for color data), NVPIPE\_UINT4, NVPIPE\_UINT8, NVPIPE\_UINT16 (for IR data), NVPIPE\_UINT32 * **bit\_rate** - target (average) bit rate for the encoder to use in bits. ## ZDepthEncoder Zdepth is from this github repo: In config.yaml: **type: “zdepth”**\ **send\_I\_frame\_interval: “30”** #### **Descriptions of Codec Fields** * **send\_I\_frame\_interval** - Omit to send only an I frame once. Check the discussion regarding these parameter [here](https://github.com/moetsi/Sensor-Stream-Pipe/issues/9) ZDepth encoder can send both full (“I”) and partial (“P”) frames. In the best case scenario, you would only need to send the first frame of the stream as a full frame; all other frames can be compressed and sent as partial frames. This is the default behaviour, as ZeroMQ guarantees that all frames are delivered. If ZeroMQ is compiled to not block when waiting for a frame; the previous guarantee does not hold. Thus, we’ve added the send\_I\_frame\_interval parameter to the ZDepth YAML config. This value defines how often to send full frames. This is implemented by setting a counter in the FrameServer’s ZDepth encoder, and setting the encoder to send a full frame when the counter is a multiple of the interval. This interaction is defined here ## **Codecs (to be used in LibAv or Nvenc)** ### H265 ``` type: "libav" codec_name: “libx265” pix_fmt: “yuv420p” bit_rate: 40000 options: preset: "veryfast" crf: "23" ``` #### **Descriptions of Codec Fields** This is a subset of common parameters. You can get the full list by running ffmpeg -h encoder=libx265 or accessing **preset** - Sets the tradeoff between the speed of encoding and compression efficiency\ **crf** - Sets video quality for variable bitrate video. The default is 28, and it should visually correspond to libx264 video at CRF 23. Setting both CRF and bitrate results on bitrate being ignored. #### Performance on Frame Types **Color** - The default yuv420p pixel format (it is most cameras, including Azure Kinect default pixel format) and CRF value of 28 achieve good results.\ **Depth** - Although you can encode 12 bit grayscale video, this is not recommended, as it will limit depth data to 4096 millimeters. We recommend you use the zdepth codec \ **IR** - We haven’t tested many encoding options for IR data. As with depth data, although you can encode 12 bit grayscale video, this is not recommended, as it will limit depth data to 4096. ZDepth is not ideal for IR data. On iOS can use `h265_videotoolbox`as a codec name, see the [ios.yaml](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/configs/serve_ios_raw.yaml.in) ### H264 ``` type: "libav" codec_name: “libx265” pix_fmt: “yuv420p” bit_rate: 40000 options: preset: "veryfast" crf: "23" ``` #### **Descriptions of Codec Fields** This is a subset of common parameters. You can get the full list by running ffmpeg -h encoder=libx264 or accessing **preset** - Sets the tradeoff between the speed of encoding and compression efficiency, \ **crf** - Sets video quality for variable bitrate video. The default is 28, and it should visually correspond to libx264 video at CRF 23. Setting both CRF and bitrate results on bitrate being ignored. #### Performance on Frame Types **Color** - As with h265, The default yuv420p pixel format and CRF value of 28 achieve good results, at the cost of a higher bitrate. \ **Depth** - No adequate settings. We recommend you use the zdepth codec \ **IR** - No adequate settings. On iOS can use `h264_videotoolbox`as a codec name, see the [ios.yaml](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/configs/serve_ios_raw.yaml.in) ## Decoding The decoder works “in reverse” compared with the encoder. For each frame type, when you first sent a frame, it sends all the information required to decode those frames (codec type and all other coded information needed, remember the data and extra data fields). The decoder of the client has a hashmap that keeps track of all these decoder information for all received frame streams. The correct decoder class is chosen according to the frame type. The hashmap keys are the randomly generated stream ids. For the FFMPEG, this information is used to build a full decoder, as you’d do when decoding a video locally. When a frame arrives, the decoder that was previously step-up is retrieved, and the encoded frame data is passed to the decoder. The decoder transforms this frame into a raw OpenCV image, with the same number of channels, width, height, .. as the original. The flow of calling this decoder is the same as with the encoder, but in reverse. You pass an encoded FrameStruct and get a decoded cv::Mat frame. The decoder is meant to be called as a part of a while hasNext(), get next() frame process of a network reader. # Receiving Frames ## NetworkReader The [NetworkReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/network_reader.cc) pulls information from the zmq socket. This follows the same IReader implementations as the other IReaders but is used on the client (receiving) side. ## Sensor Stream Client with OpenCV [ssp\_client\_opencv](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_opencv.cc) is a good template Sensor Stream Client implementation because it can visualize incoming data streams to show that the transfer is taking place. Sensor Stream Client creates a NetworkReader on the specified port from the command line. Once the zmq message has been decoded and provided as a vector of FrameStructs by the NetworkReader, the client then can visualize the frame data using an opencv window. # Add New Sensor Interface ## Overview Going to first make a new implementation of [IReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/ireader.h) to talk to new hardware. Checkout the [iPhone IReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/iphone_reader.mm) or [Kinect IReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/kinect_reader.cc) for inspiration on to get it done. You might then need to extend IFrame for possibly new frame types or data types. You can use parameters in the config.yaml to help define your IReader and make it flexible to things like fps, data types, resolution. Constructing your IReader will happen from those parameters. Make an example config.yaml to be used by ssp\_server.cc. Then update ssp\_server.cc to be able to read the config.yaml and create the IReader. If new data types are used and you want to use libav\_encoder.cc you will need to update it to handle new data types. Finally, you will need to update image\_converter.cc for any of the new data types in FrameStructToMat if you want to see it in sensor stream client. You made do some final rescaling to visualize in ssp\_client\_opencv.cc. If you need to add any libraries or deploy to new platforms will need to update cmake. ## 1. Create new IReader Implementation and example config file * Start by creating a new .yaml in /configs * Start with null encoders for each of the types * Good place to start is with [Kinect example](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/configs/serve_kinect_raw.yaml) config because of of how many parameters it has * From the parameters the IReader will need to return at least GetTypes() and GetFps() * You can create "parameters" under "frame\_source" to define what you need to correctly grab from your new frame source * Examples: * Kinect has stream rate and what streams * VideoReader has a path to the file * Then implement a new IReader * Requires the [IReader interface](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/image_reader.h) * IReader implementation can depend on the .yaml configuration parameters to alter function * Include error checking and graceful handling for hardware interface issues or configuration issues * Good example is [Kinect IReader](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/readers/kinect_reader.cc) If implementing a frame type that requires local processing, like body detection, you will need to have a function that can be called that returns detected bodies. This will require opening a connection to a sensor feed that automatically detects bodies each frame, then pulling in the bodies into frame data in the FrameStruct. ## **2. Update FrameStruct (if necessary, most likely it is)** * FrameStruct has fields * `frame_type` * `frame_data_type` * Likely that the new interface will possibly have a new frame\_type (color, depth, ir, confidence) * object will be a new frame\_type * Likely that the frame\_data\_type might be new * Maybe a new color space * Or confidence level is an int ## 3. Update encoder to handle frame\_type (optional) * Can create a new encoder or can update an existing encoder * Most likely will be updating libav if it has to do with color frames * If you create a new encoder will need to also implement decoding encoding frames * The encoder converts the raw frame into an encoded frame (represented as a packet in the code). Multiple raw frames may be needed to produce a packet. The code accounts for that and sends frames to the encoder until the encoder returns a frame. * For each frame type, when you first sent a frame, it sends all the information required to decode those frames (codec type and all other coded information needed, remember the data and extra data fields). * The decoder of the client has a hashmap that keeps track of all these decoder information for all received frame streams * The decoder transforms this frame into a raw OpenCV image, with the same number of channels, width, height, .. as the original. * Can just use null encoder if do not want to implement a new IEncoder or update an existing IEncoder ## 4. Update FrameStructToMat in image\_converter.cc and ssp\_client\_opencv * If planning to visualize a new frame\_type or frame\_data\_type will need to update image\_converter.cc * Follow the pattern for the different supported types currently in [image\_converter.cc](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/utils/image_converter.cc) * If necessary can also update [ssp\_client\_opencv](https://github.com/moetsi/Sensor-Stream-Pipe/blob/master/clients/ssp_client_opencv.cc) to normalize or change color data * Can see example of this for confidence streamed from iPhone into black, grey, white * Can see example of this for depth streamed from iPhone ## 5. Update cmake with additional libraries needed to build updated SSP * Current approach is pre-built binaries for each platform * If can create dep.tar.gz for the platform moetsi will host and support the new platform dependency * ffmpeg 4.3 as shared libraries without the GPL option. Path in the `dylib` is changed to use `@rpath` for easier linking. * OpenCV 3.4.13 as a static library, only core, imgproc, imgcodecs and highgui modules are built. * Cereal 1.3.0, header only * spdlog 1.8.2, header only but built as static library for faster compile * Zdepth (commit 9b333d9aec520 which includes a patch to generate `zdepthConfig.cmake`) * yaml-cpp 0.6.3 as a static library * libzmq 4.3.4 as a static library * cppzmq 4.7.1, header only * Can require more libraries on the platform to interact with specific hardware possibly ## 6. Update ssp\_server.cc to be able to read the config.yaml and create an IReader * Will need to add to the conditionals when reading the config.yaml and implement for the new frame source * Will need to feed the parameters to the IReader implementation constructor to create the correct IReader * This can require looking for the config.yaml file as is the case for iOS * In the case of plug-ins will also need to update to provide callable functionality as a plugin