FrameStruct
IReader pulls data from the frame source into a FrameStruct on GetCurrentFrame(). FrameStruct contains all the information necessary to be received and decoded by a Sensor Stream Client.
FrameStructs are a sample of sensor data of a certain data type. Sensors that collect different data can send different frame types.
Azure Kinect can stream: RGB (color), depth, and IR data. This means that there are 3 frame types that Sensor Stream Server can send when streaming Azure Kinect data. Each frame type gets its own FrameStruct when sampling data.
Sensor Stream Server reads the config yaml and depending on its configuration will create an IReader that pulls data from the frame source and generates FrameStructs for each frame type.
FrameStruct contains additional information to binary sensor data. It also contains information like the CameraCalibrationStruct, which provides the intrinsics of the sensor. This can be used in downstream computer vision/spatial computing pipelines.
FrameStruct also contains information for how it has been encoded in CodecParamsStruct. This is so the receiving end, Sensor Stream Pipe, knows how to decode the frame struct.
FrameStructs are sent by the Sensor Stream Pipe by being converted to string through CerealStructToString method. The string is then packaged as a zmq message and sent through a zmq socket.

​FrameStruct Interface (frame_struct.hpp)

Interfaces
Description
FrameStruct
Actual frame data captured (one for each frame type)
CodecParamsStruct
Codec information (how the frame was encoded by the IEncoder)
CameraCalibrationStruct
Intrinsics of camera (information about the sensor that captured the FrameStruct- for computer vision/spatial computing pipelines)
CerealStructToString
Converts FrameStruct to a string (which will be used by Server to send message)
ParseCerealStructFromString
Creates FrameStruct from string (will be used by Client to read message from server)
FrameStructToString
Same as CerealStructToString but takes in a pointer instead of a reference
1
#pragma once
2
​
3
#include <iterator>
4
#include <vector>
5
​
6
#include <cereal/archives/binary.hpp>
7
#include <cereal/cereal.hpp>
8
#include <cereal/types/base_class.hpp>
9
#include <cereal/types/memory.hpp>
10
#include <cereal/types/vector.hpp>
11
​
12
#include "../utils/utils.h"
13
​
14
struct CameraCalibrationStruct {
15
// 0: Kinect parameters
16
short type = -1;
17
std::vector<unsigned char> data;
18
std::vector<unsigned char> extra_data;
19
​
20
CameraCalibrationStruct() {}
21
​
22
CameraCalibrationStruct(unsigned int t, std::vector<unsigned char> d,
23
std::vector<unsigned char> ed)
24
: type(t), data(d), extra_data(ed) {}
25
​
26
template <class Archive> void serialize(Archive &ar) {
27
ar(type, data, extra_data);
28
}
29
};
30
​
31
struct CodecParamsStruct {
32
// 0: av parameters, 1: nvPipe parameters, 2: zDepth parameters
33
short type = -1;
34
std::vector<unsigned char> data;
35
std::vector<unsigned char> extra_data;
36
​
37
CodecParamsStruct() {}
38
​
39
CodecParamsStruct(unsigned int t, std::vector<unsigned char> d,
40
std::vector<unsigned char> ed)
41
: type(t), data(d), extra_data(ed) {}
42
​
43
void SetData(std::vector<unsigned char> &d) { data = d; }
44
​
45
void SetExtraData(std::vector<unsigned char> &ed) { extra_data = ed; }
46
​
47
template <class Archive> void serialize(Archive &ar) {
48
ar(type, data, extra_data);
49
}
50
};
51
​
52
struct FrameStruct {
53
​
54
// message id, currenly set to 0
55
//This is to be used as "versioning", so if how messages are updated so that Sensor Stream Client
56
//must interpret different "versions" of messages then this field will indicate the message version
57
unsigned short message_type;
58
​
59
// 0 for color, 1 for depth, 2 for ir, 3 for confidence
60
unsigned short frame_type;
61
​
62
// 0 for image frames, 1 for libav packets, 2 for raw RGBA data, 3 for raw
63
// GRAY16LE data, 4 for NvPipe packets, 5 for raw 32FC1 data, 6 for YUV data
64
// 7 for raw U8C1 data
65
//This is used to select the decoder on the "receiving" side of the Pipe
66
//Not all frame_type + frame_data_type combinations "make sense" or will be used
67
unsigned short frame_data_type;
68
​
69
// random 16 char string that uniquely ids the frame stream
70
//Some decoders (like video) are stateful and so must keep track of streams
71
//This is automatically generated
72
std::string stream_id;
73
​
74
// frame binary data
75
//We use a vector to know the size, basically a vector of bytes to store binary data
76
std::vector<unsigned char> frame;
77
​
78
// codec info for video frames, null for image frames
79
//Video decoder needs to know about the last receive frame
80
//Requires to know the codec as well as additional parameters
81
CodecParamsStruct codec_data;
82
​
83
// codec info for video frames, null for image frames
84
CameraCalibrationStruct camera_calibration_data;
85
​
86
// optional: scene description
87
std::string scene_desc;
88
​
89
// 0 for color, 1 for depth: currently redundant with frameType, but
90
// distinction may be needed in the future
91
unsigned int sensor_id;
92
​
93
// integer device id: distingish between devices in the same scene
94
//Can be set by user
95
unsigned int device_id;
96
​
97
// current frame number (increases over time)
98
//Increases by 1 for each frame automatically when SSP server starts
99
unsigned int frame_id;
100
​
101
//Use for logging and timing to understand processing speeds
102
std::vector<unsigned long> timestamps;
103
​
104
//Serialize method (not used by Server but is available)
105
template <class Archive> void serialize(Archive &ar) {
106
ar(message_type, frame_type, frame_data_type, stream_id, frame, codec_data,
107
camera_calibration_data, scene_desc, sensor_id, device_id, frame_id,
108
timestamps);
109
}
110
};
111
​
112
template <typename T>
113
static const std::string CerealStructToString(const T &t) {
114
std::ostringstream os(std::ios::binary);
115
{
116
cereal::BinaryOutputArchive oarchive(os);
117
oarchive(t);
118
}
119
​
120
return os.str();
121
}
122
​
123
template <typename T> static const std::string FrameStructToString(const T *t) {
124
std::ostringstream os(std::ios::binary);
125
{
126
cereal::BinaryOutputArchive oarchive(os);
127
oarchive(*t);
128
}
129
​
130
return os.str();
131
}
132
​
133
template <typename T> static T ParseCerealStructFromString(std::string &data) {
134
T frame_in;
135
std::istringstream is(data, std::ios::binary);
136
{
137
cereal::BinaryInputArchive iarchive(is);
138
iarchive(frame_in);
139
}
140
return frame_in;
141
}
Copied!
Last modified 3mo ago