TrustyAI-KServe conversions

Synopsis

Inference data is represented in TrustyAI¹ using objects implementing the Prediction interface. The objects always include two general entities representing collections of inputs (PredictionInput) and outputs (PredictionOutput). These in turn are comprised of Features and Outputs, respectively.

Communication with KServe/ModelMesh deployed models is done, however, using gRPC, with the KServe v2 protocol.

The basis of prediction requests is the KServe ModelInferenceRequest message, where model responses are based on the ModelInferenceResponse message. Payloads (and data in general) are encoded as InferInputTensor and InferResponseTensor. Here we look at several scenarios to convert data to/from KServe tensors and TrustyAI Predictions.

Single input

np codec

Currently, the conversion between PredictionInput and PredictionOutput to KServe Tensor format works in the following way: We assume a consistent datatype between the features and serialise it as a $(1, N_f)$ shaped tensor.

Note that although a JSON representation of the data is used, it is only for illustration of the structure. The TrustyAI structures are converted to Protobuf format.

This is using the default KServe conversion codec np.

pd codec

KServe also allows to use the pd conversion codec, where each feature is a $(1, 1)$ shaped Tensor. This allows for greater flexibility, especially in the common case where each feature has a different datatype. In TrustyAI, this conversion works the following way:

Outputs

For the single input case, the output conversion is straightforward. We get a $(1, N_f)$ shaped tensor for the outputs, which allows for direct conversion to a single PredictionOutput, such that

Multiple inputs

There are basically two ways of dealing with the multiple inputs, such as in the single inputs case. In this case we will consider $N_{obs}$ inputs, each with $N_f$ features.

np codec

With the np codec, we simply produce a $(N_{obs}, N_f)$ shaped tensor, sharing the same datatype.

pd codec

Outputs

With multiple outputs come multiple outputs and here the situation is not as straightforward. Let’s assume that for the previous inputs we get two features as the inference result. Usually, the outputs will come as a $(1, N_{obs}\times N_f)$ shaped tensor, so we will need to reshape it. So we’ll have something like

{ 
	"model_name": "dummy3", 
	"model_version": "v0.0.1", 
	"id": "3aacc8a0-a3cc-47ce-bd03-614df833d18e", 
	"parameters": {}, 
	"outputs": [ 
		{ 
			"name": "predict", 
			"shape": [ 3, 2 ], 
			"datatype": "INT64", 
			"data": [ 2, 2, 1, 4, 2, 2 ] 
		} 
	] 
}

Since we know the final shape of the multiple outputs (with the shape value), we can reshape it using:

Dataframe utilities

A collection of Prediction can be also represented as a Dataframe. Dataframe will represent the data in tabular format along with the necessary metadata related to the feature types, feature domains and if the feature belongs in the inputs or outputs.

• .rowAsSingleArrayInputTensor(), which converts a single row’s inputs to the np tensor format
• .rowAsSingleDataframeInputTensor(), which converts a single row’s inputs to the pd tensor format

  

  If an entire dataset needs to be converted (say for batch requests), we can use:
• .rowBatchArrayInputTensor(), converts an entire dataframe to the np tensor format
• rowBatchDataframeInputTensor(), converts an entire dataframe to the pd tensor format

  

Schemas

A table with the values of the schemas for each scenario.

Which schemas have $>1$ I/O counts?

If the input/output counts are $>1$, then we definitely have a PD payload. This is regardless of batch size.

Which schemas have $=1$ I/O counts?

If the top input/ouput counts are $=1$, then we definitely have an NP (the PD in there just means a 1 input, 1 output PD, which is indistinguishable).

Let’s look at the inputs only. The first case is when the count is $=1$.

Now let’s look at the case were $>1$

Logic:

flowchart TD
	A{Input count = 1?} -->|Yes| B{Shape 0 > 1}
	B -->|Yes| D[NP batch]
	B -->|No| C{Just one\nelement?}
	C -->|No| F[Either NP, no batch, multi-featute\nOR\nPD, batch, 1 feature]
	C -->|Yes| G[1 feature, no batch]
	A -->|No| H{Single element\nshapes?}
	H -->|Yes| I[Multi-feature\nPD\nNo batch]
	H -->|No| J[Multi-feature\nPD\nBatch]

Regarding the outputs, let’s first look at the case where the output counts are $=1$

We can see the same logic as in the inputs. When the output counts $>1$:

flowchart TD
	A{Output count = 1?} -->|Yes| B{Shape 0 > 1}
	B -->|Yes| D[NP batch]
	B -->|No| C{Just one\nelement?}
	C -->|No| F[Either NP, no batch, multi-featute\nOR\nPD, batch, 1 feature]
	C -->|Yes| G[1 feature, no batch]
	A -->|No| H{Single element\nshapes?}
	H -->|Yes| I[Multi-feature\nPD\nNo batch]
	H -->|No| J[Multi-feature\nPD\nBatch]