TrustyAI-KServe conversions

Synopsis

Inference data is represented in TrustyAI1 using objects implementing the Prediction interface. The objects always include two general entities representing collections of inputs (PredictionInput) and outputs (PredictionOutput). These in turn are comprised of Features and Outputs, respectively.

{{< figure src="/Excalidraw/TrustyAI predictions.excalidraw.svg" alt="TrustyAI predictions.excalidraw.svg" >}}

Communication with KServe/ModelMesh deployed models is done, however, using gRPC, with the KServe v2 protocol.

{{< figure src="/Excalidraw/TrustyAI KServe.excalidraw.svg" alt="TrustyAI KServe.excalidraw.svg" >}}

The basis of prediction requests is the KServe ModelInferenceRequest message, where model responses are based on the ModelInferenceResponse message. Payloads (and data in general) are encoded as InferInputTensor and InferResponseTensor. Here we look at several scenarios to convert data to/from KServe tensors and TrustyAI Predictions. ### Single input

np codec

Currently, the conversion between PredictionInput and PredictionOutput to KServe Tensor format works in the following way: We assume a consistent datatype between the features and serialise it as a \((1, N_f)\) shaped tensor.

Note that although a JSON representation of the data is used, it is only for illustration of the structure. The TrustyAI structures are converted to Protobuf format.

{{< figure src="/Excalidraw/KServe TrustyAI schema 1.excalidraw.svg" alt="KServe TrustyAI schema 1.excalidraw.svg" >}}

This is using the default KServe conversion codec np.

pd codec

KServe also allows to use the pd conversion codec, where each feature is a \((1, 1)\) shaped Tensor. This allows for greater flexibility, especially in the common case where each feature has a different datatype. In TrustyAI, this conversion works the following way:

{{< figure src="/Excalidraw/KServe TrustyAI schema 2.excalidraw.svg" alt="KServe TrustyAI schema 2.excalidraw.svg" >}} #### Outputs

For the single input case, the output conversion is straightforward. We get a \((1, N_f)\) shaped tensor for the outputs, which allows for direct conversion to a single PredictionOutput, such that

{{< figure src="/Excalidraw/KServe TrustyAI schema 3.excalidraw.svg" alt="KServe TrustyAI schema 3.excalidraw.svg" >}}

Multiple inputs

There are basically two ways of dealing with the multiple inputs, such as in the single inputs case. In this case we will consider \(N_{obs}\) inputs, each with \(N_f\) features.

np codec

With the np codec, we simply produce a \((N_{obs}, N_f)\) shaped tensor, sharing the same datatype.

{{< figure src="/Excalidraw/KServe TrustyAI schema 4.excalidraw.svg" alt="KServe TrustyAI schema 4.excalidraw.svg" >}}

pd codec

{{< figure src="/Excalidraw/KServe TrustyAI schema 5.excalidraw.svg" alt="KServe TrustyAI schema 5.excalidraw.svg" >}}

Outputs

With multiple outputs come multiple outputs and here the situation is not as straightforward. Let’s assume that for the previous inputs we get two features as the inference result. Usually, the outputs will come as a \((1, N_{obs}\times N_f)\) shaped tensor, so we will need to reshape it. So we’ll have something like

{ 
    "model_name": "dummy3", 
    "model_version": "v0.0.1", 
    "id": "3aacc8a0-a3cc-47ce-bd03-614df833d18e", 
    "parameters": {}, 
    "outputs": [ 
        { 
            "name": "predict", 
            "shape": [ 3, 2 ], 
            "datatype": "INT64", 
            "data": [ 2, 2, 1, 4, 2, 2 ] 
        } 
    ] 
}

Since we know the final shape of the multiple outputs (with the shape value), we can reshape it using: {{< figure src="/Excalidraw/KServe TrustyAI schema 6.excalidraw.svg" alt="KServe TrustyAI schema 6.excalidraw.svg" >}}

Dataframe utilities

A collection of Prediction can be also represented as a Dataframe. Dataframe will represent the data in tabular format along with the necessary metadata related to the feature types, feature domains and if the feature belongs in the inputs or outputs.

{{< figure src="/Excalidraw/TrustyAI dataframe.excalidraw.svg" alt="TrustyAI dataframe.excalidraw.svg" >}} A new data structure, called a TensorDataframe is an extension of the Dataframe, sharing a common API, but adding a few convenience methods to convert either individual rows (or the entire data) to tensors. The first methods are - .rowAsSingleArrayInputTensor(), which converts a single row’s inputs to the np tensor format - .rowAsSingleDataframeInputTensor(), which converts a single row’s inputs to the pd tensor format {{< figure src="/Excalidraw/TrustyAI dataframe 2.excalidraw.svg" alt="TrustyAI dataframe 2.excalidraw.svg" >}} If an entire dataset needs to be converted (say for batch requests), we can use: - .rowBatchArrayInputTensor(), converts an entire dataframe to the np tensor format - rowBatchDataframeInputTensor(), converts an entire dataframe to the pd tensor format {{< figure src="/Excalidraw/TrustyAI dataframe 3.excalidraw.svg" alt="TrustyAI dataframe 3.excalidraw.svg" >}} ## Schemas

A table with the values of the schemas for each scenario.

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 1 output, no batch NP 1 1 1 1 [0,1] 1 [0,1]
1 input, 2 outputs, no batch NP 1 2 1 1 [1,1] 1 [1,2]
2 inputs, 1 output, no batch NP 2 1 1 1 [1,2] 1 [1,1]
2 inputs, 2 outputs, no batch NP 2 2 1 1 [1,2] 1 [1,2]
1 input, 1 output, no batch PD 1 1 1 1 [1] 1 [1]
1 input, 2 outputs, no batch PD 1 2 1 1 [1] 2 [1],[1]
2 inputs, 1 output, no batch PD 2 1 1 2 [1],[1] 1 [1]
2 inputs, 2 outputs, no batch PD 2 2 1 2 [1],[1] 2 [1],[1]
1 input, 1 output, batch NP 1 1 10 1 [10, 1] 1 [10, 1]
1 input, 2 outputs, batch NP 1 2 10 1 [10,1] 1 [10,2]
2 inputs, 1 output, batch NP 2 1 10 1 [10,2] 1 [10,1]
2 inputs, 2 outputs, batch NP 2 2 10 1 [10,2] 1 [10,2]
1 input, 1 output, batch PD 1 1 10 1 [1, 10] 1 [1, 10]
1 input, 2 outputs, batch PD 1 2 10 1 [1, 10] 2 [1, 10],[1, 10]
2 inputs, 1 output, batch PD 2 1 10 2 [1, 10],[1, 10] 1 [1, 10]
2 inputs, 2 outputs, batch PD 2 2 10 2 [1, 10],[1, 10] 2 [1, 10],[1, 10]

Which schemas have \(>1\) I/O counts?

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 2 outputs, no batch PD 1 2 1 1 [1] 2 [1],[1]
2 inputs, 1 output, no batch PD 2 1 1 2 [1],[1] 1 [1]
2 inputs, 2 outputs, no batch PD 2 2 1 2 [1],[1] 2 [1],[1]
1 input, 2 outputs, batch PD 1 2 10 1 [1, 10] 2 [1, 10],[1, 10]
2 inputs, 1 output, batch PD 2 1 10 2 [1, 10],[1, 10] 1 [1, 10]
2 inputs, 2 outputs, batch PD 2 2 10 2 [1, 10],[1, 10] 2 [1, 10],[1, 10]

If the input/output counts are \(>1\), then we definitely have a PD payload. This is regardless of batch size.

Which schemas have \(=1\) I/O counts?

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 1 output, no batch NP 1 1 1 1 [1] 1 [1]
1 input, 2 outputs, no batch NP 1 2 1 1 [1,1] 1 [1,2]
2 inputs, 1 output, no batch NP 2 1 1 1 [1,2] 1 [1,1]
2 inputs, 2 outputs, no batch NP 2 2 1 1 [1,2] 1 [1,2]
1 input, 1 output, no batch PD 1 1 1 1 [1] 1 [1]
1 input, 1 output, batch NP 1 1 10 1 [10, 1] 1 [10, 1]
1 input, 2 outputs, batch NP 1 2 10 1 [10,1] 1 [10,2]
2 inputs, 1 output, batch NP 2 1 10 1 [10,2] 1 [10,1]
2 inputs, 2 outputs, batch NP 2 2 10 1 [10,2] 1 [10,2]
1 input, 1 output, batch PD 1 1 10 1 [1, 10] 1 [1, 10]

If the top input/ouput counts are \(=1\), then we definitely have an NP (the PD in there just means a 1 input, 1 output PD, which is indistinguishable).

Let’s look at the inputs only. The first case is when the count is \(=1\).

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 1 output, no batch NP 1 1 1 1 [0,1] 1 [1]
1 input, 2 outputs, no batch NP 1 2 1 1 [1,1] 1 [1,2]
2 inputs, 1 output, no batch NP 2 1 1 1 [1,2] 1 [1,1]
2 inputs, 2 outputs, no batch NP 2 2 1 1 [1,2] 1 [1,2]
1 input, 1 output, no batch PD 1 1 1 1 [1] 1 [1]
1 input, 2 outputs, no batch PD 1 2 1 1 [1] 2 [1],[1]
1 input, 1 output, batch NP 1 1 10 1 [10, 1] 1 [10, 1]
1 input, 2 outputs, batch NP 1 2 10 1 [10,1] 1 [10,2]
2 inputs, 1 output, batch NP 2 1 10 1 [10,2] 1 [10,1]
2 inputs, 2 outputs, batch NP 2 2 10 1 [10,2] 1 [10,2]
1 input, 1 output, batch PD 1 1 10 1 [1, 10] 1 [1, 10]
1 input, 2 outputs, batch PD 1 2 10 1 [1, 10] 2 [1, 10],[1, 10]

Now let’s look at the case were \(>1\)

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
2 inputs, 1 output, no batch PD 2 1 1 2 [1],[1] 1 [1]
2 inputs, 2 outputs, no batch PD 2 2 1 2 [1],[1] 2 [1],[1]
2 inputs, 1 output, batch PD 2 1 10 2 [1, 10],[1, 10] 1 [1, 10]
2 inputs, 2 outputs, batch PD 2 2 10 2 [1, 10],[1, 10] 2 [1, 10],[1, 10]

Logic:

flowchart TD
    A{Input count = 1?} -->|Yes| B{Shape 0 > 1}
    B -->|Yes| D[NP batch]
    B -->|No| C{Just one\nelement?}
    C -->|No| F[Either NP, no batch, multi-featute\nOR\nPD, batch, 1 feature]
    C -->|Yes| G[1 feature, no batch]
    A -->|No| H{Single element\nshapes?}
    H -->|Yes| I[Multi-feature\nPD\nNo batch]
    H -->|No| J[Multi-feature\nPD\nBatch]

Regarding the outputs, let’s first look at the case where the output counts are \(=1\)

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 1 output, no batch NP 1 1 1 1 [1] 1 [1]
1 input, 2 outputs, no batch NP 1 2 1 1 [1,1] 1 [1,2]
2 inputs, 1 output, no batch NP 2 1 1 1 [1,2] 1 [1,1]
2 inputs, 2 outputs, no batch NP 2 2 1 1 [1,2] 1 [1,2]
1 input, 1 output, no batch PD 1 1 1 1 [1] 1 [1]
2 inputs, 1 output, no batch PD 2 1 1 2 [1],[1] 1 [1]
1 input, 1 output, batch NP 1 1 10 1 [10, 1] 1 [10, 1]
1 input, 2 outputs, batch NP 1 2 10 1 [10,1] 1 [10,2]
2 inputs, 1 output, batch NP 2 1 10 1 [10,2] 1 [10,1]
2 inputs, 2 outputs, batch NP 2 2 10 1 [10,2] 1 [10,2]
1 input, 1 output, batch PD 1 1 10 1 [1, 10] 1 [1, 10]

We can see the same logic as in the inputs. When the output counts \(>1\):

Scenario Format # Input features # Output features Batch size Top inputs count Input(s) shape Top output counts Output(s) shape
1 input, 2 outputs, no batch PD 1 2 1 1 [1] 2 [1],[1]
2 inputs, 2 outputs, no batch PD 2 2 1 2 [1],[1] 2 [1],[1]
1 input, 2 outputs, batch PD 1 2 10 1 [1, 10] 2 [1, 10],[1, 10]
2 inputs, 2 outputs, batch PD 2 2 10 2 [1, 10],[1, 10] 2 [1, 10],[1, 10] 
flowchart TD
    A{Output count = 1?} -->|Yes| B{Shape 0 > 1}
    B -->|Yes| D[NP batch]
    B -->|No| C{Just one\nelement?}
    C -->|No| F[Either NP, no batch, multi-featute\nOR\nPD, batch, 1 feature]
    C -->|Yes| G[1 feature, no batch]
    A -->|No| H{Single element\nshapes?}
    H -->|Yes| I[Multi-feature\nPD\nNo batch]
    H -->|No| J[Multi-feature\nPD\nBatch]

Footnotes

  1. https://github.com/trustyai-explainability/trustyai-explainability↩︎