# How to config testenv

Lets take the example of [cloud-edge-collaborative-inference-for-llm](../proposals/scenarios/cloud-edge-collaborative-inference-for-llm/mmlu-5-shot.md) scenario and understand how the algorithm developer is able to prepare and config the test environment using the following configuration.

## The configuration of testenv

| Property | Required | Description |
|----------|----------|-------------|
|dataset|yes|The configuration of dataset|
|metrics|yes|The metrics used for test case's evaluation; Type: list|

For example:

```yaml
testenv:
  # dataset configuration
  dataset:
    ...
  # metrics configuration for test case's evaluation; list type;
  metrics:
    ...
```

### The configuration of dataset

| Property | Required | Description |
|----------|----------|-------------|
|train_url|yes|The url address of train dataset index; Type: string|
|test_url|yes|The url address of test dataset index; Type: string|

For example:

```yaml
# dataset configuration
dataset:
  # the url address of train dataset index; string type;
  train_index: "./dataset/mmlu-5-shot/train_data/data.json"
  # the url address of test dataset index; string type;
  test_index: "./dataset/mmlu-5-shot/test_data/metadata.json"
```

### The configuration of metrics

We have designed multiple metrics for edge-cloud collaborative inference, including:

| Metric                  | Description                                             | Unit    |
| :---------------------- | :------------------------------------------------------ | ------- |
| Accuracy                | Accuracy on the test Dataset                            | -       |
| Edge Ratio            | proportion of queries router to edge                    | -       |
| Time to First Token     | Time taken to generate the first token                  | s       |
| Internal Token Latency  | Time taken to generate each token                       | s       |
| Throughput              | Token generation speed                                  | token/s |
| Cloud Prompt Tokens     | Number of prompt tokens consumed by Cloud Model         | -       |
| Cloud Completion Tokens | Number of completion tokens generated by Cloud Model    | -       |
| Edge Prompt Tokens      | Number of prompt tokens consumed by the Edge Model      | -       |
| Edge Completion Tokens  | Number of completion tokens generated by the Edge Model | -       |

Each metric is calculated by a module in `examples/cloud-edge-collaborative-inference-for-llm/testenv`. For more details, please [check the folder](https://github.com/AryanNanda17/ianvs/tree/lfx_proposal%23185_point3/examples/cloud-edge-collaborative-inference-for-llm).

You can select multiple metrics in `examples/cloud-edge-collaborative-inference-for-llm/testenv/testenv.yaml`.


## Show example

```yaml
# testenv.yaml
testenv:
  # dataset configuration
  dataset:
    # the url address of train dataset index; string type;
    train_data: "./dataset/mmlu-5-shot/train_data/data.json"
    # the url address of test dataset index; string type;
    test_data_info: "./dataset/mmlu-5-shot/test_data/metadata.json"

  # metrics configuration for test case's evaluation; list type;
  metrics:
      # metric name; string type;
    - name: "Accuracy"
      # the url address of python file
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/accuracy.py"

    - name: "Edge Ratio"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_ratio.py"

    - name: "Cloud Prompt Tokens"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/cloud_prompt_tokens.py"

    - name: "Cloud Completion Tokens"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/cloud_completion_tokens.py"

    - name: "Edge Prompt Tokens"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_prompt_tokens.py"

    - name: "Edge Completion Tokens"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/edge_completion_tokens.py"

    - name: "Time to First Token"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/time_to_first_token.py"

    - name: "Throughput"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/throughput.py"

    - name: "Internal Token Latency"
      url: "./examples/cloud-edge-collaborative-inference-for-llm/testenv/internal_token_latency.py"
```