5.4 KiB

Raw Blame History

Introduction to benchmarks

Overview of supported benchmarks

The benchmark suite contains several benchmarks that can be used to evaluate the performance of the Asterinas platform. The following benchmarks are supported:

Sysbench
Membench
Iperf

Sysbench

Sysbench is a scriptable benchmark tool that evaluates system performance. It includes five kinds of tests: CPU, memory, file I/O, mutex performance, and thread performance. Detailed usage and options can be found by using:

sysbench --help
sysbench --test=<test_name> help

Here we list some general commands for evaluation:

# CPU test
sysbench --test=cpu --cpu-max-prime=<N> --num-threads=<N> run

# Thread test
sysbench --test=threads --thread-yields=<N> --num-threads=<N> --max-time=<N> run

# Mutex test
sysbench --test=mutex --mutex-num=<N> --mutex-locks=<N> --num-threads=<N>

# File test, the file-total-size and file-num of prepare and run must be consistent 
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> prepare
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> --file-test-mode=<Type> --file-block-size=<N><K,M,G> --max-time=<N> run

# Memory test
sysbench --test=memory --memory-block-size=<N><K,M,G> --memory-access-mode=<Type> --memory-oper=<Type> run

Membench

Membench is used to establish a baseline for memory bandwidth and latency. For specific usage and options, use:

membench --help

Here we list some general commands to use membench:

# Measure the latency of mmap
membench -runtime=5 -dir=/dev/zero -size=<N><K,M,G> -engine=mmap_lat

# Measure the latency of page fault handling. The size must be consistent with the file size.
membench -runtime=5 -dir=path_to_a_file -size=<N><K,M,G> -copysize=<N><K,M,G> -mode=<Type> -engine=page_fault 

# This is a easy way to generate a file with target size in Linux.
# The following command can create a file named 512K.file with the size 512K.
dd if=/dev/zero of=512K.file bs=1K count=512

Iperf

iPerf is a tool for actively measuring the maximum achievable bandwidth on IP networks. Usage and options are detailed in:

iperf3 -h

iperf can run in the following instructions:

export HOST_ADDR=127.0.0.1
export HOST_PORT=8888
iperf3 -s -B $HOST_ADDR -p $HOST_PORT -D # Start the server as a daemon
iperf3 -c $HOST_ADDR -p $HOST_PORT # Start the client

Note that a variant of iperf3 can measure the performance of vsock. But the implemented vsock has not been verified to work well in it.

Add benchmark to benchmark CI

To add a new benchmark to the Asternias Continuous Integration (CI) system, follow these detailed steps:

Step 1: Add the Benchmark to the `asterinas/test/benchmarks` Directory

Create the Benchmark Directory:
- Navigate to asterinas/test/benchmarks.
- Create a new directory named after your benchmark, e.g., getpid.
Create the Necessary Files:
- config.json:
```
{
    "alert_threshold": "125%",
    "pattern": "Syscall average latency:",
    "field": "4"
}
```
  - alert_threshold: Set the threshold for alerting. If the benchmark result exceeds this threshold, an alert will be triggered.
  - pattern: Define the pattern to extract the benchmark result from the output.
  - field: Specify the index of the result in the extracted output.
  For example, if the benchmark output is "Syscall average latency: 1000 ns", the pattern is "Syscall average latency:", and the field is "4". jq will extract "1000" as the benchmark result.
- result_template.json:
```
[
    {
        "name": "Average Syscall Latency on Linux",
        "unit": "ns",
        "value": 0,
        "extra": "linux_avg"
    },
    {
        "name": "Average Syscall Latency on Asterinas",
        "unit": "ns",
        "value": 0,
        "extra": "aster_avg"
    }
]
```
  - Adjust name and unit according to your benchmark specifics.
- run.sh:
```
#!/bin/bash

/benchmark/bin/getpid
```
  - This script runs the benchmark. Ensure the path to the benchmark binary is correct. asterinas/test/Makefile handles the benchmark binaries.

Step 2: Update the `asterinas/.github/benchmarks.yml` File

Edit the Benchmarks Configuration:
- Open asterinas/.github/benchmarks.yml.
- Add your benchmark to the strategy.matrix.benchmark list:
```
strategy:
  matrix:
    benchmark: [getpid]
  fail-fast: false
```

Step 3: Test the Benchmark Locally

Run the Benchmark:
- Execute the following command to test the benchmark locally:
```
cd asterinas
bash test/benchmark/bench_linux_aster.sh getpid
```
- Ensure the benchmark runs successfully and check the results in asterinas/result_getpid.json.

Additional Considerations

Validation: After adding and testing the benchmark, ensure that the CI pipeline correctly integrates the new benchmark by triggering a CI build.
Documentation: Update any relevant documentation to include the new benchmark, explaining its purpose and how to interpret its results.

By following these steps, you will successfully integrate a new benchmark into the Asternias CI system, enhancing its capability to evaluate platform performance.

5.4 KiB Raw Blame History