asterinas/test/benchmark

Introduction to benchmarks

Overview of supported benchmarks

The benchmark suite contains several benchmarks that can be used to evaluate the performance of the Asterinas platform. The following benchmarks are supported:

• Sysbench
• Membench
• Iperf

Sysbench

Sysbench is a scriptable benchmark tool that evaluates system performance. It includes five kinds of tests: CPU, memory, file I/O, mutex performance, and thread performance. Detailed usage and options can be found by using:

sysbench --help
sysbench --test=<test_name> help

Here we list some general commands for evaluation:

# CPU test
sysbench --test=cpu --cpu-max-prime=<N> --num-threads=<N> run

# Thread test
sysbench --test=threads --thread-yields=<N> --num-threads=<N> --max-time=<N> run

# Mutex test
sysbench --test=mutex --mutex-num=<N> --mutex-locks=<N> --num-threads=<N>

# File test, the file-total-size and file-num of prepare and run must be consistent 
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> prepare
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> --file-test-mode=<Type> --file-block-size=<N><K,M,G> --max-time=<N> run

# Memory test
sysbench --test=memory --memory-block-size=<N><K,M,G> --memory-access-mode=<Type> --memory-oper=<Type> run

Membench

Membench is used to establish a baseline for memory bandwidth and latency. For specific usage and options, use:

membench --help

Here we list some general commands to use membench:

# Measure the latency of mmap
membench -runtime=5 -dir=/dev/zero -size=<N><K,M,G> -engine=mmap_lat

# Measure the latency of page fault handling. The size must be consistent with the file size.
membench -runtime=5 -dir=path_to_a_file -size=<N><K,M,G> -copysize=<N><K,M,G> -mode=<Type> -engine=page_fault 

# This is a easy way to generate a file with target size in Linux.
# The following command can create a file named 512K.file with the size 512K.
dd if=/dev/zero of=512K.file bs=1K count=512

Iperf

iPerf is a tool for actively measuring the maximum achievable bandwidth on IP networks. Usage and options are detailed in:

iperf3 -h

iperf can run in the following instructions:

export HOST_ADDR=127.0.0.1
export HOST_PORT=8888
iperf3 -s -B $HOST_ADDR -p $HOST_PORT -D # Start the server as a daemon
iperf3 -c $HOST_ADDR -p $HOST_PORT # Start the client

Note that a variant of iperf3 can measure the performance of vsock. But the implemented vsock has not been verified to work well in it.

Add benchmark to benchmark CI

To add a new benchmark to the Asternias Continuous Integration (CI) system, follow these detailed steps:

Step 1: Add the Benchmark to the asterinas/test/benchmarks Directory

1. Create the Benchmark Directory:

   • Navigate to asterinas/test/benchmarks.
   • Create a new directory named after your benchmark, e.g., getpid.

2. Create the Necessary Files:

   • config.json:

      {
        "alert_threshold": "125%",
        "alert_tool": "customBiggerIsBetter",
        "search_pattern": "134.22",
        "result_index": "2",
        "description": "The memory bandwidth for copying 128 MB of data on a single processor using the fcp (fast copy) method."
      }
     

   • alert_threshold: Set the threshold for alerting. If the benchmark result exceeds this threshold, an alert will be triggered. Note that the threshold should usually be greater than 100%.

   • alert_tool: Choose the validation tool to use. The available options are customBiggerIsBetter and customSmallerIsBetter. Refer to this for more details. If using customBiggerIsBetter, the alert will be triggered when prev.value / current.value exceeds the threshold. If using customSmallerIsBetter, the alert will be triggered when current.value / prev.value exceeds the threshold.

   • search_pattern: Define a regular expression to extract benchmark results from the output using awk. This regular expression is designed to match specific patterns in the output, effectively isolating the benchmark results and producing a set of fragments.

   • result_index: Specify the index of the result in the extracted output. This field is aligned with awk's action.

   • description: Provide a brief description of the benchmark.

   For example, if the benchmark output is "Syscall average latency: 1000 ns", the search_pattern is "Syscall average latency:", and the result_index is "4". awk will extract "1000" as the benchmark result. See the awk manual for more information.

   • result_template.json:

     [
         {
             "name": "Average Syscall Latency on Linux",
             "unit": "ns",
             "value": 0,
             "extra": "linux_avg"
         },
         {
             "name": "Average Syscall Latency on Asterinas",
             "unit": "ns",
             "value": 0,
             "extra": "aster_avg"
         }
     ]
     

     • Adjust name and unit according to your benchmark specifics.

   • run.sh:

     #!/bin/bash
     
     /benchmark/bin/getpid
     

     • This script runs the benchmark. Ensure the path to the benchmark binary is correct. asterinas/test/Makefile handles the benchmark binaries.

Step 2: Update the asterinas/.github/benchmarks.yml File

1. Edit the Benchmarks Configuration:
   • Open asterinas/.github/benchmarks.yml.
   • Add your benchmark to the strategy.matrix.benchmark list:

     strategy:
       matrix:
         benchmark: [getpid]
       fail-fast: false
     

Step 3: Test the Benchmark Locally

1. Run the Benchmark:
   • Execute the following command to test the benchmark locally:

     cd asterinas
     bash test/benchmark/bench_linux_aster.sh getpid
     

   • Ensure the benchmark runs successfully and check the results in asterinas/result_getpid.json.

Additional Considerations

• Validation: After adding and testing the benchmark, ensure that the CI pipeline correctly integrates the new benchmark by triggering a CI build.
• Documentation: Update any relevant documentation to include the new benchmark, explaining its purpose and how to interpret its results.

By following these steps, you will successfully integrate a new benchmark into the Asternias CI system, enhancing its capability to evaluate platform performance.