Server Benchmarks

A benchmark suite which, transparently, stress-tests web servers and generates a report in markdown. It measures the requests per second, data transferred and time between requests and responses.

Why YABS (Yet Another Benchmark Suite)

It's true, there already enough of benchmark suites to play around. However, most of them don't even contain real-life test applications to benchmark, therefore the results are not always accurate e.g. a route handler executes SQL queries or reads and sends JSON. This benchmark suite is a fresh start, it can contain any type of tests as the tests are running as self-executables and the measuring is done by a popular and trusted 3rd-party software which acts as a real HTTP Client (one more reason of transparency). Contributions and improvements are always welcomed here.

Use case

Measure the performance of application(s) between different versions or implementations (or web frameworks).

This suite can be further customized, through its tests.yml file, in order to test personal or internal web applications before their public releases.

Installation

The only requirement for the benchmark tool is the Go Programming Language.

$ go get -u github.com/kataras/server-benchmarks
$ go get -u github.com/codesenberg/bombardier

Depending on your test cases you may want to install Node.js and .NET Core too.

How to run

1. Navigate to your tests directory, the one which includes a tests.yml file
2. Open a temrinal and execute: server-benchmarks
3. Wait for the executable server-benchmarks (or server-benchmarks.exe for windows) to finish
4. That's all, now open the the results from the generated RESULTS.md file.

Advanced usage

• Read the tests from the ./tests.dev.yml file
• Wait 3 seconds between tests
• Output the results to the ./dev directory
• Write the results to a remote google spreadsheet table, which you can convert to a graph later on (as shown above).

$ server-benchmarks --wait-run=3s -i ./tests.dev.yml -o ./dev -g-spreadsheet $GoogleSpreadsheetID -g-secret client_secret.json

Run using Docker

The only requirement is Docker.

$ docker run -v ${PWD}:/data kataras/server-benchmarks

Benchmarks

The following generated README contains benchmark results from builtin tests between popular HTTP/2 web frameworks as of 2020.

Note: it's possible that the contents of this file will be updated regularly to accept even more tests cases and frameworks.

System

Last updated: Jul 18, 2020 at 10:46am (UTC)

Terminology

Name is the name of the framework(or router) used under a particular test.

Reqs/sec is the avg number of total requests could be processed per second (the higher the better).

Latency is the amount of time it takes from when a request is made by the client to the time it takes for the response to get back to that client (the smaller the better).

Throughput is the rate of production or the rate at which data are transferred (the higher the better, it depends from response length (body + headers).

Time To Complete is the total time (in seconds) the test completed (the smaller the better).

Results

Test:Static

📖 Fires 1000000 requests, receives a static message as response.

Test:Parameterized

📖 Fires 550000 requests with a dynamic parameter of string, receives a hello text based on the parameter as response.

Test:Sessions

📖 Fires 250000 requests, sets a session and displays its value.

Test:REST

📖 Fires 200000 requests with a dynamic parameter of int, sends JSON as request body and receives JSON as response.

License

This project is licensed under the MIT License.