Test Methodology

Test Methodology is the page that defines on which setup, with which workload and under which rules each value in the Benchmark section is produced. A performance value is meaningful only together with the condition under which it was measured; every row on the platform pages is based on a definition on this page.

The setup, load tools, metric definitions, measurement rules and value classes are given below.

2

Physical servers

Measured device and load generator; both bare metal, no virtualization

7

Metrics

L7 HTTP, SSL/TLS, Throughput, Software Compression and L4

Measurement Setup

The setup consists of two physical servers: the measured device (DUT, device under test) and the load generator. The two servers are cabled without a managed switch in between; the generator sends client traffic from one leg and simulates Backend Services on the other leg.

Role	Task
Measured device	TR7 hardware platform; load balancing, TLS termination, compression and L4 forwarding
Load generator	A separate server; client load, Backend Service simulation and counter sampling

Connection	Speed	Usage
Client leg	40 GbE	Client traffic; the vService address is on this leg
Backend Service leg	40 GbE	Traffic going to Backend Services
Second Backend Service path	2 × 10 GbE	In a separate PCIe slot; additional path
Management	1 GbE	Separate from measurement traffic; counter reading and control

The connections in the table show the general layout. Each test is set up with a different scenario according to the measured metric: link speeds, cabling, address counts and the load generator configuration vary by test.

The client and Backend Service legs are on separate IP subnets. 48 source addresses are used on the client side; 8 server addresses are on the Backend Service side. The cabling is verified with MAC addresses in every setup.

Frame size

Measurements Were Taken with a 1500-Byte Frame

Jumbo frames were not measured. A jumbo frame reduces the packet count for the same data by about six times; the Throughput values therefore stay on the pessimistic side.

Load Generation and Tools

Tool	Type	Measures	Parameters in the measurement
h1load	Open-source HTTP/1 load generator	HTTP requests/s	24 threads; concurrency sweep 1280 / 1536 / 2048 / 2560 connections (16 s); 3 × 25 s at the peak
dperf	Open-source, DPDK-based L4/L7 load generator and Backend Service simulation	L4 HTTP RPS, L4 Concurrent Connections	Client 8 cores, Backend Service 8 cores (separate physical cores); 48 client addresses, 8 Backend Service addresses
tls-perf	Open-source TLS handshake meter	TLS TPS (ECDSA-P256)	52 threads × 60 concurrent connections, 30 s; the connection is closed after the handshake, no application data is sent
wrk	Open-source HTTP load generator	Throughput, TLS throughput, Compression throughput	Throughput 24-32 threads, 200-400 connections; compression 24 threads, 1000 connections, 35 s
nginx	Open-source web server	Backend Service serving fixed objects in Throughput tests	64 KB, 256 KB and 1 MB objects

Metric Definitions

Metric	What is measured	Workload	Definition of the value
HTTP requests/s	Requests per second over kept-open HTTP/1.1 connections	GET /; TR7 generates the response itself (200 without a body), no Backend Service; ~210 bytes, 2 packets per request on the wire	Total requests / duration; latency is the time to first byte (TTFB)
TLS TPS (ECDSA-P256)	TLS sessions established from scratch per second (full handshake), ECDSA P-256 certificate	Session ticket off, zero bytes of application data	30 s average
Throughput	Full proxy: the device establishes two separate TCP connections, with the client and with the Backend Service	64 KB object	Sum of both legs and both directions; the amount delivered to the client is written separately. On H7140 and H7200, the highest value on the L4 path
TLS throughput	Bulk encryption over an established TLS connection	64 KB object, TLS 1.3 AES-GCM	Bytes sent by the network card
Compression throughput	The device compressing the HTTP response with gzip	64 KB body, by content class	Pre-compression (input) bytes; the ratio is in the content class table on the platform pages
L4 HTTP RPS	HTTP requests forwarded over established persistent connections; no handshake in steady state	1 million persistent connections, 128-byte request	Requests sent per second
L4 Concurrent Connections	Connections kept open at the same time	The connections are opened and kept idle	Number of entries in the connection table of the device

Measurement Rules

1. Fresh boot. The release value is taken on a freshly booted device.
2. Zero errors. A run containing connection, socket or HTTP errors does not produce a value.
3. Repetition. Peaks are taken with at least three independent runs.
4. Both sides are measured. In each run, the processor usage of both the measured device and the generator is sampled in the same time window.
5. Generator rule. If the processor of the generator is above 90%, the value is not counted as the ceiling of the device, and the device capacity is found separately.
6. The card and the controller are limits too. Even if the processor of the generator appears idle, the packet rate of the network card, the PCIe slot or the network controller may be saturated; the card counters are read in every run.
7. The ceiling is found with a target sweep. Coming out clean at a single low target is not proof of the ceiling; the target is raised gradually, and the point at which the value starts to fall behind the target is sought.
8. The build is pinned. Each run is recorded with an immutable build ID.

Value Classes

The values on the platform pages fall into one of the classes below. Values that exceed the limit of the measurement setup are in the capacity class.

Class	Condition	Meaning of the value
Measured	The value is taken directly in the measurement setup	Measured value
Capacity	When the value exceeds the limit of the measurement setup, it is calculated from the measured points	Capacity of the device