00 /What You Will Learn
- Say why an average response time can look healthy while real customers suffer, and read p50, p95 and p99.
- Explain why p99 matters more as a page makes more calls (1 - 0.99 to the power of N).
- Find the saturation knee: the point where more users stop adding throughput and only add waiting.
- Use Little's Law (users in the system = arrivals per second x time each spends) with real numbers.
- Tell a closed-loop test from an open-loop test, and say what coordinated omission is.
- Run a k6 test and read its result.
Before you start
Finish Lab 01 (Docker Compose Basics) and Lab 03 (HTTP, Caching and DNS) first. You should know what a request, a response and a status code are. You do not need to know k6 or statistics. Everything is explained when it first appears.
01 /Why It Matters
Imagine the checkout service of a big sale, Flipkart on Big Billion Days. The dashboard shows a green number: average response time 29.9 ms. The team relaxes. Meanwhile customer support is flooded: the checkout spinner froze for two seconds, payments timed out, orders failed.
How can the average be 29.9 ms while people wait two seconds? Out of 1000 requests, 990 took 10 ms and 10 took 2000 ms. The average is (990 x 10 + 10 x 2000) / 1000 = 29.9 ms. The average buried the 10 people who had a terrible time. One customer in a hundred is a lot of customers on a big day.
Averages hide the slow ones. A percentile says "this fraction of requests were faster than this number". p99 means 99 out of 100 requests were faster. It shows the experience of your unluckiest 1 percent.
Why p99 matters more than it sounds
A modern page does not make one call, it makes many: prices, stock, offers, delivery date. If each call has a 1 percent chance of being slow, a page that makes 50 calls meets at least one slow call with probability 1 - 0.99 to the power of 50, which is about 39.5 percent. With 100 calls it is 63.4 percent. At scale, your p99 becomes your customers' everyday experience.
02 /What We Are Building
A small web service with a fixed pool of 10 worker threads. Each request takes about 15 ms of work. In 2 percent of requests something slow happens (a pause, a lock) and the request takes 200 ms more. Two load generators send traffic to it: a Python script, and k6.
When more than 10 requests arrive at once, the extra ones wait in line. That waiting is what grows when the service is overloaded.
Maximum throughput is the number of workers divided by the average time one request needs: 10 / 0.0187 s is about 530 requests per second. (The average time is 0.98 x 15 ms + 0.02 x 215 ms = 18.7 ms.)
03 /Words You Need
| Word | Plain meaning |
|---|---|
| Latency | How long one request takes, from sending to getting the answer. |
| Throughput | How many requests the service finishes per second. |
| Average (mean) | Add all the times, divide by the count. Pulled around by slow outliers. |
| p50 (median) | Half the requests were faster than this. The typical request. |
| p95, p99 | 95 or 99 out of 100 requests were faster than this. The slow tail. |
| Virtual user (VU) | One simulated customer in the load test. |
| Worker pool | A fixed number of threads that do the work. Extra requests wait for a free worker. |
| Saturation knee | The load where the pool is full. Past it, throughput stays flat and waiting time shoots up. |
| Little's Law | Users in the system = arrival rate x time each one spends there. |
| Closed loop | Each virtual user waits for the answer before sending the next request. |
| Open loop | New requests arrive at a fixed rate whether or not the server keeps up. Like real customers. |
| Queue | A line. Requests that cannot start yet wait in it, first come first served. |
| Lock | Only one at a time may use a thing, and the others wait. A slow request may be a request waiting for a lock. |
| Coordinated omission | A closed-loop test slows down when the server slows down, so it stops sending the very requests that would have shown the problem. |
04 /Predict Before You Run
Question 1: A service has 10 workers and each request takes 19 ms on average. What is the most it can do per second?
About 10 / 0.019 = 530 requests per second. Ten workers can each finish about 53 requests a second. No amount of extra users changes that. Extra users only wait.
Question 2: You go from 8 users to 32 users on that service. What happens to throughput and to the median latency?
Throughput stays about the same, because the pool is already full. Median latency goes up by roughly three to four times, because 22 of the 32 users are standing in a queue at any moment. More users does not mean more work done.
Question 3: A test uses 32 users who each wait for their answer. The server gets slow. What does the test do?
It sends fewer requests, because users are still waiting. The test politely backs off exactly when the server is in trouble, so the overload looks mild in the results. Real customers do not wait for each other. This is coordinated omission.
05 /Step by Step
Step 0: Get the lab
Download the zip with the button at the top of this page. Unzip it into a new folder and open a terminal inside that folder. Every command below runs from there.
mkdir latency-load-testing-k6 && cd latency-load-testing-k6
unzip ~/Downloads/latency-load-testing-k6.zipmkdir latency-load-testing-k6; cd latency-load-testing-k6
Expand-Archive -Path "$HOME/Downloads/latency-load-testing-k6.zip" -DestinationPath .To run everything with checks, use ./test-load.sh or .\test-load.ps1. It takes about a minute. To run the pieces by hand:
Step 1: Start the service
docker compose up -d --build app
curl http://127.0.0.1:18100/healthdocker compose up -d --build app
curl.exe http://127.0.0.1:18100/healthYou should see: {"status": "healthy", "workers": 10}.
Step 2: Averages lie, and the saturation knee (Python)
python3 load_test.pypython load_test.pyThe script runs two experiments with real requests. Experiment 1: 4 users send 1000 requests, with 5 percent of requests hitting a 300 ms slow spot. You should see: the average around 30 to 40 ms, the median under 30 ms, and p99 around 300 ms. The average looks fine. p99 is ten times the median or more. Experiment 2: users go 2, 4, 8, 12, 32. You should see: requests per second climb until the pool is full (about 10 users) and then stay flat, while the median latency keeps rising. The flat part sits below 530 because this Python script is slower than the service. That is fine for the lesson, and k6 in step 3 reaches the real ceiling. At 32 users there is a histogram of how long people waited. Last, Little's Law: requests per second times average time is about 32, the number of users.
Step 3: Closed loop versus open loop (k6)
mkdir -p out
docker compose --profile k6 run --rm k6
python3 check_k6.pymkdir out
docker compose --profile k6 run --rm k6
python check_k6.pyk6 is a load testing tool. The test file below runs two scenarios against the same service:
// Two ways to send the same load to the same service, one after the other.
//
// closed_loop : 32 users. Each user waits for the answer before sending the next request.
// If the server slows down, the users slow down too, so the test quietly sends LESS load.
// open_loop : 900 new requests every second, whatever the server does. Like real customers,
// who do not wait for each other. If the server cannot keep up, the queue grows.
//
// The service can do about 530 requests per second, so 900 per second is more than it can take.
// The thresholds below only exist so k6 reports a separate result for each scenario in the summary.
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
closed_loop: {
executor: 'constant-vus',
vus: 32,
duration: '8s',
exec: 'work',
},
open_loop: {
executor: 'constant-arrival-rate',
rate: 900,
timeUnit: '1s',
duration: '8s',
startTime: '10s',
preAllocatedVUs: 400,
maxVUs: 1500,
exec: 'work',
},
},
thresholds: {
'http_req_duration{scenario:closed_loop}': ['p(95)<60000'],
'http_req_duration{scenario:open_loop}': ['p(95)<60000'],
'http_reqs{scenario:closed_loop}': ['count>=0'],
'http_reqs{scenario:open_loop}': ['count>=0'],
},
};
const BASE_URL = __ENV.TARGET_URL || 'http://app:8080';
export function work() {
const res = http.get(`${BASE_URL}/api/work`);
check(res, { 'status is 200': (r) => r.status === 200 });
}You should see: the closed-loop scenario sends only what the service can take and shows a calm p95. The open-loop scenario keeps sending 900 requests per second, more than the 530 the service can do, so its queue grows and its p95 is many times worse. The service is the same. Only the test changed. The closed loop hid the problem.
06 /What You Should See
The real output of the automated run (test-load), including the table of users against throughput and the k6 comparison.
The numbers above come from one run on one laptop with Docker Desktop. Your machine will show different timings and counters. What must match is the shape of the result, the [OK] lines.
07 /Look Inside
1. Ask the service for its numbers
Run: curl http://127.0.0.1:18100/api/stats
You should see: the total requests so far, how many are in flight right now, and the worker limit of 10. During a test, the in-flight count stays at 10 while more are waiting in the queue.
2. Change how slow the slow requests are
Run: curl "http://127.0.0.1:18100/api/work?tail=1&tail_ms=300"
You should see: "is_tail_spike": true and a processed_time_ms of 300. The query parameters base_ms, tail (the chance, from 0 to 1) and tail_ms let you shape the service. The load tests above use them.
3. Watch the queue form
In one terminal run python3 load_test.py. In another, run curl http://127.0.0.1:18100/api/stats a few times during experiment 2. You should see: active_in_flight jumping to 32 while worker_concurrency_limit stays 10. The difference is the queue.
08 /Break It, Then Fix It
Predict: open app.py, change DEFAULT_WORKERS = 10 to 20, rebuild with docker compose up -d --build app, and run experiment 2 again. Where does the knee move?
Answer: to about 20 users, and the ceiling doubles to about 1000 requests per second (20 / 0.0187). The queue at 32 users is shorter, so the median latency is lower. More workers moves the knee, it does not remove it. Your Python load script may now become the limit, which is its own lesson: the test tool needs headroom too.
Stretch: in k6-test.js change rate: 900 to rate: 300. The open loop is now under the ceiling, so its p95 should be close to the closed loop's. The difference between the two tests only shows up when the service is overloaded.
09 /What Changes in Real Projects
- Write SLOs on percentiles, for example "p99 under 300 ms", never on the average.
- Measure from the outside. Latency the server measures leaves out the time spent in queues before the request is read.
- Use an open-loop test to find capacity. In k6 that is
constant-arrival-rate(orramping-arrival-rate). Use closed loop for questions about a fixed number of users. - Give the load generator its own machine. A test tool on the same laptop competes for CPU, and then you are measuring the tool.
- Warm up first and run long enough. p99 needs many requests to be meaningful. 1000 samples give you only 10 requests above p99.
- Size pools with Little's Law. If you want 2000 requests per second and each takes 50 ms, you need about 100 requests in flight, so at least 100 workers or connections.
Percentiles tell the truth about the slow requests and cost more to collect, store and explain than one average. Open-loop tests are more honest and need a stronger load generator and more care. Mean latency is cheap and comfortable, and it is the number that misleads you on the worst day.
10 /Interview Corner
- Why is p99 more useful than the average? The average is pulled around by a few outliers and hides them. p99 shows what the slowest 1 percent of users experience, which is what causes timeouts, retries and complaints.
- What is the saturation point of a service? The load at which all workers are busy. Beyond it throughput stops growing and requests queue, so latency rises sharply.
- What is coordinated omission? When a load generator waits for slow responses, it sends fewer requests during the slow period, so the slow period is under-represented in the results. Open-loop load avoids it.
Follow-up questions the interviewer will ask
- Your dashboard shows a 40 ms average and users complain. What do you look at next?
- How many workers do you need for 3000 requests per second at 80 ms each?
- Why can a load test make a slow service look healthy?
11 /Clean Up
docker compose --profile k6 down -vThe automated script cleans up after itself. Next: Lab 10, Nginx Basics. The front door of a website: serving files, forwarding requests and reading logs.