5 Pre Run Checks for Web Load Testing: Fail Criteria, Sizing, CI/CD

A load test simulates concurrent user traffic against a web application to verify response time, throughput and error rate against defined thresholds, and it should never begin without a written objective and pass or fail criteria. Before writing a single script, define what “acceptable” means for your system in measurable terms. Apache JMeter, Locust and Grafana k6 cover most self-managed scenarios, while managed platforms such as Azure Load Testing suit teams that need less operational overhead.

TL;DR:

• Load tests should be conducted with clear, measurable objectives and fail criteria to ensure reliable evaluation of response times, error rates, and resource usage.
• Using percentile-based thresholds for response times and error percentages helps detect systemic issues that averages may hide, especially during automated CI/CD checks.
• Tools like Apache JMeter, Locust, and Grafana k6 suit different scripting preferences and operational scales, with managed services like Azure Load Testing reducing infrastructure overhead for teams.
• Ensuring a stable test environment involves disabling unnecessary background processes, verifying network configurations, and running baseline tests with minimal virtual users to identify environment-related issues.
• Performance testing is most effective when integrated into recurring development cycles, with results feeding into infrastructure decisions and architecture improvements.

Table of Contents

• What load testing means and how it differs from stress and soak tests
• Choosing tools and frameworks for your test strategy
• Building a reliable and reproducible test environment
• Setting fail criteria and configuring auto-stop rules
• Diagnosing a failed load test
• How a structured delivery process handles load testing
• Why raw concurrency numbers mislead more than they inform
• Where Vicedomini Softworks fits into your performance engineering plan
• Where to go deeper on load testing implementation
• Sources
• FAQ

What load testing means and how it differs from stress and soak tests

The terminology matters because each test type answers a different engineering question, and confusing them produces misleading conclusions about production readiness. A load test measures behaviour under expected, sustained traffic. A stress test pushes volume beyond expected peaks to find the breaking point. A soak test runs a moderate load for an extended period to expose memory leaks, connection pool exhaustion or degradation that only appears after hours of operation. A capacity test determines the maximum throughput a given infrastructure configuration can sustain while still meeting service level objectives.

The metrics that matter across all four are largely the same:

• Tail latency: p95 and p99 response times, since averages mask the slow requests that frustrate real users.
• Error rate: the percentage of failed or timed-out requests under load.
• Throughput: requests per second sustained without degradation.
• Resource saturation: CPU, memory and connection pool usage on the system under test.

Load tests belong in every sprint as a lightweight regression check, stress and soak tests belong in pre-production gates, and capacity tests are typically run once per major architectural change or before a known traffic event.

Choosing tools and frameworks for your test strategy

The right tool depends on your team’s scripting preference, the scale you need to simulate and how tightly the test must integrate with your delivery pipeline.

• Apache JMeter remains the most widely deployed open-source engine, with a mature GUI for test design, broad protocol support (HTTP, JDBC, JMS, SOAP) and a large plugin ecosystem, though it demands careful tuning for large-scale distributed runs.
• Locust uses Python to define user behaviour as code, which suits teams that prefer version-controlled scripting over drag-and-drop test plans and want lightweight, event-driven load generation.
• Grafana k6 is scripted in JavaScript, integrates natively with Grafana dashboards for visualisation, and is built with CI/CD pipelines in mind from the outset.

Managed services such as Azure Load Testing remove the burden of provisioning and maintaining generator infrastructure, which matters when a team lacks the capacity to manage distributed runners. A comparable option on AWS is the Distributed Load Testing on AWS solution, which automates deployment of JMeter, k6 or Locust runners across regions as infrastructure-as-code. Whichever engine is chosen, reporting should feed into Grafana or an equivalent dashboard so that results are visible alongside application metrics rather than sitting in an isolated report file.

Building a reliable and reproducible test environment

Measurement error is the most common reason a load test result gets disputed, and most of it is preventable with a short pre-run checklist.

1. Run generators in CLI or non-GUI mode for any test beyond a handful of virtual users, since the GUI consumes memory and skews throughput measurements.
2. Disable antivirus scanning and unnecessary background services on generator machines, as these processes compete for CPU and distort response-time readings.
3. Verify network paths, firewall rules and RMI ports between JMeter controller and remote hosts before a distributed run, since misconfigured connectivity causes silent sampler failures.
4. Configure the controller to use a simple data writer rather than aggregating results in memory, and monitor network I/O on the controller itself.
5. Set cloud quotas and billing alerts before launching any large-scale run against managed infrastructure.

The JMeter distributed testing manual specifically warns that the controller node collecting results from remote workers can itself become the bottleneck, which produces a false signal that the system under test is failing when the test harness is actually saturated.

Pro Tip: Run a short baseline test with a single virtual user before scaling up: if response times are already inflated at minimal load, the fault lies in the environment, not the application.

Setting fail criteria and configuring auto-stop rules

A load test without explicit fail criteria produces a report, not a verdict. The pattern used across most engines and managed platforms follows the same structure: an aggregate function applied to a metric, compared against a threshold, evaluated over a time window.

Azure Load Testing supports fail criteria built from client metrics such as response time, error percentage and requests per second, combined with auto-stop functionality that halts a run once a threshold is breached within a defined window. Percentile-based thresholds are preferable to averages because an average can hide a subset of requests that time out entirely while the mean still looks acceptable. A request-level criterion catches a single misbehaving endpoint; a test-level criterion catches systemic degradation. Server-side metrics, such as database lock contention, are harder to wire into managed-service fail criteria and usually require a separate observability integration rather than the load testing tool itself.

Diagnosing a failed load test

A failed run needs triage before it needs a fix, since the failure could originate in the script, the generators, the network or the application itself.

• Check sampler statistics first to see whether failures cluster on a single request type, which points to a script or endpoint issue rather than a systemic one.
• Review client-side metrics for signs that generators themselves were resource-constrained, since generator saturation produces the same symptoms as application slowness.
• Download and inspect worker logs and engine health data, a workflow Azure Load Testing documents explicitly for separating script errors from application bottlenecks.
• Replay a small sample of failing requests in debug mode to confirm whether the error is deterministic or load-dependent.
• Compare the run against a retained baseline, since a regression is only meaningful relative to a known-good result.

Exporting results after every run and retaining them as a comparison baseline turns each subsequent test into a regression check rather than an isolated experiment.

How a structured delivery process handles load testing

Load testing produces the most value when it is treated as a recurring gate rather than a one-off event before launch. A disciplined cadence runs lightweight load checks within sprints, fuller stress and soak tests at pre-production gates, and a capacity test ahead of any known traffic event or major architectural change. Each run is paired with observability, so that response-time and error-rate metrics from the test sit alongside the same dashboards used to monitor production, rather than existing as a separate artefact nobody revisits.

A test plan without a retained fail-criteria definition and comparison baseline is not a repeatable process, it is a single measurement with no way to detect regression.

Performance testing can be built into the same delivery pipeline as the application code, with test plans, fail criteria, run logs and an improvement backlog treated as artefacts that persist across engagements, not disposable output from a single sprint.

Why raw concurrency numbers mislead more than they inform

Enterprise teams often chase headline concurrency figures instead of asking whether their p95 latency holds under realistic, repeatable load. A test run weekly in CI catches more regressions than one giant annual load test ever will. Self-managed runners suit teams with steady traffic patterns; managed cloud testing earns its overhead when traffic is unpredictable or infrastructure expertise is scarce.

— Pepe F.

Where Vicedomini Softworks fits into your performance engineering plan

Performance testing only pays off when it is connected to the architecture decisions that caused the bottleneck in the first place, and that connection is where most in-house teams run out of time or specialist depth. Vicedomini Softworks works directly with engineers rather than through account managers, so fail criteria, generator sizing and diagnosis findings move straight into architecture recommendations without a layer of translation in between.

A typical engagement follows a straightforward path:

• An initial assessment reviews current architecture, traffic patterns and existing test coverage before any load test is designed.
• Architecture and technology stack review identifies the components most likely to bottleneck under sustained load.
• Custom software development work addresses the fixes the tests surface, from connection pooling to caching layers.
• Ongoing oversight through a CTO advisory service helps keep performance testing cadence and monitoring aligned with delivery, without adding a permanent in-house role.

Details on architecture review, code audits and testing engagements sit on the services page, and teams looking for ongoing technical leadership can review the CTO subscription options starting with the Initial assessment.

Where to go deeper on load testing implementation

For fail-criteria syntax and auto-stop configuration, the Azure Load Testing documentation and its diagnostic guide are the most complete references available. The AWS Well-Architected framework covers integrating performance testing into delivery pipelines, and readers choosing which KPIs to track alongside a load test may also find this overview of website KPIs useful.

Sources

• Define load test fail criteria - Azure Load Testing | Microsoft Learn
• Apache JMeter Distributed Testing step-by-step

FAQ

What is the difference between load testing and stress testing?

Load testing verifies that a system meets its performance targets under expected traffic, while stress testing deliberately pushes volume past expected peaks to find the point of failure. Both use the same tooling but different traffic profiles and different pass or fail thresholds.

Which load testing tool should I start with?

Apache JMeter suits teams wanting a mature GUI and broad protocol support, Locust fits teams that prefer scripting user behaviour in Python, and Grafana k6 works well for JavaScript-based teams already using Grafana for dashboards. Managed options such as Azure Load Testing reduce infrastructure overhead when a team lacks capacity to run distributed generators.

How do I set fail criteria for a load test?

Fail criteria follow a pattern of an aggregate function applied to a metric compared against a threshold, such as a p95 response time above 500 milliseconds or an error percentage above 5%. Percentile-based thresholds are generally preferred over averages because they capture the slow requests an average can hide, a distinction covered in the Azure fail-criteria documentation.

Can load testing affect a live production environment?

Yes, high-volume traffic generation can strain shared infrastructure and may trigger a cloud provider’s anti-abuse defences, so tests against production should follow the provider’s stated testing policies. Setting billing alerts before a large run also avoids unexpected costs from scaled-up generator infrastructure.

How does Vicedomini Softworks support performance testing engagements?

Vicedomini Softworks integrates load testing into its custom software development and architecture review work, connecting fail criteria and diagnosis findings directly to the engineering decisions that caused a bottleneck. Ongoing oversight is available through the CTO Advisory service listed on the CTO subscription page.