In the course of supporting our customers at QA Cafe, we test a lot of devices in our own lab, and see the results of many devices tested in the field. It’s for this reason that we’ve come to understand stability testing as one of the most important types of testing that you can do for your product. As mentioned in this video and brief article with Matt Langlois, stability testing is long-term testing that mixes performance and functional tests together to uncover issues your devices may encounter when subjected to real use over time. These issues are hard to catch and can have the most impact on the end-user experience.
But what are the key tests you should run, and in what combination, for a good stability/stress test process? Here are our recommendations when performing this kind of testing with CDRouter.
CDRouter contains thousands of test cases, and there is a very large subset of them that are appropriate for long-term stability testing. The tests we are looking for in particular, however, are those that replicate common user behavior that can be repeated. We recommend developers and project teams narrow things down to the following:
The CDRouter Top 100 is a curated test list for any broadband gateway or Wi-Fi router that has a dedicated WAN and LAN, uses NAT, has firewall features, interacts with the DUT's DHCP server on the LAN. Coverage includes tests that exercise:
Most devices of this kind will pass 95 out of the 100 tests with minimal configuration. Beyond its use as a good starting point, however, the top 100 covers protocols that represent a good general set of IPv4 behavior. For devices that support IPv6, there’s an IPv6 top 100 as well, and products that have dual-stack functionality should add both of these test lists to their mix.
The best way uncover stability issues is to exercise repeated behavior in a short time that might normally take a long time to occur in the field. For any broadband gateway or router that operates a DHCP server, this can be accomplished with CDRouter’s stress tests in the dhcp-s.tcl test module.
Tests cdrouter_dhcp_server_800 and cdrouter_dhcp_server_801 are both designed to verify that your device’s DHCP server does not become exhausted after a large number of DHCP releases and restarts. This helps uncover stability issues by simulating many devices connecting and reconnecting to your device under test and requesting an address.
Wi-Fi connectivity is the primary way that most users interact with their gateways and access the Internet. It is, therefore, one of the most common sources of issues in a CPE and the subject of a lot of support and device management solutions. To help uncover performance issues that may arise from regular Wi-Fi use, add CDRouter’s wifi.tcl module to exercise the Wi-Fi functionality of your device.
In particular, test wifi_30 in simulates a large number of Wi-Fi associations from the same Wi-Fi client (or more, when combined with multi-client virtualization - more on that in a moment). The biggest advantage of this test is that it does, in a short amount of time, what might normally take days or weeks to happen in a customer’s network, like the DHCP tests above. It’s one of the most common behaviors to exercise, especially as more and more Wi-Fi devices are added to customers’ networks.
The key to stability testing is to compare performance before and after running functional tests on your device. We’ve seen issues in many devices where performance will degrade drastically over time, so performance (i.e., throughput testing) should be run after your functional test set to compare the results of each test loop.
In CDRouter, the performance expansion adds the ability to drive up to 10-gigabit line-rate traffic through your device under test. This testing is automated when added to your overall test package and should be configured to provide clear pass/fail results based on your target throughput and latency metrics.
For stability testing, the easiest benchmarks are the perf1 through perf4 test cases, which exercise downstream and upstream throughput for both TCP and UDP independently. However, any or all of the tests in the performance expansion can be considered for stability test benchmarks. And as always, be sure to add the IPv6 performance tests if your device supports it.
If your CDRouter system includes Wi-Fi client virtualization, activate 16 additional clients as a good number to start with during your stability testing. Virtualization creates multiple virtual clients that connect to your device under test. Each take turns as the source of test traffic while the others remain connected. This is very valuable for stability testing as it normalizes your results by making sure your test traffic isn’t coming from a single source.
To create a good stability test run, add the above tests to a test package in the following order:
With this package created, use a configuration that activates the 16 virtual clients mentioned above. Then, set the test to loop as long as your test strategy allows, preferably over several days. Use the CDRouter Performance Graphs tool to get a good picture of the results from one test run to the next - you will see interesting results!
As with all testing, it is important to understand the failures that might occur in your top 100 and Wi-Fi tests before you loop them for stability testing. Make sure you have a clean test run of your functional tests (steps 1, 2 and 3) before committing to performance tests. Similarly, if a functional test starts failing after several loops that include performance testing, it may indicate a stability problem.
As we cover in our definitive guide, “How to Build an Automated Test Strategy for Broadband CPE, Wi-Fi, and Smart Home Products,” various types of testing should be applied at different stages in your development process and test cycle. Stability testing is best done when you have the opportunity to run a much longer testing cycle. Better yet, if your system supports parallel test instances, you can run stability tests while still giving your developers the flexibility of running short-cycle feature tests simultaneously!
Want to get started? Download a sample package of the above here. You can also see more about creating test packages in our CDRouter training series with Brad Ritchie. And be sure to reach out to us about adding parallel testing to your system!
