Download this whitepaper (pdf)
There’s no question that broadband is the premier service delivery platform of the 21st century. With the explosion of new devices now aware of the network, and new services looking to exploit them, the devices you produce or deploy are becoming more complex than ever before. This puts pressure on everyone in the supply chain, from chipset vendor to service provider, to ensure their devices are tested well at every stage of the testing lifecycle.
The Testing Lifecycle
We break this term, testing lifecycle, into four parts. Each part may have a slightly different meaning for vendors as opposed to service providers rolling out a new product or service.
New Product Testing/Vendor Qualification
For vendors, the testing lifecycle starts with beta testing of a new product. It may involve a device with newer hardware and more advanced features than the last model, or a product that branches into new territory for the vendor. It may come from customer demand, such a service provider planning a new or innovating service offering. For vendors, at this stage, it may rest entirely in the hands of the system-on-a-chip vendor.
Editor’s note: For incumbent vendors of broadband home devices, examples of the latter that we’ve seen include smart home hubs or cloud managed Wifi systems.
For service providers, cable providers, and other MSOs, this stage usually begins with a plan to upgrade their existing device deployment in order to roll out a new service offering. Requests for Proposals (RFPs) are put out and vendors come forward with proposals for the equipment to be used in the deployment. Providers will often perform testing in their own labs, ask for certification testing to be done, or ask for reports from their vendors before choosing a device or system to deploy.
Editor’s note: Smart Wifi is one of the hottest topics providers are pushing for today - whole-home mesh systems that can be managed to provide the end user with quality Wifi throughout their premises.
Testing at this stage
In this initial stage, the most important testing involves creating a baseline of successful functionality. This includes testing for:
- Conformance of connectivity protocols to the official standards - a device should be at least compliant with the minimum expectations outlined in most protocols, such as those that will be required for WAN connectivity, or for management protocol behavior, such as ensuring TR-069 commands are processed and real changes are made to the device configuration. Often, vendors will rightly use open source protocol stacks, but even these need to be vetted for basic bugs in implementation.
- Feature testing - Does the DUT support any advanced features or ALGs? This is the stage to exercise any features that are above baseline routing or other network functionality to prove that certain applications will work as advertised for the end user.
- Product expectations - while it might not be necessary to perform rigorous performance testing until the integration/deployment stage, you want to make sure the new device meets initial expectations, or conforms to its own product specifications. For example, if a device needs to support at least 64 Wifi clients, you should perform scaling testing to ensure it meets that expectation.
Using CDRouter at this stage
Nearly every CDRouter test can be used at this stage, but for broadband gateways, we’ve produced a list we call the “top 100”, designed to verify the basic IPv4 functionality of a typical device with the following features:
- Dedicated LAN and WAN
- NAT
- Firewall
- DHCP server on the LAN
See the CDRouter Top 100
These top 100 cases demonstrate that a device can boot, connect to a remote WAN termination, allow LAN clients to connect via DHCP, allow basic user traffic requiring DNS and HTTP, etc. Most devices of these types For IPv6 devices, a similar set of tests exist that can be easily built onto a CDRouter test package.
For devices that support TR-069, this is the stage to perform conformance testing with CDRouter’s TR-069 add-on and the CDRouter BBF.069 add-on. Testing TR-069 at this stage will make integration/deployment testing with a real ACS or simulated ACS installation go much more quickly.
Lastly, testing at this stage will probably uncover the most bugs and require several revisions of the baseline code for the device. Using an automated platform like CDRouter in conjunction with continuous integration tools will make sure nothing is missed when establishing this baseline level of validity.
Integration/Deployment Testing
After the initial baseline behavior of a device is established and bugs worked out, the next stage is integration/deployment testing. For both providers and vendors, this involves building a simulated network that mirrors the deployment scenario in which the device will be used, including the expected WAN termination, required network services like DNS or the plethora of DOCSIS related services, and simulating real user behavior on the LAN.
Testing at this stage
At this stage, we exercise how a device will behave in the face of real-world test setups and user behavior. This includes testing for:
- Performance - The most popular metrics are those that demonstrate what a device can actually do under pressure. However, basic layer 2 throughput may be good during baseline testing, but isn’t useful when showing how a device will perform in the real world. For this, it’s better to test throughput of different protocols (IPv4 vs. v6, TCP vs. UDP, etc.), as well as application latency (DHCP renews, DNS, etc.). This is especially important for Wifi, which may experience application latency for physical layer reasons that will generate erratic results.
Editor’s note: QA Cafe has an in-depth webinar on the importance of testing higher layer performance and application latency here.
- Robustness - Equally important to how a device performs in a real-world environment is how it performs over time when operating in the environment in which it will be deployed. This means testing user behavior like consistent web requests and other applications, client association/de-association, DHCP renewal, etc. It’s at this stage that problems like memory leaks will be revealed that would not have surfaced when doing baseline testing, causing device performance to suffer in unexpected ways.
- Bootstrapping and ongoing management - This is the stage to test your device’s behavior when either interacting with the actual TR-069 ACS/SNMP NMS/WebPA server that will be used in deployment, or simulating the same bootstrapping process using an ACS/NMS/WebPA simulator. For DOCSIS, this also means testing that the DOCSIS configuration is correctly downloaded, installed, and utilized so that the device is correctly configured. For management protocols, this means checking the device will correctly change configurations in accordance with service profiles.
Using CDRouter at this stage
CDRouter is built to simulate a real broadband network on the WAN and home network on the LAN, replicating a number of WAN types and the behavior of LAN hosts. For DOCSIS deployments, CDRouter can simulate the entire DOCSIS environment, including DNS, ToD, DHCP, TFTP, TR-069, SNMP, WebPA, etc., as well as let you build DOCSIS config files as part of your test configuration to examine if the DUT will correctly download and use them, or switch configurations to measure behavior.
CDRouter’s Performance add-on adds some of the biggest value at this stage, letting you perform application layer throughput over TCP and UDP, and most importantly, repeat that testing in a consistent environment over long periods of time to see how your device holds up to constant use.
With CDRouter’s TR-069 and SNMP “scenario” tests, you can build a set of test cases that replicate the exact bootstrap or configuration procedure that your device will undergo in deployment.
Additionally, you can use CDRouter’s ICS add-on to connect to actual real-world network resources required for your device to operate. This is important for devices that rely on cloud or mobile applications in order to properly bootstrap.
Automated Regression/Firmware Testing
When devices are in deployment, testing still continues as either new issues arise or to eliminate known issues that weren’t critical but part of the initial release. No matter how much simulation we do, there is nothing like a production deployment to illuminate the edge cases and unexpected behavior inherent to any complex software.
Testing at this stage
Testing at this stage focuses on resolving reported issues, building new firmware, and then re-testing baseline to ensure that nothing that changed in the process of updating the firmware broke any older functionality. This includes:
- Replicating issues - Use a proper network simulation that can replicate the scenario in which the issue occurred. It’s important to make sure this is not only consistent with the deployment scenario, but able to be safely repeated from one test to the next.
- Developing a regression test suite - Decide which tests from both baseline testing and integration testing are necessary to feel comfortable that the firmware is stable. Run each new firmware through this entire test suite as changes are made to resolve discovered issues.
- Working with continuous integration - Using a system to track test failures and compare them to each firmware revision is most important at this stage. Continuous integration systems like Jenkins, Bamboo, or other systems built for large development enterprises can be used to show improvement over time and isolate edge cases.
Using CDRouter at this stage
CDRouter shines at this testing phase. Certainly, the thousands of test cases and hundreds of protocols that CDRouter supports are of key value, but the power of its overall automation platform and API integration cannot go overstated.
The key to using CDRouter here is to build test packages that can be run multiple times, overnight, as each new firmware revision is built and pushed to the test environment. When used with continuous integration, tests can be triggered and results collected using the CDRouter API to build a picture of the gains or losses made with each firmware revision.
Adding New Functions/Security Patches
Besides resolving firmware issues that arise over the course of deployment, vendors and providers must also be ready to add new functionality to existing platforms to avoid the expense of a massive hardware upgrade. They must also be cognizant of security vulnerabilities that are discovered by the networking community that would not have been known a-priori or revealed during normal operation.
Testing at this stage
This stage uses concepts from all of the previous stages, as developers must balance testing between testing new functions, resolving vulnerabilities, and making sure that these changes don’t affect functionality elsewhere. As such, the key steps here include the rigorous use of baseline testing, performance, and working with continuous integration, but also includes:
- Building test scripts that match malicious attacks - this stage also requires that developers build or obtain test scripts that are able to replicate the attack as outlined in official (CVE listings) or unofficial vulnerability reports.
- Being proactive - Testing as vulnerabilities are discovered is important, but so is being proactive when it comes to data validation on device API and management interactions, such as those sent as part of CWMP or SNMP commands.
Using CDRouter at this stage
In addition to the benefits outlined in previous stages, the QA Cafe team makes a concerted effort to stay on top of the latest vulnerabilities that are specific to home/broadband network devices. As these vulnerabilities arise, our team makes an attempt to build test cases that are able to replicate and reveal the issue and test that it has been patched.
Conclusions
Testing, and particularly quality assurance testing, is often overlooked at various stages of the development process. Testing at every stage will help ensure robust, best-in-class devices, reduce support calls and maintenance, and limit liability from security exploits. It’s also important to make sure that results are fed back to those responsible for previous stages to avoid the “waterfall effect” common to software development.
CDRouter, as a holistic automation platform, is specifically designed to tackle these problems for developers and providers of broadband and consumer networking devices. Keeping your CDRouter system in service, and using it at each stage of the testing lifecycle, will significantly improve your development, procurement, deployment, and troubleshooting processes.
