Articles

Big data analytics using TR-069 or TR-369 (USP) HTTP bulk data collection

6 min read

Big data analytics and statistics monitoring are critical in almost any industry today. Broadband service providers and MSOs can make use of mass data to monitor and anticipate network problems, optimize Wi-Fi, and provide input to machine learning/AI systems.

For CPE and agents managed with TR-069 or USP/TR-369, the Device:2 data model (also known as TR-181) provides standardized parameters to map this information. However, TR-069 wasn’t designed for the frequent, periodic transmission of large amounts of data. To assist this and enable these use cases for service providers, TR-069 Amendment 6 defines an “HTTP bulk data collection mechanism” for collecting mass telemetry.

What is HTTP bulk data collection?

The HTTP bulk data collection mechanism is an extended feature of CPE and other agents implementing TR-069 Amendment 6 (CWMP) or TR-369 (USP), defined by the Broadband Forum. It provides a means by which an Auto-Configuration Server (ACS), or USP Controller, can configure an agent to periodically send a JSON or CSV formatted set of Device:2 (TR-181) information to an HTTP server running a data collection application.

The TR-069/TR-369 Bulk Data Collection Mechanism

This creates a channel outside of the management channel that can be dedicated to collecting mass telemetry. This is powerful, as it allows providers:

  • a fast, lightweight way to collect data
  • the ability to use separate application servers to collect data (taking the load off of the traditional ACS)
  • the option to partner with application providers that offer security, optimization, and other data analytics services.

These are great use cases that are becoming more and more prevalent as connected home and enterprise networks become a focal-point for advanced connectivity and services.

Use of the BulkData object in Device:2 (TR-181)

The Device.BulkData. object contains the objects and parameters used to configure the bulk data collection mechanism. The main object contains the Enable parameter, which enables or disables all bulk data collection, plus a set of parameters that let the agent specify the protocols, encoding types (i.e., JSON or CSV), and the minimum reporting interval it supports. 

The ‘Device.BulkData.Profile.{i}’ table lets the ACS or Controller create new sets of data that it wants the client to send to the collector, and an associated reporting interval. Within that table is the .Parameters.{i}. table, which lets an ACS or Controller add parameters to the data set.

The bulk data collection workflow usually follows these steps:

  1. First, an ACS or Controller creates a new instance of the Device.BulkData.Profile.{i} object using the AddObject RPC (in TR-069) or the Add message (in USP).
  2. The ACS or Controller configures the profile’s settings using SetParameterValues (in USP this can be done as part of the Add message). For example:
    • Device.BulkData.Profile.{i}.HTTP.URL - the URL of the data collector
    • Device.BulkData.Profile.{i}.HTTP.Method - whether the agent should use an HTTP PUT or POST, based on what it supports
    • Device.BulkData.Profile.{i}.ReportingInterval - the interval, in seconds, between reports to the data collector
    • Device.BulkData.Profile.{i}.TimeReference - the time reference (dateTime) from which the reporting interval should be calculated
    • Device.BulkData.Profile.{i}.EncodingType - whether the agent should use JSON or CSV, based on what it support. 
  3.  The agent returns a successful response and indicates the profile’s instance identifier.
  4. The ACS or Controller creates a set of instances of the Device.BulkData.Profile.{i}.Parameter.{i}. object, for each of the parameters it wants in the reports, using one or more AddObject RPCs (TR-069) or Add messages (USP).
  5. The ACS or Controller configures the .Parameter.{i}. instances by setting the Name (optional) and Reference parameters using SetParameterValues (and again, in USP this can be done as part of the Add message). The Name parameter is used to specify what the record label in the report; if it is left blank, the value of Reference is used. Like all reference parameters, Reference contains a full path name to the desired parameter (for example, Device.WiFi.Radio.1.Stats.TotalChannelChangeCount).
  6. The ACS or Controller configures the profile using SetParameterValues or the Set message to change the Device.BulkData.Profile.{i}.Enable parameter to true.
  7. The agent begins sending reports to the data collector at the specified interval. If a record fails to be delivered because of an HTTP error, the agent follows a specified retry mechanism - one that is very similar to the timing and back-off algorithm of the CWMP notification retry mechanism.

If the provider wants to change anything about a report, it can use the SetParameterValues RPC (TR-069) or Set message (USP) to change an instance, or delete instances using the DeleteObject RPC (TR-069) or Delete message (USP).

CDRouter Example

The CDRouter TR-069 add-on has a test module specific to testing bulk data collection. In our testing, we use simple parameters that we know will be on every device - in the first test, we set a single parameter for reporting, using Device.ManagementServer.ConnectionRequestURL. Take a look at this packet capture of the test, hosted on CloudShark Personal SaaS.

 

 

In our simulated environment, the endpoint at 7.1.1.2 is our device under test. 6.0.0.1 is the address of CDRouter’s simulated ACS. 3.3.3.3 is the address of our bulk data collector. You can see a majority of the traffic going between the DUT and the ACS. This is the ACS setting up the BulkData profile and configuring the parameter, plus the Inform RPC messages between the DUT and ACS.

If we follow the HTTP streams, you can see the steps we outlined above play out.

  1. The ACS adds a Profile object.
  2. The ACS configures the Profile object.
  3. The ACS adds a parameter to the profile.
  4. The ACS configures the parameter reference.
  5. The ACS enables the Profile object.
  6. The DUT sends the report to the bulk data collector at 3.3.3.3

 

In this case, the DUT sends the following CSV formatted data to the collector:

ReportTimestamp,ParameterName,ParameterValue,ParameterType 1558370940,Device.ManagementServer.ConnectionRequestURL,http://7.1.1.2/,string

It will then send the data again after another 60 seconds have passed.

Where can I find more?

That’s a quick summary of how to implement basic bulk data collection functionality between an ACS/Controller, agent, and a bulk data collector. The entire bulk data collection mechanism is specified in Annex N of TR-069 Amendment 6. You can also find the TR-369/USP version of the mechanism specified on the usp.technology website.

Developers looking to use the bulk data collection mechanism should test their implementations thoroughly! There’s a lot of permutations using different encodings, timings, numbers of parameters, and whether or not the data is being sent securely over HTTPS. The CDRouter TR-069 add-on has a suite of tests for this feature. Request a demo if you don’t have it already!