How We Developed the Best Broadband Performance Testing Solution for FCC Compliance

Last year, U.S. federal funding for broadband deployment projects reached more than $4.5 billion. These programs are terrific opportunities for broadband service providers (BSPs) to help their communities get connected with the high-speed internet service subscribers want and need today.  

Securing such funding requires compliance with rigorous speed and latency performance standards set out by the Federal Communications Commission (FCC). Applicants must successfully complete this testing or risk losing up to 25 percent of their monthly funding from the program.  

If you’re eligible to receive funds from Connect America Fund (CAF) high-cost universal service support, including price cap carriers, rate-of-return carriers, rural broadband experiment (RBE) support recipients, Alaska Plan carriers, and CAF Phase II auction winners—you probably know testing done directly from the gateway in the subscriber’s home eliminates complexity. But how do you determine the right testing methodology? Let’s look closely at the pros and cons of the available options. 

Option #1: TR-069, TR-143, and an Enhanced ACS To Cover the TR-143 Gaps 

TR-069 

Originally published in 2004 by the Broadband Forum, the TR-069 specification simplifies device management. However, it’s not ideal for bulk data collection. According to a technical report released by the Broadband Forum: “CWMP (TR-069) is an undesirable protocol for the collection of this data as it is not efficient or flexible enough to meet the Service Provider's needs.” 

If you’re not sure about the amount of data collected for FCC testing, consider these points.   

• The FCC requires one latency test per minute for six hours, for seven consecutive days, for up to 50 CPE per funded service Tier, per funded state. That equals 126,000 latency tests. 
• The FCC requires one speed test per hour for six hours, for seven consecutive days, for up to 50 CPE per funded service Tier, per funded state. Also consider these caveats:  
  • The download speed test can be deferred if there is greater than 64kbps (crosstalk) on the WAN connection; the test must attempt one minute later, and a time-stamped log for the deferred minutes for crosstalk is required. That means it’s possible to have 252,000 rows of timestamps for speed testing. 

Takeaway: All this means that one week of tests can be up to 378,000 rows of test attempts/results. That is, without a doubt, bulk data collection—and the Broadband Forum itself states TR-069 is not the right protocol for bulk data collection. 

TR-143 

TR-143 is a specification created in 2008, 10 years before CAF or ACAM testing came into the picture. It defines the CPE data model objects for network service providers to initiate performance throughput tests and monitor data on the IP interface of a CPE using the diagnostic mechanism defined in TR-069.   

TR-143 has some mechanisms that could possibly be cobbled together to meet the FCC requirements.  Those items include IP ping diagnostics, download diagnostics and upload diagnostics. Three sets of TR-098 and TR-181 objects can be used to initiate a latency test and speed tests. 

Unfortunately, those diagnostics objects have four major gaps to meet the FCC testing requirements. 

1. No scheduling mechanism Each of the three objects for testing are initiated on demand and do not support scheduling in any manner. There is no way to program the CPE to do a test at a particular time—say, a 6 p.m. test, or every minute (for latency) or every hour (for speed) for six straight hours. TR-069 requires the ACS to ask the CPE to initiate a test every single time it is required. 
2. No ability to measure crosstalk pretest As part of the FCC requirements, tests can be deferred for one minute if crosstalk is greater than 64 kbps (upload speed test is 32 kbps). Unfortunately, there is no way to ask the CPE what its current WAN speed is, nor does TR-143 define a mechanism. However, TR-069 can get parameter values and some timestamps and make educated decisions so a mechanism can be created in the ACS to cover this TR-143 gap. 
3. No latency results timestamp As previously mentioned, every test attempt must have an associated timestamp. Download and upload diagnostics both have timestamps to tell the ACS exactly when the test was started and completed. IP ping diagnostics do not have a parameter and value with a timestamp to tell the ACS when the test was executed. So, the ACS must assume the test was executed when the ACS requested it. 
4. No available network of test servers TR-143 requires an Apache server for the CPE to “ping”, to download a file from and to upload a self-generated file. Also note that generally the download and the upload from the CPE are not authenticated. This means the Apache server must support HTTP with no authentication for HTTP “gets” and “puts.” Unfortunately, there isn’t a network of servers across the US that meet those requirements to allow a service provider to choose from hundreds of servers to meet the FCC testing requirements. 

Takeaway: TR-143 has too many gaps, simply cannot scale, and was not designed for the heavy-duty requirements of FCC performance testing. TR-143 was designed for one off tests, not 60 tests per hour for six hours for seven days. 

An Enhanced ACS To Cover the TR-143 Gaps 

Around the world, most CPE is managed by TR-069 (more than one billion)—it’s without a doubt the industry standard for remote management and provisioning and a standard Calix is proud to use and support. On average, a TR-069-managed CPE talks to its ACS 1.3 times per day. Periodic Inform is commonly set to every 24 hours; a CPE might get a new IP address or reboot, or the ACS might issue a connection request for something it wants to do right now. All the reasons average out to 1.3 times per day, per CPE. 

Takeaway: If you develop an ACS to cover all the shortcomings of TR-143 mentioned above—as Calix has for third- party certified systems—one CPE under FCC performance testing will now have to communicate with the ACS up to 244 times per hour, for six straight hours. That is 1,464 times in six hours, versus 1.3 times in 24 hours, or over a 100,000 percent increase in load for just one CPE. That would be like taking your ACS that was managing 1,000 CPE, and magically adding 1,000,000 CPE to it overnight. Not something we would recommend.  

Option #2: The Calix Solution 

After carefully evaluating the FCC requirements and the TR-069 option, as well as how to scale while overcoming the gaps mentioned above, Calix considered all possible scenarios. Here are the questions we asked:  

1. What are the different reasons a service provider might want to run speed test beyond just meeting FCC requirements?  
2. What about the need for on-demand and real time troubleshooting and support?  
3. What about a subscriber app for the end user?  
4. What do subscribers use for speed tests all the time?  
5. How do we deploy a wide network of speed test server locations to meet all the use cases mentioned above and meet all the geographical locations of the service providers? 

The simplest way to answer these questions was to partner with the industry standard in speed testing. It checks all the boxes. Working with that partner has allowed Calix to embed a client in our premises solutions that was highly optimized to do speed tests at up to 2.5 Gbps. 

 Here’s how we did it:  

• Created a scheduling mechanism directly in the Calix premises systems.  
• Developed the ability to monitor its WAN speed and defer tests based on cross talk.  
• Used Secure Socket protocols to transfer 378,000 rows of results to the cloud (per CPE), eliminating the need to rely on TR-069 (a protocol that we can all agree is not suitable for bulk data collection).  

Takeaway: We satisfy all the different speed testing scenarios our customers have been requesting with one elegant solution and partnership. 

Calix Offers the Simplest, Most Scalable Broadband Performance Testing Solution 

The comprehensive Broadband Performance Testing solution is comprised of Calix Support Cloud (CSC), GigaSpire® BLAST and GigaCenter systems, and the Broadband Performance Testing Service. With this solution, we’re helping Calix customers like Scott Nyman, CEO and general manager of Wittenberg Telephone-Cirrinity. Calix Professional Services saved the company considerable time and effort by leading his team through a series of key steps in a journey that ensured performance testing success, including High Cost Universal Broadband (HUBB) portal readiness, test endpoint setup, test server vetting, pre-testing, actionable reporting, and analysis. The detailed customized reporting and analysis enabled proactive problem resolution.  

Nyman’s team and Calix leveraged this reporting and the broadband framework to achieve some impressive results. They were able to:   

• Uncover provisioning issues, along with test server problems that had been dragging down test results.   
• Cut through the noise from hundreds of thousands of test results—enabling the team to identify and correct poor performing test endpoints and test servers which improved results by almost 20 percent on their 4Mbps service tests and eight percent on their 25Mbps service tests.  
• Gain additional visibility into broadband QoS delivered to the various regions of his network—highlighting areas where his team can improve the great subscriber experience they’re already getting with managed Wi-Fi.  

For the service providers that just could not swap out the deployed systems fast enough to meet the testing deadlines, Calix also supports FCC performance testing with third-party systems and TR-143. However, Calix has some advantages that allow us to support a 100,000 percent load increase. We are fully cloud, SaaS-enabled and have developed an architecture that allows us to scale to meet those demands.  

Since Calix first released the solution in Q1 of 2019, we have had two years of live deployments and in fact were in pretesting with 30 percent of the funding eligible service providers in 2020. Additionally, a full year of real-world testing and FCC submissions, plus a prior year of generally available tests with service providers to make sure that load increase was not going to be a problem. 

As of today, Calix has conducted more than 50 million performance tests, per FCC requirements, 30 million in the last 5 months. Schedule a consultation to learn more about our FCC Performance Testing Solution.