Simulated RUM testing with PhantomJS 2

Today, many companies are synthetically measuring web performance with various scripts and services. Now that everyone is able to measure those performance numbers and visualize the problem areas, it’s time to raise the bar in terms of scalability, portability and the use of newer DOM APIs.

The following charts show a snapshot of data collected over the period of one year (2012-2013) from the CNN.com home page using Loadreport.js.

Loadreport data from 2012-2013 for CNN.com

I started the Loadreport project while working on the CNN homepage in 2012. It was one of the first PhantomJS scripts that was built solely to collect as many perf numbers from the DOM as possible.
However, the performance numbers collected from events like DOMContentLoaded, onload, and others are all based around old, inaccurate or poorly measured JavaScript APIs provided by the DOM.

These measurements were a great start and gave us some kind of baseline, but as the Web Performance industry has grown over the last several years, more understanding has been gained and better tools and dashboards are being written.
Now, Real User Monitoring (RUM), Synthetic Monitoring, Client Side Performance and many more terms seek to describe this way of capturing web performance and providing some kind of actionable insights around the data.

##Enter Speedgun PhantomJS 2, released this year with a newer version of WebKit, implements the Navigation Timing API specification giving us the ability to measure page performance in an automated fashion. Now we have microsecond precision around a much more robust set of APIs. This is why I rewrote Loadreport.js and named it Speedgun - to utilize the newer APIs and refactor a lot of things that didn’t work quite right… and because the Loadreport name was not super cool. It’s the classic story of rebranding an open source project :) The rest of this article reviews the details of how speedgun.js extracts and stores performance data across many different regions and nodes to achieve a Simulated RUM environment.

##What is Simulated RUM? I don’t know. It’s a frankenstein term where I’m trying to depict a remote control environment that represents real users. This environment should have:

  1. The ability to execute a script (PhantomJS in this case) in a controlled, one at a time manner.
  2. A REST api that allows master/slave communication and is publicly accessible.
  3. A beacon that sends it’s availability to a centralized(parent) server.
  4. Data storage of reports in a db that treats JSON as a first class citizen.

All of these requirements have been built into speedgun.io.

After starting development of speedgun with Docker I realized that this could be easily distributed to any geographic region as a virtualized RUM node. I wanted to execute on the idea that this testing tool could be setup on an old mac mini at grandma’s house with a DSL connection.

Or, with a simple git clone and execution of one command, it can be run on Digital Ocean, AWS, and more environments in every region.

Today, there are 2 available nodes to run a Speedgun report, DigitalOcean SF and NY. Hopefully there will be more in the list by the time you read this ;)

##The Speedgun API There are 2 parts to the speedgun REST API…

1) Creating and retrieving reports

Start a report

Method

http://localhost:8081/rest/performance/go

        <p class="c5 c0"><span class="c8">Parameters:</span></p>

        <p class="c5 c0"><span class="c8">url, cached, email</span></p></td>
</tr>
<tr class="c3">
    <td class="c9" colspan="1" rowspan="1"><p class="c5 c0"><span class="c8a">Example</span></p></td>
    <td class="c9" colspan="1" rowspan="1"><p class="c5 c0"><span>http://localhost:8081/rest/performance/go?cached=false&amp;email=&amp;url=http:%2F%2Fwww.google.com</span></p>

        <p class="c5 c0 c4"><span class="c8"></span></p></td>
</tr> 
<tr class="c3">
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span class="c8a">Response</span></p></td>
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span class="c8">JSON - with uuid and status of where this report is in the queue</span>
    </p></td>
</tr>
<tr class="c13">
    <td class="c15" colspan="2" rowspan="1"><p class="c2"><span class="c17 c14">Retrieve a report</span></p></td>
</tr>
<tr class="c3">
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span class="c8a">Method</span></p></td>
    <td class="c9" colspan="1" rowspan="1"><p class="c5 c0"><span>http://localhost:8081/rest/performance/report</span></p>

        <p class="c5 c0"><span class="c8">Parameters</span></p>

        <p class="c5 c0"><span class="c8">uuid (returned from &ldquo;Start a report&rdquo;)</span></p></td>
</tr>
<tr class="c3">
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span class="c8a">Example</span></p></td>
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span>http://localhost:8081/rest/performance/report?uuid=62990112-48fc-4a95-bf32-2a09acf67dcd</span></p></td>
</tr>
<tr class="c3">
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span class="c8a">Response</span></p></td>
    <td class="c9" colspan="1" rowspan="1"><p class="c2"><span
            class="c8">JSON - All speedgun report data (except screenshots)</span></p></td>
</tr>
</tbody>


2) Beacon service

Start the beacon

Method

http://localhost:8081/rest/beacon/init

Response

text/plain Notification message

Retrieve list of available RUM nodes

Method

http://speedgun.io/rest/beacon/getlist

Response

JSON - This gives us the JSON data for displaying available clients on the UI.



###About the beacon service By default this is turned off. To start sending your beacon with system information to the central speedgun.io server, you only have to curl or load this url in your browser http://localhost:8081/rest/beacon/init.

The beacon sends the following data to the speedgun server every 60 seconds.

The only caveat to this working correctly is opening the required port on the firewall of the client machine’s network. You don’t have to open the port to send the beacon, only to allow speedgun reports to be ran by the centralized server.

Here’s a netgear router admin UI example:

After you open up port 8081 on your firewall, speedgun reports can be remotely ran from anywhere. Currently the speedgun.io URL is hardcoded as the centralized server, this will soon be broken out to a configuration file.
##Why? Why on earth would I want to set this up or standup a speedgun node? A few good reasons…. First, having a consistent and stable environment to run browser performance testing is a must. Second, having the ability to run multiple tests from many different geographic locations with varying connections speeds is hard to simulate. Speedgun gives us both of these.
##Futures The issues list will give you an idea of where the project is heading. If you have any ideas or input feel free to leave them there.
##Contributors It was also an honor to have a few performance gurus from the community helping out with this. Thanks goes to: Peter Hedgenskog who created sitespeed.io and speaks at a ton of conferences about web performance. And Jarrod Overson author of Plato and the new O’Reilly Web Components book!
##Other Page Testing Services I also want to give a quick shout out to all the other page testing services.