TestEngineering/Performance/Talos/Tests: Difference between revisions

Jump to navigation Jump to search

TestEngineering/Performance/Talos/Tests (view source)

Revision as of 10:34, 8 August 2019

2,995 bytes removed ,  8 August 2019
Fix internal wiki links
No edit summary
(Fix internal wiki links)
Line 24: Line 24:
=== Startup Tests ===
=== Startup Tests ===
[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test Startup tests] launch Firefox and measure the time to the onload or paint events. We run this in a series of cycles (default to 20) to generate a full set of data.  Tests that currently are startup tests are:
[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test Startup tests] launch Firefox and measure the time to the onload or paint events. We run this in a series of cycles (default to 20) to generate a full set of data.  Tests that currently are startup tests are:
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#ts_paint ts_paint]
* [[#ts_paint|ts_paint]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#tpaint tpaint]
* [[#tpaint|tpaint]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#tresize tresize]
* [[#tresize|tresize]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#sessionrestore.2Fsessionrestore_no_auto_restore.2Fsessionrestore_many_windows sessionrestore[_no_auto_restore/_many_windows]]
* [[#sessionrestore.2Fsessionrestore_no_auto_restore.2Fsessionrestore_many_windows|sessionrestore[_no_auto_restore/_many_windows]]]


=== Page Load Tests ===
=== Page Load Tests ===
Line 84: Line 84:


Tests which are imported benchmarks are:
Tests which are imported benchmarks are:
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#ares6 ARES-6]
* [[#ares6|ARES-6]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#JSS.2FDomaeo_Tests Dromaeo]
* [[#JSS.2FDomaeo_Tests|Dromaeo]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#jetstream JetStream]
* [[#jetstream|JetStream]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#kraken Kraken]
* [[#kraken|Kraken]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#motionmark MotionMark]
* [[#motionmark|MotionMark]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#speedometer speedometer]
* [[#speedometer|speedometer]]
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#stylebench stylebench]
* [[#stylebench|stylebench]]


=== Row Major vs. Column Major ===
=== Row Major vs. Column Major ===
Line 120: Line 120:
* data: we load 2 pages 25 times each, collect 2 sets of 25 data points
* data: we load 2 pages 25 times each, collect 2 sets of 25 data points
* summarization
* summarization
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 2 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 2 subtest results.


* reporting: test time in ms (lower is better)
* reporting: test time in ms (lower is better)
Line 152: Line 152:
* data: We load 5 urls 1 time each, and repeat for 25 cycles; collecting 25 sets of 5 data points
* data: We load 5 urls 1 time each, and repeat for 25 cycles; collecting 25 sets of 5 data points
* summarization
* summarization
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] five data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the rest; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] five data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the rest; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the the subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the the subtest results.


* reporting: test time in ms (lower is better)
* reporting: test time in ms (lower is better)
Line 180: Line 180:
* contact: :jandem
* contact: :jandem
* source: [https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/ARES-6 ARES-6]
* source: [https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/ARES-6 ARES-6]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: 6 cycles of the entire benchmark
* data: 6 cycles of the entire benchmark
** [https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#259 geometric mean] self reported from the benchmark
** [https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#259 geometric mean] self reported from the benchmark
Line 189: Line 189:
* contact: :davidb
* contact: :davidb
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/video video]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/video video]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: 12 cycles of the entire benchmark, each subtest will have 12 data points (see below)
* data: 12 cycles of the entire benchmark, each subtest will have 12 data points (see below)
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 24 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 24 subtest results.
* '''Lower is better'''
* '''Lower is better'''
* unit: ms/frame
* unit: ms/frame
Line 240: Line 240:
* contact: :bholley
* contact: :bholley
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest perf-reftest]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest perf-reftest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Page_Load_Tests PageLoader]
* type: [[#Page_Load_Tests|PageLoader]]
* reporting: intervals in ms (lower is better)
* reporting: intervals in ms (lower is better)
* data: each test loads 25 times
* data: each test loads 25 times
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] 5 data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] 5 data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
** suite: identical to subtest


Line 277: Line 277:
* contact: :bholley
* contact: :bholley
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest-singletons perf-reftest-singletons]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest-singletons perf-reftest-singletons]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Page_Load_Tests PageLoader]
* type: [[#Page_Load_Tests|PageLoader]]
* reporting: intervals in ms (lower is better)
* reporting: intervals in ms (lower is better)
* data: each test loads 25 times
* data: each test loads 25 times
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] 5 data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] 5 data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
** suite: identical to subtest


Line 303: Line 303:
* measuring: Time from opening a new tab (which creates a new content process) to having that new content process be ready to load URLs.
* measuring: Time from opening a new tab (which creates a new content process) to having that new content process be ready to load URLs.
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/cpstartup cpstartup]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/cpstartup cpstartup]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* bug: {{bug|1336389}}
* bug: {{bug|1336389}}
* data: 20 cycles of the entire benchmark
* data: 20 cycles of the entire benchmark
Line 326: Line 326:
* data: there are 36 reported subtests from DAMP which we load 25 times, resulting in 36 sets of 25 data points.
* data: there are 36 reported subtests from DAMP which we load 25 times, resulting in 36 sets of 25 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l356 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l356 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 36 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 36 subtest results.


To run this locally, you'll need to pull down the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5 page set] and run it in a local web server.  See the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#tp5 tp5 section] or contact jmaher for more information.
To run this locally, you'll need to pull down the [[#page_sets|tp5 page set]] and run it in a local web server.  See the [[#tp5|tp5 section]] or contact jmaher for more information.


{| class="mw-collapsible mw-collapsed wikitable"
{| class="mw-collapsible mw-collapsed wikitable"
Line 377: Line 377:
* contact: :mattwoodrow
* contact: :mattwoodrow
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/layout/benchmarks/displaylist_mutate.html displaylist_mutate.html]]
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/layout/benchmarks/displaylist_mutate.html displaylist_mutate.html]]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load the displaylist_mutate.html page five times, measuring pageload each time, generating 5 data points.
* data: we load the displaylist_mutate.html page five times, measuring pageload each time, generating 5 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]


This measures the amount of time it takes to render a page after changing its display list. The page has a large number of display list items (10,000), and mutates one every frame. The goal of the test is to make displaylist construction a bottleneck, rather than painting or other factors, and thus improvements or regressions to displaylist construction will be visible. The test runs in ASAP mode to maximize framerate, and the result is how quickly the test was able to mutate and re-paint 600 items, one during each frame.
This measures the amount of time it takes to render a page after changing its display list. The page has a large number of display list items (10,000), and mutates one every frame. The goal of the test is to make displaylist construction a bottleneck, rather than painting or other factors, and thus improvements or regressions to displaylist construction will be visible. The test runs in ASAP mode to maximize framerate, and the result is how quickly the test was able to mutate and re-paint 600 items, one during each frame.
Line 386: Line 386:
=== Dromaeo Tests ===
=== Dromaeo Tests ===


Dromaeo suite of tests for JavaScript performance testing. See the [https://wiki.mozilla.org/Dromaeo Dromaeo wiki] for more information.  
Dromaeo suite of tests for JavaScript performance testing. See the [[Dromaeo|Dromaeo wiki]] for more information.


This suite is divided into several sub-suites.  
This suite is divided into several sub-suites.  
Line 399: Line 399:
* data: Dromaeo has 6 subtests which run internal benchmarks, each benchmark reports about 180 raw data points each
* data: Dromaeo has 6 subtests which run internal benchmarks, each benchmark reports about 180 raw data points each
summarization:
summarization:
* subtest: Dromaeo is a custom benchmark which has a lot of micro tests inside each subtest, because of this we use a custom [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#dromaeo dromaeo filter] to summarize the subtest.  Each micro test produces 5 data points and for each 5 data points we take the mean, leaving 36 data points to represent the subtest (assuming 180 points).  These 36 micro test means, are then run through a geometric_mean to produce a single number for the dromaeo subtest. [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l527 source: filter.py]
* subtest: Dromaeo is a custom benchmark which has a lot of micro tests inside each subtest, because of this we use a custom [[TestEngineering/Performance/Talos/Data#dromaeo|dromaeo filter]] to summarize the subtest.  Each micro test produces 5 data points and for each 5 data points we take the mean, leaving 36 data points to represent the subtest (assuming 180 points).  These 36 micro test means, are then run through a geometric_mean to produce a single number for the dromaeo subtest. [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l527 source: filter.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 6 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 6 subtest results.


Each page in the manifest is part of the dromaeo css benchmark.  Each page measures the performance of searching the DOM for nodes matching various CSS selectors, using different libraries for the selector implementation (jQuery, Dojo, Mootools, ExtJS, Prototype, and Yahoo UI).
Each page in the manifest is part of the dromaeo css benchmark.  Each page measures the performance of searching the DOM for nodes matching various CSS selectors, using different libraries for the selector implementation (jQuery, Dojo, Mootools, ExtJS, Prototype, and Yahoo UI).
Line 443: Line 443:
* contact: :jgilbert
* contact: :jgilbert
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/terrain glterrain]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/terrain glterrain]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load the perftest.html page (which generates 4 metrics to track) 25 times, resulting in 4 sets of 25 data points
* data: we load the perftest.html page (which generates 4 metrics to track) 25 times, resulting in 4 sets of 25 data points
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l381 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l381 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 4 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 4 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 474: Line 474:
* contact: :jgilbert
* contact: :jgilbert
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/video glvideo]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/video glvideo]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: 5 cycles of the entire benchmark, each subtest will have 5 data points (see below)
* data: 5 cycles of the entire benchmark, each subtest will have 5 data points (see below)
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 4 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 4 subtest results.


* '''Lower is better'''
* '''Lower is better'''
Line 505: Line 505:
* contact: :jandem
* contact: :jandem
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/jetstream/jetstream.manifest jetstream.manifest]] and jetstream.zip from tooltool
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/jetstream/jetstream.manifest jetstream.manifest]] and jetstream.zip from tooltool
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* measuring: JavaScript performance
* measuring: JavaScript performance
* reporting: geometric mean from the benchmark
* reporting: geometric mean from the benchmark
Line 516: Line 516:
* contact: :sdetar
* contact: :sdetar
* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/kraken kraken.manifest]]
* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/kraken kraken.manifest]]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* measuring: JavaScript performance
* measuring: JavaScript performance
* reporting: Total time for all tests, in ms (lower is better)
* reporting: Total time for all tests, in ms (lower is better)
* data: there are 14 subtests in kraken, each subtest is an internal benchmark and generates 10 data points, or 14 sets of 10 data points.
* data: there are 14 subtests in kraken, each subtest is an internal benchmark and generates 10 data points, or 14 sets of 10 data points.
* summarization:
* summarization:
** subtest: For all of the 10 data points, we take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#mean mean] to report a single number.
** subtest: For all of the 10 data points, we take the [[TestEngineering/Performance/Talos/Data#mean|mean]] to report a single number.
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 14 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 14 subtest results.


This is the [[Kraken]] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
This is the [[Kraken]] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
Line 550: Line 550:
* contact: :davidb
* contact: :davidb
* source: [[https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark source]] [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark manifests]]
* source: [[https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark source]] [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark manifests]]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* measuring: benchmark measuring the time to animate complex scenes
* measuring: benchmark measuring the time to animate complex scenes
* summarization:
* summarization:
Line 576: Line 576:
* contact: :rhunt
* contact: :rhunt
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_svg.html rasterflood_svg.html]
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_svg.html rasterflood_svg.html]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load the rasterflood_svg.html page ten times, measuring pageload each time, generating 10 data points.
* data: we load the rasterflood_svg.html page ten times, measuring pageload each time, generating 10 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]


This page animates some complex SVG patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate. The result is how quickly the browser is able to render 600 frames of the animation.
This page animates some complex SVG patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate. The result is how quickly the browser is able to render 600 frames of the animation.
Line 588: Line 588:
* contact: :rhunt
* contact: :rhunt
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_gradient.html rasterflood_gradient.html]
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_gradient.html rasterflood_gradient.html]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load the rasterflood_gradient.html page ten times, computing a score each time, generating 10 data points.
* data: we load the rasterflood_gradient.html page ten times, computing a score each time, generating 10 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]


This page animates some complex gradient patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate.
This page animates some complex gradient patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate.
Line 606: Line 606:
* data: we load the session restore index page 10 times to collect 1 set of 10 data points.
* data: we load the session restore index page 10 times to collect 1 set of 10 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Buildbot/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Buildbot/Talos/Data#median median] of the remaining 9 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 9 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** suite: identical to subtest
** suite: identical to subtest


Line 624: Line 624:
* contact: :selena
* contact: :selena
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/speedometer/speedometer.manifest speedometer.manifest]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/speedometer/speedometer.manifest speedometer.manifest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* measuring: responsiveness of web applications
* measuring: responsiveness of web applications
* reporting: runs/minute score
* reporting: runs/minute score
Line 630: Line 630:
* summarization:
* summarization:
** subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
** subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
** suite: geometric [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#mean mean] of the 16 sums
** suite: geometric [[TestEngineering/Performance/Talos/Data#mean|mean]] of the 16 sums


This is the [http://browserbench.org/Speedometer/ Speedometer] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
This is the [http://browserbench.org/Speedometer/ Speedometer] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
Line 804: Line 804:
* contact: :emilio
* contact: :emilio
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/stylebench/stylebench.manifest stylebench.manifest]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/stylebench/stylebench.manifest stylebench.manifest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* measuring: speed of dynamic style recalculation
* measuring: speed of dynamic style recalculation
* reporting: runs/minute score
* reporting: runs/minute score
Line 846: Line 846:


* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 19 data points
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 19 data points
** suite: geometric_mean(subtests)
** suite: geometric_mean(subtests)


Line 857: Line 857:
* data: there are 30 reported subtests from TART which we load 25 times, resulting in 30 sets of 25 data points.
* data: there are 30 reported subtests from TART which we load 25 times, resulting in 30 sets of 25 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 30 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 30 subtest results.


TART is the '''Tab Animation Regression Test'''.
TART is the '''Tab Animation Regression Test'''.
Line 922: Line 922:
=== tp5 ===
=== tp5 ===
* contact: :jmaher
* contact: :jmaher
* source: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5n.zip]
* source: [[#page_sets|tp5n.zip]]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load each of the 51 tp5o pages 25 times, resulting in 51 sets of 25 data points.
* data: we load each of the 51 tp5o pages 25 times, resulting in 51 sets of 25 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5n.zip].
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l449 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l449 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 51 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 51 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 939: Line 939:
|}
|}


Tests the time it takes Firefox to load the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5 web page test set]. The web set was culled from the Alexa top 500 April 8th, 2011 and consists of 51 pages.  
Tests the time it takes Firefox to load the [[#page_sets|tp5 web page test set]]. The web set was culled from the Alexa top 500 April 8th, 2011 and consists of 51 pages.  


Here are the broad steps we use to create the test set:
Here are the broad steps we use to create the test set:
Line 1,039: Line 1,039:
=== tp5o_scroll ===
=== tp5o_scroll ===
* contact: :kats
* contact: :kats
* source: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5n.zip]
* source: [[#page_sets|tp5n.zip]]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos#Page_Load_Tests PageLoader]
* type: [[TestEngineering/Performance/Talos#Page_Load_Tests|PageLoader]]
* data: we load each of the 51 tp5o pages 12 times, resulting in 51 sets of 12 data points.
* data: we load each of the 51 tp5o pages 12 times, resulting in 51 sets of 12 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5n.zip].
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l470 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l470 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 51 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 51 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 1,125: Line 1,125:
* contact: :rwood, :jmaher
* contact: :rwood, :jmaher
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/quantum_pageload tp6]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/quantum_pageload tp6]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Page_Load_Tests PageLoader]
* type: [[#Page_Load_Tests|PageLoader]]
* reporting: intervals in ms (lower is better)
* reporting: intervals in ms (lower is better)
* data: we load each of the tp6 pages 25 times
* data: we load each of the tp6 pages 25 times
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first five] data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20; [http://searchfox.org/mozilla-central/rev/8a61c71153a79cda2e1ae7d477564347c607cc5f/testing/talos/talos/test.py#246 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first five]] data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20; [http://searchfox.org/mozilla-central/rev/8a61c71153a79cda2e1ae7d477564347c607cc5f/testing/talos/talos/test.py#246 source: test.py]
** suite: identical to subtest; each tp6_* test is reported individually (they're NOT aggregated)
** suite: identical to subtest; each tp6_* test is reported individually (they're NOT aggregated)


Line 1,138: Line 1,138:
The mitmproxy tool is started, the proxy setup in Firefox, and the mitmproxy https security certificate is loaded into Firefox, all automatically. The pagesets are made available for playback via the proxy, at which point each of the tp6 test pages are ready for load and measurement.
The mitmproxy tool is started, the proxy setup in Firefox, and the mitmproxy https security certificate is loaded into Firefox, all automatically. The pagesets are made available for playback via the proxy, at which point each of the tp6 test pages are ready for load and measurement.


Note: For local debugging purposes, you can find the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets page set here]
Note: For local debugging purposes, you can find the [[#page_sets|page set here]]


The tp6 suite currently consists of these individual pageload tests:
The tp6 suite currently consists of these individual pageload tests:
Line 1,220: Line 1,220:
* data: we load the tpaint test window  20 times, resulting in 1 set of 20 data points.
* data: we load the tpaint test window  20 times, resulting in 1 set of 20 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
** suite: identical to subtest


Line 1,251: Line 1,251:
* data: we open a new browser window  20 times, resulting in 1 set of 20 data points.
* data: we open a new browser window  20 times, resulting in 1 set of 20 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Buildbot/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Buildbot/Talos/Data#median median] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
** suite: identical to subtest


Line 1,285: Line 1,285:
* reporting:
* reporting:
* data: we load 50 web pages 5 times each, resulting in 50 sets of 5 data points.
* data: we load 50 web pages 5 times each, resulting in 50 sets of 5 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#page_sets tp5n.zip].
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 4 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 4 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 50 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 50 subtest results.


{| class="mw-collapsible mw-collapsed wikitable"
{| class="mw-collapsible mw-collapsed wikitable"
Line 1,354: Line 1,354:
* data: we run the tresize test page 20 times, resulting in 1 set of 20 data points.
* data: we run the tresize test page 20 times, resulting in 1 set of 20 data points.
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 15 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l205 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 15 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l205 source: test.py]
** suite: same as subtest result
** suite: same as subtest result


Line 1,385: Line 1,385:
* summarization:
* summarization:
** subtest: identical to suite
** subtest: identical to suite
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 19 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l135 source: test.py]
** suite: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 19 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l135 source: test.py]


Starts the browser to display tspaint_test.html with the start time in the url, waits for [[Buildbot/Talos/Tests#Paint_Tests|MozAfterPaint and onLoad]] to fire, then records the end time and calculates the time to startup.
Starts the browser to display tspaint_test.html with the start time in the url, waits for [[Buildbot/Talos/Tests#Paint_Tests|MozAfterPaint and onLoad]] to fire, then records the end time and calculates the time to startup.
Line 1,408: Line 1,408:
* data: we load 6 pages 25 times each, collecting 6 sets of 25 data points
* data: we load 6 pages 25 times each, collecting 6 sets of 25 data points
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] data point, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 6 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 6 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 1,443: Line 1,443:
* data: we load the 2 svg opacity pages 25 times, resulting in 2 sets of 25 data points
* data: we load the 2 svg opacity pages 25 times, resulting in 2 sets of 25 data points
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 2 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 2 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 1,476: Line 1,476:
* data: we load the 5 svg pages 25 times, resulting in 5 sets of 25 data points
* data: we load the 5 svg pages 25 times, resulting in 5 sets of 25 data points
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 5 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 5 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 1,506: Line 1,506:
* data: we load the 7 svg pages 25 times, resulting in 7 sets of 25 data points
* data: we load the 7 svg pages 25 times, resulting in 7 sets of 25 data points
* summarization:
* summarization:
** subtest: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#ignore_first ignore first] '''5''' data points, then take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** subtest: [[TestEngineering/Performance/Talos/Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos/Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]
** suite: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#geometric_mean geometric mean] of the 7 subtest results.
** suite: [[TestEngineering/Performance/Talos/Data#geometric_mean|geometric mean]] of the 7 subtest results.


{| cellspacing="1" cellpadding="1" border="1"
{| cellspacing="1" cellpadding="1" border="1"
Line 1,578: Line 1,578:
* contact: :hsivonen
* contact: :hsivonen
* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]
* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* type: [[#Microbench|Microbench]]
* reporting: intervals in ms (lower is better)
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
* summarization: take the [[TestEngineering/Performance/Talos/Data#median|median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]


Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
Line 1,598: Line 1,598:
* type: Custom GTest micro-benchmarking
* type: Custom GTest micro-benchmarking
* data: Time taken for a GTest function to execute
* data: Time taken for a GTest function to execute
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[https://wiki.mozilla.org/Performance_sheriffing/Talos/Sheriffing#Microbench_Policy Microbench Sheriffing Policy]] for some notes on how to treat regressions.
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[[TestEngineering/Performance/Talos/Sheriffing#Microbench_Policy|Microbench Sheriffing Policy]]] for some notes on how to treat regressions.


=== PerfStrip Tests ===
=== PerfStrip Tests ===
Line 1,621: Line 1,621:
* contact: :bholley, :SimonSapin
* contact: :bholley, :SimonSapin
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Microbench Microbench]
* type: [[#Microbench|Microbench]]
* reporting: intervals in ms (lower is better)
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* data: each test is run and measured 5 times
* summarization: take the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Data#median median] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
* summarization: take the [[TestEngineering/Performance/Talos/Data#median|median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]




Davehunt
Confirmed users
2,196

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1216151"

Navigation menu

OSZAR »