Greatly reduce number of level controller tests.

Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725
Committed: https://crrev.com/aff96361d3936e34a270f93ca11d0cf868239517
Cr-Commit-Position: refs/heads/master@{#15329}

[phoglund, 2016-11-30 09:40:35]

phoglund@webrtc.org changed reviewers:
+ peah@webrtc.org

[phoglund, 2016-11-30 09:41:51]

Description was changed from

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

BUG=chromium:666725
==========

to

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725
==========

[peah-webrtc, 2016-11-30 11:33:07]

On 2016/11/30 09:41:51, phoglund wrote:
> Description was changed from
> 
> ==========
> Greatly reduce number of level controller tests.
> 
> These tests have been causing a large number of false alerts, which is
> tons of work for the perf sheriff. It's infeasible to test every single
> permutation, so we must choose carefully. This CL reduces the number
> of RESULT lines from the test from ~450 to ~50. I attempted to choose
> interesting permutations, but you probably know better what's
> interesting...
> 
> BUG=chromium:666725
> ==========
> 
> to
> 
> ==========
> Greatly reduce number of level controller tests.
> 
> These tests have been causing a large number of false alerts, which is
> tons of work for the perf sheriff. It's infeasible to test every single
> permutation, so we must choose carefully. This CL reduces the number
> of RESULT lines from the test from ~450 to ~50. I attempted to choose
> interesting permutations, but you probably know better what's
> interesting...
> 
> https://chromeperf.appspot.com/report
> sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
> also suggests most of the metrics have enormous standard deviations,
> so maybe you could look into how stable the metrics really are
> and remove/stabilize the ones that aren't?
> 
> BUG=chromium:666725
> ==========
Great! Thanks!

lgtm

Reducing the number of tests definitely makes sense. My major concern is,
however, why these tests are suddenly starting to fail. The part of the code
that is running has not been touched since July, and suddenly the tests start
causing false alerts during the last 2 weeks with regressions that are related
to unrelated CLs. My guess was that these started occurring due to either new
trybots or the change in how DCHECKS are treated in these tests and my plan is
to monitor these to see whether they stop occurring and take action after that.
Regardless of that, though, reducing the number of tests definitely makes sense
and this change won't hide any issues with the tests or code regressions as that
code has been stable for a long while.

As soon as I was aware of this happening, I've been closing the levelcontroller
issues as soon as I've seen them and also notified the sheriffs so that there
should be no overhead work for them.

The standard deviation issue was expected to be resolved by
https://codereview.webrtc.org/2518523002/ but I'll need to look into that as
that seems not to yet be the case.

[phoglund, 2016-11-30 12:38:58]

The CQ bit was checked by phoglund@webrtc.org

[commit-bot: I haz the power, 2016-11-30 12:39:02]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...

[commit-bot: I haz the power, 2016-11-30 13:04:40]

CQ is committing da patch.

Bot data: {"patchset_id": 1, "attempt_start_ts": 1480509538277500, "parent_rev":
"905b52ccc394ec51e7b794c509aca22acf37cd55", "commit_rev":
"ca7710dad4c67c0446b4535feb9eba7542594352"}

[commit-bot: I haz the power, 2016-11-30 13:04:41]

Description was changed from

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725
==========

to

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725
==========

[commit-bot: I haz the power, 2016-11-30 13:04:44]

Committed patchset #1 (id:1)

[commit-bot: I haz the power, 2016-11-30 13:04:52]

Description was changed from

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725
==========

to

==========
Greatly reduce number of level controller tests.

These tests have been causing a large number of false alerts, which is
tons of work for the perf sheriff. It's infeasible to test every single
permutation, so we must choose carefully. This CL reduces the number
of RESULT lines from the test from ~450 to ~50. I attempted to choose
interesting permutations, but you probably know better what's
interesting...

https://chromeperf.appspot.com/report?
sid=a7193c96f708018848ca07ad9c78ac657cadab3c70b3939c42bd7d70a092d61a
also suggests most of the metrics have enormous standard deviations,
so maybe you could look into how stable the metrics really are
and remove/stabilize the ones that aren't?

BUG=chromium:666725

Committed: https://crrev.com/aff96361d3936e34a270f93ca11d0cf868239517
Cr-Commit-Position: refs/heads/master@{#15329}
==========

[commit-bot: I haz the power, 2016-11-30 13:04:53]

Patchset 1 (id:??) landed as
https://crrev.com/aff96361d3936e34a270f93ca11d0cf868239517
Cr-Commit-Position: refs/heads/master@{#15329}