Improve run-webkit-tests error reporting when ref tests have missing images.

Improve run-webkit-tests error reporting when ref tests have missing images.

BUG=343966
Review-Url: https://codereview.chromium.org/2781733003
Cr-Commit-Position: refs/heads/master@{#459986}
Committed: https://chromium.googlesource.com/chromium/src/+/10db17210b3b675ba92c3d4d0a25baa7687546f1

[skobes, 2017-03-27 21:28:20]

The CQ bit was checked by skobes@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-27 21:28:52]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[skobes, 2017-03-27 21:31:32]

skobes@chromium.org changed reviewers:
+ qyearsley@chromium.org

[commit-bot: I haz the power, 2017-03-27 22:35:01]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-27 22:35:02]

Dry run: This issue passed the CQ dry run.

[qyearsley, 2017-03-27 23:15:09]

The overall effect of this is that if there's an -expected.html but the test
doesn't have image results (because dumpAsText() is called), then on the console
we're expecting the messages:
  reference test didn't generate pixel results
  -expected.html didn't generate pixel results
Is that right?

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py
(right):

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py:93:
self.assertEqual(written_files, {'/tmp/exists-expected.html': 'yup'})
It's not very obvious how to interpret these tests; what does it mean when a
file is written to /tmp? Does that just mean the reference test failed?

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py
(right):

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:41:
)).intersection(failure_types)
It doesn't make a big difference, but this could also be written a little
differently:

    input_failure_types = {type(f) for f in failure_list}
    reftest_failure_types = {
        FailureReftestMismatch,
        FailureReftestMismatchDidNotOccur,
        FailureReftestNoImageGenerated,
        FailureReftestNoReferenceImageGenerated,
    }
    return bool(input_failure_types & reftest_failure_types)

WDYT?

[skobes, 2017-03-27 23:30:10]

On 2017/03/27 23:15:09, qyearsley wrote:
> The overall effect of this is that if there's an -expected.html but the test
> doesn't have image results (because dumpAsText() is called), then on the
console
> we're expecting the messages:
>   reference test didn't generate pixel results
>   -expected.html didn't generate pixel results
> Is that right?

Right.

[skobes, 2017-03-27 23:30:17]

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py
(right):

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer_unittest.py:93:
self.assertEqual(written_files, {'/tmp/exists-expected.html': 'yup'})
On 2017/03/27 23:15:09, qyearsley wrote:
> It's not very obvious how to interpret these tests; what does it mean when a
> file is written to /tmp? Does that just mean the reference test failed?

I cargo-culted this from test_reference_exists, it's verifying that the
-expected.html file is written even when missing-image failures are passed into
write_test_result.

I suppose ideally we'd check that when only one image is missing, the other is
still written, but that requires some refactoring of run_test that I'm not sure
is worth the trouble.

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py
(right):

https://codereview.chromium.org/2781733003/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:41:
)).intersection(failure_types)
On 2017/03/27 23:15:09, qyearsley wrote:
> It doesn't make a big difference, but this could also be written a little
> differently:
> 
>     input_failure_types = {type(f) for f in failure_list}
>     reftest_failure_types = {
>         FailureReftestMismatch,
>         FailureReftestMismatchDidNotOccur,
>         FailureReftestNoImageGenerated,
>         FailureReftestNoReferenceImageGenerated,
>     }
>     return bool(input_failure_types & reftest_failure_types)
> 
> WDYT?

Done.

[skobes, 2017-03-28 00:00:53]

The CQ bit was checked by skobes@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-28 00:01:13]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[qyearsley, 2017-03-28 00:02:46]

LGTM, thanks for doing this :-)

[commit-bot: I haz the power, 2017-03-28 01:41:08]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-28 01:41:08]

Dry run: This issue passed the CQ dry run.

[skobes, 2017-03-28 02:06:01]

The CQ bit was checked by skobes@chromium.org

[commit-bot: I haz the power, 2017-03-28 02:06:21]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-28 02:12:23]

CQ is committing da patch.

Bot data: {"patchset_id": 20001, "attempt_start_ts": 1490666761577710,
"parent_rev": "2b6e31dfcf2062a3dec7942daf86dc79864c14e8", "commit_rev":
"10db17210b3b675ba92c3d4d0a25baa7687546f1"}

[commit-bot: I haz the power, 2017-03-28 02:13:14]

Description was changed from

==========
Improve run-webkit-tests error reporting when ref tests have missing images.

BUG=343966
==========

to

==========
Improve run-webkit-tests error reporting when ref tests have missing images.

BUG=343966

Review-Url: https://codereview.chromium.org/2781733003
Cr-Commit-Position: refs/heads/master@{#459986}
Committed:
https://chromium.googlesource.com/chromium/src/+/10db17210b3b675ba92c3d4d0a25...
==========

[commit-bot: I haz the power, 2017-03-28 02:13:15]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/chromium/src/+/10db17210b3b675ba92c3d4d0a25...