Enable webrtc_nonparallel_tests on iOS simulator

Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
only one test needs to be disabled: VirtualSocketServerTest.delay_v4

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios" ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3 out/x64-Debug/webrtc_nonparallel_tests.app

Review-Url: https://codereview.webrtc.org/2909073002
Cr-Commit-Position: refs/heads/master@{#18519}
Committed: https://chromium.googlesource.com/external/webrtc/+/c131bf944e939131a04753bf7c6ec06b56378024

[kjellander_webrtc, 2017-05-29 15:25:21]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

BUG=webrtc:5566
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

BUG=webrtc:5566
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-05-29 15:25:30]

Patchset #1 (id:1) has been deleted

[kjellander_webrtc, 2017-05-29 20:03:58]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

BUG=webrtc:5566
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-05-29 20:03:58]

kjellander@webrtc.org changed reviewers:
+ deadbeef@webrtc.org

[kjellander_webrtc, 2017-05-29 20:05:29]

Patchset #2 (id:40001) has been deleted

[kjellander_webrtc, 2017-05-29 20:09:32]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled all the tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-05-29 20:13:32]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled all the tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled most tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
Some tests that passed on retry was left enabled though.

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-05-29 20:14:03]

Patchset #3 (id:80001) has been deleted

[kjellander_webrtc, 2017-05-29 20:14:17]

Patchset #2 (id:60001) has been deleted

[kjellander_webrtc, 2017-05-29 20:14:55]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled most tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
Some tests that passed on retry was left enabled though.

BUG=webrtc:7727
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled most tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
Some tests that passed on retry was left enabled though.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-05-30 11:58:21]

Taylor, let me know if there's a nicer way to do this. The sockets seems
generally unstable on the simulator somehow.

[Taylor Brandstetter, 2017-05-31 06:54:55]

On 2017/05/30 11:58:21, kjellander_webrtc wrote:
> Taylor, let me know if there's a nicer way to do this. The sockets seems
> generally unstable on the simulator somehow.

When I run these tests, all I see is that they're killed after a certain period
of time (20 seconds):

SpringBoard[2617]: [Watchdog] [org.chromium.gtest.generic-unit-test] Provision
violated for watchdog process-launch: <FBSProcessResourceProvision:
0x6080016e6900; allowance: <; FBSProcessResourceAllowance; type: realTime;
timeValue: 20.0s>; violated: YES>

This sounds like "OS is killing process if it doesn't launch within 20 seconds".
Has there been any update on this issue? Just disabling the longest-running
tests wouldn't be a good long-term solution.

[kjellander_webrtc, 2017-05-31 08:14:10]

On 2017/05/31 06:54:55, Taylor Brandstetter wrote:
> On 2017/05/30 11:58:21, kjellander_webrtc wrote:
> > Taylor, let me know if there's a nicer way to do this. The sockets seems
> > generally unstable on the simulator somehow.
> 
> When I run these tests, all I see is that they're killed after a certain
period
> of time (20 seconds):
> 
> SpringBoard[2617]: [Watchdog] [org.chromium.gtest.generic-unit-test] Provision
> violated for watchdog process-launch: <FBSProcessResourceProvision:
> 0x6080016e6900; allowance: <; FBSProcessResourceAllowance; type: realTime;
> timeValue: 20.0s>; violated: YES>
> 
> This sounds like "OS is killing process if it doesn't launch within 20
seconds".
> Has there been any update on this issue? Just disabling the longest-running
> tests wouldn't be a good long-term solution.

Thanks for digging into the details (which I didn't). Oleh will take a closer
look and try to isolate if there's actually a 20 second per-test timeout and see
if we can possibly change it.

[kjellander_webrtc, 2017-05-31 08:22:32]

+oprypin

[oprypin_webrtc, 2017-06-01 16:46:38]

On 2017/05/31 06:54:55, Taylor Brandstetter wrote:
> On 2017/05/30 11:58:21, kjellander_webrtc wrote:
> > Taylor, let me know if there's a nicer way to do this. The sockets seems
> > generally unstable on the simulator somehow.
> 
> When I run these tests, all I see is that they're killed after a certain
period
> of time (20 seconds):
> 
> SpringBoard[2617]: [Watchdog] [org.chromium.gtest.generic-unit-test] Provision
> violated for watchdog process-launch: <FBSProcessResourceProvision:
> 0x6080016e6900; allowance: <; FBSProcessResourceAllowance; type: realTime;
> timeValue: 20.0s>; violated: YES>
> 
> This sounds like "OS is killing process if it doesn't launch within 20
seconds".
> Has there been any update on this issue? Just disabling the longest-running
> tests wouldn't be a good long-term solution.

Hi Taylor,
This was a very helpful lead, thanks. This is indeed what's happening.
Chromium's test runner on iOS shows a text label on the screen, so it works
fine, while for WebRTC nothing is happening so the OS kills the app as "frozen".
I'll try switching the provider of the main function to the one that Chromium
uses.
https://codereview.webrtc.org/2917903002/

[kjellander_webrtc, 2017-06-09 11:21:57]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

I disabled most tests that showed up in:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType...
Some tests that passed on retry was left enabled though.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os=\"ios\"
ios_enable_code_signing=false is_component_build=false target_cpu=\"x64\"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
this should be good to go.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-06-09 11:22:41]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
this should be good to go.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.2
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
this should be good to go.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-06-09 12:14:59]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
this should be good to go.

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
only one test needs to be disabled: VirtualSocketServerTest.delay_v4

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3
out/x64-Debug/webrtc_nonparallel_tests.app
==========

[kjellander_webrtc, 2017-06-09 12:16:00]

PTAL, I updated this after https://chromium-review.googlesource.com/528173 was
landed and it looks promising!

The VirtualSocketServerTest.delay_v4 test fails like this:
../../webrtc/base/virtualsocket_unittest.cc:739: Failure
Expected: (500u) <= (receiver.samples), actual: 500 vs 284

Can we submit this as it is now?

[Taylor Brandstetter, 2017-06-09 18:39:58]

lgtm. delay_v4 is a test that seems unsafe to run on a simulator in the first
place; it relies on being able to process packets fast enough in real time.

[kjellander_webrtc, 2017-06-09 19:56:40]

On 2017/06/09 18:39:58, Taylor Brandstetter wrote:
> lgtm. delay_v4 is a test that seems unsafe to run on a simulator in the first
> place; it relies on being able to process packets fast enough in real time.

Alright. I updated the comment with that and won't reference the bug (which can
then be closed).
It makes me think the test belongs in webrtc_perf_tests though, not being a
correctness test after all.

[kjellander_webrtc, 2017-06-09 19:56:46]

The CQ bit was checked by kjellander@webrtc.org

[kjellander_webrtc, 2017-06-09 19:56:47]

The patchset sent to the CQ was uploaded after l-g-t-m from deadbeef@webrtc.org
Link to the patchset: https://codereview.webrtc.org/2909073002/#ps180001 (title:
"Updated comment agaiin")

[commit-bot: I haz the power, 2017-06-09 19:56:57]

CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...

[commit-bot: I haz the power, 2017-06-09 19:59:13]

CQ is committing da patch.

Bot data: {"patchset_id": 180001, "attempt_start_ts": 1497038206881060,
"parent_rev": "b82487b19a3a5a35b25d8261b641d2dfaea33a2f", "commit_rev":
"c131bf944e939131a04753bf7c6ec06b56378024"}

[commit-bot: I haz the power, 2017-06-09 19:59:17]

Description was changed from

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
only one test needs to be disabled: VirtualSocketServerTest.delay_v4

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3
out/x64-Debug/webrtc_nonparallel_tests.app
==========

to

==========
Enable webrtc_nonparallel_tests on iOS simulator

After landing https://chromium-review.googlesource.com/528173
only one test needs to be disabled: VirtualSocketServerTest.delay_v4

BUG=webrtc:7727
NOTRY=True
TESTED=gn gen out/x64-Debug --args='target_os="ios"
ios_enable_code_signing=false is_component_build=false target_cpu="x64"'
ninja -C out/x64-Debug webrtc_nonparallel_tests
out/x64-Debug/iossim -d "iPhone 6s" -s 10.3
out/x64-Debug/webrtc_nonparallel_tests.app

Review-Url: https://codereview.webrtc.org/2909073002
Cr-Commit-Position: refs/heads/master@{#18519}
Committed:
https://chromium.googlesource.com/external/webrtc/+/c131bf944e939131a04753bf7...
==========

[commit-bot: I haz the power, 2017-06-09 19:59:18]

Committed patchset #6 (id:180001) as
https://chromium.googlesource.com/external/webrtc/+/c131bf944e939131a04753bf7...

[Taylor Brandstetter, 2017-06-09 22:22:31]

On 2017/06/09 19:56:40, kjellander_webrtc wrote:
> On 2017/06/09 18:39:58, Taylor Brandstetter wrote:
> > lgtm. delay_v4 is a test that seems unsafe to run on a simulator in the
first
> > place; it relies on being able to process packets fast enough in real time.
> 
> Alright. I updated the comment with that and won't reference the bug (which
can
> then be closed).
> It makes me think the test belongs in webrtc_perf_tests though, not being a
> correctness test after all.

I think it should be rewritten to use the fake clock, actually. It's not trying
to test performance per se, it's just testing the "transit delay distribution"
feature of VirtualSocketServer, and sends a bunch of packets over time to do so.
I'll make a Cl since it should be trivial.