|
|
DescriptionOnly store sequence numbers for media stream in FlexFEC end-to-end test.
This should remove the test flakiness, as before this change there
could be collisions from sequence numbers coming from two sequence
number spaces (the media SSRC and the FlexFEC SSRC). The probability
of collisions was low, and hence the flakes were far between.
This change also reduces the packet loss to 5% (down from ~50%), in
order for the BWE to have an easier time to ramp up.
BUG=webrtc:6825
R=philipel@webrtc.org, mflodman@webrtc.org
Committed: https://crrev.com/3536463e7e808747ddc5da2dc0933dbec97ceeeb
Cr-Commit-Position: refs/heads/master@{#15512}
Patch Set 1 #
Total comments: 2
Patch Set 2 : mflodman response 1. #Patch Set 3 : Rebase. #Patch Set 4 : Try another seed that would work better on win-implementations. #Messages
Total messages: 38 (22 generated)
No flakes in ~10000 local runs.
One comment, then LGTM. https://codereview.webrtc.org/2554403003/diff/1/webrtc/video/end_to_end_tests.cc File webrtc/video/end_to_end_tests.cc (right): https://codereview.webrtc.org/2554403003/diff/1/webrtc/video/end_to_end_tests... webrtc/video/end_to_end_tests.cc:763: Random random_; Add #include "webrtc/base/random.h"
lgtm
https://codereview.webrtc.org/2554403003/diff/1/webrtc/video/end_to_end_tests.cc File webrtc/video/end_to_end_tests.cc (right): https://codereview.webrtc.org/2554403003/diff/1/webrtc/video/end_to_end_tests... webrtc/video/end_to_end_tests.cc:763: Random random_; On 2016/12/08 09:52:26, mflodman wrote: > Add #include "webrtc/base/random.h" Done.
The CQ bit was checked by brandtr@webrtc.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by brandtr@webrtc.org
The patchset sent to the CQ was uploaded after l-g-t-m from philipel@webrtc.org, mflodman@webrtc.org Link to the patchset: https://codereview.webrtc.org/2554403003/#ps20001 (title: "mflodman response 1.")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by commit-bot@chromium.org
Failed to apply patch for webrtc/video/end_to_end_tests.cc: While running git apply --index -p1; error: patch failed: webrtc/video/end_to_end_tests.cc:19 error: webrtc/video/end_to_end_tests.cc: patch does not apply Patch: webrtc/video/end_to_end_tests.cc Index: webrtc/video/end_to_end_tests.cc diff --git a/webrtc/video/end_to_end_tests.cc b/webrtc/video/end_to_end_tests.cc index 86c181bdf46fc8cd2eba6d086dc8ac9e694d7504..0b602d9af502480d6afa3ef5f3e7cace079e3777 100644 --- a/webrtc/video/end_to_end_tests.cc +++ b/webrtc/video/end_to_end_tests.cc @@ -19,6 +19,7 @@ #include "webrtc/base/event.h" #include "webrtc/base/file.h" #include "webrtc/base/optional.h" +#include "webrtc/base/random.h" #include "webrtc/base/rate_limiter.h" #include "webrtc/call.h" #include "webrtc/common_video/include/frame_callback.h" @@ -690,7 +691,7 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { public rtc::VideoSinkInterface<VideoFrame> { public: FlexfecRenderObserver() - : EndToEndTest(kDefaultTimeoutMs), state_(kFirstPacket) {} + : EndToEndTest(kDefaultTimeoutMs), random_(0xcafef00d) {} size_t GetNumFlexfecStreams() const override { return 1; } @@ -705,36 +706,30 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { EXPECT_EQ(kFlexfecPayloadType, payload_type); } - auto seq_num_it = protected_sequence_numbers_.find(header.sequenceNumber); - if (seq_num_it != protected_sequence_numbers_.end()) { - // Retransmitted packet, should not count. - protected_sequence_numbers_.erase(seq_num_it); - auto ts_it = protected_timestamps_.find(header.timestamp); - EXPECT_NE(ts_it, protected_timestamps_.end()); - protected_timestamps_.erase(ts_it); - return SEND_PACKET; + // Is this a retransmitted media packet? From the perspective of FEC, this + // packet is then no longer dropped, so remove it from the list of + // dropped packets. + if (payload_type == kFakeVideoSendPayloadType) { + auto seq_num_it = dropped_sequence_numbers_.find(header.sequenceNumber); + if (seq_num_it != dropped_sequence_numbers_.end()) { + dropped_sequence_numbers_.erase(seq_num_it); + auto ts_it = dropped_timestamps_.find(header.timestamp); + EXPECT_NE(ts_it, dropped_timestamps_.end()); + dropped_timestamps_.erase(ts_it); + + return SEND_PACKET; + } } - switch (state_) { - case kFirstPacket: - state_ = kDropEveryOtherPacketUntilFlexfec; - break; - case kDropEveryOtherPacketUntilFlexfec: - if (payload_type == kFlexfecPayloadType) { - state_ = kDropNextMediaPacket; - return SEND_PACKET; - } - if (header.sequenceNumber % 2 == 0) - return DROP_PACKET; - break; - case kDropNextMediaPacket: - if (payload_type == kFakeVideoSendPayloadType) { - protected_sequence_numbers_.insert(header.sequenceNumber); - protected_timestamps_.insert(header.timestamp); - state_ = kDropEveryOtherPacketUntilFlexfec; - return DROP_PACKET; - } - break; + // Simulate 5% packet loss. Record what media packets, and corresponding + // timestamps, that were dropped. + if (random_.Rand(1, 100) <= 5) { + if (payload_type == kFakeVideoSendPayloadType) { + dropped_sequence_numbers_.insert(header.sequenceNumber); + dropped_timestamps_.insert(header.timestamp); + } + + return DROP_PACKET; } return SEND_PACKET; @@ -744,17 +739,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { rtc::CritScope lock(&crit_); // Rendering frame with timestamp of packet that was dropped -> FEC // protection worked. - auto it = protected_timestamps_.find(video_frame.timestamp()); - if (it != protected_timestamps_.end()) + auto it = dropped_timestamps_.find(video_frame.timestamp()); + if (it != dropped_timestamps_.end()) observation_complete_.Set(); } - enum { - kFirstPacket, - kDropEveryOtherPacketUntilFlexfec, - kDropNextMediaPacket, - } state_; - void ModifyVideoConfigs( VideoSendStream::Config* send_config, std::vector<VideoReceiveStream::Config>* receive_configs, @@ -768,10 +757,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { } rtc::CriticalSection crit_; - std::set<uint32_t> protected_sequence_numbers_ GUARDED_BY(crit_); + std::set<uint32_t> dropped_sequence_numbers_ GUARDED_BY(crit_); // Since several packets can have the same timestamp a multiset is used // instead of a set. - std::multiset<uint32_t> protected_timestamps_ GUARDED_BY(crit_); + std::multiset<uint32_t> dropped_timestamps_ GUARDED_BY(crit_); + Random random_; } test; RunBaseTest(&test);
The CQ bit was checked by brandtr@webrtc.org
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by commit-bot@chromium.org
Failed to apply patch for webrtc/video/end_to_end_tests.cc: While running git apply --index -p1; error: patch failed: webrtc/video/end_to_end_tests.cc:19 error: webrtc/video/end_to_end_tests.cc: patch does not apply Patch: webrtc/video/end_to_end_tests.cc Index: webrtc/video/end_to_end_tests.cc diff --git a/webrtc/video/end_to_end_tests.cc b/webrtc/video/end_to_end_tests.cc index 86c181bdf46fc8cd2eba6d086dc8ac9e694d7504..0b602d9af502480d6afa3ef5f3e7cace079e3777 100644 --- a/webrtc/video/end_to_end_tests.cc +++ b/webrtc/video/end_to_end_tests.cc @@ -19,6 +19,7 @@ #include "webrtc/base/event.h" #include "webrtc/base/file.h" #include "webrtc/base/optional.h" +#include "webrtc/base/random.h" #include "webrtc/base/rate_limiter.h" #include "webrtc/call.h" #include "webrtc/common_video/include/frame_callback.h" @@ -690,7 +691,7 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { public rtc::VideoSinkInterface<VideoFrame> { public: FlexfecRenderObserver() - : EndToEndTest(kDefaultTimeoutMs), state_(kFirstPacket) {} + : EndToEndTest(kDefaultTimeoutMs), random_(0xcafef00d) {} size_t GetNumFlexfecStreams() const override { return 1; } @@ -705,36 +706,30 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { EXPECT_EQ(kFlexfecPayloadType, payload_type); } - auto seq_num_it = protected_sequence_numbers_.find(header.sequenceNumber); - if (seq_num_it != protected_sequence_numbers_.end()) { - // Retransmitted packet, should not count. - protected_sequence_numbers_.erase(seq_num_it); - auto ts_it = protected_timestamps_.find(header.timestamp); - EXPECT_NE(ts_it, protected_timestamps_.end()); - protected_timestamps_.erase(ts_it); - return SEND_PACKET; + // Is this a retransmitted media packet? From the perspective of FEC, this + // packet is then no longer dropped, so remove it from the list of + // dropped packets. + if (payload_type == kFakeVideoSendPayloadType) { + auto seq_num_it = dropped_sequence_numbers_.find(header.sequenceNumber); + if (seq_num_it != dropped_sequence_numbers_.end()) { + dropped_sequence_numbers_.erase(seq_num_it); + auto ts_it = dropped_timestamps_.find(header.timestamp); + EXPECT_NE(ts_it, dropped_timestamps_.end()); + dropped_timestamps_.erase(ts_it); + + return SEND_PACKET; + } } - switch (state_) { - case kFirstPacket: - state_ = kDropEveryOtherPacketUntilFlexfec; - break; - case kDropEveryOtherPacketUntilFlexfec: - if (payload_type == kFlexfecPayloadType) { - state_ = kDropNextMediaPacket; - return SEND_PACKET; - } - if (header.sequenceNumber % 2 == 0) - return DROP_PACKET; - break; - case kDropNextMediaPacket: - if (payload_type == kFakeVideoSendPayloadType) { - protected_sequence_numbers_.insert(header.sequenceNumber); - protected_timestamps_.insert(header.timestamp); - state_ = kDropEveryOtherPacketUntilFlexfec; - return DROP_PACKET; - } - break; + // Simulate 5% packet loss. Record what media packets, and corresponding + // timestamps, that were dropped. + if (random_.Rand(1, 100) <= 5) { + if (payload_type == kFakeVideoSendPayloadType) { + dropped_sequence_numbers_.insert(header.sequenceNumber); + dropped_timestamps_.insert(header.timestamp); + } + + return DROP_PACKET; } return SEND_PACKET; @@ -744,17 +739,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { rtc::CritScope lock(&crit_); // Rendering frame with timestamp of packet that was dropped -> FEC // protection worked. - auto it = protected_timestamps_.find(video_frame.timestamp()); - if (it != protected_timestamps_.end()) + auto it = dropped_timestamps_.find(video_frame.timestamp()); + if (it != dropped_timestamps_.end()) observation_complete_.Set(); } - enum { - kFirstPacket, - kDropEveryOtherPacketUntilFlexfec, - kDropNextMediaPacket, - } state_; - void ModifyVideoConfigs( VideoSendStream::Config* send_config, std::vector<VideoReceiveStream::Config>* receive_configs, @@ -768,10 +757,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { } rtc::CriticalSection crit_; - std::set<uint32_t> protected_sequence_numbers_ GUARDED_BY(crit_); + std::set<uint32_t> dropped_sequence_numbers_ GUARDED_BY(crit_); // Since several packets can have the same timestamp a multiset is used // instead of a set. - std::multiset<uint32_t> protected_timestamps_ GUARDED_BY(crit_); + std::multiset<uint32_t> dropped_timestamps_ GUARDED_BY(crit_); + Random random_; } test; RunBaseTest(&test);
The CQ bit was checked by brandtr@webrtc.org
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by commit-bot@chromium.org
Failed to apply patch for webrtc/video/end_to_end_tests.cc: While running git apply --index -p1; error: patch failed: webrtc/video/end_to_end_tests.cc:19 error: webrtc/video/end_to_end_tests.cc: patch does not apply Patch: webrtc/video/end_to_end_tests.cc Index: webrtc/video/end_to_end_tests.cc diff --git a/webrtc/video/end_to_end_tests.cc b/webrtc/video/end_to_end_tests.cc index 86c181bdf46fc8cd2eba6d086dc8ac9e694d7504..0b602d9af502480d6afa3ef5f3e7cace079e3777 100644 --- a/webrtc/video/end_to_end_tests.cc +++ b/webrtc/video/end_to_end_tests.cc @@ -19,6 +19,7 @@ #include "webrtc/base/event.h" #include "webrtc/base/file.h" #include "webrtc/base/optional.h" +#include "webrtc/base/random.h" #include "webrtc/base/rate_limiter.h" #include "webrtc/call.h" #include "webrtc/common_video/include/frame_callback.h" @@ -690,7 +691,7 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { public rtc::VideoSinkInterface<VideoFrame> { public: FlexfecRenderObserver() - : EndToEndTest(kDefaultTimeoutMs), state_(kFirstPacket) {} + : EndToEndTest(kDefaultTimeoutMs), random_(0xcafef00d) {} size_t GetNumFlexfecStreams() const override { return 1; } @@ -705,36 +706,30 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { EXPECT_EQ(kFlexfecPayloadType, payload_type); } - auto seq_num_it = protected_sequence_numbers_.find(header.sequenceNumber); - if (seq_num_it != protected_sequence_numbers_.end()) { - // Retransmitted packet, should not count. - protected_sequence_numbers_.erase(seq_num_it); - auto ts_it = protected_timestamps_.find(header.timestamp); - EXPECT_NE(ts_it, protected_timestamps_.end()); - protected_timestamps_.erase(ts_it); - return SEND_PACKET; + // Is this a retransmitted media packet? From the perspective of FEC, this + // packet is then no longer dropped, so remove it from the list of + // dropped packets. + if (payload_type == kFakeVideoSendPayloadType) { + auto seq_num_it = dropped_sequence_numbers_.find(header.sequenceNumber); + if (seq_num_it != dropped_sequence_numbers_.end()) { + dropped_sequence_numbers_.erase(seq_num_it); + auto ts_it = dropped_timestamps_.find(header.timestamp); + EXPECT_NE(ts_it, dropped_timestamps_.end()); + dropped_timestamps_.erase(ts_it); + + return SEND_PACKET; + } } - switch (state_) { - case kFirstPacket: - state_ = kDropEveryOtherPacketUntilFlexfec; - break; - case kDropEveryOtherPacketUntilFlexfec: - if (payload_type == kFlexfecPayloadType) { - state_ = kDropNextMediaPacket; - return SEND_PACKET; - } - if (header.sequenceNumber % 2 == 0) - return DROP_PACKET; - break; - case kDropNextMediaPacket: - if (payload_type == kFakeVideoSendPayloadType) { - protected_sequence_numbers_.insert(header.sequenceNumber); - protected_timestamps_.insert(header.timestamp); - state_ = kDropEveryOtherPacketUntilFlexfec; - return DROP_PACKET; - } - break; + // Simulate 5% packet loss. Record what media packets, and corresponding + // timestamps, that were dropped. + if (random_.Rand(1, 100) <= 5) { + if (payload_type == kFakeVideoSendPayloadType) { + dropped_sequence_numbers_.insert(header.sequenceNumber); + dropped_timestamps_.insert(header.timestamp); + } + + return DROP_PACKET; } return SEND_PACKET; @@ -744,17 +739,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { rtc::CritScope lock(&crit_); // Rendering frame with timestamp of packet that was dropped -> FEC // protection worked. - auto it = protected_timestamps_.find(video_frame.timestamp()); - if (it != protected_timestamps_.end()) + auto it = dropped_timestamps_.find(video_frame.timestamp()); + if (it != dropped_timestamps_.end()) observation_complete_.Set(); } - enum { - kFirstPacket, - kDropEveryOtherPacketUntilFlexfec, - kDropNextMediaPacket, - } state_; - void ModifyVideoConfigs( VideoSendStream::Config* send_config, std::vector<VideoReceiveStream::Config>* receive_configs, @@ -768,10 +757,11 @@ TEST_P(EndToEndTest, CanReceiveFlexfec) { } rtc::CriticalSection crit_; - std::set<uint32_t> protected_sequence_numbers_ GUARDED_BY(crit_); + std::set<uint32_t> dropped_sequence_numbers_ GUARDED_BY(crit_); // Since several packets can have the same timestamp a multiset is used // instead of a set. - std::multiset<uint32_t> protected_timestamps_ GUARDED_BY(crit_); + std::multiset<uint32_t> dropped_timestamps_ GUARDED_BY(crit_); + Random random_; } test; RunBaseTest(&test);
Rebase.
The CQ bit was checked by brandtr@webrtc.org
The patchset sent to the CQ was uploaded after l-g-t-m from philipel@webrtc.org, mflodman@webrtc.org Link to the patchset: https://codereview.webrtc.org/2554403003/#ps40001 (title: "Rebase.")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: win_x64_dbg on master.tryserver.webrtc (JOB_FAILED, http://build.chromium.org/p/tryserver.webrtc/builders/win_x64_dbg/builds/4229)
The CQ bit was checked by brandtr@webrtc.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
The CQ bit was unchecked by brandtr@webrtc.org
The CQ bit was checked by brandtr@webrtc.org
The patchset sent to the CQ was uploaded after l-g-t-m from philipel@webrtc.org, mflodman@webrtc.org Link to the patchset: https://codereview.webrtc.org/2554403003/#ps60001 (title: "Try another seed that would work better on win-implementations.")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...
CQ is committing da patch. Bot data: {"patchset_id": 60001, "attempt_start_ts": 1481294863178300, "parent_rev": "a4480f8500fc14597f1d6c5b626e7ce1154dbff6", "commit_rev": "1eba780147440a0d936796487d271ad0d930d91e"}
Message was sent while issue was closed.
Description was changed from ========== Only store sequence numbers for media stream in FlexFEC end-to-end test. This should remove the test flakiness, as before this change there could be collisions from sequence numbers coming from two sequence number spaces (the media SSRC and the FlexFEC SSRC). The probability of collisions was low, and hence the flakes were far between. This change also reduces the packet loss to 5% (down from ~50%), in order for the BWE to have an easier time to ramp up. BUG=webrtc:6825 R=philipel@webrtc.org, mflodman@webrtc.org ========== to ========== Only store sequence numbers for media stream in FlexFEC end-to-end test. This should remove the test flakiness, as before this change there could be collisions from sequence numbers coming from two sequence number spaces (the media SSRC and the FlexFEC SSRC). The probability of collisions was low, and hence the flakes were far between. This change also reduces the packet loss to 5% (down from ~50%), in order for the BWE to have an easier time to ramp up. BUG=webrtc:6825 R=philipel@webrtc.org, mflodman@webrtc.org Review-Url: https://codereview.webrtc.org/2554403003 ==========
Message was sent while issue was closed.
Committed patchset #4 (id:60001)
Message was sent while issue was closed.
Description was changed from ========== Only store sequence numbers for media stream in FlexFEC end-to-end test. This should remove the test flakiness, as before this change there could be collisions from sequence numbers coming from two sequence number spaces (the media SSRC and the FlexFEC SSRC). The probability of collisions was low, and hence the flakes were far between. This change also reduces the packet loss to 5% (down from ~50%), in order for the BWE to have an easier time to ramp up. BUG=webrtc:6825 R=philipel@webrtc.org, mflodman@webrtc.org Review-Url: https://codereview.webrtc.org/2554403003 ========== to ========== Only store sequence numbers for media stream in FlexFEC end-to-end test. This should remove the test flakiness, as before this change there could be collisions from sequence numbers coming from two sequence number spaces (the media SSRC and the FlexFEC SSRC). The probability of collisions was low, and hence the flakes were far between. This change also reduces the packet loss to 5% (down from ~50%), in order for the BWE to have an easier time to ramp up. BUG=webrtc:6825 R=philipel@webrtc.org, mflodman@webrtc.org Committed: https://crrev.com/3536463e7e808747ddc5da2dc0933dbec97ceeeb Cr-Commit-Position: refs/heads/master@{#15512} ==========
Message was sent while issue was closed.
Patchset 4 (id:??) landed as https://crrev.com/3536463e7e808747ddc5da2dc0933dbec97ceeeb Cr-Commit-Position: refs/heads/master@{#15512} |