Report timing frames info in GetStats.

webrtc/api/video/video_timing.h

 /*
  *  Copyright (c) 2017 The WebRTC project authors. All Rights Reserved.
  *
  *  Use of this source code is governed by a BSD-style license
  *  that can be found in the LICENSE file in the root of the source
  *  tree. An additional intellectual property rights grant can be found
  *  in the file PATENTS.  All contributing project authors may
  *  be found in the AUTHORS file in the root of the source tree.
  */
 
 #ifndef WEBRTC_API_VIDEO_VIDEO_TIMING_H_
 #define WEBRTC_API_VIDEO_VIDEO_TIMING_H_
 
 #include <stdint.h>
-#include <limits>
+
+#include <string>
+
 #include "webrtc/base/checks.h"
 #include "webrtc/base/safe_conversions.h"
 
 namespace webrtc {
 
-// Video timing timstamps in ms counted from capture_time_ms of a frame.
-struct VideoTiming {
+// Video timing timestamps in ms counted from capture_time_ms of a frame.
+// This structure represents data sent in video-timing RTP header extension.
+struct VideoSendTiming {
   static const uint8_t kEncodeStartDeltaIdx = 0;
   static const uint8_t kEncodeFinishDeltaIdx = 1;
   static const uint8_t kPacketizationFinishDeltaIdx = 2;
   static const uint8_t kPacerExitDeltaIdx = 3;
   static const uint8_t kNetworkTimestampDeltaIdx = 4;
   static const uint8_t kNetwork2TimestampDeltaIdx = 5;
 
   // Returns |time_ms - base_ms| capped at max 16-bit value.
   // Used to fill this data structure as per
   // https://webrtc.org/experiments/rtp-hdrext/video-timing/ extension stores
   // 16-bit deltas of timestamps from packet capture time.
   static uint16_t GetDeltaCappedMs(int64_t base_ms, int64_t time_ms) {
     RTC_DCHECK_GE(time_ms, base_ms);
     return rtc::saturated_cast<uint16_t>(time_ms - base_ms);
   }
 
   uint16_t encode_start_delta_ms;
   uint16_t encode_finish_delta_ms;
   uint16_t packetization_finish_delta_ms;
   uint16_t pacer_exit_delta_ms;
   uint16_t network_timstamp_delta_ms;
   uint16_t network2_timstamp_delta_ms;
   bool is_timing_frame;
 };
 
+// Used to report precise timings of a 'timing frames'. Contains all important
+// timestamps for a lifetime of that specific frame. Reported as a string via
+// GetStats(). Only frame which took the longest between two GetStats calls is
+// reported.
+struct TimingFrameInfo {
+  TimingFrameInfo();
+
+  // Returns end-to-end delay of a frame, if sender and receiver timestamps are
+  // synchronized, -1 otherwise.
+  int64_t EndToEndDelay() const;

[hbos, 2017/06/27 14:08:30]

Generally speaking, is there any way to know if the timestamps are synchronized?
Is this only useful an application-specific contexts where you know that peers
all synchronize to the same NTP servers?

[ilnik, 2017/06/27 15:23:56]

We know when timestamps are synchronised, because they are derived from ntp
capture time of a frame, which is estimated from rtp timestamp using continious
clock synchronization via rtcp messages. We don't in any way rely on how system
synchronizes its ntp time.

[hbos, 2017/07/06 13:27:29]

Is this synchronization method and the video-timing RTP header extension
standards? Is end-to-end delay possible to calculate through standard-supported
means?
A stat for end-to-end delay was blocked on not being able to calculate this with
satisfactory estimates, https://github.com/w3c/webrtc-stats/issues/222.

[ilnik, 2017/07/06 14:30:56]

I don't know about synchronization method standard. We utilize Receiver Reports
to make it. I guess, nothing non-standardized is used. And, yes, it's not very
precise. It just estimates RTT and assumes that packets take RTT/2 to go from
sender to receiver.  The timing frames extension is not standardized at all.
End-to-end delay is already calculated without blatant standards violation, but
reporting timings for all parts of a video pipeline, like i am trying to do, is
impossible with such an extension.

+
+  // Returns true if current frame took longer to process than |other| frame.
+  // If other frame's clocks are not synchronized, current frame is always
+  // preferred.
+  bool IsLongerThan(const TimingFrameInfo& other) const;
+
+  std::string ToString() const;
+
+  uint32_t rtp_timestamp;  // Identifier of a frame.
+  int64_t capture_time_ms;

[hbos, 2017/06/27 14:08:30]

Are these relative to session or relative to unix epoch, etc? RTP or NTP? Can
you add a comment and/or clarify the names.

[ilnik, 2017/06/27 15:23:56]

These are in some local monotonous clock. Added clarifications.

+  int64_t encode_start_ms;
+  int64_t encode_finish_ms;
+  int64_t packetization_finish_ms;
+  int64_t pacer_exit_ms;
+  int64_t network_timestamp_ms;  // In network RTP processor timestamps:
+  int64_t network2_timestamp_ms;  // Meaning is application specific.

[hbos, 2017/06/27 14:08:30]

nit: If comment spans more than one line add it to before the variable instead
of on the right side of it.

[ilnik, 2017/06/27 15:23:56]

Done.

+  int64_t receive_start_ms;  // First received packet time.
+  int64_t receive_finish_ms;  // Last received packet time.
+  int64_t decode_start_ms;
+  int64_t decode_finish_ms;
+  int64_t render_time_ms;  // Inferred smooth render time.

[hbos, 2017/06/27 14:08:31]

What does "Inferred smooth render time" mean?

[ilnik, 2017/06/27 15:23:56]

WebRTC provides intended render time for each frame, so there will be some
playout delay to provide smooth playback.

+};
+
 }  // namespace webrtc
 
 #endif  // WEBRTC_API_VIDEO_VIDEO_TIMING_H_