New rtc dump analyzing tool in Python

tools/py_event_log_analyzer/pb_parse.py

Index: tools/py_event_log_analyzer/pb_parse.py
diff --git a/tools/py_event_log_analyzer/pb_parse.py b/tools/py_event_log_analyzer/pb_parse.py
new file mode 100644
index 0000000000000000000000000000000000000000..d28a96e765ee575fe46c129489589d68630c1312
--- /dev/null
+++ b/tools/py_event_log_analyzer/pb_parse.py
@@ -0,0 +1,64 @@
+#  Copyright (c) 2016 The WebRTC project authors. All Rights Reserved.
+#
+#  Use of this source code is governed by a BSD-style license
+#  that can be found in the LICENSE file in the root of the source
+#  tree. An additional intellectual property rights grant can be found
+#  in the file PATENTS.  All contributing project authors may
+#  be found in the AUTHORS file in the root of the source tree.
+
+"""Parses protobuf RTC dumps.
+"""

[phoglund, 2016/05/31 13:47:43]

Nit: pull up line #10 to #9.

+
+from __future__ import division
+
+import os
+import struct
+import sys
+
+
+# makes sure to import correct protobuf

[phoglund, 2016/05/31 13:47:43]

Nit: uppercase M, end with .

+sys.path += [os.path.join("../../", path) for path in
+             ["out/Debug/pyproto/webrtc/call/",

[phoglund, 2016/05/31 13:47:43]

We should not do import hacks, and especially not if they build on brittle
assumptions as this one. If you build with GN, for instance, things will tend to
end up in out/Default (but the user can really choose whatever name they want). 

This will not even do what you expect for gyp, in all cases. Suppose for
instance you build with debug first but then switch to release. Since debug is
before release in the path, it will not pick up any protobuf changes if you
change + rebuild in release. Furthermore, these scripts do not properly depend
on the protobufs, so they count on that the user has hand-built those before
running. That's ugly and barbaric, so we need to fix that :)

Instead, make gyp/gn rules to copy this script to the appropriate output folder.
This means you can't just run the scripts, but you need to "build" them and then
run them. If we build, for instance, ninja -C out/Debug rtp_analyzer, all
scripts in here are copied to out/Debug. To run, you'd do
out/Debug/rtp_analyzer.py

From this file you can then just do

import pyproto.webrtc.call.rtc_event_log_pb2 as rtc_pb

since those files are already in out/Debug.

Create a py_event_log_analyzer.gyp file with something like this in it:

    { 'targets': [
        {
          'target_name': 'rtp_analyzer',
          'type': 'none',
          'actions': [
            {
              'action_name': 'copy_py_scripts',
              'variables': {
                'copy_output_dir%': '<(PRODUCT_DIR)',
              }
              'inputs': [
                '<(DEPTH)/pb_parse.py',
                '<(DEPTH)/misc.py',
                # ... etc ...
              ],
              'outputs': [
                '<(copy_output_dir)',
              ],
              'action': ['python',
                         '<(DEPTH)/copy_files.py',
                         '--input_dir=<(DEPTH)',
                         '--output_dir=<(copy_output_dir)'],
                  'message': 'Copying *py files code to <(copy_output_dir)',
                  'process_outputs_as_sources': 1,
                },
              ],
              'dependencies': [
                # Depend on protos which makes sure they are built before this
                # target builds. Not sure if this is the right target to depend
                # on (does it generate python protobufs?)
                'webrtc/webrtc.gyp:rtc_event_log_proto',
              ],
            }],
          ],
        },
      ],
    }

We could also simply invoke cp or cp -r for the action, but this needs to work
on Windows, so write a simple python script that copies files from input_dir to
output_dir. Note that PRODUCT_DIR will be out/Debug or whatever the user has
chosen - don't ever hard-code that! DEPTH is the dir of the gyp file.

[aleloi, 2016/06/02 14:01:56]

I added the changes to /webrtc/webrtc.gyp and /webrtc/BUILD.gn. Both gyp and gn
seem to work as intended, but there are two python problems:

python2 won't import pyproto.webrtc.call..., because the folders lack
__init__.py files.

../../third_party/protobuf/python still needs to be added to path.

+              "out/Release/pyproto/webrtc/call/",
+              "third_party/protobuf/python/"]]
+import rtc_event_log_pb2 as rtc_pb
+
+
+class DataPoint(object):
+  """Simple container class for RTP events.
+  """
+
+  def __init__(self, rtp_header_str, packet_size,
+               arrival_timestamp_us):
+    """Builds a data point by parsing an RTP header, and extra information
+    about packet size and arrival time.
+
+    RTP header structure is defined in RFC 3550 section 5.1.
+
+    """
+    self.size = packet_size
+    self.arrival_timestamp_ms = arrival_timestamp_us / 1000
+    (first2header_bytes, self.sequence_number,
+     self.timestamp, self.ssrc
+    ) = struct.unpack_from("!HHII", rtp_header_str, 0)
+    self.payload_type = first2header_bytes & 0b01111111
+
+
+def parse_protobuf(file_path):
+  """Parses RTC event log from protobuf file.
+
+  Args:
+       file_path: path to protobuf file of RTC event stream
+
+  Returns:
+    all RTP packet events from the event stream as a list of DataPoints
+  """
+  event_stream = rtc_pb.EventStream()
+  with open(file_path, "rb") as f:
+    event_stream.ParseFromString(f.read())
+
+  return [DataPoint(event.rtp_packet.header,
+                    event.rtp_packet.packet_length,
+                    event.timestamp_us)
+          for event in event_stream.stream
+          if event.HasField("rtp_packet")]