Add H264 bitstream rewriting to limit frame reordering marker in header

webrtc/modules/rtp_rtcp/source/rtp_format_h264.h

 /*
  *  Copyright (c) 2014 The WebRTC project authors. All Rights Reserved.
  *
  *  Use of this source code is governed by a BSD-style license
  *  that can be found in the LICENSE file in the root of the source
  *  tree. An additional intellectual property rights grant can be found
  *  in the file PATENTS.  All contributing project authors may
  *  be found in the AUTHORS file in the root of the source tree.
  */
 
 #ifndef WEBRTC_MODULES_RTP_RTCP_SOURCE_RTP_FORMAT_H264_H_
 #define WEBRTC_MODULES_RTP_RTCP_SOURCE_RTP_FORMAT_H264_H_
 
+#include <deque>
 #include <queue>
 #include <string>
 
+#include "webrtc/base/buffer.h"
 #include "webrtc/base/constructormagic.h"
 #include "webrtc/modules/rtp_rtcp/source/rtp_format.h"
 
 namespace webrtc {
 
 class RtpPacketizerH264 : public RtpPacketizer {
  public:
   // Initialize with payload from encoder.
   // The payload_data must be exactly one encoded H264 frame.
   RtpPacketizerH264(FrameType frame_type, size_t max_payload_len);
@@ skipping 15 matching lines @@
                   size_t* bytes_to_send,
                   bool* last_packet) override;
 
   ProtectionType GetProtectionType() override;
 
   StorageType GetStorageType(uint32_t retransmission_settings) override;
 
   std::string ToString() override;
 
  private:
-  struct Packet {
-    Packet(size_t offset,
-           size_t size,
-           bool first_fragment,
-           bool last_fragment,
-           bool aggregated,
-           uint8_t header)
-        : offset(offset),
-          size(size),
+  // Input fragments (NAL units), with an optionally owned temporary buffer,
+  // used in case the fragment gets modified.
+  struct Fragment {
+    Fragment(const uint8_t* buffer, size_t length);
+    explicit Fragment(std::unique_ptr<rtc::Buffer> buffer);
+    explicit Fragment(const Fragment& fragment);
+    const uint8_t* buffer = nullptr;
+    size_t length = 0;
+    std::unique_ptr<rtc::Buffer> tmp_buffer;
+  };
+
+  // A packet unit (H264 packet), to be put into an RTP packet:
+  // If a NAL unit is too large for an RTP packet, this packet unit will
+  // represent a FU-A packet of a single fragment of the NAL unit.
+  // If a NAL unit is small enough to fit within a single RTP packet, this
+  // packet unit may represent a single NAL unit or a STAP-A packet, of which
+  // there may be multiple in a single RTP packet (if so, aggregated = true).
+  struct PacketUnit {
+    PacketUnit(const Fragment& source_fragment,
+               bool first_fragment,
+               bool last_fragment,
+               bool aggregated,
+               uint8_t header)
+        : source_fragment(source_fragment),
           first_fragment(first_fragment),
           last_fragment(last_fragment),
           aggregated(aggregated),
           header(header) {}
 
-    size_t offset;
-    size_t size;
+    const Fragment source_fragment;
     bool first_fragment;
     bool last_fragment;
     bool aggregated;
     uint8_t header;
   };
-  typedef std::queue<Packet> PacketQueue;
 
   void GeneratePackets();
-  void PacketizeFuA(size_t fragment_offset, size_t fragment_length);
-  int PacketizeStapA(size_t fragment_index,
-                     size_t fragment_offset,
-                     size_t fragment_length);
+  void PacketizeFuA(size_t fragment_index);
+  size_t PacketizeStapA(size_t fragment_index);
   void NextAggregatePacket(uint8_t* buffer, size_t* bytes_to_send);
   void NextFragmentPacket(uint8_t* buffer, size_t* bytes_to_send);
 
-  const uint8_t* payload_data_;
-  size_t payload_size_;
   const size_t max_payload_len_;
-  RTPFragmentationHeader fragmentation_;
-  PacketQueue packets_;
+  std::deque<Fragment> input_fragments_;
+  std::queue<PacketUnit> packets_;
 
   RTC_DISALLOW_COPY_AND_ASSIGN(RtpPacketizerH264);
 };
 
 // Depacketizer for H264.
 class RtpDepacketizerH264 : public RtpDepacketizer {
  public:
-  virtual ~RtpDepacketizerH264() {}
+  RtpDepacketizerH264();
+  virtual ~RtpDepacketizerH264();
 
   bool Parse(ParsedPayload* parsed_payload,
              const uint8_t* payload_data,
              size_t payload_data_length) override;
+
+ private:
+  bool ParseFuaNalu(RtpDepacketizer::ParsedPayload* parsed_payload,
+                    const uint8_t* payload_data);
+  bool ParseSingleNalu(RtpDepacketizer::ParsedPayload* parsed_payload,
+                       const uint8_t* payload_data);
+
+  size_t offset_;
+  size_t length_;
+  std::unique_ptr<rtc::Buffer> modified_buffer_;

[stefan-webrtc, 2016/05/22 23:11:50]

Did you consider propagating the copy on modify buffer all the way in from the
network layers instead?

Are we sure that a second packet won't overwrite this buffer while the first
packet is still being processed? Should that be ensured by this class? Seems a
bit brittle to leave that to the caller.

[sprang_webrtc, 2016/05/25 09:06:03]

Yes, but that turned out to be a bit more work than I'd like - and something
mflodman@ has already begun working on but hasn't finished yet. We can revisit
this at that time instead, I'll add a TODO.
I've added an assert that we don't have two SPS blocks, that both needs to be
modified, packet together within a single RTP packet using STAP-A. That should
really not happen.

[stefan-webrtc, 2016/05/26 18:08:13]

Acknowledged.
That's not what I was thinking of. What if the first packet is parsed and sent
off to the jitter buffer, but before the jitter buffer is done with it a new
packet comes in which also needs to be modified, effectively overwriting the
first packet.

I think that will become increasingly likely with philipel's new jitter buffer
which doesn't copy the packets until they are passed to the decoder, see
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/webrtc....

I think we'll need a better way of handling this... The best way would really be
to have a copy-on-modify, or if we could simply pass on packet ownership?

[sprang_webrtc, 2016/05/27 13:12:21]

I've talked to Philip about this. We'll need a copy somewhere anyway, to hand
ownership over to the jitter buffer, and this seems like the right place. For
now, let's keep this and then in a follow up CL, have ParsedPayload contain a
unique_ptr with a copy of the parsed buffer. Then we can std::move that into the
jitter buffer making ownership clear, and any codec specific pre-processing
needed can take place in the depacketizer instead of inside the jitter buffer,
which is the case today.

[stefan-webrtc, 2016/05/27 20:56:59]

Yes, a copy will be needed, at least for the near future. It's possible to pass
ownership of the packet all the way from the socket to the jitter buffer though.

But if we keep this for now, does that mean we're making the assumption that we
won't run into the risk of races when one packet is on its way to the JB (but
hasn't been copied yet) and another packet with an sps to be modified arrives?
Is that safe? If you think it is I think you at least have to document it. Or
should we add a lock or something for now to ensure it's not an issue?

[sprang_webrtc, 2016/05/30 08:57:23]

The incoming packet goes out of scope when the IncomingRtpPacket() callback
returns, so the jitter buffer can really only use it directly if it's a single
packet frame that's received. Otherwise a copy will be needed somewhere even in
the future; it can never just hold on to the pointer without being racy.

After discussing with with philipel@, we agreed that _always_ cloning incoming
payload in the depackterizer is the right thing to do. Not only for H264, but
all codecs. I'll add a comment to this effect in the RtpDepacketizer interface.

[stefan-webrtc, 2016/05/30 11:00:55]

I was thinking that ownership would be passed on to the jitter buffer. Would
that not be possible? Not sure if it would make sense, but you could for
instance have a pool of packets at receiver and handle it similarly to how we
handle frames.
But if we do that now we'll be doing two memcpys for all packets with the old
jitter buffer, since the old jitter buffer does this memcpy already. We should
only do it for all packets with the new one, right?

[sprang_webrtc, 2016/05/30 12:39:30]

Then I don't think CopyOnWriteBuffer is the way to go, either std::unique_ptr or
something ref-counted.
In either case, that's a lot of plumbing. I think that should be done in
conjunction with enabling the new jitterbuffer instead of this CL.
Yes, this needs to be enabled in conjunction with enabling the new jitter
buffer, so it's something that should be done in this CL. Copying h264 packets
with a suboptimal sps in it should be fine though.

[danilchap, 2016/05/30 12:55:59]

CopyOnWriteBuffer is actually a ref-counted rtc::Buffer underneath.
(with an extra feature: copy on write if needed, i.e. ref_counter > 1).

 };
 }  // namespace webrtc
 #endif  // WEBRTC_MODULES_RTP_RTCP_SOURCE_RTP_FORMAT_H264_H_