Add texture support to HardwareVideoEncoder.

webrtc/sdk/android/src/java/org/webrtc/HardwareVideoEncoder.java

 /*
  *  Copyright 2017 The WebRTC project authors. All Rights Reserved.
  *
  *  Use of this source code is governed by a BSD-style license
  *  that can be found in the LICENSE file in the root of the source
  *  tree. An additional intellectual property rights grant can be found
  *  in the file PATENTS.  All contributing project authors may
  *  be found in the AUTHORS file in the root of the source tree.
  */
 
 package org.webrtc;
 
 import android.annotation.TargetApi;
+import android.graphics.Matrix;
 import android.media.MediaCodec;
 import android.media.MediaCodecInfo;
 import android.media.MediaFormat;
+import android.opengl.GLES20;
 import android.os.Bundle;
+import android.view.Surface;
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.util.Arrays;
 import java.util.Deque;
 import java.util.HashSet;
 import java.util.Set;
 import java.util.concurrent.LinkedBlockingDeque;
 
 /** Android hardware video encoder. */
 @TargetApi(19)
@@ skipping 38 matching lines @@
   // Thread that delivers encoded frames to the user callback.
   private Thread outputThread;
 
   // Whether the encoder is running.  Volatile so that the output thread can watch this value and
   // exit when the encoder stops.
   private volatile boolean running = false;
   // Any exception thrown during shutdown.  The output thread releases the MediaCodec and uses this
   // value to send exceptions thrown during release back to the encoder thread.
   private volatile Exception shutdownException = null;
 
+  // Surface objects for texture-mode encoding.

[pthatcher1, 2017/07/17 22:50:19]

Would it make sense to write a comment explaining what they are and how they
interact?

[mellem, 2017/07/17 23:22:00]

Done.

+  private EglBase14.Context sharedContext;
+  private EglBase14 eglBase;
+  private Surface inputSurface;
+  private GlRectDrawer drawer;

[pthatcher1, 2017/07/17 22:50:19]

It would be nice if these had a common name that made it clear it was for
texture encoding.  Perhaps something like textureContext, textureBase,
textureSurface, textureDrawer?

[mellem, 2017/07/17 23:22:01]

Done.

+
   private MediaCodec codec;
   private Callback callback;
 
   private int width;
   private int height;
 
   // Contents of the last observed config frame output by the MediaCodec. Used by H.264.
   private ByteBuffer configBuffer = null;
 
   /**
    * Creates a new HardwareVideoEncoder with the given codecName, codecType, colorFormat, key frame
    * intervals, and bitrateAdjuster.
    *
    * @param codecName the hardware codec implementation to use
    * @param codecType the type of the given video codec (eg. VP8, VP9, or H264)
    * @param colorFormat color format used by the input buffer
    * @param keyFrameIntervalSec interval in seconds between key frames; used to initialize the codec
    * @param forceKeyFrameIntervalMs interval at which to force a key frame if one is not requested;
    *     used to reduce distortion caused by some codec implementations
    * @param bitrateAdjuster algorithm used to correct codec implementations that do not produce the
    *     desired bitrates
    * @throws IllegalArgumentException if colorFormat is unsupported
    */
   public HardwareVideoEncoder(String codecName, VideoCodecType codecType, int colorFormat,
-      int keyFrameIntervalSec, int forceKeyFrameIntervalMs, BitrateAdjuster bitrateAdjuster) {
+      int keyFrameIntervalSec, int forceKeyFrameIntervalMs, BitrateAdjuster bitrateAdjuster,
+      EglBase14.Context sharedContext) {
     this.codecName = codecName;
     this.codecType = codecType;
     this.colorFormat = colorFormat;
-    this.inputColorFormat = ColorFormat.valueOf(colorFormat);
+    if (sharedContext == null) {
+      this.inputColorFormat = ColorFormat.valueOf(colorFormat);
+    } else {
+      this.inputColorFormat = null;

[pthatcher1, 2017/07/17 22:50:19]

Can you write a comment explaining why the inputColorFormat is null when the
texture shared context is non-null?

[mellem, 2017/07/17 23:22:00]

Done.

+    }
     this.keyFrameIntervalSec = keyFrameIntervalSec;
     this.forcedKeyFrameMs = forceKeyFrameIntervalMs;
     this.bitrateAdjuster = bitrateAdjuster;
     this.outputBuilders = new LinkedBlockingDeque<>();
+    this.sharedContext = sharedContext;
   }
 
   @Override
   public VideoCodecStatus initEncode(Settings settings, Callback callback) {
     return initEncodeInternal(
         settings.width, settings.height, settings.startBitrate, settings.maxFramerate, callback);
   }
 
   private VideoCodecStatus initEncodeInternal(
       int width, int height, int bitrateKbps, int fps, Callback callback) {
@@ skipping 18 matching lines @@
     }
     try {
       MediaFormat format = MediaFormat.createVideoFormat(codecType.mimeType(), width, height);
       format.setInteger(MediaFormat.KEY_BIT_RATE, adjustedBitrate);
       format.setInteger(KEY_BITRATE_MODE, VIDEO_ControlRateConstant);
       format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
       format.setInteger(MediaFormat.KEY_FRAME_RATE, bitrateAdjuster.getAdjustedFramerate());
       format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, keyFrameIntervalSec);
       Logging.d(TAG, "Format: " + format);
       codec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
+
+      if (sharedContext != null) {
+        // Surface mode.

[pthatcher1, 2017/07/17 22:50:19]

Above it was called "texture mode", not it says "surface mode".  That's the same
thing, I'm assuming.  If so, can we pick one?

[mellem, 2017/07/17 23:22:00]

Done.

+        eglBase = new EglBase14(sharedContext, EglBase.CONFIG_RECORDABLE);
+        inputSurface = codec.createInputSurface();
+        eglBase.createSurface(inputSurface);
+        drawer = new GlRectDrawer();
+      }
+
       codec.start();
     } catch (IllegalStateException e) {
       Logging.e(TAG, "initEncode failed", e);
       release();
       return VideoCodecStatus.ERROR;
     }
 
     running = true;
     outputThread = createOutputThread();
     outputThread.start();
 
     return VideoCodecStatus.OK;
   }
 
   @Override
   public VideoCodecStatus release() {
     try {
+      if (outputThread == null) {
+        return VideoCodecStatus.OK;
+      }

[pthatcher1, 2017/07/17 22:50:19]

Is this only true when we're in texture mode?  If we're not an there's so
outputThread, wouldn't that be an error?

[mellem, 2017/07/17 23:22:00]

No, this is just a fix for a bug that I whacked.  I noticed it because I had
test errors while writing texture support.  If init fails, the output thread may
still be null when release() is called.  In that case, we can't join the output
thread.

       // The outputThread actually stops and releases the codec once running is false.
       running = false;
       if (!ThreadUtils.joinUninterruptibly(outputThread, MEDIA_CODEC_RELEASE_TIMEOUT_MS)) {
         Logging.e(TAG, "Media encoder release timeout");
         return VideoCodecStatus.TIMEOUT;
       }
       if (shutdownException != null) {
         // Log the exception and turn it into an error.
         Logging.e(TAG, "Media encoder release exception", shutdownException);
         return VideoCodecStatus.ERROR;
       }
     } finally {
       codec = null;
       outputThread = null;
       outputBuilders.clear();
+
+      if (drawer != null) {
+        drawer.release();
+        drawer = null;
+      }
+      if (eglBase != null) {
+        eglBase.release();
+        eglBase = null;
+      }
+      if (inputSurface != null) {
+        inputSurface.release();
+        inputSurface = null;
+      }

[pthatcher1, 2017/07/17 22:50:19]

Would it make sense to put the 4 texture things in one "TextureModeState" class?
 Then it could have a release() and you'd only need one of these checks.

[mellem, 2017/07/17 23:22:01]

It's possible to fail init partway through allocating these (eg. if
createInputSurface throws).  We need to release whatever we created.  So I think
each needs a separate null check.

     }
     return VideoCodecStatus.OK;
   }
 
   @Override
   public VideoCodecStatus encode(VideoFrame videoFrame, EncodeInfo encodeInfo) {
     if (codec == null) {
       return VideoCodecStatus.UNINITIALIZED;
     }
 
     // If input resolution changed, restart the codec with the new resolution.
     int frameWidth = videoFrame.getWidth();
     int frameHeight = videoFrame.getHeight();
     if (frameWidth != width || frameHeight != height) {
       VideoCodecStatus status = resetCodec(frameWidth, frameHeight);
       if (status != VideoCodecStatus.OK) {
         return status;
       }
     }
 
-    // No timeout.  Don't block for an input buffer, drop frames if the encoder falls behind.
-    int index;
-    try {
-      index = codec.dequeueInputBuffer(0 /* timeout */);
-    } catch (IllegalStateException e) {
-      Logging.e(TAG, "dequeueInputBuffer failed", e);
-      return VideoCodecStatus.FALLBACK_SOFTWARE;
-    }
-
-    if (index == -1) {
-      // Encoder is falling behind.  No input buffers available.  Drop the frame.
-      Logging.e(TAG, "Dropped frame, no input buffers available");
-      return VideoCodecStatus.OK; // See webrtc bug 2887.
-    }
     if (outputBuilders.size() > MAX_ENCODER_Q_SIZE) {
       // Too many frames in the encoder.  Drop this frame.
       Logging.e(TAG, "Dropped frame, encoder queue full");
       return VideoCodecStatus.OK; // See webrtc bug 2887.
     }
 
-    // TODO(mellem):  Add support for input surfaces and textures.
-    ByteBuffer buffer;
-    try {
-      buffer = codec.getInputBuffers()[index];
-    } catch (IllegalStateException e) {
-      Logging.e(TAG, "getInputBuffers failed", e);
-      return VideoCodecStatus.FALLBACK_SOFTWARE;
-    }
-    VideoFrame.I420Buffer i420 = videoFrame.getBuffer().toI420();
-    inputColorFormat.fillBufferFromI420(buffer, i420);
-
     boolean requestedKeyFrame = false;
     for (EncodedImage.FrameType frameType : encodeInfo.frameTypes) {
       if (frameType == EncodedImage.FrameType.VideoFrameKey) {
         requestedKeyFrame = true;
       }
     }
 
     // Frame timestamp rounded to the nearest microsecond and millisecond.
     long presentationTimestampUs = (videoFrame.getTimestampNs() + 500) / 1000;
     long presentationTimestampMs = (presentationTimestampUs + 500) / 1000;
     if (requestedKeyFrame || shouldForceKeyFrame(presentationTimestampMs)) {
       requestKeyFrame(presentationTimestampMs);
     }
 
+    VideoFrame.Buffer videoFrameBuffer = videoFrame.getBuffer();
     // Number of bytes in the video buffer. Y channel is sampled at one byte per pixel; U and V are
     // subsampled at one byte per four pixels.
-    int bufferSize = videoFrame.getBuffer().getHeight() * videoFrame.getBuffer().getWidth() * 3 / 2;
+    int bufferSize = videoFrameBuffer.getHeight() * videoFrameBuffer.getWidth() * 3 / 2;
     EncodedImage.Builder builder = EncodedImage.builder()
                                        .setCaptureTimeMs(presentationTimestampMs)
                                        .setCompleteFrame(true)
                                        .setEncodedWidth(videoFrame.getWidth())
                                        .setEncodedHeight(videoFrame.getHeight())
                                        .setRotation(videoFrame.getRotation());
     outputBuilders.offer(builder);
-    try {
-      codec.queueInputBuffer(
-          index, 0 /* offset */, bufferSize, presentationTimestampUs, 0 /* flags */);
-    } catch (IllegalStateException e) {
-      Logging.e(TAG, "queueInputBuffer failed", e);
-      // Keep the output builders in sync with buffers in the codec.
-      outputBuilders.pollLast();
-      // IllegalStateException thrown when the codec is in the wrong state.
-      return VideoCodecStatus.FALLBACK_SOFTWARE;
+
+    if (videoFrameBuffer instanceof VideoFrame.TextureBuffer) {

[sakal, 2017/07/17 12:25:41]

nit: I would prefer these cases in separate methods.

[mellem, 2017/07/17 17:49:29]

Done.

+      VideoFrame.TextureBuffer textureBuffer = (VideoFrame.TextureBuffer) videoFrameBuffer;
+
+      // TODO(mellem):  Put this matrix manipulation in a helper.

[sakal, 2017/07/17 12:25:41]

I have implemented such helper here
https://chromium-review.googlesource.com/c/555516/4/webrtc/sdk/android/api/or...

feel free to copy

[mellem, 2017/07/17 17:49:30]

Done.  Thanks for writing that.

+      // The android.graphics.Matrix looks like this:
+      // [x1 y1 w1]
+      // [x2 y2 w2]
+      // [x3 y3 w3]
+      // We want to contruct a matrix that looks like this:
+      // [x1 y1  0 w1]
+      // [x2 y2  0 w2]
+      // [ 0  0  1  0]
+      // [x3 y3  0 w3]
+      Matrix matrix = videoFrame.getTransformMatrix();
+      float[] matrix3x3 = new float[9];
+      matrix.getValues(matrix3x3);
+
+      float[] transformationMatrix = new float[16];
+      transformationMatrix[0 * 4 + 0] = matrix3x3[0 * 3 + 0];
+      transformationMatrix[0 * 4 + 1] = matrix3x3[0 * 3 + 1];
+      transformationMatrix[0 * 4 + 3] = matrix3x3[0 * 3 + 2];
+      transformationMatrix[1 * 4 + 0] = matrix3x3[1 * 3 + 0];
+      transformationMatrix[1 * 4 + 1] = matrix3x3[1 * 3 + 1];
+      transformationMatrix[1 * 4 + 3] = matrix3x3[1 * 3 + 2];
+      transformationMatrix[2 * 4 + 2] = 1; // Z-scale should be 1.
+      transformationMatrix[3 * 4 + 0] = matrix3x3[2 * 3 + 0];
+      transformationMatrix[3 * 4 + 1] = matrix3x3[2 * 3 + 1];
+      transformationMatrix[3 * 4 + 3] = matrix3x3[2 * 3 + 2];
+
+      try {
+        eglBase.makeCurrent();
+        // TODO(perkj): glClear() shouldn't be necessary since every pixel is covered anyway,
+        // but it's a workaround for bug webrtc:5147.
+        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
+        drawer.drawOes(

[sakal, 2017/07/17 12:25:42]

nit: TextureFrame might be RGB frame in the future. Can we call drawRgb if
getType is RGB?

[mellem, 2017/07/17 17:49:29]

Done.

+            textureBuffer.getTextureId(), transformationMatrix, width, height, 0, 0, width, height);
+        eglBase.swapBuffers(videoFrame.getTimestampNs());
+      } catch (RuntimeException e) {
+        Logging.e(TAG, "encodeTexture failed", e);
+        // Keep the output builders in sync with buffers in the codec.
+        outputBuilders.pollLast();
+        return VideoCodecStatus.ERROR;
+      }
+      return VideoCodecStatus.OK;
+    } else {
+      // No timeout.  Don't block for an input buffer, drop frames if the encoder falls behind.

[sakal, 2017/07/17 12:25:42]

Can we at least log a clear error if the mode doesn't match what kind of frames
we receive? How hard would it be to reinitialize the encoder?

[mellem, 2017/07/17 17:49:30]

Logging added.  We can actually process a texture buffer in non-texture mode
(texture buffers still have toI420), just not the other way around.  Or maybe we
can draw the byte-buffer onto a surface right here?

As it stands, it would be quite hard to re-initialize into a different mode.  I
would have to refactor code so that we can find an appropriate color format
during init.  Byte to texture would not work anyway if we're missing the EGL
context.  Texture to byte might not work if the codec supports texture-mode but
none of our byte-buffer color formats.

I'd rather not attempt it, since existing code CHECK-fails if it gets handed a
frame in the wrong mode:

https://cs.chromium.org/chromium/src/third_party/webrtc/sdk/android/src/jni/a...

https://cs.chromium.org/chromium/src/third_party/webrtc/sdk/android/src/jni/a...

[sakal, 2017/07/18 08:46:13]

We reconfigure the encoder before we hit that check:
https://cs.chromium.org/chromium/src/third_party/webrtc/sdk/android/src/jni/a...

But we can implement this in a follow-up if needed.

[mellem, 2017/07/18 17:11:57]

Acknowledged.

+      int index;
+      try {
+        index = codec.dequeueInputBuffer(0 /* timeout */);
+      } catch (IllegalStateException e) {
+        Logging.e(TAG, "dequeueInputBuffer failed", e);
+        return VideoCodecStatus.FALLBACK_SOFTWARE;

[sakal, 2017/07/17 12:25:42]

Can you just replace all FALLBACK_SOTWAREs with ERRORs and we will deal with the
fallback in C++.

[mellem, 2017/07/17 17:49:29]

Done.

+      }
+
+      if (index == -1) {
+        // Encoder is falling behind.  No input buffers available.  Drop the frame.
+        Logging.e(TAG, "Dropped frame, no input buffers available");
+        return VideoCodecStatus.OK; // See webrtc bug 2887.
+      }
+
+      ByteBuffer buffer;
+      try {
+        buffer = codec.getInputBuffers()[index];
+      } catch (IllegalStateException e) {
+        Logging.e(TAG, "getInputBuffers failed", e);
+        return VideoCodecStatus.FALLBACK_SOFTWARE;
+      }
+      VideoFrame.I420Buffer i420 = videoFrameBuffer.toI420();

[sakal, 2017/07/17 12:25:41]

toI420 will return "a new instance". Therefore, we should release the i420
buffer.

[mellem, 2017/07/17 17:49:29]

Done.

+      inputColorFormat.fillBufferFromI420(buffer, i420);
+
+      try {
+        codec.queueInputBuffer(
+            index, 0 /* offset */, bufferSize, presentationTimestampUs, 0 /* flags */);
+      } catch (IllegalStateException e) {
+        Logging.e(TAG, "queueInputBuffer failed", e);
+        // Keep the output builders in sync with buffers in the codec.
+        outputBuilders.pollLast();
+        // IllegalStateException thrown when the codec is in the wrong state.
+        return VideoCodecStatus.FALLBACK_SOFTWARE;
+      }
+      return VideoCodecStatus.OK;
     }
-    return VideoCodecStatus.OK;
   }
 
   @Override
   public VideoCodecStatus setChannelParameters(short packetLoss, long roundTripTimeMs) {
     // No op.
     return VideoCodecStatus.OK;
   }
 
   @Override
   public VideoCodecStatus setRateAllocation(BitrateAllocation bitrateAllocation, int framerate) {
@@ skipping 177 matching lines @@
         case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar:
         case MediaCodecInfo.CodecCapabilities.COLOR_QCOM_FormatYUV420SemiPlanar:
         case MediaCodecUtils.COLOR_QCOM_FORMATYUV420PackedSemiPlanar32m:
           return NV12;
         default:
           throw new IllegalArgumentException("Unsupported colorFormat: " + colorFormat);
       }
     }
   }
 }