Android Screen Capturer.

webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java

Index: webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java
diff --git a/webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java b/webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java
new file mode 100644
index 0000000000000000000000000000000000000000..cfcce1f14e0b8476404881f009d3d78fdd56b78d
--- /dev/null
+++ b/webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java
@@ -0,0 +1,213 @@
+/*
+ *  Copyright 2016 The WebRTC project authors. All Rights Reserved.
+ *
+ *  Use of this source code is governed by a BSD-style license
+ *  that can be found in the LICENSE file in the root of the source
+ *  tree. An additional intellectual property rights grant can be found
+ *  in the file PATENTS.  All contributing project authors may
+ *  be found in the AUTHORS file in the root of the source tree.
+ */
+
+package org.webrtc;
+
+import android.annotation.TargetApi;
+import android.content.Context;
+import android.hardware.display.DisplayManager;
+import android.hardware.display.VirtualDisplay;
+import android.media.projection.MediaProjection;
+import android.util.Log;
+import android.view.Surface;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * An implementation of VideoCapturer to capture the screen content as a video stream.
+ * Capturing is done by {@code MediaProjection} on a {@code SurfaceTexture}. We interact with this
+ * {@code SurfaceTexture} using a {@code SurfaceTextureHelper}.
+ * The {@code SurfaceTextureHelper} is created by the native code and passed to this capturer in
+ * {@code VideoCapturer.initialize()}. On receiving a new frame, this capturer passes it
+ * as a texture to the native code via {@code CapturerObserver.onTextureFrameCaptured()}. This takes
+ * place on the HandlerThread of the given {@code SurfaceTextureHelper}. When done with each frame,
+ * the native code returns the buffer to the  {@code SurfaceTextureHelper} to be used for new
+ * frames. At any time, at most one frame is being processed.
+ *
+ * Note that startCapture(), stopCapture(), and dispose() are called from native code.

[sakal, 2016/08/26 07:21:49]

This is only the case using the old deprecated API that will be removed.

[arsany, 2016/08/30 00:52:15]

Thanks, Sami.
I wasn't aware of API deprecation, would it be OK to land this CL with the old
API assumptions and migrate later when all code is migrated off the deprecated
APIs ?
Also, is there documentation for the change somewhere? What do we need to change
to use the new API? Is the new API ready for use ? is it used in any apps yet?

[sakal, 2016/08/30 07:30:13]

The only real documentation of the API change is the CL implementing the change.
https://codereview.webrtc.org/2127893002/ The changes needed from the app are
described in the CL description. No changes should be needed to the capturer.
See also how AppRTC Demo Android was ported:
https://codereview.webrtc.org/2127893002/patch/180001/190015

The new API is already used by some apps downstream. I tried to remove the old
API yesterday but there were a few remaining that I wasn't aware of. I will
email you more details.

[arsany, 2016/08/30 20:23:17]

Acknowledged.

+ * Normally, a Java application should interact with {@code }VideSource} API, which indirectly calls
+ * these methods on the underlying {@code VideoCapturer}.
+ */
+@TargetApi(21)
+public class ScreenCapturerAndroid implements
+    VideoCapturer, SurfaceTextureHelper.OnTextureFrameAvailableListener {
+
+  private static final int DISPLAY_FLAGS = DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC
+      | DisplayManager.VIRTUAL_DISPLAY_FLAG_PRESENTATION;
+  // DPI for VirtualDisplay, does not seem to matter for us.
+  private static final int VIRTUAL_DISPLAY_DPI = 400;
+
+  private int width;
+  private int height;
+  private VirtualDisplay virtualDisplay;
+  private SurfaceTextureHelper surfaceTextureHelper;
+  private CapturerObserver capturerObserver;
+  private long numCapturedFrames = 0;
+  private MediaProjection mediaProjection;
+  private boolean isDisposed = false;
+  private boolean isCapturing = false;
+
+  /**
+   * Constructs a new Screen Capturer. To avoid distortion, make sure width and height have the
+   * same aspect ratio of the captured screen.
+   *
+   * @param mediaProjection media projection to be used to capture screen
+   * @param width output video width
+   * @param width output video height
+  **/
+  public ScreenCapturerAndroid(MediaProjection mediaProjection, int width, int height) {
+    this.mediaProjection = mediaProjection;
+    this.width = width;
+    this.height = height;
+  }
+
+  /**
+   * Sets the media projection used by this capturer.
+   * This method is useful when you need to pause the capture, and resume again, and make Android
+   * remove the platform cast icon while the video stream is paused (end users can be
+   * confused/suspicious if the app notifies them that screen cast has stopped while Android
+   * still shows the cast icon because the app is still holding the media projection token).
+   *
+   * Call {@code VideoSource.stop()} before calling this method and
+   * {@code VideoSource.restart()} after that.
+   */
+  public void updateMediaProjection(MediaProjection mediaProjection) {
+    if (isCapturing) {
+      throw new RuntimeException("updateMediaProjection can be called only"
+          + " when capturer is stopped");
+    }
+    this.mediaProjection = mediaProjection;
+  }
+
+  private void checkNotDisposed() {
+    if (isDisposed) {
+      throw new RuntimeException("capturer is disposed.");
+    }
+  }
+
+  @Override
+  public synchronized void initialize(
+      final SurfaceTextureHelper surfaceTextureHelper,
+      final Context ignored_applicationContext,
+      final VideoCapturer.CapturerObserver capturerObserver) {
+    checkNotDisposed();
+
+    if (capturerObserver == null) {
+      throw new RuntimeException("capturerObserver not set.");
+    }
+    this.capturerObserver = capturerObserver;
+
+    if (surfaceTextureHelper == null) {
+      throw new RuntimeException("surfaceTextureHelper not set.");
+    }
+    this.surfaceTextureHelper = surfaceTextureHelper;
+  }
+
+  @Override
+  public synchronized List<CameraEnumerationAndroid.CaptureFormat> getSupportedFormats() {
+    List<CameraEnumerationAndroid.CaptureFormat> supportedFormats = new ArrayList<>();
+    supportedFormats.add(new CameraEnumerationAndroid.CaptureFormat(
+        width, height, 1 /* minFrameRate */, 30 /* maxFrameRate */));

[sakal, 2016/08/26 07:21:49]

Maybe minFrameRate 0? maxFrameRate 30 also seems arbitrary, what if the screen
supports 60 fps?

[arsany, 2016/08/30 00:52:15]

Done. Changed to 0, 60. But I am actually not certain about the impact of these
params in our case. Could you please explain briefly how they are used
internally. Thanks.

[sakal, 2016/08/30 07:30:13]

I didn't really say we should change it to 60. I was just saying that maybe it
should be something more eloquent than magic constant. However, if there is no
easy way of for example querying the monitor sync rate, I am okay with it.

[arsany, 2016/08/30 20:23:17]

Removed this altogether.

+    return supportedFormats;
+  }
+
+  // Initially called by native code. This can also be called from native code when the

[sakal, 2016/08/26 07:21:49]

Only the case in the old deprecated API.

[arsany, 2016/08/30 00:52:14]

Acknowledged.

[arsany, 2016/08/30 20:23:17]

Done.

+  // enclosing VideoSource is "restarted" after being stopped.
+  @Override
+  public synchronized void startCapture(
+      final int ignored_width,

[sakal, 2016/08/26 07:21:49]

I would prefer using these variables as the format (this.width = width,
this.height = height). In the old API these will always be width and height
reported by getSupportedFormats. In the new API app will call this method.

[arsany, 2016/08/30 00:52:15]

Within the old API, getSupportedFormats() get called once in the beginning.
We want the app to change format freely (using changeOutputFormat), that's why
we can't use the input params here because they will always be the same as the
first start.
It sounds like the new API will solve this issue. For now, using this old API, I
think we need to continue ignoring those params so that the app can have
control.
The app needs control for example to switch dimensions when the device
orientation changes.

[sakal, 2016/08/30 07:30:13]

Hmm, I see. I would prefer you use the new API though. The old API will be
removed soon.

[magjed_webrtc, 2016/08/30 10:50:54]

Yes, please update to the new API in your client, and then use the input
parameters here. getSupportedFormats() will not even be called in the new API,
and we might remove it as a requirement from the VideoCapturer interface,
because I'm not sure it makes sense for e.g. a ScreenCapturer. So in your
client, you basically need to change:
peerConnectionFactory.createVideoSource(videoCapturer, new MediaConstraints());
to
peerConnectionFactory.createVideoSource(videoCapturer);
and
videoSource.restart();
to
videoCapturer.startCapture(width, height, framerate);
and
videoSource.stop();
to
videoCapturer.stopCapture();

[arsany, 2016/08/30 20:23:17]

Done.

+      final int ignored_height,
+      final int ignored_framerate) {
+    checkNotDisposed();
+    isCapturing = true;
+
+    this.surfaceTextureHelper.getSurfaceTexture().setDefaultBufferSize(width, height);
+    virtualDisplay = mediaProjection.createVirtualDisplay(
+        "WebRTC_ScreenCapture", width, height, VIRTUAL_DISPLAY_DPI,
+        DISPLAY_FLAGS, new Surface(surfaceTextureHelper.getSurfaceTexture()),
+        null /* callback */, null /* callback handler */);
+
+    capturerObserver.onCapturerStarted(true);
+    surfaceTextureHelper.startListening(ScreenCapturerAndroid.this);
+  }
+
+  // Called by native code to pause capturing.

[sakal, 2016/08/26 07:21:48]

Only the case in the old deprecated API.

[arsany, 2016/08/30 00:52:14]

Acknowledged.

[arsany, 2016/08/30 20:23:17]

Done.

+  @Override
+  public synchronized void stopCapture() {
+    checkNotDisposed();
+    isCapturing = false;
+    ThreadUtils.invokeAtFrontUninterruptibly(surfaceTextureHelper.getHandler(), new Runnable() {
+      @Override
+      public void run() {
+        surfaceTextureHelper.stopListening();

[sakal, 2016/08/26 07:21:49]

You should call capturerObserver.onCapturerStopped() after this.

[arsany, 2016/08/30 00:52:15]

Done.

+        virtualDisplay.release();
+      }
+    });
+  }
+
+  // Called from native code to dispose the capturer when the enclosing VideoSource is disposed.

[sakal, 2016/08/26 07:21:48]

Only the case in the old API. In the new API, app should call this.

[arsany, 2016/08/30 00:52:14]

Acknowledged.

[arsany, 2016/08/30 20:23:17]

Done.

+  // The native code calls stopCapture() before calling this method.
+  @Override
+  public synchronized void dispose() {
+    isDisposed = true;
+  }
+
+  @Override
+  public void onOutputFormatRequest(final int width, final int height, final int framerate) {
+    checkNotDisposed();
+    surfaceTextureHelper.getHandler().post(new Runnable() {
+      @Override
+      public void run() {
+        capturerObserver.onOutputFormatRequest(width, height, framerate);
+      }
+    });
+  }
+
+  /**
+   * Changes output video format. This method can be used to scale the output
+   * video, or to change orientation when the captured screen is rotated for example.
+   *
+   * Call {@code VideoSource.stop()}
+   * before calling this method and {@code VideoSource.restart()} after that.
+   *
+   * @param width new output video width
+   * @param height new output video height
+   * @param ignored_framerate ignored
+   */
+  @Override
+  public void changeCaptureFormat(int width, int height, int ignored_framerate) {

[sakal, 2016/08/26 07:21:49]

I think this should match the behavior of camera implementations and allow
calling this while capturing.

[arsany, 2016/08/30 00:52:15]

Done. Agreed. PTAL

+    checkNotDisposed();
+    if (isCapturing) {
+      throw new RuntimeException("changeCaptureFormat can be called only" +
+          " when capturer is stopped");
+    }
+    this.width = width;
+    this.height = height;
+  }
+
+  // This is called on the internal looper thread of {@Code SurfaceTextureHelper}.
+  @Override
+  public void onTextureFrameAvailable(int oesTextureId, float[] transformMatrix, long timestampNs) {
+    numCapturedFrames++;
+    capturerObserver.onTextureFrameCaptured(width, height, oesTextureId, transformMatrix,
+        0 /* rotation */, timestampNs);
+  }
+
+  @Override
+  public boolean isScreencast() {
+    return true;
+  }
+
+  public long getNumCapturedFrames() {
+    return numCapturedFrames;
+  }
+}
+