Add support to Capture to a texture in VideoCapturerAndroid

Add support to Capture to a texture instead of memory.

This adds support for capturing to a texture in the Java part of VideoCapturerAndroid.
After this cl, the C++ also needs modification.

https://codereview.webrtc.org/1375953002/ contains the idea and have a working version where textures can be rendered in local preview.

BUG=webrtc:4993
R=magjed@webrtc.org

Committed: https://crrev.com/3d06eca5e3310ea068625a9c95d492a16854f421
Cr-Commit-Position: refs/heads/master@{#10213}

[perkj_webrtc, 2015-10-06 06:35:08]

perkj@webrtc.org changed reviewers:
+ magjed@webrtc.org

[perkj_webrtc, 2015-10-06 06:35:57]

Can you please review? I will land the C++ part independently once this and your
last decoder patch have landed.

[magjed_webrtc, 2015-10-06 07:43:46]

The CL looks good in general.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
File talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTest.java
(right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTest.java:80:
public void testCreateAndRelease() throws Exception {
All these 'throws Exception' are probably 'throws InterruptedException' or not
needed at all.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
File
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTestFixtures.java
(right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTestFixtures.java:299:
final boolean identicalResolution = (observer.frameWidth() == format.width &&
&& should be on the next line.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
File talk/app/webrtc/java/android/org/webrtc/RendererCommon.java (right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/RendererCommon.java:70: // Matrix with
transform y' = 1 - y.
You can add a new matrix here:
// Matrix with transform x' = 1 - x.
public static final float[] horizontalFlipMatrix() {
  return new float[] {
      -1, 0, 0, 0,
       0, 1, 0, 0,
       0, 0, 1, 0,
       1, 0, 0, 1};
}

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/RendererCommon.java:103: final float[]
mirrorMatrix = new float[16];
If you have horizontalFlipMatrix(), you can write this as:
public static float[] mirrorTextureMatrixHorizontally(float[] textureMatrix) {
  return multiplyMatrices(textureMatrix, horizontalFlipMatrix());
}
You can even remove this function if you want and inline that one-liner in
VideoCapturerAndroid.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
File talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java (right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:179: for (Long
ts : timeStampsNs) {
nit: You can iterate with 'long ts'

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:293: public
boolean captureToSurfaceTexture() {
This function name sounds like it will change something. Maybe rename to
isCapturingToTexture() or something like that? ditto variable
|captureToTexture|.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:669: // Undo
the mirror that the OS "helps" us with.
This mirroring for front cameras is described here:
http://developer.android.com/reference/android/hardware/Camera.html#setDispla...:
"Note that preview display of front-facing cameras is flipped horizontally
before the rotation, that is, the image is reflected along the central vertical
axis of the camera sensor. So the users can see themselves as looking into a
mirror.

This does not affect the order of byte array passed in onPreviewFrame(byte[],
Camera)"
Add a reference if you want, it's not obvious how Camera works and why this
mirror is needed.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:795: private
String pendingFramesTimeStamps() {
Why is this function needed, can't you use
cameraStatistics.pendingFramesTimeStamps()?

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:811: //
Delivers a captured frame. Called on a Java thread owned by
I think we need to make a distinction between ByteBuffer frames and and texture
frames. So this function should probably be renamed to something like
"onByteBufferFrameCaptured()".

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:818: abstract
void onTextureFrameAvailable(
Change name to OnTextureFrameCaptured() to be consistent with the function
above? Also I think you can remove the comment about |oesTextureId| being
updated. It's true, and |oesTextureId| does not change, but I think clients
should not be aware of this. The risk is just that they store away
|oesTextureId| like VideoRendererGui did.

[perkj_webrtc, 2015-10-06 09:25:36]

PTAL

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
File talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTest.java
(right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTest.java:80:
public void testCreateAndRelease() throws Exception {
On 2015/10/06 07:43:45, magjed_webrtc wrote:
> All these 'throws Exception' are probably 'throws InterruptedException' or not
> needed at all.

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
File
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTestFixtures.java
(right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/androidtests/...
talk/app/webrtc/androidtests/src/org/webrtc/VideoCapturerAndroidTestFixtures.java:299:
final boolean identicalResolution = (observer.frameWidth() == format.width &&
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> && should be on the next line.

in java.....

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
File talk/app/webrtc/java/android/org/webrtc/RendererCommon.java (right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/RendererCommon.java:70: // Matrix with
transform y' = 1 - y.
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> You can add a new matrix here:
> // Matrix with transform x' = 1 - x.
> public static final float[] horizontalFlipMatrix() {
>   return new float[] {
>       -1, 0, 0, 0,
>        0, 1, 0, 0,
>        0, 0, 1, 0,
>        1, 0, 0, 1};
> }

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/RendererCommon.java:103: final float[]
mirrorMatrix = new float[16];
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> If you have horizontalFlipMatrix(), you can write this as:
> public static float[] mirrorTextureMatrixHorizontally(float[] textureMatrix) {
>   return multiplyMatrices(textureMatrix, horizontalFlipMatrix());
> }
> You can even remove this function if you want and inline that one-liner in
> VideoCapturerAndroid.

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
File talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java (right):

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:179: for (Long
ts : timeStampsNs) {
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> nit: You can iterate with 'long ts'

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:293: public
boolean captureToSurfaceTexture() {
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> This function name sounds like it will change something. Maybe rename to
> isCapturingToTexture() or something like that? ditto variable
> |captureToTexture|.

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:669: // Undo
the mirror that the OS "helps" us with.
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> This mirroring for front cameras is described here:
>
http://developer.android.com/reference/android/hardware/Camera.html#setDispla...:
> "Note that preview display of front-facing cameras is flipped horizontally
> before the rotation, that is, the image is reflected along the central
vertical
> axis of the camera sensor. So the users can see themselves as looking into a
> mirror.
> 
> This does not affect the order of byte array passed in onPreviewFrame(byte[],
> Camera)"
> Add a reference if you want, it's not obvious how Camera works and why this
> mirror is needed.

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:795: private
String pendingFramesTimeStamps() {
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> Why is this function needed, can't you use
> cameraStatistics.pendingFramesTimeStamps()?

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:811: //
Delivers a captured frame. Called on a Java thread owned by
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> I think we need to make a distinction between ByteBuffer frames and and
texture
> frames. So this function should probably be renamed to something like
> "onByteBufferFrameCaptured()".

Done.

https://codereview.webrtc.org/1383413002/diff/1/talk/app/webrtc/java/android/...
talk/app/webrtc/java/android/org/webrtc/VideoCapturerAndroid.java:818: abstract
void onTextureFrameAvailable(
On 2015/10/06 07:43:46, magjed_webrtc wrote:
> Change name to OnTextureFrameCaptured() to be consistent with the function
> above? Also I think you can remove the comment about |oesTextureId| being
> updated. It's true, and |oesTextureId| does not change, but I think clients
> should not be aware of this. The risk is just that they store away
> |oesTextureId| like VideoRendererGui did.

Done.

[magjed_webrtc, 2015-10-06 14:04:55]

lgtm

[perkj_webrtc, 2015-10-08 10:53:52]

Committed patchset #3 (id:40001) manually as
3d06eca5e3310ea068625a9c95d492a16854f421 (presubmit successful).

[commit-bot: I haz the power, 2015-10-08 10:53:53]

Patchset 3 (id:??) landed as
https://crrev.com/3d06eca5e3310ea068625a9c95d492a16854f421
Cr-Commit-Position: refs/heads/master@{#10213}