Autoroller: Support for rolling individual DEPS entries.

Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium:
  those are rolled to the same revision as the Chromium DEPS file of the revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision,
but generally these should be close (and if it passes the bots, we don't really mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_deps.py

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True

Committed: https://crrev.com/38b6dbc7939cd8b4339592d12b21c345ae29504d
Cr-Commit-Position: refs/heads/master@{#15570}

[kjellander_webrtc, 2016-12-13 09:13:02]

Patchset #1 (id:1) has been deleted

[kjellander_webrtc, 2016-12-13 09:13:15]

kjellander@webrtc.org changed reviewers:
+ phoglund@webrtc.org

[kjellander_webrtc, 2016-12-13 09:13:40]

Description was changed from

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
==========

to

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True
==========

[phoglund, 2016-12-13 09:57:02]

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
File tools/autoroller/roll_chromium_revision.py (right):

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:23: # Skip these dependencies (list
without solution name prefix).
It's not entirely clear what "skipping" means in this context. Maybe
DONT_AUTOROLL_THESE is a better name? :)

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:224: as the new_cr_deps dict, if
available. Otherwise HEAD unless the deps is
I find this confusing. When exactly is something rolled to HEAD? If it isn't
specified in the new_cr_deps dict? What does it mean for it not to be specified
in the new_cr_deps dict? Does that mean new_cr_deps specifies a subset of all
possible deps, and the ones not in the subset are rolled to head? In that case,
what is the list of all possible deps?

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:226: 2. Entried present in the
setup_links.py file. If the dir has changed between
Nit: entries

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/unittest...
File tools/autoroller/unittests/roll_chromium_revision_test.py (right):

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/unittest...
tools/autoroller/unittests/roll_chromium_revision_test.py:145:
self.assertEquals(changed_deps[0].new_rev, BUILDTOOLS_NEW_REV)
Add an explicit test for winsdk and gflags to ensure they're skipped.

[kjellander_webrtc, 2016-12-13 10:23:59]

Description was changed from

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True
==========

to

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium: 
  those are rolled to the same revision as the Chromium DEPS file of the
revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision, 
but generally these should be close (and if it passes the bots, we don't really
mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True
==========

[kjellander_webrtc, 2016-12-13 10:25:00]

PTAL

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
File tools/autoroller/roll_chromium_revision.py (right):

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:23: # Skip these dependencies (list
without solution name prefix).
On 2016/12/13 09:57:02, phoglund wrote:
> It's not entirely clear what "skipping" means in this context. Maybe
> DONT_AUTOROLL_THESE is a better name? :)

Done.

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:224: as the new_cr_deps dict, if
available. Otherwise HEAD unless the deps is
On 2016/12/13 09:57:02, phoglund wrote:
> I find this confusing. When exactly is something rolled to HEAD?

I clarified it now.

> If it isn't specified in the new_cr_deps dict? What does it mean for it not to
be specified
> in the new_cr_deps dict? Does that mean new_cr_deps specifies a subset of all
> possible deps, and the ones not in the subset are rolled to head? In that
case,
> what is the list of all possible deps?

The new_cr_deps dict contains a parsed Chromium DEPS file for the revision the
autoroller is trying to roll to (i.e. the --revision flag passed to the script).

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/roll_chr...
tools/autoroller/roll_chromium_revision.py:226: 2. Entried present in the
setup_links.py file. If the dir has changed between
On 2016/12/13 09:57:02, phoglund wrote:
> Nit: entries

Done.

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/unittest...
File tools/autoroller/unittests/roll_chromium_revision_test.py (right):

https://codereview.webrtc.org/2570603003/diff/20001/tools/autoroller/unittest...
tools/autoroller/unittests/roll_chromium_revision_test.py:145:
self.assertEquals(changed_deps[0].new_rev, BUILDTOOLS_NEW_REV)
On 2016/12/13 09:57:02, phoglund wrote:
> Add an explicit test for winsdk and gflags to ensure they're skipped.

By checking the number of changed deps, I already verify that they aren't rolled
(in the  testCalculateChangedDepsProper test).

[phoglund, 2016-12-13 10:40:57]

lgtm

[kjellander_webrtc, 2016-12-13 11:31:55]

The CQ bit was checked by kjellander@webrtc.org

[commit-bot: I haz the power, 2016-12-13 11:31:59]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.webrtc.org/...

[commit-bot: I haz the power, 2016-12-13 11:35:23]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1481628715143580,
"parent_rev": "aad667b4eb1df5e61d003e2addbb28bcfe11ff5c", "commit_rev":
"c6d3ab5be2f3ace993201e7eb034863cd8b97d66"}

[commit-bot: I haz the power, 2016-12-13 11:35:26]

Description was changed from

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium: 
  those are rolled to the same revision as the Chromium DEPS file of the
revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision, 
but generally these should be close (and if it passes the bots, we don't really
mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True
==========

to

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium: 
  those are rolled to the same revision as the Chromium DEPS file of the
revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision, 
but generally these should be close (and if it passes the bots, we don't really
mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True

Review-Url: https://codereview.webrtc.org/2570603003
==========

[commit-bot: I haz the power, 2016-12-13 11:35:26]

Committed patchset #2 (id:40001)

[commit-bot: I haz the power, 2016-12-13 11:35:33]

Description was changed from

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium: 
  those are rolled to the same revision as the Chromium DEPS file of the
revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision, 
but generally these should be close (and if it passes the bots, we don't really
mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True

Review-Url: https://codereview.webrtc.org/2570603003
==========

to

==========
Autoroller: Support for rolling individual DEPS entries.

Add functionality for automatic rolling of individual DEPS
entries. This will make it possible to move away from the
links created by setup_links.py to real DEPS entries.

There are two kinds of such deps we intend to use:
1. Third party dependencies used by both WebRTC and Chromium:
  those are rolled to the same revision as the Chromium DEPS file of the
revision passed
  to the script
2. Chromium sub-directories needed for WebRTC (mainly for //build and BUILD.gn
   files of third_party deps): those are rolled to the HEAD revision
Notice that the latter kind could be rolled ahead of the chromium_revision,
but generally these should be close (and if it passes the bots, we don't really
mind).

The new functionality can coexist with the old one, to
enable a smooth transition (not everything needs to change at once).

Some of the updating logic was inspired by
https://cs.chromium.org/chromium/build/scripts/slave/recipes/v8/auto_roll_v8_...

Add extensive tests for the logic of figuring out deps changes,
including mocking git ls-remote call to keep the tests hermetic.

BUG=webrtc:5006
NOTRY=True

Committed: https://crrev.com/38b6dbc7939cd8b4339592d12b21c345ae29504d
Cr-Commit-Position: refs/heads/master@{#15570}
==========

[commit-bot: I haz the power, 2016-12-13 11:35:34]

Patchset 2 (id:??) landed as
https://crrev.com/38b6dbc7939cd8b4339592d12b21c345ae29504d
Cr-Commit-Position: refs/heads/master@{#15570}