Rename tools-webrtc -> tools_webrtc

tools-webrtc/mb/docs/design_spec.md

-# The MB (Meta-Build wrapper) design spec
-
-[TOC]
-
-## Intro
-
-MB is intended to address two major aspects of the GYP -> GN transition
-for Chromium:
-
-1. "bot toggling" - make it so that we can easily flip a given bot
-   back and forth between GN and GYP.
-
-2. "bot configuration" - provide a single source of truth for all of
-   the different configurations (os/arch/`gyp_define` combinations) of
-   Chromium that are supported.
-
-MB must handle at least the `gen` and `analyze` steps on the bots, i.e.,
-we need to wrap both the `gyp_chromium` invocation to generate the
-Ninja files, and the `analyze` step that takes a list of modified files
-and a list of targets to build and returns which targets are affected by
-the files.
-
-For more information on how to actually use MB, see
-[the user guide](user_guide.md).
-
-## Design
-
-MB is intended to be as simple as possible, and to defer as much work as
-possible to GN or GYP. It should live as a very simple Python wrapper
-that offers little in the way of surprises.
-
-### Command line
-
-It is structured as a single binary that supports a list of subcommands:
-
-* `mb gen -c linux_rel_bot //out/Release`
-* `mb analyze -m tryserver.chromium.linux -b linux_rel /tmp/input.json /tmp/output.json`
-
-### Configurations
-
-`mb` will first look for a bot config file in a set of different locations
-(initially just in //ios/build/bots). Bot config files are JSON files that
-contain keys for 'GYP_DEFINES' (a list of strings that will be joined together
-with spaces and passed to GYP, or a dict that will be similarly converted),
-'gn_args' (a list of strings that will be joined together), and an
-'mb_type' field that says whether to use GN or GYP. Bot config files
-require the full list of settings to be given explicitly.
-
-If no matching bot config file is found, `mb` looks in the
-`//tools/mb/mb_config.pyl` config file to determine whether to use GYP or GN
-for a particular build directory, and what set of flags (`GYP_DEFINES` or `gn
-args`) to use.
-
-A config can either be specified directly (useful for testing) or by specifying
-the master name and builder name (useful on the bots so that they do not need
-to specify a config directly and can be hidden from the details).
-
-See the [user guide](user_guide.md#mb_config.pyl) for details.
-
-### Handling the analyze step
-
-The interface to `mb analyze` is described in the
-[user\_guide](user_guide.md#mb_analyze).
-
-The way analyze works can be subtle and complicated (see below).
-
-Since the interface basically mirrors the way the "analyze" step on the bots
-invokes `gyp_chromium` today, when the config is found to be a gyp config,
-the arguments are passed straight through.
-
-It implements the equivalent functionality in GN by calling `gn refs
-[list of files] --type=executable --all --as=output` and filtering the
-output to match the list of targets.
-
-## Analyze
-
-The goal of the `analyze` step is to speed up the cycle time of the try servers
-by only building and running the tests affected by the files in a patch, rather
-than everything that might be out of date. Doing this ends up being tricky.
-
-We start with the following requirements and observations:
-
-* In an ideal (un-resource-constrained) world, we would build and test
-  everything that a patch affected on every patch. This does not
-  necessarily mean that we would build 'all' on every patch (see below).
-
-* In the real world, however, we do not have an infinite number of machines,
-  and try jobs are not infinitely fast, so we need to balance the desire
-  to get maximum test coverage against the desire to have reasonable cycle
-  times, given the number of machines we have.
-
-* Also, since we run most try jobs against tip-of-tree Chromium, by
-  the time one job completes on the bot, new patches have probably landed,
-  rendering the build out of date.
-
-* This means that the next try job may have to do a build that is out of
-  date due to a combination of files affected by a given patch, and files
-  affected for unrelated reasons. We want to rebuild and test only the
-  targets affected by the patch, so that we don't blame or punish the
-  patch author for unrelated changes.
-
-So:
-
-1. We need a way to indicate which changed files we care about and which
-   we don't (the affected files of a patch).
-
-2. We need to know which tests we might potentially want to run, and how
-   those are mapped onto build targets. For some kinds of tests (like
-   GTest-based tests), the mapping is 1:1 - if you want to run base_unittests,
-   you need to build base_unittests. For others (like the telemetry and
-   layout tests), you might need to build several executables in order to
-   run the tests, and that mapping might best be captured by a *meta*
-   target (a GN group or a GYP 'none' target like `webkit_tests`) that
-   depends on the right list of files. Because the GN and GYP files know
-   nothing about test steps, we have to have some way of mapping back
-   and forth between test steps and build targets. That mapping
-   is *not* currently available to MB (or GN or GYP), and so we have to 
-   enough information to make it possible for the caller to do the mapping.
-
-3. We might also want to know when test targets are affected by data files
-   that aren't compiled (python scripts, or the layout tests themselves).
-   There's no good way to do this in GYP, but GN supports this.
-
-4. We also want to ensure that particular targets still compile even if they
-   are not actually tested; consider testing the installers themselves, or
-   targets that don't yet have good test coverage. We might want to use meta
-   targets for this purpose as well.
-
-5. However, for some meta targets, we don't necessarily want to rebuild the
-   meta target itself, perhaps just the dependencies of the meta target that
-   are affected by the patch. For example, if you have a meta target like
-   `blink_tests` that might depend on ten different test binaries. If a patch
-   only affects one of them (say `wtf_unittests`), you don't want to
-   build `blink_tests`, because that might actually also build the other nine
-   targets.  In other words, some meta targets are *prunable*.
-
-6. As noted above, in the ideal case we actually have enough resources and
-   things are fast enough that we can afford to build everything affected by a
-   patch, but listing every possible target explicitly would be painful. The
-   GYP and GN Ninja generators provide an 'all' target that captures (nearly,
-   see [crbug.com/503241](crbug.com/503241)) everything, but unfortunately
-   neither GN nor GYP actually represents 'all' as a meta target in the build
-   graph, so we will need to write code to handle that specially.
-
-7. In some cases, we will not be able to correctly analyze the build graph to
-   determine the impact of a patch, and need to bail out (e.g,. if you change a
-   build file itself, it may not be easy to tell how that affects the graph).
-   In that case we should simply build and run everything.
-
-The interaction between 2) and 5) means that we need to treat meta targets
-two different ways, and so we need to know which targets should be
-pruned in the sense of 5) and which targets should be returned unchanged
-so that we can map them back to the appropriate tests.
-
-So, we need three things as input:
-
-* `files`: the list of files in the patch
-* `test_targets`: the list of ninja targets which, if affected by a patch,
-  should be reported back so that we can map them back to the appropriate
-  tests to run. Any meta targets in this list should *not* be pruned.
-* `additional_compile_targets`: the list of ninja targets we wish to compile
-  *in addition to* the list in `test_targets`. Any meta targets
-  present in this list should be pruned (we don't need to return the
-  meta targets because they aren't mapped back to tests, and we don't want
-  to build them because we might build too much).
-
-We can then return two lists as output:
-
-* `compile_targets`, which is a list of pruned targets to be
-  passed to Ninja to build. It is acceptable to replace a list of
-  pruned targets by a meta target if it turns out that all of the
-  dependendencies of the target are affected by the patch (i.e.,
-  all ten binaries that blink_tests depends on), but doing so is
-  not required.
-* `test_targets`, which is a list of unpruned targets to be mapped
-  back to determine which tests to run.
-
-There may be substantial overlap between the two lists, but there is
-no guarantee that one is a subset of the other and the two cannot be
-used interchangeably or merged together without losing information and
-causing the wrong thing to happen.
-
-The implementation is responsible for recognizing 'all' as a magic string
-and mapping it onto the list of all root nodes in the build graph.
-
-There may be files listed in the input that don't actually exist in the build
-graph: this could be either the result of an error (the file should be in the
-build graph, but isn't), or perfectly fine (the file doesn't affect the build
-graph at all). We can't tell these two apart, so we should ignore missing
-files.
-
-There may be targets listed in the input that don't exist in the build
-graph; unlike missing files, this can only indicate a configuration error,
-and so we should return which targets are missing so the caller can
-treat this as an error, if so desired.
-
-Any of the three inputs may be an empty list:
-
-* It normally doesn't make sense to call analyze at all if no files
-  were modified, but in rare cases we can hit a race where we try to
-  test a patch after it has already been committed, in which case
-  the list of modified files is empty. We should return 'no dependency'
-  in that case.
-
-* Passing an empty list for one or the other of test_targets and
-  additional_compile_targets is perfectly sensible: in the former case,
-  it can indicate that you don't want to run any tests, and in the latter,
-  it can indicate that you don't want to do build anything else in
-  addition to the test targets.
-
-* It doesn't make sense to call analyze if you don't want to compile
-  anything at all, so passing [] for both test_targets and 
-  additional_compile_targets should probably return an error.
-
-In the output case, an empty list indicates that there was nothing to
-build, or that there were no affected test targets as appropriate.
-
-Note that passing no arguments to Ninja is equivalent to passing
-`all` to Ninja (at least given how GN and GYP work); however, we
-don't want to take advantage of this in most cases because we don't
-actually want to build every out of date target, only the targets
-potentially affected by the files. One could try to indicate
-to analyze that we wanted to use no arguments instead of an empty
-list, but using the existing fields for this seems fragile and/or
-confusing, and adding a new field for this seems unwarranted at this time.
-
-There is an "error" field in case something goes wrong (like the
-empty file list case, above, or an internal error in MB/GYP/GN). The
-analyze code should also return an error code to the shell if appropriate
-to indicate that the command failed.
-
-In the case where build files themselves are modified and analyze may
-not be able to determine a correct answer (point 7 above, where we return
-"Found dependency (all)"), we should also return the `test_targets` unmodified
-and return the union of `test_targets` and `additional_compile_targets` for
-`compile_targets`, to avoid confusion.
-
-### Examples
-
-Continuing the example given above, suppose we have the following build
-graph:
-
-* `blink_tests` is a meta target that depends on `webkit_unit_tests`,
-  `wtf_unittests`, and `webkit_tests` and represents all of the targets
-  needed to fully test Blink. Each of those is a separate test step.
-* `webkit_tests` is also a meta target; it depends on `content_shell`
-  and `image_diff`.
-* `base_unittests` is a separate test binary.
-* `wtf_unittests` depends on `Assertions.cpp` and `AssertionsTest.cpp`.
-* `webkit_unit_tests` depends on `WebNode.cpp` and `WebNodeTest.cpp`.
-* `content_shell` depends on `WebNode.cpp` and `Assertions.cpp`.
-* `base_unittests` depends on `logging.cc` and `logging_unittest.cc`.
-
-#### Example 1
-
-We wish to run 'wtf_unittests' and 'webkit_tests' on a bot, but not
-compile any additional targets.
-
-If a patch touches WebNode.cpp, then analyze gets as input:
-
-    {
-      "files": ["WebNode.cpp"],
-      "test_targets": ["wtf_unittests", "webkit_tests"],
-      "additional_compile_targets": []
-    }
-
-and should return as output:
-
-    {
-      "status": "Found dependency",
-      "compile_targets": ["webkit_unit_tests"],
-      "test_targets": ["webkit_tests"]
-    }
-
-Note how `webkit_tests` was pruned in compile_targets but not in test_targets.
-
-#### Example 2
-
-Using the same patch as Example 1, assume we wish to run only `wtf_unittests`,
-but additionally build everything needed to test Blink (`blink_tests`):
-
-We pass as input:
-
-    {
-      "files": ["WebNode.cpp"],
-      "test_targets": ["wtf_unittests"],
-      "additional_compile_targets": ["blink_tests"]
-    }
-
-And should get as output:
-
-    {
-      "status": "Found dependency",
-      "compile_targets": ["webkit_unit_tests"],
-      "test_targets": []
-    }
-
-Here `blink_tests` was pruned in the output compile_targets, and
-test_targets was empty, since blink_tests was not listed in the input
-test_targets.
-
-#### Example 3
-
-Build everything, but do not run any tests.
-
-Input:
-
-    {
-      "files": ["WebNode.cpp"],
-      "test_targets": [],
-      "additional_compile_targets": ["all"]
-    }
-
-Output:
-
-    {
-      "status": "Found dependency",
-      "compile_targets": ["webkit_unit_tests", "content_shell"],
-      "test_targets": []
-    }
-
-#### Example 4
-
-Same as Example 2, but a build file was modified instead of a source file.
-
-Input:
-
-    {
-      "files": ["BUILD.gn"],
-      "test_targets": ["wtf_unittests"],
-      "additional_compile_targets": ["blink_tests"]
-    }
-
-Output:
-
-    {
-      "status": "Found dependency (all)",
-      "compile_targets": ["webkit_unit_tests", "wtf_unittests"],
-      "test_targets": ["wtf_unittests"]
-    }
-
-test_targets was returned unchanged, compile_targets was pruned.
-
-## Random Requirements and Rationale
-
-This section is collection of semi-organized notes on why MB is the way
-it is ...
-
-### in-tree or out-of-tree
-
-The first issue is whether or not this should exist as a script in
-Chromium at all; an alternative would be to simply change the bot
-configurations to know whether to use GYP or GN, and which flags to
-pass.
-
-That would certainly work, but experience over the past two years
-suggests a few things:
-
-  * we should push as much logic as we can into the source repositories
-    so that they can be versioned and changed atomically with changes to
-    the product code; having to coordinate changes between src/ and
-    build/ is at best annoying and can lead to weird errors.
-  * the infra team would really like to move to providing
-    product-independent services (i.e., not have to do one thing for
-    Chromium, another for NaCl, a third for V8, etc.).
-  * we found that during the SVN->GIT migration the ability to flip bot
-    configurations between the two via changes to a file in chromium
-    was very useful.
-
-All of this suggests that the interface between bots and Chromium should
-be a simple one, hiding as much of the chromium logic as possible.
-
-### Why not have MB be smarter about de-duping flags?
-
-This just adds complexity to the MB implementation, and duplicates logic
-that GYP and GN already have to support anyway; in particular, it might
-require MB to know how to parse GYP and GN values. The belief is that
-if MB does *not* do this, it will lead to fewer surprises.
-
-It will not be hard to change this if need be.
-
-### Integration w/ gclient runhooks
-
-On the bots, we will disable `gyp_chromium` as part of runhooks (using
-`GYP_CHROMIUM_NO_ACTION=1`), so that mb shows up as a separate step.
-
-At the moment, we expect most developers to either continue to use
-`gyp_chromium` in runhooks or to disable at as above if they have no
-use for GYP at all. We may revisit how this works once we encourage more
-people to use GN full-time (i.e., we might take `gyp_chromium` out of
-runhooks altogether).
-
-### Config per flag set or config per (os/arch/flag set)?
-
-Currently, mb_config.pyl does not specify the host_os, target_os, host_cpu, or
-target_cpu values for every config that Chromium runs on, it only specifies
-them for when the values need to be explicitly set on the command line.
-
-Instead, we have one config per unique combination of flags only.
-
-In other words, rather than having `linux_rel_bot`, `win_rel_bot`, and
-`mac_rel_bot`, we just have `rel_bot`.
-
-This design allows us to determine easily all of the different sets
-of flags that we need to support, but *not* which flags are used on which
-host/target combinations.
-
-It may be that we should really track the latter. Doing so is just a
-config file change, however.
-
-### Non-goals
-
-* MB is not intended to replace direct invocation of GN or GYP for
-  complicated build scenarios (aka ChromeOS), where multiple flags need
-  to be set to user-defined paths for specific toolchains (e.g., where
-  ChromeOS needs to specify specific board types and compilers).
-
-* MB is not intended at this time to be something developers use frequently,
-  or to add a lot of features to. We hope to be able to get rid of it once
-  the GYP->GN migration is done, and so we should not add things for
-  developers that can't easily be added to GN itself.
-
-* MB is not intended to replace the
-  [CR tool](https://code.google.com/p/chromium/wiki/CRUserManual). Not
-  only is it only intended to replace the gyp\_chromium part of `'gclient
-  runhooks'`, it is not really meant as a developer-facing tool.