DevTools: Add support for installing node and running eslint over devtools.

DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046
Committed: https://crrev.com/56a90381f4d44cefeb5edb83a6227ebfa02fb885
Cr-Commit-Position: refs/heads/master@{#437431}

[chenwilliam, 2016-11-09 19:27:21]

I've updated your original CL Dirk by filling in the BUILD.gn and making some
minor fixes. After working on it, I've a few questions:

- Is there any way of "globbing" files in BUILD.gn? I'm wondering because eslint
has over 4000 files (many of them are its dependencies)
- Where do I include the actual command to run eslint? (e.g. the python script
at chromium/src/third_party/WebKit/Source/devtools/scripts/lint_javascript.py)
- In addition to setting the environment flag INSTALL_NODE_FOR_DEVTOOLS on the
buildbots, is there another step to enable the test?

Thanks for your help. In addition to this CL, I'm planning on opening a separate
CL for our Closure check so we can enable that one first (b/c it's simpler and
doesn't involve node & npm modules).

[iannucci, 2016-11-09 23:58:16]

as a refined version of the previous CL, this also lgtm

[Dirk Pranke, 2016-11-10 01:23:12]

I didn't get to this yet today (too much sheriffing and other things going on)
but will look at it either later tonight or (more likely) tomorrow morning.

[Dirk Pranke, 2016-11-11 02:11:20]

> - Is there any way of "globbing" files in BUILD.gn?  I'm wondering
> because eslint has over 4000 files (many of them are its 
> dependencies)

No, there's no globbing (there are reasons for this but I won't go into them at
the moment). 

The best thing to do is probably to ensure that the eslint and its dependencies
are all in a single directory, and then list that directory as a `data = [
".../eslint/" ]` in the BUILD.gn file. The trailing slash will ensure that
everything in the directory is included in the isolate. This is not a perfect
solution and hopefully the cipd-based thing well end up being better.

> - In addition to setting the environment flag INSTALL_NODE_FOR_DEVTOOLS on the
> buildbots, is there another step to enable the test?

The change in chromium.linux.json is what enables the test. The addition of the
env var on the builder is what will make the test actually pass.

You might start to test this by (a) running it as-is and see that you get the
failure you'd expect if node is missing and then (b) uploading a patchset that
runs the hook unconditionally (doesn't check for the env var) and ensure that it
works.

https://codereview.chromium.org/2486903002/diff/1/testing/buildbot/gn_isolate...
File testing/buildbot/gn_isolate_map.pyl (right):

https://codereview.chromium.org/2486903002/diff/1/testing/buildbot/gn_isolate...
testing/buildbot/gn_isolate_map.pyl:382: "script":
"//testing/scripts/run_devtools_eslint.py",
I think you want to add the path to the actual script you want to run in the
"args" field of this struct. See line 762 for an example.

https://codereview.chromium.org/2486903002/diff/1/testing/buildbot/gn_isolate...
testing/buildbot/gn_isolate_map.pyl:762: "args": [
i.e., here.

https://codereview.chromium.org/2486903002/diff/1/third_party/WebKit/Source/d...
File third_party/WebKit/Source/devtools/scripts/local_node/node.py (right):

https://codereview.chromium.org/2486903002/diff/1/third_party/WebKit/Source/d...
third_party/WebKit/Source/devtools/scripts/local_node/node.py:108: 
I'd rewrite this section to use argparse, e.g.:

    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--running-as-hook', action='store_true')
    args, rest_args = parser.parse_known_args()
    if args.running_as_hooks and ...:

and then git rid of line 121 and pass rest_args on line 122 instead. Maybe a
line or two more of code, but a little bit more idiomatic python.

[chenwilliam, 2016-11-15 04:56:27]

Patchset #8 (id:140001) has been deleted

[chenwilliam, 2016-12-06 18:22:09]

Dirk - please take another look. 

I uploaded a test patchset without the env check to make sure eslint is running
properly. I think the last step is to enable the env variable
INSTALL_NODE_FOR_DEVTOOLS on the buildbots.

https://codereview.chromium.org/2486903002/diff/1/testing/buildbot/gn_isolate...
File testing/buildbot/gn_isolate_map.pyl (right):

https://codereview.chromium.org/2486903002/diff/1/testing/buildbot/gn_isolate...
testing/buildbot/gn_isolate_map.pyl:382: "script":
"//testing/scripts/run_devtools_eslint.py",
On 2016/11/11 02:11:20, Dirk Pranke wrote:
> I think you want to add the path to the actual script you want to run in the
> "args" field of this struct. See line 762 for an example.

Done.

https://codereview.chromium.org/2486903002/diff/1/third_party/WebKit/Source/d...
File third_party/WebKit/Source/devtools/scripts/local_node/node.py (right):

https://codereview.chromium.org/2486903002/diff/1/third_party/WebKit/Source/d...
third_party/WebKit/Source/devtools/scripts/local_node/node.py:108: 
On 2016/11/11 02:11:20, Dirk Pranke wrote:
> I'd rewrite this section to use argparse, e.g.:
> 
>     import argparse
>     parser = argparse.ArgumentParser()
>     parser.add_argument('--running-as-hook', action='store_true')
>     args, rest_args = parser.parse_known_args()
>     if args.running_as_hooks and ...:
> 
> and then git rid of line 121 and pass rest_args on line 122 instead. Maybe a
> line or two more of code, but a little bit more idiomatic python.

Done.

[Dirk Pranke, 2016-12-07 03:10:40]

[email protected] changed reviewers:
+ [email protected]

[Dirk Pranke, 2016-12-07 03:10:41]

this mostly lgtm.

I think we probably don't need a new wrapper script, though. 

@kbr, can you take a look at this and see if my understanding of the
isolated_scripts interface(s) is correct?

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
File testing/scripts/run_devtools_eslint.py (right):

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
testing/scripts/run_devtools_eslint.py:54: json.dump(['devtools_eslint'],
args.output)
Is line 54 the only difference between this file and
run_devtools_closure_compile.py ?

If so, I'm wondering if we can just use a single script for both? I *think* that
if you set the override_compile_targets in the *.json files, it should probably
work.

[Ken Russell (switch to Gerrit), 2016-12-07 06:50:25]

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
File testing/scripts/run_devtools_eslint.py (right):

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
testing/scripts/run_devtools_eslint.py:54: json.dump(['devtools_eslint'],
args.output)
On 2016/12/07 03:10:41, Dirk Pranke wrote:
> Is line 54 the only difference between this file and
> run_devtools_closure_compile.py ?
> 
> If so, I'm wondering if we can just use a single script for both? I *think*
that
> if you set the override_compile_targets in the *.json files, it should
probably
> work.

I'm 99.99% sure (unless someone rewrote the recipe-side code; I'm looking
through it now but can't verify 100%) that this is not actually used by the
recipes during the build step. Yes, it should be possible to reuse the same
script. There should be two GN targets, both of which should include the same
script as a data dependency, with their own unique dependencies as well. Note
that the isolated script run_gpu_integration_test_as_googletest.py is used to
run many different steps on the bots. I suggest using it as an example.

In addition to changing this code to dump [''], I would add a comment indicating
that recipes do not call this entry point for isolated scripts.

[chenwilliam, 2016-12-08 02:52:26]

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
File testing/scripts/run_devtools_eslint.py (right):

https://codereview.chromium.org/2486903002/diff/340001/testing/scripts/run_de...
testing/scripts/run_devtools_eslint.py:54: json.dump(['devtools_eslint'],
args.output)
On 2016/12/07 06:50:25, Ken Russell wrote:
> On 2016/12/07 03:10:41, Dirk Pranke wrote:
> > Is line 54 the only difference between this file and
> > run_devtools_closure_compile.py ?
> > 
> > If so, I'm wondering if we can just use a single script for both? I *think*
> that
> > if you set the override_compile_targets in the *.json files, it should
> probably
> > work.
> 
> I'm 99.99% sure (unless someone rewrote the recipe-side code; I'm looking
> through it now but can't verify 100%) that this is not actually used by the
> recipes during the build step. Yes, it should be possible to reuse the same
> script. There should be two GN targets, both of which should include the same
> script as a data dependency, with their own unique dependencies as well. Note
> that the isolated script run_gpu_integration_test_as_googletest.py is used to
> run many different steps on the bots. I suggest using it as an example.
> 
> In addition to changing this code to dump [''], I would add a comment
indicating
> that recipes do not call this entry point for isolated scripts.

Done. Consolidated my two run_devtools_* script into one.

[chenwilliam, 2016-12-08 02:53:06]

The CQ bit was checked by [email protected] to run a CQ dry run

[commit-bot: I haz the power, 2016-12-08 02:53:41]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Dirk Pranke, 2016-12-08 03:09:38]

lgtm

[commit-bot: I haz the power, 2016-12-08 04:07:33]

The CQ bit was unchecked by [email protected]

[commit-bot: I haz the power, 2016-12-08 04:07:34]

Dry run: Try jobs failed on following builders:
  android_n5x_swarming_rel on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_...)

[chenwilliam, 2016-12-08 04:23:57]

Description was changed from

==========
Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if the INSTALL_NODE_FOR_DEVTOOLS=1 is present in the
environment.

Because that variable is not currently set in the environment on the
builder, this CL will not actually have any effect.

This CL also adds a //third_party/WebKit/Source/devtools:eslint build
target so that the isolate will be created properly (when we're ready
for it).

[email protected], [email protected]
BUG=642046

patch from issue 2479373004 at patchset 1 (http://crrev.com/2479373004#ps1)
==========

to

==========
DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046
==========

[chenwilliam, 2016-12-08 04:35:35]

The CQ bit was checked by [email protected] to run a CQ dry run

[commit-bot: I haz the power, 2016-12-08 04:35:50]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-08 04:43:37]

The CQ bit was unchecked by [email protected]

[commit-bot: I haz the power, 2016-12-08 04:43:38]

Dry run: Try jobs failed on following builders:
  win_clang on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_clang/builds/...)

[Ken Russell (switch to Gerrit), 2016-12-08 07:09:31]

lgtm once the trybots are green.

[chenwilliam, 2016-12-08 19:02:39]

The CQ bit was checked by [email protected] to run a CQ dry run

[commit-bot: I haz the power, 2016-12-08 19:04:04]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-08 20:18:54]

The CQ bit was unchecked by [email protected]

[commit-bot: I haz the power, 2016-12-08 20:18:56]

Dry run: Try jobs failed on following builders:
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[chenwilliam, 2016-12-08 21:06:17]

The CQ bit was checked by [email protected]

[chenwilliam, 2016-12-08 21:06:18]

The patchset sent to the CQ was uploaded after l-g-t-m from
[email protected], [email protected], [email protected]
Link to the patchset: https://codereview.chromium.org/2486903002/#ps420001
(title: "fix node.py")

[commit-bot: I haz the power, 2016-12-08 21:07:11]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-08 22:01:14]

The CQ bit was unchecked by [email protected]

[commit-bot: I haz the power, 2016-12-08 22:01:15]

Try jobs failed on following builders:
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[chenwilliam, 2016-12-08 22:49:40]

The CQ bit was checked by [email protected]

[commit-bot: I haz the power, 2016-12-08 22:50:31]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-09 02:16:32]

CQ is committing da patch.

Bot data: {"patchset_id": 420001, "attempt_start_ts": 1481237380100760,
"parent_rev": "0061e5b09cc34ac2497b416aab4e7ac9d078358a", "commit_rev":
"a67490506393210a03f486d2cdd46c87a0f90465"}

[commit-bot: I haz the power, 2016-12-09 02:17:10]

Description was changed from

==========
DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046
==========

to

==========
DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046
==========

[commit-bot: I haz the power, 2016-12-09 02:17:12]

Committed patchset #21 (id:420001)

[commit-bot: I haz the power, 2016-12-09 02:19:29]

Description was changed from

==========
DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046
==========

to

==========
DevTools: Add support for installing node and running eslint over devtools.

This CL adds a step to the 'Linux Tests' builder that will run
a step (under swarming) that runs eslint over the devtools frontend
code. eslint uses node.js, so this CL also adds a hook to gclient
to download node if CHROME_HEADLESS is present in the
environment.

[email protected], [email protected]
BUG=642046

Committed: https://crrev.com/56a90381f4d44cefeb5edb83a6227ebfa02fb885
Cr-Commit-Position: refs/heads/master@{#437431}
==========

[commit-bot: I haz the power, 2016-12-09 02:19:31]

Patchset 21 (id:??) landed as
https://crrev.com/56a90381f4d44cefeb5edb83a6227ebfa02fb885
Cr-Commit-Position: refs/heads/master@{#437431}