commit f223022a4c521296a6d058862d66443ab9e355bf
author Joel E. Denny <jdenny.ornl@gmail.com> Tue Sep 19 16:38:38 2023
committer Joel E. Denny <jdenny.ornl@gmail.com> Tue Sep 19 16:38:38 2023
tree f5f13e3559e32f8d04da38001412b69917c5e91c
parent 1495d51ee9e71fa58b523ec1703b3e258283481f

[lit] Improve test output from lit's internal shell

This patch and D154984 were discussed in
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839>.

Motivation
----------

D154984 removes the "Script:" section that lit prints along with a
test's output, and it makes -v and -a imply -vv.  For example, after
D154984, the "Script:" section below is never shown, but -v is enough
to produce the execution trace following it:

```
 Script:
 --
 : 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
 --
 Exit Code: 2

 Command Output (stdout):
 --
 $ ":" "RUN: at line 1"
 $ "echo" "hello"
 # command output:
 hello

 $ "FileCheck" "bogus.txt"
 # command stderr:
 Could not open check file 'bogus.txt': No such file or directory

 error: command failed with exit status: 2

 --
```

In the D154984 review, some reviewers point out that they have been
using the "Script:" section for copying and pasting a test's shell
commands to a terminal window.  The shell commands as printed in the
execution trace can be harder to copy and paste for the following
reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to
  copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and
  are not clearly delineated.
- They don't always use proper shell quoting.  Instead, they blindly
  enclose all command-line arguments in double quotes.

Changes
-------

D154984 plus this patch converts the above example into:

```
 Exit Code: 2

 Command Output (stdout):
 --
 # RUN: at line 1
 echo hello | FileCheck bogus-file.txt && echo success
 # executed command: echo hello
 # .---command stdout------------
 # | hello
 # `-----------------------------
 # executed command: FileCheck bogus-file.txt
 # .---command stderr------------
 # | Could not open check file 'bogus-file.txt': No such file or directory
 # `-----------------------------
 # error: command failed with exit status: 2

 --
```

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a
  terminal for correct execution of the RUN lines, which are printed
  intact as they appeared in the original RUN lines except lit
  substitutions are expanded.  Everything else in the execution trace
  appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as
  it executes to show (1) that the command actually executed (e.g.,
  `echo success` above didn't) and (2) what stdout, stderr, non-zero
  exit status, and output files are associated with the command, if
  any.  Shell quoting in the command is now correct and minimal but is
  not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output
  file is now delineated clearly in the trace.

To help produce some of the above output, this patch extends lit's
internal shell with a built-in `@echo` command.  It's like `echo`
except lit suppresses the normal execution trace for `@echo` and just
prints its stdout directly.  For now, `@echo` isn't documented for use
in lit tests.

Without this patch, libcxx's custom lit test format tries to parse the
stdout from `lit.TestRunner.executeScriptInternal` (which runs lit's
internal shell) to extract the stdout and stderr produced by shell
commands, and that parse no longer works after the above changes.
This patch makes a small adjustment to
`lit.TestRunner.executeScriptInternal` so libcxx can just request
stdout and stderr without an execution trace.

(As a minor drive-by fix that came up in testing: lit's internal `not`
command now always produces a numeric exit status and never `True`.)

Caveat
------

This patch only makes the above changes for lit's internal shell.  In
most cases, we do not know how to force external shells (e.g., bash,
sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is
usually better for test portability than external shells anyway), add
this to the test suite's `lit.cfg` or other configuration file:

```
config.test_format = lit.formats.ShTest(execute_external=False)
```

Reviewed By: MaskRay, awarzynski

Differential Revision: https://reviews.llvm.org/D156954