Concourse tutorial

Target Concourse

In the spirit of declaring absolutely everything you do to get absolutely the same
result every time, the fly CLI requires that you specify the target API for
every fly request.
First, alias it with a name tutorial (this name is used by all the tutorial task scripts).

fly --target tutorial login --concourse-url http://127.0.0.1:8080 -u admin -p admin fly

--target tutorial sync
You can now see this saved target Concourse API in a local file:

cat ~/.flyrc

Shows a simple YAML file with the API, credentials etc:

When we use the fly command we will target this Concourse API using fly --target
tutorial.

The central concept of concourse is to run tasks. You can run them directly
from the command line as below, or from within pipeline jobs (as per every
other section of the tutorial).

Now clone the concourse tutorial repo, switch to the task-hello-world directory, and
run the command to execute the task_hello_world.yml task.
git clone https://github.com/starkandwayne/concourse-tutorial.git
cd concourse-tutorial/tutorials/basic/task-hello-world
fly -t tutorial execute -c task_hello_world.yml

The output starts with

executing build 1 at http://127.0.0.1:8080/builds/1 initializing

---
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path:
echo args: [hello world]
From the same directory in which you previously deployed the Docker Concourse
image (verify by running ls -l and looking for the docker-compose.yml file), start the
local Concourse server.
docker-compose up
Now clone the Concourse Tutorial repo, switch to the task-hello-world directory, and
run the command to execute the task_hello_world.yml task.
Every task in Concourse runs within a "container" (as best available on the target
platform).
The task_hello_world.yml configuration shows that we are running on
a linux platform
using the busybox container image.
You will see it downloading a Docker image busybox.
It will only need to do this once; though will recheck every time that it has the
latest busybox image.
Within this container it will run the command echo hello world.

Task Docker Images

Try changing the image_resource: and the run: and run a different task:

The reason that you can select any

base image (or image_resource when configuring a task) is that this allows your task
to have any prepared dependencies that it needs to run. Instead of installing
dependencies each time during a task you might choose to pre-bake them into
an image to make your tasks much faster.

Miscellaneous

If you're interested in creating new Docker images using Concourse (of course you
are), then there is a future section Create and Use Docker Images.

The reason that you can select anybase image (or image_resource when configuring

a task) is that this allows your task to have any prepared dependencies that it needs
to run. Instead of installing dependencies each time during a task you might choose
to pre-bake them into an image to make your tasks much faster.
Task Inputs
In the previous section the only inputs to the task container were the image used.
Base images, such as Docker images, are relatively static and relatively big, slow
things to create. So Concourse supports inputs into tasks to pass in files/folders for
processing.

Consider the working directory of a task that explicitly has no inputs:

cd ../task-inputs
fly -t tutorial e -c no_inputs.yml
The task runs ls -al to show the (empty) contents of the working folder inside the
container:
running ls -al
total 8
drwxr-xr-x 2 root root 4096 Feb 27 07:23 .
drwxr-xr-x 3 root root 4096 Feb 27 07:23 ..

Note that above we used the short-hand form of the execute command in this
example, simply e, as the action. Many commands have shortened single character
forms, for example fly s is an alias for fly sync.
In the example task inputs_required.yml we add a single input:

inputs:
- name: some-important-input

When we try to execute the task:

fly -t tutorial e -c inputs_required.yml

Commonly if wanting to run fly execute we will want to pass in the local folder (.).
Use -i name=path option to configure each of the required inputs:

fly -t tutorial e -c inputs_required.yml -i some-important-input=.

Now the fly execute command will upload the . directory as an input to the container.
It will be made available at the path some-important-input:
 To pass in a different directory as an input, provide its absolute or relative path:

fly -t tutorial e -c inputs_required.yml -i some-important-input=../task-hello-world

The fly execute -i option can be removed if the current directory is the same name as
the required input.
The task input_parent_dir.yml contains an input task-inputs which is also the current
directory. All the contents in the directory ./task-inputs will be uploaded to the docker
image. So the following command will work and return the same results as above:

fly -t tutorial e -c input_parent_dir.yml

Task Scripts

The inputs feature of a task allows us to pass in two types of inputs:

 requirements/dependencies to be processed/tested/compiled
 task scripts to be executed to perform complex behavior

A common pattern is for Concourse tasks to run: complex shell scripts rather than
directly invoking commands as we did in the Hello World tutorial (we
ran uname command with arguments -a).
Let's refactor task-hello-world/task_ubuntu_uname.yml into a new task task-
scripts/task_show_uname.yml with a separated task script task-
scripts/task_show_uname.sh
cd ../task-scripts

fly -t tutorial e -c task_show_uname.yml

The former specifies the latter as its task script:

run:
path: ./task-scripts/task_show_uname.sh

Where does the  ./task-scripts/task_show_uname.sh file come from?

From section 2 we learned that we could pass inputs into the task. The task
configuration task-scripts/task_show_uname.yml specifies one input:

inputs:
- name: task-scripts

Since input task-scripts matches the current directory task-scripts we did not need to

specify fly execute -i task-scripts=..

The current directory was uploaded to the Concourse task container and placed inside
the task-scripts directory.

Therefore its file task_show_uname.sh is available within the Concourse task container

at task-scripts/task_show_uname.sh.
The only further requirement is that task_show_uname.sh is an executable script.

Basic Pipeline
1% of tasks that Concourse runs are via fly execute. 99% of tasks that Concourse runs are
within "pipelines".
cd ../basic-pipeline
fly -t tutorial set-pipeline -c pipeline.yml -p hello-world

It will display the concourse pipeline (or any changes) and request confirmation:

1. Using the fly unpause-pipeline command (or its alias fly up):

fly -t tutorial unpause-pipeline -p hello-world
fly -t tutorial unpause-job --job hello-world/job-hello-world

Pipeline Resources

It is very fast to iterate on a job's tasks by configuring them in the  pipeline.yml YAML

file.
You edit the pipeline.yml,
run fly set-pipeline, and the entire pipeline is updated automatically.
The initial lessons introduced Tasks as standalone YAML files (which can be run via fly
execute).
Our pipeline.yml YAML files can be refactored to use these.
Also in the earlier lesson Task Scripts we looked at extracting complex run commands into
standalone shell scripts.

But with pipelines we now need to store the task file and task script somewhere outside of
Concourse.
Concourse offers no services for storing/retrieving your data. No git repositories. No
blobstores. No build numbers.
Every input and output must be provided externally. Concourse calls them "Resources".
Example resources are git, s3, and semver respectively.

See the Concourse documentation Resource Types for the list of built-in resource types and
community resource types. Send messages to Slack.

Bump a version number from 0.5.6 to 1.0.0.

Create a ticket on Pivotal Tracker. It is all possible with Concourse resource types. The
Concourse Tutorials Miscellaneous section also introduces some commonly useful Resource
Types.

be fetched via the s3 resource type from an AWS S3 file; The most common resource type to
store our task files and task scripts is the git resource type. Perhaps your task files could or
the archive resource type to extract them from a remote archive file. Or perhaps the task
files could be pre-baked into the image_resource base Docker image.
But mostly you will use a git resource in your pipeline to pull in your pipeline task files.
This tutorial's source repository is a Git repo, and it contains many task files (and their task
scripts). For example, the original tutorials/basic/task-hello-
world/task_hello_world.yml.
To pull in the Git repository, we edit pipeline-resources/pipeline.yml and add a
top-level section resources:
The Concourse Tutorial verbosely prefixes resource- to resource names, and job- to job
names, to help you identify one versus the other whilst learning. Eventually you will know
one from the other and can remove the extraneous text.

After manually triggering the job via the UI, the output will look like:

The in-progress or newly-completed job-hello-world job UI has three sections:

 Preparation for running the job - collecting inputs and dependencies

 resource-tutorial resource is fetched
 hello-world task is executed

The latter two are "steps" in the job's build plan. A build plan is a sequence of steps to
execute. These steps may fetch down or update Resources, or execute Tasks.

The first build plan step fetches down (note the down arrow to the left) a git repository for
these training materials and tutorials. The pipeline named this resource resource-
tutorial and clones the repo into a directory with the same name. This means that later in
the build-plan, we reference files relative to this folder.
The resource resource-tutorial is then used in the build plan for the job:
Any fetched resource can now be an input to any task in the job build plan. As discussed in
lessons Task Inputs and Task Scripts task inputs can be used as task scripts.

The second step runs a user-defined task. The pipeline named the task hello-world. The
task itself is not described in the pipeline. Instead it is described in a
file tutorials/basic/task-hello-world/task_hello_world.yml from the resource-
tutorial input.

The completed job looks like:

The name of resources, and later the name of task outputs, determines the name used to
access them by other tasks (and later, by updated resources).

So, hello-world can access anything from resource-tutorial (this tutorial's git repository)

under the resource-tutorial/ path. Since the relative path of task_hello_world.yml task file
inside this repo is tutorials/basic/task-hello-world/task_hello_world.yml, the task: hello-
world references it by joining the two: file: resource-tutorial/tutorials/basic/task-hello-
world/task_hello_world.yml

There is a benefit and a downside to abstracting tasks into YAML files outside of the
pipeline.

One benefit is that the behavior of the task can be kept in sync with the primary input
resource (for example, a software project with tasks for running tests, building binaries, etc).

One downside is that the pipeline.yml no longer explains exactly what commands will
be invoked. Comprehension of pipeline behavior is potentially reduced.
But one benefit of extracting inline tasks into task files is that pipeline.yml files can
get long and it can be hard to read and comprehend all the YAML. Instead, give tasks
long names so that readers can understand what the purpose and expectation of the
task is at a glance.
But one downside of extracting inline tasks into files is that fly set-pipeline is no
longer the only step to updating a pipeline.

From now onwards, any change to your pipeline might require you to do one or both:

 fly set-pipeline to update Concourse on a change to the job build plan and/or
input/output resources
 git commit and git push your primary resource that contains the task files and
task scripts

If a pipeline is not performing new behaviour then it might be you skipped one of the
two steps above.

Due to the benefits vs downsides of the two approaches - inline task configuration vs
YAML file task configuration - you will see both approaches used in this Concourse
Tutorial and in the wider community of Concourse users.

Watch Job Output in Terminal

It was very helpful that the job-hello-world job build included the terminal output from
running git commands to clone the git repo and the output of the running hello-world task.

You can also view this output from the terminal with fly watch:

fly -t tutorial watch -j hello-world/job-hello-world

The output will be similar to:

The --build NUM option allows you to see the output of a specific build number, rather than
the latest build output.
You can see the results of recent builds across all pipelines with fly builds:

fly -t tutorial builds

The output will look like:

The fly watch command can also be a battery saver on your laptop. Hear me out: I've
observed that watching jobs run in the Concourse Web UI uses a lot more battery power than
running fly watch in a terminal. Your mileage may vary.

Trigger Jobs

There are four ways for a job to be triggered:

 Clicking the + button on the web UI of a job (as we did in previous sections)

 Input resource triggering a job (see the next lesson Triggering Jobs with Resources)
 fly trigger-job -j pipeline/jobname command
 Sending POST HTTP request to Concourse API
We can re-trigger our hello-world pipeline's job-hello-world:
fly -t tutorial trigger-job -j hello-world/job-hello-world
Whilst the job is running, and after it has completed, you can then watch the output in your
terminal using 
fly watch:
fly -t tutorial watch -j hello-world/job-hello-world
Alternately, you can combine the two commands - trigger the job and watch the output with
the trigger-job -w flag:
fly -t tutorial trigger-job -j hello-world/job-hello-world -w

In the next lesson we will learn to trigger jobs after changes to an input resource.

Triggering Jobs with Resources

The primary way that Concourse jobs will be triggered to run will be by resources changing.
A git repo has a new commit? Run a job to test it. A GitHub project cuts a new release? Run
a job to pull down its attached files and do something with them.
Triggering resources are defined the same as non-triggering resources, such as the resource-
tutorial defined earlier.

The difference is in the job build plan where triggering is de sired.

By default, including get: my-resource in a build plan does not trigger its job.
To mark a fetched resource as a trigger add trigger: true
jobs: - name: job-demo plan: - get: resource-tutorial trigger: true

In the above example the job-demo job would trigger anytime the remote resource-

tutorial had a new version. For a git resource this would be new git commits.
The time resource has intrinsic purpose of triggering jobs.

If you want a job to trigger every few minutes then there is the time resource.
Now upgrade the hello-world pipeline with the time trigger and unpause it.

This adds a new resource named my-timer which triggers job-hello-world approximately

every 2 minutes.

Visit the pipeline dashboard http://127.0.0.1:8080/teams/main/pipelines/hello-world and wait

a few minutes and eventually the job will start running automatically.

The dashboard UI makes non-triggering resources distinct with a hyphenated line

connecting them into the job. Triggering resources have a full line.

Why does time resource configured with interval: 2m trigger "approximately"

every 2 minutes?
"resources are checked every minute, but there's a shorter (10sec) interval for
determining when a build should run; time resource is to just ensure a build runs on
some rough periodicity; we use it to e.g. continuously run integration/acceptance tests
to weed out flakiness" - alex
The net result is that a timer of 2m will trigger every 2 to 3 minutes.
Destroying Pipelines

The current hello-world pipeline will now keep triggering every 2-3 minutes forever. If you
want to destroy a pipeline - and lose all its build history - then may the power be granted to
you.
You can delete the hello-world pipeline:
fly -t tutorial destroy-pipeline -p hello-world

Using Resource Inputs in Job Tasks

Consider a simple application that has unit tests. In order to run those tests inside a pipeline
we need:

 a task image that contains any dependencies

 an input resource containing the task script that knows how to run the tests
 an input resource containing the application source code
For the example Go application simple-go-web-app,
the task image needs to include the Go programming language. We will use the golang:1.9-
alpine image from https://hub.docker.com/_/golang/ (see https://imagelayers.io/?
images=golang: 1.9-alpine for size of layers) The task file task_run_tests.yml includes:

The resource-app resource will place the inbound files for the input into an alternate path.
By default we have seen that inputs store their contents in a folder of the same name. The
reason for using an alternate path in this example is specific to building & testing Go
language applications and is outside the scope of the section.

To run this task within a pipeline:

View the pipeline UI http://127.0.0.1:8080/teams/main/pipelines/simple-app and notice that

the job automatically starts.
The job will pause on the first run at web-app-tests task because it is downloading
the golang:1.9-alpine image for the first time.
The web-app-tests output below corresponds to the Go language test output (in case you've
not seen it before):
ok github.com/cloudfoundry-community/simple-go-web-app 0.003s

Passing Task Outputs to Task Inputs

In previously our task web-app-tests consumed an input resource and ran a script that ran
some unit tests. The task did not create anything new. Some tasks will want to create
something that is then passed to another task for further processing (this lesson); and some
tasks will create something that is pushed back out to the external world (next lesson).
So far our pipelines' tasks' inputs have only come from resources using get: resource-
tutorial build plan steps.
A task's inputs can also come from the outputs of previous tasks. All a task needs to do is
declare that it publishes outputs, and subsequent steps can consume those as inputs by the
same name.
A task file declares it will publish outputs with the outputs section:

outputs:
- name: some-files

If a task included the above outputs section then its run: command would be responsible

for putting interesting files in the some-files directory.
Subsequent tasks (discussed in this section) or resources (discussed in the next section) could
reference these interesting files within the some-files/ directory.

In this pipeline's job-pass-files there are two task steps create-some-files and show-

some-files:

The former creates 4 files into its own some-files/ directory. The latter gets a copy of these
files placed in its own task container filesystem at the path some-files/.
The pipeline build plan only shows that two tasks are to be run in a specific order. It does not
indicate that some-files/ is an output of one task and used as an input into the next task.

jobs:
- name: job-pass-files
public: true
plan:
- get: resource-tutorial
- task: create-some-files
config:
...
inputs:
- name: resource-tutorial
outputs:
- name: some-files

run:
path: resource-tutorial/tutorials/basic/task-outputs-to inputs / create_some_
files.sh

- task: show-some-files
config:
...
inputs:
- name: resource-tutorial
- name: some-files

run:
path: resource-tutorial/tutorials/basic/task-outputs-to-inputs/show_files.sh
Note, task create-some-files build output includes the following error:
mkdir: can't create directory 'some-files': File exists
This is a demonstration that if a task includes outputs then those output directories are pre-
created and do not need creating.
Publishing Outputs
So far we have used the git resource to fetch down a git repository, and
used git & time resources as triggers. The git resource can also be used to push a modified
git repository to a remote endpoint (possibly different than where the git repo was originally
cloned from).

Pipeline dashboard http://127.0.0.1:8080/teams/main/pipelines/publishing-outputs shows that

the input resource is erroring (see orange in key):
The pipeline.yml does not yet have a git repo nor its write-access private key
credentials. Create a Github Gist with a single file bumpme, and press "Create public
gist":

Click the "Embed" dropdown, select "Clone via SSH", and copy the git URL:
And modify the resource-git section of pipeline.yml

- name: resource-gist
type: git
source:
uri: [email protected]:e028e491e42b9fb08447a3bafcf884e5.git
branch: master
private_key: |-
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAuvUl9YU...
...
HBstYQubAQy4oAEHu8osRhH...
-----END RSA PRIVATE KEY----
Also paste in your ~/.ssh/id_rsa private key (or which ever you have registered with
github) into the private_key section. Note: Please make sure that the key used here is not
generated using a passphrase. Otherwise, the key will not be accepted and you would get an
error.

Update the pipeline, force Concourse to quickly re-check the new Gist credentials, and then
run the job:

Revisit the Web UI and the orange resource will change to black if it can successfully fetch
the new [email protected]:XXXX.git repo.
After the job-bump-date job completes, refresh your gist:

This pipeline is an example of updating a resource. It has pushed up new git commits to the
git repo (your github gist).

Where did the new commit come from?

The task: bump-timestamp-file task configuration describes a single output updated-

gist:

outputs:
- name: updated-gist
The bump-timestamp-file task runs the following bump-timestamp-file.sh script:

First, it copied the input resource resource-gist into the output resource updated-

gist (using git clone as a preferred git way to do this). A trivial modification is made to
the updated-gist directory, followed by a git commit to modify the updated-
gist folder's Git repository. It is this updated-gist folder and its additional git
commit that is subsequently pushed back to the gist by the pipeline step:

- put: resource-gist
params:
repository: updated-gist
The updated-gist output from the task: bump-timestamp-file step becomes
the updated-gist input to the resource-gist resource because their names match (see
the git resource for additional configuration).
Dependencies within Tasks
The bump-timestamp-file.sh script needed the git CLI.
It could have been installed at the top of the script using apt-get update; apt-get install
git or similar, but this would have made the task very slow - each time it ran it would
have reinstalled the CLI.
Instead, the bump-timestamp-file.sh step assumes its base Docker image already
contains the git CLI.
The Docker image being used is described in the image_resources section of the task's
configuration:
The Docker image starkandwayne/concourse is described
at https://github.com/starkandwayne/dockerfiles/ and is common base Docker image used by
many Stark & Wayne pipelines.

Your organisation may wish to curate its own based Docker images to be shared across
pipelines. After finishing the Basics lessons, visit Lesson Create and Use Docker Images for
creating pipelines to create your own Docker images using Concourse.

Tragic Security
If you're feeling ill from copying your private keys into a plain text file (pipeline.yml) and
then seeing them printed to the screen (during fly set-pipeline -c pipeline.yml), then
fear not. We will get to Secret with Credential Manager soon.

Parameterized Pipelines
In the preceding section you were asked to place private credentials and personal git URLs
into the pipeline.yml files. This would make it difficult to share your pipeline.yml with
anyone who had access to the repository. Not everyone needs nor should have access to the
shared secrets.
Concourse pipelines can include ((parameter)) parameters for any value in the pipeline
YAML file. Parameters are all mandatory. There are no default values for parameters.
In the lesson's pipeline.yml there are two parameters:
If we fly set-pipeline but do not provide the parameters, we see an error when the job is
triggered to run:

This will fail with the following error:

Parameters from fly options

The output will show that the -v variables were passed into the params section of the show-
animal-names task. Values in params sections in turn become environment variables within
the task:

Parameters from local file

Alternatively, you can store your parameter values in a local file.

Use the --load-vars-from flag (aliased -l) to pass in this file instead of the -v flag. The

following command should not modify the pipeline from the preceding step as the resulting
pipeline YAML is equivalent.

fly -t tutorial sp -p parameters -c pipeline.yml -l credentials.yml

Revisiting Publishing Outputs

In the previous lesson Publishing Outputs, there were two user-provided changes to
the pipeline.yml. These can now be changed to parameters.
cd ../publishing-outputs
There is an alternate pipeline-parameters.yml that offers two parameters for resource-
gist:
cd ../publishing-outputs
There is an alternate pipeline-parameters.yml that offers two parameters for resource-
gist:
Create a credentials.yml with the Gist URL and private key:

Use the --load-vars-from or -l flag to pass the variables into the parameters:

Dynamic Parameters and Secret Parameters

Parameters are very useful. They allow you to describe your pipeline.yml in public
repositories without embedding variables nor secrets.

There are two downsides to the two approaches above.

 To change any parameter values requires you to rerun fly set-pipeline. If a value

is common across many pipelines then you must rerun fly set-pipeline for them all.
 The parameter values are not very secret. Anyone with access to the pipeline's team is
able to download the pipeline YAML and extract the secrets.
fly -t tutorial get-pipeline -p parameters

Shows that the two potentially secret parameters are visible in plain text:
... params: CAT_NAME: garfield DOG_NAME: odie run: path: env

The solution to both of these problems is to use a Concourse Credentials Manager and is
discussed in lesson Secret with Credential Manager.
Actual Pipeline - Passing Resources Between Jobs
Finally, it is time to make an actual pipeline - one job passing results to another job upon
success.

In all previous sections our pipelines have only had a single job. For all their wonderfulness,
they haven't yet felt like actual pipelines. Jobs passing results between jobs. This is where
Concourse shines even brighter.

Update the publishing-outputs pipeline with a second job job-show-date which will run

whenever the first job successfully completes:

- name: job-show-date
plan:
- get: resource-tutorial
- get: resource-gist
passed: [job-bump-date]
trigger: true
- task: show-date
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: resource-gist
run:
path: cat
args: [resource-gist/bumpme]
Update the pipeline:
cd ../pipeline-jobs
fly -t tutorial sp -p publishing-outputs -c pipeline.yml -l ../publishing-
outputs/credentials.yml
fly -t tutorial trigger-job -w -j publishing-outputs/job-bump-date

If you are missing ../publishing-outputs/credentials.yml, visit the section Revisiting

Publishing Outputs from the previous lesson.

The dashboard UI displays the additional job and its trigger/non-trigger resources.
Importantly, it shows our first multi-job pipeline:
The latest resource-gist commit fetched down in job-show-date will be the exact commit
used in the last successful job-bump-date job. If you manually created a new git commit in
your gist and manually ran the job-show-date job it would continue to use the previous
commit it used, and ignore your new commit. This is the power of pipelines.

Secret left

out