Ellin.com
Not Very Interesting
Concourse and Secret Management
Ggood CI/CD pipelines with the management of passwords and other credentials required for deployment.

This page was converted from my old blog and hasn’t been reviewed. If you see an error please let me know in the comments.

One aspect of creating good CI/CD pipelines is the management of passwords and other credentials required for deployment.

A typical concourse pipeline will poll for updates in a git repo, do a build and then push the results to a PaaS such as Kubernetes or Cloud Foundry.

A Working Example.

For this example, we will use bucc. bucc is an all in one deployment of Bosh, UAA, Credhub, and Concourse. If you don’t have access to a working Concourse/Credhub environment, this is an excellent place to start.

  1. Install bucc per the documentation in here.

  2. Install Credhub cli from here

Adding a secret to Concourse.

Concourse will retrieve credentials from Credhub by looking them up based on their path.

1
2
/concourse/TEAM_NAME/PIPELINE_NAME/s3-password
/concourse/TEAM_NAME/s3-password

Global credentials for a team can be placed directly under the team name. Credentials for a specific pipeline can be organized under the team name/pipeline name.

1
2
3
4
5
6
-> credhub set -n /concourse/main/cf-password --type value --value foobar
id: 1fc1da07-4938-47d8-a7c4-1f442a61dc33
name: /concourse/main/cf-password
type: value
value: <redacted>
version_created_at: "2019-03-15T15:07:00Z"

Properties can be referenced in a pipeline using standard property replacement in Concourse

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
---
jobs:
- name: job-hello-world
  public: true
  plan:
  - task: hello-world
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: busybox}
      run:
        path: env
      params:
        CF_PASSWD: ((cf-password))

Running this job should reveal the password.

1
2
3
4
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/root
CAT_NAME=foobar
USER=root

Using other Secret types.

Credhub can store other types of secrets besides just key-value pairs. Examples include SSH keys, JSON structures, and Certificates. A complete list of types are available here.

An SSH Key contains two parts. public_key and private_key. It can be imported as follows.

credhub set --type ssh --name /concourse/main/ssh -p '-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAwGwcKp6LJqCmwz63HKGjhDhrsHJbn/bnWnvSE0oqPCic/LnA\ncY0qlvs4DbV+a7fYRDpvYfVAGQj277CkCnoEWKc6meiH+1PHcLJdOhKWSHNSkZrA\ntQ1Wb6MsVpXejpo4YzIiyLzaW4sXmz0bhxdkPWLRQAKr34fKJ27rOIJXDFTR1Bt8\nzz0As0R72R11o2GcnVjarR/3TAK+/ADkzAPrMMz9o+1J1wZD2YNBANs1dPh/IxZZ\nwWfqwc7JXCYKVFB+Xt7UpAam5UYt8gQ0lJnnNU5+TUhaUU5LenwNANmG4tLUHzqy\nYkUtSPhJ/BbNjYlKUnsN72ystrqPkmDPDP6g+wIDAQABAoIBAHwONyqTBItmz5zY\n9h0TaOR5q5QaZk//UrDXW1zsV8ZpOK0G5LdQl8C3PjA4bsTrxhZWxjCVeTmquelW\nLKxEdkDhr7pCXEkAfnh9xfUGvrT/BKCy8MLJUoyu2osIHHA7pVbun9ZjSzPxvMps\n3y59OjcJWna2QjDezsoVjLjl71EWz3Bk42gwZ3b4bBGlAgSgssL78E5xU9sYLGQP\ntKDsfU4OOB2VSdDsqpOiYyc5246GG8bbSmxbkmtWqL42iUvlnQptNanHAjphPWC+\nIFakDW8pugjFoGOpDW6jnzZEqEywFtmvpXd6jLeBKjBc6vtPODWbNN0fARdwo/An\noRPl6sECgYEA64eLHT3RMlMRxfjEKk3mQe8+qAVU5L92rzWgR9qgvANNlb4RFONU\nuwOzG9Tkv/vtWcR70LQY5KN2hJixCs1DyJgfPWIzrR6iPhc6aN4r48SjygDhFlPw\no6+qBpliHSNKSUao0u3+Bdk2LcYfqfXU+qjGKCXpl+t09W+/M2W3r0kCgYEA0SVy\nIOmjvm69dvj9ZSi6AbzSOP2gKWBXYG3qxpNlLq121mnEBf6JNagyKTvITCxT9bd0\n8DNYrVN8nxWF3nrROvmCGtBTNLVW5MRZYoBh0o/Qh1nCXCUODy7Vhyf4WtXNsGyu\nq3lqcJdZA791gdGpk+e6miuYFH0HcRNRKa0yWiMCgYBgvS1wd0GDcAcuzzyTO6fF\nkSSlEnuJ8PIoiNgqayv1zU2CoayWbcERhzV7yvehuzID2uYYFMDcuB8n2ydsjl62\n93RtW/Zpttlgs120UPyp8sxrXe0VpKiEMtSdHUblPOd4LWOOL15UvKC6MFQ1FNnD\nkqrBNsE5OuaxIJLh43eMsQKBgCCkvJSAgws1E6NfJ4XDfozI4PL+OyJaJCkr3soR\ntWg8sOC0b2EUImxajUG8T/37qTsf4EOhcATVlAzsehGIj+GpkfIHdAU1DJP2RZFH\nQn1v7vdBPkHNks0x3SgUSAI9frY7sGOZNtDN/pnEJ14U0GgCcjCf/0OrZB71CeT8\nYHCLAoGBAKY+kEMkX3drGj4BtCtJgt6nv3KZ/j7GJTl8M+brhBjH0fCtuZJgg7sP\nhukUE4Yb/qd1zLnFmUfepikow2qKhVzzdOhsdIR44BagqJzAS2jEkV/0m5PEABr3\nhfIpaY7w/RZ4Uid/5qGrJSWQnh00c+VqvVSCbfqnIeM4lwp9+slY\n-----END RSA PRIVATE KEY-----\n' -u 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAbBwqnosmoKbDPrccoaOEOGuwcluf9udae9ITSio8KJz8ucBxjSqW+zgNtX5rt9hEOm9h9UAZCPbvsKQKegRYpzqZ6If7U8dwsl06EpZIc1KRmsC1DVZvoyxWld6OmjhjMiLIvNpbixebPRuHF2Q9YtFAAqvfh8onbus4glcMVNHUG3zPPQCzRHvZHXWjYZydWNqtH/dMAr78AOTMA+swzP2j7UnXBkPZg0EA2zV0+H8jFlnBZ+rBzslcJgpUUH5e3tSkBqblRi3yBDSUmec1Tn5NSFpRTkt6fA0A2Ybi0tQfOrJiRS1I+En8Fs2NiUpSew3vbKy2uo+SYM8M/qD7'

You can now access the ssh key in your script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
---
jobs:
- name: job-hello-world
  public: true
  plan:
  - task: hello-world
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: busybox}
      run:
        path: env
      params:
        CAT_NAME: ((ssh.private_key))
  • You must flatten the key to a single line before importing it.
1
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' key.txt

alternatively you can pass a file containing the key to import.

Using SSH keys

A typical use case in concourse is polling git for updated commits.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
---
resources:
- name: my-project-resource
  type: git
  source:
    uri: git@github.com:concourse/git-resource.git
    branch: master
    private_key: ((ssh.private_key))

jobs:
- name: my-project-resource
  public: true
  plan:
  - get: resource-tutorial
    trigger: true

Generating Secrets

In addition to storing secrets, Credhub can be used to generate them.

  • Generate a new SSH key pair
1
credhub generate -t ssh --name /concourse/main/testssh  
  • retrieve the public key
1
2
credhub get --name testssh2 --output-json | jq .value.public_key


Last modified on 2019-03-15

comments powered by Disqus