Testing SaltStack formulas on Travis-ci

SaltStack is an Open Source DevOp tool to automate administration of a computer (server or desktop) infrastructure, typically but not limited to, developing in-house PaaS. Travis-ci is an Open Source Continuous Integration platform and online-hosted for free for Open Source projects.

This article targets SaltStack formula developers who wants to have CI enabled

• and of course every SaltStack user should be a formula developer wanting CI.

Overview

First things first, we have to test the /pillar.example file located at the top of the formula repository. Unfortunately, it’s not straightforward, I’ve started a discussion on salt-users in case we find a way to improve that.

So, we’ll have to make a test pillar directory. Then, organise a travis configuration file to run our various states and check idempotence.

Setting up the test pillar directory

This is the anatomy of our test pillar directory:

test/
test/pillar   -----------> Pillar used for tests
test/pillar/top.sls  ----> Includes example only
test/pillar/example.sls -> Symlink to /pillar.example

In our case, we’re testing with the /pillar.example file located at the top of the formula repository. The trick here is to include it in a top.sls.

So our test/pillar/top.sls contains:

base:
  '*':
    - example

And we created the symlink with:

[env] 15/05 2015 02:07:19 jpic@lue ~/work/novapost/rsyncd-formula () 
ln -sfn ../../pillar.example test/pillar/example.sls 

Result:

[env] 15/05 2015 02:07:19 jpic@lue ~/work/novapost/rsyncd-formula () 
$ ls -l test/pillar/example.sls
lrwxrwxrwx 1 jpic superadmin 20 May 15 02:07 test/pillar/example.sls -> ../../pillar.example

Now we have a test pillar directory including the example. Don’t forget to add the symlink to your git repo !

travis.yml

Our humble testing plan is to test each state with the example pillar:

• Run state.show_sls to ensure that it parses properly and have some debugging output,
• Run state.sls to run the state we’re on,
• Run state.sls again, capturing output, asserting that ^Not Run: is not present in the output, because if it is then it means that a state cannot detect by itself whether it has to be run or not and thus is not idempotent.

Example:

Best practice

Note that we’re not using anything like serverspec or envassert. In most case, it’s not worth mirroring the actual code in test code just for coverage. Rather, I believe that each formula should install and run its healhchecks at the end of each deployment in-place of a serverspec behaviour test.

That’s all folks !

As you can see, we have a proper test matrix set up !

Please let me know if there’s anything we can improve.