Strategies for testing Salt states

Barney Sowood - Saltconf21

Me

  • Barney Sowood
  • Automation Consultant, Corsac Ltd
  • Currently specialising in Salt and CI/CD
  • barney at corsac.uk
  • @barneysowood (Github, Twitter)

Acknowlegements

  • Thanks to

  • Saltconf 18 - Automated Testing for SaltStack Environments - Ethan Moore

Strategies for testing Salt states

Introduction

How do you ensure your Salt states work?

Strategies for testing Salt states

Introduction

Ensuring your Salt states work now and in the future

Strategies for testing Salt states

What we will cover

  • Setting up a local environment
  • Linting
  • Building a test suite with pytest and testinfra
  • Using containers and CI to run tests
  • Testing against multiple OS and Salt versions

Local testing

  • Testing of states on local machines during development
  • Needs to be lightweight
  • Should provide a quick feedback loop
  • Will help with formatting, syntax and jinja templating errors

Local testing

Setup environment

  • Make it easy for everyone to set up with shared tooling
  • Setup script in the same repository as states or formulas
  • Create a python virtualenv with salt installed

Local testing

Setup environment

Demo

Local testing

Using salt-call locally

  • Config to allow salt-call to run against states in the repository
  • Doesn't require master or salt installation outside the repository
  • Uses Saltfile and simple minion config

Local testing

Using salt-call locally

Demo

Local testing

Using the state module to test states

  • Use salt-call to run state module functions on local states
  • state.show_sls
  • state.apply mock=True
  • state.apply test=True

Local testing

Using state.show_sls

  • Run as salt-call state.show_sls <slsname>
  • Shows the states that will be run
  • Will catch formatting issues and jinja templating errors in state file
  • Doesn't catch issues with module or function names or arguments

Local testing

Using state.show_sls

Demo

Local testing

Using state.apply mock=True

  • Run as salt-call state.apply mock=True <slsname>
  • Runs through normal state.apply process but doesn't call state functions
  • Mocks return values from states
  • Validates state function exists
  • Checks state function argspec which will catch missing required arguments
  • Won't catch misnamed keyword arguments or do more thorough validation of arguments
  • Checks requisites

Local testing

Using state.apply mock=True

Demo

Local testing

Using state.apply test=True

  • Run as salt-call state.apply test=True <slsname>
  • Runs through normal state.apply process but calls state functions in test mode
  • Actually calls state functions so will catch issues with arguments
  • Will show what changes would be made on the local system

Local testing

Using state.apply test=True

Demo

Local testing

Linting

  • Language specific
  • Validates using rules
  • Enforce standards in state files and other config
  • Use combination of yamllint and salt-lint
  • Can be added to pre-commit

Local testing

Linting

yamllint

  • Checks YAML files for valid syntax and style issues such as line length
  • Will work on state files if they don't have any jinja templating
  • Still useful for YAML data files and in repository config

Local testing

Linting

yamllint

Demo

Local testing

Linting

salt-lint

  • Checks state files against a series of rules
  • Will work on state files with jinja
  • Relative small number of rules
  • Extensible if you want to add local standards

Local testing

Linting

salt-lint

Demo

Local testing

Building a test suite with pytest and testinfra

Local testing

pytest

  • Python testing framework
  • Auto discovery of tests
  • Fixtures system makes it easy to extend
  • Rich plugin ecosystem
  • Salt project moving to pytest

Local testing

pytest

pytest + Salt Python client API

  • We can create fixtures that use the salt.client.Caller() class
  • Same functionality as using salt-call but accessible from pytest tests
  • Able to validate output and handle failures

Local testing

pytest

pytest + Salt Python client API

Demo

Local testing

pytest

pytest + testinfra

  • We want to be able to actually apply states and validate they are functioning correctly
  • Testinfra is a pytest extension that allows us to assert things about the state of a host (eg package is installed, file is owned by user)
  • Normally we don't want to actually run state.apply against a local machine so we can configure pytest to skip these tests by default

Local testing

pytest

pytest + testinfra

Demo

Testing with containers

  • Isolated and disposable environment
  • OS and software dependencies can be set to match real environments
  • Automation ensures consistent builds

Testing with containers

Building container images and testing in CI

  • Dockerfile and CI config in states repository
  • Images can be used in CI or locally for isolated testing
  • CI config has jobs for linting and pytest tests

Testing with containers

Building container images and testing in CI

Demo - Gitlab cli
Demo - Gitlab web
Demo - Github cli
Demo - Github web

Testing with containers

Multiple salt and OS versions

  • Build process makes it easy to test variations
  • Testing against multiple OSes or OS versions
  • Testing multiple Salt versions

Testing with containers

Multiple salt and OS versions

Demo - cli
Demo - web

Further work

  • Testing that requires salt-master
  • Orchestrations

Summary

  • Setting up an environment for local testing and quick feedback
  • Using pytest and infratest to write tests against states
  • Building containers to run tests in
  • Testing against multiple operating system and Salt versions