# salt.pillar.git_pillar

## Use a git repository as a Pillar source

This external pillar allows for a Pillar top file and Pillar SLS files to be sourced from a git repository.

However, since git_pillar does not have an equivalent to the `pillar_roots` parameter, configuration is slightly different. A Pillar top file is required to be in the git repository and must still contain the relevant environment, like so:

    base:
      '*':
        - foo

The branch/tag which maps to that environment must then be specified along with the repo's URL. Configuration details can be found below.

Important

Each branch/tag used for git_pillar must have its own top file. This is different from how the top file works when configuring States. The reason for this is that each git_pillar branch/tag is processed separately from the rest. Therefore, if the `qa` branch is to be used for git_pillar, it would need to have its own top file, with the `qa` environment defined within it, like this:

    qa:
      'dev-*':
        - bar

Additionally, while git_pillar allows for the branch/tag to be overridden (see here), keep in mind that the top file must reference the actual environment name. It is common practice to make the environment in a git_pillar top file match the branch/tag name, but when remapping, the environment of course no longer matches the branch/tag, and the top file needs to be adjusted accordingly. When expected Pillar values configured in git_pillar are missing, this is a common misconfiguration that may be to blame, and is a good first step in troubleshooting.

### Configuring git_pillar for Salt

Beginning with Salt version 2015.8.0, pygit2 is now supported in addition to GitPython. The requirements for GitPython and pygit2 are the same as for GitFS, as described here.

Important

git_pillar has its own set of global configuration parameters. While it may seem intuitive to use the global gitfs configuration parameters (`gitfs_base`, etc.) to manage git_pillar, this will not work. The main difference for this is the fact that the different components which use Salt's git backend code do not all function identically. For instance, in git_pillar it is necessary to specify which branch/tag to be used for git_pillar remotes. This is the reverse behavior from gitfs, where branches/tags make up your environments.

See here for documentation on the git_pillar configuration options and their usage.

Here is an example git_pillar configuration:

    ext_pillar:
      - git:
        # Use 'prod' instead of the branch name 'production' as the environment
        - production https://gitserver/git-pillar.git:
          - env: prod
        # Use 'dev' instead of the branch name 'develop' as the environment
        - develop https://gitserver/git-pillar.git:
          - env: dev
        # No per-remote config parameters (and no trailing colon), 'qa' will
        # be used as the environment
        - qa https://gitserver/git-pillar.git
        # SSH key authentication
        - master git@other-git-server:pillardata-ssh.git:
          # Pillar SLS files will be read from the 'pillar' subdirectory in
          # this repository
          - root: pillar
          - privkey: /path/to/key
          - pubkey: /path/to/key.pub
          - passphrase: CorrectHorseBatteryStaple
        # HTTPS authentication
        - master https://other-git-server/pillardata-https.git:
          - user: git
          - password: CorrectHorseBatteryStaple

The main difference between this and the old way of configuring git_pillar is that multiple remotes can be configured under one `git` section under `ext_pillar`. More than one `git` section can be used, but it is not necessary. Remotes will be evaluated sequentially.

Per-remote configuration parameters are supported (similar to gitfs), and global versions of the git_pillar configuration parameters can also be set.

To remap a specific branch to a specific Pillar environment, use the `env` per-remote parameter:

    ext_pillar:
      - git:
        - production https://gitserver/git-pillar.git:
          - env: prod

If `__env__` is specified as the branch name, then git_pillar will decide which branch to use based on the following criteria:

- If the minion has a `pillarenv` configured, it will use that pillar environment. (2016.11.2 and later)

- Otherwise, if the minion has an `environment` configured, it will use that environment.

- Otherwise, the master's `git_pillar_base` will be used.

Note

The use of `environment` to choose the pillar environment dates from a time before the `pillarenv` parameter was added. In a future release, it will be ignored and either the minion's `pillarenv` or the master's `git_pillar_base` will be used.

Here's an example of using `__env__` as the git_pillar environment:

    ext_pillar:
      - git:
        - __env__ https://gitserver/git-pillar.git:
          - root: pillar

The corresponding Pillar top file would look like this:

    "{{saltenv}}":
      '*':
        - bar

With the addition of pygit2 support, git_pillar can now interact with authenticated remotes. Authentication works just like in gitfs (as outlined in the Git Fileserver Backend Walkthrough), only with the global authentication parameter names prefixed with `git_pillar` instead of `gitfs` (e.g. `git_pillar_pubkey`, `git_pillar_privkey`, `git_pillar_passphrase`, etc.).

Note

The `name` parameter can be used to further differentiate between two remotes with the same URL and branch. When using two remotes with the same URL, the `name` option is required.

#### How Multiple Remotes Are Handled

As noted above, multiple remotes can be included in the same `git` ext_pillar configuration. Consider the following:

    my_etcd_config:
      etcd.host: 127.0.0.1
      etcd.port: 4001

    ext_pillar:
      - etcd: my_etcd_config
      - git:
        - master https://mydomain.tld/foo.git:
          - root: pillar
        - master https://mydomain.tld/bar.git
        - master https://mydomain.tld/baz.git
        - dev https://mydomain.tld/qux.git
      - git:
        - master https://mydomain.tld/abc.git
        - dev https://mydomain.tld/123.git

To understand how pillar data from these repos will be compiled, it's important to know how Salt will process them. The following points should be kept in mind:

1. Each ext_pillar is called separately from the others. So, in the above example, the `etcd` ext_pillar will be evaluated first, with the first group of git_pillar remotes evaluated next (and merged into the etcd pillar data). Lastly, the second group of git_pillar remotes will be evaluated, and then merged into the ext_pillar data evaluated before it.

2. Within a single group of git_pillar remotes, each remote will be evaluated in order, with results merged together as each remote is evaluated.

   Note

   Prior to the 2017.7.0 release, remotes would be evaluated in a non-deterministic order.

3. By default, when a repo is evaluated, other remotes' which share its pillar environment will have their files made available to the remote being processed.

The first point should be straightforward enough, but the second and third could use some additional clarification.

First, point #2. In the first group of git_pillar remotes, the top file and pillar SLS files in the `foo` remote will be evaluated first. The `bar` remote will be evaluated next, and its results will be merged into the pillar data compiled when the `foo` remote was evaluated. As the subsequent remotes are evaluated, their data will be merged in the same fashion.

But wait, don't these repositories belong to more than one pillar environments? Well, yes. The default method of generating pillar data compiles pillar data from all environments. This behavior can be overridden using a `pillarenv`. Setting a `pillarenv` in the minion config file will make that minion tell the master to ignore any pillar data from environments which don't match that pillarenv. A pillarenv can also be specified for a given minion or set of minions when `running states`, by using the `pillarenv` argument. The CLI pillarenv will override one set in the minion config file. So, assuming that a pillarenv of `base` was set for a minion, it would not get any of the pillar variables configured in the `qux` remote, since that remote is assigned to the `dev` environment. The only way to get its pillar data would be to specify a pillarenv of `dev`, which would mean that it would then ignore any items from the `base` pillarenv. A more detailed explanation of pillar environments can be found here.

Moving on to point #3, and looking at the example ext_pillar configuration, as the `foo` remote is evaluated, it will also have access to the files from the `bar` and `baz` remotes, since all three are assigned to the `base` pillar environment. So, if an SLS file referenced by the `foo` remotes's top file does not exist in the `foo` remote, it will be searched for in the `bar` remote, followed by the `baz` remote. When it comes time to evaluate the `bar` remote, SLS files referenced by the `bar` remote's top file will first be looked for in the `bar` remote, followed by `foo`, and `baz`, and when the `baz` remote is processed, SLS files will be looked for in `baz`, followed by `foo` and `bar`. This "failover" logic is called a directory overlay, and it is also used by `file_roots` and :conf_minion`pillar_roots`. The ordering of which remote is checked for SLS files is determined by the order they are listed. First the remote being processed is checked, then the others that share the same environment are checked. However, before the 2017.7.0 release, since evaluation was unordered, the remote being processed would be checked, followed in no specific order by the other repos which share the same environment.

Beginning with the 2017.7.0 release, this behavior of git_pillar remotes having access to files in other repos which share the same environment can be disabled by setting `git_pillar_includes` to `False`. If this is done, then all git_pillar remotes will only have access to their own SLS files. Another way of ensuring that a git_pillar remote will not have access to SLS files from other git_pillar remotes which share the same pillar environment is to put them in a separate `git` section under `ext_pillar`. Look again at the example configuration above. In the second group of git_pillar remotes, the `abc` remote would not have access to the SLS files from the `foo`, `bar`, and `baz` remotes, and vice-versa.

#### Mountpoints

Added in version 2017.7.0.

Assume the following pillar top file:

    base:
      'web*':
        - common
        - web.server.nginx
        - web.server.appdata

Now, assume that you would like to configure the `web.server.nginx` and `web.server.appdata` SLS files in separate repos. This could be done using the following ext_pillar configuration (assuming that `git_pillar_includes` has not been set to `False`):

    ext_pillar:
      - git:
        - master https://mydomain.tld/pillar-common.git
        - master https://mydomain.tld/pillar-nginx.git
        - master https://mydomain.tld/pillar-appdata.git

However, in order to get the files in the second and third git_pillar remotes to work, you would need to first create the directory structure underneath it (i.e. place them underneath `web/server/` in the repository). This also makes it tedious to reorganize the configuration, as changing `web.server.nginx` to `web.nginx` in the top file would require you to also move the SLS files in the `pillar-nginx` up a directory level.

For these reasons, much like gitfs, git_pillar now supports a "mountpoint" feature. Using the following ext_pillar configuration, the SLS files in the second and third git_pillar remotes can be placed in the root of the git repository:

    ext_pillar:
      - git:
        - master https://mydomain.tld/pillar-common.git
        - master https://mydomain.tld/pillar-nginx.git:
          - mountpoint: web/server/
        - master https://mydomain.tld/pillar-appdata.git:
          - mountpoint: web/server/

Now, if the top file changed the SLS target from `web.server.nginx`, instead of reorganizing the git repository, you would just need to adjust the mountpoint to `web/` (and restart the `salt-master` daemon).

Note

- Leading and trailing slashes on the mountpoints are optional.

- Use of the `mountpoint` feature requires that `git_pillar_includes` is not disabled.

- Content from mounted git_pillar repos can only be referenced by a top file in the same pillar environment.

- Salt versions prior to 2018.3.4 ignore the `root` parameter when `mountpoint` is set.

#### all_saltenvs

Added in version 2018.3.4.

When `__env__` is specified as the branch name, `all_saltenvs` per-remote configuration parameter overrides the logic Salt uses to map branches/tags to pillar environments (i.e. pillarenvs). This allows a single branch/tag to appear in all saltenvs. Example:

    ext_pillar:
      - git:
        - __env__ https://mydomain.tld/top.git
          - all_saltenvs: master
        - __env__ https://mydomain.tld/pillar-nginx.git:
          - mountpoint: web/server/
        - __env__ https://mydomain.tld/pillar-appdata.git:
          - mountpoint: web/server/

#### git_pillar_update_interval

Added in version 3000.

This option defines the default update interval (in seconds) for git_pillar remotes. The update is handled within the global loop, hence `git_pillar_update_interval` should be a multiple of `loop_interval`.

    git_pillar_update_interval: 120

#### fallback

Added in version 3001.

Setting `fallback` per-remote or global configuration parameter will map non-existing environments to a default branch. Example:

    ext_pillar:
      - git:
        - __env__ https://mydomain.tld/top.git
          - all_saltenvs: master
        - __env__ https://mydomain.tld/pillar-nginx.git:
          - mountpoint: web/server/
          - fallback: master
        - __env__ https://mydomain.tld/pillar-appdata.git:
          - mountpoint: web/server/
          - fallback: master

salt.pillar.git_pillar.ext_pillar(minion_id, pillar, *repos)

  Checkout the ext_pillar sources and compile the resulting pillar SLS