class: center, middle # Chef Development Workflow --- # Policy Objects - role + cookbooks + environment = policy ## Role Roles define: - attributes - run list ★ ## Environment Environments define: - attributes - cookbook versions ★ --- # Policy Objects - Role defines the cookbook(s) to be used - Environment defines the version of a cookbook to use - Versioning is important because different versions of a cookbook may have different features and capabilities - This is particularly critical when using 3rd party (i.e. community) cookbooks as we don't control releases > _N.B.:_ A node may have multiple roles! --- # Cookbook Patterns ## _role cookbook_ pattern - the _role_ object is defined on the server with only the _role recipe_'s default recipe in the run list - the _role recipe_ runs additional recipes as necessary to configure the components for the role ## _environment cookbook_ pattern - Defines dependencies only on role cookbooks - Used to resolve cookbook dependencies within an environment - `Berksfile.lock` is checked into SCM > This is important- by checking in a working lock file you can > revert to a previous environment should something break --- # Role Cookbooks A role cookbook is named for the role object it supports. - a role "server" will have a cookbook named "role" behind it - the "role" cookbook contains all the necessary recipes to configure the node - the role object will set its run list to the default recipe of the role cookbook Roles get stacked up: - we have a "cit-base" role that configures a minimum of things that are globally acceptable for any node on the network - ntp - dns - then a "scicomp-base" role that configures the minimum for a node to be managed by SciComp: - sudoers - ssh keys --- # Role Cookbooks The role object on the server: ``` $ knife role show scicomp-base chef_type: role default_attributes: description: Minimal configuration to participate on Hutch Network env_run_lists: json_class: Chef::Role name: scicomp-base override_attributes: run_list: recipe[scicomp-base] ``` Note that the run list is more precisely indicated as: ``` run_list: recipe[scicomp-base::default] ``` --- # Role Cookbooks The recipe `scicomp-base/recipes/default.rb`: ``` # . # Cookbook Name:: scicomp-base # Recipe:: default # . ... include_recipe 'nscd::default' include_recipe 'sudo::default' include_recipe 'krb5::default' include_recipe 'scicomp-base::keys' # Create privilege separation directory directory '/var/run/sshd' do owner 'root' group 'root' mode '0755' action :create end ... ``` --- # Role Cookbooks # Development - Check out, branch, and hack away - When tests succeed and the features are done: - merge to prod - version - upload itself and dependencies to server - upload to kwik-e-mart --- # Environment Cookbooks - The environment cookbook has only a `metadata.rb` file containing the role cookbooks that will operate inside this environment ## Example - Assume one environment for database servers and another for web servers - The *database* environment would have roles: - base - db\_server - The *web* environment would have roles: - base - apache - nginx > The base role has rules common to any host. Though note that the database > and web environments could use different versions of the base role cookbook. --- # Environment Cookbooks `scicomp-env-database/metadata.rb`: ``` name 'scicomp-env-database' version '0.1.0' depends 'scicomp-role-base', '= 1.0.2' depends 'scicomp-role-db_server', '= 2.1.7' ``` `scicomp-env-web/metadata.rb`: ``` name 'scicomp-env-web' version '0.5.2' depends 'scicomp-role-base', '= 0.9.5' depends 'scicomp-role-apache', '= 4.0.1' depends 'scicomp-role-nginx', '= 2.2.3' ``` --- # Environment Cookbooks ## Environment Branches - Each branch has a corresponding environment on the server - An environment cookbook `scicomp-env-web` with branches `prod` and `dev`, there would have these environment objects on the Chef server: - `scicomp-env-web` (the `prod` branch doesn't get a suffix) - `scicomp-env-web-dev` > The "-dev" environment has different version pins, presumable targetting > newer versions of cookbooks with additional features --- # Environment Cookbooks The environment for most SciComp nodes: ``` $ knife environment show scicomp-env-compute chef_type: environment cookbook_versions: apt: = 4.0.1 apt-chef: = 1.0.0 ... java: = 1.39.0 scicomp-base: = 0.1.7 scicomp-env-compute: = 0.1.2 scicomp-roles: = 1.2.8 selinux: = 0.9.0 seven_zip: = 2.0.1 sudo: = 2.9.0 sysctl: = 0.7.5 vim: = 2.0.1 windows: = 1.44.1 yum: = 3.11.0 yum-epel: = 0.7.0 default_attributes: description: Environment for Scientific Computing Compute Hosts json_class: Chef::Environment name: scicomp-env-compute ``` --- # Berkshelf Management `berkshelf` manages cookbook dependencies, allows liberal use of 3rd party cookbooks without losing one's sanity - Using community cookbooks quickly leads to dependency hell - Declare dependencies and let `berks` sort out which versions are required --- # Berkshelf Management ## Basic `Berksfile`: - created by `chef generate cookbook` ``` source 'https://supermarket.chef.io' metadata ``` - indicates url of a server running the supermarket API - indicates that dependencies are located in the `metadata.rb` file ``` source 'https://knife.fhcrc.org' source 'https://supermarket.chef.io' metadata ``` - this file adds the URL of the local supermarket with our cookbooks --- # Berkshelf Management - `berks` reads `metadata.rb` and starts walking the dependencies - first the ones declared in the file - then the dependencies for each of those - iterate till no more dependencies declared  > This graph is generated by `berks viz`, using `graphviz` as the back-end for > creating the image --- # Berkshelf Management ## Building the Lockfile - `Berksfile.lock` is a file containing a graph indicating cookbooks and versions required - The text version of the image shown in the last slide - `berks install` and `berks update` create and manage the file ## Artifact Management - `berks upload` uploads the cookbook to the server and freezes it - `berks apply <environment>` applies the versions to the environment --- # Working with Chef - The exact steps depend on the need ## New Hosts - Adding new hosts to existing roles- just bootstrap `knife bootstrap --run-list role[cit-base] ...` ## Changes to Existing Roles - Update the cookbook(s) and check in changes - Rev the cookbook and create a release - Update the environment with the new role version - Deploy the environment --- # Working with Chef ## New Applications - Create a role that configures _just_ the application - Rely on base roles to configure the basic stuff --- # Automation - Replace manual steps with one action: `git push` ## Current State - Role, Application, and Library cookbooks built with Jenkins - Check the "automatic-garbanzo" github project - Jenkins responds to Github webhook, does heavy lifting - Multibranch job "builds" each branch, deploys artifacts for "prod" branch - Role cookbooks need an additional step, the creation of the policy object - Environment cookbooks aren't automated yet (but close) - Likely very similar to the automatic-garbanzo - Instead of artifact upload, `berks apply` - Automating Jenkins is not trivial ☹ --- # Closing Notes  - Think like a developer... - No more isolated changes, rather releases - Think about collaboration with application devs - Think about handing off management to application devs!