TSM - Configuration management with Chef
Andrei Ivanov
- DevOps Engineer @Endava
Building large scale applications is a very rewarding thing. However, the maintenance and scaling process/part of such apps can be quite challenging sometimes. The applications and the operating system can each have different, possible configurations to fit the environment they are deployed in. Moreover, they have to be consistent across the development, the QA and the production stages, and point out to the information relevant at each stage. For example, the information contained in a database URL will be different than that of a URL used in the development stage, which is obviously different than the one used in production. Some of this information is not available for all the people involved in the development process, because they should not care about it either.
There are many configuration management tools to configure applications across environments and one of them is *Chef*. The definition on official website reads: "Chef is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes"."
### Chef concepts
Some of **Chef**'s essential components are:
\- **Nodes** - A node can be anything from a physical machine, a virtual one, a network device that is managed by **Chef**
\- **Chef server** - This is the central part of the product, and it is the place where the configuration data is stored. The nodes use **Chef**-client to get the proper configuration and deploy it accordingly.
\- **Workstation** - This is a computer that is used to configure **Chef**-related tasks via a set of command line tools, a computer that interacts with the **Chef** server and nodes, also allowing developers to write cookbooks.
\- **Chef client** - This is an agent that sits on every node that **Chef** is managing and which will handle the actual deployment part.
\- **Cookbooks** - These can be defined as the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything required to support that scenario.
\- **[Supermarket](https://supermarket.chef.io)** - This is the community site for cookbooks. Some cookbooks in the supermarket are developed and maintained by **Chef**. For some simple tasks, like installing **Apache** or **iptables**, it is better not to reinvent the wheel and get a tested, maintained ready to go cookbook which you can find in the supermarket.
\- **Others** (compliance, analytics, delivery) - These are just some other components, used in **Chef**, which we are going to leave aside for now.
The **Chef** architecture overview:
[ ](images/articles/tsm46/c1.png)
### The practical scenario
In this article, we will get straight into the practical part, by setting up a very simple scenario. We will take 2 virtual machines in the **aws** cloud. Then, with the help of a basic **Chef** cookbook, we will be deploying a simple **http** server and a static webpage, both of them displaying some server and environment information. Then, we will update the cookbook to dynamically search the environment and generate an html image tag pointing to the http location of the second web server. At the end we will simulate an environment change (from dev to prod). Sounds pretty simple, doesn't it?
### Prerequisites:
This example will be done using:
\- **Linux**, **Vagrant**, **virtualbox/aws**
\- **Chef** server, **chefdk** installed on the workstation
### Implementation
First, let's create a Chef environment called **chefdemodev** in which we're going to deploy some services.
```
knife environment create chefdemodev
```
When issuing the previous command, **Chef** will be filling in some defaults for us. Note that all the configuration in **Chef** is expressed in **json** format.
```
{
"name": "chefdemodev",
"description": "",
"cookbook_versions": {
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
}
}
```
For this example we will provision 2 machines in **aws** using **Vagrant**. **Vagrant** will undertake the task of setting up these instances, everything from talking to the cloud provider to setting up the **Chef** bootstrapping (installing **Chef**-client on the node and adding that node to the **Chef** server). The relevant **Vagrant** file contents are listed below. Notice that the **Chef** environment has the same value for both instances.
```
ENVIRONMENT = "chefdemodev"
VMNAME1 = "webserver01"
VMNAME2 = "webserver02"
VIRTUAL_MACHINES={
"#{ENVIRONMENT}-#{VMNAME1}" => {
:hostname => "#{ENVIRONMENT}-#{VMNAME1}.altidev.net",
:subnetid => "#{SUBNET}",
:instancetype => "#{VMTYPE}",
:ami => "#{VMAMI}", #centos image
:securitygroup => "#{SECGROUP}",
:role => ['all'],
:environment => "#{ENVIRONMENT}",
:recipe => ['iptables::disabled'],
:tags => {
:owner => "#{AWS_TAG_OWNER}",
:group => "#{AWS_TAG_GROUP}",
:cost_center => "#{AWS_TAG_COST_CENTER}",
:environment => "#{ENVIRONMENT}"
}
},
"#{ENVIRONMENT}-#{VMNAME2}" => {
:hostname => "#{ENVIRONMENT}-#{VMNAME2}.altidev.net",
:subnetid => "#{SUBNET}",
:instancetype => "#{VMTYPE}",
:ami => "#{VMAMI}", #centos image
:securitygroup => "#{SECGROUP}",
:role => ['all'],
:environment => "#{ENVIRONMENT}",
:recipe => ['iptables::disabled'],
:tags => {
:owner => "#{AWS_TAG_OWNER}",
:group => "#{AWS_TAG_GROUP}",
:cost_center => "#{AWS_TAG_COST_CENTER}",
:environment => "#{ENVIRONMENT}"
}
}
}
```
We are going to bring those 2 vm's up:
```
vagrant up
Bringing machine 'chefdemodev-webserver01' up with 'aws' provider...
Bringing machine 'chefdemodev-webserver02' up with 'aws' provider...
```
We can also verify the status of those vm's:
```
vagrant status
Current machine states:
chefdemodev-webserver01 running (aws)
chefdemodev-webserver02 running (aws)
```
Let's come back to the **Chef** part. Having those machines provisioned for us by **Vagrant**, we can also see them added to the **Chef** server, in the **chefdemodev** environment:
```
knife node list --environment=chefdemodev
chefdemodev-webserver01.altidev.net
chefdemodev-webserver02.altidev.net
```
Next, we are going to set up the **Chef** environment with some basic settings, so that we can deploy a basic **Apache** service.
```
Knife environment edit chefdemodev
```
And we will add
```
"apache": {
"default_site_enabled": true
}
```
Then, we are going to create our cookbook called **chefdemoweb**. In order to do this, we are going to use a tool called **Berkshelf**. You can read more about **Berkshelf** at , since I will not go into the details here. **Berkshelf** is a dependency manager for cookbooks and it will also create a basic cookbook structure for us.
```
berks cookbook chefdemoweb
create chefdemoweb/files/default
create chefdemoweb/templates/default
……
```
If we inspect the **chefdemoweb** directory we'll see that **Berkshelf** has created this structure for us:
attributes
Berksfile
CHANGELOG.md
chefignore
files
default
Gemfile
libraries
LICENSE
metadata.rb
providers
README.md
recipes
default.rb
resources
templates
default
test
integration
default
Thorfile
Vagrantfile
Next, we'll add the following code to the default recipe ( recipe/default.rb), note that we are using a community cookbook named *apache2* to do the heavy lifting for us.
```
include_recipe 'apache2'
template "#{node['apache']['docroot_dir']}/index.html" do
source "index.html.erb"
owner 'root'
group node['apache']['root_group']
mode '0644'
variables(
:environment => node.chef_environment
)
end
service "httpd" do
action :restart
end
```
We will also add a template for our webpage in templates/default/index.html.erb
```
Chefdemo
Hi
This is <%= node['fqdn'] %>
This is the <%= @environment %> environment
```
We'll also add the apache2 dependency in the metadata.rb file. This will tell chef that this cookbook depends on the apache2 cookbook.
```
depends 'apache2'
```
Next, we'll use **Berkshelf** to resolve the dependencies between cookbooks and also upload the cookbook to the **Chef** server, so that all **Chef** nodes will be able to access it.
```
berks install && berks upload
Resolving cookbook dependencies...
Fetching 'chefdemoweb' from source at .
Using chefdemoweb (0.1.0) from source at .
Using apache2 (3.2.0)
Uploaded apache2 (3.2.0) to: 'https://chef.altidev.net:443/'
Uploaded chefdemoweb (0.1.0) to: 'https://chef.altidev.net:443/'
….
```
Now, into the previously created chefdemodev-webserver01, we will run our newly created **chefdemoweb** cookbook.
```
chef-client -o "recipe[chefdemoweb]"
```
A **Chef**-client run output will be quite large, so I am not going to include it here. However, we notice a few important lines that will tell us that the cookbook has configured a basic webpage, and restarted the webserver:
```
create new file /etc/httpd/sites-available/default.conf
update content in file /etc/httpd/sites-available/default.conf from none to 15a87b
update content in file /var/www/html/index.html
restart service service[apache2]
```
Opening the address in a browser will test out if everything worked as intended.
[ ](images/articles/tsm46/c2.png)
We will repeat the deployment for the second vm in the same way as above with a small change. We will set a *boolean* attribute on the node, **is_cdn** with the value **true**.
```
knife node edit chefdemodev-webserver02.altidev.net
"name": "chefdemodev-webserver02.altidev.net",
"chef_environment": "chefdemodev",
"normal": {
"is_cdn": true,
….
```
Now we will have the cookbook dynamically search the environment for the node that has the **is_cdn** attribute set, so the recipe becomes:
```
include_recipe 'apache2'
webcdn = ''
search("node", "chef_environment:#{node.chef_environment} AND is_cdn:true").each do |server|
webcdn << server['fqdn']
end
template "#{node['apache']['docroot_dir']}/index.html" do
source "index.html.erb"
owner 'root'
group node['apache']['root_group']
mode '0644'
variables(
:environment => node.chef_environment,
:webcdn => webcdn
)
end
```
The index.html.erb template will then become:
```
Chefdemo
Hi
This is <%= node['fqdn'] %>
This is the <%= @environment %> environment
```
Afterwards, we are going to increment the version in metadata.rb, do a berks install && berks upload once again and redeploy the cookbook with **Chef**-client -o "recipe[chefdemoweb]" on the chefdemodev-webserver01 vm. We can now see the change occurring in the **Chef** output.
```
- update content in file /var/www/html/index.html from 9cf426 to 336e5b
--- /var/www/html/index.html 2016-04-11 11:49:00.020707241 +0000
+++ /tmp/chef-rendered-template20160411-11512-67feql 2016-04-11 11:49:31.556190114 +0000
@@ -9,6 +9,7 @@
Hi
This is chefdemodev-webserver01.altidev.net
This is the chefdemodev environment
+