{"id":419,"date":"2019-10-11T13:26:52","date_gmt":"2019-10-11T17:26:52","guid":{"rendered":"https:\/\/www.bu.edu\/engit\/?page_id=419"},"modified":"2019-11-12T17:15:52","modified_gmt":"2019-11-12T22:15:52","slug":"linux-configuration-management-with-ansible","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/engit\/knowledge-base\/linux\/linux-configuration-management-with-ansible\/","title":{"rendered":"Linux Configuration Management with Ansible"},"content":{"rendered":"<div>\n<div class=\"table-of-contents\">\n<p class=\"table-of-contents-heading\">Contents<\/p>\n<ol>\n<li><a href=\"#LinuxConfigurationManagementwithAnsible\">Linux Configuration Management with Ansible<\/a><\/li>\n<\/ol>\n<\/div>\n<ol>\n<li><a href=\"#One-timeSetup\">One-time Setup<\/a><\/li>\n<li><a href=\"#UsageExamples.3Aone-offcommands\">Usage Examples: one-off commands<\/a><\/li>\n<li><a href=\"#UsageExamples.3AOngoingconfigurationmanagement\">Usage Examples: Ongoing configuration management<\/a><\/li>\n<li><a href=\"#ManagingConfigurations\">Managing Configurations<\/a><\/li>\n<li><a href=\"#ImplementationDetails\">Implementation Details<\/a><\/li>\n<li><a href=\"#Links\">Links<\/a><\/li>\n<\/ol>\n<\/div>\n<h2 id=\"One-timeSetup\">One-time Setup<\/h2>\n<p>First, grab a working copy of the repository in your home directory. You should include both the github and eng_linux (NFS) remotes so you can keep both copies up to date.<\/p>\n<pre class=\"darkSnippet\">$ cd\r\n$ git clone git@github.com:eng-it\/ansible.git\r\n$ cd ansible\r\n$ git remote add eng_linux \/ad\/eng\/support\/software\/linux\/etc\/ansible.git<\/pre>\n<p>See the <a class=\"https\" href=\"https:\/\/github.com\/eng-it\/ansible\/blob\/master\/README.md\">README.md<\/a> file there for more details on setting up the environment.<\/p>\n<h2 id=\"UsageExamples.3Aone-offcommands\">Usage Examples: one-off commands<\/h2>\n<p>These examples show some mpc-like tasks to run commands, copy over files, etc. The lab names are the &#8220;host groups&#8221; specified in <tt>~\/ansible\/hosts<\/tt>. There are four meta-groups: grid (all grid server machines), instruction (instruction lab computers), research (research lab workstations), and servers (ENG-IT Linux servers).<\/p>\n<p>Basic arguments:<\/p>\n<ul>\n<li style=\"list-style-type: none;\"><tt>-m<\/tt> specifies what ansible module to use (default is &#8220;command&#8221;. To use shell things like pipes and redirects specify &#8220;-m shell&#8221;)<\/li>\n<li style=\"list-style-type: none;\"><tt>-a<\/tt> specifies the arguments for the module (use quotes to group spaces)<\/li>\n<li style=\"list-style-type: none;\"><tt>-o<\/tt> will keep per-host output to one line.<\/li>\n<\/ul>\n<p>Check the uptime report for each compsim system:<\/p>\n<pre class=\"darkSnippet\">$ ansible compsim -a uptime -o\r\nbme-compsim-1 | success | rc=0 | (stdout)  16:23:27 up 91 days,  1:54,  4 users,  load average: 1.00, 1.00, 1.00\r\nbme-compsim-5 | success | rc=0 | (stdout)  16:23:27 up 13 days,  2:29,  3 users,  load average: 1.03, 1.00, 1.00\r\nbme-compsim-4 | success | rc=0 | (stdout)  16:23:27 up 134 days,  1:07,  3 users,  load average: 1.07, 1.02, 1.00\r\nbme-compsim-7 | success | rc=0 | (stdout)  16:23:29 up 26 days,  3:30,  5 users,  load average: 1.05, 1.04, 1.01\r\n...<\/pre>\n<p>Copy a configuration file to each signals system. If the file was already there it will report that as changed: false.<\/p>\n<pre class=\"darkSnippet\">$ ansible signals -m copy -a \"src=~\/something.conf dest=\/etc\/something.conf\" -o\r\n\r\nsignals11 | success &gt;&gt; {\"changed\": true}\r\n\r\n\r\nsignals12 | success &gt;&gt; {\"changed\": true}\r\n\r\n\r\nsignals09 | success &gt;&gt; {\"changed\": false}\r\n\r\n...<\/pre>\n<h2 id=\"UsageExamples.3AOngoingconfigurationmanagement\">Usage Examples: Ongoing configuration management<\/h2>\n<p>These examples use ansible <em>playbooks<\/em> to track how hosts should be configured in general.<\/p>\n<p>Basic arguments:<\/p>\n<ul>\n<li style=\"list-style-type: none;\"><tt>--check<\/tt> will do its best to figure out what <em>would<\/em> be changed if the playbook were actually run.<\/li>\n<li style=\"list-style-type: none;\"><tt>--diff<\/tt> shows diffs of any changing files<\/li>\n<li style=\"list-style-type: none;\"><tt>--tags<\/tt> restricts the run to just tasks that have that tag listed.<\/li>\n<li style=\"list-style-type: none;\"><tt>--skip-tags<\/tt> does the reverse.<\/li>\n<\/ul>\n<p>Have all compsim machines re-join AD as needed:<\/p>\n<pre class=\"darkSnippet\">$ kinit jesse08-adm -c adm_ticket\r\nPassword for jesse08-adm@AD.BU.EDU:\r\n$ ansible-playbook grid.yml -l compsim --tags ad-client<\/pre>\n<p>Check what <em>would<\/em> be changed to bring all bungee systems up to date for the grid settings:<\/p>\n<pre class=\"darkSnippet\">$ ansible-playbook grid.yml -l bungee-nodes --tags grid-computenode --check<\/pre>\n<p>Apply all configuration steps defined for each of Ultra&#8217;s Linux workstations, from start to finish, but leaving out any lists of package installs. (This will still check and install packages for specific things, like AD, NFS, SSSD, etc., just won&#8217;t do all the locally-installed packages in the list.) In this example there are two workstations, with one currently unplugged.<\/p>\n<pre class=\"darkSnippet\">$ ansible-playbook research.yml -l ultra-workstations --skip-tags packages\r\n\r\nPLAY [ultra-workstations] *****************************************************\r\n\r\nGATHERING FACTS ***************************************************************\r\nfatal: [ece-pho810-02] =&gt; SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh\r\nok: [ece-pho810-03]\r\n\r\nTASK: [common | Ensure that all SSH keys in our list are present in \/root\/.ssh\/authorized_keys] ***\r\nok: [ece-pho810-03] =&gt; (item=\/ad\/eng\/users\/j\/e\/jesse08\/ansible\/roles\/common\/files\/ssh-keys\/batista.pub)\r\nok: [ece-pho810-03] =&gt; (item=\/ad\/eng\/users\/j\/e\/jesse08\/ansible\/roles\/common\/files\/ssh-keys\/jaredb.key)\r\nok: [ece-pho810-03] =&gt; (item=\/ad\/eng\/users\/j\/e\/jesse08\/ansible\/roles\/common\/files\/ssh-keys\/jesse08.key)\r\nok: [ece-pho810-03] =&gt; (item=\/ad\/eng\/users\/j\/e\/jesse08\/ansible\/roles\/common\/files\/ssh-keys\/jkgoebel.key)\r\nok: [ece-pho810-03] =&gt; (item=\/ad\/eng\/users\/j\/e\/jesse08\/ansible\/roles\/common\/files\/ssh-keys\/mskramer.pub)\r\n\r\nTASK: [common | Delete the annoying message for root logins] ******************\r\nok: [ece-pho810-03]\r\n\r\nTASK: [common | Set SELinux to permissive mode.] ******************************\r\nok: [ece-pho810-03]\r\n\r\n...\r\n\r\n\r\nPLAY RECAP ********************************************************************\r\n           to retry, use: --limit @\/home\/jesse08\/research-ece-ultra.retry\r\n\r\nece-pho810-02              : ok=0    changed=0    unreachable=1    failed=0\r\nece-pho810-03              : ok=53   changed=0    unreachable=0    failed=0<\/pre>\n<h2 id=\"ManagingConfigurations\">Managing Configurations<\/h2>\n<p>Here&#8217;s a general overview of how to update and manage the configuration files themselves, and coordinate any changes with others in ENG-IT. The descriptions here about how Ansible works just scratch the surface, but the official documentation is very thorough:<\/p>\n<ul>\n<li><a class=\"http\" href=\"http:\/\/docs.ansible.com\/\">http:\/\/docs.ansible.com\/<\/a><\/li>\n<li><a class=\"http\" href=\"http:\/\/docs.ansible.com\/YAMLSyntax.html\">http:\/\/docs.ansible.com\/YAMLSyntax.html<\/a><\/li>\n<\/ul>\n<h3 id=\"Playbooks.2CRoles.2CTasks.2CandModules\">Playbooks, Roles, Tasks, and Modules<\/h3>\n<p>All configuration data for Ansible is stored in <a class=\"http\" href=\"http:\/\/yaml.org\/\">YAML<\/a> format, with strings, lists, dictionaries, and so on stored in a compact tree structure of simple text. At the lowest level Ansible uses a set of <em>modules<\/em> called within config files to actually implement tasks. Some common modules are command, shell, copy, mount, service, and yum (which all do basically what the names imply).<\/p>\n<p>To actually use the modules, <em>tasks<\/em> give the modules arguments about what to do. For example this standalone <em>playbook<\/em> uses the &#8220;file&#8221; module to ensure that two symlinks exist:<\/p>\n<pre class=\"darkSnippet\">---\r\n- hosts: all\r\n  tasks:\r\n    - file: state=link src=\/usr\/lib64\/libGL.so.1 dest=\/usr\/lib64\/libGL.so\r\n    - file: state=link src=\/usr\/lib64\/libGL.so.1.2.0 dest=\/usr\/lib64\/libGL.so.1<\/pre>\n<p>Playbooks can include other playbooks (see next section below) and some more parameters aside from tasks, but mainly group sets of tasks together.<\/p>\n<p>The last major concept is that of <em>roles<\/em>. A role is a directory containing all the configuration data and files defining a particular role a computer is meant to play or a service it provides. Ansible uses some conventions for paths inside the role directories that make it easy to refer to other files within the role, and roles can be linked into playbooks easily. All of this means roles are good for compartmentalizing all the details for each service. For example, all the details related to eng-grid-monitor.bu.edu&#8217;s configuration, from start to finish, can be summarized in servers.yml as shown below, with five roles in total. First there are roles that apply to all servers, and separately there is an entry to apply &#8220;ganglia-server&#8221; only to eng-grid-monitor.<\/p>\n<pre class=\"darkSnippet\">- hosts: servers\r\n  roles:\r\n    - common\r\n    - networking\r\n    - sshd\r\n    - ad-client\r\n\r\n# (... skipping ...)\r\n\r\n- hosts: eng-grid-monitor\r\n  roles:\r\n    - ganglia-server<\/pre>\n<p>For the full description of how the roles directories are organized see the <a class=\"http\" href=\"http:\/\/docs.ansible.com\/playbooks_roles.html\">Ansible role documentation<\/a>, and a brief example in the following section.<\/p>\n<h3 id=\"HowOurFilesareOrganized\">How Our Files are Organized<\/h3>\n<p>Start here for an overview of what&#8217;s actually inside our ansible configuration repository.<\/p>\n<h4 id=\"Thetopleveldirectory\">The top level directory<\/h4>\n<p>At the highest level there&#8217;s a <tt>site.yml<\/tt> file that is just a list containing all of the per-hostgroup playbooks.<\/p>\n<p>The <tt>inventory<\/tt> directory contains definitions for what hosts belong to what groups. Along with specific playbooks like <tt>research.yml<\/tt> this defines how specific hosts should be configured. Each role listed in a playbook corresponds to one of the sub-directories in the <tt>roles<\/tt> directory. If a dictionary is used for a role instead of just a role name, we can set variables that the role can use, like for the sshd role in this example.<\/p>\n<pre class=\"darkSnippet\">---\r\n- hosts: research\r\n  roles:\r\n    - common\r\n    - networking\r\n    - kace-client\r\n    - { role: sshd, sshd_restrict_to_admins: true }\r\n    - ad-client\r\n    - kerberized-nfs\r\n    - sssd\r\n    - software-admin\r\n    - software-scitech\r\n    - eng-shell-modules\r\n    - workstation\r\n    - printing<\/pre>\n<h4 id=\"Therolesdirectory\">The roles directory<\/h4>\n<p>For a quick reference of how a role is set up, look at <tt>roles\/sshd<\/tt>:<\/p>\n<pre class=\"darkSnippet\">$ ls -1 roles\/sshd\/*\r\nroles\/sshd\/defaults:\r\nmain.yml\r\n\r\nroles\/sshd\/handlers:\r\nmain.yml\r\n\r\nroles\/sshd\/tasks:\r\nmain.yml\r\nsshd_config.yml\r\n\r\nroles\/sshd\/templates:\r\nsshd_config<\/pre>\n<p><tt>tasks\/main.yml<\/tt> contains a block of comments describing what the role does, and a short block of <tt>include<\/tt> lines that include more specific sub-tasks. Tagging the includes with the role name make it easy to select specific roles when running ansible-playbook with the <tt>--tags<\/tt> argument, as shown below. <tt>sshd_config.yml<\/tt> contains tasks to actually enable and configure sshd.<\/p>\n<pre class=\"darkSnippet\">- include: sshd_config.yml\r\n  tags: [ sshd, sshd_config ]<\/pre>\n<p><tt>templates\/sshd_config<\/tt> is a sshd config file, but using Ansible&#8217;s jinja2 templating support. (It can insert variables into configuration files dynamically and contain flow control and loops and things.) Here we just use it to decide whether or not to include &#8220;<tt>AllowGroups\u00a0root\u00a0wheel<\/tt>&#8221; based on the variable sshd_restrict_to_admins.<\/p>\n<p><tt>defaults\/main.yml<\/tt> is a set of default variables the role will use. This way sshd_restrict_to_admins can be set to false by default, and then enabled for just those hosts where it&#8217;s needed.<\/p>\n<p><tt>handlers\/main.yml<\/tt> defines the &#8220;handlers&#8221; that tasks can &#8220;notify&#8221; (see sshd_config.yml) that something has changed and a service should be restarted. It&#8217;s done separately from the tasks so that multiple changes might notify a handler, but it runs just once after the tasks are complete.<\/p>\n<pre class=\"darkSnippet\">---\r\n- name: reload sshd\r\n  service: name=sshd state=reloaded<\/pre>\n<h3 id=\"CoordinatingChangeswithinENG-IT\">Coordinating Changes within ENG-IT<\/h3>\n<p>There are two main branches always in the git repository: master (for production use, and the default for <a href=\"\/engit\/it\/eng-deploy\">auto-installs<\/a>) and testing:<\/p>\n<pre class=\"darkSnippet\">$ git branch -a\r\n  master\r\n* testing\r\n  remotes\/eng_linux\/master\r\n  remotes\/eng_linux\/testing<\/pre>\n<p>One strategy is to branch off from testing to add specific features, merge them back in as they&#8217;re finished, and then merge testing back into master when it&#8217;s stable. <a class=\"http\" href=\"http:\/\/nvie.com\/posts\/a-successful-git-branching-model\/\">This article<\/a> presents the idea nicely. With that approach we can add new roles or features in specifically-named branches, and only impact others&#8217; use when we&#8217;re sure the code is ready and agree to merge it in.<\/p>\n<p>Quick tip: before committing anything to anywhere, check the YAML syntax across all playbooks included via site.yml. This won&#8217;t actually connect to anywhere or run anything but it will sanity-check your YAML syntax.<\/p>\n<pre class=\"darkSnippet\">$ ansible-playbook --check-syntax site.yml<\/pre>\n<h4 id=\"AcontrivedexampleforaddinganAFSclientconfiguration\">A contrived example for adding an AFS client configuration<\/h4>\n<p>(First create a new branch off of testing for the new feature.)<\/p>\n<pre class=\"darkSnippet\">$ git checkout -b afs testing\r\nSwitched to a new branch 'afs'\r\n$ git branch -a\r\n* afs\r\n  master\r\n  testing\r\n  remotes\/eng_linux\/master\r\n  remotes\/eng_linux\/testing<\/pre>\n<p>(So, the &#8220;afs&#8221; branch only exists in this copy of the repo. Now add files and commit them.)<\/p>\n<pre class=\"darkSnippet\">$ git status\r\n# On branch afs\r\n# Untracked files:\r\n#   (use \"git add &lt;file&gt;...\" to include in what will be committed)\r\n#\r\n#       roles\/afs\/\r\nnothing added to commit but untracked files present (use \"git add\" to track)\r\n$ git add roles\/afs\/\r\n$ git status\r\n# On branch afs\r\n# Changes to be committed:\r\n#   (use \"git reset HEAD &lt;file&gt;...\" to unstage)\r\n#\r\n#       new file:   roles\/afs\/files\/CellAlias\r\n#       new file:   roles\/afs\/files\/CellServDB\r\n#       new file:   roles\/afs\/files\/ThisCell\r\n#       new file:   roles\/afs\/tasks\/afs.yml\r\n#       new file:   roles\/afs\/tasks\/main.yml\r\n#\r\n$ git commit -m 'AFS client configuration for BU network'\r\n[afs c32e9c4] AFS client configuration for BU network\r\n 5 files changed, 16 insertions(+), 0 deletions(-)\r\n create mode 100644 roles\/afs\/files\/CellAlias\r\n create mode 100644 roles\/afs\/files\/CellServDB\r\n create mode 100644 roles\/afs\/files\/ThisCell\r\n create mode 100644 roles\/afs\/tasks\/afs.yml\r\n create mode 100644 roles\/afs\/tasks\/main.yml<\/pre>\n<p>(Now merge that local branch into the real testing branch, and delete the local branch.)<\/p>\n<pre class=\"darkSnippet\">$ git checkout testing\r\nSwitched to branch 'testing'\r\n$ git merge --no-ff afs\r\n$ git branch -d afs\r\n$ git push eng_linux<\/pre>\n<h2 id=\"ImplementationDetails\">Implementation Details<\/h2>\n<p>A self-contained Ansible installation is on the network, installed to <tt>\/ad\/eng\/opt\/64\/ansible<\/tt>. This was done with a dedicated <a class=\"http\" href=\"http:\/\/conda.pydata.org\/miniconda.html\">miniconda<\/a> install, with ansible on top of that. This can be updated by running <tt>\/ad\/eng\/support\/software\/linux\/opt\/64\/ansible\/bin\/pip\u00a0install\u00a0--upgrade\u00a0ansible<\/tt> (note the long path for the read\/write mountpoint). The definitive configuration data is kept as a bare git repository in <tt>\/ad\/eng\/support\/software\/linux\/etc\/ansible.git<\/tt> and at <a class=\"https\" href=\"https:\/\/github.com\/eng-it\/ansible\">https:\/\/github.com\/eng-it\/ansible<\/a>.<\/p>\n<h2 id=\"Links\">Links<\/h2>\n<ul>\n<li><a class=\"http\" href=\"http:\/\/docs.ansible.com\/ansible\/list_of_all_modules.html\">http:\/\/docs.ansible.com\/ansible\/list_of_all_modules.html<\/a><\/li>\n<li><a class=\"http\" href=\"http:\/\/www.servermanaged.it\/ansible\/ansible-simple-rollback-strategy\/\">http:\/\/www.servermanaged.it\/ansible\/ansible-simple-rollback-strategy\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Contents Linux Configuration Management with Ansible One-time Setup Usage Examples: one-off commands Usage Examples: Ongoing configuration management Managing Configurations Implementation Details Links One-time Setup First, grab a working copy of the repository in your home directory. You should include both the github and eng_linux (NFS) remotes so you can keep both copies up to date. [&hellip;]<\/p>\n","protected":false},"author":16541,"featured_media":0,"parent":868,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/419"}],"collection":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/users\/16541"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/comments?post=419"}],"version-history":[{"count":3,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/419\/revisions"}],"predecessor-version":[{"id":687,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/419\/revisions\/687"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/868"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/media?parent=419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}