Ensure installation of same openstack client

Updated url of docker registry.

make a loop for more flexibility.

Introducing listen_ip variable that overrides

the default listen_ip.

make a loop for more flexibility.

Get a listen ip specifically for that host.

see if components have listen_ip defined before using ansible_default_ipv4

Make service files look for listen_ip variable.

implemented listen_ip variable here too

map to different port to avoid clashes.

Make PHYSICAL_INTERFACE_MAPPINGS variable...

instead of just one provuider interface.
it should contain something like:
physnet2:eth1,physnet3:eth2

add openstack client on machine that is running keystone

Added delay to check.

enable all services are started at boot

Inventory for gcc openstack03 all in one.

added volume for glance images

Added gcc all in one specific config.

Prevent an error when there is no secrets.yml..

to back up

removed reference to empty dir.

added empty meta/main.yml

And now with a list of roles

..and removed the list

removed reference to empty dir.

Added galera cluster support

When at least three database nodes are installed, the playbook will
install a galera cluster across them.

The galera cnf can be the same template across...

nodes.

made environment file for the service.

I am unable to reproduce systemctl set-environment to work as
advertized.

Reverted to updating init file by ansible.

entrypoint.sh of the mariadb container seems unable to cope when a blank
variable is passed by systemd.

give the galera master node some time

It seems to be nessecary to run in host mode..

for galera to work.

I misunderstood pause. need sleep.

Inventory for physical test cluster.

Added CINDER_PASSWORD

Make sure docker is started.

If docker was already installed but not running
it was not started.

fixed refernce to neutron controller

Added heat

Added port for metadata service

Passed metadata secret to be used in config.

Listen ip should be the machine's ip...

Added openstack client from repo.

changed name of subnet

added horizon

Changed rabbitmq default user to openstack.

This makes it no longer nesseccary to create a separate openstack user,
which is lost on rabbitmq restart.

Added sleep because hosts were usually..

not discovered.

Removed unnessecary port mapping

Making /dev/lvm available to container.

fixed os-test inventory

Make iscsi devices available

(needed to attach cinder volumes to machines)

command module no longer works with &&

add cinder to test setup

README.md

@@ -1,8 +1,42 @@
 # hpc-cloud
 
-This repository will contain playbooks to bring up openstack components inside docker containers.
+This repository contains playbooks to bring up openstack components inside docker containers.
+It makes use of ansible roles for the openstack components and the supporting infrastructure.
+The following roles are installed.
+
+### Openstack components.
+
+* keystone
+* glance-controller
+* horizon
+* neutron-controller
+* nova-controller
+* nova-compute
+* cinder-controller
+* cinder-storage
+
+### Auxilary components:
+
+* database (mariadb)
+* rabbitmq (cluster of three nodes)
+* memcached
+
+## Getting started:
+
+### Prerequisites:
+* A cluster of servers to install the components on.
+  * The machines running nova-compute and neutron-controller need a separate interface for neutron to use.
+* ubuntu 16.04 with python installed (usually already present).
+* Access to the webhost12.service.rug.nl docker repository.
+
+### Settings:
+Passwords need be added to `secrets.yml.topol` and it needs to be saved as `secrets.yml`.
+This can be done by running `./generate_secrets.py`.
+Optionally, one can encrypt the secrtets by running `ansible-vault encrypt secrets.yml`.
+
+
+### Secrets:
 
-It makes use of ansible roles.
 The roles can be set in the inventory file (hosts)
 
 To bring up one role, for instance keystone, use:

ansible.cfg

@@ -1,2 +1,2 @@
 [defaults]
-hostfile = hosts
+inventory = merlin

cinder-controller.yml

@@ -0,0 +1,9 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: cinder-controller
+  become: True
+  roles:
+     - cinder-controller

cinder-storage.yml

@@ -0,0 +1,9 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: cinder-storage
+  become: True
+  roles:
+     - cinder-storage

gcc-post-install.yml

@@ -0,0 +1,35 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: keystone
+  become: True
+  vars_files:
+    - settings.yml
+  tasks:
+    - name: copy public key
+      copy:
+        content: "{{ rsa_pub }}"
+        dest: /srv/keystone/root/id_rsa.pub
+    - name: post install configuration
+      command: docker exec -i keystone.service bash -c "source /root/admin-openrc.sh && {{ item }}"
+      with_items:
+        - openstack network create --share --external --provider-physical-network provider --provider-network-type vlan --provider-segment 985 vlan985
+        - >
+            openstack subnet create --subnet-range 172.23.34.0/24 --gateway 172.23.34.1
+            --network vlan985 --allocation-pool start=172.23.34.50,end=172.23.34.60
+            --dns-nameserver 8.8.4.4 vlan985_subnet
+        - openstack network create --share --external --provider-physical-network provider --provider-network-type vlan --provider-segment 16 vlan16
+        - >
+            openstack subnet create --subnet-range 195.169.22.0/23 --gateway 195.169.23.251
+            --network vlan16 --allocation-pool start=195.169.22.237,end=195.169.22.237
+            --dns-nameserver 8.8.4.4 vlan16_subnet
+
+        - openstack flavor create --ram 4096 --disk 40 --vcpus 2  "Molgenis Dual"
+        - openstack flavor create --ram 16384 --disk 40 --vcpus 4  "Molgenis Quad 16GB"
+        - openstack flavor create --ram 8192 --disk 40 --vcpus 4  "Molgenis Quad 8GB"
+
+        - openstack keypair create --public-key /root/id_rsa.pub adminkey
+
+

gcc-site.yml

@@ -0,0 +1,14 @@
+---
+- include: common.yml
+- include: rabbitmq.yml
+- include: memcached.yml
+- include: mariadb.yml
+- include: keystone.yml
+- include: glance-controller.yml
+- include: nova-controller.yml
+- include: neutron-controller.yml
+- include: cinder-controller.yml
+- include: cinder-storage.yml
+- include: nova-compute.yml
+- include: horizon.yml
+- include: gcc-post-install.yml

generate_secrets.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python
+
+"""
+Open the secrets.yml and replace all passwords.
+Original is backed up.
+"""
+
+from os import path
+import random
+import string
+from subprocess import call
+from yaml import load, dump
+
+try:
+    from yaml import CLoader as Loader, CDumper as Dumper
+except ImportError:
+    from yaml import Loader, Dumper
+
+# length of generated passwords.
+pass_length = 20
+
+with open('secrets.yml.topol', 'r') as f:
+    data = load(f, Loader=Loader)
+
+for key, value in data.iteritems():
+    data[key] = ''.join(
+        random.choice(string.ascii_letters + string.digits)
+        for _ in range(pass_length))
+
+# Make numbered backups of the secrets file.
+if path.isfile('secrets.yml'):
+    call(['cp', '--backup=numbered', 'secrets.yml', 'secrets.yml.bak'])
+
+with open('secrets.yml', 'w') as f:
+    dump(data, f, Dumper=Dumper, default_flow_style=False)

heat.yml

@@ -0,0 +1,9 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: heat
+  become: True
+  roles:
+     - heat

hosts

@@ -1,5 +1,9 @@
+# A demo cluster of three nodes.
+
 [databases]
+openstack01-node01
 openstack01-node02
+openstack01-node03
 
 [keystone]
 openstack01-node03
@@ -15,29 +19,20 @@ openstack01-node01
 openstack01-node02
 openstack01-node03
 
-#[cassandra]
-#openstack01-node[01:03]
-
-#openstack01-node01
-#
-#[next_cassandra]
-#openstack01-node02
-#openstack01-node03
-
 [memcached]
 openstack01-node03
 
-#[first_cassandra:vars]
-#run_options=""
-#
-#[next_cassandra:vars]
-#run_options="-e CASSANDRA_SEEDS=172.23.41.1"
-
 [neutron-controller]
-openstack01-node01 provider_interface_name=ens192
+openstack01-node01 physical_interface_mappings=provider:ens192
 
 [nova-controller]
 openstack01-node03
 
+[cinder-controller]
+openstack01-node03
+
+[cinder-storage]
+openstack01-node01 storage_volume=/dev/loop0
+
 [nova-compute]
-openstack01-node04 provider_interface_name=dummy0
+openstack01-node04 physical_interface_mappings=provider:enp4s0f0

mariadb.yml

@@ -4,3 +4,10 @@
   become: True
   roles:
       - mariadb
+  vars:
+      hostname_node0: "{{ hostvars[groups['databases'][0]]['ansible_hostname'] }}"
+      hostname_node1: "{{ hostvars[groups['databases'][1]]['ansible_hostname'] }}"
+      hostname_node2: "{{ hostvars[groups['databases'][2]]['ansible_hostname'] }}"
+      ip_node0: "{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
+      ip_node1: "{{ hostvars[groups['databases'][1]]['listen_ip'] | default(hostvars[groups['databases'][1]]['ansible_default_ipv4']['address']) }}"
+      ip_node2: "{{ hostvars[groups['databases'][2]]['listen_ip'] | default(hostvars[groups['databases'][2]]['ansible_default_ipv4']['address']) }}"

merlin

@@ -0,0 +1,37 @@
+[nova-compute]
+merlin-managementnode002 physical_interface_mappings=provider:eno3
+merlin-managementnode003 physical_interface_mappings=provider:eno3
+merlin-node001 physical_interface_mappings=provider:eno3
+merlin-node003 physical_interface_mappings=provider:eno3
+merlin-node004 physical_interface_mappings=provider:eno3
+
+[databases]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[rabbitmq]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[horizon]
+merlin-managementnode001
+
+[memcached]
+merlin-managementnode001
+
+[nova-controller]
+merlin-managementnode001
+
+[keystone]
+merlin-managementnode001
+
+[neutron-controller]
+merlin-managementnode001 physical_interface_mappings=provider:eno3
+
+[heat]
+merlin-managementnode001
+
+[glance-controller]
+merlin-managementnode001

meta/main.yml

@@ -0,0 +1 @@
+---

nuke.yml

@@ -0,0 +1,21 @@
+---
+# This playbook will reset the instalation to facilitate a new installation.
+# All data is lost!
+- hosts: all
+  become: True
+  name: Dummy to gather facts
+  tasks:
+    - name: Stop docker service
+      shell: "systemctl stop docker"
+    - name: Verify docker is stopped.
+      systemd:
+        name: docker
+        state: stopped
+    - name: remove volumes
+      shell: "rm -rf /srv"
+
+- hosts: cinder-storage
+  become: True
+  tasks:
+    - name: wipe cinder storage.
+      shell: "dd if=/dev/zero of={{ storage_volume }} bs=1M count=1 && sync"

openstack03

@@ -0,0 +1,35 @@
+[databases]
+openstack03
+
+[keystone]
+openstack03
+
+[glance-controller]
+openstack03
+
+[horizon]
+openstack03
+
+[rabbitmq]
+openstack03
+
+[memcached]
+openstack03
+
+[neutron-controller]
+openstack03 physical_interface_mappings=provider:enp4s0f0
+
+[nova-controller]
+openstack03
+
+[cinder-controller]
+openstack03
+
+[cinder-storage]
+openstack03 storage_volume=/dev/sdb1
+
+[nova-compute]
+openstack03 physical_interface_mappings=provider:enp4s0f0
+
+[all:vars]
+listen_ip=172.23.40.243

os-test

@@ -0,0 +1,37 @@
+# An all in one
+
+[databases]
+os-test
+
+[keystone]
+os-test
+
+[glance-controller]
+os-test
+
+[horizon]
+os-test
+
+[rabbitmq]
+os-test
+
+[memcached]
+os-test
+
+[neutron-controller]
+os-test physical_interface_mappings=provider:enp4s0f0
+
+[nova-controller]
+os-test
+
+[cinder-controller]
+os-test
+
+[cinder-storage]
+os-test storage_volume=/dev/sdb
+
+[nova-compute]
+os-test physical_interface_mappings=provider:enp4s0f0
+
+[all:vars]
+listen_ip=129.125.60.194

post-install.yml

@@ -0,0 +1,26 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: keystone
+  become: True
+  vars_files:
+    - settings.yml
+  tasks:
+    - name: copy public key
+      copy:
+        content: "{{ rsa_pub }}"
+        dest: /srv/keystone/root/id_rsa.pub
+    - name: post install configuration
+      command: docker exec -i keystone.service bash -c "source /root/admin-openrc.sh && {{ item }}"
+      with_items:
+        - openstack network create  --share --external  --provider-physical-network provider  --provider-network-type flat provider
+        - >
+            openstack subnet create --network provider
+            --allocation-pool start={{ allocation_pool['start'] }},end={{ allocation_pool['end'] }}
+            --dns-nameserver {{ dns_nameserver }} --gateway {{ gateway }} --subnet-range {{ subnet_range }} providersub
+        - openstack flavor create --id 0 --vcpus 1 --ram 64 --disk 1 m1.nano
+        - openstack keypair create --public-key /root/id_rsa.pub adminkey
+
+

roles/cassandra/tasks/main.yml

@@ -7,9 +7,14 @@
     mode: 644
     owner: root
     group: root
+
 - name: install service file
   command: systemctl daemon-reload
+
 - name: make sure service is started
   systemd:
     name: cassandra.service
     state: started
+
+- name: start service at boot.
+  command: systemctl reenable cassandra.service

roles/cinder-controller/tasks/main.yml

@@ -0,0 +1,62 @@
+# Build and install a docker image for cinder.
+---
+- name: include secrets
+  include_vars:
+    file: ../../secrets.yml
+    name: secrets
+
+- set_fact:
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-cinder-controller:latest
+    env_vars: >
+        -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}"
+        -e "CINDER_HOST={{ listen_ip | default(hostvars[groups['cinder-controller'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "CINDER_PASSWORD={{ secrets['CINDER_PASSWORD'] }}"
+        -e "CINDER_USER=cinder"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
+        -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
+        -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
+        -e "RABBIT_USER=openstack"
+
+- name: pull docker image
+  docker_image:
+    name: "{{ docker_image }}"
+  tags: pull
+
+- name: Make build and persistent directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    mode: 0777
+  with_items:
+      - /srv/cinder-controller
+      - /srv/cinder-controller/root
+
+- name: install service file.
+  template:
+    src: templates/cinder-controller.service
+    dest: /etc/systemd/system/cinder-controller.service
+    mode: 644
+    owner: root
+    group: root
+
+- name: start service at boot.
+  command: systemctl reenable cinder-controller.service
+
+- command: systemctl daemon-reload
+
+- name: Initialize database.
+  command: >
+             /usr/bin/docker run --rm
+             {{ env_vars }}
+             -v /srv/cinder-controller/root:/root \
+             {{ docker_image }} /etc/bootstrap.sh
+  tags: bootstrap
+
+- name: make sure service is started
+  systemd:
+    name: cinder-controller.service
+    state: restarted

roles/cinder-controller/templates/cinder-controller.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Openstack Glance Container
+After=docker.service
+Requires=docker.service
+
+[Service]
+TimeoutStartSec=0
+Restart=always
+ExecStartPre=-/usr/bin/docker kill %n
+ExecStartPre=-/usr/bin/docker rm %n
+ExecStart=/usr/bin/docker run --name %n \
+  {{ env_vars | replace('\n', '') }} \
+  -v /srv/cinder-controller/root:/root \
+  -p 8776:8776 \
+  {{ docker_image }}
+
+[Install]
+WantedBy=multi-user.target

roles/cinder-storage/tasks/main.yml

@@ -0,0 +1,68 @@
+# Build and install a docker image for cinder.
+---
+- name: include secrets
+  include_vars:
+    file: ../../secrets.yml
+    name: secrets
+  tags: vars
+
+- set_fact:
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-cinder-storage:latest
+    env_vars: >
+        -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}"
+        -e "CINDER_HOST={{ listen_ip | default(hostvars[groups['cinder-storage'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "CINDER_PASSWORD={{ secrets['CINDER_PASSWORD'] }}"
+        -e "CINDER_USER=cinder"
+        -e "GLANCE_HOST={{ listen_ip | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
+        -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
+        -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
+        -e "RABBIT_USER=openstack"
+  tags: vars
+
+- name: pull docker image
+  docker_image:
+    name: "{{ docker_image }}"
+  tags: pull
+
+- name: Make build and persistent directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    mode: 0777
+  with_items:
+      - /srv/cinder-storage
+      - /srv/cinder-storage/root
+
+- name: initial setup
+  command: >
+             /usr/bin/docker run --rm
+             --privileged
+             {{ env_vars }}
+             -v /srv/cinder-storage/root:/root \
+             -v "{{ storage_volume }}":/dev/cinder_storage_volume \
+             {{ docker_image }} /etc/bootstrap.sh
+  tags: bootstrap
+
+- name: install service file.
+  template:
+    src: templates/cinder-storage.service
+    dest: /etc/systemd/system/cinder-storage.service
+    mode: 644
+    owner: root
+    group: root
+  tags: systemd
+
+- command: systemctl daemon-reload
+  tags: systemd
+
+- name: start service at boot.
+  command: systemctl reenable cinder-storage.service
+
+- name: make sure service is started
+  systemd:
+    name: cinder-storage.service
+    state: restarted

roles/cinder-storage/templates/cinder-storage.service

@@ -0,0 +1,24 @@
+[Unit]
+Description=Openstack Cinder Storage container
+After=docker.service
+Requires=docker.service
+
+[Service]
+TimeoutStartSec=0
+Restart=always
+ExecStartPre=-/usr/bin/docker kill %n
+ExecStartPre=-/usr/bin/docker rm %n
+ExecStart=/usr/bin/docker run --name %n \
+  --privileged \
+  {{ env_vars | replace('\n', '') }} \
+  -v "/dev/cinder-volumes/":/dev/cinder-volumes \
+  -v /srv/cinder-storage/root:/root \
+  -v "{{ storage_volume }}":/dev/cinder_storage_volume \
+  -v "/dev/lvm":/dev/lvm \
+  -v "/srv/cinder-storage/volumes/:/var/lib/cinder/volumes/" \
+  -p 8777:8776 \
+  -p 3260:3260 \
+  {{ docker_image }}
+
+[Install]
+WantedBy=multi-user.target

roles/common/tasks/docker.yml

@@ -13,3 +13,8 @@
   with_items:
      - docker-engine
      - python-docker
+
+- name: make sure service is started
+  systemd:
+    name: docker.service
+    state: started

roles/dockerregistry/tasks/main.yml

@@ -7,13 +7,18 @@
     mode: 644
     owner: root
     group: root
+
 - name: install service file
   command: systemctl daemon-reload
+
 - name: make sure service is started
   systemd:
     name: dockerregistry.service
     state: started
 
+- name: start service at boot.
+  command: systemctl reenable dockerregistry.service
+
 - name: Copy certificates and passwd file
   copy:
     src: "{{ item }}"

roles/glance-controller/tasks/main.yml

@@ -6,17 +6,17 @@
     name: secrets
 
 - set_fact:
-    docker_image: webhost12.service.rug.nl/hpc/openstack-glance:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-glance:latest
     env_vars: >
-        -e "GLANCE_HOST={{ hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "GLANCE_HOST={{ listen_ip | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "GLANCE_PASSWORD={{ secrets['GLANCE_PASSWORD'] }}"
         -e "GLANCE_USER=glance"
-        -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
         -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
         -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
-        -e "RABBIT_HOST={{ hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
         -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
         -e "RABBIT_USER=openstack"
 
@@ -42,14 +42,18 @@
     owner: root
     group: root
 
+- name: start service at boot.
+  command: systemctl reenable glance.service
+
 - command: systemctl daemon-reload
 
 - name: Initialize database.
   command: >
              /usr/bin/docker run --rm
              {{ env_vars }}
-             --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }} \
+             --add-host=keystone:{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
              -v /srv/glance/root:/root \
+             -v /var/lib/glance/images:/var/lib/glance/images \
              {{ docker_image }} /etc/bootstrap.sh
   tags: bootstrap
 

roles/glance-controller/templates/glance.service

@@ -11,6 +11,7 @@ ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
   {{ env_vars | replace('\n', '') }} \
   -v /srv/glance/root:/root \
+  -v /var/lib/glance/images:/var/lib/glance/images \
   -p 9292:9292 \
   {{ docker_image }}
 

roles/heat/tasks/main.yml

@@ -0,0 +1,62 @@
+# Build and install a docker image for heat.
+---
+- name: include secrets
+  include_vars:
+    file: ../../secrets.yml
+    name: secrets
+
+- set_fact:
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-heat:latest
+    env_vars: >
+        -e "HEAT_HOST={{ listen_ip | default(hostvars[groups['heat'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "HEAT_PASSWORD={{ secrets['HEAT_PASSWORD'] }}"
+        -e "HEAT_USER=heat"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
+        -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
+        -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
+        -e "RABBIT_USER=openstack"
+
+- name: pull docker image
+  docker_image:
+    name: "{{ docker_image }}"
+  tags: pull
+
+- name: Make build and persistent directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    mode: 0777
+  with_items:
+      - /srv/heat
+      - /srv/heat/root
+
+- name: install service file.
+  template:
+    src: templates/heat.service
+    dest: /etc/systemd/system/heat.service
+    mode: 644
+    owner: root
+    group: root
+
+- name: start service at boot.
+  command: systemctl reenable heat.service
+
+- command: systemctl daemon-reload
+
+- name: Initialize database.
+  command: >
+             /usr/bin/docker run --rm
+             {{ env_vars }}
+             --add-host=keystone:{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
+             -v /srv/heat/root:/root \
+             {{ docker_image }} /etc/bootstrap.sh
+  tags: bootstrap
+
+- name: make sure service is started
+  systemd:
+    name: heat.service
+    state: restarted

roles/heat/templates/heat.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=Openstack heat Container
+After=docker.service
+Requires=docker.service
+
+[Service]
+TimeoutStartSec=0
+Restart=always
+ExecStartPre=-/usr/bin/docker kill %n
+ExecStartPre=-/usr/bin/docker rm %n
+ExecStart=/usr/bin/docker run --name %n \
+  {{ env_vars | replace('\n', '') }} \
+  -v /srv/heat/root:/root \
+  -p 8000:8000 \
+  -p 8004:8004 \
+  {{ docker_image }}
+
+[Install]
+WantedBy=multi-user.target

roles/horizon/tasks/main.yml

@@ -1,11 +1,12 @@
 # Run hpc/horizon
 ---
 - set_fact:
-    docker_image: webhost12.service.rug.nl/hpc/openstack-horizon:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-horizon:latest
 
 - name: pull docker image
   docker_image:
     name: "{{ docker_image }}"
+    force: True
   tags: pull
 
 - name: install service file.
@@ -18,6 +19,9 @@
 
 - command: systemctl daemon-reload
 
+- name: start service at boot.
+  command: systemctl reenable horizon.service
+
 - name: make sure service is started
   systemd:
     name: horizon.service

roles/horizon/templates/horizon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=Openstack Glance Container
+Description=Openstack Horizon Container
 After=docker.service
 Requires=docker.service
 
@@ -9,9 +9,9 @@ Restart=always
 ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
-  -e "MEMCACHED_SERVER={{ hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address'] }}" \
+  -e "MEMCACHED_SERVER={{ hostvars[groups['memcached'][0]]['listen_ip'] | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}" \
-  -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}" \
+  -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}" \
-  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
   -p 80:80 \
   {{ docker_image }}
 

roles/keystone/tasks/main.yml

@@ -17,11 +17,12 @@
     - /srv/keystone/root
 
 - set_fact:
-    docker_image: webhost12.service.rug.nl/hpc/openstack-keystone:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-keystone:latest
 
 - name: pull docker image
   docker_image:
     name: "{{ docker_image }}"
+    force: True
   tags: pull
 
 - name: install service file.
@@ -35,11 +36,14 @@
 - name: install service file
   command: systemctl daemon-reload
 
+- name: start service at boot.
+  command: systemctl reenable keystone.service
+
 - name: Initialize db
   script: scripts/initialize_db.sh
   environment:
     MYSQL_ROOT_PASSWORD: "{{ secrets['MYSQL_ROOT_PASSWORD'] }}"
-    DB_HOST: "{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}"
+    DB_HOST: "{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
   register: result
   until: result|succeeded
   # sometimes the initial connect fails.
@@ -48,12 +52,12 @@
   delay: 3
   ignore_errors: yes
 
-- name: keystone manage commands to setup db
+- name: keystone manage commands to setup db_sync
   command: >
              /usr/bin/docker run --rm
-             --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}
+             --add-host=mariadb:{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}
              -v /srv/keystone/fernet-keys:/etc/keystone/fernet-keys
-             -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+             -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
              {{ docker_image }} keystone-manage {{ item }}
   with_items:
       - db_sync
@@ -61,10 +65,15 @@
       - credential_setup --keystone-user keystone --keystone-group keystone
       - >
           bootstrap --bootstrap-password {{ secrets['OS_PASSWORD'] }}
-                    --bootstrap-admin-url http://{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}:35357/v3/
+                    --bootstrap-admin-url http://{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}:35357/v3/
-                    --bootstrap-internal-url http://{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}:35357/v3/
+                    --bootstrap-internal-url http://{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}:35357/v3/
-                    --bootstrap-public-url http://{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}:5000/v3/
+                    --bootstrap-public-url http://{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}:5000/v3/
                     --bootstrap-region-id RegionOne
+  # sometimes the initial connect fails.
+  # Retry until it succeeds.
+  retries: 7
+  delay: 3
+  ignore_errors: yes
 
 - name: make sure service is started
   systemd:
@@ -74,10 +83,35 @@
 - name: Create a domain, projects users and roles
   command: >
              /usr/bin/docker run --rm
-             --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}
+             --add-host=mariadb:{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}
              -v /srv/keystone/fernet-keys:/etc/keystone/fernet-keys
              -v /srv/keystone/root:/root
-             -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+             -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
              -e "OS_AUTH_URL=http://${KEYSTONE_HOST}:35357/v3"
              -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
              {{ docker_image }} bash /etc/bootstrap.sh
+  register: result
+  retries: 7
+  delay: 3
+
+- name: install openstack repo host key.
+  command: apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5EDB1B62EC4926EA
+  tags: openstackclient
+
+- name: install openstack repo on host.
+  apt_repository:
+      repo: "deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main"
+      filename: ocata
+  tags: openstackclient
+
+- name: install openstack client for management
+  apt:
+    name: python-openstackclient
+    state: latest
+    update_cache: yes
+  tags: openstackclient
+
+- name: source admin-openrc.sh in root .bashrc
+  lineinfile:
+    path: /root/.bashrc
+    line: 'source /srv/keystone/root/admin-openrc.sh'

roles/keystone/templates/keystone.service

@@ -9,8 +9,8 @@ Restart=always
 ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
-  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }} \
-  -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}" \
+  -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}" \
   -p 5000:5000 -p 35357:35357 \
   -v /srv/keystone/fernet-keys:/etc/keystone/fernet-keys \
   -v /srv/keystone/root:/root \

roles/mariadb/files/galera.cnf

@@ -0,0 +1,20 @@
+[mysqld]
+binlog_format=ROW
+default-storage-engine=innodb
+innodb_autoinc_lock_mode=2
+bind-address=0.0.0.0
+
+# Galera Provider Configuration
+wsrep_on=ON
+wsrep_provider=/usr/lib/galera/libgalera_smm.so
+
+# Galera Cluster Configuration
+wsrep_cluster_name="test_cluster"
+wsrep_cluster_address="gcomm://{{ ip_node0 }},{{ ip_node1 }},{{ ip_node2 }}"
+
+# Galera Synchronization Configuration
+wsrep_sst_method=rsync
+
+# Galera Node Configuration
+wsrep_node_address="{{ listen_ip | default(ansible_default_ipv4.address) }}"
+wsrep_node_name="{{ ansible_nodename }}"

roles/mariadb/tasks/main.yml

@@ -5,14 +5,6 @@
     file: ../../secrets.yml
     name: secrets
 
-- name: install service file.
-  template:
-    src: templates/mysql.service
-    dest: /etc/systemd/system/mysql.service
-    mode: 644
-    owner: root
-    group: root
-
 - name: make mariadb settings volume
   file:
     path: "{{ item }}"
@@ -21,16 +13,60 @@
   with_items:
       - /srv/mariadb/lib/mysql
       - /srv/mariadb/etc/mysql
+      - /srv/mariadb/etc/mysql/conf.d
 
 - name: place settings file
   copy:
     src: files/my.cnf
-    dest: /srv/mariadb/etc/mysql
+    dest: /srv/mariadb/etc/mysql/conf.d/my.cnf
     mode: 660
 
-- command: systemctl daemon-reload
+- name: Set galara.cnf on node if we have at least three nodes.
+  template:
+    src: files/galera.cnf
+    dest: /srv/mariadb/etc/mysql/conf.d/galera.cnf
+    mode: 660
+  when: groups['databases'] | length >= 3
+
+  # This mimics galera_new_cluster.sh
+- name: Initialize a new cluster.
+  block:
+    - set_fact:
+        mariadb_args: "--wsrep-new-cluster"
+
+    - template:
+        src: templates/mysql.service
+        dest: /etc/systemd/system/mysql.service
+        mode: 644
+        owner: root
+        group: root
+
+    - command: systemctl daemon-reload
+
+    - systemd:
+        name: mysql.service
+        state: started
+
+  when: groups['databases'] | length >= 3 and ansible_hostname == hostname_node0
+
+- name: install service file.
+  block:
+    - set_fact:
+        mariadb_args: ""
+    - template:
+        src: templates/mysql.service
+        dest: /etc/systemd/system/mysql.service
+        mode: 644
+        owner: root
+        group: root
+
+- name: Give the master node some time to initialize the cluster.
+  command: bash -c "sleep 60 && systemctl daemon-reload"
 
 - name: make sure service is started
   systemd:
     name: mysql.service
     state: started
+
+- name: start service at boot.
+  command: systemctl reenable mysql.service

roles/mariadb/templates/mysql.service

@@ -9,10 +9,11 @@ Restart=always
 ExecStartPre=-/usr/bin/docker kill %n || /bin/true
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStartPre=/usr/bin/docker pull mariadb:10.2
-ExecStart=/usr/bin/docker run  -p 3306:3306 --name %n \
+ExecStart=/usr/bin/docker run --name %n \
+ --network host \
  -v /srv/mariadb/lib/mysql:/var/lib/mysql \
- -v /srv/mariadb/etc/mysql:/etc/mysql \
+ -v /srv/mariadb/etc/mysql/conf.d:/etc/mysql/conf.d \
- -e MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }} mariadb:10.2
+ -e MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }} mariadb:10.2 {{ mariadb_args }}
 
 [Install]
 WantedBy=multi-user.target

roles/memcached/tasks/main.yml

@@ -7,8 +7,13 @@
     mode: 644
     owner: root
     group: root
+
 - name: install service file
   command: systemctl daemon-reload
+
+- name: start service at boot.
+  command: systemctl reenable memcached.service
+
 - name: make sure service is started
   systemd:
     name: memcached.service

roles/neutron-controller/tasks/main.yml

@@ -6,28 +6,31 @@
     name: secrets
 
 - set_fact:
-    docker_image: "webhost12.service.rug.nl/hpc/openstack-neutron-controller:latest"
+    docker_image: "registry.webhosting.rug.nl/hpc/openstack-neutron-controller:latest"
 
 - name: pull docker image
   docker_image:
     name: "{{ docker_image }}"
+    force: True
   tags: pull
 
 - set_fact:
     env_vars: >
-        -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
         -e "METADATA_SECRET={{ secrets['METADATA_SECRET'] }}"
-        -e "MY_IP={{ hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MY_IP={{ listen_ip | default(hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
         -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
         -e "NEUTRON_PASSWORD={{ secrets['NEUTRON_PASSWORD'] }}"
         -e "NEUTRON_USER=neutron"
-        -e "NOVA_CONTROLLER_HOST={{ hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "NOVA_USER=nova"
+        -e "NOVA_PASSWORD={{ secrets['NOVA_PASSWORD'] }}"
+        -e "NOVA_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "NOVA_PLACEMENT_PASSWORD={{ secrets['NOVA_PLACEMENT_PASSWORD'] }}"
         -e "NOVA_PLACEMENT_USER=placement"
         -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
-        -e "PROVIDER_INTERFACE_NAME={{ provider_interface_name }}"
+        -e "PHYSICAL_INTERFACE_MAPPINGS={{ physical_interface_mappings }}"
         -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
         -e "RABBIT_USER=openstack"
   tags: env
@@ -42,12 +45,15 @@
 
 - command: systemctl daemon-reload
 
+- name: start service at boot.
+  command: systemctl reenable neutron-controller.service
+
 - name: Initialize neutron
   command: >
     /usr/bin/docker run --rm
     {{ env_vars }}
-    --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}
+    --add-host=mariadb:{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}
-    --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}
+    --add-host=keystone:{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}
     --network host
     {{ docker_image }}
     /etc/bootstrap.sh

roles/neutron-controller/templates/neutron-controller.service

@@ -10,9 +10,10 @@ ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
   {{ env_vars | replace('\n', '') }} \
-  --add-host=nova-controller:{{ hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=nova-controller:{{ hostvars[groups['nova-controller'][0]]['listen_ip'] | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }} \
-  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }} \
-  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
+  --add-host={{ ansible_nodename }}:{{ ansible_default_ipv4.address }} \
   --privileged \
   --network host \
   -v /lib/modules:/lib/modules \

roles/nova-compute/tasks/main.yml

@@ -4,14 +4,16 @@
   include_vars:
     file: ../../secrets.yml
     name: secrets
+  tags: vars
 
 - set_fact:
-    docker_image: webhost12.service.rug.nl/hpc/openstack-nova-compute:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-nova-compute:latest
-  tags: facts
+  tags: vars
 
 - name: pull docker image
   docker_image:
     name: "{{ docker_image }}"
+    force: True
   tags: pull
 
 - name: install service file.
@@ -21,11 +23,13 @@
     mode: 644
     owner: root
     group: root
+  tags: systemd
 
 - command: systemctl daemon-reload
+  tags: systemd
 
 - apt:
-    name: '{{ item }}'
+    name: "{{ item }}"
   with_items:
       - kvm
       - libvirt0
@@ -36,3 +40,10 @@
   systemd:
     name: nova-compute.service
     state: restarted
+
+- name: start service at boot.
+  command: systemctl reenable nova-compute.service
+
+- name: let nova controler discover new host
+  shell: "sleep 10 && docker exec -i nova-controller.service nova-manage cell_v2 discover_hosts"
+  delegate_to: "{{ hostvars[groups['nova-controller'][0]]['ansible_hostname'] }}"

roles/nova-compute/templates/nova-compute.service

@@ -9,30 +9,32 @@ Restart=always
 ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
-    -e "GLANCE_CONTROLLER_HOST={{ hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "GLANCE_CONTROLLER_HOST={{ hostvars[groups['glance-controller'][0]]['listen_ip'] | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}" \
-    -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}" \
-    -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "METADATA_SECRET={{ secrets['METADATA_SECRET'] }}" \
-    -e "MY_IP={{ hostvars[groups['nova-compute'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['listen_ip'] | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}" \
-    -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}" \
+    -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}" \
-    -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['neutron-controller'][0]]['listen_ip'] | default(hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "NEUTRON_PASSWORD={{ secrets['NEUTRON_PASSWORD'] }}" \
     -e "NEUTRON_USER=neutron" \
     -e "NOVA_COMPUTE_USER=nova_compute" \
-    -e "NOVA_CONTROLLER_HOST={{ hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "NOVA_CONTROLLER_HOST={{ hostvars[groups['nova-controller'][0]]['listen_ip'] | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "NOVA_PASSWORD={{ secrets['NOVA_PASSWORD'] }}" \
     -e "NOVA_PASSWORD={{ secrets['NOVA_PASSWORD'] }}" \
     -e "NOVA_PLACEMENT_PASSWORD={{ secrets['NOVA_PLACEMENT_PASSWORD'] }}" \
     -e "NOVA_PLACEMENT_USER=placement" \
     -e "NOVA_USER=nova" \
     -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}" \
-    -e "PROVIDER_INTERFACE_NAME={{ provider_interface_name }}" \
+    -e "PHYSICAL_INTERFACE_MAPPINGS={{ physical_interface_mappings }}" \
-    -e "RABBIT_HOST={{ hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address'] }}" \
+    -e "RABBIT_HOST={{ hostvars[groups['rabbitmq'][0]]['listen_ip'] | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}" \
     -e "RABBIT_USER=openstack" \
-    --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }} \
+    --add-host=keystone:{{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
-    --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }} \
+    --add-host=mariadb:{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }} \
     --privileged \
+    -v /dev:/dev \
     -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock \
     -v /var/lib/nova/instances:/var/lib/nova/instances \
     -v /lib/modules:/lib/modules \

roles/nova-controller/tasks/main.yml

@@ -15,18 +15,19 @@
     - /srv/nova-controller/root
 
 - set_fact:
-    docker_image: webhost12.service.rug.nl/hpc/openstack-nova-service:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-nova-service:latest
     env_vars: >
-        -e "GLANCE_CONTROLLER_HOST={{ hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "GLANCE_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "METADATA_SECRET={{ secrets['METADATA_SECRET'] }}"
-        -e "MY_IP={{ hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
         -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
-        -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['neutron-controller'][0]]['listen_ip'] | default(hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "NEUTRON_PASSWORD={{ secrets['NEUTRON_PASSWORD'] }}"
         -e "NEUTRON_USER=neutron"
-        -e "NOVA_CONTROLLER_HOST={{ hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address'] }}"
+        -e "NOVA_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "NOVA_PASSWORD={{ secrets['NOVA_PASSWORD'] }}"
         -e "NOVA_PLACEMENT_PASSWORD={{ secrets['NOVA_PLACEMENT_PASSWORD'] }}"
         -e "NOVA_PLACEMENT_USER=placement"
@@ -39,6 +40,7 @@
 - name: pull docker image
   docker_image:
     name: "{{ docker_image }}"
+    force: True
   tags: pull
 
 - name: install service file.
@@ -51,12 +53,15 @@
 
 - command: systemctl daemon-reload
 
+- name: start service at boot.
+  command: systemctl reenable nova-controller.service
+
 - name: Initialize database.
   command: >
     /usr/bin/docker run --rm
     {{ env_vars }}
-    --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }}
+    --add-host=mariadb:{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}
-    --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }}
+    --add-host=keystone:{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}
     -v /srv/nova-controller/root:/root
     {{ docker_image }}
     /etc/bootstrap.sh

roles/nova-controller/templates/nova-controller.service

@@ -10,11 +10,12 @@ ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
   {{ env_vars | replace('\n', '') }} \
-  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=mariadb:{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }} \
-  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address'] }} \
+  --add-host=keystone:{{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
   --privileged \
   -v /srv/nova-controller/root:/root \
   -p 8774:8774 \
+  -p 8775:8775 \
   -p 8778:8778 \
   -p 6080:6080 \
   {{ docker_image }} /etc/run.sh

roles/rabbitmq/files/rabbitmq.service

@@ -10,11 +10,11 @@ ExecStartPre=-/usr/bin/docker kill %n
 ExecStartPre=-/usr/bin/docker rm %n
 ExecStartPre=/usr/bin/docker pull rabbitmq:latest
 ExecStart=/usr/bin/docker run \
-          --add-host "{{ hostvars[groups['rabbitmq'][0]]['ansible_hostname'] }}:{{ hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address'] }}" \
+{% for host in groups['rabbitmq'] %}
-          --add-host "{{ hostvars[groups['rabbitmq'][1]]['ansible_hostname'] }}:{{ hostvars[groups['rabbitmq'][1]]['ansible_default_ipv4']['address'] }}" \
+          --add-host "{{ host }}:{{ hostvars[host]['listen_ip'] | default(hostvars[host]['ansible_default_ipv4']['address']) }}" \
-          --add-host "{{ hostvars[groups['rabbitmq'][2]]['ansible_hostname'] }}:{{ hostvars[groups['rabbitmq'][2]]['ansible_default_ipv4']['address'] }}" \
+{% endfor %}
           -p 4369:4369 -p 25679:25679 -p 25672:25672 -p 5671-5672:5671-5672 -p 8080:15672 \
-          -e "RABBITMQ_DEFAULT_USER=user" -e "RABBITMQ_DEFAULT_PASS={{ secrets['RABBIT_PASSWORD'] }}" \
+          -e "RABBITMQ_DEFAULT_USER=openstack" -e "RABBITMQ_DEFAULT_PASS={{ secrets['RABBIT_PASSWORD'] }}" \
           -e "RABBITMQ_ERLANG_COOKIE={{ secrets['RABBITMQ_ERLANG_COOKIE'] }}" \
           -e "RABBITMQ_NODENAME=rabbit_{{ ansible_nodename }}" \
           --hostname "{{ ansible_nodename }}" --name %n rabbitmq:3-management

roles/rabbitmq/tasks/main.yml

@@ -5,9 +5,6 @@
     file: ../../secrets.yml
     name: secrets
 
-- include_vars:
-    dir: 'vars'
-
 - name: install service file.
   template:
     src: files/rabbitmq.service
@@ -19,6 +16,9 @@
 - name: install service file
   command: systemctl daemon-reload
 
+- name: start service at boot.
+  command: systemctl reenable rabbitmq.service
+
 - name: make sure service is started
   systemd:
     name: rabbitmq.service
@@ -27,6 +27,7 @@
 - name: wait for container to be started
   wait_for:
       port: 5672
+      delay: 5
 
 - name: setup the cluster
   command: "docker exec -i rabbitmq.service {{ item }}"
@@ -36,11 +37,3 @@
       - rabbitmqctl start_app
   when: ansible_nodename != hostname_node0
 
-- name: create openstack user
-  command: "docker exec -i rabbitmq.service {{ item }}"
-  with_items:
-      - rabbitmqctl add_user openstack "{{ secrets['RABBIT_PASSWORD'] }}"
-      - rabbitmqctl set_permissions openstack ".*" ".*" ".*"
-  when: ansible_nodename == hostname_node0
-  register: command_result
-  failed_when: "command_result.rc not in (0, 70)"

secrets.yml

@@ -0,0 +1,32 @@
+$ANSIBLE_VAULT;1.1;AES256
+32313865346461323861313234623330633830663561353636396161643566353733393931303232
+3134353364393034626564333866383065633162383561390a656463303464616434303435303037
+63313232373630616136626464373464646535353030396136656361343162666165366566383737
+6133326539393432390a386162393639636137326532393939633665326637373461663766656565
+36636661653939373134313165383261353832373362613533366431626330313831643836363932
+38623937663335646134323130636539333335636265313564323032623065393031343235633832
+33336138653336633932383133366566656464356331306265663662356166613135663234326434
+64343765366439303766613062363830346238376435366138323662366463353634623937376663
+39313263613764623533323562333932656564346462333961663538353366313065343263383431
+39643734353632336134663965343062373933656461393264323938303261316563313962343837
+66623562333031646633643734383439386130653033343962633930613031313433633033383561
+65366230663263346661343339636537336332323266623230646534323563373934356332663136
+30626231623534616562323033333437353239306538653835623931363164383536336562336136
+30666265366536313436646535383632626137613831633132666666653830383566323532306332
+34353534336331653330663137323936303337306134333036633932363664643864333534316438
+31623062303137376637353061643838383831663561663436663130663064323665316261316531
+32616533333165636130623334373130316339326538343330646366383933353137623631323530
+39653437343432383161323661643931346664663265326664336461373033646563666333353661
+61633865373764346131623131346266373331626336663735303439376230333537386562303939
+65363139346564653464663633326639323930333464376136353064393039373535613231623138
+61373434323065646238356436373730333939613965666237616265653033333230353466316266
+30383939376335656632626232663061623332636337646234653565363561353334643462376666
+30326438303333336265306463313137656334313235643434616238333564373761333235633639
+66346161316130633463623435646639366136386335386139613230653064663230366265633036
+33613132633035393337653436613031383765616638323663363866393165613030306637393134
+38333734373939626364343533306662393463646264666161346434363832623239643864303431
+39383931333139633338663761646335613935636239636439383333313531633364653439323036
+35373639363164386666366335313934336231333261623763633133393562656237313761356631
+39663234653339313466326534333435306662316461333035623339353435383137383735373733
+32373535303338646266346539386364356233616631316661633037346665353762353138376538
+386535333439313233663464353534376535

secrets.yml.topol

@@ -0,0 +1,13 @@
+---
+GLANCE_PASSWORD:
+METADATA_SECRET:
+MYSQL_ROOT_PASSWORD:
+NEUTRON_PASSWORD:
+NOVA_PASSWORD:
+NOVA_PLACEMENT_PASSWORD:
+OS_PASSWORD:   # Keystone admin password
+OS_DEMO_PASSWORD: # Keystone demo user password
+RABBIT_PASSWORD:
+RABBITMQ_ERLANG_COOKIE:
+CINDER_PASSWORD:
+HEAT_PASSWORD:

settings.yml

@@ -0,0 +1,12 @@
+---
+- allocation_pool:
+    start: 172.23.128.50
+    end: 172.23.128.249
+
+- dns_nameserver: 129.125.4.6
+
+- gateway: 172.23.128.250
+
+- subnet_range: 172.23.128.0/24
+
+- rsa_pub: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDStPUPXkcu81onUm/le54JCu174yXJJDsthDr96Mv8irBVBWuy5FxnaASuDpmC4QE4s0UAIg1iq/SWrr8qdBQ4OVuYFiW0S7ZJvcoKr/40Wh+T5MeltGQfmkDp6kBsfaMSo6M4tF1c8i+XgOgxb4fxHYb8mFhseztRLx6McxJJJLB0nu+T12WQ01nl0XtwD+3EsZWfxRH0KA59VHZSe3Anc5z+Fm7WU+1Vzy6/pkiIhVReI1L6VVhZsIdSu3fQK6fHQcujtfuw6RKEpisZQqnxMUviWQ98yeQXHk6Nx840WCh3vvKveEAoC4Y/UEZa1TMe6PczfUaLjaidUkpulJsP egon@egon-pc

site.yml

@@ -7,5 +7,9 @@
 - include: glance-controller.yml
 - include: nova-controller.yml
 - include: neutron-controller.yml
+- include: cinder-controller.yml
+- include: cinder-storage.yml
 - include: nova-compute.yml
 - include: horizon.yml
+- include: heat.yml
+- include: post-install.yml

test_hosts

@@ -0,0 +1,48 @@
+[databases]
+openstack-test05
+openstack-test06
+openstack-test07
+
+[keystone]
+openstack-test05
+
+[glance-controller]
+openstack-test05
+
+[horizon]
+openstack-test05
+
+[rabbitmq]
+openstack-test05
+openstack-test06
+openstack-test07
+
+[memcached]
+openstack-test05
+
+[neutron-controller]
+openstack-test05 physical_interface_mappings=provider:enp4s0f0
+
+[nova-controller]
+openstack-test05
+
+[cinder-controller]
+openstack-test05
+
+[heat]
+openstack-test05
+
+[cinder-storage]
+openstack-test05 storage_volume=/dev/openstack-test05-vg/cinder
+openstack-test06 storage_volume=/dev/openstack-test06-vg/cinder
+openstack-test07 storage_volume=/dev/openstack-test07-vg/cinder
+openstack-test08 storage_volume=/dev/openstack-test08-vg/cinder
+openstack-test09 storage_volume=/dev/openstack-test09-vg/cinder
+openstack-test10 storage_volume=/dev/openstack-test10-vg/cinder
+
+[nova-compute]
+openstack-test06 physical_interface_mappings=provider:enp4s0f0
+openstack-test07 physical_interface_mappings=provider:enp4s0f0
+openstack-test08 physical_interface_mappings=provider:enp4s0f0
+openstack-test09 physical_interface_mappings=provider:enp4s0f0
+openstack-test10 physical_interface_mappings=provider:enp4s0f0