added ceph-conf to systemd-unitfile nova-compute

Updated url of docker registry.

make a loop for more flexibility.

Introducing listen_ip variable that overrides

the default listen_ip.

make a loop for more flexibility.

Get a listen ip specifically for that host.

see if components have listen_ip defined before using ansible_default_ipv4

Make service files look for listen_ip variable.

implemented listen_ip variable here too

map to different port to avoid clashes.

Make PHYSICAL_INTERFACE_MAPPINGS variable...

instead of just one provuider interface.
it should contain something like:
physnet2:eth1,physnet3:eth2

add openstack client on machine that is running keystone

Added delay to check.

enable all services are started at boot

Inventory for gcc openstack03 all in one.

added volume for glance images

Added gcc all in one specific config.

Prevent an error when there is no secrets.yml..

to back up

removed reference to empty dir.

added empty meta/main.yml

And now with a list of roles

..and removed the list

removed reference to empty dir.

Added galera cluster support

When at least three database nodes are installed, the playbook will
install a galera cluster across them.

The galera cnf can be the same template across...

nodes.

made environment file for the service.

I am unable to reproduce systemctl set-environment to work as
advertized.

Reverted to updating init file by ansible.

entrypoint.sh of the mariadb container seems unable to cope when a blank
variable is passed by systemd.

give the galera master node some time

It seems to be nessecary to run in host mode..

for galera to work.

I misunderstood pause. need sleep.

Inventory for physical test cluster.

Added CINDER_PASSWORD

Make sure docker is started.

If docker was already installed but not running
it was not started.

fixed refernce to neutron controller

Added heat

Added port for metadata service

Passed metadata secret to be used in config.

Listen ip should be the machine's ip...

Added openstack client from repo.

changed name of subnet

added horizon

Changed rabbitmq default user to openstack.

This makes it no longer nesseccary to create a separate openstack user,
which is lost on rabbitmq restart.

Added sleep because hosts were usually..

not discovered.

Removed unnessecary port mapping

Making /dev/lvm available to container.

fixed os-test inventory

Make iscsi devices available

(needed to attach cinder volumes to machines)

command module no longer works with &&

add cinder to test setup

ansible.cfg

@@ -1,2 +1,2 @@
 [defaults]
-hostfile = hosts
+inventory = merlin

generate_secrets.py

@@ -5,6 +5,7 @@ Open the secrets.yml and replace all passwords.
 Original is backed up.
 """
 
+from os import path
 import random
 import string
 from subprocess import call
@@ -27,7 +28,8 @@ for key, value in data.iteritems():
         for _ in range(pass_length))
 
 # Make numbered backups of the secrets file.
-call(['cp', '--backup=numbered', 'secrets.yml', 'secrets.yml.bak'])
+if path.isfile('secrets.yml'):
+    call(['cp', '--backup=numbered', 'secrets.yml', 'secrets.yml.bak'])
 
 with open('secrets.yml', 'w') as f:
     dump(data, f, Dumper=Dumper, default_flow_style=False)

heat.yml

@@ -0,0 +1,9 @@
+---
+- hosts: all
+  name: Dummy to gather facts
+  tasks: []
+
+- hosts: heat
+  become: True
+  roles:
+     - heat

hosts

@@ -1,5 +1,9 @@
+# A demo cluster of three nodes.
+
 [databases]
+openstack01-node01
 openstack01-node02
+openstack01-node03
 
 [keystone]
 openstack01-node03
@@ -15,24 +19,9 @@ openstack01-node01
 openstack01-node02
 openstack01-node03
 
-#[cassandra]
-#openstack01-node[01:03]
-
-#openstack01-node01
-#
-#[next_cassandra]
-#openstack01-node02
-#openstack01-node03
-
 [memcached]
 openstack01-node03
 
-#[first_cassandra:vars]
-#run_options=""
-#
-#[next_cassandra:vars]
-#run_options="-e CASSANDRA_SEEDS=172.23.41.1"
-
 [neutron-controller]
 openstack01-node01 physical_interface_mappings=provider:ens192
 
@@ -46,4 +35,4 @@ openstack01-node03
 openstack01-node01 storage_volume=/dev/loop0
 
 [nova-compute]
-openstack01-node04 physical_interface_mappings=provider:dummy0
+openstack01-node04 physical_interface_mappings=provider:enp4s0f0

hosts-openstack03

@@ -0,0 +1,26 @@
+[databases]
+openstack03.gcc.rug.nl
+
+[keystone]
+openstack03.gcc.rug.nl
+
+[glance-controller]
+openstack03.gcc.rug.nl
+
+[horizon]
+openstack03.gcc.rug.nl
+
+[rabbitmq]
+openstack03.gcc.rug.nl
+
+[memcached]
+openstack03.gcc.rug.nl
+
+[neutron-controller]
+openstack03.gcc.rug.nl
+
+[nova-controller]
+openstack03.gcc.rug.nl
+
+[nova-compute]
+openstack03.gcc.rug.nl

merlin

@@ -0,0 +1,53 @@
+[nova-compute]
+merlin-managementnode002 physical_interface_mappings=provider:enp5s0f1.1064 overlay_ip=172.23.43.102
+merlin-managementnode003 physical_interface_mappings=provider:enp5s0f1.1064 overlay_ip=172.23.43.103
+merlin-node001 physical_interface_mappings=provider:enp130s0f0.1064 overlay_ip=172.23.43.1
+merlin-node002 physical_interface_mappings=provider:enp130s0f0.1064 overlay_ip=172.23.43.2
+merlin-node003 physical_interface_mappings=provider:enp130s0f0.1064 overlay_ip=172.23.43.3
+merlin-node004 physical_interface_mappings=provider:enp130s0f0.1064 overlay_ip=172.23.43.4
+merlin-node005 physical_interface_mappings=provider:enp130s0f0.1064 overlay_ip=172.23.43.5
+
+[databases]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[rabbitmq]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[horizon]
+merlin-managementnode001
+
+[memcached]
+merlin-managementnode001
+
+[nova-controller]
+merlin-managementnode001
+
+[keystone]
+merlin-managementnode001
+
+[neutron-controller]
+merlin-managementnode001 physical_interface_mappings=provider:enp5s0f1.1064 overlay_ip=172.23.43.101 
+
+[heat]
+merlin-managementnode001
+
+[glance-controller]
+merlin-managementnode002
+
+[cinder-controller]
+merlin-managementnode002
+
+[cinder-storage]
+merlin-managementnode002 
+merlin-managementnode003 
+merlin-node001 
+merlin-node002 
+merlin-node003 
+merlin-node004 
+merlin-node005 
+
+

merlinsdn

@@ -0,0 +1,37 @@
+[nova-compute]
+merlin-managementnode002 physical_interface_mappings=provider:eno3
+merlin-managementnode003 physical_interface_mappings=provider:eno3
+merlin-node001 physical_interface_mappings=provider:eno3
+merlin-node003 physical_interface_mappings=provider:eno3
+merlin-node004 physical_interface_mappings=provider:eno3
+
+[databases]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[rabbitmq]
+merlin-managementnode001
+merlin-managementnode002
+merlin-managementnode003
+
+[horizon]
+merlin-managementnode001
+
+[memcached]
+merlin-managementnode001
+
+[nova-controller]
+merlin-managementnode001
+
+[keystone]
+merlin-managementnode001
+
+[neutron-controller]
+merlin-managementnode001 physical_interface_mappings=provider:eno3
+
+[heat]
+merlin-managementnode001
+
+[glance-controller]
+merlin-managementnode001

meta/main.yml

@@ -0,0 +1 @@
+---

nuke.yml

@@ -0,0 +1,16 @@
+---
+# This playbook will reset the instalation to facilitate a new installation.
+# All data is lost!
+- hosts: all
+  become: True
+  name: Dummy to gather facts
+  tasks:
+    - name: Stop docker service
+      shell: "systemctl stop docker"
+    - name: Verify docker is stopped.
+      systemd:
+        name: docker
+        state: stopped
+    - name: remove volumes
+      shell: "rm -rf /srv"
+

os-test

@@ -0,0 +1,37 @@
+# An all in one
+
+[databases]
+os-test
+
+[keystone]
+os-test
+
+[glance-controller]
+os-test
+
+[horizon]
+os-test
+
+[rabbitmq]
+os-test
+
+[memcached]
+os-test
+
+[neutron-controller]
+os-test physical_interface_mappings=provider:enp4s0f0
+
+[nova-controller]
+os-test
+
+[cinder-controller]
+os-test
+
+[cinder-storage]
+os-test storage_volume=/dev/sdb
+
+[nova-compute]
+os-test physical_interface_mappings=provider:enp4s0f0
+
+[all:vars]
+listen_ip=129.125.60.194

post-install.yml

@@ -16,10 +16,7 @@
       command: docker exec -i keystone.service bash -c "source /root/admin-openrc.sh && {{ item }}"
       with_items:
         - openstack network create  --share --external  --provider-physical-network provider  --provider-network-type flat provider
-        - >
+        - openstack network set --external provider
-            openstack subnet create --network provider
-            --allocation-pool start={{ allocation_pool['start'] }},end={{ allocation_pool['end'] }}
-            --dns-nameserver {{ dns_nameserver }} --gateway {{ gateway }} --subnet-range {{ subnet_range }} provider
         - openstack flavor create --id 0 --vcpus 1 --ram 64 --disk 1 m1.nano
         - openstack keypair create --public-key /root/id_rsa.pub adminkey
 

roles/cinder-storage/files/ceph.client.volumes.keyring

@@ -0,0 +1,9 @@
+$ANSIBLE_VAULT;1.1;AES256
+39313161646365373665646331613930316437363735326262376531636166346138303139613138
+3361353633616136303365646165643339333130393031380a373934636436626336326436306666
+34316532333165346139633239313930326238333134633365666138326338386632373937343335
+3262383863653136300a393464646365623763663063303936646462313764633736613562633661
+62313961626165363761656363393538396461653936353932303137626435626161316239623338
+65656132353136656430613462663466616432643761303366396461653066616162366666356533
+39386261623861323861633739343237386266306264356436666430313531303238636235393665
+31396533306261393835

roles/cinder-storage/files/ceph.conf

@@ -0,0 +1,14 @@
+[global]
+fsid = ef0b40a2-bc8c-4432-9cde-0ca7c82c8717
+mon_initial_members = merlin-managementnode002
+mon_host = 172.23.59.102
+auth_cluster_required = cephx
+auth_service_required = cephx
+auth_client_required = cephx
+
+# Your network address
+public network = 172.23.59.0/24
+osd pool default size = 2
+
+[client.volumes]
+keyring = /etc/ceph/ceph.client.volumes.keyring

roles/cinder-storage/files/uuid

@@ -0,0 +1 @@
+d0db6ba7-a0c9-4da6-b0bc-aa7846325333

roles/cinder-storage/tasks/main.yml

@@ -4,9 +4,13 @@
   include_vars:
     file: ../../secrets.yml
     name: secrets
+  tags: vars
+
+#- command: uuidgen
+#  register: uuid
 
 - set_fact:
-    docker_image: registry.webhosting.rug.nl/hpc/openstack-cinder-storage:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-cinder-storage-merlin:latest
     env_vars: >
         -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}"
         -e "CINDER_HOST={{ listen_ip | default(hostvars[groups['cinder-storage'][0]]['ansible_default_ipv4']['address']) }}"
@@ -20,6 +24,7 @@
         -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
         -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
         -e "RABBIT_USER=openstack"
+  tags: vars
 
 - name: pull docker image
   docker_image:
@@ -34,6 +39,7 @@
   with_items:
       - /srv/cinder-storage
       - /srv/cinder-storage/root
+      - /srv/cinder-storage/etc/ceph
 
 - name: initial setup
   command: >
@@ -41,10 +47,22 @@
              --privileged
              {{ env_vars }}
              -v /srv/cinder-storage/root:/root \
-             -v "{{ storage_volume }}":/dev/cinder_storage_volume \
              {{ docker_image }} /etc/bootstrap.sh
   tags: bootstrap
 
+- name: copy ceph-client configurationfile
+  copy:
+    src: files/ceph.conf
+    dest: /srv/cinder-storage/etc/ceph/ceph.conf
+    mode: 0644
+
+- name: copy ceph-client-keyring
+  copy:
+    src: files/ceph.client.volumes.keyring
+    dest: /srv/cinder-storage/etc/ceph/ceph.client.volumes.keyring
+    mode: 0644
+
+
 - name: install service file.
   template:
     src: templates/cinder-storage.service
@@ -52,8 +70,10 @@
     mode: 644
     owner: root
     group: root
+  tags: systemd
 
 - command: systemctl daemon-reload
+  tags: systemd
 
 - name: start service at boot.
   command: systemctl reenable cinder-storage.service

roles/cinder-storage/templates/cinder-storage.service

@@ -12,8 +12,9 @@ ExecStart=/usr/bin/docker run --name %n \
   --privileged \
   {{ env_vars | replace('\n', '') }} \
   -v /srv/cinder-storage/root:/root \
-  -v "{{ storage_volume }}":/dev/cinder_storage_volume \
+  -v /srv/cinder-storage/etc/ceph:/etc/ceph \
   -p 8777:8776 \
+  -p 3260:3260 \
   {{ docker_image }}
 
 [Install]

roles/common/tasks/docker.yml

@@ -13,3 +13,8 @@
   with_items:
      - docker-engine
      - python-docker
+
+- name: make sure service is started
+  systemd:
+    name: docker.service
+    state: started

roles/glance-controller/files/ceph.client.images.keyring

@@ -0,0 +1,9 @@
+$ANSIBLE_VAULT;1.1;AES256
+38396565316335323234336366643364636639306234383033383664383131343861326231633437
+3164323532393735663065313036316664623065383466310a393665393238323265316464393338
+35303535306434363538303032323366353138303363386463313364313864663535386138323634
+3138393163353063650a386532356566396636636365343065343934396163663732663965343038
+63343539643964656538396664633235636639356532616530613838613839623830396565656538
+62613236643632346539613036383230653431643733343737626666633532613030366462326430
+63303339373563366633363130363361643731303030323635323832333165363439623831316636
+64613533313664383164

roles/glance-controller/files/ceph.conf

@@ -0,0 +1,14 @@
+[global]
+fsid = ef0b40a2-bc8c-4432-9cde-0ca7c82c8717
+mon_initial_members = merlin-managementnode002
+mon_host = 172.23.59.102
+auth_cluster_required = cephx
+auth_service_required = cephx
+auth_client_required = cephx
+
+# Your network address
+public network = 172.23.59.0/24
+osd pool default size = 2
+
+[client.images]
+keyring = /etc/ceph/ceph.client.images.keyring

roles/glance-controller/tasks/main.yml

@@ -6,7 +6,7 @@
     name: secrets
 
 - set_fact:
-    docker_image: registry.webhosting.rug.nl/hpc/openstack-glance:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-glance-merlin:latest
     env_vars: >
         -e "GLANCE_HOST={{ listen_ip | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "GLANCE_PASSWORD={{ secrets['GLANCE_PASSWORD'] }}"
@@ -20,10 +20,11 @@
         -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
         -e "RABBIT_USER=openstack"
 
-- name: pull docker image
+
-  docker_image:
+#- name: pull docker image
-    name: "{{ docker_image }}"
+#  docker_image:
-  tags: pull
+#    name: "{{ docker_image }}"
+#  tags: pull
 
 - name: Make build and persistent directories
   file:
@@ -33,6 +34,19 @@
   with_items:
       - /srv/glance
       - /srv/glance/root
+      - /srv/glance/etc/ceph
+
+- name: copy ceph-client configurationfile
+  copy:
+    src: files/ceph.conf
+    dest: /srv/glance/etc/ceph/ceph.conf
+    mode: 0644
+
+- name: copy ceph-client-keyring
+  copy:
+    src: files/ceph.client.images.keyring
+    dest: /srv/glance/etc/ceph/ceph.client.volumes.keyring
+    mode: 0644
 
 - name: install service file.
   template:

roles/glance-controller/templates/glance.service

@@ -11,7 +11,7 @@ ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
   {{ env_vars | replace('\n', '') }} \
   -v /srv/glance/root:/root \
-  -v /var/lib/glance/images:/var/lib/glance/images \
+  -v /etc/ceph:/etc/ceph \
   -p 9292:9292 \
   {{ docker_image }}
 

roles/heat/tasks/main.yml

@@ -0,0 +1,62 @@
+# Build and install a docker image for heat.
+---
+- name: include secrets
+  include_vars:
+    file: ../../secrets.yml
+    name: secrets
+
+- set_fact:
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-heat:latest
+    env_vars: >
+        -e "HEAT_HOST={{ listen_ip | default(hostvars[groups['heat'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "HEAT_PASSWORD={{ secrets['HEAT_PASSWORD'] }}"
+        -e "HEAT_USER=heat"
+        -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
+        -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
+        -e "RABBIT_HOST={{ listen_ip | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"
+        -e "RABBIT_USER=openstack"
+
+- name: pull docker image
+  docker_image:
+    name: "{{ docker_image }}"
+  tags: pull
+
+- name: Make build and persistent directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    mode: 0777
+  with_items:
+      - /srv/heat
+      - /srv/heat/root
+
+- name: install service file.
+  template:
+    src: templates/heat.service
+    dest: /etc/systemd/system/heat.service
+    mode: 644
+    owner: root
+    group: root
+
+- name: start service at boot.
+  command: systemctl reenable heat.service
+
+- command: systemctl daemon-reload
+
+- name: Initialize database.
+  command: >
+             /usr/bin/docker run --rm
+             {{ env_vars }}
+             --add-host=keystone:{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
+             -v /srv/heat/root:/root \
+             {{ docker_image }} /etc/bootstrap.sh
+  tags: bootstrap
+
+- name: make sure service is started
+  systemd:
+    name: heat.service
+    state: restarted

roles/heat/templates/heat.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=Openstack heat Container
+After=docker.service
+Requires=docker.service
+
+[Service]
+TimeoutStartSec=0
+Restart=always
+ExecStartPre=-/usr/bin/docker kill %n
+ExecStartPre=-/usr/bin/docker rm %n
+ExecStart=/usr/bin/docker run --name %n \
+  {{ env_vars | replace('\n', '') }} \
+  -v /srv/heat/root:/root \
+  -p 8000:8000 \
+  -p 8004:8004 \
+  {{ docker_image }}
+
+[Install]
+WantedBy=multi-user.target

roles/horizon/templates/horizon.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=Openstack Glance Container
+Description=Openstack Horizon Container
 After=docker.service
 Requires=docker.service
 

roles/keystone/tasks/main.yml

@@ -52,7 +52,7 @@
   delay: 3
   ignore_errors: yes
 
-- name: keystone manage commands to setup db
+- name: keystone manage commands to setup db_sync
   command: >
              /usr/bin/docker run --rm
              --add-host=mariadb:{{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}
@@ -69,6 +69,11 @@
                     --bootstrap-internal-url http://{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}:35357/v3/
                     --bootstrap-public-url http://{{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}:5000/v3/
                     --bootstrap-region-id RegionOne
+  # sometimes the initial connect fails.
+  # Retry until it succeeds.
+  retries: 7
+  delay: 3
+  ignore_errors: yes
 
 - name: make sure service is started
   systemd:
@@ -85,12 +90,27 @@
              -e "OS_AUTH_URL=http://${KEYSTONE_HOST}:35357/v3"
              -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
              {{ docker_image }} bash /etc/bootstrap.sh
+  register: result
+  retries: 7
+  delay: 3
+
+
+- name: install openstack repo key host.
+  command: apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5EDB1B62EC4926EA
+  tags: openstackclient
+
+- name: install openstack repo on host.
+  apt_repository:
+      repo: "deb http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata main"
+      filename: ocata
+  tags: openstackclient
 
 - name: install openstack client for management
   apt:
     name: python-openstackclient
     state: latest
     update_cache: yes
+  tags: openstackclient
 
 - name: source admin-openrc.sh in root .bashrc
   lineinfile:

roles/mariadb/tasks/main.yml

@@ -70,3 +70,8 @@
 
 - name: start service at boot.
   command: systemctl reenable mysql.service
+
+- name: Give the cluster some time to initialize replication.
+  command: bash -c "sleep 60 && systemctl daemon-reload"
+  when: groups['databases'] | length >= 3
+

roles/neutron-controller/tasks/main.yml

@@ -6,7 +6,7 @@
     name: secrets
 
 - set_fact:
-    docker_image: "registry.webhosting.rug.nl/hpc/openstack-neutron-controller:latest"
+    docker_image: "registry.webhosting.rug.nl/hpc/openstack-neutron-controller-merlin:latest"
 
 - name: pull docker image
   docker_image:
@@ -29,6 +29,7 @@
         -e "NOVA_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "NOVA_PLACEMENT_PASSWORD={{ secrets['NOVA_PLACEMENT_PASSWORD'] }}"
         -e "NOVA_PLACEMENT_USER=placement"
+        -e "OVERLAY_IP={{ overlay_ip }}"
         -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}"
         -e "PHYSICAL_INTERFACE_MAPPINGS={{ physical_interface_mappings }}"
         -e "RABBIT_PASSWORD={{ secrets['RABBIT_PASSWORD'] }}"

roles/nova-compute/files/ceph.client.nova.keyring

@@ -0,0 +1,8 @@
+$ANSIBLE_VAULT;1.1;AES256
+35383561643837376265356534363538386266646466313034383231313261343566626338613433
+3464373930663738316366383835336235636137356431390a626334343661353733616366633963
+33306166646465356531666364656236633766653639393636396234663538373136373266353136
+3433626231323638300a306437383862316463363533373131343561303934326135663532656630
+38616139373964313963326539346338353664613035633436326433343835343030653936363836
+65313537623937663438663734323265323634376238373761626236353665613931616566306536
+343334346138313539333338336663343863

roles/nova-compute/files/ceph.conf

@@ -0,0 +1,14 @@
+[global]
+fsid = ef0b40a2-bc8c-4432-9cde-0ca7c82c8717
+mon_initial_members = merlin-managementnode002
+mon_host = 172.23.59.102
+auth_cluster_required = cephx
+auth_service_required = cephx
+auth_client_required = cephx
+
+# Your network address
+public network = 172.23.59.0/24
+osd pool default size = 2
+
+[client.nova]
+keyring = /etc/ceph/ceph.client.nova.keyring

roles/nova-compute/files/uuid

@@ -0,0 +1 @@
+b5044271-1918-4070-822c-f19ed14d7494

roles/nova-compute/tasks/ceph.conf

@@ -0,0 +1,2 @@
+[client.nova]
+keyring = /etc/ceph/ceph.client.nova.keyring

roles/nova-compute/tasks/main.yml

@@ -4,10 +4,11 @@
   include_vars:
     file: ../../secrets.yml
     name: secrets
+  tags: vars
 
 - set_fact:
-    docker_image: registry.webhosting.rug.nl/hpc/openstack-nova-compute:latest
+    docker_image: registry.webhosting.rug.nl/hpc/openstack-nova-compute-merlin:latest
-  tags: facts
+  tags: vars
 
 - name: pull docker image
   docker_image:
@@ -15,6 +16,27 @@
     force: True
   tags: pull
 
+- name: Make build and persistent directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    mode: 0777
+  with_items:
+      - /srv/nova-compute
+      - /srv/nova-compute/etc/ceph
+
+- name: copy ceph-client configurationfile
+  copy:
+    src: files/ceph.conf
+    dest: /srv/nova-compute/etc/ceph/ceph.conf
+    mode: 0644
+
+- name: copy ceph-client-keyring
+  copy:
+    src: files/ceph.client.nova.keyring
+    dest: /srv/nova-compute/etc/ceph/ceph.client.nova.keyring
+    mode: 0644
+
 - name: install service file.
   template:
     src: templates/nova-compute.service
@@ -22,11 +44,13 @@
     mode: 644
     owner: root
     group: root
+  tags: systemd
 
 - command: systemctl daemon-reload
+  tags: systemd
 
 - apt:
-    name: '{{ item }}'
+    name: "{{ item }}"
   with_items:
       - kvm
       - libvirt0
@@ -42,5 +66,5 @@
   command: systemctl reenable nova-compute.service
 
 - name: let nova controler discover new host
-  command: docker exec -i nova-controller.service nova-manage cell_v2 discover_hosts
+  shell: "sleep 10 && docker exec -i nova-controller.service nova-manage cell_v2 discover_hosts"
   delegate_to: "{{ hostvars[groups['nova-controller'][0]]['ansible_hostname'] }}"

roles/nova-compute/templates/nova-compute.service

@@ -11,8 +11,9 @@ ExecStartPre=-/usr/bin/docker rm %n
 ExecStart=/usr/bin/docker run --name %n \
     -e "GLANCE_CONTROLLER_HOST={{ hostvars[groups['glance-controller'][0]]['listen_ip'] | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "KEYSTONE_HOST={{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}" \
+    -e "METADATA_SECRET={{ secrets['METADATA_SECRET'] }}" \
     -e "MEMCACHED_HOST={{ hostvars[groups['memcached'][0]]['listen_ip'] | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}" \
-    -e "MY_IP={{ hostvars[groups['nova-compute'][0]]['listen_ip'] | default(hostvars[groups['nova-compute'][0]]['ansible_default_ipv4']['address']) }}" \
+    -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}" \
     -e "MYSQL_HOST={{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}" \
     -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}" \
     -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['neutron-controller'][0]]['listen_ip'] | default(hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address']) }}" \
@@ -25,6 +26,7 @@ ExecStart=/usr/bin/docker run --name %n \
     -e "NOVA_PLACEMENT_PASSWORD={{ secrets['NOVA_PLACEMENT_PASSWORD'] }}" \
     -e "NOVA_PLACEMENT_USER=placement" \
     -e "NOVA_USER=nova" \
+    -e "OVERLAY_IP={{ overlay_ip }}" \
     -e "OS_PASSWORD={{ secrets['OS_PASSWORD'] }}" \
     -e "PHYSICAL_INTERFACE_MAPPINGS={{ physical_interface_mappings }}" \
     -e "RABBIT_HOST={{ hostvars[groups['rabbitmq'][0]]['listen_ip'] | default(hostvars[groups['rabbitmq'][0]]['ansible_default_ipv4']['address']) }}" \
@@ -33,10 +35,12 @@ ExecStart=/usr/bin/docker run --name %n \
     --add-host=keystone:{{ hostvars[groups['keystone'][0]]['listen_ip'] | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }} \
     --add-host=mariadb:{{ hostvars[groups['databases'][0]]['listen_ip'] | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }} \
     --privileged \
+    -v /dev:/dev \
     -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock \
     -v /var/lib/nova/instances:/var/lib/nova/instances \
     -v /lib/modules:/lib/modules \
     -v /etc/machine-id:/etc/machine-id \
+    -v /srv/nova-compute/etc/ceph:/etc/ceph \
     --network host \
     {{ docker_image }} /etc/run.sh
 

roles/nova-controller/tasks/main.yml

@@ -19,11 +19,12 @@
     env_vars: >
         -e "GLANCE_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['glance-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "KEYSTONE_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "METADATA_SECRET={{ secrets['METADATA_SECRET'] }}"
         -e "MEMCACHED_HOST={{ listen_ip | default(hostvars[groups['memcached'][0]]['ansible_default_ipv4']['address']) }}"
-        -e "MY_IP={{ listen_ip | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "MY_IP={{ listen_ip | default(ansible_default_ipv4.address) }}"
         -e "MYSQL_HOST={{ listen_ip | default(hostvars[groups['databases'][0]]['ansible_default_ipv4']['address']) }}"
         -e "MYSQL_ROOT_PASSWORD={{ secrets['MYSQL_ROOT_PASSWORD'] }}"
-        -e "NEUTRON_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['keystone'][0]]['ansible_default_ipv4']['address']) }}"
+        -e "NEUTRON_CONTROLLER_HOST={{ hostvars[groups['neutron-controller'][0]]['listen_ip'] | default(hostvars[groups['neutron-controller'][0]]['ansible_default_ipv4']['address']) }}"
         -e "NEUTRON_PASSWORD={{ secrets['NEUTRON_PASSWORD'] }}"
         -e "NEUTRON_USER=neutron"
         -e "NOVA_CONTROLLER_HOST={{ listen_ip | default(hostvars[groups['nova-controller'][0]]['ansible_default_ipv4']['address']) }}"

roles/nova-controller/templates/nova-controller.service

@@ -15,6 +15,7 @@ ExecStart=/usr/bin/docker run --name %n \
   --privileged \
   -v /srv/nova-controller/root:/root \
   -p 8774:8774 \
+  -p 8775:8775 \
   -p 8778:8778 \
   -p 6080:6080 \
   {{ docker_image }} /etc/run.sh

roles/rabbitmq/files/rabbitmq.service

@@ -14,7 +14,7 @@ ExecStart=/usr/bin/docker run \
           --add-host "{{ host }}:{{ hostvars[host]['listen_ip'] | default(hostvars[host]['ansible_default_ipv4']['address']) }}" \
 {% endfor %}
           -p 4369:4369 -p 25679:25679 -p 25672:25672 -p 5671-5672:5671-5672 -p 8080:15672 \
-          -e "RABBITMQ_DEFAULT_USER=user" -e "RABBITMQ_DEFAULT_PASS={{ secrets['RABBIT_PASSWORD'] }}" \
+          -e "RABBITMQ_DEFAULT_USER=openstack" -e "RABBITMQ_DEFAULT_PASS={{ secrets['RABBIT_PASSWORD'] }}" \
           -e "RABBITMQ_ERLANG_COOKIE={{ secrets['RABBITMQ_ERLANG_COOKIE'] }}" \
           -e "RABBITMQ_NODENAME=rabbit_{{ ansible_nodename }}" \
           --hostname "{{ ansible_nodename }}" --name %n rabbitmq:3-management

roles/rabbitmq/tasks/main.yml

@@ -37,11 +37,3 @@
       - rabbitmqctl start_app
   when: ansible_nodename != hostname_node0
 
-- name: create openstack user
-  command: "docker exec -i rabbitmq.service {{ item }}"
-  with_items:
-      - rabbitmqctl add_user openstack "{{ secrets['RABBIT_PASSWORD'] }}"
-      - rabbitmqctl set_permissions openstack ".*" ".*" ".*"
-  when: ansible_nodename == hostname_node0
-  register: command_result
-  failed_when: "command_result.rc not in (0, 70)"

secrets.yml

@@ -1,30 +1,32 @@
 $ANSIBLE_VAULT;1.1;AES256
-35643437313834633532373265366630663035336231306639623561613765386332663334343237
+32313865346461323861313234623330633830663561353636396161643566353733393931303232
-3339363162303463353437326331656532336138373066620a623137643762383532376361353364
+3134353364393034626564333866383065633162383561390a656463303464616434303435303037
-37646236386466353636396535376463333133323664316634663466663164303063383830653039
+63313232373630616136626464373464646535353030396136656361343162666165366566383737
-3535666361303562630a316137376531636537383138663662373865383431343035646539356137
+6133326539393432390a386162393639636137326532393939633665326637373461663766656565
-38323866643831353537366630363333663865383261633938346664633362343661343839383766
+36636661653939373134313165383261353832373362613533366431626330313831643836363932
-66363733356333303334323136376136353738376362376231353338343763663131363731343639
+38623937663335646134323130636539333335636265313564323032623065393031343235633832
-61383138626235633663666430383964616239363035663663646133636434363032626633663865
+33336138653336633932383133366566656464356331306265663662356166613135663234326434
-30663732646630393163653461626435333463396463333236313930346461626364626166386365
+64343765366439303766613062363830346238376435366138323662366463353634623937376663
-66323736316230376165666366363136666533376335316132343361393532616536383965363339
+39313263613764623533323562333932656564346462333961663538353366313065343263383431
-30376362356665633630393561653532613139366236663961643864383738353430666562623730
+39643734353632336134663965343062373933656461393264323938303261316563313962343837
-34663166393665653265663836623731386235633062306562373935633737363639383336303539
+66623562333031646633643734383439386130653033343962633930613031313433633033383561
-37663763623664623038316438356138363134646230643261646262353163333430616462393866
+65366230663263346661343339636537336332323266623230646534323563373934356332663136
-31666233636233356464633436626637313633623736343264613037353432386131393964386663
+30626231623534616562323033333437353239306538653835623931363164383536336562336136
-36353236613662633764366437306461316138366461653731373436613039346663663536653362
+30666265366536313436646535383632626137613831633132666666653830383566323532306332
-38656636303935626563303732666261373665303035333661643865393166653330646336393961
+34353534336331653330663137323936303337306134333036633932363664643864333534316438
-31646539396131626464313733383638656438613530663166393035343630353764313232323432
+31623062303137376637353061643838383831663561663436663130663064323665316261316531
-34386334666231323261343765623636313032373835396332623037613866613636393038653266
+32616533333165636130623334373130316339326538343330646366383933353137623631323530
-36336531356534633933383432646663663364376130386239613836336263623161326563346661
+39653437343432383161323661643931346664663265326664336461373033646563666333353661
-33636232313866613662353661373533383138393434396338343934326333326238336638396462
+61633865373764346131623131346266373331626336663735303439376230333537386562303939
-65376133343038313437343934373265333632663133653133656130636533663237623839623634
+65363139346564653464663633326639323930333464376136353064393039373535613231623138
-35363764363763363465363437623964363362616261663166633066373033633864336532633031
+61373434323065646238356436373730333939613965666237616265653033333230353466316266
-32323733616562663031303230383561373637326436336462363461313532623262653866323862
+30383939376335656632626232663061623332636337646234653565363561353334643462376666
-34643631333533626537373538353564306261313035303530666462326534633638363932363037
+30326438303333336265306463313137656334313235643434616238333564373761333235633639
-65336230373034643966656561303164373463353638316632613431643535303930373334383134
+66346161316130633463623435646639366136386335386139613230653064663230366265633036
-38323731363535313065326330653666323934636466386238616664316635303333653631396639
+33613132633035393337653436613031383765616638323663363866393165613030306637393134
-39303737613361653862343964303231393164346134633366633262326230643137303331373231
+38333734373939626364343533306662393463646264666161346434363832623239643864303431
-31323832363937663935333737613133323265323863623933633962633230386339636432643937
+39383931333139633338663761646335613935636239636439383333313531633364653439323036
-66653763376663666637353738646565343835333937343765356539383734316231623466343634
+35373639363164386666366335313934336231333261623763633133393562656237313761356631
-30663135663938393561333133663737653635393432333534306466366332333338
+39663234653339313466326534333435306662316461333035623339353435383137383735373733
+32373535303338646266346539386364356233616631316661633037346665353762353138376538
+386535333439313233663464353534376535

secrets.yml.topol

@@ -9,3 +9,5 @@ OS_PASSWORD:   # Keystone admin password
 OS_DEMO_PASSWORD: # Keystone demo user password
 RABBIT_PASSWORD:
 RABBITMQ_ERLANG_COOKIE:
+CINDER_PASSWORD:
+HEAT_PASSWORD:

site.yml

@@ -11,4 +11,5 @@
 - include: cinder-storage.yml
 - include: nova-compute.yml
 - include: horizon.yml
-- include: post-install.yml
+- include: heat.yml
+  #- include: post-install.yml

test_hosts

@@ -1,30 +1,48 @@
 [databases]
-ansible-test
+openstack-test05
-ansible-test-2
+openstack-test06
-ansible-test-3
+openstack-test07
 
 [keystone]
-ansible-test-3
+openstack-test05
 
 [glance-controller]
-ansible-test-2
+openstack-test05
 
 [horizon]
-ansible-test-3
+openstack-test05
 
 [rabbitmq]
-ansible-test
+openstack-test05
-ansible-test-2
+openstack-test06
-ansible-test-3
+openstack-test07
 
 [memcached]
-ansible-test-3
+openstack-test05
 
 [neutron-controller]
-ansible-test physical_interface_mappings=provider:ens10
+openstack-test05 physical_interface_mappings=provider:enp4s0f0
 
 [nova-controller]
-ansible-test
+openstack-test05
+
+[cinder-controller]
+openstack-test05
+
+[heat]
+openstack-test05
+
+[cinder-storage]
+openstack-test05 storage_volume=/dev/openstack-test05-vg/cinder
+openstack-test06 storage_volume=/dev/openstack-test06-vg/cinder
+openstack-test07 storage_volume=/dev/openstack-test07-vg/cinder
+openstack-test08 storage_volume=/dev/openstack-test08-vg/cinder
+openstack-test09 storage_volume=/dev/openstack-test09-vg/cinder
+openstack-test10 storage_volume=/dev/openstack-test10-vg/cinder
 
 [nova-compute]
-ansible-test-2 physical_interface_mappings=provider:ens10
+openstack-test06 physical_interface_mappings=provider:enp4s0f0
+openstack-test07 physical_interface_mappings=provider:enp4s0f0
+openstack-test08 physical_interface_mappings=provider:enp4s0f0
+openstack-test09 physical_interface_mappings=provider:enp4s0f0
+openstack-test10 physical_interface_mappings=provider:enp4s0f0