MIST logo

MIST as a Self-Service Platform

How-to guide for project teams

Using MIST as a Self-Service Platform

The original version of this guide is available in the MIST documentation repository. Access to the repository is required.

Steps

  1. Prerequisites
  2. Configure CI/CD variables
  3. Request access to the MIST repository group
  4. Create project repositories
  5. Set up the Terraform repository
  6. Create virtual machines and optional DNS zones
  7. Run the Terraform pipeline
  8. Set up the Ansible repository
  9. Run the Ansible pipeline

Introduction

MIST is a self-service platform for managing Linux virtual machines in NREC. Infrastructure is defined in Git repositories and deployed through pipelines, using standardised modules. The platform currently supports AlmaLinux 10 virtual machines.

MIST is available to project teams as a self-service offering. This guide helps you use MIST independently within your project, from meeting the prerequisites and initial setup to day-to-day use. For more information about the platform, see mist.uib.no. Available standardised modules are listed in the MIST module catalogue.

Its goal is to enable you to provision and manage the MIST resources your project needs without having to open a separate ticket in hjelp.uib.no for every step. Clear responsibilities, applicable security requirements, and transparent operational procedures remain in place.

The following sections explain which access and information you need, how to configure MIST for your project, and whom to contact if you have questions or encounter issues.

Prerequisites

Before you begin, make sure that you have the following:

Configure CI/CD variables

Configure the required CI/CD variables in your project repository group in git.app.uib.no. Set these as group variables only, so that they are available to both the Ansible and Terraform repositories. Mark every variable indicated as masked and hidden accordingly.

VariableRequiredDescriptionConfiguration
GITLAB_ACCESS_TOKENYesPersonal access token for GitLab.Masked and hidden
OS_PASSWORDYesPassword for your NREC user.Masked and hidden
OS_USERNAMEYesNREC username, for example username@uib.no.Masked and hidden
OS_PROJECT_NAMEYesName of your OpenStack project.—
SSH_PUB_KEYYesBase64-encoded public SSH key. This key is added to the default user on the virtual machines.—
SSH_KEYYesBase64-encoded private SSH key. Ansible uses this key to access the virtual machines.Masked and hidden
VAULT_PASSWORDNoPassword for group_vars/all/vault.yml.Masked and hidden
ANSIBLE_DIFFNoRuns Ansible with --diff.Set to true; unset or false runs without diff
FORCE_DEPLOYNoStarts deployment automatically on the main branch.Set to true; otherwise deployment is manual

You can create the base64 values for the SSH keys with the following commands:

base64 < sshkey.pub
base64 < sshkey

Copy the output of the first command to SSH_PUB_KEY and the output of the second command to SSH_KEY.

For the current CI/CD variable documentation, see the MIST pipeline documentation.

Request access to the MIST repository group

Request access to the Infrastructure as Code repository group. This group contains the MIST repositories and standardised modules needed to configure your project repositories and pipelines.

Create project repositories

In git.app.uib.no, create a repository group for your project. We recommend using the name of your NREC project for the group, but this is not required.

Within this group, create two repositories:

We also recommend creating a third repository for project documentation. Keeping it separate from the infrastructure and configuration repositories makes project-specific operational information easier to maintain and share.

Set up the Terraform repository

You can use the central MIST Terraform pipeline in your Terraform repository. Create a .gitlab-ci.yml file with the following configuration:

include:
  - project: sc/infrastructure-as-code/mist-pipelines/terraform
    ref: 1.0.0
    file: /nrec.yml

default:
  tags: []

The central pipeline includes a runner that is not available through self-service access. Setting default.tags to an empty list removes this runner tag, allowing the pipeline to run with the runner available to your project.

Create a main.tf file in the Terraform repository with the following initial configuration:

terraform {
  backend "http" {}
}

terraform {
  required_version = ">= 1.0"
  required_providers {
    openstack = {
      source = "terraform-provider-openstack/openstack"
      version = ">= 3.0.0"
    }
  }
}

resource "openstack_compute_keypair_v2" "ci_key_uib-it-vb-admin" {
  name       = "pipelinekey-uib-it-vb-admin"
  public_key = var.ssh_pub_key
}

provider "openstack" {
  region = var.nrec_region
}

Create a security.tf file to define the security group for your virtual machines. The SSH rules below allow connections from the UiB network and from the GitLab runners. The ssh_scrunner rule is required: it allows the GitLab runner that runs Ansible to connect to the virtual machines over SSH.

resource "openstack_networking_secgroup_v2" "defaultMNREC" {
  name        = "defaultMNREC"
  description = "Allow ICMP and SSH from UiB"
}

resource "openstack_networking_secgroup_rule_v2" "icmp_rule" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "icmp"
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

resource "openstack_networking_secgroup_rule_v2" "ssh_ipv4" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "129.177.0.0/16"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

resource "openstack_networking_secgroup_rule_v2" "ssh_git1" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "158.39.74.247/32"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

resource "openstack_networking_secgroup_rule_v2" "ssh_git2" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "158.37.64.190/32"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

resource "openstack_networking_secgroup_rule_v2" "ssh_scrunner" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "158.37.65.6/32"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

resource "openstack_networking_secgroup_rule_v2" "ssh_ipv6" {
  direction         = "ingress"
  ethertype         = "IPv6"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "2001:700:200::/48"
  security_group_id = openstack_networking_secgroup_v2.defaultMNREC.id
}

Create a variables.tf file with the variables used by the Terraform configuration:

variable "nrec_region" {
  description = "OpenStack Region"
  type        = string
}

variable "ssh_pub_key" {
  description = "SSH public key for OpenStack instances"
  type        = string
}

Example virtual machine

The following example creates a virtual machine, a persistent volume, and public and internal DNS records. It is a template and must be adapted for your project. In particular, update the DNS zones and names, image ID, flavor, key pair, security groups, network, and volume size to match your requirements.

resource "openstack_dns_recordset_v2" "prjman_record" {
  zone_id     = openstack_dns_zone_v2.mist-uib-no_zone.id
  name        = "projects.mist.uib.no."
  type        = "A"
  records     = [openstack_compute_instance_v2.prjman.access_ip_v4]
  ttl         = 3600
  description = "DNS entry prjman"
}

resource "openstack_dns_recordset_v2" "internalprjman_record" {
  zone_id     = openstack_dns_zone_v2.uib-it-vb-admin_zone.id
  name        = "projects.uib-it-vb-admin.uib.no."
  type        = "A"
  records     = [openstack_compute_instance_v2.prjman.access_ip_v4]
  ttl         = 3600
  description = "DNS entry internal project"
}

resource "openstack_compute_instance_v2" "prjman" {
  name            = "prjman"
  image_id        = "36ab8e92-a967-42ef-b618-bc6abdf0b98d"
  flavor_name     = "m1.xlarge"
  key_pair        = openstack_compute_keypair_v2.ci_key_uib-it-vb-admin.name
  security_groups = ["default", "defaultMNREC"]

  network {
    name = "dualStack"
  }

  depends_on = [
    openstack_blockstorage_volume_v3.prjman_vol
  ]
}

resource "openstack_blockstorage_volume_v3" "prjman_vol" {
  name        = "prjman_vol"
  size        = "20"
  description = "volume for prjman"

  lifecycle {
    prevent_destroy = true
  }
}

resource "openstack_compute_volume_attach_v2" "attach_prjman" {
  instance_id = openstack_compute_instance_v2.prjman.id
  volume_id   = openstack_blockstorage_volume_v3.prjman_vol.id
}

Optional: manage DNS zones with Terraform

If you want to manage DNS zones with Terraform, use the following as a starting point. Adapt the zone names, contact email, and descriptions to your project. The DNS record example above must reference the corresponding zone resources.

resource "openstack_dns_zone_v2" "uib-it-vb-admin_zone" {
  name        = "uib-it-vb-admin.uib.no."
  email       = "nrec-vb@uib.no"
  ttl         = 3600
  description = "DNS-Zone uib-it-vb-admin"
  type        = "PRIMARY"
}

resource "openstack_dns_zone_v2" "mist-uib-no_zone" {
  name        = "mist.uib.no."
  email       = "nrec-vb@uib.no"
  ttl         = 3600
  description = "DNS-Zone mist.uib.no"
  type        = "PRIMARY"
}

Run the Terraform pipeline

After you have completed the configuration and set the CI/CD variables described above, run the pipeline in your Terraform repository. The pipeline applies the Terraform configuration and creates the NREC resources defined in your repository.

Set up the Ansible repository

The following example shows a typical directory structure for an Ansible project repository. Adapt it to the roles and services used by your project.

.
├── ansible.cfg
├── files
│   ├── elabftw
│   │   ├── metadata.xml
│   │   └── metadata.xml.bkp
│   ├── proxy
│   │   ├── favicon.svg
│   │   └── index.html
│   └── rspace
│       └── opt
│           └── rspace
│               ├── configs
│               │   ├── apache.setup
│               │   ├── db-config.cnf
│               │   ├── server.xml
│               │   └── tomcat-locate.sh
│               └── templates
│                   └── db-template.sql
├── group_vars
│   ├── all
│   │   ├── main.yml
│   │   └── vault.yml
│   ├── uib_ita_fd_eln_dev_app
│   │   └── main.yml
│   └── uib_ita_fd_eln_dev_proxy
│       └── main.yml
├── host_vars
│   ├── chemotion
│   │   └── main.yml
│   ├── elabftw
│   │   └── main.yml
│   ├── main.yml
│   └── rspace
│       └── main.yml
├── inventory
├── postchecks
│   ├── chemotion.gitlab-ci.yml
│   ├── elabftw.gitlab-ci.yml
│   ├── proxy.gitlab-ci.yml
│   └── rspace.gitlab-ci.yml
├── precheck.yml
├── README.md
├── requirements.yml
└── site.yml

Use the central MIST Ansible pipeline in your Ansible repository. Create a .gitlab-ci.yml file with the following configuration:

---
include:
  - project: sc/infrastructure-as-code/mist-pipelines/ansible
    ref: 1.0.0
    file: /ansible.yml

default:
  tags: []

The central pipeline defines a runner tag that is not available through self-service access. Setting default.tags to an empty list unsets this tag, so that the pipeline uses the shared runner available to your project.

Create an ansible.cfg file in the root of the Ansible repository:

[defaults]
inventory   = inventory
host_key_checking = False
retry_files_enabled = False
stdout_callback = yaml
deprecation_warnings = False
roles_path = ./roles:./roles/galaxy:/etc/ansible/roles
pipelining = True
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 86400

Create an inventory file in the root of the Ansible repository. The following is an example only: replace the inventory group names, virtual machine names, DNS names, and Ansible users with the values for your own virtual machines.

[uib_it_vb_admin_prod]
prometheus ansible_host=prometheus.uib-it-vb-admin.uib.no
content ansible_host=content.uib-it-vb-admin.uib.no

[uib_it_vb_admin_prod:vars]
ansible_user=rocky
ansible_python_interpreter=/usr/libexec/platform-python

[uib_it_vb_admin_alma10_prod]
mist ansible_host=mist.uib-it-vb-admin.uib.no
loki ansible_host=loki.uib-it-vb-admin.uib.no
grafana ansible_host=grafana.uib-it-vb-admin.uib.no

[uib_it_vb_admin_alma10_prod:vars]
ansible_user=almalinux
ansible_python_interpreter=/usr/bin/python3

[uib_it_vb_admin_alma10_test]
prjman ansible_host=projects.uib-it-vb-admin.uib.no

[uib_it_vb_admin_alma10_test:vars]
ansible_user=almalinux

Create a precheck.yml file in the root of the Ansible repository. This file is required by the central pipeline and runs the prechecks role against all hosts.

---
- name: Run Prechecks only
  hosts: all
  become: true
  roles:
    - role: prechecks

If your virtual machines do not use Puppet images, set the following variable in inventory variables instead of in precheck.yml. You can set it for all hosts in group_vars/all/main.yml, or set it in the relevant group_vars or host_vars file:

---
prechecks_puppet_image: false

Create a requirements.yml file to declare the Ansible collections and roles used by your project. Add only the roles your project needs. The following is an example:

---
collections:
  - name: community.general
    version: ">=8.0.0"
  - name: ansible.posix

roles:

# standard roles

  - name: hostfile
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-hostfile.git"
    scm: git
    version: "${ROLE_HOSTFILE_VERSION}"

  - name: lvm
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-lvm.git"
    scm: git
    version: "${ROLE_LVM_VERSION}"

  - name: prechecks
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-prechecks.git"
    scm: git
    version: "${ROLE_PRECHECKS_VERSION}"

  - name: prometheus_node_exporter
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-prometheus_node_exporter.git"
    scm: git
    version: "${ROLE_PROMETHEUS_NODE_EXPORTER_VERSION}"

  - name: promtail
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-promtail.git"
    scm: git
    version: "${ROLE_PROMTAIL_VERSION}"

  - name: sudoadd
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-sudoadd.git"
    scm: git
    version: "${ROLE_SUDOADD_VERSION}"

  - name: sysprep
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-sysprep.git"
    scm: git
    version: "${ROLE_SYSPREP_VERSION}"

  - name: userlogins
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-userlogins.git"
    scm: git
    version: "${ROLE_USERLOGINS_VERSION}"

# NFS NETAPP

  - name: msdefender
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-msdefender.git"
    scm: git
    version: "${ROLE_MSDEFENDER_VERSION}"

  - name: nfsmount
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-nfsmount.git"
    scm: git
    version: "${ROLE_NFSMOUNT_VERSION}"

  - name: nginx
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-nginx.git"
    scm: git
    version: "${ROLE_NGINX_VERSION}"

  - name: docker
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-docker.git"
    scm: git
    version: "${ROLE_DOCKER_VERSION}"

  - name: local_users_and_groups
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-local_users_and_groups.git"
    scm: git
    version: "${ROLE_LOCUSERSGROUPS_VERSION}"

  - name: iptables
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-iptables.git"
    scm: git
    version: "${ROLE_IPTABLES_VERSION}"

  - name: files_and_pkgs
    src: "git+https://oauth2:${GITLAB_ACCESS_TOKEN}@git.app.uib.no/sc/infrastructure-as-code/shared-ansible-roles/ansible-role-files_and_pkgs.git"
    scm: git
    version: "${ROLE_FILESANDPKGS_VERSION}"

All available roles can be found in the shared Ansible roles repository.

Create a site.yml file to define which roles are applied to each inventory group. The following is a simple example; adapt the host groups and roles to your project, and make sure that every referenced role is listed in requirements.yml.

---
- name: Configure entire project
  hosts: uib_clarino_repo
  become: true
  roles:
    - role: sysprep
    - role: hostfile
    - role: prometheus_node_exporter
    - role: promtail
    - role: userlogins
    - role: sudoadd

- name: Configure proxy server
  hosts: proxy
  become: true
  roles:
    - role: iptables
    - role: lvm
    - role: apache

- name: Configure backup server
  hosts: backup
  become: true
  roles:
    - role: lvm
    - role: local_users_and_groups
    - role: nfsmount
    - role: files_and_pkgs
    - role: sourcebuild
    - role: cronjobs

Create group_vars files for settings shared by an inventory group, and host_vars files for settings that apply to individual hosts. Before configuring a role, read the README.md in that role's repository to understand the available options and their expected values. The MIST roles are documented in the shared Ansible roles repository.

Run the Ansible pipeline

After you have added and committed the Ansible configuration, run the pipeline in the Ansible repository. The pipeline installs the declared roles, validates the configuration, and runs the prechecks. After the checks have completed successfully, start the deployment job to configure your virtual machines. Set FORCE_DEPLOY to true if the deployment should start automatically on the main branch.

Support & Contact