# Translating an Openshift Template into an Operator
## Operators
Operator is a piece of software in Kubernetes, that encodes the domain knowledge of the application, and also extends the Kubernetes API through the Third Party Resources mechanism. Using the extended API users can create, configure and manage the applications. Operators in general manage multiple instances across the cluster. In simple words, 'Operator is like a code representation of operational logic with expertise'. Operators extend the Kubernetes features for the stateless applications can handle complex stateful application with complex logic related to databases, migration, etc.
### Operator-Framework
The Operator Framework is an open source toolkit to manage Kubernetes native applications or Operators, in an effective, automated, and scalable way.
### Operator Directory Structure
The operator directory is divided into different sub-directories & files. Each of them are explained below
* build – This folder contains the scripts for building the docker image of the operator
* deploy - This folder contains the generic set of deployment manifests, which are used to deploy the operator on a Kubernetes cluster
* molecule - This folder contains ansible files used for testing the operator
* playbook.yaml - This file is the main playbook file for the operator to deploy an instance. It contains the information about what all roles to run when requested for an instance
* roles - All the roles listed in the playbook.yaml file are included in this with their respective sub-directories
* watches.yaml - This contains the information related to Group, Version, Kind, Ansible invocation method and Ansible configuration
## Operator-SDK
Operator-sdk is a component of the operator-framework project which provides workflows to build operators in either Go, Ansible or Helm.
### Installing operator-sdk
The first step to building an operator in Ansible using operator SDK is to install the operator-sdk CLI (command line tool). The tool can be further used to create skeleton, build a docker image and others. The operator-sdk CLI can be installed using the following commands
Set the release version variable
`RELEASE_VERSION=v0.10.0`
Download the binary
`curl -OJL https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu`
The installation can be verified by running the following command with executable `./operator-sdk –version`
## Building the Discourse operator
### Initializing the operator
The operator directory can be initialized by running the command
`operator-sdk new discourse-operator --api-version=cache.example.com/v1alpha1 --kind=Discourse --type=ansible`
The flag kind specifies the name of the CRD, while type specifies the option chosen to write the operator in
### Adding & modifying the necessary files
Now that we have the directory structure ready. The next step is to add manifest files and roles. For the Discourse operator, we have 2 set of roles namely ‘Discourse’ and ‘Redis’. In order to create the directory structure for each of these roles, I have used ansible-galaxy. ansible-galaxy is a command line tool for installing, creating & managing roles. This comes installed with the ansible package. Although the directories & sub-directories can be created manually without the ansiblegalaxy, this is a clean and efficient way of doing it. In order to create the role directory structure, navigate to the roles in the root directory & run the following commands
```
ansible-galaxy init discourse
ansible-galaxy init redis
```
These commands will create directories discourse and redis, with many other sub-directories inside. The next step is to add roles & change the configuration files.
#### Adding variables to var/main.yml
Add the following content to the file ‘discourse-operator/roles/discourse/vars/main.yml’.
```
db_host_value: "{{ lookup('env','db_host_value') }}"
db_port_value: "{{ lookup('env','db_port_value') }}"
db_name_value: "{{ lookup('env','db_name_value') }}"
db_username_value: "{{ lookup('env','db_username_value') }}"
db_password_value: "{{ lookup('env','db_password_value') }}"
developers_email_value: "{{ lookup('env','developer_emails_value') }}"
```
These can be considered as variables within the scope of all the ansible roles. In other words, these are the parameters to be given as input by the user to the operator in order to
create an instance.
#### Adding tasks to discourse/tasks
From the openshift-template file, each of the discourse components namely configmaps,
deploymentconfigs, persistent volumes, routes and services are add as a separate file. Each
of the file’s contents are as follows
*configmaps.yml*
```
###################
# CONFIGMAPS #
###################
- name: Create env configmap
k8s:
definition:
kind: ConfigMap
apiVersion: v1
metadata:
name: env-configmap
namespace: '{{ namespace }}'
data:
DISCOURSE_DB_HOST_KEY: "{{ db_host_value }}"
DISCOURSE_DB_PORT_KEY: "{{ db_port_value }}"
DISCOURSE_DB_NAME_KEY: "{{ db_name_value }}"
DISCOURSE_DB_USERNAME_KEY: "{{ db_username_value }}"
DISCOURSE_DB_PASSWORD_KEY: "{{ db_password_value }}"
DISCOURSE_DEVELOPER_EMAILS_KEY: "{{ developers_email_value }}"
DISCOURSE_DB_POOL_KEY: "8"
LANG: "en_US.UTF-8"
SIDEKIQ_CONCURRENCY_KEY: "5"
UNICORN_PID_PATH: "/var/run/unicorn.pid"
UNICORN_PORT: "3000"
UNICORN_SIDEKIQS: "1"
UNICORN_WORKERS: "1"
RUBY_GC_HEAP_GROWTH_MAX_SLOTS: "40000"
RUBY_GC_HEAP_INIT_SLOTS: "400000"
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR: "1.5"
- name: Create nginx configmap
k8s:
namespace: '{{ namespace }}'
resource_definition: "{{ lookup('file', 'nginx-configmap.yml') }}"
- name: Create discourse configmap
k8s:
namespace: '{{ namespace }}'
resource_definition: "{{ lookup('file', 'discourse-configmap.yml') }}"
```
*deploymentconfig.yml*
```
###################
# DEPLOYMENTS #
###################
- name: Create discourse deployment config
k8s:
definition:
kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
labels:
app: discourse-cern
name: webapp
namespace: '{{ namespace }}'
spec:
replicas: 1
selector:
app: discourse-cern
deploymentconfig: webapp
strategy:
rollingParams:
timeoutSeconds: 1200
type: Rolling
template:
metadata:
labels:
app: discourse-cern
deploymentconfig: webapp
spec:
containers:
-
name: nginx
command:
- ./run-nginx.sh
image: discourse-cern:v2.3.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: 400Mi
cpu: 200m
requests:
memory: 20Mi
cpu: 50m
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /discourse/public/uploads
name: discourse-uploads
- mountPath: /discourse/public/assets
name: discourse-public-assets
- mountPath: /discourse/public/backups
name: discourse-backups
- mountPath: /tmp/nginx/
name: nginx-configmap
- mountPath: /var/cache/nginx
name: var-cache-nginx
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 10
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 8080
timeoutSeconds: 10
-
name: webapp
image: discourse-cern:v2.3.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
protocol: TCP
resources:
limits:
# limit as per https://github.com/discourse/discourse/blob/master/docs/ADMIN-QUICK-START-GUIDE.md#maintenance
memory: 1Gi
cpu: 1
requests:
memory: 640Mi
cpu: 400m
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /tmp/discourse-configmap
name: discourse-configmap
- mountPath: /discourse/public/uploads
name: discourse-uploads
- mountPath: /discourse/public/backups
name: discourse-backups
- mountPath: /discourse/public/assets
name: discourse-public-assets
- mountPath: /discourse/tmp
name: discourse-tmp
- mountPath: /discourse/logs
name: discourse-logs
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 10
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 900
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 10
env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
value: "$(NAMESPACE).web.cern.ch"
- name: DISCOURSE_CONFIG_DB_HOST
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_HOST_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PORT
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PORT_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_NAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_NAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_USERNAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_USERNAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PASSWORD
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PASSWORD_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DEVELOPER_EMAILS
valueFrom:
configMapKeyRef:
key: DISCOURSE_DEVELOPER_EMAILS_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_POOL
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_POOL_KEY
name: env-configmap
- name: UNICORN_PID_PATH
valueFrom:
configMapKeyRef:
key: UNICORN_PID_PATH
name: env-configmap
- name: UNICORN_PORT
valueFrom:
configMapKeyRef:
key: UNICORN_PORT
name: env-configmap
- name: UNICORN_SIDEKIQS
valueFrom:
configMapKeyRef:
key: UNICORN_SIDEKIQS
name: env-configmap
- name: UNICORN_WORKERS
valueFrom:
configMapKeyRef:
key: UNICORN_WORKERS
name: env-configmap
- name: SIDEKIQ_CONFIG_CONCURRENCY
valueFrom:
configMapKeyRef:
key: SIDEKIQ_CONCURRENCY_KEY
name: env-configmap
- name: RUBY_GC_HEAP_GROWTH_MAX_SLOTS
valueFrom:
configMapKeyRef:
key: RUBY_GC_HEAP_GROWTH_MAX_SLOTS
name: env-configmap
- name: RUBY_GC_HEAP_INIT_SLOTS
valueFrom:
configMapKeyRef:
key: RUBY_GC_HEAP_INIT_SLOTS
name: env-configmap
- name: RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR
valueFrom:
configMapKeyRef:
key: RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR
name: env-configmap
- name: LANG
valueFrom:
configMapKeyRef:
key: LANG
name: env-configmap
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext:
capabilities: {}
privileged: false
initContainers:
- name: init-dbmigration
command:
- ./init-dbmigration.sh
env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
value: $(NAMESPACE).web.cern.ch
- name: DISCOURSE_CONFIG_DB_HOST
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_HOST_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PORT
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PORT_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_NAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_NAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_USERNAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_USERNAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PASSWORD
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PASSWORD_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DEVELOPER_EMAILS
valueFrom:
configMapKeyRef:
key: DISCOURSE_DEVELOPER_EMAILS_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_POOL
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_POOL_KEY
name: env-configmap
- name: LANG
valueFrom:
configMapKeyRef:
key: LANG
name: env-configmap
image: discourse-cern:v2.3.0
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 200m
memory: 320Mi
volumeMounts:
- mountPath: /tmp/discourse-configmap
name: discourse-configmap
- mountPath: /discourse/public/uploads
name: discourse-uploads
- name: init-assets
command:
- ./init-assets.sh
env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
value: $(NAMESPACE).web.cern.ch
- name: DISCOURSE_CONFIG_DB_HOST
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_HOST_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PORT
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PORT_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_NAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_NAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_USERNAME
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_USERNAME_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_PASSWORD
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_PASSWORD_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DEVELOPER_EMAILS
valueFrom:
configMapKeyRef:
key: DISCOURSE_DEVELOPER_EMAILS_KEY
name: env-configmap
- name: DISCOURSE_CONFIG_DB_POOL
valueFrom:
configMapKeyRef:
key: DISCOURSE_DB_POOL_KEY
name: env-configmap
- name: LANG
valueFrom:
configMapKeyRef:
key: LANG
name: env-configmap
image: discourse-cern:v2.3.0
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 200m
memory: 320Mi
volumeMounts:
- mountPath: /tmp/discourse-configmap
name: discourse-configmap
- mountPath: /discourse/public/assets
name: discourse-public-assets
volumes:
- name: discourse-configmap
configMap:
name: discourse-configmap
- name: nginx-configmap
configMap:
name: nginx-configmap
- name: discourse-public-assets
emptyDir: {}
- name: discourse-tmp
emptyDir: {}
- name: discourse-logs
emptyDir: {}
- name: var-cache-nginx
emptyDir: {}
- name: discourse-uploads
emptyDir: {}
# persistentVolumeClaim:
# claimName: discourse-uploads
- name: discourse-backups
emptyDir: {}
# persistentVolumeClaim:
# claimName: discourse-backups
triggers:
- type: ConfigChange
- type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- "init-dbmigration"
- "init-assets"
- "webapp"
- "nginx"
from:
kind: "ImageStreamTag"
name: 'discourse-cern:v2.3.0'
namespace: openshift
```
*pvc.yml*
For this operator, I have used a local OKD cluster which comes with default PV’s. Therefore, for this reason, pvc are not configured for now. When deploying to production, these roles need to be configured and added.
*route.yml*
```
###################
# ROUTES #
###################
- name: Create route
k8s:
definition:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
labels:
app: discourse-cern
name: nginx
namespace: '{{ namespace }}'
spec:
port:
targetPort: 8080-tcp
to:
kind: Service
name: nginx
weight: 100
tls:
termination: "edge"
insecureEdgeTerminationPolicy: Redirect
```
*services.yml*
```
###################
# SERVICES #
###################
- name: Create a nginx service
k8s:
definition:
kind: Service
apiVersion: v1
metadata:
labels:
app: discourse-cern
name: nginx
namespace: '{{ namespace }}'
spec:
ports:
- name: 8080-tcp
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: discourse-cern
deploymentconfig: webapp
sessionAffinity: None
type: ClusterIP
- name: Create a webapp service
k8s:
definition:
kind: Service
apiVersion: v1
metadata:
labels:
app: discourse-cern
name: webapp
namespace: '{{ namespace }}'
spec:
ports:
- name: 3000-tcp
port: 3000
protocol: TCP
targetPort: 3000
selector:
app: discourse-cern
deploymentconfig: webapp
sessionAffinity: None
type: ClusterIP
```
*main.yml*
Configuring the main.yml in roles/discourse/tasks
Now that we have added all the ansible roles, all these roles have to be connected in the main.yml file. The contents of the file are as follows
```
##############################################################################
## Provision Discourse site
##############################################################################
- name: Provision Discourse site
block:
- name: Configure Configmaps
include_tasks: configmaps.yml
# - name: Configure PVCs
# include_tasks: pvc.yml
- name: Configure DeploymentConfig
include_tasks: deploymentconfig.yml
- name: Configure Services
include_tasks: services.yml
- name: Configure Routes
include_tasks: route.yml
rescue:
#- include_role:
# name: mail-handler
# tasks_from: failure
- name: Writing Termination Message '/dev/termination-log'
shell: >
echo "Failed task -> {{ ansible_failed_task.name }}.
Error was -> {{ ansible_failed_result.msg }}"
> /dev/termination-log
- fail:
msg: "Error in task {{ ansible_failed_task.name }}: {{ ansible_failed_result.msg }}"
```
#### Configmaps of discourse roles
Since the configmaps are huge to put in a single file, for the purpose of brevity they are separated into two files discourse-configmap.yml and nginx-configmap.yml in the directory discourse-operator/roles/discourse/files. And these files are configured respectively the configmaps roles i.e discourse-operator/roles/discourse/tasks/configmaps.yml
There are no changes required besides these in the discourse roles.
#### Redis role
Like the discourse roles, config files have to be added to the redis roles directory. The tasks in the redis roles are as follows
*deploymentconfig.yml*
```
- name: Create a redis deployment config
k8s:
definition:
- kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
labels:
app: discourse-cern
name: redis
namespace: '{{ namespace }}'
spec:
replicas: 1
selector:
app: discourse-cern
deploymentconfig: redis
strategy:
type: Rolling
template:
metadata:
labels:
app: discourse-cern
deploymentconfig: redis
spec:
containers:
- image: ' '
imagePullPolicy: Always
name: redis
ports:
- containerPort: 6379
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/lib/redis/data
name: redis-1
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 6379
timeoutSeconds: 10
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 6379
timeoutSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
volumes:
- emptyDir: {}
name: redis-1
test: false
triggers:
- type: ConfigChange
- type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- "redis"
from:
kind: "ImageStreamTag"
name: 'redis:3.2'
namespace: openshift
```
*service.yml*
```
- name: Create a redis service
k8s:
definition:
kind: Service
apiVersion: v1
metadata:
labels:
app: discourse-cern
service: redis
name: redis
namespace: '{{ namespace }}'
spec:
type: ClusterIP
sessionAffinity: None
ports:
- name: 6379-tcp
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: discourse-cern
deploymentconfig: redis
```
*main.yml*
The main.yml also needs to be configured specifying the tasks to run
```
##############################################################################
## Provision Redis
##############################################################################
- name: Provision Redis
block:
- name: Provision Redis DeploymentConfig
include_tasks: deploymentconfig.yml
- name: Provision Redis Service
include_tasks: service.yml
rescue:
#- include_role:
# name: mail-handler
# tasks_from: failure
- name: Writing Termination Message '/dev/termination-log'
shell: >
echo "Failed task -> {{ ansible_failed_task.name }}.
Error was -> {{ ansible_failed_result.msg }}"
> /dev/termination-log
- fail:
msg: "Error in task {{ ansible_failed_task.name }}: {{ ansible_failed_result.msg }}"
```
#### Integrating all the roles in the watches.yaml file & playbook.yaml file
The playbook.yaml file should contain the all roles that are to be run by the operator. The file simply lists all the roles, which in our case are discourse and redis.
The contents of the file are follows
```
- hosts: localhost
tasks:
- debug: msg="Running Discourse Operator Playbook"
- import_role:
name: "redis"
- import_role:
name: "discourse"
```
The watches.yaml, besides Ansible configuration, should also include the path of the playbook.yaml file. The playbook.yaml file path should be relative to the docker image, i.e the path to which the file is copied to when building the Docker image. The contents of the file are as follows
```
---
- version: v1alpha1
group: discourse.cern
kind: Discourse
playbook: /opt/ansible/playbook.yaml
# reconcilePeriod: 5m
# manageStatus: false
watchDependentResources: False
# role: /opt/ansible/roles/discourse
```
#### Modifying the Dockerfile to include all the roles & tasks
The Dockerfile present in the directory `discourse-operator/build/` should look as follows
```
FROM quay.io/operator-framework/ansible-operator:v0.9.0
COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbook.yaml ${HOME}/playbook.yaml
```
#### Adding manifest files for the operator deployment
Now that we have added logic of what to do when a Discourse instance if requested, we have to now add Kubernetes manifest files that will deploy the Discourse operator on the cluster, which will then manage all the instances for us. These files are auto generated by the operator-sdk in the deploy directory. The files that are created are as follows
1. crds
i. discourse_v1alpha1_discourse_cr.yaml
ii. discourse_v1alpha1_discourse_crd.yaml
2. imagestream.yaml
3. operator.yaml
4. role.yaml
5. role-binding.yaml
6. service_account.yaml
The files that are to be modified are operator.yml and crds/discourse_v1alpha1_discourse_cr.yaml.
In the operator.yml, only the name of the image and imagePullPolicy have to be modified. These parameters will be discussed in the next sub-section
The crds/discourse_v1alpha1_discourse_cr.yaml has to be modified to include the input parameters that are to be sent to the operator when creating an instance. The contents of this file are follows
```
apiVersion: discourse.cern/v1alpha1
kind: Discourse
metadata:
name: example-discourse
spec:
# Add fields here
size: 3
namespace: discourse-operator
version: latest
category: personal
db_host_value: <host>
db_name_value: <db_name>
db_username_value: <username>
db_password_value: <password>
developers_email_value: rajula.vineet.reddy@cern.ch
```
### Building the operator image
Once all the files are put in the respective directories, the docker image can be built by running the command operator-sdk build <image_name>. This will use the Dockerfile from the deploy folder in order to build the image. In order to automate this, I have integrated GitLab CI/CD along with SVC to build the image. The setup is in such a way that, all my code for the operator is hosted on GitLab, and whenever I make a commit with a tag, GitLab triggers it’s CI, which will then build the Docker image in it’s runner and stores the image in it’s Registry Server. I can then use the image URL from the registry and configure it in the deploy/operator.yml file.
The configuration of gitlab-ci.yml, The GitLab CI files is as follows
```
stages:
- build
build docker image with host daemon:
tags:
- docker-privileged
stage: build
image: docker:latest
script:
- docker info
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- docker build --pull -f $CI_PROJECT_DIR/build/Dockerfile -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- tags
```
### Running the operator
Once the operator image is built & stored in a local/ remote registry, using the image URL and the manifest files, the operator can be deployed. Detailed steps to deploy an operator are as follows
1. Clone the repository containing the operator files
`git clone https://gitlab.cern.ch/rvineetr/discourse-operator.git`
`cd discourse-operator`
2. Create the CRD (Custom Resource Definition)
`oc create -f deploy/crds/discourse_v1alpha1_discourse_crd.yaml`
3. Create all the components of the operator in a new project
`oc new-project discourse-operator`
`oc create -f deploy/`
4. Give admin permissions to the operator user
`oc adm policy add-cluster-role-to-user admin -z discourse-operator`
5. Verify if the operator is running
`oc get pods`
`oc logs <pod_name> -c operator -f`
### Provisioning an instance
Now that the operator is running, an instance can be created using the operator. To create an instance, the CR (Custom Resource) file deploy/crds/discourse_v1alpha1_discourse_cr.yaml has to be executed. Before running it, the parameters have to be configured. For now, since there is no way to get a database server, I have started a PostgreSQL server locally and have configured it with the CR. Once I have the CR ready, it can be created by running the command `oc create -f deploy/crds/discourse_v1alpha1_discource_cr.yaml` This will trigger the operator, which will then create an instance using the ansible roles provided by us. It is important to configure the namespace parameter in the CR. The CR will be deployed in the namespace specified.
## Postgres Operator
Instead of writing an operator for the database from scratch, we have decided to explore existing open source options from Operatorhub.io. There were 2-3 options available for PostgreSQL. After trying to deploy one of them namely Postgres-Operator by Zalanado, I have realized the architecture of this operator was complex and there were compatibility issues with the OLM (Operator Lifecycle Manager) and the OKD 3.11 version. So, I have moved on to a different option. The other option was Crunchy PostgreSQL Enterprise by Crunchy Data. This one had good amount of documentation and support from the community. After trying to deploy an older version 3.9.x for about a week I ended with lot of bugs. Later found the documentation & release of a newer version 4.0.0, which worked perfectly. I succeeded in deploying the postgres-operator and then connecting it with the Discourse operator. Although, there were some openshift issues I encountered during this process, was able to resolve/ by pass them by taking to Alexandre Lossent and Iago Santos Pardo. More about these issues in the Miscellaneous section.
This operator has a CLI client, that comes with the operator, which is mandatory to communicate with the postgres-operator to create/ delete/ manage instances. The steps to deploy this Crunchy PostgreSQL operator are as follows
- `mkdir -p $HOME/odev/src/github.com/crunchydata $HOME/odev/bin $HOME/odev/pkg`
- `cd $HOME/odev/src/github.com/crunchydata`
- `git clone https://github.com/CrunchyData/postgres-operator.git`
- `cd postgres-operator`
- `git checkout 4.0.0`
- `cat $HOME/odev/src/github.com/crunchydata/postgres-operator/examples/envs.sh >> $HOME/.bashrc`
- `source $HOME/.bashrc`
- `export NAMESPACE=pgouser1,pgouser2`
- `export PGO_OPERATOR_NAMESPACE=pgo`
- `make setupnamespaces`
- Change the storage options in `conf/postgresql-operator/pgo.yaml` to `hostpathstorage`
Change from
```
PrimaryStorage: storageos
BackupStorage: storageos
ReplicaStorage: storageos
BackrestStorage: storageos
```
to
```
PrimaryStorage: hostpathstorage
BackupStorage: hostpathstorage
ReplicaStorage: hostpathstorage
BackrestStorage: hostpathstorage
```
- Install expenv
`wget https://github.com/blang/expenv/releases/download/v1.2.0/expenv_amd64.tar.gz`
`tar -xzf expenv_amd64.tar.gz expenv`
`cp expenv /usr/bin`
- `cp ./conf/postgres-operator/pgouser $HOME/.pgouser`
- `cp ./conf/postgres-operator/pgorole $HOME/.pgorole`
- `make installrbac`
- `make deployoperator`
- `wget https://github.com/CrunchyData/postgres-operator/releases/download/4.0.0/pgo -O /usr/bin/pgo` - Since the version of postgres operator is 4.0.0
- `chmod 777 /usr/bin/pgo` - Give executable permissions to the downloaded 'pgo' file
<!-- - `oc get service postgres-operator -n pgo` -->
- `IP=$(oc get svc postgres-operator -n pgo -o jsonpath='{range.spec}{.clusterIP}')`
- `export PGO_APISERVER_URL=https://$(oc get svc postgres-operator -n pgo -o jsonpath='{range.spec}{.clusterIP}'):8443`
- `pgo version`
- `pgo create cluster mycluster -n pgouser1`
- `pgo show cluster mycluster -n pgouser1`
- `pgo create user user1 --selector=name=mycluster --password=newpass` - Create a new user with a password
- `pgo user --change-password=postgres --selector=name=mycluster --password=newpass` - Updated an existing user with a given password
- `pgo test mycluster -n pgouser1` - Testing the cluster
- `pgo scale mycluster -n pgouser1` - Scaling the cluster
# Misc
## Setting up a OKD 3.11 cluster
- Pre-requisities -
- `oc cluster up`
- SSH tunneling inorder to access the console from the VM - https://github.com/openshift/origin/issues/19699#issuecomment-434367748 `sudo ssh -L 8443:localhost:8443 -f -N user@host`
- Login with *developer* & any password
- Giving admin privileges to user 'developer' - `oc adm policy add-cluster-role-to-user cluster-admin developer`
## Namespace/ Project stuck in terminating
- https://stackoverflow.com/a/52412965
# Future work
- Install 'pgo' inside the operator image and configure it with the postgres operator API server using the DNS name and trigger postgres whenever request for a Discourse instance
- Add tests
- Integrate with the OLM\
# References
- https://gitlab.cern.ch/rvineetr/discourse-operator
- https://gitlab.cern.ch/webservices/discourse-cern/blob/v2.3.0/templates/discourse-cern.yaml
- https://github.com/operator-framework/operator-sdk/blob/ff8d91200872c7255c7a1c4c2bc2f6b7539ea1f1/doc/ansible/dev/dependent_watches.md
- https://gitlab.cern.ch/drupal/paas/drupal8-operator
- https://operatorhub.io/operator/postgresql
- https://indico.cern.ch/event/830002/contributions/3523481/attachments/1892895/3122120/Rajula_Vineet_Reddy_-_Discourse_Forum_Automation.pptx
- https://codimd.web.cern.ch/s/SyySaIkNr#
- https://gitlab.cern.ch/rvineetr/discourse-operator/-/jobs/4774013
- https://access.crunchydata.com/documentation/postgres-operator/4.0.0/installation/operator-install/
- https://docs.openshift.com/container-platform/4.1/applications/operators/olm-understanding-olm.html
- https://medium.com/@fabiojose/working-with-oc-cluster-up-a052339ea219
- https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/user-guide.md