Core Compute Services - Cloud & IaaS

## Core Compute Services #### Cloud & IaaS ###### Giacomo Tenaglia - CSC on IT Services - 2024-11-05 ---  ## Who am I * :spaghetti: MSc Computer Science * :computer: Joined CERN in 2007 (TECH) * :male-construction-worker: Config Services @CERN IT-CD-CC * :blue_heart: CERN OSPO founder & IT representative * :penguin: since 1999 ---  ## My assumptions on you * Love interrupting when things are not clear. * Have an `ssh` client installed on your laptop. * Are familiar with Linux administration concepts. ---  ## Traditional computing models ---  ---  ---  ## Cloud computing ---  ---  ---  ---  ---  ## The CERN Private Cloud * Full-blown IaaS in production since 2013: * Hosted at CERN; * Self-service, programmable, elastic, efficient; * Based on [Openstack](https://www.openstack.org/) (open-source); * Integrated with CERN infra (network/LanDB, authentication, Linux/Windows, ...). * https://cern.ch/clouddocs --- ## Exercise 0.1 ###### Get the environment setup ``` ssh lxplus.cern.ch openstack help openstack project list env | grep OS_ export OS_PROJECT_NAME="CSC IT Services" openstack server list ``` * Notes: * We use `bash` (sorry I'm a boomer). * `bash` completion for `openstack` is awesome. Does it work? * `--fit-width` helps. --- ## Exercise 0.2 ###### Get the environment setup * Horizon UI: https://openstack.cern.ch/ * Locate the `CSC IT Services` project and inspect its quota. * Locate `hello-csc-2024.cern.ch` and its properties. ---  ## Compute instance management ###### Openstack Nova * Create and manage VMs. --- ## Exercise 1.1 ###### Instance management * Create a `keypair` (Horizon/CLI) or import an existing one. * Create an instance (`openstack server create`) with the following specs: * Flavour: `m2.small` * Image: `RHEL9 - x86_64` * Name: `$USER-csc-2024` * Your key-pair * Verify in Horizon that the instance is being created. * [ASYNC] Access the instance as `root` via SSH. ---  ## Instance management: extras * [LanDB properties](https://clouddocs.web.cern.ch/using_openstack/properties.html#other-cern-specific-properties) * Setting properties: "Main user", "Responsible", "IPv6 ready", ... * [VM placement](https://clouddocs.web.cern.ch/availability_techniques/vm_placement.html): * Availability zones: a, b, c (Meyrin) * Different local network switch and power inputs. * Data centre: `cern` vs `pdc` (need dedicated resource request). * Server groups: affinity/anti-affinity (host/rack, `pdc` first). ---  ## A word on [data centres](https://clouddocs.web.cern.ch/datacentre.html#differences-between-datacentres) * Meyrin Data Centre (MDC, Building 513): `OS_REGION_NAME=cern` * Prévessin Data Centre (PDC, Building 775): `OS_REGION_NAME=pdc` ---  ---  ## Image management ###### Openstack Glance * Predominant format: `qcow2` (QEMU CoW). * CERN-supported image library: * Alma, RHEL, Windows, ... * `openstack image list` * *Golden* images management: * Reference/master image to help standardisation. * Complementary to config management. * Backup/restore: * Regular vs ad-hoc. ---  ## Block storage management ###### Openstack Cinder * Persistent virtualised block devices independent of any particular instance. * Can only be attached to a single instance at a time. * Several [volume types](https://clouddocs.web.cern.ch/block_storage/create_block_volumes.html#available-volume-types): * Performances, location, placement. * Use-cases: "Boot from volume", snapshots, backups. * At CERN: using IT-SD Ceph service (RBD): * HDD: replica x3, SSD: erasure-coding 8+3. * *NOTE: please do not do SW RAID!* ---  ## File shares management ###### Openstack Manila * Network accessible filesystem for Linux. * Mountable from several clients at once. * Several [share types](https://clouddocs.web.cern.ch/file_shares/share_types.html): * HDD/SSD, location. * At CERN: using IT-SD Ceph service (CephFS). ---  ## Other Openstack components * Ironic: bare-metal machines management. * Magnum: containers/k8s as a service. --- ## Exercise 2.1 ###### Image management: clone an instance * Install the Apache web-server: ``` dnf install -y httpd systemctl enable httpd systemctl start httpd firewall-cmd --permanent --zone=public --add-port=80/tcp ``` * Drop a file in `/var/www/html`: ``` echo "My CSC test" > /var/www/html/content ``` * Make sure you can access it: ``` curl http://${USER}-csc-2024.cern.ch/content ``` --- ## Exercise 2.2 ###### Image management: clone an instance (Linux only!) * Create a snapshot: ``` openstack server image create --name $USER-snapshot --wait $USER-csc-2024 ``` * Download the image: ``` mkdir /tmp/$USER # should already exist openstack image save --file /tmp/$USER/my-snapshot.qcow2 $USER-snapshot ``` * Clean it up (note: use actual `username`): ``` LIBGUESTFS_BACKEND=direct virt-sysprep \ --add /tmp/gtenagli/my-snapshot.qcow2 --delete /etc/krb5.keytab \ --delete /var/lib/cern-private-cloud-addons/state ``` * (Bonus) inspect: ``` LIBGUESTFS_BACKEND=direct guestfish -i --ro -a /tmp/gtenagli/my-snapshot.qcow2 ``` --- ## Exercise 2.3 ###### Image management: clone an instance (Linux only!) * Upload the image: ``` openstack image create \ --file /tmp/$USER/my-snapshot.qcow2 \ --property os=LINUX --disk-format=qcow2 \ --container-format=bare $USER-snapshot-clean ``` * Create an instance from the clean image: ``` openstack server create --key-name your_key_name \ --flavor m2.small --image $USER-snapshot-clean \ $USER-clone-1 ``` * [ASYNC] Check the web content is available on the newly created instance: ``` curl http://${USER}-clone-1.cern.ch/content ``` --- ## Exercise 3.1 ###### Block storage: attach an extra disk * Create a volume and attach it: ``` openstack volume create --description "CSC Test volume" --size 5 $USER-vol openstack server add volume $USER-csc-2024 $USER-vol ``` * Inspect the volume from the VM (?) * Format and mount: ``` # mkfs -t ext4 -L csc-test /dev/vdb # mount -L csc-test /mnt ``` --- ## Exercise 3.2 ###### Block storage: extend a volume * Fill the volume (hint: `df -k && fallocate -l`) * Extend the volume (new! note the magic API version call): ``` openstack --os-volume-api-version 3.42 volume set $USER-vol --size 6 # resize2fs /dev/vdb ``` --- ## Thanks! ---