Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latest exercise revisions. #2152

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@

In the previous exercise, we used a job template/playbook to analyze our CentOS7 app servers. Behind the scenes, the `analysis` module from the `infra.convert2rhel` Ansible collection launched a Convert2RHEL pre-conversion analysis on each of our CentOS 7 app servers and once complete, parsed the pre-conversion analysis report for any issues that would either block the conversion from initiating or potentially failing. If any problems were found, then the Ansible Automation Platform automation job would have reported a failure.

- If we we're using the Convert2RHEL framework to manually convert just a single CentOS host, we could simply get to a shell prompt on the host and look at the local report file output. In [Exercise 1.1, Step 2](../1.1-setup/README.md#step-2---open-a-terminal-session), we learned how to open an ssh session to one of our app servers. Follow those steps and after logging in, use this command to review the local Convert2RHEL pre-conversion report file:
- If we are using the Convert2RHEL utility to manually convert just a single CentOS host, we could simply get to a shell prompt on the host and look at the local report file output. In [Exercise 1.1, Step 2](../1.1-setup/README.md#step-2---open-a-terminal-session), we learned how to open an ssh session to one of our app servers. Follow those steps and after logging in, use this command to review the local Convert2RHEL pre-conversion report file:

```
less /var/log/convert2rhel/convert2rhel-pre-conversion.txt
Expand All @@ -36,7 +36,7 @@ In the previous exercise, we used a job template/playbook to analyze our CentOS7
>
> Use the up and down arrow keys to scroll through the file and type `q` when you are ready to quit the `less` command.

- CentOS 7 includes an optional administration web console based on [Cockpit](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/managing_systems_using_the_rhel_8_web_console/index#what-is-the-RHEL-web-console_getting-started-with-the-rhel-8-web-console) that we call the CentOS Web Console. We will explore how to review the Convert2RHEL pre-conversion reports using the CentOS Web Console in the next step of this exercise.
- CentOS 7 includes an optional administration web console based on [Cockpit](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/managing_systems_using_the_rhel_8_web_console/index#what-is-the-RHEL-web-console_getting-started-with-the-rhel-8-web-console) called the CentOS Web Console. We will explore how to review the Convert2RHEL pre-conversion reports using the CentOS Web Console in the next step of this exercise.

- In addition to writing the plain text `convert2rhel-pre-conversion.txt` file, a JSON format `convert2rhel-pre-conversion.json` file is also generated. This file includes the same report results as the plain text file, but in JSON format which is perfect for being ingested by log management tools like Elastic/Kibana or Splunk. Many large enterprises will push their pre-conversion report data to one of these tools to develop their own custom dashboards that can filter reports by environment (e.g., Dev/Test/Prod), location, app ID, owning team, etc. <!-- FIXME: add Splunk example here when https://issues.redhat.com/browse/RIPU-35 gets done. -->

Expand Down Expand Up @@ -66,13 +66,13 @@ For this workshop, we will be using the CentOS Web Console to access the Convert

### Step 3 - Review Convert2RHEL Pre-conversion Report of CentOS 7 Host

Now we are ready to use the Web Console to review the Convert2RHEL pre-conversion reports. Let's start by looking at one of the CentOS 7 hosts.

We are now here in the automation approach workflow:

![Automation approach workflow diagram with review report step highlighted](images/conversion-workflow-hl-review.svg)

- Navigate to the RHEL Web Console remote host menu and click on the hostname of one of your CentOS 7 app servers. Remember as we learned in the previous step, you can confirm the CentOS version on the system overview page.
Now, let's use the Web Console to review the Convert2RHEL pre-conversion reports. We will start by looking at one of the CentOS 7 hosts.

- Navigate to the RHEL Web Console remote host menu and click on the hostname of one of your CentOS 7 app servers (should be node4, node5, or node6). Remember as we learned in the previous step, you can confirm the CentOS version on the system overview page.

- Having verified you are looking at one of the CentOS 7 app servers, use the main menu to navigate to Tools > Terminal. This will display a shell terminal on the host system. In order to review the Convert2RHEL pre-conversion report that was generated for host, copy the following command line and paste into the terminal and press `Enter` or `return`:

Expand Down Expand Up @@ -110,15 +110,15 @@ Of course, the answer is our automated snapshot/rollback capability.

- If any of the findings listed in the pre-conversion analysis report ultimately leads to the conversion failing or results in application compatibility impact, we can quickly get back to where we started by rolling back the snapshot. Before rolling back, we can debug the root cause and use the experience to understand the best way to develop remediation and eliminate the risk of that failure or impact happening in the future.

- There is a concept explained quite well in the famous article [Fail Fast](http://www.martinfowler.com/ieeeSoftware/failFast.pdf) published in *IEEE Software*. The article dates back to 2004, so this is hardly a new concept. Unfortunately, there is a stigma associated with failure that can lead to excessively risk-averse behavior. The most important benefit of having automated snapshots is being able to quickly revert failures. That allows us to safely adopt a fail fast and fail smart mantra.
- There is a concept explained quite well in the famous article [Fail Fast](http://www.martinfowler.com/ieeeSoftware/failFast.pdf) published in *IEEE Software*. The article dates back to 2004, so this is hardly a new concept. Unfortunately, there is a stigma associated with failure that can lead to excessively risk-averse behavior. The most important benefit of having automated snapshots is being able to quickly revert failures. That allows us to safely adopt a "fail fast / fail smart" mantra.

- Of course, there are many best practices we can follow to reduce risk. Obviously, test for application impacts by trying conversions in your lower environments first. Any issues that can be worked out with Dev and Test servers will help you be prepared to avoid those issues in Production.
- Of course, there are many best practices we can follow to reduce risk. Obviously, start with testing for application impacts by trying conversions in your lower environments first. Any issues that can be worked out with Dev and Test servers will help you be prepared to avoid those issues in Production.

- When present, the findings reported by the Convert2RHEL pre-conversion analysis report are there to make us aware of potential failure modes, but experience has shown that sometimes they are not a problem. Do not become petrified when you see those warnings on the report. Assess findings, develop automation to remediate any issues, and then run through the conversion process, reverting back to step one to loop through the process as your arsenal of remediation automation builds over time. Convert early _and_ often!

## Conclusion

In this exercise, we learned about the different options for managing Convert2RHEL pre-conversion analysis reports. We used the RHEL/CentOS Web Console to look at the reports we generated in the previous exercise. In the challenge lab, we reviewed the importance of snapshots and learned to embrace failure.
In this exercise, we learned about the different options for managing Convert2RHEL pre-conversion analysis reports. We used the CentOS Web Console to look at the reports we generated in the previous exercise. In the challenge lab, we reviewed the importance of snapshots and learned to embrace failure.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ We are ready to start the conversion phase of our CentOS to RHEL automation jour

![Automation approach workflow diagram with conversion step highlighted](images/conversion-workflow-hl-conversion.svg)

It is during this phase that the conversion playbooks are executed using a workflow job template. The first playbook creates a snapshot that can be used for rolling back if anything goes wrong with the conversion. After the snapshot is created, the second playbook uses the Convert2RHEL utility to perform the conversion where the host is advanced to the new RHEL major version.
It is during this phase that the conversion playbooks are executed using a workflow job template. The first playbook clears yum cache in order to ensure that the maximum amount of space is available for the conversions process. The second playbook creates a snapshot that can be used for rolling back if anything goes wrong with the conversion. After the snapshot is created, the third playbook uses the Convert2RHEL utility to perform the conversion where the host is advanced to the new RHEL major version.

### Step 1 - Launch the Conversion Workflow Job Template

We are about to start the CentOS conversion of our application servers. The hosts will not be available for login or application access during the conversion. When the conversion is finished, the hosts will reboot under the newly converted RHEL major version.
We are about to start the CentOS conversion of our application servers. When the conversion is finished, the hosts will reboot under the newly converted RHEL major version.

Conversions typically take less than an hour, although they can run for longer if there are applications that shutdown slowly or with bare metal hosts that have a long reboot cycle. The cloud instances provisioned for our workshop lab environment will convert fairly quickly as they are very lightweight compared to traditional enterprise app servers.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,17 @@ Automating snapshots can be one of the most difficult features of the CentOS con

Let's start by defining exactly what we mean when we talk about snapshots.

### Step 1 - What are Snapshots and What are They Not
### Step 1 - What are Snapshots and What They are Not

Most organizations with a mature traditional computing environment will have standards and tools implemented for doing backups. Typically, backups will be performed on a periodic schedule. More critical or more dynamic data might be backed up more frequently than mostly static data. There is often a strategy where full backups are performed only occasionally and incremental backups are used to save changed file more often.
Most organizations with a mature, traditional computing environment will have standards and tools implemented for doing backups. Typically, backups will be performed on a periodic schedule. More critical or more dynamic data might be backed up more frequently than mostly static data. There is often a strategy where full backups are performed only occasionally and incremental backups are used to save changed files and data more often.

The reason for doing backups is to be able to recover data that has been lost for any reason. If data is corrupted because of an operations issue or software defect or accidentally deleted, backups make it easy to turn the clock back and restore the lost data.
The reason for doing backups is to be able to recover data that has been lost for any reason. If data is corrupted because of an operations issue, software defect or accidentally deleted, backups make it easy to turn the clock back and restore the lost data.

But when an entire server is lost, using backups to recover is more difficult because a new operating system must first be installed before anything can be restored from the backup. The data can be spread out across a full backup as well as multiple incremental backups, further increasing the time for a full server recovery. Most organizations only use their backup solution to restore individual files or directories, but they are not as prepared to recover everything on a server. Even if they are, such a recovery will take a long time.

Snapshots are different in that they do not backup and restore individual files. Instead, backups operate at a storage device level, instantly saving the contents of an entire logical volume or virtual disk. Unlike backups, snapshots do not make a copy of the data being backed up, but rather mark a point in time after which a copy of all modified data is copied going forward. For this reason, the underlying technique used for snapshots is often referred to as "copy-on-write" or COW.

While COW snapshots are not a substitute for traditional full and incremental backups, they do offer a number of advantages. The most important advantage is that creating and rolling back snapshots happens almost instantaneously as compared to hours or longer for traditional backup and restore. It is this ability to quickly take an entire server back in time that makes snapshots ideal for reducing the risk of performing CentOS conversions.
While COW snapshots are not a substitute for traditional full and incremental backups, they do offer a number of advantages. The most important advantage is that creating and rolling back snapshots happens almost instantaneously as compared to hours or longer for traditional backup and restore. It is this ability to quickly take an entire server "back in time" that makes snapshots ideal for reducing the risk of performing CentOS conversions.

### Step 2 - Assessing Different Snapshot Solutions

Expand Down Expand Up @@ -82,9 +82,9 @@ If there is not enough free space in the volume group, there are a few ways we c

- Adding another physical volume to the volume group (i.e., `pvcreate` and `vgextend`). For a VM, you would first configure an additional virtual disk.
- Temporarily remove a logical volume you don't need. For example, on bare metal servers, there is often a large /var/crash empty filesystem. Removing this filesystem from `/etc/fstab` and then using `lvremove` to remove the logical volume from which it was mounted will free up space in the volume group.
- Reducing the size of one or more logical volumes. This is tricky because first the filesystem in the logical volume needs to be shrunk. XFS filesystems do not support shrinking. EXT filesystems do support shrinking, but not while the filesystem is mounted. Until recently, this way of freeing up volume group space was considered a last resort to be attempted by only the most skilled Linux admin, but it now possible to safely automate shrinking logical volumes using the [`shrink_lv`](https://github.com/swapdisk/infra.lvm_snapshots/tree/main/roles/shrink_lv#readme) role of the aforementioned `infra.lvm_snapshots` collection.
- Reducing the size of one or more logical volumes. This is tricky because first, the filesystem in the logical volume needs to be shrunk. XFS filesystems do not support shrinking. EXT filesystems do support shrinking, but not while the filesystem is mounted. Until recently, this way of freeing up volume group space was considered a last resort to be attempted by only the most skilled Linux admin, but it is now possible to safely automate shrinking logical volumes using the [`shrink_lv`](https://github.com/swapdisk/infra.lvm_snapshots/tree/main/roles/shrink_lv#readme) role of the aforementioned `infra.lvm_snapshots` collection.

After a snapshot is created, COW data will start to utilize the free space of the snapshot logical volume as blocks are written to the origin logical volume. Unless the snapshot is create with the same size as the origin, there is a chance that the snapshot could fill up and become invalid. Testing should be performed during the development of the LVM snapshot automation to determine snapshot sizings with enough cushion to prevent this. The `snapshot_autoextend_percent` and `snapshot_autoextend_threshold` settings in lvm.conf can also be used to reduce the risk of snapshots running out of space. The [`lvm_snapshots`](https://github.com/swapdisk/infra.lvm_snapshots/tree/main/roles/lvm_snapshots#readme) role of the `infra.lvm_snapshots` collection supports variables that may be used to automatically configure the autoextend settings.
After a snapshot is created, COW data will start to utilize the free space of the snapshot logical volume as blocks are written to the origin logical volume. Unless the snapshot is created with the same size as the origin, there is a chance that the snapshot could fill up and become invalid. Testing should be performed during the development of the LVM snapshot automation to determine snapshot sizings with enough cushion to prevent this. The `snapshot_autoextend_percent` and `snapshot_autoextend_threshold` settings in lvm.conf can also be used to reduce the risk of snapshots running out of space. The [`lvm_snapshots`](https://github.com/swapdisk/infra.lvm_snapshots/tree/main/roles/lvm_snapshots#readme) role of the `infra.lvm_snapshots` collection supports variables that may be used to automatically configure the autoextend settings.

Unless you have the luxury of creating snapshots with the same size as their origin volumes, LVM snapshot sizing needs to be thoroughly tested and free space usage carefully monitored. However, if that challenge can be met, LVM snapshots offer a reliable snapshot solution without the headache of depending on external infrastructure such as VMware.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The first thing we want to do is see if the job running the conversion playbook

### Step 2 - Verify the Hosts are Upgraded to Next RHEL Version

Now let's make sure our application servers are actually converted to the next RHEL version.
Now let's make sure our application servers have actually converted to RHEL.

- In [Exercise 1.3: Step 2](../1.3-report/README.md#step-2---navigating-the-rhel-web-console), you used the RHEL Web Console to check the installed CentOS versions on your application servers. Let's repeat those steps to see the RHEL versions reported after our conversions.

Expand All @@ -69,7 +69,7 @@ Now let's make sure our application servers are actually converted to the next R

- Additionally, if you recorded the `Asset Tag` and `Machine ID`, note that they are the same, serving as verification that this is indeed the same system that was previously operating with CentOS 7.

- You can also check the RHEL and kernel versions from the command line following the steps you used with [Exercise 1.1: Step 2](../1.1-setup/README.md#step-2---open-a-terminal-session).
- You can also check the RHEL and kernel versions from the command line following the steps you used with [Exercise 1.4: Step 2](../1.4-report/README.md###step-3---review-convert2rhel-pre-conversion-report-of-centos-7-host).

At the shell prompt of your application servers, use the `cat /etc/redhat-release` and `uname -r` commands. Here's an example showing that application server `node4` was converted to RHEL 7:

Expand All @@ -83,7 +83,7 @@ Now let's make sure our application servers are actually converted to the next R

In this exercise, we observed that the conversion playbook runs completed successfully. We then used the RHEL Web Console,the command line, as well as Satellite to verify the new RHEL versions were installed.

Continue to the following exercise to verify the three tier application is still functioning as expected after the CentOS conversions:
Continue to the following exercise to verify the three tier application is still functioning as expected after the CentOS to RHEL conversions:

- [Exercise 2.4 - How is the Three Tier App Doing?](../2.4-check-three-tier-app/README.md)

Expand Down
Loading