Fixes #615 - Add documentation for upgrade procedure

[aidenfine]

Why are you introducing these changes? (Problem description, related links)

#615

What are the changes introduced in this pull request?

• Introduce user documentation to cover the manual update process for connected RPM, disconnected RPM, and from-source installs.

Note: this pull request has a few TODOs which are open questions that need to be answered by the foremanctl team. Please feel free to push to this branch to address these issues (no AI fluff please). This PR was originally opened #616. The author (@qcjames53) is on paternity leave. This PR will replace the existing PR.

How to test this pull request

• Attempt the upgrade procedure from source and the portions of the RPM procedures which can be currently attempted.
• Validate the document for accuracy. Ensure it is concise, useful, and well organized.

Checklist

• Tests added/updated (if applicable)
• Documentation updated (if applicable)

[aidenfine]

Based on previous comments on #616. I added the health check mention in docs/user/upgrade.md and moved Upgrading from source section into docs/developer/development-environment.md

[ehelms]

Is the RPM enough? Or will they need the Foreman repository for dependencies?

[aidenfine]

If I recall you should also need the repo for the deps, let me test right now.

[aidenfine]

@ehelms Take a look I added an extra bullet point what do you think?

[ianballou]

I would recommend telling users to stage the entire repository locally. python3.12-obsah is a dependency of foremanctl for example, and users may want to update it even if the update isn't a hard requirement. One reason could be for a CVE, for example.

[ehelms]

Minor wording mismatch -- the PR title and commits mention update which is different than upgrade in our nomenclature. I think we should try to be clear about which one we are talking about. Hopefully the procedure for the two becomes the same, but for now I like the differentiation:

1. upgrade: X.Y -> X.Y+1
2. update: X.Y.Z -> X.Y.Z+1

upgrade requires things like switching repositories, different version tags on the images and possibly "bigger" upgrade actions happening. Where as "update" should feel smaller, and more direct to achieve for the user.

[ianballou]

Just one note so far, thanks for putting all this together:

[aidenfine]

Minor wording mismatch -- the PR title and commits mention update which is different than upgrade in our nomenclature. I think we should try to be clear about which one we are talking about. Hopefully the procedure for the two becomes the same, but for now I like the differentiation:

1. upgrade: X.Y -> X.Y+1
2. update: X.Y.Z -> X.Y.Z+1

upgrade requires things like switching repositories, different version tags on the images and possibly "bigger" upgrade actions happening. Where as "update" should feel smaller, and more direct to achieve for the user.

Updated commits and PR title.

[ehelms]

This procedure doesn't explain to the user where to get the new version of foremanctl. I would expect it to include managing the Foreman repository. I think the title here is also confusing as it talks about "upgrading foremanctl". But upgrading foremanctl inherently means also updating or upgrading Foreman. Right now, we do not support one without the other. Therefore, I think the focus needs to be on "How do I upgrade my installation from X.Y to X.Y+1"

[ianballou]

@aidenfine it might help to run through the upgrading and updating guides in the existing documentation to understand how we handle defining repositories. I think the general strategy for how we manage the repo will stay the same.

[ianballou]

The troubleshooting section should probably have a section about varying Ansible errors, after which a user should probably open up an issue on GitHub or a post at community.theforeman.org.

[aidenfine]

Is there any link to common Ansible errors that may occur?

[ianballou]

We don't support anything on development environments, so people can git pull at will.

[ianballou]

While it's a good practice, I'm not sure how many people will be taking backups of their development environment. Actually, it makes me wonder if they're somehow incompatible with backup/restore.

[ianballou]

I would recommend telling users to stage the entire repository locally. python3.12-obsah is a dependency of foremanctl for example, and users may want to update it even if the update isn't a hard requirement. One reason could be for a CVE, for example.

[aidenfine]

@ianballou Are you talking about the Foreman repository?

[ianballou]

@ianballou Are you talking about the Foreman repository?

Yep! Like https://yum.theforeman.org/releases/3.19/el9/x86_64/. Where users will find the foremanctl RPM. The repository contains the dependencies that foremanctl needs.

Check this out:

# rpm -q --requires foremanctl 
/usr/bin/bash
/usr/bin/env
/usr/bin/python3
config(foremanctl) = 2.3.0-1.el9sat
python3.12-obsah >= 1.8.1
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsZstd) <= 5.4.18-1

The basic deps likely come from the main OS repos. But then there's python3.12-obsah, which we package up. But anyway - when a user is installing Foreman we want them to have the whole repo. For disconnected users in Satellite, the full setup typically looks like having a connected Satellite host the repositories and the disconnected Satellite consuming from it in some fashion - possibly via import/export. The foremanctl docs don't need to fully define this whole setup though, it could be as basic as saying the user needs to bundle up the repository in some manner and present it to the host.

It might be acceptable if we identify all the dependencies and tell them to copy them over, but it's technically safer to use the entire repo where foremanctl lives.

I think the tricky thing here is that "disconnected" installation is truly a bit complicated and I don't think we've really documented it before for regular Foreman or Katello, just Satellite.

So - with all this said, my recommendation would be to suggest that the disconnected user creates a local repository mirror of the foreman distribution of their choice, but leave the complexities of how to their choosing. See https://access.redhat.com/solutions/7019225 for an example of how Red Hat suggests users make local mirrors of RHEL repos.

[aidenfine]

@ianballou I added the suggestion let me know what you think.

[qcjames53]

Thank you guys for all the movement on this :)

Something I think is worth bringing up is that downgrading rpm version troubleshooting section was intended for people with a full backup. If you upgrade from x.1 to x.2 and something breaks, there's a path to restore your working x.1 backup. It could use better wording.

Another thought: is there a way to force upgrade jobs to run sequentially if a user goes from foremanctl x.1 to x.10 in one dnf command? Eric mentioned we require version bumping foreman one release at a time so if we can't automate that in packaging there should be clear instructions to the user here.

[ianballou]

Hi @qcjames53 !

Thank you guys for all the movement on this :)

Something I think is worth bringing up is that downgrading rpm version troubleshooting section was intended for people with a full backup. If you upgrade from x.1 to x.2 and something breaks, there's a path to restore your working x.1 backup. It could use better wording.

I think we need to test that the downgrade is possible, with containers it might be easier to support. One question that comes to mind, for example, is what happens if the new deployment pull some newer container images. If you downgrade and run foremanctl restore, will the older container images be downgraded too?

Another thought: is there a way to force upgrade jobs to run sequentially if a user goes from foremanctl x.1 to x.10 in one dnf command? Eric mentioned we require version bumping foreman one release at a time so if we can't automate that in packaging there should be clear instructions to the user here.

+1 to this, we need a note saying that users must upgrade to the next Z or Y version and it's the only upgrade path we support.