First, we note that it is not always possible to upgrade from a major release to the next without upgrading through intermediate, Minor releases… This link provides an explanation of the nuances of each release. We’re currently on Release 16.1, which is detailed here.
Here’s a very useful GitLab Tool to determine correct migration path and notes about intermediate updates
The add on packages such as Gitaly and GEO (multiple sites hosting your GitLab) have significant impact on path to updates. Fortunately, we are running a simplified instance…
(this is the same location we found the script above – by choosing DEBIAN-type packages since we’re running GitLab on an Ubuntu Server (ver 22.04)
We have identified a Script (Click the Debian button at this GitLab packages page):
curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
Since we're carrying out this update remotely, we paste the following command into our (Putty) CLI page and hit Enter.
If this script runs successfully, we receive and acknowlegement and can move on to installing intermediate update packages - as follows;
tamer@git:/etc/apt/trusted.gpg.d$ curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
Detected operating system as Ubuntu/jammy.
Checking for curl...
Detected curl...
Checking for gpg...
Detected gpg...
Running apt-get update... done.
Installing apt-transport-https... done.
Installing /etc/apt/sources.list.d/gitlab_gitlab-ee.list...done.
Importing packagecloud gpg key... done.
Running apt-get update... done.
The repository is setup! We can now install the packages.
Beware: A bug was introduced in Revision 16.5 - for installations with GEO . It relates to space running out on the Secondary instance/server. We should be careful about this and ONLY install intermediate versions if essential. Ideally we want to end up with version 17.... The issue is debated in detail here.
As shown above, to reach version 17, we simply need to go through four intermediate updates. The beauty of this tool is that it gives you CLI commands.
Another word of caution:
I had stopped GitLab
sudo gitlab-ctl stopbefore updating and found this was not the correct way to go since the Database is backed up during the update and cna’t go ahead if GitLab hence database is not running…
So I re-started it using the command
sudo gitlab-ctl start1st of four updates – to GitLab 16.3.7
$ apt-get install -y gitlab-ee=16.3.7-ee.0Sit back and observe a rigorous update process – you can go grab a cuppa during this first of four updates…
Once complete, a restart of .. is required. Select Ok to confirm this step.
At the end of this you should see;
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart before anything else.
We go to the web interface to verify update…
That was successful. So we proceed to 2nd update.
2nd of four updates – to GitLab 16.7.8
$ apt-get install -y gitlab-ee=16.7.8-ee.0Following another rigorous update process – enjoy sipping on your hot drink…
(Note, I did wonder if I ought to have created an intermediate Snapshot, through XCP-ng to secure that update… Too late but perhaps upon next step…)
Again, we are asked to restart
This time round, we see a warning message. Reading through the updates notes this warning implies that REDIS has been updated and needs to be restarted..
Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.
sudo gitlab-ctl restart redis
[2024-07-06T16:22:52+00:00] WARN: This release of Cinc Client became end of life (EOL) on May 1st 2024. Please update to a supported release to receive new features, bug fixes, and security updates.
gitlab Reconfigured!
Restarting previously running GitLab services
ok: run: crond: (pid 651801) 0s
ok: run: gitaly: (pid 651793) 0s
ok: run: gitlab-workhorse: (pid 651752) 2s
ok: run: logrotate: (pid 651814) 0s
ok: run: nginx: (pid 651769) 1s
ok: run: postgresql: (pid 651592) 158s
ok: run: puma: (pid 651822) 1s
ok: run: redis: (pid 628256) 966s
ok: run: registry: (pid 651782) 2s
ok: run: sidekiq: (pid 651829) 0s
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).
Firing up our web site and checking the HELP page, we see that the update has been successful. We also note some updates to do with Left Menu and Forced Expiration of tokens…
A quick lookup shows that We can restart REDIS as follows;
Accordingly we try the following commands from our Putty Window to address warning about REDIS but, none seem to work. Says Unit redis-server.service not found....
In fact it seems unnecessary as our GitLab version 16.7.8 instance is up and running already (see below).Step 1. Enable Redis service on reboot:
$ sudo systemctl enable redis-server
Step 2. Start Redis service:
4 sudo systemctl start redis-server
Step 3. Stop Redis service (the recommended way):
redis-cli shutdown
$ sudo systemctl stop redis-server works too.
Step 4. Restart Redis service.
systemctl restart redis-serverThis time we take a snapshot of our VM to safeguard against potential issues during next two steps…
3rd of four updates – to GitLab 16.11.5
$ apt-get install -y gitlab-ee=16.11.5-ee.0Following a penultimate rigorous update process – It’s good news. According to our web HELP page, we’re over this hurdle, too and now running GitLab v16.11.5-ee.
4th and final update – to 17.1.1
$ apt-get install -y gitlab-ee=17.1.1-ee.0We note that there were errors during this last leg.
Running handlers:
[2024-07-06T17:23:42+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
Reading unsupported config value grafana.
Running handlers complete
[2024-07-06T17:23:42+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 04 seconds
[2024-07-06T17:23:42+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-07-06T17:23:42+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-06T17:23:42+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-07-06T17:23:42+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-06T17:23:42+00:00] FATAL: Mixlib::Config::UnknownConfigOptionError: Reading unsupported config value grafana.This was exciting as the web site was not accessible.
We tried again, to execute the command
gitlab-ctl reconfigurewith same results. The configuration file needs to be reviewed and we need to also investigate what’s changed with Grafana between this and prior version… Clearly a configuration value, issue.
We followed this up with However, by executing
gitlab-ctl startThis got us back up and running and the site seems to be working.
That’s almost perfect with the bulk of the update session achieving objectives. This leaves a minor follow up needed and the only other thing I will say is that perhaps this needs to be done more frequently as the entire four step update has taken over an hour to complete…
Time to take a Snapshot of the VM in XCP-ng, where we’re hosting this…
Will need to repeat the Snapshot as there were continued issues with Grafana error. I could see some pages including initial log-in page but couldn’t actually log-in….
I went on to take a look at the gitlab.rb file where I noticed grafana [‘enable’] = false. which seemed correct. However, at this GitLab web site I noticed that Grafana is no longer supported in GitLab, so I simply commented out the line / entry in gitlab.rb (the line with green box in the following editor window).
This then allowed me to execute the command
sudo gitlab-ctl reconfigure I believe this has fixed it!! Yes. I have verified that it has…
Time to backup the VM, again for the last time.