Upgrading Splunk Universal Forwarders from the Deployment Server – Source: securityboulevard.com

One of the most requested features I hear from clients as a Splunk Managed Services Provider (MSSP) is to have a mechanism for managing the version of the Splunk Universal Forwarder across the environment from the Deployment Server.  We could easily manage Splunk configurations for our clients via the Deployment Server, but any UF version upgrades would have to be handled by the client using their typical process for managing software in their environment. 

You might be thinking, “Hey Tom, can’t they just upgrade this using Ansible/Puppet/SCCM/whatever?”  That’s a great point, and was our typical recommendation.  However, not every environment has the same level of technical maturity to have all of their package management automated across their environment. Informing a client that they have to manually upgrade a Splunk agent on dozens or more different systems wasn’t the most fun news to share. 

Fortunately, Splunk has recently introduced a new app to help solve part of this problem, Splunk Remote Upgrader for Linux Universal Forwarders.  As the name suggests, this app is designed to handle UF upgrades on Linux systems.  I’ve always found that the best way to learn something new is to test it out in a lab environment and see what happens, so let’s take this new app for a test drive!

Test environment and Assumptions

For my test, I’ve set up a very simple Splunk environment consisting of the following:

• A Splunk Enterprise 9.4.0 system functioning as an all-in-one system with a deployment server
• A Splunk Universal Forwarder running a supported version of Splunk.  For my example, I picked 9.0.7, but the oldest version supported is 8.0.

The Remote UF Upgrader is a service that runs external to the Splunk UF itself, and will need to be installed separately.  This means that you’ll have to at least deploy this package to all of your Linux systems with a UF installed, but once this is deployed, you’ll be able to do future upgrades via the Deployment Server.

Since this app is a bit unique in that it has components that need to be installed on both the Deployment Server and the Universal Forwarders, lets look at each part separately.

Deploying the App on the Deployment Server

From the perspective of the deployment server, the app is fairly straightforward and deploys like a normal app, with some additional steps needed to include the UF packages you wish to deploy.  When you extract the app to $SPLUNK_HOME/etc/deployment-apps, there will be a directory splunk_app_uf_remote_upgrade_linux that is created.  

Inside the app, you should create a local/packages directory to store the UF installers that are used for the upgrade.  You’ll also need to include the signature file for the UF installer – you can download this from Splunk at the same time that you download the UF installer. 

Click the “More” link to download the x509 signature for your UF installer

If I want to provision the tarball installer of the Splunk 9.4.0 UF to be installed on remote UFs, my local/packages directory will end up looking like this:

splunk@tk-demo-uf-upgrade-ds:/opt/splunk/etc/deployment-apps/splunk_app_uf_remote_upgrade_linux/local/packages$ ls -ltrah

total 130M

-rw-rw-r– 1 splunk splunk 130M Dec 13 13:40 splunkforwarder-9.4.0-6b4ebe426ca6-linux-amd64.tgz

-rw-r–r– 1 splunk splunk 8.7K Feb  3 12:34 splunkforwarder-9.4.0-6b4ebe426ca6-linux-amd64.tgz.sig

Note that I have both the .tgz UF installer and the corresponding signature file.

You can also copy future versions of the splunk-upgrader app (which is named splunk-upgrader-100.tgz in the initial release of the app) into local/packages to handle future upgrades of the UF upgrader on your remote endpoints.  Since this is the first version of the app, I’ve not yet been able to test upgrades of the upgrader.  If you do make any configuration changes to the upgrader package, you can also include those in the local/packages directory. 

Finally, you’ll want to set up a serverclass for distributing the UF upgrader to your Linux endpoints.  This is where the machineTypesFilter in serverclass.conf can come in handy, as you want to limit this upgrader to Linux systems only.  You’ll also probably want to make this a dedicated serverclass, so that you can better control what systems get upgraded and when. In my lab environment, however, I’m just going to push this to all the Linux systems like this:

[serverClass:all_linux]

machineTypesFilter = linux*

whitelist.0 = *

[serverClass:all_linux:app:splunk_app_uf_remote_upgrade_linux]

restartSplunkWeb = 0

restartSplunkd = 1

I’ve put together a quick video to show you how the app looks when configured on the deployment server: 

Once you reload the deployment server, all of your UFs in this serverclass will end up with the splunk_app_uf_remote_upgrade_linux app deployed to etc/apps and we’ll be ready for the next step!

Installing the Remote UF Upgrader

While you will need to install the UF upgrader on each system, you can still use the Deployment Server to distribute the installer to all of your Linux UFs. With the default app packaging in version 1.0.0 of the app, you’ll find the installer at splunk_app_uf_remote_upgrade_linux/default/packages/splunk-upgrader-100.tgz.

As a user with administrative privileges, we can untar this to a directory such as /opt:

root@tk-demo-uf-upgrade-uf2:/opt# tar -zxvf /opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/default/packages/splunk-upgrader-100.tgz

splunkupgrader/

splunkupgrader/history/

splunkupgrader/history/README

splunkupgrader/bin/

splunkupgrader/bin/permission.sh

splunkupgrader/bin/logging.sh

splunkupgrader/bin/monitor.sh

splunkupgrader/bin/pkg_validation_util.sh

splunkupgrader/bin/install.sh

splunkupgrader/bin/upgrade_rpm.sh

splunkupgrader/bin/upgrade.sh

splunkupgrader/bin/preflight_constants.sh

splunkupgrader/bin/upgrade_deb.sh

splunkupgrader/bin/splunk_util.sh

splunkupgrader/bin/constants.sh

splunkupgrader/bin/util.sh

splunkupgrader/bin/uninstall.sh

splunkupgrader/bin/self_upgrade.sh

splunkupgrader/bin/upgrade_tgz.sh

splunkupgrader/bin/validate_version.sh

splunkupgrader/bin/init_config.sh

splunkupgrader/bin/splunk_updater_launcher.sh

splunkupgrader/bin/history.sh

splunkupgrader/bin/process.sh

splunkupgrader/bin/install_sudo_prereqs.sh

splunkupgrader/bin/report.sh

splunkupgrader/config/

splunkupgrader/config/default_config

splunkupgrader/resource/

splunkupgrader/resource/Splunk_Root_CA.cer

splunkupgrader/resource/SplunkPGPKey.pub

splunkupgrader/resource/splunk-upgrader.service

splunkupgrader/resource/eula.txt

splunkupgrader/log/

splunkupgrader/log/.keep

splunkupgrader/VERSION

Next, run the installer for the UF upgrader!  I’m going to use the flag to have it create the default user for this example.  You’ll see the installer step through the installation process, and then start the splunk-upgrader.service via systemd:

root@tk-demo-uf-upgrade-uf2:/opt# /opt/splunkupgrader/bin/install.sh –accept-license –create-user

2025-02-04-12:35:04 ### Running pre-install checks ###

2025-02-04-12:35:04 Checking if Splunk updater is already running

Unit splunk-upgrader.service could not be found.

2025-02-04-12:35:05 Checking if the current user can run all required commands with sudo

2025-02-04-12:35:05 Verified that the current user can run all base required commands with sudo

2025-02-04-12:35:05 ### Checking user ###

2025-02-04-12:35:05 Checking if user “splunkupgrader” exists

2025-02-04-12:35:05 Creating new user with configured name (splunkupgrader)

2025-02-04-12:35:05 Creating group splunkupgrader

id: ‘splunkupgrader’: no such user

2025-02-04-12:35:05 Creating user splunkupgrader

2025-02-04-12:35:05 ### Discovering SPLUNK_HOME ###

2025-02-04-12:35:12 Adding new SPLUNK_HOME=/opt/splunkforwarder

SPLUNK_HOME=/opt/splunkforwarder

2025-02-04-12:35:12 Successfully set SPLUNK_HOME to the given location: /opt/splunkforwarder

2025-02-04-12:35:12 ### Ensure location not under SPLUNK_HOME ###

2025-02-04-12:35:12 Check if current location is under SPLUNK_HOME.

2025-02-04-12:35:12 ### Verifying that the required package-type dependent binaries are present ###

/opt/splunkupgrader/bin/splunk_util.sh: line 354: rpm: command not found

dpkg-query: package ‘splunkforwarder’ is not installed

Use dpkg –contents (= dpkg-deb –contents) to list archive files contents.

2025-02-04-12:35:13 Checking for any required binaries for package type “tgz”

2025-02-04-12:35:13 ### Adding permissions for the new SPLK_USER ###

2025-02-04-12:35:13 ### Validating user permissions ###

2025-02-04-12:35:13 Checking in advance if the current user has permissions to verify the target user “splunkupgrader”‘s sudo permissions

2025-02-04-12:35:13 Verified that the current user can run all required verification commands for target user “splunkupgrader” with sudo

2025-02-04-12:35:13 Checking if the given user “splunkupgrader” can run all required commands with passwordless sudo

2025-02-04-12:35:14 Verified that the given user “splunkupgrader” can run all required commands with passwordless sudo

2025-02-04-12:35:14 ### Beginning install ###

2025-02-04-12:35:14 Updating splunk-upgrader.service unit file to run as the configured user (splunkupgrader) + group (splunkupgrader)

2025-02-04-12:35:15 Setting file and directory permissions in SPLUNK_UPDATER_HOME (/opt/splunkupgrader/bin/../) to the configured user (splunkupgrader) + group (splunkupgrader)

2025-02-04-12:35:15 Enabling and starting Splunk updater service “splunk-upgrader”

Created symlink /etc/systemd/system/multi-user.target.wants/splunk-upgrader.service → /etc/systemd/system/splunk-upgrader.service.

• splunk-upgrader.service – Splunk Upgrader daemon that monitors for new UF packages to upgrade to

     Loaded: loaded (/etc/systemd/system/splunk-upgrader.service; enabled; vendor preset: enabled)

     Active: active (running) since Tue 2025-02-04 12:35:15 EST; 37ms ago

   Main PID: 2695 (splunk_updater_)

      Tasks: 2 (limit: 2308)

     Memory: 664.0K

        CPU: 27ms

     CGroup: /system.slice/splunk-upgrader.service

             └─2695 /bin/bash /opt/splunkupgrader/bin/splunk_updater_launcher.sh

Feb 04 12:35:15 tk-demo-uf-upgrade-uf2 systemd[1]: Started Splunk Upgrader daemon that monitors for n…e to.

Hint: Some lines were ellipsized, use -l to show in full.

2025-02-04-12:35:15 ### Finished installing ###

2025-02-04-12:35:15 Appending the current Splunk Updater installation log from /opt/splunkupgrader/bin/..//log/install.log to /opt/splunkforwarder/var/log/splunk/splunk-upgrade-install.log

You’ll also see that a new directory named SPLUNK_UPDATER_MONITORED_DIR is created in /tmp by the installer:

root@tk-demo-uf-upgrade-uf2:~# ls /tmp | grep -i splunk

SPLUNK_UPDATER_MONITORED_DIR

 At this point, if you already have the UF package in place in the app being pushed from the deployment server, you’ll see your UF get upgraded automatically. Logs for the upgrade are stored in $SPLUNK_HOME/var/log/splunk, and include splunk-upgrade-install.log (logs for installing the splunkupgrader app),  upgrader_package_delivery.log (logs for the delivery of the UF installation package), and splunk-upgrade.log (for the Splunk upgrade itself).

A successful upgrade will look like this in the upgrader_package_delivery.log:

2025-02-04-12:36:16 Checking if any forwarder packages are available

2025-02-04-12:36:16 Found files in /opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/bin/../local/packages. Will deliver them.

2025-02-04-12:36:16 Copying files from /opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/bin/../local/packages to /tmp/SPLUNK_UPDATER_MONITORED_DIR

2025-02-04-12:36:16 Running cmd: cp -r “/opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/bin/../local/packages/.” “/tmp/SPLUNK_UPDATER_MONITORED_DIR”

2025-02-04-12:36:17 Creating a trigger file to start upgrade: /tmp/SPLUNK_UPDATER_MONITORED_DIR/start_uf_upgrade

2025-02-04-12:36:17 Running cmd: touch “/tmp/SPLUNK_UPDATER_MONITORED_DIR/start_uf_upgrade”

2025-02-04-12:36:17 Completed the package delivery. Creating a file to make sure it only happens once.

2025-02-04-12:36:17 Running cmd: touch “/opt/splunkforwarder/etc/apps/splunk_app_uf_remote_upgrade_linux/bin/pkg_delivered”

2025-02-04-12:36:17 Completed!

When the upgrade is complete, you’ll see that the UF now reflects the upgraded version:

root@tk-demo-uf-upgrade-uf2:~# /opt/splunkforwarder/bin/splunk version

Warning: Attempting to revert the SPLUNK_HOME ownership

Warning: Executing “chown -R root:root /opt/splunkforwarder”

Couldn’t change ownership for /opt/splunkforwarder/lib: No such file or directory

Splunk Universal Forwarder 9.4.0 (build 6b4ebe426ca6)

You’ll also see the new version of the UF reflected in the Forwarder Management UI on your Deployment Server (if you’re running an updated version of Splunk Enterprise on your DS, as it looks better now in 9.4!)

Want to see the whole process at once?  This video will show you how: 

Wrap up

And that’s it – you’ve now successfully set up your environment so that you can deploy UF version upgrades on your Linux hosts from your Deployment Server.  Splunk says that there’s a Windows version of this that they’re working on as well, so I’ll be sure to test that out when it becomes available too. 

—–

Troubleshooting information:

While getting this set up, I ran into a couple issues, some of which likely were just the result of me doing something wrong.  However, I wanted to pass that information along in case it helps you later. 

Don’t run from $SPLUNK_HOME

Since this app is designed to upgrade the Splunk Universal Forwarder, you can’t deploy the app inside the Splunk UF’s installation directory.  If you try to do that, you’ll see this error when attempting to run the install script: 

2025-02-03-15:47:55 ### Discovering SPLUNK_HOME ###

2025-02-03-15:48:03 Adding new SPLUNK_HOME=/opt/splunkforwarder

SPLUNK_HOME=/opt/splunkforwarder

2025-02-03-15:48:03 Successfully set SPLUNK_HOME to the given location: /opt/splunkforwarder

2025-02-03-15:48:03 ### Ensure location not under SPLUNK_HOME ###

2025-02-03-15:48:03 Check if current location is under SPLUNK_HOME.

2025-02-03-15:48:03 This location is under SPLUNK_HOME, abort installation is required to prevent permission issues.

Permissions/ownership issues 

This seems to happen if the installation is interrupted by another check failing, and you need to run the script to install a second time.  In this case, the splunkupgrader user didn’t have complete ownership of everything in the application directory, and threw the following error:

2025-02-03-15:55:18 ### Beginning install ###

2025-02-03-15:55:18 Updater user “splunkupgrader” does not have execute permissions for SPLUNK_UPDATER_HOME=”/opt/splunkupgrader/bin/../”, aborting

I was able to fix this with the following command: 

chown -R splunkupgrader: /opt/splunkupgrader