This article was written to help organizations prepare for challenges when upgrading their Splunk Enterprise infrastructure from Splunk version 7.x to 8.x. We will review how deepwatch has been able to do this at scale for its customers. If you are still running a major Splunk version below 7.x, please visit Splunk’s website for the recommended upgrade path to upgrade to Splunk 8.
Why Should I Upgrade to Splunk 8?
There are many reasons that you and your organization should upgrade your Splunk environment to version 8. These reasons include providing the end-user with the most up-to-date features and functionality available, maintaining a mature application lifecycle policy across your organization, and to simply stay within the boundaries of supported Splunk versions. Running out-of-date or end-of-life applications can pose more risk to an organization than maintaining a reasonable patching cycle that is in line with vendor recommendations for their products. There is no denying that Splunk is a very consistent and reliable product and we only see Splunk getting better and better. For a full list of added features and enhancements, please review Splunk’s official release notes for the version you are targeting.
What Are the Risks?
Calculating risks will depend on your architectural design and organization requirements. We will focus on what we believe are the most common risks associated with upgrading to Splunk 8 from Splunk 7.
First and foremost, Splunk made a difficult decision to switch its primary Python interpreter from Python 2 to Python 3. This was due to Python 2 becoming an end-of-life version as described in PEP 373. The most notable impact of the Python version change pertains to Splunk’s data ingestion mechanisms. While more and more apps and add-ons are getting updated and built for compatibility with Python 3, there are still a few legacy add-ons available for download on Splunkbase that currently still require Python 2. This change, combined with the amount of legacy apps or add-ons you might have deployed in your environment, will determine the amount of effort required to ensure minimal impact to functionality post-upgrade.
The second compatibility challenge is caused by Splunk completely removing support for the embedding advanced XML within Splunk applications – something Splunk has discouraged in any application or dashboard since Splunk 6. Because of this, the embedding of advanced XML has largely been abandoned by developers since the launch of Splunk 6. If you are currently running some older or home-grown apps that use advanced XML, there will be compatibility issues.
To learn more about Splunk 8.0 upgrade pitfalls, please see the Splunk installation manual.
How can I Identify what Risks Apply to me?
The good news is that Splunk provides a very useful tool to help identify areas of concern and organize risks per app associated with the compatibility challenges within this particular upgrade path. The Splunk Platform Upgrade Readiness App can be installed on any infrastructure running Splunk Enterprise just like any other Splunk application. Once installed, there are a series of scans and searches this app can perform to identify potential Python compatibility issues as well as if any advanced XML are embedded in your applications. It even provides the ability to export findings for further analysis. This tool will save you a massive amount of time identifying any potential risks for a subset or all apps installed on any Splunk Enterprise host. The app is highly recommended by both deepwatch and Splunk, and is a big reason deepwatch has had great success upgrading over 100 unique customer environments to Splunk 8.
I’ve Identified my Compatibility Risks. What Now?
Scope out your work
After you receive the output from your environment scan using the Upgrade Readiness App, search for “Warnings” and “Blockers”. “Blockers” will need a bit more care and focus for this upgrade process and will typically require manual remediation per app. For any app that is identified as a “Warning” or a “Blocker”, check to see if there is a newer version available on Splunkbase. The majority of apps or add-ons identified by the scan will likely be out of date and will need to be updated. Check if Splunkbase has a newer version of your app or add-on. If an update is not available on Splunkbase for the identified app or add-on, reach out to Splunk support or the technology vendor to validate that there isn’t a newer version of that app or add-on available. Should you find that there isn’t a new version available to upgrade to, you will need to consult with your Splunk admin on what functionality can be salvaged by either converting the applicable Python code to be compatible with Python 3 or by utilizing a different ingestion method for that data source. Splunk provides some high-level guidelines on how to convert any Python 2 script to become compatible with Python 3 here.
Identify key stakeholders
It will prove to be valuable to your organization if you identify key stakeholders for this upgrade project and to track the changes that will happen in your environment. These individuals should be able to articulate the impact and scope of work for this effort, as it will inevitably require one or many maintenance windows to complete it; which depends on the size and complexity of the project and your Splunk deployment.
Test in a lab environment
The great thing about Splunk is that it is machine agnostic. If you have the available infrastructure or can quickly provision infrastructure for testing, you can simply copy over the working installation to a separate environment to test the pending changes to your production environment to help mitigate any potential loss in functionality. If you are an existing Splunk customer, you can apply for a 50GB developer license to cover any additional licensing costs, if you feel that ingesting live data is required for an accurate upgrade simulation. To apply for a dev/test license, visit Splunk’s developer licensing page. Testing these changes in a lab environment will let you get ahead of any potential challenges you will experience during the Splunk 8.0 upgrade process, without impacting your production environment.
Automate what you can
Depending on the size of your deployment, it may prove beneficial to automate particular portions of your upgrade, or the entire process depending on the tools and expertise at your disposal. deepwatch, on account of being an MDR provider for a large number of customers, needed a scalable, repeatable solution to perform this task across our customer base. We have found that tools like Ansible, Terraform, Python, and even bash/powershell scripts are valuable time-savers during our upgrade process. If you choose to automate any portion of your upgrade, make sure you develop with the intention to reuse and iterate upon your codebase. This will likely not be your last Splunk upgrade, so this a perfect opportunity to start capturing and improving upon your upgrade process from the start. Doing so will reduce load and complexity for each upgrade your organization completes in the future. Version control technologies such as git are perfectly applicable in this situation and will give you insights into how your team is accounting for changes and improvements to your automations with every upgrade cycle.
Keeping Splunk Enterprise up to date should be a priority for any organization. Maintaining a reasonable Splunk version lifecycle and making sure it is a priority for your business, will ensure you are getting the best return on your investment and allow you to take full advantage of the many of the new features that Splunk releases with each new version.