Splunk certificate management is confusing at best, and the lack of detailed documentation along with the plethora of ‘gotchas’ is daunting when you are trying to implement a more secure Splunk environment. In this article, I’ll cover some basic tips, and provide advance guidance on some of the unexpected issues that can arise during an implementation of certificate management. So, why should you change the defaults?
Splunk’s Root Certificate Authority (CA)
Splunk comes pre-bundled with an internal root certificate authority (CA). This CA is the same root certificate on all Splunk instances for the last several years. Go ahead and check, it’s located in ‘$SPLUNK_HOME/etc/auth/ca.pem.default’ and has a serial number of “8d:1c:13:7c:b6:3f:c5:c3”. So what does this mean? This means the “secure” communications are not so secure and can be decrypted rather trivially using the Splunk provided defaults. Best practice is to enable TLS on all Splunk communications and replace the certificates with private or third party signed CA.
When enabling TLS on Splunk there are four main certificates that you should be aware of to enable and set up properly:
- The first is the Splunkd (server) certificate; this certificate is used by the underlying Splunk process that is bound to port 8089. This certificate is used for all Splunk to Splunk communications by default.
- Next is the web certificate, this certificate is served up to users when they browse to the HTTPS Splunk webpage (not enabled by default).
- Third is a certificate to be used for incoming data on port 9997, again not used by default.
- Lastly, the HTTP endpoint collector (HEC) certificate should be set up and in many instances requires a third party trusted certificate (looking at you, AWS firehose).
| Splunk Service | Default Cert | Default Status | Port |
| Splunkd | Server.pem | enabled | 8089 |
| Web | Web.pem | Disabled | 8000 |
| Indexer data | NA | Disabled | 9997 |
| HEC | NA | Disabled | 8088 |
Customizing the Splunk Certificate Authority
So how do you change these certificates? It’s pretty easy, but Splunk has several nuisances you should be aware of first. The biggest being not explicitly calling out the differences between TLS and mTLS. When enabling certain settings in this article, we will focus on the TLS settings and getting the data encrypted, similar to how the world wide web operates. The second big issue is when you change the “sslRootCAPath” setting in the server.conf. This has downstream effects of what certificates Splunk will trust. Alright, let’s get into it!
Splunk Certificate Management Setup
Are you using a private or third-party certificate, like Lets Encrypt? If you’re using a private certificate authority, I’d recommend using an organizational-wide private CA if this is available. This can simplify certificate management and address trust issues after the certificates are installed. In this article, we will focus on the private CA, as this will be the most common; we will also be using the same certificate for all of the different Splunk components. To start you will need to work with the CA team to retrieve the organization’s root certificate, any intermediate certificates, generate the Splunk certificate, and private key in the PEM format.
Once all required files are gathered, it is time to assemble them because Splunk expects the certificate files to follow a specific format. The web cert is the outlier as the other three primary certs all follow the same formatting as the server certificate. This format is one file with the server certificate, encrypted private key, intermediate certificates, and root certificate – in that order, from top to bottom.
The below chart shows a representation of how each certificate should look, and what files will be needed.
| Certificate | File Format | Filename | Comments |
| Splunkd
Indexer
HEC | —–BEGIN CERTIFICATE—–
….SERVER….. —–END CERTIFICATE—– —–BEGIN RSA PRIVATE KEY—– ….ENCRYPTED KEY…. —–END RSA PRIVATE KEY—– —–BEGIN CERTIFICATE—– ….INTERMEDIATE CA….. —–END CERTIFICATE—– —–BEGIN CERTIFICATE—– ….ROOT CA….. —–END CERTIFICATE—– | server.pem
indexer.pem
hec.pem | Requires intermediate and root CA certs.
Supports an encrypted key. |
| Web | —–BEGIN CERTIFICATE—–
….SERVER…. —–END CERTIFICATE—– | web.pem | Only the one cert is allowed |
| Web key | —–BEGIN PRIVATE KEY—–
….UNENCRYPTED KEY…. —–END PRIVATE KEY—– | web.key | Cannot be encrypted |
Once the formatting is complete from the above step you should have 5 certs as shown below.
These certs can be copied to a new directory, that will hold our custom certificates in the $SPLUNK_HOME/etc/auth/ path. The name of the new directory does not matter; it is just a logical separator for us.
Read More: Upgrading to Splunk 8
How to Configure Splunk!
The primary files we will have to modify are the server.conf, web.conf, and inputs.conf files. These all most likely already exist in $SPLUNK_HOME/etc/system/local and can be modified there. However, I’d recommend making a separate app in $SPLUNK_HOME/etc/apps/ to keep similar settings together.
The first file web.conf is the easiest to configure and has the least amount of ramifications.
| # web.conf
[settings] enableSplunkWebSSL = 1 httpport = 8000 privKeyPath = $SPLUNK_HOME/etc/auth/myCerts/web.key serverCert = $SPLUNK_HOME/etc/auth/mCerts/web.pem |
Next is the inputs.conf
| #inputs.conf
[splunktcp-ssl://9997] serverCert = $SPLUNK_HOME/etc/auth/myCerts/indexer.pem sslPassword = <password> # Used to encrypted private key requireClientCert = false cipherSuite = TLSv1.2:!eNULL:!aNULL [http] disabled = 0 enableSSL = 1 sslPassword = <password> # Used to encrypted private key serverCert = $SPLUNK_HOME/etc/auth/dwcerts/server.pem |
The last file is the server.conf
| # server.conf
[sslConfig] enableSplunkdSSL = true serverCert = $SPLUNK_HOME/etc/auth/dwcerts/server.pem sslPassword = <password> # Used to encrypted private key sslRootCAPath = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem sslVersions = tls1.2 cipherSuite = TLSv1.2:!eNULL:!aNULL |
Final Considerations on Configuration Settings
A few quick notes about the server.conf file, the most important is the sslRootCAPath setting, this setting tells Splunk what root certificate authorities it will trust and by default is not set, which means it trusts all CA’s. I would recommend setting this to be the CA bundle of the OS, which in my example is CentOS7. If you are running a different OS this can be found by searching the Internet for OS version root CA store. When you set this to the OS CA bundle and you’re using a private CA you will need to add your private Root CA certificate to the bundle if it’s not already installed. A good how-to on this can be found here.
The other comment is that there are several more settings that can be modified to further secure the communications. Some of these settings are sslVerifyServerCert, sslCommonNameToCheck, sslAltNameToCheck, and requireClientCert. These settings go along with mTLS and I would not recommend setting these until the basic TLS deployment has been fully tested as described in this article. Setting these configurations can break Splunk-to-Splunk communications and some Splunk CLI commands as the CLI is not able to provide a certificate to the Splunkd process.
Read More: Why It’s Time to Move from On-Prem to Splunk Cloud
Time to Restart Splunk!
With these files updated, it is now safe to restart the Splunk instance and validate Splunk is using your new private certificates. The easiest way to do this is by opening a web browser and navigating to the main Splunk login. Click the lock icon near the URL bar to display the certificate information, it should show that it was issued by your organization CA. Next is to validate that the Splunkd process is using the new certificate. This can be performed by opening a terminal session and running the command ‘openssl s_cleint -connect hostname:port’ which will retrieve the certificate information bond to that port. Again, the issuer should show your organization’s CA and not the Splunk Common CA.
Deepwatch + Splunk
Deepwatch is an Elite Tier Splunk MSP with a large team of experienced Splunk engineers. Deepwatch is Splunk’s #1 MSSP in terms of the amount of data we manage across our customers. Splunk customers trust Deepwatch as a true extension of their security team. Whether you have an existing Splunk license or you are looking to get started with Splunk, Deepwatch can help. Let’s talk.
Happy Splunking!
↑
Share