Overcoming startup challenges with Apache Solr on Ubuntu 24.04

Recently, after upgrading to Ubuntu 24.04, we encountered a significant challenge with our Apache Solr service—it refused to restart. This post documents the steps I took to diagnose and resolve the issues, providing a clear guide for anyone facing similar troubles.

Initial troubleshooting

The journey began with a simple command to check the status of our Solr service:

sudo service solr status

The response indicated that the service had failed:

× solr.service - LSB: Controls Apache Solr as a Service
     Loaded: loaded (/etc/init.d/solr; generated)
     Active: failed (Result: exit-code) since Fri 2024-06-07 04:29:06 UTC; 6h ago
       Docs: man:systemd-sysv-generator(8)
    Process: 5755 ExecStart=/etc/init.d/solr start (code=exited, status=1/FAILURE)
        CPU: 11ms
Jun 07 04:29:06 UAT systemd[1]: Starting solr.service - LSB: Controls Apache Solr as a Service...
Jun 07 04:29:06 UAT su[5757]: (to solr) root on none
Jun 07 04:29:06 UAT su[5757]: pam_unix(su-l:session): session opened for user solr(uid=114) by (uid=0)
Jun 07 04:29:06 UAT su[5757]: pam_unix(su-l:session): session closed for user solr
Jun 07 04:29:06 UAT systemd[1]: solr.service: Control process exited, code=exited, status=1/FAILURE
Jun 07 04:29:06 UAT systemd[1]: solr.service: Failed with result 'exit-code'.
Jun 07 04:29:06 UAT systemd[1]: Failed to start solr.service - LSB: Controls Apache Solr as a Service.

To probe deeper, I tried starting the service

sudo service solr start

Giving the following response

Job for solr.service failed because the control process exited with error code.
See "systemctl status solr.service" and "journalctl -xeu solr.service" for details.

This attempt also failed, prompting us to consult the Solr logs and the systemd journal.

Diving into the logs

Solr logs, located either in /var/solr/logs or /opt/solr/logs depending on the installation, was my next stop. Here I hoped to find clues about what might be going wrong during the startup.

Exploring the init script output

The Solr service on our system is managed via an init script (/etc/init.d/solr). Executing this script manually revealed several warnings and an error

*** [WARN] *** Your open file limit is currently 1024.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 31733.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Port 8983 is already being used by another process (pid: 915)
Please choose a different port using the -p option.

These messages pointed out two major issues:

1. Inadequate System Limits: Both the open file limit and the max processes limit were far below the recommended settings; and
2. Port Conflict: Port 8983 was occupied by another process, preventing Solr from using it.

Addressing the Configuration Issues

You can increase the open file and max processes limits for the Solr user by modifying system limits.  Edit the limits configuration by opening the /etc/security/limits.conf file with an editor like nano or vim:

sudo nano /etc/security/limits.conf

I appended the following lines to configure the appropriate limits for the Solr user to the end of the file

solr soft nofile 65000
solr hard nofile 65000
solr soft nproc 65000
solr hard nproc 65000

For these changes to take effect, you may need to log out and log back in, or reboot the system. This ensures that the new limits are applied when you start Solr.  In our situation I rebooted the server.

After rebooting the server to ensure all configurations were applied, I restarted Solr

sudo service solr start

A subsequent status check confirmed that Solr was active and running smoothly

● solr.service - LSB: Controls Apache Solr as a Service
    Loaded: loaded (/etc/init.d/solr; generated)
    Active: active (exited) since Fri 2024-06-07 10:39:30 UTC; 38s ago
      Docs: man:systemd-sysv-generator(8)
   Process: 778 ExecStart=/etc/init.d/solr start (code=exited, status=0/SUCCESS)
       CPU: 14ms
Jun 07 10:39:14 UAT systemd[1]: Starting solr.service - LSB: Controls Apache Solr as a Service...
Jun 07 10:39:15 UAT su[788]: (to solr) root on none
Jun 07 10:39:15 UAT su[788]: pam_unix(su-l:session): session opened for user solr(uid=114) by (uid=0)
Jun 07 10:39:30 UAT solr[1472]: Started Solr server on port 8983 (pid=1466). Happy searching!
Jun 07 10:39:30 UAT systemd[1]: Started solr.service - LSB: Controls Apache Solr as a Service.

The wrap

This experience underscores the importance of thorough system checks and log reviews when troubleshooting services like Apache Solr. By adjusting the system limits and resolving port conflicts, I was able to get Solr back up and running on Ubuntu 24.04.