server 500 error - 'Internal server error' on Nginx

Finding the issue with a server 500 error

The Nginx error logs are showing too much detail.

Nginx logs and test

There are no errors in the nginx log and sudo nginx -t looks fine

sudo nginx -t

Response

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Are the services running?

I'll start by doing a status check for 

• Nginx
• PostgreSQL
• jetty9
• Supervisor

Nginx

sudo service nginx status

Response

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-08-05 02:49:49 UTC; 24min ago
       Docs: man:nginx(8)
    Process: 15758 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 15760 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 15761 (nginx)
      Tasks: 4 (limit: 9530)
     Memory: 4.0M
     CGroup: /system.slice/nginx.service
             ├─15761 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─15762 nginx: worker process
             ├─15763 nginx: worker process
             └─15764 nginx: cache manager process

systemd[1]: Starting A high performance web server and a reverse proxy server...
systemd[1]: Started A high performance web server and a reverse proxy server.

No stand-out issues above.

PostgreSQL

sudo service postgresql status

Response

● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2022-08-05 00:21:25 UTC; 2h 54min ago
    Process: 1257 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
   Main PID: 1257 (code=exited, status=0/SUCCESS)

systemd[1]: Starting PostgreSQL RDBMS...
systemd[1]: Finished PostgreSQL RDBMS.

No stand-out issues above.

jetty9

sudo service jetty9 status

Response

lines 1-22/22 (END)
● jetty9.service - Jetty 9 Web Application Server
     Loaded: loaded (/lib/systemd/system/jetty9.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jetty9.service.d
             └─solr-permissions.conf, solr.conf
     Active: active (running) since Fri 2022-08-05 02:47:48 UTC; 28min ago
       Docs: https://www.eclipse.org/jetty/documentation/current/
   Main PID: 15567 (java)
      Tasks: 30 (limit: 9530)
     Memory: 228.9M
     CGroup: /system.slice/jetty9.service
             └─15567 /usr/bin/java -Djetty.home=/usr/share/jetty9 -Djetty.base=/usr/share/jetty9 -Djava.io.tmpdir=/tmp -jar /usr/share/jetty9/start.jar jetty.state=/var/lib/jetty9/jetty.state jetty-started.xml

jetty9[15567]: 2022-08-05 02:47:51.855:WARN:oeju.DeprecationWarning:main: Using @Deprecated Class org.apache.solr.servlet.SolrUpdateServlet
jetty9[15567]: Aug 05, 2022 2:47:51 AM org.apache.solr.core.SolrResourceLoader locateSolrHome
jetty9[15567]: INFO: No /solr/home in JNDI
jetty9[15567]: Aug 05, 2022 2:47:51 AM org.apache.solr.core.SolrResourceLoader locateSolrHome
jetty9[15567]: INFO: using system property solr.solr.home: /usr/share/solr
jetty9[15567]: Aug 05, 2022 2:47:51 AM org.apache.solr.servlet.SolrUpdateServlet init
jetty9[15567]: INFO: SolrUpdateServlet.init() done
jetty9[15567]: 2022-08-05 02:47:51.871:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3ad2e17{/solr,file:///usr/share/solr/web/,AVAILABLE}{/usr/share/solr/web}
jetty9[15567]: 2022-08-05 02:47:51.891:INFO:oejs.AbstractConnector:main: Started ServerConnector@32148249{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
jetty9[15567]: 2022-08-05 02:47:51.892:INFO:oejs.Server:main: Started @3507ms

No stand-out issues above.

Supervisor

sudo service supervisor status

Response

● supervisor.service - Supervisor process control system for UNIX
     Loaded: loaded (/lib/systemd/system/supervisor.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-08-05 01:16:01 UTC; 2h 2min ago
       Docs: http://supervisord.org
   Main PID: 7431 (supervisord)
      Tasks: 4 (limit: 9530)
     Memory: 67.5M
     CGroup: /system.slice/supervisor.service
             ├─7431 /usr/bin/python3 /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
             ├─7923 /usr/lib/ckan/default/bin/uwsgi -i /etc/ckan/default/ckan-uwsgi.ini
             ├─7933 /usr/lib/ckan/default/bin/uwsgi -i /etc/ckan/default/ckan-uwsgi.ini
             └─7934 /usr/lib/ckan/default/bin/uwsgi -i /etc/ckan/default/ckan-uwsgi.ini

supervisord[7431]: 2022-08-05 01:16:13,812 INFO success: ckan-uwsgi-00 entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
supervisord[7431]: 2022-08-05 01:21:25,176 INFO waiting for ckan-uwsgi-00 to die
supervisord[7431]: 2022-08-05 01:21:27,176 INFO stopped: ckan-uwsgi-00 (exit status 0)
supervisord[7431]: 2022-08-05 01:21:27,183 INFO Included extra file "/etc/supervisor/conf.d/ckan-uwsgi.conf" during parsing
supervisord[7431]: 2022-08-05 01:21:27,183 INFO Set uid to user 0 succeeded
supervisord[7431]: 2022-08-05 01:21:27,420 INFO RPC interface 'supervisor' initialized
supervisord[7431]: 2022-08-05 01:21:27,421 INFO RPC interface 'supervisor' initialized
supervisord[7431]: 2022-08-05 01:21:27,423 INFO supervisord started with pid 7431
supervisord[7431]: 2022-08-05 01:21:28,426 INFO spawned: 'ckan-uwsgi-00' with pid 7923
supervisord[7431]: 2022-08-05 01:21:38,855 INFO success: ckan-uwsgi-00 entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

Again no critical errors

Check the log files

View the ckan.ini file (/etc/ckan/default/ckan.ini)