ckan-worker:ckan-worker-00 FATAL Exited too quickly (process log may have details)

Working through a CKAN installation on Ubuntu 20.04

User sees a server 500 error.

syslog shows: ckan-worker:ckan-worker-00 FATAL Exited too quickly (process log may have details)

Information:

syslog

Jul 19 03:59:43 {project}-CKAN supervisord[773]: 2022-07-19 03:59:43,515 INFO spawned: 'ckan-worker-00' with pid 4337
Jul 19 03:59:44 {project}-CKAN supervisord[773]: 2022-07-19 03:59:44,900 INFO exited: ckan-worker-00 (exit status 1; not expected)

The error is pointing to supervisord, run the command

sudo supervisorctl status

Response:

ckan-datapusher:ckan-datapusher-00   RUNNING   pid 1149, uptime 0:24:31
ckan-uwsgi:ckan-uwsgi-00             RUNNING   pid 1150, uptime 0:24:31
ckan-worker:ckan-worker-00           BACKOFF   Exited too quickly (process log may have details)

Restarted supervisor by executing

sudo supervisorctl reload

To which you'll get the response

Restarted supervisord

No success.

Supervisor

Configuration files

/etc/supervisor/
/etc/supervisor/conf.d/

Supervisor log files location:

stdout_logfile=/var/log/ckan/ckan-worker.stdout.log
stderr_logfile=/var/log/ckan/ckan-worker.stderr.log

The error log is showing

2022-07-19 04:16:21,572 INFO  [ckan.cli] Using configuration file /etc/ckan/default/ckan.ini
2022-07-19 04:16:21,572 INFO  [ckan.config.environment] Loading static files from public
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/ckan", line 11, in <module>
    load_entry_point('ckan', 'console_scripts', 'ckan')()
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 115, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 125, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 134, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 56, in __init__
    self.app = make_app(self.config)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 161, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 208, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 236, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 153, in plugins_update
    environment.update_config()
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 198, in update_config
    raise RuntimeError(
RuntimeError: ckan.site_url is not configured and it must have a value. Please amend your .ini file.
2022-07-19 04:16:57,816 INFO  [ckan.cli] Using configuration file /etc/ckan/default/ckan.ini
2022-07-19 04:16:57,817 INFO  [ckan.config.environment] Loading static files from public
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/ckan", line 11, in <module>
    load_entry_point('ckan', 'console_scripts', 'ckan')()
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 115, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 125, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 134, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/usr/lib/ckan/default/src/ckan/ckan/cli/cli.py", line 56, in __init__
    self.app = make_app(self.config)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 161, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 208, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 236, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 153, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 198, in update_config
    raise RuntimeError(
RuntimeError: ckan.site_url is not configured and it must have a value. Please amend your .ini file.

Focusing on the ckan.site_url issue, let's hit that first.  The location of this file is

/etc/ckan/default/ckan.ini

To edit this file using vim

sudo vim /etc/ckan/default/ckan.ini

Find the site_url and update accordingly.  Once completed, you will need to provide the location to the configuration file.

Initialize the CKAN database

sudo ckan db init

OR

sudo ckan -c /etc/ckan/default/ckan.ini db init

However, this command had the following response

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "ckan_default"
FATAL:  password authentication failed for user "ckan_default"

ckan_default is the default user in PostgreSQL

This issue is due to the password for this user not being set in the sqlalchemy.url setting in the ckan.ini file.

## Database Settings
sqlalchemy.url = postgresql://ckan_default:{pass}@localhost/ckan_default

Make sure you have replaced {pass} with the password you have used.

Running the init command again

ckan -c /etc/ckan/default/ckan.ini db init

Response:

2022-07-19 04:36:28,561 INFO  [ckan.cli.db] Initialize the Database
2022-07-19 04:36:31,104 INFO  [ckan.model] CKAN database version upgraded: base -> ccd38ad5fced (head)
2022-07-19 04:36:31,105 INFO  [ckan.model] Database initialised
Initialising DB: SUCCESS

Now is a good time to retest the supervisor status

sudo supervisorctl status

Response

ckan-datapusher:ckan-datapusher-00   RUNNING   pid 4846, uptime 0:46:00
ckan-uwsgi:ckan-uwsgi-00             RUNNING   pid 4847, uptime 0:46:00
ckan-worker:ckan-worker-00           RUNNING   pid 7844, uptime 0:15:06

Back on track and working

Reload the Supervisor daemon so the new processes are picked up:

sudo supervisorctl reload

After a few seconds run the following command to check the status of the processes:

sudo supervisorctl status

Like above, you will hopefully see three processes running without errors:

ckan-datapusher:ckan-datapusher-00   RUNNING   pid 25124, uptime 0:03:18
ckan-uwsgi:ckan-uwsgi-00             RUNNING   pid 25125, uptime 0:03:18
ckan-worker:ckan-worker-00           RUNNING   pid 25126, uptime 0:03:18

If some of the processes report an error, ensure you’ve run all the previous steps and check the logs located in /var/log/ckan for more details.