How to set up a CKAN environment using Docker with Solr schema configuration

Setting up CKAN in a Docker environment can sometimes require some additional troubleshooting, particularly when working with Solr for search functionality. In this article, we’ll walk through how to set up CKAN 2.11 in a Docker environment with a specific focus on resolving issues related to Solr schema configuration.

Prerequisites

Before we start, ensure that you have the following installed on your system:

- Docker
- Docker Compose

Step 1: Setting up your Docker environment

First, we need to create a `docker-compose.yml` file to define the required services for CKAN (PostgreSQL, Solr, Redis, CKAN, and a Datastore). Below is a sample `docker-compose.yml` file for CKAN 2.11:

version: '3.8'
services:
 db:
   image: postgres:12
   environment:
     POSTGRES_DB: ckan
     POSTGRES_USER: ckan
     POSTGRES_PASSWORD: mysecretpassword
   volumes:
     - ./db_data:/var/lib/postgresql/data
 solr:
   image: solr:8.11.1
   entrypoint:
     - docker-entrypoint.sh
     - solr-precreate
     - ckan
   volumes:
     - ./solr_data:/var/solr
   ports:
     - "8983:8983"
 redis:
   image: redis:6
 ckan:
   build: .
   environment:
     - CKAN_SITE_URL=http://localhost:5000
     - CKAN_SQLALCHEMY_URL=postgresql://ckan:mysecretpassword@db/ckan
     - CKAN_SOLR_URL=http://solr:8983/solr/ckan
     - CKAN_REDIS_URL=redis://redis:6379/0
     - CKAN_DATASTORE_WRITE_URL=postgresql://ckan:mysecretpassword@datastore/datastore
     - CKAN_DATASTORE_READ_URL=postgresql://ckan:mysecretpassword@datastore/datastore
   depends_on:
     - db
     - solr
     - redis
     - datastore
   volumes:
     - ./ckan_storage:/var/lib/ckan
     - ./config:/etc/ckan
   ports:
     - "5001:5000"
 datastore:
   image: postgres:12
   environment:
     POSTGRES_DB: datastore
     POSTGRES_USER: ckan
     POSTGRES_PASSWORD: mysecretpassword
   volumes:
     - ./datastore_data:/var/lib/postgresql/data
volumes:
 db_data:
 solr_data:
 datastore_data:
 ckan_storage:

Step 2: Configuring CKAN with Docker

Next, we’ll create a `Dockerfile` to define the CKAN image. This Dockerfile will use Python 3.12.7, install CKAN and its dependencies, and configure CKAN to run on port 5000.

Here’s the `Dockerfile`:

# Use Python 3.12.7 slim image
FROM python:3.12.7-slim
# Install necessary system dependencies, including libmagic
RUN apt-get update && apt-get install -y \
   git-core \
   libpq-dev \
   libffi-dev \
   libxml2-dev \
   libxslt1-dev \
   python3-dev \
   libjpeg-dev \
   zlib1g-dev \
   g++ \
   curl \
   libmagic1 \
   libmagic-dev && \
   rm -rf /var/lib/apt/lists/*
# Upgrade pip and install CKAN
RUN pip install --upgrade pip
# Install CKAN using pip for version 2.11
RUN pip install ckan==2.11.0
# Install CKAN's dependencies (including 'click')
RUN pip install -r https://raw.githubusercontent.com/ckan/ckan/ckan-2.11.0/requirements.txt
# Set CKAN_HOME environment variable
ENV CKAN_HOME=/usr/local/lib/python3.12/site-packages/ckan
# Add config file directory for CKAN settings
RUN mkdir -p /etc/ckan
# Copy entrypoint script into the container
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
# Make the entrypoint script executable
RUN chmod +x /usr/local/bin/entrypoint.sh
# Create volume for CKAN storage
VOLUME /var/lib/ckan
# Expose CKAN's internal port 5000
EXPOSE 5000
# Use the entrypoint script to generate config and start CKAN
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
# Start CKAN using the CKAN CLI
CMD ["ckan", "-c", "/etc/ckan/ckan.ini", "run"]

Step 3: Entry point for CKAN configuration

You need to create an `entrypoint.sh` script that will generate the `ckan.ini` file if it doesn’t already exist, and then start CKAN:

#!/bin/bash
# Check if the config file exists, and if not, generate it
if [ ! -f "/etc/ckan/ckan.ini" ]; then
   echo "Generating CKAN configuration file..."
   ckan generate config /etc/ckan/ckan.ini
fi
# Start CKAN
exec "$@"

Step 4: Solr schema configuration

If you encounter issues with CKAN connecting to Solr, it’s likely due to the Solr schema version. CKAN requires a specific schema configuration for Solr.

Here’s how you can resolve this:

1. Log into the Solr container

docker exec -it project-ckan-solr-1 /bin/bash

2. Change to the correct directory

cd /var/solr/data/ckan/conf

3. Download the required Solr schema

curl -o schema.xml https://raw.githubusercontent.com/ckan/ckan/dev-v2.11/ckan/config/solr/schema.xml

4. Exit the container

 exit

Step 5: Restart the environment

To apply all changes, you’ll need to restart the Docker environment:

docker compose down
docker compose up --build -d

The wrap

By following these steps, you should have a fully functional CKAN 2.11 environment running in Docker. We resolved some common issues such as Solr schema compatibility and ensured that CKAN runs smoothly with Docker Compose.

If you encounter further issues, you can check the CKAN container logs with:

docker logs <ckan_container_name>

This setup provides a solid foundation for running CKAN in Docker.