Using synonyms in Solr

Have you tried adding custom synonyms to Solr on Drupal?

How did you go?

What do you need to do to connect a custom synonym list to your Solr?

Set-up : the tools

What am I using?

Search API Synonym- https://www.drupal.org/project/search_api_synonym

Solr : 8.1

Steps to get a custom Solr working

Install Search API Synonym

Read through the instructions on the module page or have a look at the following article

Setting up synonyms on your Drupal site with a few errors you might experience along the way

Adjust the settings

Fine tune the settings to your requirements. I set the following parameters

Type: All

Filter: No filter

Separate file: false

Only export if changes: true

Add / manage synonyms

After installation can you start managing your synonyms and spelling errors at

admin/config/search/search-api-synonyms

For initial testing I added a simple phrase:

Export your synonym list

While the Search Synonym API module has a cron export function, it doesn't move the file to the Solr directory.  Therefore, it will be more efficient to handle this through Terminal.  You can do this by using Drush:

drush search-api-synonym:export --plugin=solr --langcode=en

This script will add a file to the files (sites/default/files) directory.  In a directory called synonyms.  If the synonyms directory doesn't exist, create it and run the script again.

Note, there are many variations to this script, including but not limited to:

Export all English synonyms and spelling errors in the Solr format.

drush search-api-synonym:export --plugin=solr --langcode=en

Export all English spelling errors in the Solr format.

drush search-api-synonym:export --plugin=solr --langcode=en --type=spelling_error

Export all English synonyms without white spaces in the Solr format.

drush search-api-synonym:export --plugin=solr --langcode=en --type=synonym --filter=nospace

The file that was generated from the above command was synonyms__lang_en__type_all__filter_all.txt

Move the file to Solr

First cross the location of your Solr directory.  My Solr directory is located

/var/solr/data/{name}/conf/

Moving your file.  There are several methods to move a file.

• rsync
• cp

To list a couple.  I prefer rsync.  If you want you can read more about this in the article Are rsync commands are faster than cp.

rsync -au /var/www/html/path/to/your/synonym/directory/synonyms__lang_en__type_all__filter_all.txt /var/solr/data/{name}/conf/

Configure synonyms in Solr

To configure Solr to use synonyms you need to add a filter to the field type where you want synonyms to be used. For example, to enable synonyms for the text field in Solr I added a filter using the Synonym Graph Filter in our schema.xml.  Note the Synonym Filter is now deprecated.

For more details about this change see Solr Filters.

In the schema.xml file make the following additions:

  <field name="synonym_extra" type="string" indexed="true" stored="false" multiValued="true"/>

    <!-- A text field checks for synonyms -->
    <fieldType name="synonym_extra" stored="false" indexed="true" class="solr.TextField" >
      <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms__lang_en__type_all__filter_all.txt"/>
        <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters -->
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms__lang_en__type_all__filter_all.txt"/>
      </analyzer>
    </fieldType>

You can add this filter for indexing, for querying, or both.  In the example above I am only configuring the use of synonyms at query time.

Notice how the SynonymGraphFilterFactory references a synonyms__lang_en__type_all__filter_all.txt file. This text file is where synonyms are defined. Notice also the expanded=true setting.

As you have made changes to the Solr core directory, you need to restart Solr.

service solr restart

When you restart the Solr service, in your shell you will see an output similar to:

Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 23125 to stop gracefully.
Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=870). Happy searching!

The third line means Solr is running again.  You can cross check by viewing your dashboard.

Testing synonym matching in Solr

To see how synonyms are applied you can use the “Analysis” option available on the Solr dashboard page.

Image

The following picture shows how this tool can be used to verify how Solr is handling synonyms at index time. Notice, in the highlighted rectangle, how “setup” was indexed as:

• Set-up
• set-up
• setup
• set
• setup
• up

We are also able to use this tool to see how values are handled at query time. The following image shows how a query for “how “setup” indexed” is handled and matched to an original text “how “setup” indexed”. Specifically, in this case synonyms are enabled in the Solr configuration only at query time which explains why the indexed value (left side) only has “setup” but the value used at query time has been expanded to include the variations noted above that resulted in a match.

Image

Index vs Query time

When configuring synonyms in Solr is important to consider the advantages and disadvantages of using them at index time, query time or both.

Using synonyms at query time is easy because you don’t have to change your index to add or remove synonyms. You just add/remove lines from your synonym txt file.  Remember that any changes you make to the solr conf directory, that you will need to restart your Solr core.  Only post the restart will the synonyms be applied in subsequent searches.