Lime CRM Search Indexer

In the Lime CRM application, it is important that all data intended for searchability is indexed in Elasticsearch. The responsibility of automatically indexing objects when they are updated in the server lies with the Lime CRM Search Indexer service. Whenever a searchable object undergoes modification or addition, an internal index event is triggered within Lime CRM. This event is subsequently dispatched to a message queue (RabbitMQ) and eventually retrieved by the search indexer service. Consequently, the service proceeds to index the respective change.

Information

Data that is not indexed will not be possible to find from the global search in the web client.

What happens when an object is created, updated or deleted?

When data is added, changed or deleted through Lime CRM Server, the data is also updated in the search index.

Warning

If data is inserted directly to the SQL database, for instance via a SSIS integration, Lime CRM Server has no possibility to detect this. This means that that data will not be indexed and searchable in the global search.

The search index is updated if data is added, updated or deleted via:

• The web client
• The desktop client
• The REST API
• Custom endpoints
• The importer

If data that needs to be searchable in the web client is added in any other way, you must manually trigger a reindex of the table that you added this data to. Again, this is a time consuming and CPU intensive process, and is not recommended. But sometimes it's necessary and then it's best solved with a scheduled task that runs a reindex operation periodically (nightly, weekly, monthly...).

Changes to the database structure

Prior versions of Lime CRM required a reindex to synchronize changes to properties and limetypes in LISA. However, with the introduction of Lime CRM 2023.1 (Hvannadalsnjúkur), utilizing Elasticsearch 7, the need for reindexing has been significantly reduced.

Now, properties and limetypes can be added, updated, or deleted without a mandatory reindex. The search functionality remains effective and unchanged, even with modifications to the database. However, certain situations may still require a reindex. These include:

• Addition of New Properties: When a new property (field) is added, it will be indexed and assumed to be of type string. For instance, if you add a new option field, Elasticsearch will treat that field as a string, which may lead to undefined behaviors.

• Deletion of Properties: If a property is deleted, the corresponding data remains stored in the index, despite being removed from the database. This means the deleted data can still be searched. That data will be removed when an object is updated, but a complete removal of all associated data is most conveniently achieved through a reindex.

Search results

When searching in the web client, the results are boosted depending on:

• Limetype (Table) labels
• Property (Field) labels

This means that a match in the search query on a limetype with a limetype label and in a property with a property label is more likely to get a higher boost, and be presented higher in the search result list.

The following property types are possible to search for in the web client:

• text properties
• option properties
• relation properties

Relation properties

It's possible to search for information in related objects. However, it's only the descriptive of a related limeobject that is indexed together with the object. This means that it's important to set labels on all limeobject's limetype's that you want to have related information from.

What this means is that if you search for a deal connected to "Company A", you may get a search hit for "Company A", if the deals label is set to name.

Working with the Search Indexer

The Search Indexer is interacted with via limefu.

Configuration

Connection details are configured as described here.

Service configuration:

search_indexer: 
    log_level: INFO

Exclude lime types

From lime-crm version 2.413.0, you can configure your application to exclude certain lime types from being indexed. This can be very useful if a certain lime type has a huge number of records but does not need to be searchable, i.e. indexed ¹. Lime types such as participant, orderrow or invoicerow might exist in the quantities of millions, due to integrations. By excluding such lime types from being indexed, the index becomes smaller, thus making a re-index much faster.

Add this section to your configuration in CAFE (cloud) or application_config.yaml (on-premise) to exclude specific lime types. Replace participant and invoicerow with the lime types you wish to exclude.

search:
    exclude_limetypes:
        - participant
        - invoicerow

Managing indexes in Elasticsearch 7

With the introduction of Elasticsearch 7 in Lime CRM 2023.1 (Hvannadalsnjúkur), index management within Lime CRM has undergone significant changes.

In prior versions, a single active index contained all application data. However, this model has been updated in Lime CRM 2023.1.

Here's an overview of the revised structure:

The concept of an active index has been phased out. Now, a single application can possess multiple indexes, all of which can be searched simultaneously. Upon the creation of a new index, it is assigned an alias that corresponds to the application name (e.g., app). Concurrently, the is_write_index flag is set to true.

Should an existing index already be associated with the application, the is_write_index flag for that index will be reset to false. This alteration enables the use of the index alias in lieu of the index name when operating with the index. New data is directed to the most recent index (where is_write_index is true), while searches can be conducted across all application indexes.

This modification facilitates simultaneous searching and updating of the index while a re-index is underway. Under the old structure, a re-index would discard any updates made during the process as writes were directed to two distinct indexes, and the old index was discarded post-re-indexing.

If a document is updated, it is transferred to the new index. However, the previous indexes may still house the old, now-outdated, data. To manage this issue, any document added to the new index is removed from the old indexes.

When a new index is established, the preceding indexes are granted a new alias featuring the __old suffix, e.g., app__old. This system simplifies the process of removing documents from old indexes, ensuring that no out-of-date data remains searchable.

Example

To illustrate these changes, consider an application named app with the following indexes and aliases:

• Index ab123-app associated with aliases:
  • app
  • app__old
• Index cd456-app associated with alias:
  • app

In this scenario, data searches within the application will use the app alias. Data writing operations also use the app alias, with the cd456-app index being the targeted destination since it's is_write_index is set to true for the app alias.

The app__old alias serves as a marker for identifying old indexes. This identification enables the removal of stale data from the associated index, ensuring data relevancy and accuracy.

---

1. Please note that these records will still show up in lists, tables, reports and such. They will only be excluded from features where Elasticsearch is used, such as the global search and relation pickers. ↩