Lime CRM Search Indexer¶
In the Lime CRM application, it is important that all data intended for searchability is indexed in Elasticsearch. The responsibility of automatically indexing objects when they are updated in the server lies with the Lime CRM Search indexer service. Whenever a searchable object undergoes modification or addition, an internal index event is triggered within Lime CRM. This event is subsequently dispatched to a message queue (Rabbit MQ) and eventually retrieved by the search indexer service. Consequently, the service proceeds to index the respective change.
Data that is not indexed will not be possible to find from the global search in the webclient
What happens when an object is created, updated or deleted?¶
When data is added, changed or deleted through Lime CRM Server, the data is also updated in the search index.
If data is inserted directly to the sql database, for instance via a SSIS integration, Lime CRM Server has no possibility to detect this. This means that the data will not be searchable in the global search.
The search index is updated if data is added, updated or deleted via:
- The webclient
- The desktop client
- The REST Api
- A custom endpoint
- The importer
If data that needs to be searchable in the webclient is added in any other way, you must manually trigger a reindex of the table that you added this data to. Again, this is a time consuming and CPU intensive process, and is not recommended to do. But sometimes it's neccessarry and then its best solved with a scheduled task that runs a reindex operation periodically (nightly, weekly, monthly...).
Changes to the database structure¶
Prior versions of Lime CRM necessitated a reindex to synchronize changes to properties and limetypes in LISA. However, with the introduction of CRM version 2023.1 Hvannadalsnjúkur, utilizing Elasticsearch 7, the need for reindexing has been significantly reduced.
Now, properties and limetypes can be added, updated, or deleted without a mandatory reindex. The search functionality remains effective and unchanged, even with modifications to the database. However, certain situations may still require a reindex. These include:
Addition of New Properties: When a new property (field) is added, it will be indexed and assumed to be of type
string. For instance, if you add a new option field, Elasticsearch will treat that field as a string, which may lead to undefined behaviors.
Deletion of Properties: If a property is deleted, the corresponding data remains stored in the index, despite its removal from the database. This means the deleted data can still be searched. The data will be eliminated if an object is updated, but a complete removal of all associated data is most conveniently achieved through a reindex.
When searching in the web client, the results are boosted depending on:
- Limetype (Table) labels
- Property (Field) labels
This means that a match in the search query on a limetype with a limetype label and in a property with a property label is more likely to get a higher boost, and be presented higher in the search result list.
The following property types are possible to search for in the web client:
- text properties
- option properties
- relation properties
It's possible to search for information in related objects. However, it's only the descriptive of a related limeobject that is indexed together with the object. This means that it's important to set labels on all limeobjects limetypes that you want to have related information from.
What this means is that if you search for a deal connected to "Company A",
you may get a search hit for "Company A", if the deals label is set to
Working with the search indexer¶
The search indexer is interacted with via limefu
Connection details are configured as described here.
# config.yaml search_indexer: log_level: INFO
Exclude lime types¶
From lime-crm version 2.413.0, you can configure your application so that certain lime types are not indexed.
This can become very useful if a certain lime type has a huge number of records but
does not need to be searchable, i.e. indexed 1. Lime types such as
invoicerow might exist in the quantities of millions, due to integrations.
By excluding such lime types from being indexed, the index becomes smaller,
thus making a re-index much faster.
Add this section to your Config in CAFE (cloud) or application_config.yaml (on-prem) to exclude certain lime types.
invoicerow with the lime types you wish to exclude.
search: exclude_limetypes: - participant - invoicerow
Managing Indexes in Elasticsearch 7¶
With the introduction of Elasticsearch version 7 in Lime CRM's 2023.1 Hvannadalsnjúkur release, index management within Lime CRM has undergone significant changes.
In prior versions, a single active index contained all application data. However, this model has been updated in the latest release. Here's an overview of the revised structure:
The concept of an active index has been phased out. Now, a single application
can possess multiple indexes, all of which can be searched simultaneously. Upon
the creation of a new index, it is assigned an alias that corresponds to the
application name (e.g.,
app). Concurrently, the
is_write_index flag is set
Should an existing index already be associated with the application, the
is_write_index flag for that index will be reset to
false. This alteration
enables the use of the index alias in lieu of the index name when operating
with the index. New data is directed to the most recent index (where
true), while searches can be conducted across all
This modification facilitates simultaneous searching and updating of the index while a re-index is underway. Under the old structure, a re-index would discard any updates made during the process as writes were directed to two distinct indexes, and the old index was discarded post-re-indexing.
If a document is updated, it is transferred to the new index. However, the previous indexes may still house the old, now-outdated, data. To manage this issue, any document added to the new index is removed from the old indexes.
When a new index is established, the preceding indexes are granted a new alias
__old suffix, e.g.,
app__old. This system simplifies the
process of removing documents from old indexes, ensuring that no out-of-date
data remains searchable.
To illustrate these changes, consider an application named
app with the
following indexes and aliases:
ab123-appassociated with aliases:
cd456-appassociated with alias:
In this scenario, data searches within the application will use the
Data writing operations also use the
app alias, with the
being the targeted destination since its
is_write_index is set to
the app alias.
app__old alias serves as a marker for identifying old indexes. This
identification enables the removal of stale data from the associated index,
ensuring data relevancy and accuracy.
Please note that these records will still show up in lists, tables, reports and such. They will only be excluded from features where Elastic is used, such as The Global Search and relation pickers. ↩