Refer to this doc if you're only interested in indexing a new field of an existing entity
Modeling is the most important and crucial part of your design.
Search document model contains a list of fields that need to be indexed along with the names and their data types.
Check here to learn more about search document model.
Please note that all fields in the search document model (except the urn) are optional.
This is because we want to support partial updates to search documents.
A mapping is created using the information of search document model.
It defines how a document, and the fields it contains, are stored and indexed by various [tokenizers], [analyzers] and data type for the fields.
For certain fields, sub-fields are created using different analyzers.
The analyzers are chosen depending on the needs for each field.
This is currently created manually using curl commands, and we plan to automate the process in the near future.
Check index mappings & settings for dataset search index.
The actual indexing process for each entity is powered by [index builders]. The builders register the metadata [aspects] of their interest against MAE Consumer Job and will be invoked whenever an MAE of same interest is received. Index builders should be extended from BaseIndexBuilder. Check DatasetIndexBuilder as an example. For the consumer job to consume those MAEs, you should add your index builder to the index builder registry.
Once you have the search index built, it's ready to be queried! The search query is constructed and executed through Search DAO. The raw search hits are retrieved and extracted using the base model. Besides the regular full text search, run time aggregation and relevance are provided in the search queries as well.
ESSearchDAO is the implementation for the BaseSearchDAO for Elasticsearch. It's still a generic class which can be used for a specific entity and configured using BaseSearchConfig.
BaseSearchConfig is the abstraction for all query related configurations such as query templates, default field to execute autocomplete on etc.
public abstract class BaseSearchConfig<DOCUMENT extends RecordTemplate> {
public abstract Set<String> getFacetFields();
public String getIndexName() {
return getSearchDocument().getSimpleName().toLowerCase();
}
public abstract Class<DOCUMENT> getSearchDocument();
public abstract String getDefaultAutocompleteField();
public abstract String getSearchQueryTemplate();
public abstract String getAutocompleteQueryTemplate();
}DatasetSearchConfig is the implementation of search config for dataset entity.
Search query templates for various entities can be found here.
Finally, you need to create rest.li APIs to serve your search queries. BaseSearchableEntityResource provides an abstract implementation of search and autocomplete APIs. Any top level rest.li resource implementation could extend it and easily add search and autocomplete ACTION methods. Refer to CorpUsers rest.li resource implementation as an example.