Project

General

Profile

Actions

Freetext Search aka Hibernate Search in the CDM Library

TODO what is this site about


Hibernate Search brings the power of full text search engines to the persistence domain model by combining Hibernate Core with the capabilities of the Apache Lucene search engine. Therefore it looks like a good idea to use Hibernate Search in the CDM Library to perform free text searches. Due to the architecture of some parts of the EDIT platform there are some caveats and problems which have to be considered carefully before deciding on making full use of Hibernate Search / Lucene in the CDM Library:

Benefits

With plain HQL it is possible to search for example for text snippets contained in TextData.multilanguageText: LIKE '%ext snip%'. But the situation gets a bit more complex when taking a look at some specific use cases, like for example the following tickets: implement advanced search, ALL. Image search, Implement search for multiple areas. For example the image search would require to perform a LIKE search over the following fields simultaneously. The performance of this query would not be the best.

  • Media.title

  • Media.description

  • Media.representations.parts.uri

  • Description.title

  • Description.taxon

  • Description.elements.multilanguageText

  • Description.elements.name

Hibernate search / Lucene allows to build documents which combine multiple fields which are distributed in the object graph. These docuemts are indexed and thus are serchable in a very quickly without the need to join multiple tables. Further benefits over plain hql are:

  • normalization

    • lowercase/uppercase - 'lactuca' finds 'Lactuca'
    • unicode (diacritics) - 'Angstrom' finds 'Ångström'
    • removing special characters from words - 'donalds' finds 'donald's'
  • real term based free text search over a phrase based search with wildcards as 'term_1 te*rm_2 ter' in HQL

  • can speed up existing find*() methods in the CDM Library

  • Lucene can handle spacial searches

  • retrieve information from the Lucene index (titelCache, UUID, etc,) without the need to initialize any CDM entity

  • retrieve lucene documents together with associated cdm entities, this is very nice if we for example search for taxa based on scientific and on common names,

we want to return not only the matching taxa associated with the common name but also the common name itself.

  • ...

Existing implementations

  • Hibernate search base configuration is done, individual index location for each instance in the CDM Server

  • Indexing works

  • findDescriptionElements() is based on hibernate search

  • findTaxaByDescriptionElements() is based on hibernate search over 10 fold performance increase over plain HQL (performance benchmark in TaxonServiceSearchTest.java )

  • retrieve lucene documents together with associated cdm entities (see above)

Open questions

  1. is the index for the type A always updated when an associated object D like in A.B.C.D has been changed?

Projects and tasks which require Hibernate search

  • some ticket can only be solved satisfyingly if we use hibernate search:

    • #842 (ALL. Image search)
    • #424 (Implement advanced search)
    • #2432 (Implement search for multiple areas)
    • ...

Problems

Lucene index in the local filesystem

Lucene is storing the index in the file system, whereas the data is stored in the database. Usually multiple Taxonomic Editors are connected directly to a central database. When a cdm entity is updated by an editor the new data is stored in the central database, but the lucene index exists in the local file system where the editor is installed, thus the central index is not updated.

Hibernate Search registers an update listener which triggers the indexing of inserted and updated entities.

Solutions

All solutions are only appicable if one database server + cdmlib-remote instance is defined as the central repository.

Solutions which target at the Lucene indexing:

  1. Implement a custom HibernateUpdateListener which sends a notification to a web service, which then in turn triggers the indexing process in the same way as the local update listener

  2. Could Lucene clustering be a solution?

  3. Provide a central index into which all Lucene instances are writing. e.g. a webdav share ?? - Conflicts?

  4. any further ideas?

Solutions on the editor side:

  1. Fully implement the HTTP Invoker remoting solution for the editor. There are some major problems to be solved for this solution: @Niels: please could you name the existing problems here?

  2. Fully implement the RAP solution for the editor. There are some major problems to be solved for this solution: @Niels: please could you name the existing problems here?

Updated by Andreas Müller about 1 year ago · 18 revisions