Project

General

Profile

CdmLibraryFreetextSearch » History » Version 1

Andreas Kohlbecker, 01/24/2012 12:54 PM

1 1 Andreas Kohlbecker
2
# EDIT - Freetext Search
3
4
5
 [Hibernate Search](http://www.hibernate.org/subprojects/search.html)  brings the power of full text search engines to the persistence domain model by combining Hibernate Core with the capabilities of the [Apache Lucene](http://lucene.apache.org/) search engine. Therefore it looks like a good idea to use Hibernate Search in the CDM Library to perform free text searches. Due to the architecture of some parts of the EDIT platform there are some caveats and problems which have to be considered carefully before deciding on making full use of Hibernate Search / Lucene in the CDM Library:
6
7
8
9
## Benefits
10
11
12
With plain HQL it is possible to search for example for text snippets contained in TextData.multilanguageText: LIKE '%ext snip%'. But the situation gets a bit more complex when taking a look at some specific use cases, like for example the following tickets: [[#842|implement advanced search]], [[#424|ALL. Image search[#2432 Implement search for multiple areas]]. For example the image search would require to perform a LIKE search over the following fields simultaneously. The performance of this query would not be the best.
13
14
* Media.title
15
16
* Media.description
17
18
* Media.representations.parts.uri 
19
20
* Description.title
21
22
* Description.taxon
23
24
* Description.elements.multilanguageText
25
26
* Description.elements.name
27
28
29
Hibernate search / Lucene allows to build documents which combine multiple fields which are distributed in the object graph. These docuemts are indexed and thus are serchable in a very quickly without the need to join multiple tables. Further benefits over plain hql are:
30
31
32
* normalization 
33
34
  * lowercase/uppercase - 'lactuca' finds 'Lactuca'
35
36
  * unicode (diacritics) - 'Angstrom' finds 'Ångström' 
37
38
  * removing special characters from words - 'donalds' finds 'donald's'
39
40
* real term based free text search over a phrase based search with wildcards as '*term_1 te*rm_2 ter*' in HQL
41
42
* can speed up existing find*() methods in the CDM Library
43
44
45
46
## Open questions
47
48
49
1. is the index for the type A always updated when an associated object D like in A.B.C.D has been changed?
50
51
52
53
54
## Projects which require Hibernate search
55
56
57
* Vibrant: Task 2 - CDM Datastore as a ViBRANT Index ( ... allows humans to perform full text searches ...)
58
59
60
61
62
## Problems
63
64
65
66
## Side effects
67
68
69
70
## Solutions