Project

General

Profile

CdmLibraryFreetextSearch » History » Version 5

Andreas Kohlbecker, 01/24/2012 01:50 PM

1 1 Andreas Kohlbecker
2
# EDIT - Freetext Search
3
4
5
 [Hibernate Search](http://www.hibernate.org/subprojects/search.html)  brings the power of full text search engines to the persistence domain model by combining Hibernate Core with the capabilities of the [Apache Lucene](http://lucene.apache.org/) search engine. Therefore it looks like a good idea to use Hibernate Search in the CDM Library to perform free text searches. Due to the architecture of some parts of the EDIT platform there are some caveats and problems which have to be considered carefully before deciding on making full use of Hibernate Search / Lucene in the CDM Library:
6
7
8
9
## Benefits
10
11
12 2 Andreas Kohlbecker
With plain HQL it is possible to search for example for text snippets contained in TextData.multilanguageText: LIKE '%ext snip%'. But the situation gets a bit more complex when taking a look at some specific use cases, like for example the following tickets: [[#842|implement advanced search]], [[#424|ALL. Image search]], [[#2432|Implement search for multiple areas]]. For example the image search would require to perform a LIKE search over the following fields simultaneously. The performance of this query would not be the best.
13 1 Andreas Kohlbecker
14
* Media.title
15
16
* Media.description
17
18
* Media.representations.parts.uri 
19
20
* Description.title
21
22
* Description.taxon
23
24
* Description.elements.multilanguageText
25
26
* Description.elements.name
27
28
29
Hibernate search / Lucene allows to build documents which combine multiple fields which are distributed in the object graph. These docuemts are indexed and thus are serchable in a very quickly without the need to join multiple tables. Further benefits over plain hql are:
30
31
32
* normalization 
33
34
  * lowercase/uppercase - 'lactuca' finds 'Lactuca'
35
36
  * unicode (diacritics) - 'Angstrom' finds 'Ångström' 
37
38
  * removing special characters from words - 'donalds' finds 'donald's'
39
40
* real term based free text search over a phrase based search with wildcards as '*term_1 te*rm_2 ter*' in HQL
41
42
* can speed up existing find*() methods in the CDM Library
43 2 Andreas Kohlbecker
44
* Lucene can handle [spacial searches](http://wiki.apache.org/lucene-java/SpatialSearch)
45
46
* retrieve information from the Lucene index (titelCache, UUID, etc,) without the need to initialize any CDM entity
47
48
* ...
49 1 Andreas Kohlbecker
50
51
52
## Open questions
53
54
55
1. is the index for the type A always updated when an associated object D like in A.B.C.D has been changed?
56
57
58
59
60
## Projects which require Hibernate search
61
62
63 3 Andreas Kohlbecker
* Vibrant: Task 2 - CDM Datastore as a ViBRANT Index ( ... allows humans to perform full text searches ...) **Due 30 July 2012** 
64 1 Andreas Kohlbecker
65
66
67 5 Andreas Kohlbecker
----
68 3 Andreas Kohlbecker
69 1 Andreas Kohlbecker
70
71 5 Andreas Kohlbecker
## Problems
72 1 Andreas Kohlbecker
73 5 Andreas Kohlbecker
74
75 4 Andreas Kohlbecker
### Lucene index in the local filesystem
76 1 Andreas Kohlbecker
77 3 Andreas Kohlbecker
Lucene is storing the index in the file system, whereas the data is stored in the database. Usually multiple Taxonomic Editors are connected directly to a central database. When a cdm entity is updated by an editor the new data is stored in the central database, but the lucene index exists in the local file system where the editor is installed, thus the central index is not updated. 
78 4 Andreas Kohlbecker
79
Hibernate Search registers an update listener which triggers the indexing of inserted and updated entities.
80 1 Andreas Kohlbecker
81
82
83 5 Andreas Kohlbecker
## Solutions
84
85
86
All solutions are only appicable if one database server + cdmlib-remote instance is defined as the central repository.
87
88
89
Solutions which target at the Lucene indexing:
90
91
 
92
1. Implement a custom HibernateUpdateListener which sends a notification to a web service, which then in turn triggers the indexing process in the same way as the local update listener
93
94
1. Could Lucene clustering be a solution?
95
96
1. Provide a central index into which all Lucene instances are writing. e.g. a webdav share ?? - Conflicts?
97
98
1. any further ideas?
99
100
101
Solutions on the editor side:
102
103
104
1. Fully implement the HTTP Invoker remoting solution for the editor. There are some major problems to be solved for this solution: **@Niels: please could you name the existing problems here?** 
105
106
1. Fully implement the RAP solution for the editor. There are some major problems to be solved for this solution: **@Niels: please could you name the existing problems here?**