bug #4962: improve portal/classification/{taxon_uuid}/childNodesAt/{rank_uuid} web service - EDIT - EDIT Project Management

Custom queries

5.33
cdm-dataportal (open)
last 7 days tickets
mine in current release
phycobank
phycobank-index-closed
phycobank-registration-closed
potentially missed issues
recently updated
Release3.7
Release3.8
Release4.0
Release4.1
Release4.10
Release4.11
Release4.12
Release4.13
Release4.14
Release4.16
Release4.2
Release4.3
Release4.4
Release4.5
Release4.6
Release4.7
Release4.8
Release4.9
Release4.9
Release5.0
Release5.1
Release5.10
Release5.11
Release5.12
Release5.13
Release5.14
Release5.15
Release5.16
Release5.17
Release5.18
Release5.19
Release5.2
Release5.20
Release5.21
Release5.22
Release5.23
Release5.24
Release5.25
Release5.26
Release5.27
Release5.28
Release5.29
Release5.3
Release5.30
Release5.31
Release5.32
Release5.33
Release5.34
Release5.35
Release5.36
Release5.37
Release5.38
Release5.39
Release5.4
Release5.40
Release5.41
Release5.5
Release5.6
Release5.7
Release5.8
Release5.9
Remoting-Versions
Taxeditor-UX (Platform Etablierung)

Actions

Copy link

bug #4962

open

task #3336: Improve overall dataportal page loading performance

improve portal/classification/{taxon_uuid}/childNodesAt/{rank_uuid} web service

Added by Andreas Kohlbecker almost 9 years ago. Updated about 2 months ago.

Status:

Resolved

Priority:

New

Assignee:

Andreas Müller

Category:

cdmlib

Target version:

Release 5.43

Start date:

Due date:

% Done:

Estimated time:

Severity:

normal

Found in Version:

Tags:

performance faunaEuropaea

Description

the @portal/classification/{taxon_uuid}/childNodesAt/{rank_uuid} @-service is one of the slowest service endpoints.

for example in the case of Bellis perennis: [classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6/portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6] it takes about 2,5 seconds on a cdmserver with 22 instances(see #4925)

with fauna europaea this web service is extremely slow:

http://cdm.biodiv.naturkundemuseum-berlin.de/cdm/cdmfauneur2/portal/classification/e66c33cc-47d8-4215-9946-7790d0d4889e/childNodesAt/b0785a65-c1c1-4eb4-88c7-dbd3df5aaad1.json about 15s as of 2015 July 14th

this is a problem in cdmlib-services or in cdmlib-persistence.

for documentation on the behaviour of this method see http://dev.e-taxonomy.eu/trac/wiki/CdmClassificationRankSpecificRootnodes

performance improvements that have been applied in the past to this method

f989e26 avoiding LEFT OUTER JOIN by splitting query in two
ba84f3f performance improvement for listChildNodes of a taxo

Related issues

Related to EDIT - bug #7045: portal/classification/{uuid}/childNodesOf/{uuid} service slow

Feedback

Katja Luther

Actions

Related to EDIT - feature request #9271: Improve E+M dataportal performance

In Progress

Andreas Müller

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Andreas Kohlbecker almost 9 years ago

Keywords changed from performance, to performance, FaunaEuropaea,

Actions

Copy link

Updated by Andreas Kohlbecker over 8 years ago

results of more detailed performance measures. In the following log the time (ms) of specific parts of the whole process are printed out, in the dao method two queries are run sequentially to fetch all nodes:

service.listRankSpecificRootNodes() : total time of the service method, this includes the subprocesses
- dao.listRankSpecificRootNodes() - query[x].list() : time of the first or second queries
- dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() : execution of the bean initializer
create JSON Object 392 : time taken by the JSONView for serialization

2015-11-19 22:59:50,975 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 1275
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 4840
dao.listRankSpecificRootNodes() - query[1].list() 1101
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 9426
create JSON Object 392
2015-11-19 23:00:14,704 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 937
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 4067
dao.listRankSpecificRootNodes() - query[1].list() 1156
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 8285
create JSON Object 190
2015-11-19 23:00:37,157 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 932
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 3338
dao.listRankSpecificRootNodes() - query[1].list() 1098
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 7424
create JSON Object 93

The bean initializer clearly dominates in these results. Therefore the next steps to improve this should be to

reduce or avoid the use of the initialization strategy in the controller.
let the Dao method return the TaxonNodeDTO directly instead of constructing it in the view. This DTO class is by the way named TaxonNodeDao which is wrong and thus should be changed.

Actions

Copy link

Updated by Andreas Kohlbecker over 8 years ago

Status changed from New to Resolved
Target version changed from cdmlib - Next Major Release to Release 3.12

Improvements implemented:

f36a078 [r26575 commenting performance measure print-out via Systemerr|* 256c8f8 r26574] avoiding bean initialization in getChildNodesAtRank() as much as possible - performance improvement

The attempt to avoid the use of an initialization strategy by adding JOIN FETCH statements to the hql statements in the dao but keeping the default initialization strategy '$' already leads to a significant speed up:

before

7030.257 ms|* 7965 [ms]

after

2898.261 ms|* 5120 [ms]

further improvements can not be applied. First of all because of the name cache strategies which require a big part of the object graph to be loaded, secondly because of the TaxonNodeComparator which also needs detailed and atomized information on the taxon name and the authorship cache. I created a new ticket for this major topic #5387

Actions

Copy link

Updated by Andreas Kohlbecker over 8 years ago

I think we can close this ticket since no further improvement is possible without solving #5387

Actions

Copy link

Updated by Andreas Kohlbecker over 8 years ago

the commits which removed the init strategy introduced a LIE, so i am re-adding the init strategy for taxonNames: 77db7b5 r26928

Actions

Copy link

Updated by Andreas Kohlbecker over 8 years ago

2b1209a r26947 : using LEFT FETCH JOIN for ranks and sec fixed problems and improves performance (all times in ms):

tested with euro+med_

##before
2016-01-11 10:55:55,975 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 252
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 455
dao.listRankSpecificRootNodes() - query[1].list() 1
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 66
service.listRankSpecificRootNodes() 808

## Using LEFT JOIN FETCH for n.rank and t.sec
2016-01-11 10:57:38,471 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 279
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 340
dao.listRankSpecificRootNodes() - query[1].list() 1
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 45
service.listRankSpecificRootNodes() 695

tested with fauna europaea:

##before
2016-01-11 11:02:07,901 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/7d85033b-2f6b-49d3-a9b7-073f2756f78d/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 2747
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 1215
dao.listRankSpecificRootNodes() - query[1].list() 1811
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 211
service.listRankSpecificRootNodes() 10016

## Using LEFT JOIN FETCH for n.rank and t.sec
2016-01-11 11:03:24,429 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/7d85033b-2f6b-49d3-a9b7-073f2756f78d/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 2847
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 958
dao.listRankSpecificRootNodes() - query[1].list() 1834
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 175
service.listRankSpecificRootNodes() 5854

Actions

Copy link