Project

General

Profile

Actions
Copy link

bug #4962

open

task #3336: Improve overall dataportal page loading performance

improve portal/classification/{taxon_uuid}/childNodesAt/{rank_uuid} web service

Added by Andreas Kohlbecker almost 9 years ago. Updated about 2 months ago.

Status:
Resolved
Priority:
New
Assignee:
Andreas Müller
Category:
cdmlib
Target version:
Release 5.43
Start date:
Due date:
% Done:

0%

Estimated time:
Severity:
normal
Found in Version:
Tags:
performance faunaEuropaea

Description

the @portal/classification/{taxon_uuid}/childNodesAt/{rank_uuid} @-service is one of the slowest service endpoints.

for example in the case of Bellis perennis: [classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6/portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6] it takes about 2,5 seconds on a cdmserver with 22 instances(see #4925)

with fauna europaea this web service is extremely slow:

http://cdm.biodiv.naturkundemuseum-berlin.de/cdm/cdmfauneur2/portal/classification/e66c33cc-47d8-4215-9946-7790d0d4889e/childNodesAt/b0785a65-c1c1-4eb4-88c7-dbd3df5aaad1.json about 15s as of 2015 July 14th

this is a problem in cdmlib-services or in cdmlib-persistence.

for documentation on the behaviour of this method see http://dev.e-taxonomy.eu/trac/wiki/CdmClassificationRankSpecificRootnodes

performance improvements that have been applied in the past to this method

  • f989e26 avoiding LEFT OUTER JOIN by splitting query in two

  • ba84f3f performance improvement for listChildNodes of a taxo

Related issues

Related to EDIT - bug #7045: portal/classification/{uuid}/childNodesOf/{uuid} service slowFeedbackKatja Luther

Actions
Related to EDIT - feature request #9271: Improve E+M dataportal performanceIn ProgressAndreas Müller

Actions
Actions
Copy link
 #1

Updated by Andreas Kohlbecker almost 9 years ago

  • Keywords changed from performance, to performance, FaunaEuropaea,
Actions
Copy link
 #2

Updated by Andreas Kohlbecker over 8 years ago

results of more detailed performance measures. In the following log the time (ms) of specific parts of the whole process are printed out, in the dao method two queries are run sequentially to fetch all nodes:

  • service.listRankSpecificRootNodes() : total time of the service method, this includes the subprocesses

    • dao.listRankSpecificRootNodes() - query[x].list() : time of the first or second queries
    • dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() : execution of the bean initializer

  • create JSON Object 392 : time taken by the JSONView for serialization

2015-11-19 22:59:50,975 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 1275
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 4840
dao.listRankSpecificRootNodes() - query[1].list() 1101
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 9426
create JSON Object 392
2015-11-19 23:00:14,704 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 937
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 4067
dao.listRankSpecificRootNodes() - query[1].list() 1156
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 8285
create JSON Object 190
2015-11-19 23:00:37,157 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/f65ce83e-3856-40ad-b8e2-ba046f984f07/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 932
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 3338
dao.listRankSpecificRootNodes() - query[1].list() 1098
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.listRankSpecificRootNodes() 7424
create JSON Object 93

The bean initializer clearly dominates in these results. Therefore the next steps to improve this should be to

  1. reduce or avoid the use of the initialization strategy in the controller.

  2. let the Dao method return the TaxonNodeDTO directly instead of constructing it in the view. This DTO class is by the way named TaxonNodeDao which is wrong and thus should be changed.

Actions
Copy link
 #3

Updated by Andreas Kohlbecker over 8 years ago

  • Status changed from New to Resolved
  • Target version changed from cdmlib - Next Major Release to Release 3.12

Improvements implemented:

  • f36a078 [r26575 commenting performance measure print-out via Systemerr|* 256c8f8 r26574] avoiding bean initialization in getChildNodesAtRank() as much as possible - performance improvement

The attempt to avoid the use of an initialization strategy by adding JOIN FETCH statements to the hql statements in the dao but keeping the default initialization strategy '$' already leads to a significant speed up:

before

  • 7030.257 ms|* 7965 [ms]

after

  • 2898.261 ms|* 5120 [ms]

further improvements can not be applied. First of all because of the name cache strategies which require a big part of the object graph to be loaded, secondly because of the TaxonNodeComparator which also needs detailed and atomized information on the taxon name and the authorship cache. I created a new ticket for this major topic #5387

Actions
Copy link
 #4

Updated by Andreas Kohlbecker over 8 years ago

I think we can close this ticket since no further improvement is possible without solving #5387

Actions
Copy link
 #5

Updated by Andreas Kohlbecker over 8 years ago

the commits which removed the init strategy introduced a LIE, so i am re-adding the init strategy for taxonNames: 77db7b5 r26928

Actions
Copy link
 #6

Updated by Andreas Kohlbecker over 8 years ago

2b1209a r26947 : using LEFT FETCH JOIN for ranks and sec fixed problems and improves performance (all times in ms):

tested with euro+med_ 

##before
2016-01-11 10:55:55,975 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 252
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 455
dao.listRankSpecificRootNodes() - query[1].list() 1
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 66
service.listRankSpecificRootNodes() 808

## Using LEFT JOIN FETCH for n.rank and t.sec
2016-01-11 10:57:38,471 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/314a68f9-8449-495a-91c2-92fde8bcf344/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 279
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 340
dao.listRankSpecificRootNodes() - query[1].list() 1
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 45
service.listRankSpecificRootNodes() 695

tested with fauna europaea: 

##before
2016-01-11 11:02:07,901 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/7d85033b-2f6b-49d3-a9b7-073f2756f78d/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 2747
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 1215
dao.listRankSpecificRootNodes() - query[1].list() 1811
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 211
service.listRankSpecificRootNodes() 10016

## Using LEFT JOIN FETCH for n.rank and t.sec
2016-01-11 11:03:24,429 INFO [eu.etaxonomy.cdm.remote.controller.ClassificationPortalListController] - getChildNodesAtRank() /portal/classification/7d85033b-2f6b-49d3-a9b7-073f2756f78d/childNodesAt/af5f2481-3192-403f-ae65-7c957a0f02b6.json
dao.listRankSpecificRootNodes() - query[0].list() 2847
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 958
dao.listRankSpecificRootNodes() - query[1].list() 1834
dao.listRankSpecificRootNodes() - defaultBeanInitializer.initializeAll() 0
service.pageRankSpecificRootNodes() - Collections.sort(results,  taxonNodeComparator) 175
service.listRankSpecificRootNodes() 5854
Actions
Copy link
 #7

Updated by Andreas Kohlbecker over 6 years ago

  • Related to bug #7045: portal/classification/{uuid}/childNodesOf/{uuid} service slow added
Actions
Copy link
 #8

Updated by Andreas Kohlbecker over 6 years ago

  • Description updated (diff)
  • Private changed from Yes to No
Actions
Copy link
 #9

Updated by Andreas Müller over 3 years ago

Actions
Copy link
 #10

Updated by Andreas Müller about 2 years ago

  • Target version changed from Release 3.12 to Release 5.44

Please review and move back to 3.12 when done.

Also check if new implementation is needed (follow up ticket)

Actions
Copy link
 #11

Updated by Andreas Müller about 2 years ago

See also #7045 which indicates that the problem is not fully solved yet.

Actions
Copy link
 #12

Updated by Belen Escobari about 1 year ago

  • Target version changed from Release 5.44 to Release 5.37
Actions
Copy link
 #13

Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 5.37 to Release 5.44
Actions
Copy link
 #14

Updated by Andreas Müller 2 months ago

  • Target version changed from Release 5.44 to Release 5.43
Actions
Copy link
 #15

Updated by Andreas Müller about 2 months ago

Andreas Müller wrote in #note-11:

See also #7045 which indicates that the problem is not fully solved yet.

Not sure if this is true. #7045 is about another method (childnodes of nodes in the tree/sibblings of nodes in the tree path).

Actions
Copy link

Also available in: Atom PDF