Project

General

Profile

task #8651

Unify description aggregation methods (distribution and structured descriptive data)

Added by Andreas Müller about 1 month ago. Updated 27 days ago.

Status:
Closed
Priority:
Highest
Category:
cdmlib
Target version:
Start date:
11/05/2019
Due date:
% Done:

70%

Severity:
normal

Description

Currentyl we have at least 2 methods/classes handling description aggregation, 1 for distribution data called TransmissionEngineDistribution, 1 for structured descriptive data currently handled in DescriptionDataSetService.

Both do similar things like aggregating descriptive data first on taxon level and in a second step aggregating descriptive data to higher rank taxa, flagging the created descriptions with flag "aggregated" and deleting older aggregation when aggregation is running anew.

Therefore both algorithms may use same base functionality and may learn from each other. Also the handling as long running task should be unified.


Related issues

Related to Edit - bug #8670: Unwanted distribution aggregation/display on 2 levels Resolved 11/09/2019
Related to Edit - task #8671: Distribution in E+M (BM) on different levels In Progress 11/09/2019
Related to Edit - feature request #5176: TransmissionEngineDistribution can run on selected taxa Closed 09/01/2015
Precedes Edit - task #8679: Further unify description aggregation methods In Progress 11/13/2019

Associated revisions

Revision 7c5cb694 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 first base classes for unified description aggregation and move SDD aggregation from service to new class

Revision 8b518049 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 fix compile error on linux

Revision 753627ae (diff)
Added by Andreas Müller about 1 month ago

ref #8651 fix compile error on linux (2nd try)

Revision b997cab9 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 minor changes to TransmissionEngineDistribution and moving test

Revision ac2cee48 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 fix compile error on linux (3rd try)

Revision 7ef0f93e (diff)
Added by Andreas Müller about 1 month ago

ref #8651 rename TransmissionEngineDistribution to DistributionAggregation

Revision 3bab771f (diff)
Added by Andreas Müller about 1 month ago

ref #8651 remove @Service annotation from DistributionAggregation and moving some functionality to base class + explicit Configurator for DistributionAggregation

Revision 8bbb7d89 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 try to fix compile error in DescriptionListController

Revision 8049524f (diff)
Added by Andreas Müller about 1 month ago

ref #8651 try to fix failing repository dependency injection in DescriptionListController

Revision d68e1b58 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 try to fix failing repository dependency injection in DescriptionListController (2nd try)

Revision 3aa37a25 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 remove unnecessary dependency DistributionAggregation in DescriptionElementListController

Revision 6be3adaa (diff)
Added by Andreas Müller about 1 month ago

ref #5176, ref #8651 make DistributionAggregation treeIndex based and use TaxonNodeFilter

Revision 5c148b4e (diff)
Added by Andreas Müller about 1 month ago

ref #8651 more specific description type for DistributionAggregation

Revision 7c03912b (diff)
Added by Andreas Müller about 1 month ago

ref #8651 merge area and rank aggregation for DistributionAggregation

Revision d062baaf (diff)
Added by Andreas Müller about 1 month ago

ref #8651 fix failing tests due to use of old description markers

Revision 3a97a754 (diff)
Added by Andreas Müller about 1 month ago

ref #8651 remove some comments and handle AggregationMode correctly

Revision 291ea3f3 (diff)
Added by Andreas Müller 30 days ago

ref #8651 fix failing tests due to order of aggregation

Revision 00fafe2a (diff)
Added by Andreas Müller 30 days ago

ref #8651 remove MarkerType.Computed where possible

Revision 46278434 (diff)
Added by Patrick Plitzner 30 days ago

ref #8651, #8467 Use unified aggregation class for matrix aggregation

  • Link specimen descriptions as sources as direct CDM entities

Revision 7f601c27 (diff)
Added by Patrick Plitzner 30 days ago

ref #8651 Use unified aggregation class for matrix aggregation

Revision a2dd1526 (diff)
Added by Andreas Müller 29 days ago

ref #8651 use explicit conifguration for structured descriptions

Revision a51edae3 (diff)
Added by Andreas Müller 29 days ago

ref #8651 unify invocation call in LongRunninTasksService

Revision c655943d (diff)
Added by Andreas Müller 29 days ago

ref #8651 factory method and base constructor for aggregation configurators

Revision 9d192006 (diff)
Added by Andreas Müller 29 days ago

ref #8651 register UUID lists for distribution accumulation webservice

Revision 58ea183c (diff)
Added by Andreas Müller 29 days ago

ref #8651 smaller changes to distribution accumulation webservice

Revision 4eedd13b (diff)
Added by Patrick Plitzner 29 days ago

ref #8651 Adapt to cdmlib aggregation configuration

Revision d1292203 (diff)
Added by Andreas Müller 29 days ago

ref #8651 always add description type to aggregated descriptions

Revision c68519c0 (diff)
Added by Patrick Plitzner 29 days ago

ref #8651 Remove aggregation methods(now in DescriptionAggregationBase)

Revision a914faf1 (diff)
Added by Andreas Müller 28 days ago

ref #8651 Introduce new description type "Aggregated Structured Description"

Revision f24fb983 (diff)
Added by Andreas Müller 28 days ago

ref #8651 fix missing uuid

Revision a7df73f0 (diff)
Added by Andreas Müller 28 days ago

ref #8651 cast failing in LINUX

Revision 3fa32caf (diff)
Added by Patrick Plitzner 28 days ago

ref #8651 Adapt to finer grained aggregation description type

Revision ec44caf9 (diff)
Added by Andreas Müller 28 days ago

ref #8651 cleanup DistributionAggregation

Revision 0cfec87b (diff)
Added by Andreas Müller 28 days ago

ref #8651 further cleanup DistributionAggregation

Revision 85884141 (diff)
Added by Andreas Müller 28 days ago

ref #8651 further cleanup DistributionAggregation

Revision c1ede4f8 (diff)
Added by Andreas Müller 28 days ago

ref #8651 implement reuse of distributions in DistributionAggregation

Revision 91006ade (diff)
Added by Andreas Müller 28 days ago

ref #8651 cleanup DistributionAggregation and remove old computed markers and avoid not saving of descriptions due to readonly

Revision 9449fa3d (diff)
Added by Andreas Müller 28 days ago

ref #8651 useClearExistingDistribution = false as default

Revision ed98102a (diff)
Added by Andreas Müller 27 days ago

ref #8651 adapt EuroMedActivator to changes in DistributionAggregation/TransmissionEngine

Revision 38969108 (diff)
Added by Andreas Müller 24 days ago

ref #8651 adapt EuroMedActivator to changes in DistributionAggregation/TransmissionEngine

History

#1 Updated by Andreas Müller about 1 month ago

  • Subject changed from Unify description aggregation methods to Unify description aggregation methods (distribution and structured descriptive data)

#2 Updated by Andreas Müller about 1 month ago

  • Related to bug #8670: Unwanted distribution aggregation/display on 2 levels added

#3 Updated by Andreas Müller about 1 month ago

  • Related to task #8671: Distribution in E+M (BM) on different levels added

#4 Updated by Andreas Müller about 1 month ago

#5 Updated by Andreas Müller 29 days ago

  • Status changed from New to In Progress
  • Priority changed from New to Highest
  • Target version changed from Release 5.12 to Release 5.11
  • % Done changed from 0 to 40

This is basically done but there are mutliple open tasks

#6 Updated by Andreas Müller 29 days ago

  • Precedes task #8679: Further unify description aggregation methods added

#7 Updated by Andreas Müller 29 days ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Andreas Müller to Andreas Kohlbecker
  • % Done changed from 40 to 70

We do now have a common base class and also the LongRunningTaskService used only 1 method (with config as parameter).

Further work is to be done and will be handled in #8679.

AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.

Before closing we should delete the redundant methods from DescriptiveDataSetService. PP can you do this?

#8 Updated by Patrick Plitzner 29 days ago

Andreas Müller wrote:

Before closing we should delete the redundant methods from DescriptiveDataSetService. PP can you do this?

redundant methods are deleted.

#9 Updated by Andreas Kohlbecker 27 days ago

  • Assignee changed from Andreas Kohlbecker to Andreas Müller

Andreas Müller wrote:

AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.

I tried to review the changes but it turned out to be quite time consuming, since a profound review would require having an in depth understanding of all the new requirements and decisions that have been made. I am having no objections against closing this issue as long as the result of the aggregation conforms to the expectations as as long as the performance is good.

#10 Updated by Andreas Müller 27 days ago

  • Assignee changed from Andreas Müller to Andreas Kohlbecker

Andreas Kohlbecker wrote:

Andreas Müller wrote:

AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.

I tried to review the changes but it turned out to be quite time consuming, since a profound review would require having an in depth understanding of all the new requirements and decisions that have been made. I am having no objections against closing this issue as long as the result of the aggregation conforms to the expectations as as long as the performance is good.

I agree that it is difficult to review. I only wanted to link you to the changes in case you are interested.
The requirements did not really change, tests have not been changed (and helped me a lot). Only the algorithm is more generic now and also subtree aggregation is now possible (required a lot of changes).

I think we can close this ticket. There is a follow up: #8679

#11 Updated by Andreas Müller 27 days ago

  • Status changed from Resolved to Closed
  • Assignee changed from Andreas Kohlbecker to Andreas Müller

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)