task #8651
Unify description aggregation methods (distribution and structured descriptive data)
70%
Description
Currentyl we have at least 2 methods/classes handling description aggregation, 1 for distribution data called TransmissionEngineDistribution, 1 for structured descriptive data currently handled in DescriptionDataSetService.
Both do similar things like aggregating descriptive data first on taxon level and in a second step aggregating descriptive data to higher rank taxa, flagging the created descriptions with flag "aggregated" and deleting older aggregation when aggregation is running anew.
Therefore both algorithms may use same base functionality and may learn from each other. Also the handling as long running task should be unified.
Related issues
Associated revisions
ref #8651 first base classes for unified description aggregation and move SDD aggregation from service to new class
ref #8651 fix compile error on linux
ref #8651 fix compile error on linux (2nd try)
ref #8651 minor changes to TransmissionEngineDistribution and moving test
ref #8651 fix compile error on linux (3rd try)
ref #8651 rename TransmissionEngineDistribution to DistributionAggregation
ref #8651 remove @Service annotation from DistributionAggregation and moving some functionality to base class + explicit Configurator for DistributionAggregation
ref #8651 try to fix compile error in DescriptionListController
ref #8651 try to fix failing repository dependency injection in DescriptionListController
ref #8651 try to fix failing repository dependency injection in DescriptionListController (2nd try)
ref #8651 remove unnecessary dependency DistributionAggregation in DescriptionElementListController
ref #8651 more specific description type for DistributionAggregation
ref #8651 merge area and rank aggregation for DistributionAggregation
ref #8651 fix failing tests due to use of old description markers
ref #8651 remove some comments and handle AggregationMode correctly
ref #8651 fix failing tests due to order of aggregation
ref #8651 remove MarkerType.Computed where possible
ref #8651 Use unified aggregation class for matrix aggregation
ref #8651 use explicit conifguration for structured descriptions
ref #8651 unify invocation call in LongRunninTasksService
ref #8651 factory method and base constructor for aggregation configurators
ref #8651 register UUID lists for distribution accumulation webservice
ref #8651 smaller changes to distribution accumulation webservice
ref #8651 Adapt to cdmlib aggregation configuration
ref #8651 always add description type to aggregated descriptions
ref #8651 Remove aggregation methods(now in DescriptionAggregationBase)
ref #8651 Introduce new description type "Aggregated Structured Description"
ref #8651 fix missing uuid
ref #8651 cast failing in LINUX
ref #8651 Adapt to finer grained aggregation description type
ref #8651 cleanup DistributionAggregation
ref #8651 further cleanup DistributionAggregation
ref #8651 further cleanup DistributionAggregation
ref #8651 implement reuse of distributions in DistributionAggregation
ref #8651 cleanup DistributionAggregation and remove old computed markers and avoid not saving of descriptions due to readonly
ref #8651 useClearExistingDistribution = false as default
ref #8651 adapt EuroMedActivator to changes in DistributionAggregation/TransmissionEngine
ref #8651 adapt EuroMedActivator to changes in DistributionAggregation/TransmissionEngine
History
#1 Updated by Andreas Müller about 2 years ago
- Subject changed from Unify description aggregation methods to Unify description aggregation methods (distribution and structured descriptive data)
#2 Updated by Andreas Müller about 2 years ago
- Related to bug #8670: Unwanted distribution aggregation/display on 2 levels added
#3 Updated by Andreas Müller about 2 years ago
- Related to task #8671: Distribution in E+M (BM) on different levels added
#4 Updated by Andreas Müller about 2 years ago
- Related to feature request #5176: TransmissionEngineDistribution can run on selected taxa added
#5 Updated by Andreas Müller about 2 years ago
- Status changed from New to In Progress
- Priority changed from New to Highest
- Target version changed from Release 5.12 to Release 5.11
- % Done changed from 0 to 40
This is basically done but there are mutliple open tasks
#6 Updated by Andreas Müller about 2 years ago
- Precedes task #8679: Further unify description aggregation methods added
#7 Updated by Andreas Müller about 2 years ago
- Status changed from In Progress to Resolved
- Assignee changed from Andreas Müller to Andreas Kohlbecker
- % Done changed from 40 to 70
We do now have a common base class and also the LongRunningTaskService used only 1 method (with config as parameter).
Further work is to be done and will be handled in #8679.
AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.
Before closing we should delete the redundant methods from DescriptiveDataSetService. PP can you do this?
#8 Updated by Patrick Plitzner about 2 years ago
Andreas Müller wrote:
Before closing we should delete the redundant methods from DescriptiveDataSetService. PP can you do this?
redundant methods are deleted.
#9 Updated by Andreas Kohlbecker about 2 years ago
- Assignee changed from Andreas Kohlbecker to Andreas Müller
Andreas Müller wrote:
AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.
I tried to review the changes but it turned out to be quite time consuming, since a profound review would require having an in depth understanding of all the new requirements and decisions that have been made. I am having no objections against closing this issue as long as the result of the aggregation conforms to the expectations as as long as the performance is good.
#10 Updated by Andreas Müller about 2 years ago
- Assignee changed from Andreas Müller to Andreas Kohlbecker
Andreas Kohlbecker wrote:
Andreas Müller wrote:
AK and PP as you have developed the orginal classes maybe you want to have a look to the new implementation. Afterwards you may close this ticket.
I tried to review the changes but it turned out to be quite time consuming, since a profound review would require having an in depth understanding of all the new requirements and decisions that have been made. I am having no objections against closing this issue as long as the result of the aggregation conforms to the expectations as as long as the performance is good.
I agree that it is difficult to review. I only wanted to link you to the changes in case you are interested.
The requirements did not really change, tests have not been changed (and helped me a lot). Only the algorithm is more generic now and also subtree aggregation is now possible (required a lot of changes).
I think we can close this ticket. There is a follow up: #8679
#11 Updated by Andreas Müller about 2 years ago
- Status changed from Resolved to Closed
- Assignee changed from Andreas Kohlbecker to Andreas Müller
#12 Updated by Andreas Müller 9 months ago
- Related to feature request #2625: [E+M Overview] Data aggregation functionalities for E+M (TransmissionEngine) added
#13 Updated by Andreas Müller 9 months ago
- Related to bug #8312: Test, fix, improve and run Transmission engine for E+M added
