Project

General

Profile

feature request #8096

Implement CacheUpdater as long running task with correct progress monitoring

Added by Andreas Müller 10 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Priority14
Assignee:
Category:
cdmlib-remote
Target version:
Start date:
02/13/2019
Due date:
% Done:

100%

Severity:
normal

picture638-1.png View (30.4 KB) Patrick Plitzner, 04/03/2019 01:41 PM

picture686-1.png View (4.43 KB) Andreas Müller, 04/04/2019 11:13 AM

picture787-1.png View (1.63 KB) Andreas Müller, 04/04/2019 11:17 AM

picture455-1.png View (21 KB) Katja Luther, 04/05/2019 10:09 AM

picture115-1.png View (2.55 KB) Andreas Müller, 04/08/2019 05:48 PM


Related issues

Related to Edit - feature request #8030: Cache updater service methods externalized and base class specific Closed 01/29/2019
Related to Edit - feature request #8232: Show updated objects of cache updater separately New 04/09/2019
Copied to Edit - bug #8237: Cache updating sometimes does not work as expected for references and names New 04/10/2019

Associated revisions

Revision c01a0dfe (diff)
Added by Katja Luther 10 months ago

ref #8096: restructure repair menu and use longrunningTaskService for repair functionality

Revision 112ab4f7 (diff)
Added by Katja Luther 10 months ago

ref #8096: move cache updater and service methods return updateResult

Revision 95371079 (diff)
Added by Katja Luther 10 months ago

ref #8096: move cache updater and service methods return updateResult -continue

Revision 0da5fc18 (diff)
Added by Katja Luther 10 months ago

ref #8096: delete old cache updater and adapt cache updater to moved terms

Revision 66172879 (diff)
Added by Katja Luther 10 months ago

ref #8096: termservice adapted

Revision d2deb7ca (diff)
Added by Katja Luther 9 months ago

ref #8096: use subprogressmonitor for cache updater

Revision 4478751c (diff)
Added by Katja Luther 9 months ago

ref #8096: set correct task name in progress monitor manager

Revision 72631511 (diff)
Added by Katja Luther 9 months ago

ref #8096: minor

Revision e627dccd (diff)
Added by Katja Luther 9 months ago

ref #8096: subtask name is the same as task name

Revision 428a2961 (diff)
Added by Katja Luther 9 months ago

ref #8096: deduplicate message in progress bar

Revision 1a1cc8f8 (diff)
Added by Katja Luther 8 months ago

ref #8096: smaller changes for update caches

Revision 4624cd0d (diff)
Added by Katja Luther 8 months ago

ref #8096: remove doubled colon

Revision aa20ae50 (diff)
Added by Katja Luther 8 months ago

ref #8096: move cache updater and service methods return updateResult

Revision d173c335 (diff)
Added by Katja Luther 8 months ago

ref #8096: move cache updater and service methods return updateResult -continue

Revision aa7a50d3 (diff)
Added by Katja Luther 8 months ago

ref #8096: delete old cache updater and adapt cache updater to moved terms

Revision 4ce3ad94 (diff)
Added by Katja Luther 8 months ago

ref #8096: termservice adapted

Revision 2462bed3 (diff)
Added by Katja Luther 8 months ago

ref #8096: use subprogressmonitor for cache updater

Revision 69e38fad (diff)
Added by Katja Luther 8 months ago

ref #8096: minor

Revision 3412cfed (diff)
Added by Katja Luther 8 months ago

ref #8096: smaller changes for update caches

History

#1 Updated by Andreas Müller 10 months ago

  • Related to feature request #8030: Cache updater service methods externalized and base class specific added

#2 Updated by Katja Luther 10 months ago

  • Status changed from New to In Progress
  • Priority changed from New to Priority14

#3 Updated by Katja Luther 9 months ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Katja Luther to Andreas Müller

please review

#4 Updated by Katja Luther 9 months ago

  • % Done changed from 0 to 60

#5 Updated by Katja Luther 9 months ago

  • Assignee changed from Andreas Müller to Patrick Plitzner

#6 Updated by Patrick Plitzner 9 months ago

  • File picture638-1.png View added
  • Status changed from Resolved to Feedback
  • Assignee changed from Patrick Plitzner to Katja Luther

I tested updating caches with all options on on the lates nightly. The progress bar shows the current work done and the progress increases over time. I did not check if the caches were really updated as this functionality was already implemented.

One minor issue:

It still says "No subtask name" if you do not expand the details. This text should also be set. Can be the same as the text in the details.

#7 Updated by Katja Luther 9 months ago

  • Status changed from Feedback to Resolved
  • Assignee changed from Katja Luther to Andreas Müller

Patrick Plitzner wrote:

I tested updating caches with all options on on the lates nightly. The progress bar shows the current work done and the progress increases over time. I did not check if the caches were really updated as this functionality was already implemented.

One minor issue:

It still says "No subtask name" if you do not expand the details. This text should also be set. Can be the same as the text in the details.

this is fixed.

#8 Updated by Andreas Müller 9 months ago

  • File picture686-1.png View added
  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Katja Luther

I get a strange duplication of task and progress now:

#9 Updated by Andreas Müller 9 months ago

When running on references (in rem_conf_am) I only get no task shown:

#10 Updated by Andreas Müller 9 months ago

Also when rerunning on references I always get the same result (updated objects 2669), second time I would expect 0.

Can you check if there is still an error in the algorithm?

The same happens for taxon names (always 6 updated objects).

Please also check for taxa and specimens. I havn't tested yet.

#11 Updated by Katja Luther 8 months ago

Andreas Müller wrote:

When running on references (in rem_conf_am) I only get no task shown:

now it looks like this:

#12 Updated by Katja Luther 8 months ago

Andreas Müller wrote:

Also when rerunning on references I always get the same result (updated objects 2669), second time I would expect 0.

Can you check if there is still an error in the algorithm?

The same happens for taxon names (always 6 updated objects).

Please also check for taxa and specimens. I havn't tested yet.

Maybe it is the same issue as for teams we found a few weeks ago. I will have a look.

#13 Updated by Katja Luther 8 months ago

Katja Luther wrote:

Andreas Müller wrote:

Also when rerunning on references I always get the same result (updated objects 2669), second time I would expect 0.

Can you check if there is still an error in the algorithm?

The same happens for taxon names (always 6 updated objects).

Please also check for taxa and specimens. I havn't tested yet.

Maybe it is the same issue as for teams we found a few weeks ago. I will have a look.

It is a problem of the cache update:

if (this.protectedAbbrevTitleCache == false){
           String oldAbbrevTitleCache = this.abbrevTitleCache;

           String newAbbrevTitleCache = cacheStrategy.getFullAbbrevTitleString(this);

           if ( oldAbbrevTitleCache == null   || ! oldAbbrevTitleCache.equals(newAbbrevTitleCache) ){
                this.setAbbrevTitleCache(null, false);
                String newCache = this.getAbbrevTitleCache();

                if (newCache == null){
                    logger.warn("New abbrevCache should never be null");
                }
                if (oldAbbrevTitleCache == null){
                    logger.info("oldAbbrevTitleCache was illegaly null and has been fixed");
                }
                result = true;
            }
        }
        return result;

first newAbbrevTitleCache is created by cacheStrategy.getFullAbbrevTitleString(this) and compared to the abbrevTitleCache of the reference and then the new cache is created by this.getAbbrevTitleCache(); and the strings are not always the same.

#14 Updated by Katja Luther 8 months ago

the problem for references is that if cacheStrategy.getFullAbbrevTitleString(this) returns null, the abbrevTitleCache is set to titleCache and then the comparison fails because the old abbrevTitleCache is the titleCache and the computed abbrevTitleCache is null.

#15 Updated by Katja Luther 8 months ago

  • Status changed from Feedback to Resolved

there are problems with objects having a null titleCache because they get a new titlecache while saving.
I think we should keep it like it is, because this can show the user dirty data because objects without titlecache should not exist.

#16 Updated by Katja Luther 8 months ago

  • Assignee changed from Katja Luther to Andreas Müller

#17 Updated by Andreas Müller 8 months ago

Katja Luther wrote:

Andreas Müller wrote:

When running on references (in rem_conf_am) I only get no task shown:

now it looks like this:

Much better now, but I get 2 colons in status list now:

(not serious)

#18 Updated by Andreas Müller 8 months ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Katja Luther

If running on multiple classes the update result returns the total number of updated objects. Better we had them separated by classes, e.g. 10 taxon names updated, 5 authors/teams updated, ...

But this may go into a new ticket if not easy to fix.

#19 Updated by Andreas Müller 8 months ago

Katja Luther wrote:

the problem for references is that if cacheStrategy.getFullAbbrevTitleString(this) returns null, the abbrevTitleCache is set to titleCache and then the comparison fails because the old abbrevTitleCache is the titleCache and the computed abbrevTitleCache is null.

Hm, in current rem_conf_am are 19 references with empty abbrevTitleCache but 40 objects (references) are updated according to the report.

I think we should further investigate here how to best handle this issue. However, this is another ticket. Can you create a new ticket for this? If the 2 above minor issues are fixed or moved to a "remaining" issue we can close this ticket.

#20 Updated by Katja Luther 8 months ago

Andreas Müller wrote:

Katja Luther wrote:

the problem for references is that if cacheStrategy.getFullAbbrevTitleString(this) returns null, the abbrevTitleCache is set to titleCache and then the comparison fails because the old abbrevTitleCache is the titleCache and the computed abbrevTitleCache is null.

Hm, in current rem_conf_am are 19 references with empty abbrevTitleCache but 40 objects (references) are updated according to the report.

I think we should further investigate here how to best handle this issue. However, this is another ticket. Can you create a new ticket for this? If the 2 above minor issues are fixed or moved to a "remaining" issue we can close this ticket.

The other references are references where the titleCache itself is null and during save/merge the titleCache is created by the CdmPreDataChangeListener

if (identifiableEntity.hasEmptyTitleCache()){
                    identifiableEntity.setTitleCache(identifiableEntity.toString(), false);
                }

#21 Updated by Katja Luther 8 months ago

Andreas Müller wrote:

If running on multiple classes the update result returns the total number of updated objects. Better we had them separated by classes, e.g. 10 taxon names updated, 5 authors/teams updated, ...

But this may go into a new ticket if not easy to fix.

This needs a separation of the different kind of objects because actually all updated objects are in one list. If we want to release soon, I would create a new ticket for this issue.(-> #8232)

#22 Updated by Katja Luther 8 months ago

  • Status changed from Feedback to Resolved
  • Assignee changed from Katja Luther to Andreas Müller

the two colons are fixed and for a better result there is a new ticket.

#23 Updated by Andreas Müller 8 months ago

#24 Updated by Andreas Müller 8 months ago

  • Status changed from Resolved to Closed
  • Assignee changed from Andreas Müller to Katja Luther

Moved null/empty cache issues to new ticket #8237.

All the rest works or has new ticket so we can close this one.

#25 Updated by Andreas Müller 8 months ago

  • % Done changed from 60 to 100

#26 Updated by Andreas Müller 8 months ago

  • Copied to bug #8237: Cache updating sometimes does not work as expected for references and names added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)