Project

General

Profile

feature request #6321

Implement parallel loading + progress bar for loading objects into the bulk editor

Added by Patrick Plitzner almost 2 years ago. Updated 4 months ago.

Status:
Closed
Priority:
Priority14
Category:
taxeditor
Target version:
Start date:
01/10/2017
Due date:
% Done:

60%

Estimated time:
8.00 h
Severity:
critical
Tags:

Related issues

Related to Edit - feature request #7463: Move taxonnodes should be available for multiple selection Closed 06/07/2018
Related to Edit - feature request #7439: Use NatTable for BulkEditor Closed 05/28/2018
Related to Edit - bug #7505: IllegalStateException: Multiple representations problem when adding Annotation to name in NameBulkEditor Closed 06/21/2018
Related to Edit - feature request #7661: Improve saving strategy for bulk editor New 08/17/2018

Associated revisions

Revision 27a07b9c (diff)
Added by Patrick Plitzner 6 months ago

fix #6321 Implement parallel loading + progress bar for bulk editor

Revision 8b4e88a6 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Avoid multiple parallel searches for same bulk editor

Revision 59277452 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Fix search service invocation for users and groups

Revision 82e85f0d (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Fix open authority handler + no multiple editor for same input

Revision 611b3a85 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Re-enable search count warning for derivate view

Revision 1e1a1052 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Cancel previous search when starting a new one

Revision 1e892f7f (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Invoke search query with default order by title cache

Revision bc04b70c (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Show search string in progress bar

Revision fd28a17f (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Show search progress in progress bar

Revision 3da52db5 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 i18n for search progress in progress bar

Revision 1f8befae (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Make OrderHint serializable

Revision 264fff91 (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Use individual page size for different editor inputs

Revision 2b7b435a (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 show progress bar in dialog

Revision fa79586c (diff)
Added by Patrick Plitzner 6 months ago

ref #6321 Add preference to run long operations in background

Revision 28c922f7 (diff)
Added by Patrick Plitzner 6 months ago

ref #7505, #6321 Disable parallel loading

  • This caused problems with the session handling because the CDM entities were not correctly loaded

Revision e7314de8 (diff)
Added by Patrick Plitzner 6 months ago

ref #7505, #6321 Enable max count check for bulk editor searches

Revision fbdeccd7 (diff)
Added by Patrick Plitzner 6 months ago

ref #7505, #6321 Update UI after bulk editor search + remove page size

Revision 36ee2736 (diff)
Added by Patrick Plitzner 4 months ago

ref #6321 Wait for canceled search to finish before starting new search

Revision 88ef5f8b (diff)
Added by Patrick Plitzner 4 months ago

ref #6321 Wait for previous search to finish

Revision 66e51de3 (diff)
Added by Patrick Plitzner 4 months ago

ref #6321 Load asynchronously loaded entities into main session

History

#1 Updated by Andreas Müller almost 2 years ago

Currently we have a warning if more than 500(?) objects are going to be loaded into an bulk editor. Instead we should have a progress monitor which allows canceling and shows how long it will take. Also the loading may take place in partitions or via a stream (new ticket?) or even in parallel (Java 8 stream API?) to show partial results and to avoid Out-of-Memory exceptions.
This is less relevant for authors (loading is fast here) but more for references and names.

#2 Updated by Andreas Müller almost 2 years ago

  • Priority changed from New to Priority14
  • Target version changed from Unassigned CDM tickets to Release 4.7
  • Estimated time set to 8.00 h

#3 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.7 to Release 4.8

#4 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.8 to Release 4.9

#5 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.9 to Release 4.10

#6 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.10 to Release 4.12

#7 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.12 to Release 4.13

#8 Updated by Andreas Müller 11 months ago

  • Target version changed from Release 4.13 to Release 4.14

#9 Updated by Andreas Müller 10 months ago

  • Target version changed from Release 4.14 to Release 5.0

#10 Updated by Patrick Plitzner 7 months ago

  • Target version changed from Release 5.0 to Release 5.1

#11 Updated by Patrick Plitzner 6 months ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 50

#12 Updated by Patrick Plitzner 6 months ago

  • Assignee changed from Patrick Plitzner to Andreas Müller

I am currently using a page size of 20 which leads to a quicker refresh of the bulk editor. But of course this will produce more server queries. But I assume that after having seen the first 100results the user is either satisfied and hits Cancel or searches again with a refined query.

Please check performance of paging with size=20 when reviewing.

#13 Updated by Andreas Müller 6 months ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Patrick Plitzner

When trying to open the progress monitor during loading I got the following exception:

login : admin
editor version : 5.1.0.201806142303
server : test.e-taxonomy.eu (edit-test) / rem_conf_am
schema version : 5.0.0.0.20180514
os : Windows Server 2012 R2 6.3 amd64
java : 1.8.0_121
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://test.e-taxonomy.eu:80/cdmserver/rem_conf_am/remoting/progressmonitor.service]; nested exception is java.io.EOFException
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:216)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:147)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy55.getRemotingMonitor(Unknown Source)
    at eu.etaxonomy.taxeditor.util.ProgressMonitorClientManager.pollMonitor(ProgressMonitorClientManager.java:84)
    at eu.etaxonomy.taxeditor.util.ProgressMonitorClientManager.pollMonitor(ProgressMonitorClientManager.java:58)
    at eu.etaxonomy.taxeditor.model.AbstractUtility$3.run(AbstractUtility.java:695)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2903)
    at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:2185)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2159)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at java.util.HashSet.readObject(HashSet.java:333)
    at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.doReadRemoteInvocationResult(AbstractHttpInvokerRequestExecutor.java:292)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.readRemoteInvocationResult(AbstractHttpInvokerRequestExecutor.java:243)
    at org.springframework.remoting.httpinvoker.HttpComponentsHttpInvokerRequestExecutor.doExecuteRequest(HttpComponentsHttpInvokerRequestExecutor.java:232)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:138)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:194)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:176)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:144)
    ... 7 more

The same happened with the progress monitor in #7463#note-6 (happens sometimes, no pattern recognized yet).
Maybe we need to open a new ticket as this seems to be a general issue with the progress monitor.

#14 Updated by Andreas Müller 6 months ago

#15 Updated by Andreas Müller 6 months ago

The loading should take place sorted so the new records are added to the end. Currently one does not really understand what happens until the load is completed. If sorted, new records will always be added to the end (so only the scroll bar will change a bit, this is the typical/expected behavior). Also, if load is canceled one does not know, which records are already loaded.

#16 Updated by Andreas Müller 6 months ago

The progress bar does not show up as dialog so the ordinary user does not know how to cancel a running load.

#17 Updated by Andreas Müller 6 months ago

The progress bar (in progress view) does not show percentage so one does not know how long it will still take to fully load the data. Also a message like xxx/yyyy records (names/references/authors/...) loaded.

#18 Updated by Andreas Müller 6 months ago

Starting a new search while the old one is still running creates a warning "Wait until last search has finished or cancel it".

This dialog should have buttons "Cancel new search", "Cancel last search" (maybe you have an idea for better labels).
The expected default behavior is to cancel the last search as there is no reason for running a new search if you don't want to stop the old one. Often this may happen if one realizes that the search was too unspecific.
We may even think about a "remember my decision" preference as one may always want to stop previous searches once running a new one.

#19 Updated by Andreas Müller 6 months ago

The progress monitor may show the pattern to search for. Currently it only shows something like "Load names". Better show "Load names for 'A*'" or something similar.

#20 Updated by Andreas Müller 6 months ago

I think we should increase the number of records per page. 50, 100 and/or 200 seems reasonable. Probably this should depend on the performance of the class. E.g. authors may have 200 records as loading is fast. References may have 100 and names 50.

#21 Updated by Andreas Müller 6 months ago

Andreas Müller wrote:

The loading should take place sorted so the new records are added to the end. Currently one does not really understand what happens until the load is completed. If sorted, new records will always be added to the end (so only the scroll bar will change a bit, this is the typical/expected behavior). Also, if load is canceled one does not know, which records are already loaded.

The sorting is also very important for re-search which takes place e.g. after save. Now it is very obvious that the data is fully reloaded, but shouldn't.

#22 Updated by Andreas Müller 6 months ago

Generally we need to think about reloading. For large datasets this might not be a good strategy as it takes too long, however reloading is the safest way to avoid session issues.

#23 Updated by Andreas Müller 6 months ago

  • % Done changed from 50 to 30

Critical: setting focus does not work after reloading.

E.g. after saving the focus is lost. Don't know if this is related to loading strategy (this ticket) or to the new NAT table (#7439).

#24 Updated by Andreas Müller 6 months ago

#25 Updated by Patrick Plitzner 6 months ago

Comment #6321#note-13: I tried opening the progress view several times and never saw this exception.
Comment #6321#note-15: Search results are ordered by title cache
Comment #6321#note-16: Progress bar shows up as dialog. Preference for long "run in background" added to taxeditor general preferences
Comment #6321#note-17: search progress is shown
Comment #6321#note-18: Now the previous search is just canceled. I also think this is the most common behavior. Preferences or dialogs are too much overload in this case.
Comment #6321#note-19: search query is shown
Comment #6321#note-20: Every editor input can specify its individual page size

#26 Updated by Patrick Plitzner 6 months ago

Andreas Müller wrote:

Critical: setting focus does not work after reloading.

E.g. after saving the focus is lost. Don't know if this is related to loading strategy (this ticket) or to the new NAT table (#7439).

I don't understand the problem of the focus loss. What were you trying to do and what did not work?

#27 Updated by Patrick Plitzner 6 months ago

  • Status changed from Feedback to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Müller

#28 Updated by Andreas Müller 6 months ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Patrick Plitzner

Patrick Plitzner wrote:

Andreas Müller wrote:

Critical: setting focus does not work after reloading.

E.g. after saving the focus is lost. Don't know if this is related to loading strategy (this ticket) or to the new NAT table (#7439).

I don't understand the problem of the focus loss. What were you trying to do and what did not work?

As a user, if you save your work, you generally don't expect that any state of your UI changes (except for the dirty state). Or do you expect in MS Word that you jump back to first page if you save your Word document. Saving should be minimal interrupting the standard workflow as possible. Especially as we ask the users to save as often as possible.

Example: You want to go through a list of 100 names and check something (e.g. identifiers). You find some identifier missing. You add it. Save (because you should always save immediately) and want to go to the next name. But what is your next name? The focus is lost and you may not know.

#29 Updated by Andreas Müller 6 months ago

Even worse than the focus loss is that we jump back to first record. This is definitely not acceptable.

#30 Updated by Patrick Plitzner 6 months ago

  • Assignee changed from Patrick Plitzner to Andreas Müller

Ah, ok. This is a misunderstanding of the terms we used. For me (and code-wise), the focus refers to a view or editor i.e. the whole bulk editor in this case. What you are talking about is the selection.

And this definitely is a problem which happens because of the re-searching. But we can reset the selection after the re-search has finished.

#31 Updated by Andreas Müller 6 months ago

  • Tags set to euro+med

See also #7439#note-29 for sorting with pre-sorted columns.

#32 Updated by Andreas Müller 6 months ago

Patrick Plitzner wrote:

Ah, ok. This is a misunderstanding of the terms we used. For me (and code-wise), the focus refers to a view or editor i.e. the whole bulk editor in this case. What you are talking about is the selection.

OK, I understand. Other UI frameworks allow to distinguish focus and selection even within a view. E.g. focus is on a single row (and makes this row editable, while selection is on a multi-select, or a row is selected but does not have the focus, so it has a right click menu but can not be edited). But it looks like RCP or NAT does not support this.

#33 Updated by Andreas Müller 6 months ago

Patrick Plitzner wrote:

And this definitely is a problem which happens because of the re-searching. But we can reset the selection after the re-search has finished.

Can't we do the re-select as soon as the record is loaded? This way the user can start working as soon as possible. Otherwise, all the parallel loading doesn't make so much sense.
We could even think about first loading the selected record and then load all the rest.

#34 Updated by Patrick Plitzner 6 months ago

  • Related to bug #7505: IllegalStateException: Multiple representations problem when adding Annotation to name in NameBulkEditor added

#35 Updated by Patrick Plitzner 6 months ago

Parallel loading is temporarily deactivated. Fix for #7505 and update problem in name details view

#36 Updated by Andreas Müller 6 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Andreas Müller to Patrick Plitzner
  • Target version changed from Release 5.1 to Release 5.2

Move this to next milestone until it is reactivated

#37 Updated by Patrick Plitzner 4 months ago

  • Status changed from In Progress to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Müller

Parallel loading is reactivated. I could not find any bugs until during random tests

#38 Updated by Andreas Müller 4 months ago

Did you change anything or did you simply reactivate?

#39 Updated by Patrick Plitzner 4 months ago

I just reactivated and tested a bit. I made some small changes I think.

#40 Updated by Andreas Müller 4 months ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Patrick Plitzner

When running a search and stopping it by running a new search the results of the first search are still (partly) available in the result of the second search. This can be reproduced by e.g. running a search on names with "" and restart a search with "R" afterwards in a large database (e.g. rem_conf_am).

It looks like the first search still delivers a couple of results before really being finished.

As we do not have a progress dialog but only the progressbar running a new search might be the preferred way to stop a long running search with a more specific one. Therefore this relevant (but not really critical as "only" more records are displayed than expected)

#41 Updated by Andreas Müller 4 months ago

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

!ENTRY org.eclipse.e4.ui.workbench 4 0 2018-08-10 20:58:01.277
!MESSAGE 
!STACK 0
eu.etaxonomy.taxeditor.remoting.CdmEagerLoadingException: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://test.e-taxonomy.eu:80/cdmserver/rem_conf_am/remoting/common.service]; nested exception is org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.hibernate.collection.internal.AbstractPersistentCollection.remoteInitialize(AbstractPersistentCollection.java:1358)
    at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:627)
    at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:167)
    at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:184)
    at org.hibernate.collection.internal.PersistentSet.size(PersistentSet.java:143)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.setSectionTitle(AbstractEntityCollectionSection.java:178)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.internalUpdateSection(AbstractEntityCollectionSection.java:204)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.setEntity(AbstractEntityCollectionSection.java:165)
    at eu.etaxonomy.taxeditor.view.detail.CdmSectionPart.setFormInput(CdmSectionPart.java:155)
    at org.eclipse.ui.forms.ManagedForm.setInput(ManagedForm.java:210)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmDataViewerE4.refresh(AbstractCdmDataViewerE4.java:161)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmDataViewerE4.setInput(AbstractCdmDataViewerE4.java:143)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmEditorPartE4.showViewer(AbstractCdmEditorPartE4.java:225)
    at eu.etaxonomy.taxeditor.view.e4.supplementaldata.SupplementalDataPartE4.selectionChanged_internal(SupplementalDataPartE4.java:121)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmEditorPartE4$DelaySelection.run(AbstractCdmEditorPartE4.java:85)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:182)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4211)
    at org.eclipse.swt.widgets.Display.msgFilterProc(Display.java:3541)
    at org.eclipse.swt.internal.win32.OS.DefWindowProcW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DefWindowProc(OS.java:2547)
    at org.eclipse.swt.widgets.Scrollable.callWindowProc(Scrollable.java:88)
    at org.eclipse.swt.widgets.Composite.WM_SYSCOMMAND(Composite.java:1864)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:4877)
    at org.eclipse.swt.widgets.Canvas.windowProc(Canvas.java:359)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:5110)
    at org.eclipse.swt.internal.win32.OS.DefWindowProcW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DefWindowProc(OS.java:2547)
    at org.eclipse.swt.widgets.Scrollable.callWindowProc(Scrollable.java:88)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:4897)
    at org.eclipse.swt.widgets.Canvas.windowProc(Canvas.java:359)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:5123)
    at org.eclipse.swt.internal.win32.OS.DispatchMessageW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DispatchMessage(OS.java:2552)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3822)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1121)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1022)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:150)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:693)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:610)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
    at eu.etaxonomy.taxeditor.Application.start(Application.java:24)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
Caused by: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://test.e-taxonomy.eu:80/cdmserver/rem_conf_am/remoting/common.service]; nested exception is org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:216)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:147)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy49.initializeCollection(Unknown Source)
    at eu.etaxonomy.taxeditor.service.CachedCommonServiceImpl.initializeCollection(CachedCommonServiceImpl.java:61)
    at org.hibernate.collection.internal.AbstractPersistentCollection.remoteInitialize(AbstractPersistentCollection.java:1340)
    ... 55 more
Caused by: org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.springframework.remoting.httpinvoker.HttpComponentsHttpInvokerRequestExecutor.executeHttpPost(HttpComponentsHttpInvokerRequestExecutor.java:338)
    at org.springframework.remoting.httpinvoker.HttpComponentsHttpInvokerRequestExecutor.doExecuteRequest(HttpComponentsHttpInvokerRequestExecutor.java:229)
    at eu.etaxonomy.taxeditor.service.CdmServiceRequestExecutor.doExecuteRequest(CdmServiceRequestExecutor.java:61)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:138)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:194)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:176)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:144)
    ... 60 more

#42 Updated by Andreas Müller 4 months ago

  • Severity changed from normal to critical

In name bulk editor changing e.g. specific epithet does NOT change the labels for "Name" and "Scientific name" and also "Name Cache" is not changed as well as the value in the bulk editor (titleCache). However, the value in "Cache" is changed. (No protected caches).

Don't know if this is related to parallel loading but might be as listeners on CDM objects may be created in other threads. Similar things happened in reference bulk editor. Did not check all bulk editors.

This is critical

#43 Updated by Andreas Müller 4 months ago

Reloading after save and setting selection after save if selection is one of the later records in a huge dataset is still an open issue as you may need to wait long time until you can continue working. Sometimes re-selection even does not work at all.

Generally reloading strategy needs to be evaluated. Hopefully there is a better solution which may load only changed records. This is related to transaction handling and we may want to open a new ticket for this.

However, did you test if it is possible to first load the selected record (if it is still in the search result) and then load all the rest. This might be an acceptable temporary solution.

#44 Updated by Andreas Müller 4 months ago

Also got an NPE when deleting (and saving the delete) of a Reference. Maybe this is also related to parallel loading? Mention just in case. See separate mail.

#45 Updated by Patrick Plitzner 4 months ago

Andreas Müller wrote:

When running a search and stopping it by running a new search the results of the first search are still (partly) available in the result of the second search. This can be reproduced by e.g. running a search on names with "" and restart a search with "R" afterwards in a large database (e.g. rem_conf_am).

It looks like the first search still delivers a couple of results before really being finished.

As we do not have a progress dialog but only the progressbar running a new search might be the preferred way to stop a long running search with a more specific one. Therefore this relevant (but not really critical as "only" more records are displayed than expected)

This is fixed now. I use a volatile boolean with a job change listener to wait for the canceled job to finish completely before starting the new search.

#46 Updated by Patrick Plitzner 4 months ago

Andreas Müller wrote:

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

Looking at #7634 Walter already mentioned the same exception and it occured with the stable version on production. I could also (not deterministically) reproduce it on cdm_rem_conf_am. Could this be the same problem that both servers, test and production, share?

#47 Updated by Andreas Müller 4 months ago

Patrick Plitzner wrote:

Andreas Müller wrote:

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

Looking at #7634 Walter already mentioned the same exception and it occured with the stable version on production. I could also (not deterministically) reproduce it on cdm_rem_conf_am. Could this be the same problem that both servers, test and production, share?

I can't really see that the issue reported in #7634 is related to this one as Walter was doing something completely different. Of course NoHttpResponseException appear sometimes and in different context. But it looks like it appears relatively often with parallel bulk loading, something that was not reported for Non-parallel bulk loading. So we need to check if parallel bulk loading creates specific problems. Have you checked the log-files?

#48 Updated by Patrick Plitzner 4 months ago

Comment #6321#note-40: I just added a delay of 500ms to wait for the previous job to finish. Waiting explicitily is very error prone for accidental endless loops during development and this is not a critical issue.
Comment #6321#note-41 #6321#note-42: The Cdm entities needed to be loaded into the main session during the parallel loading. The editing in the name details view is fixed. I also did not experience and CdmEagerLoadingException any more. This could have been related.
Comment #6321#note-43: This is fixed

#49 Updated by Patrick Plitzner 4 months ago

  • Status changed from Feedback to Resolved
  • Assignee changed from Patrick Plitzner to Andreas Müller

#50 Updated by Andreas Müller 4 months ago

  • Status changed from Resolved to Feedback
  • Assignee changed from Andreas Müller to Patrick Plitzner

I still get exceptions related to NoHttpResponseException on test server. E.g. when running a first request for "*" on names on rem_conf_am.

Maybe this is because the test server is generally very slow at the moment. But we need to further test this.

#51 Updated by Andreas Müller 4 months ago

Patrick Plitzner wrote:

Comment #6321#note-40: I just added a delay of 500ms to wait for the previous job to finish. Waiting explicitily is very error prone for accidental endless loops during development and this is not a critical issue.

Ok, this seems to work (though it is a bit dirty)

#52 Updated by Andreas Müller 4 months ago

Patrick Plitzner wrote:

Comment #6321#note-43: This is fixed

OK, this looks quite good. I think we can only improve it by changing the save strategy which I created a new ticket for #7661

#53 Updated by Andreas Müller 4 months ago

  • % Done changed from 30 to 60

Generally I wonder how to continue here. As mentioned I still get HTTPNoResponse issues on "test" when running large queries. For small queries it usually works.
Also unexpected is that before these exceptions the TaxEditor becomes unresponsive. As I thought that the queries run in a non UI thread I wouldn't expect this.

However, we may want to open a new ticket (with high priority) and close this one. Maybe behavior on production is different.

#54 Updated by Andreas Müller 4 months ago

#55 Updated by Patrick Plitzner 4 months ago

  • Status changed from Feedback to Closed

Andreas Müller wrote:

Generally I wonder how to continue here. As mentioned I still get HTTPNoResponse issues on "test" when running large queries. For small queries it usually works.
Also unexpected is that before these exceptions the TaxEditor becomes unresponsive. As I thought that the queries run in a non UI thread I wouldn't expect this.

However, we may want to open a new ticket (with high priority) and close this one. Maybe behavior on production is different.

I opened a new ticket for this issue #7667

I will close this ticket now.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)