EDIT

Currently we have a warning if more than 500(?) objects are going to be loaded into an bulk editor. Instead we should have a progress monitor which allows canceling and shows how long it will take. Also the loading may take place in partitions or via a stream (new ticket?) or even in parallel (Java 8 stream API?) to show partial results and to avoid Out-of-Memory exceptions.
This is less relevant for authors (loading is fast here) but more for references and names.

The loading should take place sorted so the new records are added to the end. Currently one does not really understand what happens until the load is completed. If sorted, new records will always be added to the end (so only the scroll bar will change a bit, this is the typical/expected behavior). Also, if load is canceled one does not know, which records are already loaded.

The progress bar does not show up as dialog so the ordinary user does not know how to cancel a running load.

The progress bar (in progress view) does not show percentage so one does not know how long it will still take to fully load the data. Also a message like xxx/yyyy records (names/references/authors/...) loaded.

Starting a new search while the old one is still running creates a warning "Wait until last search has finished or cancel it".

This dialog should have buttons "Cancel new search", "Cancel last search" (maybe you have an idea for better labels).
The expected default behavior is to cancel the last search as there is no reason for running a new search if you don't want to stop the old one. Often this may happen if one realizes that the search was too unspecific.
We may even think about a "remember my decision" preference as one may always want to stop previous searches once running a new one.

The progress monitor may show the pattern to search for. Currently it only shows something like "Load names". Better show "Load names for 'A*'" or something similar.

I think we should increase the number of records per page. 50, 100 and/or 200 seems reasonable. Probably this should depend on the performance of the class. E.g. authors may have 200 records as loading is fast. References may have 100 and names 50.

Andreas Müller wrote:

The loading should take place sorted so the new records are added to the end. Currently one does not really understand what happens until the load is completed. If sorted, new records will always be added to the end (so only the scroll bar will change a bit, this is the typical/expected behavior). Also, if load is canceled one does not know, which records are already loaded.

The sorting is also very important for re-search which takes place e.g. after save. Now it is very obvious that the data is fully reloaded, but shouldn't.

Generally we need to think about reloading. For large datasets this might not be a good strategy as it takes too long, however reloading is the safest way to avoid session issues.

Comment #6321#note-13: I tried opening the progress view several times and never saw this exception.
Comment #6321#note-15: Search results are ordered by title cache
Comment #6321#note-16: Progress bar shows up as dialog. Preference for long "run in background" added to taxeditor general preferences
Comment #6321#note-17: search progress is shown
Comment #6321#note-18: Now the previous search is just canceled. I also think this is the most common behavior. Preferences or dialogs are too much overload in this case.
Comment #6321#note-19: search query is shown
Comment #6321#note-20: Every editor input can specify its individual page size

Andreas Müller wrote:

Critical: setting focus does not work after reloading.

E.g. after saving the focus is lost. Don't know if this is related to loading strategy (this ticket) or to the new NAT table (#7439).

I don't understand the problem of the focus loss. What were you trying to do and what did not work?

Even worse than the focus loss is that we jump back to first record. This is definitely not acceptable.

Patrick Plitzner wrote:

Ah, ok. This is a misunderstanding of the terms we used. For me (and code-wise), the focus refers to a view or editor i.e. the whole bulk editor in this case. What you are talking about is the selection.

OK, I understand. Other UI frameworks allow to distinguish focus and selection even within a view. E.g. focus is on a single row (and makes this row editable, while selection is on a multi-select, or a row is selected but does not have the focus, so it has a right click menu but can not be edited). But it looks like RCP or NAT does not support this.

Patrick Plitzner wrote:

And this definitely is a problem which happens because of the re-searching. But we can reset the selection after the re-search has finished.

Can't we do the re-select as soon as the record is loaded? This way the user can start working as soon as possible. Otherwise, all the parallel loading doesn't make so much sense.
We could even think about first loading the selected record and then load all the rest.

Parallel loading is temporarily deactivated. Fix for #7505 and update problem in name details view

Did you change anything or did you simply reactivate?

I just reactivated and tested a bit. I made some small changes I think.

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

!ENTRY org.eclipse.e4.ui.workbench 4 0 2018-08-10 20:58:01.277
!MESSAGE 
!STACK 0
eu.etaxonomy.taxeditor.remoting.CdmEagerLoadingException: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://test.e-taxonomy.eu:80/cdmserver/rem_conf_am/remoting/common.service]; nested exception is org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.hibernate.collection.internal.AbstractPersistentCollection.remoteInitialize(AbstractPersistentCollection.java:1358)
    at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:627)
    at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:167)
    at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:184)
    at org.hibernate.collection.internal.PersistentSet.size(PersistentSet.java:143)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.setSectionTitle(AbstractEntityCollectionSection.java:178)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.internalUpdateSection(AbstractEntityCollectionSection.java:204)
    at eu.etaxonomy.taxeditor.ui.section.AbstractEntityCollectionSection.setEntity(AbstractEntityCollectionSection.java:165)
    at eu.etaxonomy.taxeditor.view.detail.CdmSectionPart.setFormInput(CdmSectionPart.java:155)
    at org.eclipse.ui.forms.ManagedForm.setInput(ManagedForm.java:210)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmDataViewerE4.refresh(AbstractCdmDataViewerE4.java:161)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmDataViewerE4.setInput(AbstractCdmDataViewerE4.java:143)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmEditorPartE4.showViewer(AbstractCdmEditorPartE4.java:225)
    at eu.etaxonomy.taxeditor.view.e4.supplementaldata.SupplementalDataPartE4.selectionChanged_internal(SupplementalDataPartE4.java:121)
    at eu.etaxonomy.taxeditor.view.e4.AbstractCdmEditorPartE4$DelaySelection.run(AbstractCdmEditorPartE4.java:85)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:182)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4211)
    at org.eclipse.swt.widgets.Display.msgFilterProc(Display.java:3541)
    at org.eclipse.swt.internal.win32.OS.DefWindowProcW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DefWindowProc(OS.java:2547)
    at org.eclipse.swt.widgets.Scrollable.callWindowProc(Scrollable.java:88)
    at org.eclipse.swt.widgets.Composite.WM_SYSCOMMAND(Composite.java:1864)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:4877)
    at org.eclipse.swt.widgets.Canvas.windowProc(Canvas.java:359)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:5110)
    at org.eclipse.swt.internal.win32.OS.DefWindowProcW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DefWindowProc(OS.java:2547)
    at org.eclipse.swt.widgets.Scrollable.callWindowProc(Scrollable.java:88)
    at org.eclipse.swt.widgets.Control.windowProc(Control.java:4897)
    at org.eclipse.swt.widgets.Canvas.windowProc(Canvas.java:359)
    at org.eclipse.swt.widgets.Display.windowProc(Display.java:5123)
    at org.eclipse.swt.internal.win32.OS.DispatchMessageW(Native Method)
    at org.eclipse.swt.internal.win32.OS.DispatchMessage(OS.java:2552)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3822)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1121)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1022)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:150)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:693)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:610)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
    at eu.etaxonomy.taxeditor.Application.start(Application.java:24)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
Caused by: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://test.e-taxonomy.eu:80/cdmserver/rem_conf_am/remoting/common.service]; nested exception is org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:216)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:147)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy49.initializeCollection(Unknown Source)
    at eu.etaxonomy.taxeditor.service.CachedCommonServiceImpl.initializeCollection(CachedCommonServiceImpl.java:61)
    at org.hibernate.collection.internal.AbstractPersistentCollection.remoteInitialize(AbstractPersistentCollection.java:1340)
    ... 55 more
Caused by: org.apache.http.NoHttpResponseException: test.e-taxonomy.eu:80 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.springframework.remoting.httpinvoker.HttpComponentsHttpInvokerRequestExecutor.executeHttpPost(HttpComponentsHttpInvokerRequestExecutor.java:338)
    at org.springframework.remoting.httpinvoker.HttpComponentsHttpInvokerRequestExecutor.doExecuteRequest(HttpComponentsHttpInvokerRequestExecutor.java:229)
    at eu.etaxonomy.taxeditor.service.CdmServiceRequestExecutor.doExecuteRequest(CdmServiceRequestExecutor.java:61)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:138)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:194)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:176)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:144)
    ... 60 more

Reloading after save and setting selection after save if selection is one of the later records in a huge dataset is still an open issue as you may need to wait long time until you can continue working. Sometimes re-selection even does not work at all.

Generally reloading strategy needs to be evaluated. Hopefully there is a better solution which may load only changed records. This is related to transaction handling and we may want to open a new ticket for this.

However, did you test if it is possible to first load the selected record (if it is still in the search result) and then load all the rest. This might be an acceptable temporary solution.

Also got an NPE when deleting (and saving the delete) of a Reference. Maybe this is also related to parallel loading? Mention just in case. See separate mail.

Andreas Müller wrote:

When running a search and stopping it by running a new search the results of the first search are still (partly) available in the result of the second search. This can be reproduced by e.g. running a search on names with "" and restart a search with "R" afterwards in a large database (e.g. rem_conf_am).

It looks like the first search still delivers a couple of results before really being finished.

As we do not have a progress dialog but only the progressbar running a new search might be the preferred way to stop a long running search with a more specific one. Therefore this relevant (but not really critical as "only" more records are displayed than expected)

This is fixed now. I use a volatile boolean with a job change listener to wait for the canceled job to finish completely before starting the new search.

Andreas Müller wrote:

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

Looking at #7634 Walter already mentioned the same exception and it occured with the stable version on production. I could also (not deterministically) reproduce it on cdm_rem_conf_am. Could this be the same problem that both servers, test and production, share?

Patrick Plitzner wrote:

Andreas Müller wrote:

when testing a couple of times I got an EagerLoadingException caused by an NoHttpResponseException. May the test server is swapping again but maybe this is also a general issue on the way how the search is implemented. Or the server is not able to that many request at the same time. Needs to be checked. Therefore I added AK in cc.
I did run search on >10.000 names and reloading triggered by save.

Looking at #7634 Walter already mentioned the same exception and it occured with the stable version on production. I could also (not deterministically) reproduce it on cdm_rem_conf_am. Could this be the same problem that both servers, test and production, share?

I can't really see that the issue reported in #7634 is related to this one as Walter was doing something completely different. Of course NoHttpResponseException appear sometimes and in different context. But it looks like it appears relatively often with parallel bulk loading, something that was not reported for Non-parallel bulk loading. So we need to check if parallel bulk loading creates specific problems. Have you checked the log-files?

Comment #6321#note-40: I just added a delay of 500ms to wait for the previous job to finish. Waiting explicitily is very error prone for accidental endless loops during development and this is not a critical issue.
Comment #6321#note-41 #6321#note-42: The Cdm entities needed to be loaded into the main session during the parallel loading. The editing in the name details view is fixed. I also did not experience and CdmEagerLoadingException any more. This could have been related.
Comment #6321#note-43: This is fixed

Patrick Plitzner wrote:

Comment #6321#note-40: I just added a delay of 500ms to wait for the previous job to finish. Waiting explicitily is very error prone for accidental endless loops during development and this is not a critical issue.

Ok, this seems to work (though it is a bit dirty)

Patrick Plitzner wrote:

Comment #6321#note-43: This is fixed

OK, this looks quite good. I think we can only improve it by changing the save strategy which I created a new ticket for #7661