Project

General

Profile

bug #6389

Returning a Map in cdmlib leads to data loss

Added by Patrick Plitzner almost 2 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Highest
Category:
taxeditor
Target version:
Start date:
01/26/2017
Due date:
% Done:

100%

Severity:
critical
Found in Version:

Description

I added the following method to IService/ServiceBase. I then invoked it on the taxeditor side in different places. The returned map on the taxeditor side never has the expected size of i but only contains the first half of the entries. On the cdmlib side it is completely filled.

    public Map returnMap() {
        Map map = new HashMap<>();
        for(int i=0;i<10;i++){
            map.put(i, i);
        }
        return map;
    }

In a quick search in the service methods I only found save methods also returning maps but I am not sure if the return values are used on the taxeditor side.

In Campanula production DB search for "India, Sikkim*" in the specimen editor. The type icon for the DerivedUnits seem to be placed randomly every time you search.
After a little investigation I found out that it was due to IOccurrenceService.listTypeDesignations(Collection specimens). The returned collection of SpecimenTypeDesignation is not equal to the one that you could create by iterating through all DerivedUnits and adding the result of DerivedUnit.getSpecimenTypeDesignations().
As this is nevertheless a very resource intensive call just for showing icons I replaced it.
But anyway this should be analyzed.

Associated revisions

Revision 1b7f25fe (diff)
Added by pplitzner almost 2 years ago

ref #6389 avoid service call for icon determination (performance)

  • additionally the service call returned undeterministic results

Revision 98baf5f8 (diff)
Added by Patrick Plitzner 10 months ago

fix #6389 Fixed cache loader method

  • the wrong collection size was used for iteration leaving out the second half of map entries

Revision 1c9d09b9 (diff)
Added by Andreas Kohlbecker 9 months ago

ref #7325 applying taxeditor changeset (98baf5f ref #6389) onto cdmlib-cache

History

#1 Updated by Andreas Müller almost 2 years ago

Check if this method is used elsewhere. If not make it deprecated with reason that it currently gives not the expected results but may be fixed in future and link to the ticket and set to Reviewed with priority 11.

#2 Updated by Andreas Müller almost 2 years ago

  • Priority changed from New to Highest
  • Target version changed from Unassigned CDM tickets to Release 4.6

#3 Updated by Andreas Müller almost 2 years ago

  • Target version changed from Release 4.6 to Release 4.7

#4 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.7 to Release 4.8

#5 Updated by Patrick Plitzner over 1 year ago

  • Target version changed from Release 4.8 to Release 4.9

#6 Updated by Andreas Müller over 1 year ago

  • Target version changed from Release 4.9 to Release 4.10

#7 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.10 to Release 4.11

#8 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.11 to Release 4.12

#9 Updated by Andreas Müller about 1 year ago

  • Target version changed from Release 4.12 to Release 4.13

#10 Updated by Andreas Müller 10 months ago

  • Target version changed from Release 4.13 to Release 4.14

#11 Updated by Andreas Müller 10 months ago

  • Target version changed from Release 4.14 to Release 5.0

#12 Updated by Andreas Müller 10 months ago

  • Description updated (diff)

#13 Updated by Andreas Müller 10 months ago

  • Description updated (diff)

#14 Updated by Patrick Plitzner 10 months ago

  • Subject changed from listTypeDesignations() returns undeterministic results in specimen editor to Returning a Map in cdmlib leads to data loss
  • Severity changed from normal to critical

#15 Updated by Patrick Plitzner 10 months ago

  • Description updated (diff)

#16 Updated by Patrick Plitzner 10 months ago

  • Description updated (diff)

#17 Updated by Andreas Kohlbecker 10 months ago

  • Status changed from New to Feedback

To me it looks like this problem is related to Spring HTTPInvoker and the serilazization/de-serialization of maps.

For serialization and de-serialization of Maps the equals() and hashCode() methods are crucial and that the types of value and key are implementing Serializable. In your test code you are using int as key and value. But maps hold Objects, so internally java will need to convert these into Objects by un-/boxing.

First of all I would rewrite the test method, so that the map is type save and so that un-/boxing is not needed:


    public Map<Integer, Integer> returnMap() {
        Map<Integer, Integer> map = new HashMap<>();
        for(int i=0;i<10;i++){
            map.put(Integer.valueOf(i), Integer.valueOf(i));
        }
        return map;
    }

see also e.g.:

https://stackoverflow.com/questions/2747819/serializing-and-deserializing-a-map-with-key-as-string#2747911

#18 Updated by Patrick Plitzner 10 months ago

There is no change with using Integer.valueOf(i)

#19 Updated by Andreas Kohlbecker 10 months ago

Patrick Plitzner wrote:

There is no change with using Integer.valueOf(i)

Ok, this is good news!
Maybe it helps to take a look at the serialization and deserialization of Objects in HttpInvoker.

BTW: You could also test if the serialization and deserialization with jdk is working, see ObjectStreams.

#20 Updated by Patrick Plitzner 10 months ago

Tested serialization and deserialization for plain java with the following:

    public static void main(String args[]) {

        HashMap<Integer, Integer> map = new HashMap<>();
        for(int i=0;i<100;i++){
            map.put(Integer.valueOf(i),Integer.valueOf(i));
        }
        try{
            FileOutputStream fos =
                    new FileOutputStream("map.file");
            ObjectOutputStream oos = new ObjectOutputStream(fos);
            oos.writeObject(map);
            oos.close();
            fos.close();
            System.out.println("Serialized HashMap data is saved in map.file");

            map = null;
            FileInputStream fis = new FileInputStream("map.file");
            ObjectInputStream ois = new ObjectInputStream(fis);
            map = (HashMap) ois.readObject();
            ois.close();
            fis.close();
        }catch(Exception e) {
            e.printStackTrace();
            return;
        }
        System.out.println("Deserialized HashMap");
        map.forEach((key, value)->System.out.println("key: "+ key + ",  value: "+value));
    }

This works fine. So it must have something to do with the serialization process of spring.

#21 Updated by Patrick Plitzner 10 months ago

The problem has nothing to do with the deserialization but with CacheLoader.load(Map map, List alreadyVisitedEntities, boolean update) where the Map is not reconstructed correctly.

#22 Updated by Patrick Plitzner 10 months ago

  • Status changed from Feedback to Resolved
  • % Done changed from 0 to 50

#23 Updated by Patrick Plitzner 10 months ago

  • Assignee changed from Patrick Plitzner to Andreas Kohlbecker
  • % Done changed from 50 to 0

In the CacheLoader class the map was deconstructed into an array of twice the size containing keys and values alternating. Then the map was re-built but the iteration was only over half the size of the array. So the resulting map was always half the size and only contained the first half of entries.
I checked for cdmlib service methods that have a Map as return value and there are also some, but few, methods, other than save methods, that use Maps and that are also used in the taxeditor. So if there were any bugs related to these methods they could also be solved with this fix.

#24 Updated by Patrick Plitzner 10 months ago

  • Description updated (diff)

#25 Updated by Andreas Kohlbecker 9 months ago

  • Status changed from Resolved to Closed
  • Assignee changed from Andreas Kohlbecker to Patrick Plitzner
  • % Done changed from 0 to 100

looks ok and I applied your changeset 98baf5f onto cdmlib-cache

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)