Project

General

Profile

task #3560

feature request #7491: [MASTER] Withheld unpublished taxa from webservice results and in DAOs

Withheld unpublished taxa from webservice used in E+M dataportal

Added by Andreas Müller almost 6 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Highest
Assignee:
Andreas Müller
Category:
cdmlib-remote
Target version:
Release 5.1
Start date:
09/27/2013
Due date:
% Done:

70%

Estimated time:
2.00 h
Severity:
critical
Tags:
euro+med search security permission phycobank

Description

see also #1780, #3740, #3739

the following services have been secured.

AbstractIdentifiableController:

  • open:
    • doGetTitleCache (not E+M)

BaseController:

  • doGet
  • open:
    • doGetMethod

TaxonController:

  • doGetClassifications
  • doGetDescriptionElementsByType
  • doGetDescriptions
  • doGetFieldUnits
  • doGetTaggedName (not yet correct)
  • doGetTaxonNodes (not yet correct)
  • open:
    • doGetIncludedTaxa
    • doGetTaxonNodeAgentRelations
    • doListSpecimensOrObservations
    • getAcceptedFor

TaxonPortalController:

  • doGetSynonymy
  • doGetTaxonNodes
  • doGetTaxonRelations
  • doGetMedia
  • doGetSubtreeMedia
  • doGetFromNameRelations
  • doGetToNameRelations

TaxonListController

  • doFind
  • dofindByEverythingFullText
  • doFindByFullText
  • doSearch
  • open:
    • doFindBestMatchingTaxon
    • dofindByDescriptionElementFullText
    • doFindByIdentifier
    • doFindByMarker
    • doFindByNameParts

TaxonPortalListController

  • no services

TaxonNodeController

  • doPageChildNodes
  • open:
    • getParent()

ClassificationController

  • getChildNodes
  • getChildNodesAtRank
  • getTaxonInContext
  • open:
    • getGroupedTaxaByHigherTaxon (not E+M)
    • getGroupedTaxaByMarkedParents (not E+M)

ClassificationPortalController

  • not relevant:
    • getClassificationRootNode

ClassificationPortalListController

  • getChildNodes
  • getChildNodesAtRank
  • getChildNodesOfTaxon
  • getPathFromTaxon
  • getPathFromTaxonToRank
  • getSiblingsOfTaxon

ClassificationListController

  • no services

DwcA-Controller:

  • doDwcaTaxExport

General:

  • open:
    • check property paths in all controllers if they include taxa
    • web-services from other controllers that return taxa as part of there result

ws-endpoint-usage.ods (23.2 KB) Andreas Müller, 08/15/2018 06:13 PM

Related issues

Related to Edit - feature request #3709: [E+M][Editor] sufficient rights management for E+M workflow In Progress 09/09/2013
Related to Edit - feature request #3740: Allow recursively setting the publish bit for TaxonBase Closed 09/27/2013
Related to Edit - bug #7745: Subtree filter for dataportal services Resolved 09/09/2018
Related to Edit - bug #7756: org.hibernate.QueryException in portal/taxon/{uuid}/subtree/media.json webservice related to relation direction param in prepareTaxonRelationshipQuery() Feedback 09/12/2018
Precedes Edit - feature request #7457: E+M: Allow filtering for only published in search New 09/30/2013 09/30/2013
Precedes Edit - feature request #7492: Withheld unpublished taxa from webservice used in other webservices New 06/19/2018

Associated revisions

Revision 2edda1af (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement publish filter for TaxonNodeDao.listChildrenOf and .countChildrenOf

Revision 16fe90fd (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement publish filter for ClassificationDao.listChildrenOf and .countChildrenOf

Revision acebcf17 (diff)
Added by Andreas Müller 12 months ago

ref #3560 add includeUnpublished for loadChildNodesOfTaxonNode in TaxEditor

Revision 43e22332 (diff)
Added by Andreas Müller 12 months ago

ref #3560 add includeUnpublished for loadChildNodesOfTaxonNode in Vaadin

Revision 3c1306f8 (diff)
Added by Andreas Müller 12 months ago

ref #3560 use global "unpublished" constant

Revision bb8eecc0 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement unpublished for load and find in service layer

Revision 78e8f961 (diff)
Added by Andreas Müller 12 months ago

ref #3560 unpublished for doGetTaxonNodes and doGetTaggedName and doGetSubtreeMedia

Revision 12b3d645 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement unpublished for taxon collection retrieval in occurrence service

Revision 62a08491 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement unpublished for rankSpecificRootNode in service and remote (no tests yet)

Revision e073fc85 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement unpublished for listSiblings in service and remote (no tests yet)

Revision 892f37a3 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement unpublished in BaseController.doGet

Revision 547b40a1 (diff)
Added by Andreas Müller 12 months ago

ref #3560 tests for unpublished listRankSpecificRootNodes

Revision 754a3c47 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement publish in taxon service by name search

Revision 1301eb40 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement publish in taxon service by name search

Revision 160717a9 (diff)
Added by Andreas Müller 12 months ago

ref #3560 fix test in TaxEditor for findMatchingXXX

Revision b8dc9fef (diff)
Added by Andreas Müller 12 months ago

ref #3560 set includeUnpublished as default for FindTaxaAndNames configurator

Revision 2d3df654 (diff)
Added by Andreas Müller 12 months ago

ref #3560 search also for unpublished as default in TaxEditor

Revision cac97126 (diff)
Added by Andreas Müller 12 months ago

ref #3560 implement loadTreeBranchToTaxon (pathTo) for unpublished

Revision d113f0b3 (diff)
Added by Andreas Müller 12 months ago

ref #3560 add exception package to TaxEdtior

Revision 024216db (diff)
Added by Andreas Müller 12 months ago

ref #3560 add publish flag to lucene index

Revision e293181b (diff)
Added by Andreas Müller 12 months ago

ref #3560 add publish flag to lucene search

Revision df72b543 (diff)
Added by Andreas Müller 12 months ago

ref #3560 fix pathFrom in classification service impl

Revision 7ac91dc0 (diff)
Added by Andreas Müller 11 months ago

ref #3560 further publish flag issues for lucene search and first tests on lucene search

Revision 95e2bd9c (diff)
Added by Andreas Müller 11 months ago

ref #3560 try to add publish to synonym search in findTaxaAndNamesByFullText (probably not working yet)

Revision 37b63de0 (diff)
Added by Andreas Müller 11 months ago

ref #3560 further improve full text taxonAndName search for publish flag, including tests

Revision 1771f383 (diff)
Added by Andreas Müller 11 months ago

ref #3560, ref #5477 fix missing lucene index update for synonyms and misapplied names

Revision 92a5fd30 (diff)
Added by Andreas Müller 11 months ago

ref #3560 remove unpublished synonyms from taxon.synonymy

Revision 1e9f87b1 (diff)
Added by Andreas Müller 11 months ago

ref #3560 handle access denied for unpublished data in some TaxonController methods

Revision 3e01938c (diff)
Added by Andreas Müller 11 months ago

ref #3560 remove unpublished synonyms from taxon.synonymy (fix)

Revision 7cfcc8f1 (diff)
Added by Andreas Müller 11 months ago

ref #3560 handle access denied for unpublished data in BaseController.doGet

Revision 34337128 (diff)
Added by Andreas Müller 11 months ago

ref #3560 remove unpublished synonyms from taxon.synonymy (handle homotypic syns correctly)

Revision 11fcf6af (diff)
Added by Andreas Müller 11 months ago

ref #3560 fix wrong method call in BaseController

Revision e08838f0 (diff)
Added by Andreas Müller 11 months ago

ref #3560 add includeUnpublished for taxon relationships

Revision 0d028764 (diff)
Added by Andreas Müller 11 months ago

ref #3560 fix includeUnpublished for taxon relationships and correct includePublished->includeUnpublished

Revision 53a4a1d9 (diff)
Added by Andreas Müller 11 months ago

ref #3560 secured TaxonPortalController.doGetMedia and doGetSubtreeMedia

Revision 76bdf64b (diff)
Added by Andreas Müller 11 months ago

ref #3560 secured TaxonPortalController.doGetFromNameRelations and doGetToNameRelations

Revision 87ffda36 (diff)
Added by Andreas Müller 11 months ago

ref #3560 unpublished exception handling for TaxonController.doGetAcceptedFor

Revision d499114c (diff)
Added by Andreas Müller 11 months ago

ref #3560 unpublished exception handling for TaxonController.getPathFromTaxon(ToRank)

Revision bb006ee7 (diff)
Added by Andreas Müller 11 months ago

ref #3560 fix checkPublished in service base class

Revision 68a8023a (diff)
Added by Andreas Müller 3 months ago

ref #3560

History

#1 Updated by Andreas Kohlbecker over 5 years ago

  • Category changed from cdmserver to cdmlib-remote

#2 Updated by Andreas Kohlbecker over 5 years ago

  • Severity changed from critical to blocker

#3 Updated by Andreas Kohlbecker over 5 years ago

  • Priority changed from Priority08 to Highest

This must be implemented at the persistence level

  • TaxonDao

  • SpecimenDao

  • Search functions

can we use Hibernate filtered collections for this?

#4 Updated by Andreas Kohlbecker over 5 years ago

  • Assignee changed from Andreas Kohlbecker to Andreas Müller
  • Category changed from cdmlib-remote to cdmlib
  • Subject changed from Withheld unpublished taxa and specimen from webservice results to Withheld unpublished taxa and specimen from webservice results and in DAOs

#5 Updated by Andreas Kohlbecker over 5 years ago

  • Severity changed from blocker to critical

for now we will use a second database for publishing, the entities marked as unpublished can be filtered out at the database level, so this issue is no longer a blocker.

#6 Updated by Andreas Müller about 3 years ago

  • Target version changed from CDM UML 3.3/3.4 - Postprocessing to Unassigned CDM tickets

#7 Updated by Andreas Müller almost 3 years ago

  • Target version changed from Unassigned CDM tickets to Euro+Med Portal Release

#8 Updated by Andreas Kohlbecker over 2 years ago

  • Tags set to phycobank, Euro+Med
  • Description updated (diff)
  • Due date set to 03/31/2016

this is crucial for the phycobank.

#9 Updated by Andreas Kohlbecker almost 2 years ago

#10 Updated by Andreas Müller over 1 year ago

#11 Updated by Andreas Müller 12 months ago

  • Priority changed from Highest to Priority14
  • Estimated time set to 15.00 h

#12 Updated by Andreas Müller 12 months ago

  • Due date deleted (03/31/2016)
  • Status changed from New to In Progress
  • Priority changed from Priority14 to Highest

#13 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#14 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#15 Updated by Andreas Müller 12 months ago

  • Description updated (diff)
  • % Done changed from 0 to 10

#16 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#17 Updated by Andreas Müller 12 months ago

  • Description updated (diff)
  • % Done changed from 10 to 20

#18 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#20 Updated by Andreas Müller 12 months ago

  • Subject changed from Withheld unpublished taxa and specimen from webservice results and in DAOs to Withheld unpublished taxa from webservice results and in DAOs

#21 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#22 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#23 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#24 Updated by Andreas Müller 12 months ago

  • Description updated (diff)

#25 Updated by Andreas Müller 12 months ago

#26 Updated by Andreas Müller 11 months ago

  • Tags changed from phycobank, euro+med to phycobank, euro+med, search

#27 Updated by Andreas Müller 11 months ago

  • Target version changed from Euro+Med Portal Release to Release 5.1

#28 Updated by Andreas Müller 11 months ago

  • Parent task set to #7491

#29 Updated by Andreas Müller 11 months ago

  • Subject changed from Withheld unpublished taxa from webservice results and in DAOs to Withheld unpublished taxa from webservice used in E+M dataportal

#30 Updated by Andreas Müller 11 months ago

  • % Done changed from 20 to 60

#31 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#32 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#33 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#34 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#35 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#36 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#37 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#38 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#39 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#40 Updated by Andreas Müller 11 months ago

  • Precedes feature request #7492: Withheld unpublished taxa from webservice used in other webservices added

#41 Updated by Andreas Müller 11 months ago

  • Category changed from cdmlib to cdmlib-remote

#42 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#43 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#44 Updated by Andreas Müller 11 months ago

  • % Done changed from 60 to 70
  • Estimated time changed from 15.00 h to 2.00 h

#45 Updated by Andreas Müller 11 months ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved
  • Assignee changed from Andreas Müller to Andreas Kohlbecker

I still want to unify the handling of unpublished a bit, but maybe you can already start testing if there some services that urgently need to be protected to not show unpublished data in the data portal.

The general usecase to protect ALL webservices to not show unpublished data is handled in #7492 and is not part of this ticket.

#46 Updated by Andreas Kohlbecker 11 months ago

Hi Andreas,

first of all I analyzed the logfile on the production server and created a statistics on all webservice calls that are send to the cdm-server.
By this we now have an overview on the webservice endpoints that are really used together with a usage count. The count corresponds to the number of calls to this endpoint as seen in the logfile.

I quickly went over the logfile and marked all webservice endpoints which potentially need filtering by the publish flag. Endpoints marked yellow needs further investigation, grey ones are ok. Other colors are explained in the sheet.

ws-endpoint-usage.ods

#47 Updated by Andreas Kohlbecker 11 months ago

  • File ws-endpoint-usage.ods added

#48 Updated by Andreas Müller 11 months ago

Andreas Kohlbecker wrote:

first of all I analyzed the logfile on the production server and created a statistics on all webservice calls that are send to the cdm-server.
By this we now have an overview on the webservice endpoints that are really used together with a usage count. The count corresponds to the number of calls to this endpoint as seen in the logfile.

I quickly went over the logfile and marked all webservice endpoints which potentially need filtering by the publish flag. Endpoints marked yellow needs further investigation, grey ones are ok. Other colors are explained in the sheet.

How do we want report further results/improvements? Handling in ods is not comfortable if 2 people work on it.

Some of the yellow services are already checked. See the overview in the ticket description.

#49 Updated by Andreas Kohlbecker 11 months ago

  • File ws-endpoint-usage.ods added

I updated the ws-endpoint-usage.ods spread sheet by the information from the issue description.

There are now only few controller methods left which need to be checked.

Andreas

#50 Updated by Andreas Kohlbecker 11 months ago

  • File deleted (ws-endpoint-usage.ods)

#51 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

#52 Updated by Andreas Kohlbecker 11 months ago

  • File ws-endpoint-usage.ods added

#53 Updated by Andreas Kohlbecker 11 months ago

  • File deleted (ws-endpoint-usage.ods)

#54 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

secured TaxonPortalController.doGetMedia and doGetSubtreeMedia

#55 Updated by Andreas Müller 11 months ago

  • Description updated (diff)

secured TaxonPortalController.doGetFromNameRelations and doGetToNameRelations

#56 Updated by Andreas Müller 11 months ago

  • Private changed from Yes to No

#57 Updated by Andreas Müller 11 months ago

  • Assignee changed from Andreas Kohlbecker to Andreas Müller

#59 Updated by Andreas Müller 9 months ago

  • File deleted (ws-endpoint-usage.ods)

#60 Updated by Andreas Müller 8 months ago

  • Related to bug #7745: Subtree filter for dataportal services added

#61 Updated by Andreas Kohlbecker 8 months ago

  • Related to bug #7756: org.hibernate.QueryException in portal/taxon/{uuid}/subtree/media.json webservice related to relation direction param in prepareTaxonRelationshipQuery() added

#62 Updated by Andreas Kohlbecker 4 months ago

  • Tags changed from phycobank, euro+med, search to phycobank, euro+med, search, security, permission

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 40 MB)