DS-4283: (Google Scholar) Provide Entity Type Label/Name as part of URL path

[heathergreerklein]

This is a duplicate of DS-4283

Estimate from release plan: TBD

This is a request from the Google Scholar team. I talked with them briefly today about the upcoming DSpace 7 release.

They've requested the ability to differentiate Entity Types via the URL path/pattern (e.g. a Person might have a URL of /person/[uuid], while a Publication may have a path of /publication/[uuid].

The reason they want this feature is so that their crawlers can be "trained" to concentrate on objects of specific types. Not all Entities may be of interest to Google Scholar (e.g. Persons would likely not be indexed). Ideally they'd like to be able to make a decision based on the URL, if possible.

They are aware that Entities are entirely configurable, so that this will not be a complete solution for all DSpace installations. However, for any out-of-the-box Entities, they can train their crawlers to be selective regarding which ones to concentrate on.

[benbosman]

A few options were discussed for this ticket.

Our recommendations were:

• Sitemaps are the ideal place to identify which items to index, e.g. by creating a sitemap specifically for Google Scholar, or using alternative URLs in the sitemap which allow Google Scholar to see which items are desired
• Informing Google Scholar of the entity type will imply that institutions which don't use type Publication but rather use type Article are not indexed in Google Scholar. It's better to have a config with a good default value which identifies the types to index in Google Scholar. This ensures that any repository not using the default types is still able to be indexed in Google Scholar
• We can create a Google Scholar specific sitemap containing only the relevant items. This will be generated automatically daily using the new solution which auto-generates sitemaps even without cron jobs. The relevant items will include items without an entity type and items with type "Publication", but this is configurable and can be adjusted. The configuration can link to a page explaining how it should be determined which items are relevant for Google Scholar
• Alternatively, we can use the regular sitemap, and just add info (e.g. a parameter or a different URL path, which will result in the same item once opened) for the items which are relevant for Google Scholar
• For either solution, we'd expect roughly one day of work

Google Scholar's recommendations were:

• They don't support Sitemaps because they're unable to limit the harvested data to the contents of a sitemap
• They would want item pages to use the syntax /typeditems/<:entity-type>/<:uuid> if there's an entity type
• They would only be indexing items matching the pattern /typeditems/publication/<:uuid> or /items/<:uuid> (untyped)
• They don't want the repository manager to be able to identify another entity type (e.g. article) to be relevant for indexing in Google Scholar
• This would require Angular to adjust links to the items. For any displayed item link (in the search results, related items, handle redirects, …) the metadata should be retrieved, and the URL should be /typeditems/<:entity-type>/<:uuid> if there's an entity type
• For any page opened with URL /items/<:uuid>, Angular should verify whether there's an entity type, and route the user to /typeditems/<:entity-type>/<:uuid> if there's one
• For any page opened with URL /typeditems/<:entity-type>/<:uuid>, Angular should verify whether there's an entity type, and route the user to /items/<:uuid> if there's none, or route the user to /typeditems/<:correct-entity-type>/<:uuid> if the type isn't correct
• The UUID based URLs would no longer be persistent, but handles are still persistent
• Google Scholar will adjust their harvesting system to only include items matching the pattern /typeditems/publication/<:uuid> or /items/<:uuid> (untyped)
• This will require changes to any Angular code which uses /items/<:uuid>
• This will require Angular to redirect between /items/<:uuid> and /typeditems/<:entity-type>/<:uuid> and vice versa
• This will require REST to use the different syntax in handles and sitemaps
• I don't have an estimate for this yet because it's hard to identify what's all impacted. We'd first need to identify all locations where item URLs are used. Based on the current expectations, I'd estimate at least 3 days of work. But this can also become much larger

[tdonohue]

@benbosman : I think you've accidentally misstated some of Google Scholar's recommendations on this ticket. The recommendations you listed in your comment are not entirely correct based on my discussions with Google Scholar

I've rewritten the Google Scholar recommendations below (in the same order you have above):

• GS does support sitemaps & they do use them first (After all, GS requested we update our sitemaps to be auto-enabled and auto-updated per DS-4061 expose sitemaps and schedule automatic updates DSpace#2858). However, they've found through experience that Sitemaps are not always 100% accurate. So, they do not limit their indexing to only what is listed in the sitemap.
• GS would like Entities to use a standard URL path, so that it's easy to determine if a page represents a generic Item or a typed Entity. The URL path we choose makes no difference to them, but it could be something like /entities/<:type>/<:uuid> but other suggestions are welcome (their only hard requirement is they want the entity type in the URL path).
• GS plans to create a standard "allow list" of DSpace v7 URL paths that are of interest to Google Scholar. This might be /entities/publications/<:uuid> and /items/<:uuid> initially, but it depends on which entities are available out of the box. This "allow list" can change over time/future releases.
• GS is worried about giving repository managers the power to select which custom entity types are of interest to Google Scholar. If those repository managers "guessed wrong", it could result in the GS spiders automatically blocking/flagging their DSpace site. (My vague understanding is if the spider finds a lot of uninteresting/invalid content, it may flag the site in some way so that the site is blocked or indexed less frequently.) Instead, they recommended that individual sites contact GS to notify them of custom Entities which they'd like to appear in Google Scholar. GS team would then analyze whether that custom entity is of interest, and, if so, add it to their "allow list" for that site.

Base on those recommendations, parts of your solution seems correct. But, there's other parts that I'm not as certain about:

1. I agree, for any page opened with URL /items/<:uuid>, we'd want to check if the Item has an entity type and, if so, redirect to /entities/<:type>/<:uuid>.
   • This actually creates a persistent URL based on UUID, as all Items can be guaranteed accessible via /items/<:uuid> (in some cases it may just redirect you, but that's no different than a persistent Handle URL).
   • I do not know that we need the opposite redirect. I cannot think of any real life scenario where you'd try to access a normal item under an entity URL path, so I don't see why we'd want a redirect from /entities/<:type>/<:uuid> back to /items/<:uuid>.
2. Rather than writing all these changes into the UI-layer, I wonder if we should consider indexing Entities differently in Solr, so that it's easier to tell Items & Entities apart in search/browse results, etc. For instance, would it be easier (for the UI) if Person Entities were always returned with type: entity/person (or alternatively have some other separate field(s) that denote it's an entity) instead of a normal type: item?
   • This may simplify the UI logic. If the type of object starts with "entity", then use a URL path of /entities/<:type>/<:uuid> (even in search results), but otherwise treat it like an Item. (Or if easier we could even return type: item/person to let the UI know it's still an Item, but a specific type of one). That way, browsing & searching items would result in a mix of Items with /items/<:uuid> URLs and /entities/<:type>/<:uuid> URLs.
   • This also might simplify the Sitemap logic. The Sitemap generation could be slightly refactored to use the Solr index instead of going at the database directly. That way it could easily tell which URL to use for each object in the Sitemap.

I'm not sure if these ideas help with your analysis/estimate. But, I'm also glad to add this to a future meeting agenda if we feel more discussion is necessary.

[benbosman]

Based on input from @abollini in a previous discussion, we would propose the following solution which minimizes the complexity in DSpace, and makes it easier for Google Scholar to index the items:

• There are two types of items which should be indexed by Google Scholar: items without an entity type (for repositories not using entities, similar to DSpace 6) and items with entity type Publication (for repositories using entities, where we want a Publication to be indexed in Google Scholar, but not a Person)
• In DSpace 6, the items had no entity type, and used the syntax /handle/prefix/suffix. In DSpace 6 CRIS, the Publications were items, which also used the syntax /handle/prefix/suffix. A Person used a different syntax /cris/rp/id. Similarly, organizations used the syntax /cris/ou/id.
• Since Google Scholar already has support for this, we want to propose a similar approach for DSpace 7. Items without an entity type use the syntax /items/uuid (already the current implementation). Items with entity type Publication will use the syntax /items/uuid (also the current implementation). Items with entity type Person will use the syntax /typeditems/Person/uuid (a customization), and similar for other entity types. The syntax could alternatively be /items/Person/uuid to make it more obvious it’s still an item

The changes required for this solution are quite similar to what was discussed before:

• The REST API can return two non-modifiable properties to help Angular determine the applicable URL of an item. A property isGSIndexable that’s true for items without entity type or with entity type publication, and a property entitytype that can be null or the current item’s type
• When the Angular UI creates a link to an item (e.g. in discovery, related items, handle redirects, …), it should verify the isGSIndexable property. If it’s true, it should define the URL as /items/uuid (current syntax). If it’s false, it should define the URL as /typeditems/<:entity-type>/<:uuid> (with entitytype retrieved from the entitytype property).
• For any page opened with URL /items/<:uuid>, Angular should verify whether isGSIndexable is true, and route the user to /typeditems/<:entity-type>/<:uuid> if it’s true
• For any page opened with URL /typeditems/<:entity-type>/<:uuid>, Angular should verify whether isGSIndexable is true and what the entitytype property is, and route the user to /items/<:uuid> if isGSIndexable is true, or route the user to /typeditems/<:correct-entity-type>/<:uuid> if the type doesn’t match the entitytype property (this use case is important in case an item was created with the wrong entity type

I’ve also researched the required customizations in more detail:

• The REST API should return the isGSIndexable and entitytype properties. I’d estimate 12 hours of work for this (including tests)
• Angular routing: Allow items to be opened using either /typeditems/*/<:uuid> or /items/<:uuid>: this is a change in the item-page-routing.module.ts. I’d estimate 12 hours of work for this. Using /items/*/<:uuid> or /items/<:uuid> would result in the same cost
• Angular item links: The links to items currently occur in e.g. org-unit-item-metadata-list-element.component.html, org-unit-search-result-list-element.component.html, org-unit-search-result-grid-element.component.html, org-unit.component.html, journal-search-result-list-element.component.html, publication-search-result-list-element.component.html, full-item-page.component.html, publication.component.html, … (not a complete list). The function getItemPageRoute can be adjusted (or a similar function can be created) to accept the item (not the ID), with the ability to define whether it’s a link to the full item page, and define the URL of this page. I’d estimate 16 hours of work for creating the function and using it in all these html files (including tests)
• Handles: Currently, hdl.handle.net/1234/56 will be resolved by the DSpace handle server in dspace7-demo.atmire.com/handle/1234/56, which will result in a request from Angular to get the item, and redirect to /items/<:uuid>. This can be adjusted in Angular to redirect to /items/<:uuid> or /typeditems/<:entity-type>/<:uuid> (Angular will have access to the REST representation of the item). This doesn’t require additional requests for the item metadata. I’d estimate this to 4 hours of work when reusing the Angular item links function
• Angular redirects: When on an item page, the current URL has to be compared with the expected URL, and a redirect may happen to change the URL. I’d estimate this to 8 hours of work when reusing the Angular item links component
• Sitemaps: The sitemaps currently use the /items/<:uuid> syntax. The entity type can either be retrieved from the item metadata or from solr (entity types are already indexed in discovery). The performance impact of either solution would need to be tested first. I’d estimate this to 12 hours of work.

Of course simply having the item pages contain a meta tag with the type to ensure the items which are not a publication don’t have to be requested would be less effort, but if I remember correctly that solution was not supported by Google Scholar.

[tdonohue]

As discussed in today's meeting:

Overall approach by @benbosman looks good. However, we recommend the following (minor) changes:

1. Entities should use /entities/<:entity-type>/<:uuid> paths. The term "typeditems" is no longer valid as we decided to just call them Entities.
2. All Entities should use the /entities/<:entity-type>/<:uuid> paths... this includes the Publication entity. So, there is no need for the isGSIndexable property. The REST API can just return an entitytype property, and the Angular UI can ensure that Items with an entitytype use the /entities/<:entity-type>/<:uuid> path, while Items without an entitytype use the default /items/<:uuid> path.
3. Redirects from one path to another only need to occur in one direction. If you access an Entity via the /items/<:uuid> path, then you should be redirected to /entities/<:entity-type>/<:uuid> (which is the correct URL for that Entity). However, the opposite redirect will not be implemented at this time....so we'd never redirect /entities/<:entity-type>/<:uuid> back to /items/<:uuid>. (If in the future we find a need for that additional redirect we can add it at a later date.)

Therefore, this ticket just needs an updated total estimate from @benbosman . I'd expect the estimate to decrease slightly from above as we've made some minor simplifications (namely the removal of isGSIndexable & fewer redirects). Once that estimate is added, this ticket can be assigned for work.

[benbosman]

I’m updating the estimates section:

• The REST API should return the entitytype property. I’d estimate 10 hours of work for this (including tests)
• Angular routing: Allow items to be opened using either /entities/*/<:uuid> or /items/<:uuid>: this is a change in the item-page-routing.module.ts. I’d estimate 12 hours of work for this
• Angular item links: The links to items currently occur in e.g. org-unit-item-metadata-list-element.component.html, org-unit-search-result-list-element.component.html, org-unit-search-result-grid-element.component.html, org-unit.component.html, journal-search-result-list-element.component.html, publication-search-result-list-element.component.html, full-item-page.component.html, publication.component.html, … (not a complete list). The function getItemPageRoute can be adjusted (or a similar function can be created) to accept the item (not the ID), with the ability to define whether it’s a link to the full item page, and define the URL of this page. I’d estimate 16 hours of work for creating the function and using it in all these html files (including tests)
• Handles: Currently, hdl.handle.net/1234/56 will be resolved by the DSpace handle server in dspace7-demo.atmire.com/handle/1234/56, which will result in a request from Angular to get the item, and redirect to /items/<:uuid>. This can be adjusted in Angular to redirect to /items/<:uuid> or /entities/<:entity-type>/<:uuid> (Angular will have access to the REST representation of the item). This doesn’t require additional requests for the item metadata. I’d estimate this to 4 hours of work when reusing the Angular item links function
• Angular redirects: When on an item page with syntax /items/<:uuid>, the current URL has to be compared with the expected URL, and a redirect may happen to change the URL. I’d estimate this to 7 hours of work when reusing the Angular item links component
• Sitemaps: The sitemaps currently use the /items/<:uuid> syntax. The entity type can either be retrieved from the item metadata or from solr (entity types are already indexed in discovery). The performance impact of either solution would need to be tested first. I’d estimate this to 12 hours of work.