more than 150 GET requests when navigating to Communities & Collections view

[saschaszott]

Describe the bug

Navigating to Communities & Collections leads to more than 150 HTTP GET requests per page. This results in page load times of more than 5 seconds (10+ seconds on old machines).

The page shows 50 entries. For each entry 3 GET requests are generated:

• https://host.tld/server/api/core/communities/_id_?endpointMap
• https://host.tld/server/api/core/communities/_id_/subcommunities?size=1
• https://host.tld/server/api/core/communities/_id_/collections?size=1

To Reproduce
Steps to reproduce the behavior:

1. go to https://dspace7-demo.atmire.com
2. click tab All of DSpace
3. choose entry Communities & Collections

Expected behavior

Minimize number of HTTP requests and page load time (best case: one request per page; but at most one request per entry).

[tdonohue]

To me, it sounds like the page should likely be updated to use projections/embeds in the initial request to better control what info comes back (so that many requests are not necessary).

I am able to reproduce this issue on the demo site. After the initial request occurs, I see the above 3 GET requests for each top-level community. For sites with a lot of top-level communities, that results in a large number of requests.

Flagging this as a performance issue.

@artlowel : Could you take a quick look at this? I'd like to schedule this for Beta 5, but not sure how much effort it'd take?

[artlowel]

The reason for those requests is to know whether a community has children, in order to render the chevron in front of the community name. That's why the requests all have size=1. We're not really interested in the returned communities and collections, only the total number of results in the pagination info. So ideally size=0 would be used, but that isn't supported by the backend.

Passing a page size for an embed also isn't supported by the backend. The only thing we can embed right now is subcommunities and collections using the default page size (20), and that wouldn't be a performance gain: the initial rest request would take multiple seconds for the server to return, and parsing such a big response on the angular side would take a non-negligible amount of time as well.

So the way forward here would seem to have to start with rest changes:

• either add a way to know whether a community has children without any additional request, straight as a property on the community response
• or add support for pagination options when requesting embeds: e.g. /api/core/communities/<:uuid>?embed=subcommunities&embed.subcommunities.size=1
• and ideally also add support for size=0 for when all you need is the pagination info
• and while we're at it: a single link on a community, or a search that would return both collections and communities in one list. Because there are multiple use cases in the UI where you want a single paginated list of all direct children, regardless of type.

One of the first two is required to do anything about this ticket, the last two are optional suggestions that could further improve the performance

All of this wouldn't get rid of the endpointmap requests though. That's a consequence of our policy of never hardcoding paths but discovering them using HAL links, more info on this older issue. So what's happening here is that before the UI does a api/core/communities/<:uuid>/subcommunities call, it's checking what links are available for that particular community, and it does so for each community individually.

We could improve performance here by e.g. assuming that once we've seen which links are available for one object of a certain type, we've seen them all. But honestly I'm not sure that assumption is always correct. I believe the workflowGroups links on collections are an example where the links change between specific objects of the same type.

All in all I think this is a "needs discussion" type of ticket.

[tdonohue]

Thanks for your summary here @artlowel . I agree that based on your analysis this is much more complex than it first appears -- and that the necessary changes need to start on the REST API side.

@abollini or @benbosman : I wonder if either of you have thoughts on how to properly support a way to simply check whether a resource has subresources (in this case, whether a Community has any children) more easily from the REST API, based on @artlowel 's brainstorms above. The current behavior is obviously a performance issue...we shouldn't require 3 requests per top-level community in order to build the Community List page. Ideally, we need a way to "get top-level communities, along with info on whether they have any subcommunities or collections" in a single request.

UPDATE: I dug around a bit to see if there's some easy way to achieve this in Spring Data REST. I'm not seeing one (I could be missing it though). So, I'm starting to wonder if the easiest solution is to return a new property in each response. It could be as simple as a "hasChildren": "true/false" (or if we wanted to know the exact count, it could be "childrenCount": "0").

[benbosman]

I've analyzed the preferred solution we identified in the last meeting:

• add support for pagination options when requesting embeds: e.g. /api/core/communities/<:uuid>?embed=subcommunities&embed.subcommunities.size=1

This will only require minor changes assuming the parameter syntax can be changed to e.g. /api/core/communities/<:uuid>?embed=subcommunities&embed.size=subcommunities=1 (so there's a single parameter name with all sizes)

This can also support /api/core/communities/<:uuid>?embed=subcommunities&embed=collections&embed=subcommunities/collections&embed.size=subcommunities=5&embed.size=collections=5&embed.size=subcommunities/collections=1

I expect the implementation including tests to require 4 hours of work (most time required for tests)

I would be interested in claiming this work if it's approved

[tdonohue]

@benbosman : Thanks for this summary! Honestly, it sounds great to me. My only (very minor) comment is that we obviously should ensure having 2 equals symbols in one querystring param (e.g. embed.size=collections=5) is parsed properly by our code (and Spring). I'd expect/hope it would work fine, but I know that's a slightly different syntax than usual.

@abollini : Any initial comments/feedback here? We'll obviously have a chance to touch base in our meeting this week as well, if necessary.

[benbosman]

@tdonohue It's also possible to use embed.size=collections:5 if preferred, but I've tested in REST that there's no problem with using the embed.size=collections=5 syntax (If I create such a request, the values can be retrieved as is)