Best way for resource (entity) relations loading in a REST API

Question

We have a problem of load time in our REST API, which is the result of having Fat DTO (nested DTO s) & the N+1 problem (which is off my question), means that we are loading too much data in one request, and a big chunk of this loaded data is not going to be used by the client (this is a classic problem in REST API).

ex:let's say we have endpoint that list all the buildings entities, so currently in the response we are loading all the related owners (list of UserDTO) in which each owner has a list of notifications (list of NotificationsDTO ) ... etc

GET /buildings

---- response
{
 "name" : "Building X"
 "owners" [
  {
   "name" : "Tarek B",
   "notifications" : [
    { 
      "title" : "title 1" 
      //....
    }
   ]
   //....
  }
 ]
 //....
}

I have a couple of solutions in mind (you are very welcome to add other ideas, or tools that might help me with this), so which one of the following options is the best (keep in mind the effort needed to the implementation and other factors as scaling and maintenance ... )

My current options:

Cut all the relations with Ids, and offer foreach entity an endpoint
to get multiple with a list of Ids.
make a query parameter to specify which relations will be loaded ex: /buildings?with=owners,floors and this will load the buildings with all the owners and floors. -I am a bit afraid this might expand in the future and endup re-inventing the wheel of graphQL -
rewrite only the Data requesting part (GET endpoints) of the API with GraphQL. -too much work in the backend, also i do not know if we could use the old search logic from our Specification toPredicate() -

Current tech stack : SpringBoot - Angular

Peter Csala · Answer

I think there is no one-size-fits-all solution for this problem.
There is a lot of factors which influences the to be chosen solution, like

How many different clients / consumers do you have?
1.1) Less than then?
1.2) More than a thousand?
Are these clients using your API in the same way?
2.1) Are they implementing the same functionality (just for different platforms)?
2.2) Do they conduct absolutely different flows?
Do you have influence on these clients?
3.1) Are they 1st party clients where change requests can be submitted?
3.2) Are they 3rd party clients without any influence?
Does your API have SLA / SLO and big response could harm that?
Does extra roundtrip for details is a viable option for all of your clients?
etc.

Even if you have just a single client with a well-known consuming pattern then it is still a challenging exercise. Since you have to go through all the exposed endpoints and check which approach might fit best.
For example:

Client displays data in a master-details view and extra roundtrip is a not a problem then option #1 might be the best choice
Client splits your massive data into smaller chunks and feeds several of its components  then option #2, #3 could be a better fit
etc.

Wim Ombelets · Answer

There are a number of ways to address this in a restful way:

Subresources: the /buildings endpoint would only return an array of buildings and their respective properties (properties, not linked resources). If you would need any additional linked data on a per-resource basis, you would use an endpoint like /buildings/{id}/owners/{id}/notifications. This is by far the advised way to go in general from a pure (purist) restful design standpoint but, depending on your use cases, can lead to an overhead of api calls because, as you already state, would force iteration which is to be avoided whenever possible.However, when will you need all buildings? I'm making an assumption here about your domain but /streets/{id}/buildings will already return a vastly reduced subset of buildings. If you subresource correctly, you could get away with your so-called fat DTO.
OData: making your api queryable can add some interesting capabilities to your api that would otherwise not be available in (1). Notably, for this particular problem, expansion and pagination. This would allow your endpoint to still serve buildings only when needed, but allows expanding the resource with linked subresources and support pagination so your body payload stays manageable. After all, the required data in the client is mostly limited to the human using it. You won't need 5000 records to be displayed at once.This seems a good fit for solving your issue.

Best way for resource (entity) relations loading in a REST API

2 Answers

Add your own answers!

Ask a Question