You are making some excellent points and there is definitely lots to think about. There are a few points I wanted to touch on for clarification right now though.
We are still in development of this new API, but we have our existing stack in which we need to integrate. As you’ve already identified, there are budgets and time frames and we can’t re-architect our entire data model to build a better API. We need a new approach to interact with our existing data and we have to work with certain aspects and architecture components as if they were set in stone at this time.
That being said, the problem I identified is already real in development and, given the ability to infinitely include other resources, has infinite potential to cause harm performance-wise. So I feel like this is something to think about right now. Even if it was only to familiarize yourself with the subject for when the time comes to apply the knowledge.
Given that any simple parent-child relationship has the potential to allow a user to request something with
include=parent.children.parent.children.parent.children, not limiting something like that seems like a mistake. I’d be happy to accept that it is not a mistake and if I understand you correctly, you wouldn’t treat it like one and allow as much freedom as possible.
You mention that RDBMS are quite efficient at dealing with these problems and I would agree. However, while implementing, I did not see how to utilize the DB properly to implement inclusion (I feel like CTEs are definitely something I have not looked into properly).
Instead we just perform one query and then query the database again to retrieve the records that should be included. We do this recursively (hopefully the term is used appropriately ) until we reached the end of all inclusion paths.
Regardless though, with infinite potential for inclusion (well, until you hit the string length limit the HTTP request parser accepts for a query argument), it doesn’t really matter how optimized your query will be processed. It might spend minutes in processing and cause issues all over the place.
I agree that a single slow query should not have such an impact that it takes out an entire node. But 20 might. And 20 requests is nothing that our rate limiting approach would care about.
After writing this, I feel like I’m being defensive because I’m looking for a solution to a problem in the wrong place . Thanks again for your valued input.