Filtering RESTfull JSON view with Spring

You will be able to find full code that you can actually compile and run on the GitHub (each “way” in a separate branch).

I still consider this post incomplete and as soon as I have enough time I will revise it.

todo: add GraphQL as another way

todo: use ResourceProcessor<T>


The problem

We have a backend (Spring: Boot, Data JPA, Data REST) that provides a REST API. The predetermined API users (frontend and other components of the system) only need a certain part of the JSON object they ask for. For example, when asking for User resource, one API user wants only its username and the other wants first name and last name. We will discover different ways of satisfying the API users while remaining a RESTful API, Spring integration and minimizing the effort.

Spring Boot 1.5.9.RELEASE defines further versions:


Preparation

TL;DR creating entities and their repositories: one User belongs to many Groups.

First, let’s set up a workspace. This is available on the main branch.

We will be using PostgreSQL, so we want Hibernate to properly use its sequences:

Now, we can refer to this generator name using @GeneratedValue. Learn more about this on Vlad Mihalcea’s  blog.

Now, create some entities:

P.S. about Identifiable: vote for this issue I created.

A User must belong to a Group. A Group contains zero or more Users.

Next, we need the repositories:

We have also populated the database with 3 groups: AB, and C and have assigned 3 users to each of those.

Skipping all the other minor settings (see full code on GitHub), let’s see what we have.

Accessing GET localhost:8080:

We can see the two repositories being automatically exported by Spring Data REST.

It automatically pluralizes entity names, but does it better if EVO Inflector is on the classpath. To learn more about profile resource see here.

Accessing GET http://localhost:8080/groups/1:

This is the group A and we can access its users by following users HATEOAS link.

With that, we can branch out the repository to try all the ways of filtering JSON.


Getting around LAZY fetching

TL;DR in a case you always need some lazy property, association or just part of that then fetch only what you need in the first place and do not try to bypass lazy fetching.

In case we want full related resources, we can just define them as new fields in our DTO and include this dependency in the pom.xml which will be automatically resolved by Spring Data REST:

com.fasterxml.jackson.datatype
jackson-datatype-hibernate5

This dependency helps with lazy initialized associations. It basically allows forcing additional select on serialization of lazy properties. This is not recommended because there must not be any case where application always triggers some lazy association because this collides with the meaning of the lazy initialization. In such cases, you should just fetch what you need in the first place. However, this dependency also can tell Jackson to not touch non-fetched lazy fields, so, in theory, you may serialize a JPA entity without getting a LazyInitializationException.

You can also manipulate fetching types on a per-query basis using JPA 2.1 @NamedEntityGraph and Spring’s @EntityGraph (these define what properties to fetch eagerly and what lazily). It is still not recommended since both LAZY and EAGER fetching types are just a hint and not a requirement and you should not mess with them since even if you change those types the actual properties set fetched may (and most likely will) differ from what you expected. Also, it seems that defining the fetching type on the entity itself and on the entity graph use different logic (from the DTO example, I tried to include the group field and somehow it gives an exception on “entity lazy; graph eager” and works fine on “entity eager”, so you definitely cannot trust that).


Notes on remaining RESTful

TL;DR do not modify default resource paths or users will not know which data to include for modifying. Do not expose resource’s id unless it is absolutely required.

If we want to alter the view of our hypermedia resources, we must not touch the default single resource paths (../users is a collection resource path; ../users/1 is a single resource path). Otherwise, we just go against REST principals.

If you return that view for an item resource by default, how would a client know which data it had to send to update the resource. (c) Oliver Gierke

Instead, we should specify different paths for our custom view somewhere else like ../users/views/username-only/1 or even something like ../users/1?view=username-only.

Note, Spring Data REST places all user-defined repository query methods on the path “collection resource”/search/”method name”?{param1}&{param2} and uses methods defined in CrudRepository to serve default paths (e.g. findOne is used for all single resources). We can change the path part of “method name” and set a name to our new “resource” (set the rel) using @RestResource but we cannot make parameters into path variables (except if we create a @Controller of course).

Regarding ids, Spring Data REST will always fetch them since it needs to construct the HATEOAS links and they use the ids. It is important to understand that in a RESTful API a unique identifier of a resource is its link and a database primary key is on a lower level. Even the main class ResourceSupport has the method public Link getId() { return getLink(Link.REL_SELF); }. The point of HATEOAS is for an API user to be able to simply navigate through the links rel names not even looking at the hrefs at all. It is a practice at Amazon to generate buttons on the website that are filled with these links (rel is the button name and href is its link), so in some cases it is not even required to redeploy the frontend when adding/removing links in the backend. It is, however possible to expose @Ids for certain entities through exposeIdsFor.


DTO

This approach is saved on the GitHub under the DTO branch.

We will start with the most straightforward way — just replace the object that is going to be serialized by Jackson with our custom one where we can define whatever fields we want.

Now, the only way to make this POJO work with Spring Data repositories is to define query methods on the repository using this DTO as a return type:

This approach is described in Spring Data JPA documentation Example 71. A projecting DTO. They also recommend using Lombok library for code generation, so that a DTO class definition will only need fields and a single @Value class-level annotation.

Accessing GET localhost:8080/users/search:

Accessing GET localhost:8080/users/search/username?id=1:

To make the difference more obvious, we can compare it to the default view:

Accessing GET localhost:8080/users/1:

As you can see, we lost not only the fields we excluded but also lost our links and now have to include them manually. Otherwise, we go against REST.

However, keep in mind that REST is not a law but rather a convenience convention, especially it is not important at all for a custom path that is for a specific need which does not even use these HATEOAS links. But we will go full REST here assuming we really need the links.

For that, we will refer to Spring Data REST documentation:

  • Overriding Spring Data REST Response Handlers for how to alter certain resource behavior. To integrate our controller with Spring Data REST we will use @RepositoryRestController.
  • Programmatic Links for a way to actually place the links on the resource. Spring HATEOAS defines EntityLinks interface (can be autowired) that lets you create links based on managed resources (entities) rather than based on a raw URI string or @Controllers like LinkBuilder does. In Spring Data REST there is an “EntityLinks implementation that is able to create Link for domain classes managed by Spring Data REST” that is RepositoryEntityLinks.

Accessing GET localhost:8080/users/search/username?id=1:

Now, this is a proper RESTful response. The generated SQL:

select users.username from users where users.id=1

In case we also want some fields from the related resources (this is against REST unless we put related resources inside _embedded section), we can define them as new fields in our DTO and then use the DTO constructor in the JPQL query. For example, if we define one more String field (e.g. groupName) in the UserUsername then we can just annotate UserRepository.findUsernameById with this @Query(“select new com.samkruglov.dtos.UserUsername(u.username, u.group.name) from User u where u.id=:id”) and our JSON response will also contain the group name. Learn more about using DTO with raw JPA in this Vlad Mihalcea’s blog post.


Spring Data REST Projection

This approach is saved on the GitHub under the Projection branch.

Spring Data JPA Projection technique is defined in section 4.3.11 of the documentation. Basically, we can define an interface with JavaBean getters that point to the fields we want.

To demonstrate the power of the Spring Projections, we will start from the case where we want some fields from the related resources (this is against REST unless we put related resources inside _embedded section). For that, we can simply use recursive open projections (see examples 65 and 68 of the docs):

  • line 4 — defines the return type as our interface (projection).
  • line 6 to line 22 — defines our projection
  • line 8 — refers to User.username. Spring accesses it by field, not by getter, so if User did not have the getUsername method but still had username field it would still work.
  • line 10 — default interface method makes this projection open. First, this method is ignored by mapping to User (and there is no groupName field in User anyway). Second, because this is an open projection, some optimizations that could have been applied by Spring are dropped.
  • line 16 — refers to User.group field. Notice, that the type is not Group but it is GroupName wich is also another projection that is defined at line 18.
  • line 15 — makes Jackson ignore group field which removes the field from resulting JSON. Spring Data JPA, however, will still recognize it.
  • line 18 — defines another projection that filters Group entity. Notice how it is declared inside Username projection — this is not required, we may place it anywhere.
  • line 20 — refers to Group.name.

Resulting JSON fields’ names are determined by decapitalizing these getters, so we have:

  • getUsername username
  • getGroupName groupName
  • getGroup — ignored due to @JsonIgnore annotation

Accessing localhost:8080/users/search/username?id=1:

Note, that this recursive technique is not supported for the DTO approach, only valid with interfaces.

To make the difference more obvious, we can compare it to the default view:

Accessing GET localhost:8080/users/1:

As you can see, we lost our links and embedded a group name onto the root.

Unfortunately, we cannot add HATEOAS links in this state but it is not always required so one may find this solution satisfying.

This approach uses Spring Data JPA Projection. To be able to add links we have to use Spring Data REST Projection which a tiny bit different and has these two requirements (section 7.1 of the docs):

How does Spring Data REST finds projection definitions?

  • Any @Projection interface found in the same package as your entity definitions (or one of it’s sub-packages) is registered.
  • You can manually register via RepositoryRestConfiguration.getProjectionConfiguration().addProjection(…).

In either situation, the interface with your projection MUST have the @Projection annotation.

For simplicity, further investigation will continue with redefined projection:

We have put this interface in a dedicated .java file under entities.projections package so Spring Data REST will register it. You will see the purpose of the name annotation attribule just in a moment.

As you might have noticed, there are no mentions on query methods. That is because Spring Data REST Projection extends Spring Data JPA Projection a little bit farther with aforementioned automatic registering into the REST API. We should remove our query method thus keeping the UserRepository untouched:

That is enough to make this work. Let’s see what changed in our REST API.

Accessing GET localhost:8080/users/1:

As you can see, self link is exactly the link that we accessed, but notice how user single resource link has changed. user link now also highlights to the API user that there is a projection representation available. Following the ALPS specification the API user can navigate to the profile of this resource by going to the collection resource and then to its profile link:

Accessing GET localhost:8080/users:

The users collection is omitted, it is not actually empty. We can see the profile link that we can use to observe:

Accessing GET localhost:8080/profile/users:

Other contents omitted. Here we can see that usually the returned type rt is user-representation which enumerates all the fields defined in the User entity but there is also a possibility of a projection request parameter that can be one of the defined in its descriptors blocks. There is only one — username-only which describes its representation through its descriptors blocks where each of those is a separate field. There is only one — username.

As you can see, one can easily navigate in a RESTful API.

Accessing GET localhost:8080/users/1?projection=username_only:

The request above generates the following SQL (aliases omitted, query parameters inserted):

select users.id, users.first_name, users.group_id, users.last_name, users.username from users where users.id=1

Even if we redefine our interface to be open and recursive as we did before, everything keeps working without any effort. However, the generated SQL contains two selects instead of one join (not sure if that’s a very bad thing; aliases omitted, query parameters inserted):

select users.id, users.first_name, users.group_id, users.last_name, users.username from users where users.id=1
select groups.id, groups.name from groups where groups.id=1

@JsonView and @JsonFilter

This approach is saved on the GitHub under the JsonFilter branch.

This may only be useful for limiting data transfer from the application to REST API users because this approach does not touch SQL at all as it only works with Jackson.

For a good and really quick overview of both @JsonView and @JsonFilter, you may watch this part of Eugene Paraschiv presentation at Spring I/O 2016.

We will not cover @JsonView because there is plenty of material out there about that, there is no proper support for using this with Spring HATEOAS or Spring Data REST and it is not even good (I will provide comments on the presentation examples in a bit).

Basically, you create an interface, e.g. called Views, inside that interface, you create other interfaces, e.g. one called FirstName, and then annotate your fields (e.g. only firstName) with @JsonView(Views.FirstName.class) and your @Controller method with the same annotation. And the returned result of the method will only contain this one field. Jackson has DEFAULT_VIEW_INCLUSION set to false by default. Also note, that these views (interfaces) support inheritance (child interface includes all the fields from its parents).

@JsonView is bad for our case because Spring HATEOAS provides a wrapper object Resource, that contains the _links property and our actual domain object inside contents but annotated with @JsonUnwrapped. So, if we try and do what is meant, we will fail because the @Controller method actually returns not the domain object, for which we provided a @JsonView but the Resource object that does not even have the fields that are described in that view. Because of that, we will have an empty JSON object due to DEFAULT_VIEW_INCLUSION being false. In the presentation above, Eugene returns ResourceSupport (I am guessing by the class name) from his @Controller method rather than the entity he showed a slide before which just does not work like that because @JsonView on the @Controller method must also be used on the class that is returned by the method. So, for usage with HATEOAS one actually must implement a DTO (POJO that extends ResourceSupport (or better Resource)) and use @JsonView there. For our case, to make @JsonView work we must manually, inside a @RestController (this disables Spring Data REST integration), create and add links to our HATEOAS DTO. Or we can research the ways to somehow configure Spring Data REST ObjectMapper to support that. Either way, @JsonView for Spring Data REST is not the right approach.

@JsonFilter allows an API user to define which fields he needs at runtime by sending them right inside the URI.

With @JsonFilter, we simply put one class-level annotation @JsonFilter(“userFilter”) on our entity (not shown here) and create the following controller:

Accessing GET localhost:8080/users/1?fields=username:

Accessing GET localhost:8080/users/1?fields=username,firstName:

As you can see, we lost our links but we will take care of this later. Also, we have overridden the path users/{id} and now can only access it via this new method but we can actually fix that!

Spring MVC and Spring Data REST controllers run in separate environments (like sandboxes). With that, if we register the controller with @RestController, then we drop the Spring Data REST default handling of the specified path (users/{id}) and define our own within Spring MVC environment. If we register the controller with @RepositoryRestController, then we overload the Spring Data REST default handling of the specified path (users/{id}) but for that to actually be overloading rather than overriding we must include the param annotation attribute for the mapping method for Spring to be able to distinguish our method from the Spring Data REST’s one.

Note, I also changed the method return type to be wrapped in ResponseEntity. I do not know why but without this wrapping it just does not serialize and gives an exception.

This looks good so far but we still cannot access the users/1 because it gives us this error: “Can not resolve PropertyFilter with id ‘userFilter’“. This is because if you define a @JsonFilter anywhere then you must provide the actual filter, however, we can configure ObjectMapper to not require that and we will do that because we do not require an API user to specify the fields parameter. Fortunately, it is pretty easy to configure Spring Data REST’s ObjectMapper:

Accessing GET localhost:8080/users/1?fields=username:

Accessing GET localhost:8080/users/1:

That way we can actually still access users/1 and will get the response like we did not change anything.

Now, let’s try to configure the links:

Accessing GET localhost:8080/users/1?fields=firstName:

At first sight, this seems awesome. But if we look closely, this actually messes up the links structure (you may compare the differences with other responses from above). I thought that this is a bug and reported under DATAREST-1179, so, please, vote for that. If you managed to fix the links by yourself, please, leave a comment and show us how you did that.

So, currently, @JsonFilter is only suitable if you are okay with dropping the links.

However, keep in mind that REST is not a law but rather a convenience convention, especially it is not important at all for a custom path that is for a specific need which does not even use these HATEOAS links. But we will go full REST here assuming we really need the links.

It would be cool if we could somehow implement this technique for every single resource somewhere in one place and without having to override Spring Data REST behavior for cases where the filter is not used. I believe that Spring Data REST controller’s are defined package-private, so we cannot influence that. So, if we want such behavior, we should just give up on Spring Data REST and write it by hand with Spring MVC and Spring HATEOAS. Please, vote for this feature request.


Conclusion

Well, it is quite simple — use Spring Data REST Projections — this is actually the only way of the ones we showcased that is mentioned in the Spring Data REST documentation. This provides a way to predefine a set of needed fields at compile-time, name that set of fields, and refer to it as an URL argument by the API users. This also affects the underlying SQL.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s