Agrest - Model-Driven REST Data Services

Protocol Version 1.2

1. Overview

Unlike the rest of the Agrest documentation that talks about the server framework, this document should be of interest not only to the server-side developers, but to the client-side devs as well, as it demonstrates how to access Agrest services.

"Agrest Protocol" is based on HTTP and JSON and defines how a client can communicate with Agrest endpoints. It consists of two things:

A format of JSON documents used in requests and responses. It can work with any application-specific data model.
A set of control parameters for the client to request a specific representation of the model from the server (i.e. the client can control sorting, filtering, offsets and included properties of the response objects). An endpoint may support all or a subset of these parameters, depending on its intention and capabilities.

As we’ll see below, the protocol defines some powerful features that give a lot of flexibility to the client to decide what data is retrieved from the service and how it should look like. The fact that every Agrest service looks the same regardless of a specific data model simplifies the implementation of the client. And the server implementation is assisted by the framework.

1.1. Data Model

The examples in this document are using a simple "bookstore" model. Here it is defined as pseudocode:

Author
  - id : long
  - name : string
  - dateOfBirth : date
  - books : list<Book>

Book
  - id : long
  - title : string
  - genre: string
  - author : Author

As you see, the model consists of two entities (Author and Book), each with an "id" property, some value properties (strings, dates, etc.) and related entity properties (list<Book>, Author). id, attribute and relationship are the terms used throughout Agrest to distinguish between these types of properties.

2. Protocol Conventions

As things often are in the REST world, Agrest protocol is quite loose. Many elements are conventions rather than hard requirements, but following them would produce consistent, easy-to-understand APIs. Here are some common conventions for writing Agrest endpoints, some reflected in the default server framework, others left to the developer to follow:

Content type of GET, POST, PUT responses serving message documents or collection documents is application/json
Content type of POST and PUT requests consuming update documents is application/json
Control parameters are passed as URL query parameters (e.g. /my?include=name)
Values of control parameters may contain spaces, special chars and small chunks of JSON. So they need to be URL-encoded
Individual object ids are passed as URL path components (e.g. /my/5)
Standard HTTP status codes are used to convey either a success or a failure of a request
Dates and times in requests and responses are in ISO 8601 format. E.g. 2022-04-19T11:08:53Z, 2023-04-10T11:08, 1979-04-19

3. JSON Documents

3.1. Message Response

This document is used in response to POST, PUT or DELETE when no entity data is returned to the client.

{
   "message" : "..."
}

Table 1. Message Response Document properties
`message`	string	An optional human-readable message, usually provided to explain reasons of a failure.

3.2. Collection Response

This document is used in response to GET, POST or PUT to return data from the server. This is the format used to return any kind of data to the client.

{
   "data" : [
      { "id" : 5, "name": "A" },
      { "id" : 8, "name": "B" }
   ],
   "total" : 2
}

Table 2. Collection Response Document properties
`data`	list or object	A list (or sometimes, an object) that contains entity objects. The schema of the objects is implicit, and depends on the endpoint model and request control parameters. Each object may contain other objects, with an arbitrary number of nesting levels.
`total`	int	The total number of objects in the collection. This is usually equal to the size of the `data` list. Except when pagination is in use, in which case the total corresponds to the size of the collection ignoring pagination. I.e. it may be greater than the number of the returned objects in the "data" array.

3.3. Update Request

This document is used as a body of a POST or PUT request to pass the data from the client to the server to modify an entity collection. It is either a single object or a collection of objects (so pure custom model, no Agrest-specific properties) :

{ "id" : 5, "name": "X" }

[
   { "id" : 5, "name": "X" },
   { "id" : 8, "name": "Y" }
]

While collection response documents can contain object trees with an unlimited depth (entities with their related entities, with their own related entities and so on), update documents can only have properties of a single entity. It can still have references to the related entities as described below, but no entity nesting.

3.3.1. Updating To-One Relationships

To set a to-one relationship of an updated entity, use a pair "propertyName" : "related_id" in the update request document:

{ "id" : 5, "related": 10 }

3.3.2. Updating To-Many Relationships

To set a to-many relationship of an updated entity, pass a collection of ids for the relationship property in the update request document:

{ "id" : 5, "related": [101, 2, 305] }

4. Control Parameters

Agrest protocol control parameters are exp, sort, dir, start, limit, mapBy, include, exclude. They are passed as URL query parameters (e.g. my?include=name) with the goal to shape the Collection Response document. With their help, the client can decide how to sort, filter, paginate the objects in the data collection, and which object properties to include.

These parameters are applied to GET and often to POST and PUT requests (when those latter ones return Collection Response).

Parameter values can be either simple strings/numbers or short JSON structures. JSON structures may require proper URL encoding to be done by the client.

4.1. exp

exp is used to filter objects in the response collection. It can take one of the three forms, depending on the parameter binding style:

exp=<exp_string> (1)
exp=["<exp_string>", "value1", "value2"] (2)
exp={"exp"="<exp_string>", "params":{ "param1" : "value1", "param2", "value2"}} (3)

1	Without parameters, an expression it is just a string (we’ll explain what `exp_string` is shortly)
2	An expression with parameter resolution by position is represented as a JSON list, where the expression string is the first element, and all subsequent elements are positional parameters
3	An expression with parameter resolution by name is represented as a JSON object with "exp" and "params" properties

exp_string is a conditional expression based on Agrest own "expression DSL", that should be intuitively understood by most programmers. Here are some examples:

name='Ernest Hemingway'
books.title = 'A Farewell to Arms'
title = not in ('A Farewell to Arms', 'For Whom the Bell Tolls')
title like 'A%' and author.dateOfBirth > $afterDate

TODO: A more comprehensive Agrest exp format specification is forthcoming. Until then, please check Apache Cayenne expressions that use practically the same syntax (except Agrest would not allow "db:" expressions). And see the examples below

A few things to note about the examples above:

Expressions use dot-separated paths to refer to the values of object properties
There is an implicit "root entity" to which the paths are applied. It is either the main entity of the request or, when an expression is used inside an include, a related entity. So name and books.title would work with Author, while title and author.dateOfBirth - with Book.
Parameters always start with a "$" sign, regardless of whether they are resolved by name or by position

Here are a few examples using an expression parameter. URL encoding is not shown for clarity:

Example 1: Filtering on a single property

exp=name='Ernest Hemingway'
exp=name like 'E%'
exp=name likeIgnoreCase '%ab%'

Example 2: Filtering on a relationship with a left join

exp=books+ = null

The "+" sign denotes "left join" syntax. Comparing to null returns all authors who have no books in the system.

Example 3: Parameter resolution by position

exp=["author.dateOfBirth > $afterDate","1900-01-01"]

Example 4: Parameter resolution by name

exp={"exp":"author.dateOfBirth > $afterDate", "params":{"afterDate":"1900-01-01" }}

4.2. sort / direction

sort and direction are used to order objects in the response collection. The following forms are supported:

sort=<prop_path> (1)
sort=<prop_path>&direction=<direction_spec> (2)
sort=<json_sort_object> (3)
sort=[<json_sort_object>,...] (4)

1	Ascending sort by a single property
2	Sort by a single property with explicit direction
3	Sort by a single property with a sort object
4	Sort by multiple properties with a list of sort objects

direction is added to specify sort direction when sorting is done on a single property. It can have one of these values (specified in lower or upper case) :

asc: this is also the default, when no direction is provided
desc
asc_ci: for case-insensitive ascending ordering
desc_ci: for case-insensitive descending ordering

<json_sort_object> allows to specify both the property and the direction in one place and has a form of {"path": .., "direction": .. }. direction is optional and takes the same values as direction. Let’s demonstrate all the flavors with a few examples:

Example 1: Ascending sort by a single property

sort=name

Example 2: Descending sort by a single property with explicit direction

sort=id&direction=desc

Example 3: Descending sort by a single property with a sort object

sort={"path":"id","direction":"desc"}

Example 4: Sort by multiple properties with a list of sort objects

sort=[{"path":"name"},{"path":"dateOfBirth","direction":"desc"}]

start and limit parameters are used for client-controlled pagination of a response collection. They can be used together or individually to request a range of objects in a collection.

start is an offset within the "data" array. All the objects below this offset are discarded from the response collection. Default start is 0.

limit is a maximum number of objects in the collection "data". Default is no limit.

limit is applied after start. So for a collection of 10 objects, start=2&limit=5 would result in objects 2 through 6 returned from the server. Returned Collection "total" would still be 10.

4.4. mapBy

mapBy parameter is used to reshape the data collection from a list of objects to a map of lists, keyed by some property. E.g. consider a typical list response:

{
"data" : [
    { "id" : 8, "title" : "One Hundred Years of Solitude",  "genre" : "fiction" },
    { "id" : 5, "title" : "Battle Cry of Freedom",  "genre" : "history" },
    { "id" : 12, "title" : "For Whom the Bell Tolls",  "genre" : "fiction" }
  ],
  "total":3
}

With mapBy=genre it is transformed to a map. The total here is still the number of objects in all the maps combined:

{
"data" : {
    "fiction" : [
        { "id" : 8, "title" : "One Hundred Years of Solitude",  "genre" : "fiction" },
        { "id" : 12, "title" : "For Whom the Bell Tolls",  "genre" : "fiction" }
    ],
    "history" : [
        { "id" : 5, "title" : "Battle Cry of Freedom",  "genre" : "history" }
    ]
  },
  "total" : 3
}

4.5. include / exclude

include and exclude are used to recursively shape individual objects in a response collection. These are the controls that turn your REST endpoints fixed models into graphs that can be dynamically navigated by the clients.

exclude format:

exclude=<prop_path> (1)
exclude=[<prop_path>,...] (2)

1	A single property path
2	A JSON array of property paths

include format:

include=<prop_path> (1)
include=<json_include_object> (2)
include=[<prop_path_or_json_include_object>,...] (3)

1	A single property path
2	A JSON include object
3	A JSON array of property paths and include objects

<json_include_object> has the following structure:

{
    "path": .. , // the only required property
    "exp": .. ,
    "sort": .. ,
    "start": ..,
    "limit": ..
    "mapBy": ..
    "include": ...
}

The only required property is path that determines which property is included. If the path points to a relationship, the object can contain properties corresponding to all the individual controls we’ve seen already (even a nested include!). Those controls are applied to the related entity denoted by the path.

A few more notes before we show the examples:

What is included by default? As we’ve discussed above, Agrest model entities consist of id, attribute and relationship properties. If no includes are specified, Collection Response document would contain the id and all the attributes of a given entity, and none of the relationships.
Multiple include and exclude parameters can be used in a single request. They will be combined together.

Now let’s see the examples:

In the examples below we will omit the {"data":[..],"total": .. } collection document wrapper, and will only show the structure of a single individual object within the "data" collection.

Example 1: Include the id and the attributes, but exclude the "genre" attribute

exclude=genre

{ "id" : 8, "title" : "One Hundred Years of Solitude" }

Example 2: Only include "id"

include=id

{ "id" : 8 }

Example 3: Multiple includes, one of them pointing to the attributes of a related entity

include=id&include=author.name

{ "id" : 8, "author" : {"name" : "Gabriel García Márquez"} }

Example 4: JSON include object with sorting, filtering and a nested include

include={"path":"books","exp":"title like '%a%'","sort":"title", "include":"title"}

{
   "books" : [
      { "title" : "Autumn of the Patriarch" },
      { "title" : "One Hundred Years of Solitude" }
   ]
}

Example 5: JSON include object with mapBy and a nested include

include={"path":"books","mapBy":"genre", "include":"id"}

{
   "books" : {
      "fiction" : [
        { "id" : 55 },
        { "id" : 8 }
      ]
   }
}

Example 6: Include and Exclude parameters can take an array of values

include=["id","name"]

{ "id" : 45, "name" : "Gabriel García Márquez"}

Example 7: The include array can contain a combination of paths and include objects

include=["id","books.title",{"path":"books","exp":"title like %a%'"}]

{
   "id" : 45,
   "books" : [
      { "title" : "Autumn of the Patriarch" },
      { "title" : "One Hundred Years of Solitude" }
   ]
}

Example 8: Include array is recursive

include=["id",{"books":["id", "title"]}]

{
   "id" : 45,
   "books" : [
      { "id" : 55, "title" : "Autumn of the Patriarch" },
      { "id" : 8, "title" : "One Hundred Years of Solitude" }
   ]
}

In this example attributes of both the root entity and the related entity are specified as JSON arrays. Also, there is a shortcut - instead of {"path":"books","include":[..]}}, we are using {"books":[..]}.

Agrest Protocol