Agrest 5.x Alpha Protocol Documentation

1. Overview

Unlike the rest of the Agrest documentation that talks about the server framework, this document should be of interest not only to the server-side developers, but to the client-side devs as well, as it demonstrates how to access Agrest services.

"Agrest Protocol" is based on HTTP and JSON and defines how a client can communicate with Agrest endpoints. It consists of two things:

A format of JSON documents used in requests and responses. It can work with any application-specific data model.
A set of control parameters for the client to request a specific representation of the model from the server (i.e. the client can control sorting, filtering, offsets and included properties of the response objects). An endpoint may support all or a subset of these parameters, depending on its intention and capabilities.

As we’ll see below, the protocol defines some powerful features that give a lot of flexibility to the client to decide what data is retrieved from the service and how it should look like. The fact that every Agrest service looks the same regardless of a specific data model simplifies the implementation of the client. And the server implementation is assisted by the framework.

1.1. Data Model

The examples in this document are using a simple "bookstore" model. Here it is defined as pseudocode:

Author
  - id : long
  - name : string
  - dateOfBirth : date
  - books : list<Book>

Book
  - id : long
  - title : string
  - genre: string
  - author : Author

As you see, the model consists of two entities (Author and Book), each with an "id" property, some value properties (strings, dates, etc.) and related entity properties (list<Book>, Author). id, attribute and relationship are the terms used throughout Agrest to distinguish between these types of properties.

2. Protocol Conventions

As things often are in the REST world, Agrest protocol is quite loose. Many elements are conventions rather than hard requirements, but following them would produce consistent, easy-to-understand APIs. Here are some common conventions for writing Agrest endpoints, some reflected in the default server framework, others left to the developer to follow:

Content type of GET, POST, PUT responses serving message documents or collection documents is application/json
Content type of POST and PUT requests consuming update documents is application/json
Control parameters are passed as URL query parameters (e.g. /my?include=name)
Values of control parameters may contain spaces, special chars and small chunks of JSON. So they need to be URL-encoded
Individual object ids are passed as URL path components (e.g. /my/5)
Standard HTTP status codes are used to convey either a success or a failure of a request
Dates and times in requests and responses are in ISO 8601 format. E.g. 2022-04-19T11:08:53Z, 2023-04-10T11:08, 1979-04-19

3. JSON Documents

3.1. Message Response

This document is used in response to POST, PUT or DELETE when no entity data is returned to the client.

{
   "message" : "..."
}

Table 1. Message Response Document properties
`message`	string	An optional human-readable message, usually provided to explain reasons of a failure.

3.2. Collection Response

This document is used in response to GET, POST or PUT to return data from the server. This is the format used to return any kind of data to the client.

{
   "data" : [
      { "id" : 5, "name": "A" },
      { "id" : 8, "name": "B" }
   ],
   "total" : 2
}

Table 2. Collection Response Document properties
`data`	list or object	A list (or sometimes, an object) that contains entity objects. The schema of the objects is implicit, and depends on the endpoint model and request control parameters. Each object may contain other objects, with an arbitrary number of nesting levels.
`total`	int	The total number of objects in the collection. This is usually equal to the size of the `data` list. Except when pagination is in use, in which case the total corresponds to the size of the collection ignoring pagination. I.e. it may be greater than the number of the returned objects in the "data" array.

3.3. Update Request

This document is used as a body of a POST or PUT request to pass the data from the client to the server to modify an entity collection. It is either a single object or a collection of objects (so pure custom model, no Agrest-specific properties) :

{ "id" : 5, "name": "X" }

[
   { "id" : 5, "name": "X" },
   { "id" : 8, "name": "Y" }
]

While collection response documents can contain object trees with an unlimited depth (entities with their related entities, with their own related entities and so on), update documents can only have properties of a single entity. It can still have references to the related entities as described below, but no entity nesting.

3.3.1. Updating To-One Relationships

To set a to-one relationship of an updated entity, use a pair "propertyName" : "related_id" in the update request document:

{ "id" : 5, "related": 10 }

3.3.2. Updating To-Many Relationships

To set a to-many relationship of an updated entity, pass a collection of ids for the relationship property in the update request document:

{ "id" : 5, "related": [101, 2, 305] }

4. Control Parameters

Agrest protocol control parameters are exp, sort, dir, start, limit, mapBy, include, exclude. They are passed as URL query parameters (e.g. my?include=name) with the goal to shape the Collection Response document. With their help, the client can decide how to sort, filter, paginate the objects in the data collection, and which object properties to include.

These parameters are applied to GET and often to POST and PUT requests (when those latter ones return Collection Response).

Parameter values can be either simple strings/numbers or short JSON structures. JSON structures may require proper URL encoding to be done by the client.

4.1. exp

exp is used to filter objects in the response collection. It can take one of the three forms, depending on the parameter binding style:

exp=<exp_string> (1)
exp=["<exp_string>", "value1", "value2"] (2)
exp={"exp"="<exp_string>", "params":{ "param1" : "value1", "param2", "value2"}} (3)

1	Without parameters, an expression it is just a string (we’ll explain what `exp_string` is shortly)
2	An expression with parameter resolution by position is represented as a JSON list, where the expression string is the first element, and all subsequent elements are positional parameters
3	An expression with parameter resolution by name is represented as a JSON object with "exp" and "params" properties

exp_string is a conditional expression based on Agrest expression syntax, that should be intuitively understood by most programmers.

The full syntax of exp_string is described in the Expression Syntax chapter

Below are some examples:

name='Ernest Hemingway'
books.title = 'A Farewell to Arms'
title = not in ('A Farewell to Arms', 'For Whom the Bell Tolls')
title like 'A%' and author.dateOfBirth > $afterDate

A few things to note about the examples above:

Expressions use dot-separated paths to refer to the values of object properties
There is an implicit "root entity" to which the paths are applied. It is either the main entity of the request or, when an expression is used inside an include, a related entity. So name and books.title would work with Author, while title and author.dateOfBirth - with Book.
Parameters always start with a "$" sign, regardless of whether they are resolved by name or by position

Here are a few examples using an expression parameter. URL encoding is not shown for clarity:

Example 1: Filtering on a single property

exp=name='Ernest Hemingway'
exp=name like 'E%'
exp=name likeIgnoreCase '%ab%'

Example 2: Filtering on a relationship with a left join

exp=books+ = null

The "+" sign denotes "left join" syntax. Comparing to null returns all authors who have no books in the system.

Example 3: Parameter resolution by position

exp=["author.dateOfBirth > $afterDate","1900-01-01"]

Example 4: Parameter resolution by name

exp={"exp":"author.dateOfBirth > $afterDate", "params":{"afterDate":"1900-01-01" }}

4.2. sort / direction

sort and direction are used to order objects in the response collection. The following forms are supported:

sort=<prop_path> (1)
sort=<prop_path>&direction=<direction_spec> (2)
sort=<json_sort_object> (3)
sort=[<json_sort_object>,...] (4)

1	Ascending sort by a single property
2	Sort by a single property with explicit direction
3	Sort by a single property with a sort object
4	Sort by multiple properties with a list of sort objects

direction is added to specify sort direction when sorting is done on a single property. It can have one of these values (specified in lower or upper case) :

asc: this is also the default, when no direction is provided
desc
asc_ci: for case-insensitive ascending ordering
desc_ci: for case-insensitive descending ordering

<json_sort_object> allows to specify both the property and the direction in one place and has a form of {"path": .., "direction": .. }. direction is optional and takes the same values as direction. Let’s demonstrate all the flavors with a few examples:

Example 1: Ascending sort by a single property

sort=name

Example 2: Descending sort by a single property with explicit direction

sort=id&direction=desc

Example 3: Descending sort by a single property with a sort object

sort={"path":"id","direction":"desc"}

Example 4: Sort by multiple properties with a list of sort objects

sort=[{"path":"name"},{"path":"dateOfBirth","direction":"desc"}]

start and limit parameters are used for client-controlled pagination of a response collection. They can be used together or individually to request a range of objects in a collection.

start is an offset within the "data" array. All the objects below this offset are discarded from the response collection. Default start is 0.

limit is a maximum number of objects in the collection "data". Default is no limit.

limit is applied after start. So for a collection of 10 objects, start=2&limit=5 would result in objects 2 through 6 returned from the server. Returned Collection "total" would still be 10.

4.4. mapBy

mapBy parameter is used to reshape the data collection from a list of objects to a map of lists, keyed by some property. E.g. consider a typical list response:

{
"data" : [
    { "id" : 8, "title" : "One Hundred Years of Solitude",  "genre" : "fiction" },
    { "id" : 5, "title" : "Battle Cry of Freedom",  "genre" : "history" },
    { "id" : 12, "title" : "For Whom the Bell Tolls",  "genre" : "fiction" }
  ],
  "total":3
}

With mapBy=genre it is transformed to a map. The total here is still the number of objects in all the maps combined:

{
"data" : {
    "fiction" : [
        { "id" : 8, "title" : "One Hundred Years of Solitude",  "genre" : "fiction" },
        { "id" : 12, "title" : "For Whom the Bell Tolls",  "genre" : "fiction" }
    ],
    "history" : [
        { "id" : 5, "title" : "Battle Cry of Freedom",  "genre" : "history" }
    ]
  },
  "total" : 3
}

4.5. include / exclude

include and exclude are used to recursively shape individual objects in a response collection. These are the controls that turn your REST endpoints fixed models into graphs that can be dynamically navigated by the clients.

exclude format:

exclude=<prop_path> (1)
exclude=[<prop_path>,...] (2)

1	A single property path
2	A JSON array of property paths

include format:

include=<prop_path> (1)
include=<json_include_object> (2)
include=[<prop_path_or_json_include_object>,...] (3)

1	A single property path
2	A JSON include object
3	A JSON array of property paths and include objects

<json_include_object> has the following structure:

{
    "path": .. , // the only required property
    "exp": .. ,
    "sort": .. ,
    "start": ..,
    "limit": ..
    "mapBy": ..
    "include": ...
}

The only required property is path that determines which property is included. If the path points to a relationship, the object can contain properties corresponding to all the individual controls we’ve seen already (even a nested include!). Those controls are applied to the related entity denoted by the path.

A few more notes before we show the examples:

What is included by default? As we’ve discussed above, Agrest model entities consist of id, attribute and relationship properties. If no includes are specified, Collection Response document would contain the id and all the attributes of a given entity, and none of the relationships.
Multiple include and exclude parameters can be used in a single request. They will be combined together.

Now let’s see the examples:

In the examples below we will omit the {"data":[..],"total": .. } collection document wrapper, and will only show the structure of a single individual object within the "data" collection.

Example 1: Include the id and the attributes, but exclude the "genre" attribute

exclude=genre

{ "id" : 8, "title" : "One Hundred Years of Solitude" }

Example 2: Only include "id"

include=id

{ "id" : 8 }

Example 3: Multiple includes, one of them pointing to the attributes of a related entity

include=id&include=author.name

{ "id" : 8, "author" : {"name" : "Gabriel García Márquez"} }

Example 4: JSON include object with sorting, filtering and a nested include

include={"path":"books","exp":"title like '%a%'","sort":"title", "include":"title"}

{
   "books" : [
      { "title" : "Autumn of the Patriarch" },
      { "title" : "One Hundred Years of Solitude" }
   ]
}

Example 5: JSON include object with mapBy and a nested include

include={"path":"books","mapBy":"genre", "include":"id"}

{
   "books" : {
      "fiction" : [
        { "id" : 55 },
        { "id" : 8 }
      ]
   }
}

Example 6: Include and Exclude parameters can take an array of values

include=["id","name"]

{ "id" : 45, "name" : "Gabriel García Márquez"}

Example 7: The include array can contain a combination of paths and include objects

include=["id","books.title",{"path":"books","exp":"title like %a%'"}]

{
   "id" : 45,
   "books" : [
      { "title" : "Autumn of the Patriarch" },
      { "title" : "One Hundred Years of Solitude" }
   ]
}

Example 8: Include array is recursive

include=["id",{"books":["id", "title"]}]

{
   "id" : 45,
   "books" : [
      { "id" : 55, "title" : "Autumn of the Patriarch" },
      { "id" : 8, "title" : "One Hundred Years of Solitude" }
   ]
}

In this example attributes of both the root entity and the related entity are specified as JSON arrays. Also, there is a shortcut - instead of {"path":"books","include":[..]}}, we are using {"books":[..]}.

5. Expression Syntax

Expressions passed via an exp control parameter or inside a JSON include are conditions evaluated in the context of the root request entity (in the former case), or a nested entity (in the latter case). We’ve already shown the URL parameter structure for exp (e.g. how to pass parameters). Here we will focus on the syntax of the expression itself.

The syntax presented here is a part of the Agrest protocol. Though some Agrest backends may not be able to support every single expression described here, but most should work with a reasonable subset. The Java server framework shipped by the Agrest project supports the syntax in its entirety.

5.1. Literals

There are a few kinds of literals that denote various constant scalar values in expressions - numbers, strings, dates, etc.

Null is a literal that indicates an absent value:

null

Boolean literals are either true or false:

true
false

A variety of number literals are supported - positive / negative, whole / decimal, etc.:

// positive int
1

// negative int
-2

// long
3147483647L

// double
2.1

// big decimal (arbitrary size and precision)
2.1001234065B

// TODO: hex types, etc.

String literals contain arbitrary sequences of characters, and are enclosed in either single or double quotes:

'abc'
"abc"

// if quotes are present in a string, either use non-conflicting quote symbols,
// or escape them explicitly
'a"c'
"a'c"
"a\"c"

Date / time components are just strings that follow ISO 8601 format:

"2023-09-11"
"2023-09-11T00:00:01"

5.2. Paths

Important building blocks of expressions are property paths. They are dot-separated unquoted strings that point to property names of some entity:

// Entity id. Most Agrest entities have it
id

// Entity attribute (a "simple" or "value" property)
dateOfBirth

// Entity relationship
books

// Dot-separated path spanning multiple entities
books.id
books.library.name

Paths are used not only in expressions, but also in a variety of other control parameters (sort, include, exclude, mapBy)

There are a few advanced features that allow to specify special behavior of the relationship path components:

// "Optional" semantics

// Adding "+" to a relationship anywhere in the path indicates optional" semantics.
// This is a hint to the server evaluating the expression. E.g. if the expression is
// translated to SQL, LEFT JOIN will be used for optional expressions
books+.library.name

5.3. Parameters

Parameters are placeholders in expressions that are replaced with scalar values when the expression is evaluated on the server. Parameters are unquoted strings that start with a dollar sign:

// must start with a dollar sign,
// followed by a sequence of letters, digits or underscores
$myVar
$1
$_myVar

As described in the previous chapter, parameters can be resolved either by name (the name is the part of the string that follows the dollar sign), or by position in the expression. Parameter naming syntax for both styles is the same (e.g. there’s no special significance to using a number as the param name, like $15. It is simply a parameter called "15").

5.4. Arithmetic Expressions

a + 5
1 - a
a * b
b.c / d

5.5. Bitwise Expressions

TODO …

5.6. Functions

Agrest supports a number of scalar functions. Function names are case-sensitive. Functions can take zero or more arguments that can be paths, scalars, arithmetic and bitwise expressions. Supported functions:

// date/time
currentDate()
currentTime()
currentTimestamp()
day(exp)
dayOfMonth(exp)
dayOfWeek(exp)
dayOfYear(exp)
hour(exp)
minute(exp)
month(exp)
second(exp)
week(exp)
year(exp)

// numeric
abs(exp)
mod(exp1, exp2)
sqrt(exp)

// string
concat(exp1, exp2)
length(exp)
locate(toLocateExp, inStringExp [, startAtIndexExp])
lower(exp)
substring(strExp, startIndexExp [, lenExp])
trim(exp)
upper(exp)

5.7. Simple Conditions

"Simple" conditional expressions are built from paths and literals and a variety of comparison operators:

name = 'Joe'
name != 'Alice'
name <> 'Alice'

salary < 50000
salary <= 50000
age > 16
age >= 21

salary between 50000 and 150000
salary not between 50000 and 150000

color in ('blue', 'red', 'green')
color not in ('blue', 'red', 'green')

A special kind of comparison is SQL-style pattern matching with "like" operator: a like '<pattern>'. In the pattern string, a few characters have special meanings as "wildcards": "%" stands for any sequence of characters, while "_" stands for any single character. If you want "%" or "_" to be treated as a regular character, not a wildcard, you should prefix it with an arbitrary escape character, as shown in the examples below.

name like 'A%'
name like 'A_C_'

// do not treat the first underscore as a wildcard
name like 'Ax_C_' escape 'x'

// case-insensitive matching
name likeIgnoreCase 'a%'

// not like
name not like 'A%'
name not likeIgnoreCase 'A%'

5.8. Composite Conditions

Conditions can be joined together using and, or or negated with not.

name like 'A%' and salary < 100000
name = 'Joe' or name = 'Ann'
not name like 'A%' // same as "name not like 'A%'"

Operator precedence (in decreasing order) goes like : not, and, or. Parenthesis can be used to specify an explicit order of execution that is different from the default:

(salary < 100000 or age > 40) and name like 'A%'

Agrest Protocol