Building a truly RESTful API using ASP.Net Core 2.1.

A lot of APIs that are called RESTful aren’t RESTful. This document captures the constraints a RESTful
system must adhere to be RESTful. Not all these constraints are straightforward to implement, and to be
completely correct, an architecture that skips on one of the required constraints or weakens it is
considered not conformed to REST, and that immediately means that most APIs that are built today and
are called RESTful, well, they aren't RESTful. It does mean that we need to know the consequences of
deviating from these constraints and understand the potential tradeoffs.

Table of Contents

A) REST 2

B) REST Constraints 2

C) Richardson Maturity Model 5

D) Restful API using ASP.NET core 6

E) Outer Facing Contract 7

F) Status Codes 11

G) Formatters and Content Negotiation 13

H) Handling unsupported media type 14

I) Purpose of Repository Patten 16

J) Partially updating a resource 17

K) Validation in Restful world 17

L) Paging Collection Resources 18

M) Filtering and searching 20

N) Sorting 20

O) HATEOAS 21

P) Cacheable 23

Q) Summary 25

R) References 29

S)
A) REST

The first thing that comes to mind is APIs, it's how we build APIs, and JSON, because that's what we get
back from these APIs. But, REST isn't just about building an API which consists of a few HTTP services
that return JSON. It's much broader than that.

REST is not a standard, it's an architectural style. In principle, REST is completely protocol agnostic. JSON
isn't a part of REST, but theoretically not even HTTP is a part of REST. It is an architectural style and
nothing more than that. It's up to us to fill in the blanks with the protocols and standards we want to use

Now let's say we click a link in our browser to access a specific page. For example: SVG on the Control
Logic Viewer Website. That one is again identified by a URI. A new request message is sent to the server,
and the server again sends back a representation of the page, the resource. The browser interprets it
and changes state. In other words, the client changes state depending on the representation of the
resource we're accessing. And that's representational state transfer, or REST.

The HTTP client can be a browser, but often it's an application that can be built in a variety of
technologies, like Angular or ASP.NET Core MVC. It wants to get a resource from API, so it sends a
request to the URI where that SVG resides, and the API sends back a representation of that resource in
the response body. These days that's typically a JSON representation, but it doesn't have to be. So, the
client gets back the representation and changes state, and then we can continue. The client sends a
request to the URI of another resource. The API sends back a representation and the client changes
state again. And with that we now know what REST is, but how do we build such a system? An
architectural style is described by a set of constraints, and it's these constraints we'll have to adhere to.

B) REST Constraints

REST is defined by six constraints, one of which is optional. We should see these constraints as design
decisions, and each of those can have both positive and negative impacts. So, for these constraints, the
benefits are considered to outweigh the disadvantages.

1) Client-Server Constraint

A very basic fundamental constraint. What this does is enforce client-server architecture. A client, or
consumer of the API, shouldn't be concerned with how data is stored, or how the representation is
generated, that's transparent.

A server, the API, shouldn't be concerned with client, for example, the user interface or user state or
anything related to how the client is implemented. In other words, client and server can evolve
separately.

2) Statelessness Constraint

The second constraint is statelessness. This means that the necessary state to handle every request is
contained within the request itself. When a client requests a resource, that request contains all the
information necessary to service the request. It's one of the constraints that ensures RESTful APIs can
scale so easily. We don't have things like server-side session state to keep in mind when scaling up.

3)Cacheable constraint

This one states that each response message must explicitly state if it can be cached or not. Like this we
can eliminate some client/ server interaction, and at the same time prevent clients from using out-of-
date data.

4)Layered System constraint

A REST-based solution can be comprised of multiple architectural layers, just as almost all application
architectures we use today. These layers can be modified, added, removed, but no one layer can directly
access a layer that's beyond the next one. That also means that a client cannot tell whether it's directly
connected to the final layer or to another intermediary along the way. REST restricts knowledge to a
single layer, which reduces the overall system complexity.

5)Code on demand Constraint (optional)

This one states that the server can extend or customize client functionality. For example, if the client is a
web application, the server can transfer JavaScript code to the client to extend its functionality.

6)Uniform Interface Constraint

It states that the API and the consumers of the API share one single technical interface. As we're
typically working with the HTTP protocol, this interface should be seen as a combination of resource
URIs, where we can find resources, HTTP methods, how we can interact with them, like GET and POST,
and HTTP media types, like application/json, application/xml, or more specific versions of that that are
reflected in the request and response bodies. All of these are standards, by the way. This makes it is a
good fit for cross-platform development.

By describing such a contract, this uniform interface constraint decouples the architecture, which in turn
enables each part to evolve independently.

There are four sub-constraints for this uniform interface constraint.

a) Identification of resources - sub-constraint

It states that individual resources are identified in requests using URIs, and those resources are
conceptually separate from the representations that are returned to the client. The server doesn't send
an entity from its database or possibly a combination of fields from multiple additional databases and
services. Instead it sends the data, typically for RESTful APIs, as JSON, but HTML, XML, or custom formats
are also possible. So, if the API supports XML and JSON, both representations are different from the
server's internal representation, but it is the same resource.

b) Manipulation of resources - sub-constraint

That's manipulation of resources through representations. When a client holds a representation of a

resource, including any possible metadata, it has enough information to modify or delete a resource on
the server, provided it has permission to do so.
c) Self-descriptive message - sub-constraint.

Each message must include enough information on how to process it. When a consumer requests data
from an API, we send a request message, but that message also has headers and a body. If the request
body contains a JSON representation of a resource, the message must specify that fact in its headers by
including the media type, application/json for example. Like that, the correct parser can be invoked to
process the request body, and it can be serialized into the correct class. Same goes for the other way
around. The application/json media type is a simple sample. Media types play a very important role in
REST.

d) HATEOAS (Hypermedia as the Engine of Application State) - sub-constraint

This is the one that a lot of RESTful systems fail to implement. Hypermedia is a generalization of
Hypertext. It adds other types like music, images, text, etc., and it's that hypermedia that should be the
engine of application state.

In other words, it drives how to consume and use the API. It tells the consumer what it can do with the
API. Can I delete this resource? Can I update it? How can I create it, and where can I find it? This really
boils down to a self-documenting API, and that documentation can then be used by the client
application to provide functionality to the user. It's not an easy constraint.
C) Richardson Maturity Model

The Richardson Maturity Model is a model developed by Leonard Richardson. It grades APIs by their
RESTful maturity.

The first level is level 0, the Swamp of POX, or plain-old XML. This level states that the implementing
protocol, HTTP, is used for remote interaction. But, we use it just as that and we don't use anything else
from the HTTP protocol correctly. So for example, to get some altered data, you send over a POST
request to some basic entry point URI, like host/myapi, and in the body you send some XML that
contains info on the data you want. You then get back the data you asked for in the response.

Level 1 is resources. From this moment on, multiple URIs are used and each URI is mapped to a specific
resource, so it extends on level 0 where there was only 1 URI. For example, we now have a URI,
host/api/projects, to get a list of projects, and another one, host/api/projects, followed by an ID, to get
the author with that specific ID. However, only one method like POST is still used for all interactions. So,
the HTTP methods aren't used as they should be according to the standards. This is already one little
part of the uniform interface constraint we see here. From a software design point of view, this means
reducing complexity by working with different endpoints instead of one large service endpoint.

To reach a level 2 API, the correct HTTP verbs, like GET, POST, PUT, PATCH, and DELETE, are used as they
are intended by the protocol. In the example we see a GET request to get a list of projects, and a POST
request containing a resource representation in the body for creating a project. The correct status codes
are also included in this level, i.e. use a 200 Ok after a successful GET, a 201 Created after a successful
POST, and so on. This again adds to that uniform interface constraint.

And level 3, that's hypermedia controls. This means that the API supports HATEOAS, another part of that
uniform interface constraint. A sample GET request to the projects resource would then return not only
the list of projects, but also links, hypermedia, that drive application state. From a software design point
of view, this means it is discoverability and self-documentation. What is important to know is that
according to Roy Fielding, who coined the term REST, a level 3 API is a precondition to be able to talk
about a RESTful API.

So, this maturity model does not mean there's such a thing as a level 1 RESTful API, a level 2 RESTful API,
and so on. It means that there are APIs of different levels, and only when we reach level 3 we can start
talking about a RESTful API.
D) Restful API using ASP.Net Core

RESTful APIs can be built with a variety of technologies. We're going to use ASP.NET Core and the MVC
middleware. In a few words, ASP.NET Core is an open-source and cross-platform framework for building
modern cloud-based internet connected applications, web apps, but also APIs for those web apps,
amongst others. Compare to a previous version of ASP.NET, well, this isn't an update. It was rethought
from the ground up. Now these ASP.NET Core applications allow injecting middleware into the request
pipeline. The ASP.NET Core MVC middleware is one of those. It provides us with a framework to build
APIs and web apps using the MVC, or model view controller pattern, so this is what we're going to use to
create a truly RESTful API. But it's very important to know that we don't just get a RESTful API out of the
box just because we're building an API with ASP.NET Core. That's our responsibility. We get that by
adhering to the constraints. So, we'll use ASP.NET Core MVC to wield our API. But what is that MVC
pattern then regarding an API? MVC is an architectural software pattern for implementing user
interfaces. Different interpretations exist, but in all interpretations, it's used to encourage loose coupling
and separation of concerns that then allows better testability of the app and the ability to reuse parts of
it. It isn't a full application architecture, just like REST. Typically, MVC is used in the presentation layer,
and that's about it, hence the implementing user interfaces part of the definition. But how do we map
this to an API then? Well, an API can also be regarded as a user interface; it's the interface to the
consumer of the API.

It consists of three parts.

The model handles the logic for our application data. A model in this sense can contain code to retrieve
or store data at that level. In some implementations, view models are used as a model. These contain
code to retrieve or store data, but in quite a few implementations the model doesn't contain any logic at
all, it's another component of the application that handles this.

The view represents the parts of the application that handle displaying of data. The view should be
regarded as the representation of our data or resources. Often these days, that is JSON.

And the controller then handles the interaction between the view and the model, including handling
user input. In the API world, this user that interacts with the API is the consumer, typically another
application. If you look at the dependencies between these components, both controller and view
depend on the model, and the controller also depends on the view. That's one of the key benefits of this
separation. In other words, the controller chooses the view to display to the user and provides it with
any model data it requires. So, when a request is made to an API by a consumer, an action on the
controller will be triggered. The controller sends any data that's input, like query string parameters, to
the part of the application responsible for business or data access logic. It then returns a model to the
view, in this case the resource representation in JSON, for example.
E) Outer Facing Contract

The outer facing contract consists of three big concepts a consumer of an API uses to interact with that
API.

 Resource Identifier
The URIs where the resources can be found.
For example: https://host/api/users

 HTTP Method
Resource identifier are combined with HTTP methods like GET, to get resources,

 Payload (optional)
When creating a resource, the HTTP response will contain a resource representation in its
body. The format of those representations is what media types are used for, like application
JSON. The uniform interface constraint does cover the fact that resources are identified by
URIs. Each resource has its own URI, but as far as naming of resources is concerned, there
isn't a standard that describes that, or investigate OData.

Resource Naming Guidelines

A resource name in a URI should always be a noun. In other words, a RESTful URI should
refer to a resource that is a thing, instead of referring to an action.

Shouldn't create a getusers resource, that's an action.

For example: api/getusers

Should create an users resource, that's a thing conveyed by a noun, and use the GET method
to get it.
For example: GET api/users

To get one specific user then,

For example : GET api/users/{userId}

Using these nouns conveys meaning, it describes the resource. That principle should be
followed throughout the API for predictability. It helps a client/consumer understand the
API structure.

If we have another non-hierarchical resource, say projects. A single project then shouldn't
be named as api/projects. It should be api/projects/{projectId}. This helps keep the API
contract predictable and consistent.
 A good APIs that don't pluralize nouns exist as well. Either all resources should be pluralized
nouns, or singular nouns, and not a mix.
Represent hierarchy when naming resources for data or models that have structure.
For example, an user is assigned projects that should be represented in the API contract.
- api/users/{userId}/projects
- api/users/{userId}/projects/{projectId}

APIs often expose additional capabilities such as filtering and ordering resources and they
aren't resources. These can be addressed using Query String.
- api/users?orderby=name

To get number of projects assigned to a user, a good resource URI design.

- api/users/{userId}/totalprojectsassigned

REST stops at that outer facing contract. REST is unrelated to the backend data store, APIs
that use the auto-numbered primary key Ids from the database. By using GUIDs, resource
URIs will stay the same. These GUIDs don't give anything away about the underlying
technology.

Routing

Routing matches a request URI to an action on a controller. So, once we send an HTTP request, the MVC
framework parses the URI, and tries to map it to an action on a controller, and there's two ways it can
do this, convention-based or attribute-based.

For convention-based (see below screenshot), we must configure these conventions.

We can do that by passing in these routing conventions to the UseMvc extension method. This example
would map the URI values index to an index method on a controller named values controller. This is a
typical sample of something that's used when building a web application with views that return HTML.

The MVC Middleware can be used for that, but for APIs the ASP.NET Core team recommends not using
convention-based routing, but attribute- based routing instead. Attribute-based routing, as the name
implies, allows us to use attributes at controller and action level. We provide these with a URI template,
and through that template and the attribute itself, a request is matched to a specific action on a
controller. For this we use a variety of attributes, depending on the HTTP method we want to match.
 Interact Resources with HTTP methods

Different actions can happen to resources at the same URI. For example, getting an User and deleting an
User are interactions with the same resource URI. It's the method that defines the action that will
happen, and depending on the method, we'll potentially need to send or get a payload. It's important to
follow this standard so other components of our application can rely on this being implemented
correctly. With the below table, we know about routing, and about using the correct HTTP methods and
URIs.

Http Method Request Payload Sample Response Payload Remarks

URI
GET - /api/users User collection Reading resources.
/api/users/{userId} Single User
POST Single user /api/users Single User Create resources
PUT Single user /api/users/{userId} Single User or Update resource to do full
empty update.
The request payload is a
representation of the resource
we want to update, including all
fields, and if a field is missing, it
should be put to its default
value.
PATCH JSON PATCH document on /api/users/{userId} Single User or Need partial updates to update
user empty only one or two fields instead of
all of them. The request payload
is a JSON document, essentially
a set of changes that should be
executed on that resource.
DELETE - /api/users/{userId} -
HEAD - /api/users - HEAD is identical to GET with
/api/users/{userId} the notable difference that the
API shouldn't return a response
body, so no response payload. It
can be used to obtain
information on the resource like
testing it for validity, for
example, to test if a resource
exists.
OPTIONS /api/… OPTIONS represent a request
for information about the
communication options
available on that URI. So in
other words, OPTIONS will tell
us whether or not we can GET
 the resource, POST it, DELETE it
and so on. These OPTIONS are
typically in the response
headers and not in the body, so
no response payload.
HTTP Method Request Payload Sample URI Response
Payload
We can use the Route attribute at controller level for that.
For example : [Route("api/users")]

We can refer to that by putting controller in straight brackets in the template.

[Route("api/[controller]")]

That will then match the route api/users to the UsersController. If we were to have refactoring of our
codes, and rename the Controller class to URI, to our authors resource, would automatically change. For
APIs this isn't an advantage. Resource URIs should remain the same and were we to refactor this
controller so it has another name, all our resource URIs would change. The name of the underlying class
is of no importance to the consumer/client, so that's something we want to avoid.

REST guidelines for model

When we designed the outer facing contract, we learned that REST stops at that level. What lies
underneath that outer facing contract is of no importance to REST.

In our case Reliable Collections, used to represent database roles as objects, should be different from
the outer facing model. In some application architectures there's a business layer in-between, which in
turn is different from the outer facing model and the entity model. The outer facing model does only
represents the resources that are sent over the wire in a specific format, but it also leads to possibilities.

Take an User, for example. Concatenating the FirstName and LastName from an entity into one name
field in the resource representation, and sometimes data might come from different places. An User
could have a field, say, Projects, that comes from another API our API must interact with. That alone
leads to issues when using entity classes for the outer facing contract, as they don't contain that field.
Keeping these model’s separate leads to more robust, reliably evolvable code. Imagine having to change
a reliable collection, that would lead to a change of the Entity class. If we're using that same Entity class
to directly expose data via the API, our clients might run into problems because they're not expecting an
additional renamed or removed field. So, when developing, it's fairly important to keep these separate.

AutoMapper

To avoid manual mapping codes between entities and DTOs (Data Transformation Objects), an open
source AutoMapper component can be used. AutoMapper is an object to object mapper. Object to
object mapping works by transforming an input object of one type into an output object of a different
type.
F) Status Codes

The API consumer/client tried to get a resource that doesn't exist should get a 404 Not Found but not
200 OK status codes.

For example:

Status codes and when to use them are part of the HTTP standard, and it's important to get these right
because these status codes are the only thing a client/consumer of the API can inspect to know if a
request worked out as expected, and if something went wrong, whether it's the fault of the consumer or
of the API itself. We're currently returning JSON result (above screenshot) in our code, which sends back
a 200-status code by default, but if we keep doing that for each request, the consumer/client of the API
would assume this request worked out as expected, even if it went wrong. Remember that consumers
of the API are typically non-human, all they can inspect is the status code. They can't read out and
interpret error messages.

Trying to get an User that doesn't exist is a client mistake. That also means client can correct it and try
the request again, and other times it's the server's responsibility, for example when the database
(Reliable Collection) is unavailable.

There's a lot of status codes, and an API doesn't necessarily have to support all of them.

There's five levels of status codes.

Level 100 status codes are informational and weren't part of the HTTP 1 standard. These are currently
not used by APIs.

The level 200 series status codes mean the request went well.

Level 200 Successful GET request

Level 201 Created for a successful request that resulted in the creation of a new resource
 Level 204 Successful request that shouldn't return anything.

Level 300 status codes are used for redirection, for example, to tell a search engine/client that a page
has permanently moved. Most APIs don't have a need for these.

Then there's the status codes that tell the consumer that something wrong, level 400, client mistakes.

Level 401 Unauthorized means that no, or invalid authentication details, were provided.
Level 403 Forbidden means that authentication succeeded, but the authenticated user doesn't have access to
the requested resource.
Level 404 Not found means that the requested resource doesn't exist.
Level 405 Method not allowed. This happens when we try to send a request to a resource with an HTTP method
that isn't allowed. For example we try to send a POST request to api/users, when only GET is
implemented on that resource.
Level 406 Not acceptable, and now we're diving into our presentation media types. For example a
consumer/client might request the application/xml media type, while the API only supports
application/json and it doesn't provide that as a default representation.
Level 409 Conflict then means that there's a conflicted request versus the current state of the resource the
request is sent to. This is often used when we're editing a version of a resource that has been renewed
since we started editing it, but it's also used when creating a resource, or trying to create a resource
that already exists.
Level 415 Means Unsupported media type, and that's the other way around from the 406 status codes.
Sometimes we must provide data to our API in the request body, when creating a resource for
example, and that data also has a specific media type. If the API doesn't support this, a 415-status code
is returned.
Level 422 This one is typically used for semantic mistakes, that's what we get when working with validation. If a
validation rule fails, 422 is what's returned.

And lastly there's level 500, server mistakes. This means the server made a mistake and the client can't
do anything about it, other than trying again later.

These mistakes are then categorized into two categories, faults and errors.

Errors are defined as a consumer of the API, like a web app, passing invalid data to the API, and the API
correctly rejecting that data. Examples include invalid credentials or incorrect parameters, in other
words, these are level 400 status codes and are the result of a client passing incorrect or invalid data.
Errors do not contribute to overall API availability.

Faults are defined as the API failing to correctly return a response to a valid request by a consumer. In
other words, the API made a mistake, so these are level 500 status codes, and these faults, they do
contribute to the overall API availability.
G) Formatters and Content Negotiation

When we think about a RESTful API these days, JSON has nothing to do with REST per se, it just happens
to be the format of the resource representation, and that implies that a RESTful API can also work with
other representation formats, like XML, or a proprietary format.

So that also means that we must be able to let our API know what format we want to get back, and that
brings us to one of the key concepts when working with HTTP requests and responses, and that's
content negotiation.

That's the process of selecting the best representation for a given response when there are multiple
representations available, and it's important. If you're only building an API for a specific client, it might
be okay to always get back representations in JSON format, but if we're building an API for consumption
by multiple clients, some of which we have no control over, chances are that not all these clients can
easily consume JSON representations as a response.

Some might be better off with XML or another format, so how is this done? Well that's what the Accept
header of the HTTP request message is for. An API consumer should pass in the requested media type
like application/JSON or application/xml through this header.

For example, if an Accept header has a value of application/json, the consumer states that if our API
supports the requested format, it should return that format. If it has a value of application/xml, it should
return an XML representation, and so on.

If no Accept header is available or if it doesn't support the requested format, a lot of APIs default to a
default format, like JSON. We could argue that we'd allow our API to serve up responses for requests
without an Accept header with the default representation, but if the client requests a specific format,
that means client also expect to get something back that can parse. If the API returns its default format,
if the requested format is missing, we'd likely end up with a client that cannot parse the response
anyway, so in those cases a 406 Not acceptable status code is warranted.

It's also a best practice to always include an Accept header. Okay, so now how do we do that in ASP.NET
Core? Well this is about output, the format of the resource representation.

ASP.NET Core supports this with output formatters. The consumer of the API can request a specific type
of output by setting the Accept header to the requested media type, but if there's output there's also
input. ASP.NET Core also supports input formatters. An input formatter deals with input, for example,
the body of a POST request for creating a new resource. For inputs the media type of the request body is
identified through the content-type header. These types of headers be part of the self-descriptive
message constraint, which states that each message must contain enough information to process it.
In case of the Accept header, it tells the API what format to return, so how to process the message and
return a response. While the content-type header will tell the API the format it'll have to parse the body
to be able to process the message.

H) Handling Unsupported Media Type

Need to inform ASP.NET Core to support return HTTP not acceptable.

To support XML and JSON, set the output formatter.

By default, ASP.NET core supports both XML and JSON.

Creating Resource and returning Route

Need to return 201 and the location of the newly created resource.
return CreatedAtRoute("GetUser",
new {id = userToReturn.Id},
userToReturn);

The route creates the ID.

Location →http://localhost:6058/api/users/c0422737-a9b0-4e7b-8487-b10b8d9e4260

Blocking unwanted posts based on GUID

A post should not be done based on GUID.

If the client does then it should block explicitly by using route template and return 409 if the user
already exists and 404 if not found.
//Handling POST to a single resource and blocking POST on GUID's
[HttpPost("{id}")]
public IActionResult BlockUserCreation(Guid id)
{
 if (_userRepository.UserExists(id))
return new StatusCodeResult(StatusCodes.Status409Conflict);

return NotFound();
}

I) Purpose of Repository Pattern

The repository pattern is an abstraction that reduces complexity and aims to make the code safe for the
repository implementation, persistence ignorant.

Writing code to access a backing data stored directly in a controller action easily leads to code
duplication. We might need to access Users or Projects from multiple parts in our application, so it's
preferable to write that code just once instead of in every action or part of the application. The
repository pattern helps with that.

And from that, code will also become less error prone, if only for avoiding that application. And then
there's testing, if we want to test the controller action, but the action also contains logic related to
persistence, it's harder to pinpoint why something might go wrong. Is it logic in the action, or is it a
persistence-related code in the action that fails? If there's a way to mock the persistence-related code
and test again, we know that the mistake isn't related to that persistence logic.

So, using the repository pattern allows for this mocking of persistence logic, which means better
testability of the consuming class. Sometimes it's said that through a repository, you can switch out the
persistence technology when needed, and while that is strictly true, it's not really the purpose of the
repository pattern.

What it is very useful for, however, is allowing us to choose which persistence technology to use for a
specific method on the repository. Getting a Project might be easier through Entity
Framework/NHibernate/Reliable Collections, getting an User with some complex logic might be more
advisable through ado.net, or might even call into an external service. For consumers of the repository
like controllers, it's of no interest what goes on in the implementation, rather than switching out one
persistence technology for another for the complete repository, it allows us to choose the technology
that's the best fit for a specific method on the repository, thus persistence ignorant.

One of the advantage of using repository pattern is to use POST and PUT together.

Upserting

Upserting is a principal and allows to create a new resource with Put or Patch instead of Post.

In a lot of systems, it's the server (API) that's responsible for creating the identifier of a resource. Part of
it is often the underlying key in the data store, an integer value or a GUID, for example.

A consumer might post a new User to the Users resource, and the server response with that newly-
created User in the response body, and the location of the User resource, which the server generated in
the location header. In fact, in most systems, the server decides on the resource URI, but REST does not
have this as a requirement.

It's perfectly valid to have a system where the consumer can do this, or where it's allowed for both the
consumer and the server. Now if the key in the database is part of the resource URI, this doesn't just
work with AutoNum fields. It does, however, when working with GUIDs. This is one of the reason to
choose GUIDs instead of integers to store data in the backend store.

Resource URI generation responsibility

Server Consumer/Client
To send an update request (PUT). Need to get the Send the put request to a previously nonexistent
URI for a resource from the server to be able to resource identifier that is valid, because the
update it. The resource must already exist. And if consumer can create it. No longer need to get the
it doesn't, it returns a 404 Not Found. URI from the server.

Use post to create resources and URI is not know Put can be used to creation and this is called
in advance. Upserting.
Post isn't idempotent, sending the same request Put is idempotent. If the consumer of the API
more than once will result in different outcomes. chooses the URI, sending the request once will
create a resource. Sending it again after that will
have the exact same result. The same resource
with the same Id is created.

J) Partially updating a resource

Full updates with PUT aren't always advised, if only for the overhead it creates.

Based on best practices for creating RESTful APIs, PATCH should be preferred over PUT, and PATCH,
that's partial updates. But that's not sufficient, we need a way to pass a change set to a resource using
this HTTP PATCH method. In other words, what should the body of a PATCH request look like? Luckily
there's a standard for this, the JSON PATCH standard (RFC6902). This defines a JSON document structure
for expressing a sequence of operations to apply to a JSON document.

A set of operations that'll be applied to the resource at patch request with that JSON patch document in
the body is sent to. The “application/json-patch+json” media type is used to identify such patch
documents. Each operation is applied after the other one, and a request is only successful when all
operations can be applied.

There's six different operations possible such as Add, Remove, Replace, Copy, Move and Test.

K) Validation in Restful world

When talking about validation, there's essentially following three things are required.

 Define validation rules

 Check validation rules
 Report validation errors
We don't return those errors to let the consumer know whether client or the server made a mistake,
that's what the Status Codes are for. However, often the response body contains a set of validation error
messages that the client application could use to show to the user of the application that consumes the
API.

Most applications require some rules on their objects, be it on a DTO, a business, or a domain object, or
an entity. If those rules don't check out, the request should be halted. To create these validation rules,
we can use ASP.NET Core's built-in approach or a third-party component.

In ASP.NET Core, the default approach to this is using data annotations on properties. For all common
property validation rules, annotations exist, like required or max length. We can also define custom
rules, rules we can't just easily define with annotations.

From a conceptual point of view, what's of importance is what we want to validate. The rule is that we
typically validate rules on input, and not on output, so for POST, PUT, and PATCH, but for not for GET.

Only the input needs to be validated, that's where something can go wrong. For checking validation
rules in ASP.NET Core, the building concept of Model State is used. Model State is a dictionary
containing both the state of the model and model binding validation. It represents a collection of name
and value pairs that were submitted to the API, one for each property. It also contains a collection of
error messages for each value submitted. Whenever a request comes in, the rules we define are
checked. If one of them doesn't check out, the Model State’s valid property will be false, and this
property will also be false if an invalid value for a property type is passed in. So that's what we'll use for
checking our rules.

And that leaves the final step, reporting validation errors. When a validation error happens, the
consumer of the API needs to be notified. It's a mistake of the client, so that warrants a 400 level Status
Code. There's one that's specifically used for this, 422 Unprocessable Entity. As we learned before, the
422 Status Code means that the server understood the content type of the request entity, and the
syntax of the request entity is correct, but it couldn’t process the contained instructions. For example,
the syntax is correct, but the semantics are off. But when we say reporting validation errors, we're not
only talking about the Status Code, we're also talking about response body. What we write out in the
response body isn't regulated by REST, nor by any of the standards we're using, but for validation issues,
this will often be a list of properties and their related error messages.

In APS.NET CORE MVC, typically the Model State’s validation errors are serialized, so we end up with a
list of property names and related validation errors. But this can vary depending on what we're building
the API for. Third party validations such as https://fluentvalidation.net/ can simplify writing rules. The
errors can be logged using providers available at https://github.com/aspnet/Logging. The validation
errors can be saved on disk in a file using NLog https://github.com/nlog/NLog/wiki/Configuration-file .

L) Paging Collection Resources

Most APIs, expose collection resources. For example, that's Tags and Signals, and these collection
resources can grow quite large.
It's considered best practice to always implement paging on each resource collection or, at least, on
those resources that can also be created. This is to avoid unintended negative effects on performance
when the resource collection grows.

Not having paging on the list of ten Tags might be okay, but if our API allows creating Tags, this list can
grow, and we don't want to end up with accidentally returning thousands of Tags in one response, as
that will have an adverse effect on performance.

When we design the outer-facing contract, options like paging, sorting, filtering, and others are passed
through via the query string. These are not resources. They're parameters that manipulate a resource
collection that's returned. For example: https://host/api/tags?pageNumber=1&pageSize=5

As far as paging is concerned, the consumer should be able to choose the page number and page size,
but that page size can cause problems as well. If we don't limit this, a consumer can pass through
100,000 as page size, which will still cause performance issues. The page size itself should be checked
against a specified value to see if it isn't too large, and if no paging parameters are provided, we should
only return first page by default. Always page even if it isn't specifically asked for by the consumer.

If we have thousands of Tags in our database, and we first return all those Tags from the repository to
the controller and then page them, well, we're still fetching way too much data from the database.
We're not really helping performance in that regard then, but how can this happen technically, with
LINQ and Entity Framework Core/NHibernate? The principle named deferred execution can be used.

Principal of Deferred Execution

When working with Entity Framework Core/NHibernate, we use LINQ to build queries. With deferred
execution, the query variable itself never holds the query results, and only stores the query commands.

Execution of the query is deferred until the query variable is iterated over. Deferred execution means
that query execution occurs sometime after the query has been constructed.

We can get this behavior by working with IQueryable implementing collections. IQueryable<T> allows us
to execute a query against a specific data source, and while building upon it, it creates an expression
tree. That's those query commands, but the query itself, isn't sent to the data store until iteration
happens. Iteration can happen in different ways. One way is by using an IQueryable in a loop. Another, is
by calling into something like ToList(), ToArray(), or ToDictionary() on it because that means converting
the expression tree to an actual list of items. Another way is by calling singleton queries. Singleton
queries are queries like average, count, and first, because, to get the count, or first item of an
IQueryable, the list must be iterated over. If we can avoid that, we can build our query by, for example,
adding take and skip statements for paging and assure it's only executed after that. Exactly what we'll
need in this module. Let's put it into practice in the first demo on paging.

Page Metadata

It would be useful for the consumer of the API to know about a page set of data. At the very least, we
should include URIs to request the previous and next pages, so the consumer doesn't have to construct
those himself. Additional information, like total count or total amount of pages, is often included as well,
just like page number and/or page size.
How do we include this data in a response then? Third party APIs include the paging information in the
response body itself. Often with a metadata tag or paging info tag, but this isn't correct.

If a consumer of the API requests a page of Tags, with the application/json media type as value for the
accept header, the API should return a JSON representation of the resource. An envelope that has a
results field and metadata field, well, that doesn't match the media type we asked for. It's not a JSON
representation of the Tags collection; it's a different media type. Returning the paging info like this
combined with application/json as media type, effectively breaks REST, as we are no longer adhering to
the self-descriptive message constraint. That constraint states that each message must contain enough
information on how to process it. Next to returning the wrong representation, the response will have
content-type pattern with value application/json, which doesn't match the content of the response. In
other words, we also don't tell the consumer how to process it. The consumer doesn't know how to
interpret the response judging from the content-type.

When requesting application/json, paging information isn't part of the resource representation. Its
metadata related to that resource. Metadata, well, that's something that's put in a response header.
The consumer can empower that header to get that information. We should create a custom pagination
header, like X-Pagination. For example:
XPagination →{"totalCount":6,"pageSize":5,"currentPage":1,"totalPages":2,"previousPageLink":null,"nextPageLink":"
http://localhost:6058/api/tags?pageNumber=2&pageSize=5"}

M) Filtering and Searching

Filtering a collection resource means limiting the collection resource, considering a predicate. We're
passing in the field name and the value we want that field to match, to make it part of the collection
that will be returned. Filter is applied on fields of resource, not on field of lower level objects.

For example: http://host/api/users?role=administrator

Searching goes beyond that. For searching, we don't pass in a field name that should match, we pass in a
value to search for, and it's up to the API to decide which fields shall be searched for that value. Often,
that's done with full-text search, and it's useful for cases where basic filtering isn't powerful enough.

For example: http://host/api/users?searchQuery=User1

Filtering and searching are different. As far as the rest is concerned, two things are important. First, filter
and search options are sent to the API via the query string.

N) Sorting

Resource Fields

http://host/api/users?orderBy=role

Take direction into account

http://host/api/users?orderBy=role desc

Sorting multiple fields

http://host/api/users?orderBy= role desc, name

Property Mapping Service for Sorting

Need to create a propertyMappingService and register it with the ASP.Net Core's built-in container.
IPropertyMappingService, that allows us to register it on the container on interface instead of directly
on implementation.

O) HATEOAS (Hypermedia as the engine of the application state)

HATEOAS makes APIs evolvable and even self-describing. Hypermedia like links drives how to consume
and use the API. It tells the consumer how it can interact with the API.

Can the API delete the resource? Can the API update the resource? Can the API create the resource? Can
the API access the next page of data? And so on. But, what problem does it solve?

Well, imagine we want to allow the consumer of the API to update or delete a Project. Currently it does
not know how to do that by looking at the response body. It must have some intrinsic knowledge of how
our API contract is structured. From contractor documentation, the consumer can learn that it must
send a Project or batch request to API, users, user ID, projects, project ID, to update a Project with a
specific payload. And delete request to delete it. But it cannot get this information from the response
itself. The consumer must know this in advance.

Now imagine another case. Let's say we have an additional field, NumberOfBulkLicenses available for
project. If there are still licenses in the Project, somebody can reserve a license. This is obviously an
optimistic example, set in the hopefully near future. Anyway, in this case, the consumer of the API will
have to know it has to inspect that field before it can send a POST request to, say, api/projectLicenses'.
And that's again intrinsic knowledge of how the API works.

Say an additional rule is to be implemented. This rule will effectively break the consumers of the API.
They must implement that on the client side as well. In short, if an API doesn't support HATEOAS the
consumers of the API have to know too much. And if the API evolves, this might break the consuming
applications because the assumptions made by those applications can become invalid. And that's the
issue that HATEOAS solves.

A lot of these assumptions can be thrown overboard by including links in the response that tell the
consumer which action on the API is possible. And those links, well, that's hypermedia. So HATEOAS
allows the API to truly evolve without breaking consuming applications. And that in the end results in
self-discoverable APIs. The client would just have to inspect these links.

Roy Fielding quote who invented REST. "You can't have evolvability if clients have their controls baked
into their design at deployment. Controls have to be learned on the fly. And that's what hypermedia
enables."

The logic for creating the links can't just be automated, as it will depend on the business rules. The links
should appear at the consumer only when rules are met. Essentially, there's two approaches.
The first one, a statically typed approach, involves working with base (with links) and wrapper classes.
The Data Transformation Objects class inherits a class that contains links

Second approach is a dynamically typed approach. It involves working with anonymous types and
dynamics. For example, an ExpandoObject.

HATEOAS and Content Negotiation

Are these links part of the resource or do they describe the resource, i.e. are they metadata? Same goes
for the other links, links to self and so on. If it's metadata describing the resource, they should go in the
header.

If they are explicit parts of the resource, they can go in the response body.

What we're dealing with here are two different representations of the resource. With content
negotiation, we ask through the accept header to get back a JSON representation of the resource.

Additional semantics, the links, on top of the JSON representation. So, the links should not be part of the
resource when we ask for media type application/json. We've effectively created another media type
which were wrongly returning when asking for application/json. So, by doing this, we're breaking the
self-descriptive message of constraint, which states each message must include enough info to describe
how to process the message. If we return a representation with links with an accept header of
application/json, we're not only returning the wrong representation, the response will have a content-
type header with value application/json, which does not match the content of the response. So, the
consumer of the API does not know how to interpret the response judging from the content type. In
other words, we also don't tell the consumer how to process it.

The solution is creating a new media type. There's a format for this, a vendor-specific media type. For
example: application/vnd.iacc.hateoas+json First part after application is vnd, the vendor prefix. That's a
reserve principle must begin the mime type with. It indicates that this is a content type that's vendor
specific. It's then followed by a custom identifier of the vendor and possibly additional values. What
we're stating here is that we want a resource representation in JSON with HATEOAS links. If that new
media type is requested, the links should be included. The consumer must know about this media type
and how to process it. If this media type isn't requested, so we simply request application/json, the links
should not be included.

HATEOAS Media Types

It defines the presentation of a resource. It's a central principle in the RESTful world. For example: We
have different representations for getting an User and creating an User. We just used application/json as
media types for both, as most APIs do. But, in fact, these are different representations of the same
User's resource. Using application/json is a mistake. That just tells us we want data format to be JSON.
But we need to be more specific. So how do we correct this? By using media types. Instead of using
application/json, you could create vendor-specific media types for these representations.
Versioning in RESTful world

When we talk about versioning, we're talking about adapting to change. An API changes. What we can
do with the API changes. We may be allowed to create an User today, but we might not be tomorrow.
Or resources might be added or removed. Those are functional changes.

And then there's changes in resources or representations. Sometimes, fields are added or removed. We
need to be able to adapt to this without breaking consumers of the API. There's various ways to version
an API.

One way is to change the URI for each version. So, v1 would have v1 in the URI. Version two would have
v2, and so on. For example: api/v1/users

Another way is by working with query string parameters. The requested version is added as a query
string parameter. For example: api/users?api-version=v1

And yet another way is through a custom header. For example: “api-version”=v1

Even versioning can be represented using “application/vnd.iacc.hateoas.v1+json”.

P) Cacheable

The caching is quite a big topic as it involves infrastructure and requires separate investigation. What is
mentioned here is just the basic version.

The cacheable constraint states that every response should define itself as cacheable or not. We can
simply add a response header stating the response isn't cacheable, and that's perfectly valid.

For this, we'll use HTTP Caching, which is part of the HTTP standard, RFC 2616, and it's described in full
in RFC 7234. HTTP Standard says: Caching would be useless if it did not significantly improve
performance. The goal of caching in HTTP/1.1 is to eliminate the need to send requests in many cases
and to eliminate the need to send full responses in many other cases.

Eliminating the number of requests that must be sent reduces the number of network-roundtrips
required for many operations. HTTP cache uses an expiration mechanism for that purposes.

Eliminating the need to send full responses reduces network bandwidth requirements. And for that, a
validation mechanism is used. It does depend on where the cache lives on what exactly is reduced by a
specific mechanism.

We're working with HTTP, and the cache is a separate component. It sits between the client application,
the consumer of the API, which makes the requests, and the API itself. The cache must accept requests
from the consumer and pass them to the API. It will also receive response messages from the API, and it
will store responses that are deemed cacheable and forward them to the consuming application. If the
same response is requested again, the cache can send the cached response.

So, we can look at the cache as the middle-man of the request-response communication between the
consuming application and the API.

In HTTP, there are three cache types. One is the client cache, or browser cache. It lives on the side of the
client application. A browser is an HTTP client, but we can extrapolate that. If we're building, for
example, a mobile client application with Xamarin, that client application, which communicates with the
API through an HTTP client instance will often have a private cache. It's called private, because the
resources aren't shared with anyone else. They're kept on the client for each consuming instance. So,
good example to our cache on your mobile device, but also a cache in local storage for an angular
application.

Second type is the gateway cache. This is a shared cache. The resources it caches are shared across
different applications. This type of cache is used on the server-side. Other names for this are reverse-
proxy caches, or even HTTP accelerators.

The third type is the proxy cache. It's also a shared cache, but it doesn't live with the consuming side nor
at the side of our API. It lives somewhere on the network. They are often used by large corporations and
ISPs to serve thousands of users. For example: Browser setting by enabling proxy address.

Expiration

The expiration model is a way for the server to say how long a requested resource, so the response
message, is considered fresh. A cache can then store this response, so subsequent requests are served
from the cache if it's still fresh.

For this, two response headers are used.

The first one is the expires header. It contains an HTTP date, stating at one time the response expires.
For example - Expires: Wed, 8 Aug 2018 16:00:00 GMT. It has a potential issue. The clocks between the
cache and the server must be synchronized. It also offers little control over what type of response can be
cached, when, and where.

The expires header is superseded by the cache-control header, which addresses these limitations. For
example- Cache-Control:public, max-age=60. This means this response header contains two directives,
public and max-age. Max-age states that the response can be cached for 60 seconds, so clock sync is no
longer an issue. And public states that it can be cached by both shared and private caches. So, the server
can decide on whether a response is even allowed to be cached by a gateway or proxy cache. It's just
the preferred header for expiration. There's quite a few possible directives next to public and max-age.

Imagine we have a client application, the consumer of our API, a cache, be it shared or private, and our
API, the back-ends. The application sends a request to get the Users to request it to cache. There's no
cached version of the response available, so the cache forwards the request to our API. Our API should
then return a response, including a Cache-Control header. For example: Cahe-Control:max-age:1800.
Ten minutes later our app sends a request to the User's resource again. We hit the cache, and this time
that cache has a response that hasn't expired. It sends back that response, including an age header. In
this case, that age is 10 minutes or 600 seconds. A request to the back-end is thus eliminated. Only
when the cached version reaches expiration time, which means it becomes stale, the cache will hit the
back-end again. If this cache is a private cache, like one that lives in local storage for a Desktop app or on
a mobile device for a mobile app, that's where this stop. But if we assume this cache is a shared cache,
living on the server, for example, it's a bit different. Ten minutes later, a subsequent request from
another instance of our client application, or another client application that consumes our API, is sent to
get the Users. It will hit that cache, and assuming your response hasn't expired, it will serve that up from
the cache, this time with an age header of 20 minutes, or 1200 seconds. So, there's again no
communication with our API.

Two types of caches are available. If it's a private cache, subsequent requests save bandwidth and
reduce requests to the API. If the response hasn't expired, we don't hit the server anymore for requests
coming from the same client instance. But of course, when different clients and the same request, they
still hit our back-end, as they all have their own private cache.

If it's a shared (public) cache, subsequent requests don't save bandwidth between the cache in our API,
but it does drastically eliminate requests to our API. Imagine there's 1000 simultaneous consumers our
API. The first one hits a specific resource on our API, and all the other ones who want to get that same
resource hit the cache, which also means a lot less code must be executed.

Validation Model:

Validation is used to validate the freshness of a response that's been cached. When server cache data as
a stale entry that it would like to use as a response to a client's request, it first must check with the
origin server, or possibly an intermediate cache with a fresh response, to see if its cached entry is still
usable. To be able to validate, we need something to, well, validate against. And that's validated.

These validators are used to check if they represent the same or different responses. There are two
types of validators, strong validators and weak validators.

A strong validator changes if the body or headers of the response change. A typical example of such a
validator is an ETag, or Entity Tag, sent through via the ETag response header. For example : ETag :
“123456789”. Such an ETag is an opaque identifier assigned by a web server, or API, to a specific version
of a resource. Strong validators can be used in any context with caching, but also for concurrency checks
when updating resources.

A weak validator doesn't always change when the resource changes. It's up to the server to decide when
a change is warranted. For example, only change on significant changes, not when less significant
aspects change. A typical example is the last-modified response header, containing the time when the
resource was last modified. These are implicitly weak. We've got a possible 1 second gap. Now, it is
possible to make these strong by adhering to a set of rules, like an additional date value on cached entry
that's at least 60 seconds after the last modified date, among others, but that would lead us too far. So
we treat these as weak. But even then, the clocks must be synchronized, so it suffers the same issues as
the expires header, which is why ETags are a better option, and there's something like a weak ETag as
well. When it's post-fixed with w/, it's treated as being weak. It's up to the server to decide this. When
an Entity Tag is in this format, it doesn't necessarily change if the response changes. Weak validators can
only be used when an equivalent resource is good enough, but equality isn't required.

If it's a private cache, subsequent requests save bandwidth. We must communicate with the API, but we
don't need to send over the full response from the API, only a 304 Not Modified if the response is the
same.

If it's a shared (public) cache, subsequent requests don't save bandwidth between the cache and the
client, but it does save bandwidth between the cache and the server as the server does not have to
regenerate the response and resend it if it's still valid. So, two different strategies, two different uses.
Often, expiration and validation are combined. This combination works in different way for Private cache
and Shared cache.

Private cache:

A new request is sent, and if the response hasn't expired, the cached response can be used. This reduces
communication with the API, including regenerating responses, and it reduces bandwidth requirements;
we don't even hit the API. Once it expires, we do hit the API. If we only had expiration checks, this
means the API would have to regenerate the response. But with the validation checks, we can
potentially avoid this, because it's not because the response is stale that it's invalid. So, the API checks
the validators, and if the response is still valid, it sends back a 304 Not Modified. Bandwidth usage and
response generation is thus potentially reduced even more. Because even an expired response doesn't
necessarily result in a response body.

Shared Cache:

If the cache is shared, the cached response is used if it hasn't expired. This doesn't result in bandwidth
reduction between client and cache, but it does result in bandwidth reduction between cache and API,
and dramatically reduces the number of requests to the API. A lot more than a private cache, as this one
is shared across potentially unlimited client instances. If the response expires, the cache must
communicate with the API. But here as well it doesn't mean the response has to be regenerated. If
validation checks out, a 304 Not Modified is sent to the cache without a response body. That potentially
reduces bandwidth usage between cache and API, and response body generation. The new response
must still be sent to the client from the cache, of course. Combine these two caches, a private cache for
each client instance, and a shared cache at server level and with each the Holy Grail of caching.

Cache-Control Directives:

There's quite a few cache-control directives available that control caching, both a request side and a
response side.

Response cache-control directives

a) Freshness directives represent how long a response can be considered. Directive “max-age”
defines the maximum age after which a response expires in seconds. Directive “s-maxage”
overrides the maxage value for shared caches. So, a response can expire differently in a private
cache than in a shared cache.

b) Cache directive relate to cache location. Directive “public” indicates that the response may be
cached by any cache, private or shared. And directive “private” indicates that all or part of the
response message is intended for a single user and must not be cached by a shared cache.

c) Validation directive related to validation. Directive “no-cache” indicates that the response
should not be used for subsequent requests without successful revalidation with the origin
server. Directive “must-revalidate”, the server can state that if a response becomes stale, i.e. it's
expired, well, then revalidation must happen. This allows the server to force a revalidation by
the cache, even if the client has decided that stale responses are okay. Directive “proxy-
 revalidate” is the same as must-revalidate, but it doesn't apply to private user age in caches, like
browser cache.

d) Other directives such as no-store and no-transform. Directive “no-store” states that the cache
must not store any part of the message, mostly used for confidentiality reasons. Directive “no-
transform” states that the cache shouldn't convert the media type of the response body.

Now, all of these are decided by the server, but the client can override some of these settings,
or rather use the same directives to tell the cache or server what it's willing to accept.

Request cache-control directives:

a) Freshness Directives: Directive “max-age” indicates that the client is willing to accept a
response whose age is no greater than the specified time in seconds. Directive “min-fresh”
indicates that the client is willing to accept a response whose freshness lifetime is no less
than its current age, plus the specified time in seconds. That is, the client wants a response
that will still be fresh for at least a specified number of seconds. And directive “max-stale”
indicates that the client is willing to accept a response that has exceeded its expiration time.
b) Validation Directive: There's only one related to validation, and that's a “no-cache” directive,
stating that the response list request should not be used for subsequent requests without
successful revalidation with the original server.
c) Other Directives: Directives “no-store” and “no-transform” are the same as for the response
headers. Directive “only-if-cached” states that a client wants the cache to return only
responses that it currently has stored, and not with the origin server. This one is typically
used when there's a very poor network connection.

Supporting cache headers:

To implement support for HTTP caching, according to REST constraints, we'll need a component that
generates the correct response headers for us and checks the headers that are sent through with a
request, so it can only turn a 304 Not Modified (use for Caching) or 412 Preconditioned Failed (use for
Concurrency). This component does live at the back-end behind those cache stores.

In ASP.NET Core, there is a built-in [ResponseCache] attribute, which can be used on controller actions.
It generates a cache control header and it doesn't do anything as far as storing the responses in a cache
is concerned, so it's not a caching store. It also doesn't work with ETags. Not sufficient as per REST
constraints.

In the old ASP.NET, there's a popular component called CacheCow.Server that does handle expiration
and validation, and it works with ETags. It's both generated of cache-control and ETags, and a cache
store.

A new open source project Marvin.Cache.Headers is available at GitHub. This middleware for ASP.NET
Core that adds http cache headers to responses, cache control, expires, ETag, and last modified, and
implements cache expiration and validation models. There's a reason it's preferable to separate
generation of the headers from the cache store. Those ETags are also used for other purposes, like
concurrency. Both are often used together.
The following example uses Cache-Control, Age, ETag and Expires when caching is enabled in the Web
API.

Cache Stores:

This is a component that's responsible for storing those responses and returning them if they're not
expired or invalid; it also checks with the back-end for validation. There are a few options we have,
depending on the framework used to build an application, and whether we want a private or shared
cache store. Private stores live at the client, so at the side of the consumer.

Some cache clients can act as both private or shared caches, depending on where they're used.
CacheCow.Client is one of those. It's positioned as a library for implementing HTTP caching on both
client and server in ASP.NET Web API. It uses message handlers on both client and server to intercept
requests and responses and apply caching logic and rules. It does only work with the full .NET
framework, so no PCL, and no .NET Core. As far as shared cache stores that do work with .NET Core are
concerned, Microsoft ResponseCaching middleware. This cache store takes into account the headers the
marvin.cache.headers middleware generates, and it will save responses in the cache and serve them up,
depending on the headers. Note that it does not generate these headers itself.

Concurrency:

Okay, so we can now generate the correct headers. But this isn't the cache store yet. We're okay as far
as REST is concerned, but we are not actually saving the responses anywhere. For that, we need a cache
store. Let's start with an example. Say two client applications or two users, User1 and User2, are
working with a specific resource at the same time. For example, they are working on the same Project.
User1 gets the project from the API, and a bit after that, User2 gets the same project from the API.
User1 edits the project and issues a PUT request. The project has been updated. Now User2, who is
finally done editing, issues a PUT request as well. At this moment, User1 changes are lost. They are
overwritten with User2's changes, and that's a problem. Issues like these, concurrent updates, are
handled with a concurrency strategy. Pessimistic or optimistic concurrency control.

Pessimistic concurrency control implies that the project would be locked for User1. If it remains locked,
no one else but User1 would be able to modify it. So, the name pessimistic implies that there will be
some sort of fight over the resource. Pessimistic concurrency control isn't possible in REST, as one of the
constraints is statelessness. So, we cannot use this.

But there's also optimistic concurrency control. This implies that User1 gets a token that allows him to
update the project. If a token remains valid, User1 will be able to perform the update. So, optimistic
implies that there won't be an issue, but if there is, the token will tell you that. Optimistic concurrency is
possible. And that token is actually a validator. A strong validator is required for handling issues like
these, so we can use strong ETags for that.
 Figure 1: ETag for Caching and Concurrency

Q) Rate Limiting and Throttling

Rate limiting means limiting the number of allowed requests to API (or a specific resource). It can thus
protect the API against unintended usage scenarios, like too many requests that can result in
deteriorated performance for all clients of the API. A throttle in this regard is then used to control the
rate of requests that are allowed to access API. It determines whether or not a specific request is
allowed.

For example, a specific client might have a throttle of 100 requests per hour. This can be limited even
further or combined, for example, by stating that a specific IP is allowed 1000 requests per day, but at
the same time it's limited to 100 requests per hour. Another example would be limiting the total amount
of requests to the API for a given IP to 1000 a day, of which only 50 are allowed to the project's
resource.

Now for caching there were cache-related headers on the responses that tell the consumer/client, or an
in-between cache, how to handle those. Likewise, response headers are used for rate limiting, even
though they are not part of the HTP standard. A response to a request should contain rate limiting
information in the response headers. They all start with X rate limits.

Allowed Requests:

X-Rate-Limit-Limit header: It adds limit and contains the time period for which a limit is valid.

X-Rate-Limit-Remaining header: Remaining provides the client with info on the remaining number of
requests before the limit is reached.

X-Rate-Limit-Reset header: Reset provides information on when the limits will be reset.

Disallowed Requests:

If the limits are hit, we need to return something as well. Those responses warrant a 429 - Too many
requests status code. Retry-After response header, and the response body should contain details
explaining the condition.
R) Summary

Method Safety and Idempotency

A method is considered SAFE when it does not change state (resource) even though it is called multiple
times.

A method is considered IDEMPOTENT when it can be called multiple times which return the same result.

HTTP METHOD SAFE IDEMPOTENT

GET Yes Yes
POST No No
DELETE No Yes
PUT No Yes
PATCH No No
Method safety and idempotency help us to decide which method to use for which use case. For
illustration purpose, User Management controller is taken.

HTTP Method by Use Cases

Remarks: Filtering, Searching, Sorting, Page Metadata and Caching are applied to the method without
overloads.

 Reading Resources

Collection of Users:
HTTP Method: GET
HTTP URI: api/users
HTTP Status Code: 200 (Success), 404 (Failure)
HTTP Response Body: [{user}, {user}]

Specific User:
HTTP Method: GET
HTTP URI: api/users/{userId}
HTTP Status Code: 200 (Success), 404 (Failure)
HTTP Response Body: {user}
 Deleting Resources

Deleting User:
HTTP Method: DELETE
HTTP URI: api/users/{userId}
HTTP Status Code: 204 (Success), 404 (Failure)
HTTP Response Body: None

Deleting collection of Users:

HTTP Method: DELETE
HTTP URI: api/users
HTTP Status Code: 204 (Success), 404 (Failure)
HTTP Response Body: None
Remarks: Rarely implemented as highly destructive

 Creating Resources (Resource ID generated at Server)

Creating User
HTTP Method: POST
HTTP URI: api/users
HTTP Request Body: {user}
HTTP Status Code: 201 (Success) with response body and Location header, 404 (Failure)
HTTP Response Body: {user}
HTTP Location Header: api/users/{userId}

Creating User with resource id

HTTP Method: POST
HTTP URI: api/users/{userId}
HTTP Status Code: 404 (User does not exist) or 409 (Conflict if user exists)

Creating User collections

HTTP Method: POST
HTTP URI: api/usercollections
HTTP Request Body: {userCollection}
HTTP Status Code: 201 (Success) with response body, 404 (Failure)
HTTP Response Body: {userCollection}

 Creating Resources (Resource ID generated at Client)

Creating User with resource id

HTTP Method: PUT
HTTP URI: api/users/{userId}
HTTP Request Body: {user}
HTTP Status Code: 201 (Success) with response body. No 404 as resource will be created as User
is not found.
HTTP Response Body: {user}

Creating User with resource id

 HTTP Method: PATCH
HTTP URI: api/users/{userId}
HTTP Request Body: {user}
HTTP Status Code: 201 (Success) with response body. No 404 as resource will be created as User
is not found.
HTTP Response Body: {JsonPatchDocument on User}

 Updating Resources (Full)

Update a specific user

HTTP Method: PUT
HTTP URI: api/users/{userId}
HTTP Request Body: {user}
HTTP Status Code: 200 (Success) with response body or 204 (Success), 404 (Not Found)
HTTP Response Body: {user}

Update a collection of users

HTTP Method: PUT
HTTP URI: api/users/
HTTP Request Body: [{user}, {user}]
HTTP Status Code: 200 (Success) with response body or 204 (Success), 404 (Not Found)
HTTP Response Body: [{user}, {user}]
Remarks: Rarely implemented

 Updating Resources (Partial)

Update a specific user information partially

HTTP Method: PATCH
HTTP URI: api/users/{userId}
HTTP Request Body: {JsonPatchDocument on user}
HTTP Status Code: 200 (Success) with response body or 204 (Success), 404 (Not Found)
HTTP Response Body: {user}

Update a specific user information partially

HTTP Method: PATCH
HTTP URI: api/users/
HTTP Request Body: {JsonPatchDocument on users}
HTTP Status Code: 200 (Success) with response body or 204 (Success), 404 (Not Found)
HTTP Response Body: [{user}, {user}]
Remarks: Rarely implemented
S) Patterns to centralize access to Microservices

Need for API gateways to access our API based microservices:

Microservices architecture gives us the ability to create software which is module and
decoupled and individual components can be independently deployed and changed. However,
with these advantages we inherit some complexities and these complexities can be resolved by
providing centralized access to our microservices infrastructure using an API gateway.

The following are the complexities in distributed microservices:

Distributed Functionality Complexity:

The first complexity is the fact that functionality within system is now distributed. Previously, for
example, we would have called one monolith API to update Topology device records and update
our Signal records to different endpoints on the same monolith API. With microservices,
however, we might have to call multiple APIs in the background to carry out the same action. So,
with microservices we need to remember all the different locations of the APIs that provide the
distributed functionality.

Distributed Data Complexity:

Each microservice will have a small scope and a micro database which contains data related to
that scope. And we have the same complexity here, instead of remembering one monolith API
or database in terms of location, we now need to remember several different locations of
several different microservices to extract the same data.

Distributed Security Complexity:

With microservices we also inherit complexity from the fact that our security is now distributed.
Instead of our client applications authenticating once with one monolith API, we now must
authenticate each time we access one of our APIs within our microservices architecture. So, if
we have multiple APIs for data and functionality, each one of those APIs will require
authentication and authorization. So, our security also becomes distributed across our
microservices architecture.

Client Complexity:
All these complexities that we inherit must be managed and maintained by the client's
applications, the client's applications that consume microservices.

Client consolidating API data and functionality Complexity:

And for the client application, this complexity will include remembering different APIs for
different pieces of data and remembering different APIs for difficulty functionalities and then
bringing it all together and consolidating the data and functionality.

Client managing Security Complexity:

And on top of this, the client application will have to remember the authentication details for
each microservice API it accesses in the background. Not only is this a nightmare in terms of
configuration, because we need to have a record which states how to access an API in terms of
credentials, but this also complicates the roundtrips required to get authentication for each one
of the APIs need to access. We'll basically need to make several roundtrips to get the
authentication for each API that need to access. If the client application that uses microservices
architecture is an internal client application, we might choose to live and manage these
complexities because it's an internal client application.

Hopefully from these complexities we can see why there's a need for having centralized access
to microservices architecture. Centralized access means providing a central component which
can be used to access the microservices instead of directly accessing individual microservices
and their individual endpoints directly. The design pattern this centralized access component
falls under is the API gateway pattern. So as the name suggests, this is the centralized gateway
that clients use to talk to microservices instead of accessing microservices endpoints directly.

API Gateway Pattern:

The API gateway is a component within microservices architecture. It needs to be scaled out and
have the option to either develop it in house or to use third party options.

An API itself (Restful API + Endpoints):

At a mechanical level, the API gateway is likely to be an API in itself and it's consumed by client
applications, like any other API, and it will also follow the conventions and standards of typical
API, and the recommended style for an API gateway is to adopt the RESTful API style, which
works over HTTP, and this will provide the most conventional interface for client application to
integrate with. The API gateway will provide several endpoints. And the API gateway is
configured to take incoming traffic from these endpoints and then pass the traffic on to actual
microservices.

Security (Authentication):
An API gateway can be configured to authenticate all incoming connections to its endpoints and
it can be used to take away the security and the authentication responsibility from the upstream
backend APIs. So not only can the API gateway be configured to authenticate and reject any
unknown connections to the gateway itself, but there are other ways it can control the traffic as
well.

Traffic Control:
The API gateway can be configured to only accept traffic from specific external IP addresses.
Can control the traffic in terms of frequency and can cap the number of calls accepted at the API
gateway if the external client exceeds a certain agreed threshold.

Load Balancing:
The API gateways can also be configured to do load balancing. So, for a specific API, in the
background, we might have multiple instances and the API gateway can be configured to load
balance the traffic between all the instances.

Logging and monitoring:

The API gateway can also be configured to log and monitor all the incoming traffic.

Data Transformations:
The API gateway can also be configured to do transformations, so for example, an upstream API
might respond in XML, but the API gateway then converts that data to JSON format for the
consuming client.

Circuit Breaker:
An API gateway can also be configured to provide circuit breaker functionality. So, for example,
if an upstream API instance fails, the API gateway can detect this, and it will then stop
forwarding traffic to that one specific instance.

Caching:
To improve performance an API gateway can also be configured to cache result sets, so these
are result sets it receives from upstream APIs. So, when a consumer calls an API gateway
endpoint, it might check its local cache to see if it can satisfy the query itself instead of calling
the upstream API, and overall this will improve performance.

An API gateway can do a lot more than just providing centralized access to microservices, and
what features an API gateway provides depends on how much time is invested in developing API
gateway and if we take the third-party option, it depends on what types of plugins and add-ons
the third-party option supports.

Figure 2: API Gateway Core Functionality

T) Microservices Security

1) Different Applications Security

Thick clients (WinForms or WPF applications) / Not Service-based /Tier Applications:

Windows credentials were typically used if identity-related information was required. The user would
log in on his or her PC on the domain of his or her plant and that was sufficient.

Server-side web applications:

And for internet web apps, Windows or Forms authentication was used with an in-application login
screen. These web applications were quite easy to secure, all nicely living on the same domain so the
cookie with the authentication ticket could be shared for the rudimentary single sign-on implementation
if needed. That's a strategy that's advised against today.

Service-based applications (Windows communication foundation):

When we build service-oriented applications often with the help of frameworks like Windows
communication foundation, but even then, it is about thick clients inside a company domain or server-
to-server communication from a server-side ASP.NET webforms or MVC app to a WCF service. With WCF
taking care of most of the hard work through the WS-Security standard, it wasn't that much more work
to keep that secure.

All apps nicely hosted on the same domain and/or running in the same application pool. Often, this was
then combined with IP level configuration on the firewall. Only requests from Server A would be allowed
to Server B, for example.

SAML 2.0:
A popular protocol when the exchange of authorization and authentication data between security
domains was required is SAML 2.0. That one is still used a lot, even these days, especially for server-side
web applications in enterprise environments, but it's not such a good fit for more modern application
architectures.

Building RESTful APIs or HTTP APIs:

We started building RESTful APIs or HTTP APIs. Those no longer necessarily live in the same domain and
sometimes they weren't even under our control. We started creating applications that integrated with
different APIs.

A client application could talk to our own API, but at the same time talk to third party API integration to
name one example and those client applications changed as well. It is not limited to server-side web
apps or thick clients that live inside company walls anymore. The client-side web apps with Angular, for
example, and communicate with our APIs directly. We can no longer protect our APIs by keeping them
inside of the company walls safe for using something like a VPN. A mobile application or an Angular
application runs at the client level.
So, if a client talks to an HTTP API, that API must be public. It cannot be secured with forms
authentication anymore. A mobile application doesn't even have a notion of that and server-side web
app doesn't necessarily only talk to in-company APIs. We might decide to host the API outside of
company walls and monetize it or the app might need access to other APIs and sending a
username/password combo over the wire on each request quickly proved a bad idea. All that's required
to steal password is a packet sniffer and chances are that with that password, an attacker will gain
access to much more than just this one application.

In the era of token-based security. Instead of sending over username/password combinations on each
call to the API, we're now sending tokens. These tokens represent consent. For example, consent
granted by the user to the client application to access an API. The token can then be used on the client
or can be passed to the API used at that level to authorize access.

But how do we create those tokens and how do we safely deliver them to the applications that require
them?
Typically, a call to a self-created login endpoint that would accept our username/password and return a
JSON web token. It's an improvement.
We're no longer sending the username/password on each request. We're only sending it once, get a
token, and our request after that, we'd send that token to the API.
Moreover, it'll work for all these modern types of application architectures but sending your
username/password to a self-created action on an API and getting a token in return isn't a good fit for a
lot of applications nor a good idea. It still requires us to share our username/password with that
application or API.
That might be okay for trusted application, but it's a very bad idea for third-party applications, including
other applications that might live in the company.

Moreover, once we start creating endpoints like these ourselves, we are essentially reinventing the
wheel.
We must implement expiration ourselves.
We must implement signing of tokens and token validation.
We need to think about token format and might want to implement reference token support.
We'll need to separate out authentication and authorization.
We need to find a secure way to deliver these tokens to all these different types of applications and
that's just a few concerns.

In short, mistakes are almost inevitable, and even if we don't make any mistakes, keeping home grown
systems like this secure isn't easy either as new vulnerabilities are recently discovered.

So, we're really trying to solve two issues here, one is no longer handling user credentials at application
level. That should be handled by another component, so we're looking for a central component to
handle common user credential related tasks, and the other one is finding a way to ensure these tokens
are safe enough to use for both authentication and authorization and for different types of applications.

So, we're looking for a proven protocol that handles these common concerns we will inevitably forget
when creating tokens ourselves.
2) Identity Provider
In the past, we used to create login screens inside of an application and often user management,
password tools, and resets, etc., as well. These days, we no longer do that. An application shouldn't
be responsible for checking if a user is who he or she says he is. It's the responsibility of an identity
provider, to authenticate the user, and if needed, safely provide proof of identity to an application.
Often such an identity provider is also a gateway to user and account-related tasks like letting a user
manage his or her account.

Now why do we use these IDPs or a big brother identity and access management systems. There are
a few reasons for this.

Let's think about identity and access management related tasks first, user registration, account
management, implementing password policies with rules for password strengths and resets, locking
out users are tedious tasks easily allowing for mistakes and prone to change. It's better to handle
these in one central location at IDP level so they can be reused across applications and decent
penetration testing can happen. Then let's think about how we store this information.

To safely store a password in a database in the early days, encrypting it in a non-decryptable way
was considered sufficient, but most encryption mechanisms can easily be brute forced these days.
Today, mechanisms like key stretching are used to prevent this. That means, encrypting passwords
repeatedly so it takes a few seconds to get to the correctly encrypted value. That discourages brute
forcing, and for that, strong algorithms like Algorithm2 or BeCript are currently used, and probably
in a year or two, if it even takes that long, new algorithms will supersede these.

Imagine having a custom account database for each application. You don't want to go and change all
this for each application, so a central system is a much better option. And while we're about
passwords, there is more than just username/password combos. Users might want to authenticate
themselves for different needs. Username/password is one way, but other applications or systems
might require certificates or, for example, proof of possession, ie. a second factor of authentication
like a digital pass or a smartphone. So multiple applications require access to the same user
accounts.

Tasks related to those accounts are common for all applications and not application specific.
Best practices concerning safely storing credentials change and means of authentication can change.
Those are just a few of the reasons to keep all of this in a central location.

A user will just no longer prove who he or she is in the application itself but will do so at the level of
the identity provider and that's a solution for our first concern. Solutions like these have existed for
a while, by the way. Active Directory services is an example of this, but the application does need
proof of identity and it also needs authorization to access the apps APIs or third-party APIs and that
brings us to solving that homegrown token service problem. Once we have a central identity
provider and identity and access management system, we could write some code at that level, the
homegrown token or login endpoints, but as we learned, that's just not good enough. We are now
back at our reinventing the wheel slide. So how do we solve this? Luckily, when problems arise,
standards emerge, and that's where OAuth2 and OpenID Connect come into play.
3) OAuth2
Definition: OAuth2 is an open protocol to allow secure authorization in a simple and standard
method from web, mobile, and desktop applications. From that definition, we can learn a few
things. One OAuth2 is about authorization, i.e. through OAuth2, we can authorize a client
application to access an API.

OAuth2 defines an access token and a client application can request an access token and pass it
through to the API to gain access to that API. Next to that, the definition speaks about web,
mobile, and desktop applications.

Not all applications are created equal. A server-side web app like an ASP.NET Core application
has different means of storing things like secrets than, for example, an Angular application,
which runs completely on the client and is just not to be trusted by default. So, OAuth also
defines how an application can securely get such token from a security token service, or in other
words, how a client application can achieve authorization.

We can already feel that our reinventing the wheel issue is like expiration, delivering tokens to
different applications, authorization, and so on are being solved. The homegrown endpoints (if
exist) are being replaced by endpoints defined by the OAuth2 standard and the standard also
states how we should use these endpoints from the different application types. So, our identity
provider should implement this standard. In this regard, as we're talking about authorization,
these IDPs are also named authorization service or the more generic name security token
services.

But those access tokens should only be used to access resources and API. They shouldn't be
used to sign into a client application, that's authentication, not authorization, so we're only
halfway there. OpenID Connect is the last piece of the puzzle.

4) OpenID Connect
OpenID Connect is a simple identity layer on top of the OAuth2 protocol so it extends OAuth2.

With OpenID Connect, an application can receive an identity token (next to an access token if it
needs one). This identity token can then be used to sign into an application while that same
application can use the access token to access an API. It also defines an additional endpoint that
can be used by an application to get additional user information from and that's the user info
endpoint. As for the rest, likewise principles apply.

It defines how different types of client applications can safely get those tokens from that
security token service.

In this regard, as we're now talking about authentication or identity through that identity token,
both Azure Active Directory and identity server can be named Identity Providers, or IDPs, next to
authorization servers.

OpenID Connect is the superior protocol. It extends and supersedes OAuth2. These days, even
though we're still talking about choosing OAuth2 to secure APIs, we're often without realizing it,
using the superior OpenID Connect protocol. For example, if developer worked with a security
token service before like IdentityServer3 or Azure AD and you are building an ASP.NET MVC web
 app, you've probably used Microsoft OpenID Connect middleware. That's pretty much the
default way of handling this. Even if you only needed an access token, you were probably
already using the superior OpenID Connect protocol. Lastly, OpenID Connect isn't just for new
applications or API-based applications.

5) Working of OpenID Connect

We're only focusing on identity at this moment, so identity tokens and not access tokens. The
client application requires a user's identity.

That client application is also called the relying party in this case, as it relies on the identity
provider to return the result of the authentication in a secure manner, so the client can rely on
it. An authentication request is created by the client application which redirects the user to the
identity provider. Here, the user proves who he or she is, for example, by providing a
username/password combination. When this checks out, the identity provider creates an
identity token and signs it. Such an identity token can be seen as a token that contains a user's
verifiable identity. After that, it redirects the users to the client application passing through the
identity token. The client application then gets the identity token from the URI and validates it.
If validation checks out, the client application now has proof of identity.

In an ASP.NET Core application, this identity token is then used to create claims identity from,
which in most cases, is stored as an authentication ticket in an encrypted cookie. The browser
sends that cookie on each request to the client application proving the user's identity. That's the
high-level picture, but as we know by now, not all applications are created equal and open ID
connect as it's built on OAuth2 is designed to work for various types of apps like server-side web
apps, client-side web apps, and mobile applications. To allow us to decide on the correct way to
implement this for our application, it defines client types and grants or flows.

6) Client Types
Two client types are defined, confidential clients and public clients.

Confidential Client (Server-Side Web Apps):

The confidential client is a client capable of maintaining the confidentiality of their credentials,
that's client credentials (clientid, clientsecret) typically, not the user's credentials. When user
enter his or her credentials at level of the IDP, client credentials, on the other hand, are stored
at the level of the client application.
 They live on the server.

The prime example of such an app is a web application running on a web server like an ASP.NET
Core MVC web app. As it runs on the server, it can safely store its client credentials on that web
server. They are not exposed to nor accessible by the user.

Public Client (JavaScript apps, Mobile Apps, Desktop Apps):

The second, is a public client. These are client’s incapable of maintaining the confidentiality of
their credentials (clientid, clientsecret).

They are, in other words, executed on the device, the browser, or on a mobile device.

That means that a JavaScript application like an Angular app or a native mobile application, both
are public clients. No matter what we try, a secret stored in JavaScript or on a mobile device isn't
a secret; a user can always access it.

7) Open Id Connect Flows and Endpoints

The flow determines how the code and/or tokens are returned to the client.

Depending on the type of application, confidential or public, and requirements we might have
like the need for long lifetimes, for example, we must use a different flow.

These flows will talk to different endpoints.

Authorization Endpoint (Identity Provider Level):

This endpoint is used by the client app to obtain authentication for identity tokens and/or
authorization for access tokens from the user. This is done by redirecting the client application
to the identity provider. So, from the authorization endpoint, an authorization code or tokens
can be returned to the client application.

Important to know is that in OIDC, TLS (or as it's wrongly named SSL) is a requirement. In other
words, traffic should always be encrypted.

Redirection Endpoint (Client Level):

The client redirects to the authorization endpoint at level of the identity provider and the
identity provider redirects back to the client. The URI or the IDP redirects the client back to
what's called the redirection endpoint. So it lifts at level of the client and it's used by the identity
provider to deliver the code or tokens to the client application.

Token Endpoint (Identity Provider Level):

From this endpoint, which lives at level of the identity provider, client applications can
programmatically request tokens and that's typically done via an HTTP post without redirection.
 There are more endpoints, but to explain the flows, the endpoints we mentioned above are
sufficient.

OpenID Connect Authorization Code flow:

It returns an authorization code from the authorization endpoint and tokens from the token
endpoint.
This authorization code is a short-lived single use credential used to verify that the user who
logged in at level of the identity provider is the same one who started the flow at level of the
web app/Desktop Client.
The authorization code flow is suitable for confidential clients and it allows long-lived access.

OpenID Connect Implicit flow:

The implicit flow returns all tokens from the authorization endpoint and there is no
authorization code. It's suitable for public clients. There are no client authentication as public
clients can’t safely store client secrets anyway. Therefore, there is also no long-lived access
through the refresh tokens are allowed. The implicit flow may be used by confidential clients as
well.

OpenID Connect Hybrid flow:

This one returns some tokens from the authorization endpoint and others from the token
endpoint. It's just a mix of the authorization code and implicit grant or flow. It's suitable for
confidential clients and it allows long-lived access. In some cases, the hybrid flow is allowed for
native mobile applications, even though they are technically public clients.

Now why is it so important to choose the correct flow?

Well, because choosing the wrong one might open us up to security vulnerabilities. For example,
the hybrid flow allows long-lived access through the refresh tokens, but as long lifetime is a
potential risk, they should only be delivered to authenticated clients. To be able to safely
authenticate a client, a client secret is required, and as we know, a JavaScript app/Desktop
client/Mobile client can’t safely store such a clientsecret as it's a public client. So those apps
should not use the hybrid flow as it will potentially open them up to a security risk, but the thing
is that there is nothing technically stopping us from doing that. We can perfectly use a hybrid
flow in say an Angular app, even though it's an exceptionally bad idea. And that's the thing with
security, a lot of approaches will work, but most of them are not a good idea.

8) Open Id Connect Flow for ASP.NET Core MVC

ASP.NET Core MVC Web Client is a confidential client. So, all the flows can be applied. For
server-side web apps, renewing tokens with refresh tokens is often a requirement. Refresh
tokens aren't allowed with an implicit flow, so we won't choose that. And that leaves us with
two flows, the authorization code and the hybrid flow. Before OpenID Connect was conceived,
the authorization code flow was the flow of choice. For that flow, as we learned, all tokens are
returned from the token endpoint, but the hybrid flow has an advantage over that one. It allows
us to get an identity token from the authorization endpoint first and we can then verify that
before continuing with additional roundtrips to get an access token. Next to that, OpenID
Connect links the identity token to an optional access token and these are reasons to choose
OpenID Connect over regular OAuth2. We will always use the hybrid flow, even with that flow,
 quite a few decisions must be made on how to deliver what kind of token and all of these might
have an impact on how secure our application will end up being.

9) Identity Server 4
The OpenID Connect standard does not contain a single line of code. It is a set of rules, a
guidance, so we need a component that implements this standard, so we can create an identity
provider that supports these protocols. All these endpoints, the flows, the tokens, and so on,
that's not something we want to code ourselves and that's where IdentityServer4 comes into
play. IdentityServer4 is an OpenID Connect and OAuth 2 framework for ASP.NET Core developed
by Dominick Baier and Brock Allen. It's officially certified by the OpenID foundation and its part
of the .NET Foundation. On top of that, it's completely open source. It's this component, which
is a piece of middleware, that will take care of most of the hard work for us.

10) Hybrid Flow

Each of the OpenID Connect flows start to request to the authorization endpoint.

There is the endpoint itself to the authorization URI at the level of the IDP.
For example: https://idphostaddress/connect/authorize

Then there is the client id. That's the identifier of the client application.
For example: https://idphostaddress/connect/authorize?client_id=usermanamgementclient

To redirect URI value is the redirection endpoint at the level of the client app.
For example: https://idphostaddress/connect/authorize?
client_id=usermanagementclient
&redirect_uri=https://clientapphostaddress/signin-oidc

Using scopes, the application requests access to the openid and profile scopes, so it wants
access to the user's identifier and the profile related claims like given name and family name.
For example: https://idphostaddress/connect/authorize?
client_id=usermanagementclient
&redirect_uri=https://clientapphostaddress/signin-oidc
&scope=openid profile

It's the response type value that determines the flow that's used. A response type of code
signifies that we'll use the authorization code flow. The hybrid flow with response type “code
id_token”, that's the one that's preferred. The web app creates an authentication request with
response type “code id_token”.
For example: https://idphostaddress/connect/authorize?
client_id=usermanagementclient
&redirect_uri=https://clientapphostaddress/signin-oidc
&scope=openid profile
&response_type= code id_token
 Response Type OpenID Connect Flow
code Authorization Code
id_token Implicit
id_token token Implicit
code id_token Hybrid
code token Hybrid
code id_token token Hybrid

The web app sends the authorization request. At the level of the IDP, a user authenticates, for
example, by providing his or her username and password combination. From that moment on,
the user is authenticated at level of the IDP. The identity provider optionally asks the user for
consent, i.e. it'll ask if we want to allow the application to get profile information. The identity
provider sends us back to the web application via URI redirection or by a form post with the
authorization code and the identity token, that's the response type we asked for. The WebClient
then validates the identity token it got back from the authorization endpoint. If it's invalid,
execution of the flow stops. If it's valid, the flow continues.

The middleware calls the token endpoint through the backchannel. For this, it passes through
the authorization code, clientid, and clientsecret, ie. the client application has to authenticate
itself. In the response, we get back an identity token, this is validated as well. One of the
validation steps is to check if this token matches the user in the previous token, and at this time,
the middleware extracts the user's identifier from the identity token. Now this may look a bit
weird, after all, we've already got an identity token from the authorization endpoints, why not
just use that? Well, the token endpoint requires client authentication with the secret that's not
exposed so that adds an additional layer of security. In reality, access tokens and refresh tokens
can be returned from this token endpoint as well, so it's not just a matter of getting a second
identity token. We're currently focusing on the identity token and this identity token after it has
been validated has proved that the user is who he or she says he or she is. So, it can be used by
applications for that purpose. A difficult thing to do with such a token is create a claims identity
 from it and use that to sign into our ASP.NET Core MVC app. Now that's just one of those three
response types we could use for the hybrid flow, so what do the other two mean?

The hybrid flow response type signifies what will be returned from the authorization endpoint
via the front channel. When communicating via the front channel, the information is delivered
to the browser either via the URI or via Form Post, that was that response_mode value from the
authorization request. So when we use code id_token as a response type, we're returning the
authorization code and identity token from the authorization endpoint via the browser, but not
the access token.

Other tokens or new versions of the same ones can then be returned through the back channel
from the token endpoint and that's server-to-server communication, so the browser doesn't see
this, which tends to be the safer option.

So that's quite a few steps and the client has to do some work, create URIs, catch the identity
token, validate it, create a claims identity from it. We can do all of this manually and that's
actually a good thing. All any application built in any language needs to be able to work with
OAuth2 and OpenID Connect is support for the HTTP protocol and a way to parse the tokens.
JSON parser is often sufficient, but we don't have to do this manually. There is middleware
that'll help us with it.

11) UserInfo Endpoint

By default, IdentityServer doesn't include identity claims (except sub) for the user identifier in
the identity token. We could allow this, by setting the AlwaysIncludeUserClaimsInIdToken
setting on the client configuration to true, but we don't want to do that.

The initial identity token is returned from the authorization endpoint file front channel
communication, either through a form post or to the URI. If it's returned via the URI and the
token becomes too big, we might hit URI link limits, which still are dependent on the browser
and that's not a good thing.

Not including these claims in the identity token keeps it small, which avoids these potential
issues. So how do we get access to those claims then? The token endpoints and authorization
endpoint at level of the IDP. There is another one, the UserInfo endpoint (IDP). This is an
endpoint the client application can call to get additional information on the user. Calling these
endpoints requires an access token with scopes that relate to the claims that must be returned.
As to get profile information from the UserInfo endpoint, the access token must contain the
profile scope. It's sufficient to know that the middleware needs one to call its User Info
endpoint.
Well, remember the hybrid flow diagram, I mentioned we skipped on a few parts there to keep
it simple and this is where those come into play. We're at the point where the identity token is
returned from the token endpoint. In reality, an access token is returned next to that identity
token. Up next, the middleware sends a request to the UserInfo endpoint passing in the access
token, there the access token is validated and the UserInfo endpoint returns the user's claims
that are related to the scopes in the access token.
12) Token PayLoad Data
An example of a decoded identity token can be seen on screen.

"sub": "8A6E345D-34F4-41EC-8D5D-D1F8272B25A4"
This is the user's identifier, a claim that's always returned when using OpenID Connect.

"iss": https://localhost:8265
ISS stands for issuer, the issuer of the identity token. In our case, that's the URI of our identity
provider.

"aud": "engineeringapigatewayclient"
AUD means Audience, ie. the intended audience for this token, the client application signified
through the client id.

The following numeric values represent the seconds that have passed since January 1, 1970.
"iat": 1536921595
Iat or Issued At, is a time at which JWT was issued.

"exp": 1536921895
Exp or Expiration, is the expiration time on or after which the identity token must not be
accepted for processing.
"nbf": 1536921595,
NBF, or Not Before, identifies a time before which token must not be accepted for processing.

"auth_time": 1536921590
The auth_time represents the time of the original authentication.

"amr": [
"pwd"
]
Amr means authentication methods references. That's a JSON array of strings that are identified
for authentication methods used in the authentication, in our case, it's pwd, password, but in
other cases, this might be a one-time password, or a combination used in multi-factor
authentication.

"nonce":
"636725183836155931.ZDYxNTQ5YzctMzk0NS00NjY2LWI4NzMtM2NhODc0YjgwNTM3YjQ1ZW
U4YzctOWUyNi00YzUwLWJiNzEtNTQ0YjRjNmQwNzJm"
And then there is the nonce or number only used once. This is a value that's generated client
level and that's sent back by identity provider in the token. Comparing these during validation
protects against cross-site request forgery attacks.

"at_hash": "TeMTR45ZDaMVzKKtcoMPkw"
Lastly, we have the c_hash value or code hash value and at_hash value or access token hash
value. Both are Base64 encoded values of a part of the hash code and/or access token. These
can be used to link an authorization code or access token to the specific identity token. Now this
is just one example of an identity token. Identity providers like identity server sometimes add
their own claims to a token so the results may vary depending on the IDP integrating with, but
the claims we looked at here are all part of the standard.
13) Information through UserInfo EndPoint

By default

As we remember, the authentication middleware calls the UserInfo endpoint passing in an

access token with requested scopes, so the claims related to those scopes are included.

We can also manually call that UserInfo endpoint say to get user's details. We might have
privacy or security related reasons for not including that in the claims identity, but next to that,
it keeps the cookies small, and moreover, it allows us to get the most up-to-date information on
the user exactly when we need it.

The UserInfo request:

GET idphostaddress/connect/userinfo
Authorization: Bearer S8bsx41Qml

Such a request is typically a GET request to the UserInfo endpoint at level of the IDP, but POST is
also allowed. An access token with the requested scopes should be set as a Bearer token value
for the authorization header. Such a request will return a JSON response containing the claim
values for later with the scopes in the access token.

14) Role Based Authorization

Authentication: The process of determining who you are. The identity token plays a role for that.
Authorization: The process of determining what you're allowed to do and there's a few
approaches to that.
 The typical approach a lot of people are used to working with is role-based authorization or
access control, RBAC i.e. we're going to get a role claim and that role carries a specific set of
privileges.

Attribute-based access control, or ABAC, is another way of handling this often preferred over
RBAC.

References
This material is referenced from:
https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

https://martinfowler.com/articles/richardsonMaturityModel.html

https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

https://tools.ietf.org/html/rfc6902

https://tools.ietf.org/html/rfc7234

https://tools.ietf.org/html/rfc2616

https://www.kevindockx.com/

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/routing?view=aspnetcore-2.1

https://msdn.microsoft.com/en-us/magazine/mt845654.aspx