NGINX as an API Gateway with OAuth 2.0 Authorization on AWS
Background
Recently I’ve been investigating the architecture for a microservices based web application in the cloud — AWS. There were two initial requirements we had (which is relevant to this post).
- Run a few services independent of each other on different containers.
- Secure each service with OAuth2.
This was the first experience for us with microservices so we decided to make the project as experimental as possible. Three services were written in Node.js, Go and Python to begin with. The UI was planned to be a completely independent SPA which made use of above services. Each API had some routes which needed to be secured with OAuth 2.0 tokens and some which could be allowed accessing without any authorization.
Instinctively, the first option we went for was AWS Fargate for deploying the services. It provided a serverless engine to deploy our services with auto-scaling features. Spinning up new instances was easy and it was only a matter of pushing the latest Docker image to ECR. We had the Fargate services deployed in a private subnet with a NAT Gateway setup for internet access and the respective databases in another private subnet which was completely detached from internet. We planned to have an API gateway to route the requests to Fargate services which lied behind a Network Load Balancer. A VPC PrivateLink was created so that the API Gateway has access to the NLB.
We realized that we did not want to implement token validation in each service. That was bad design resulted in code duplication and unnecessary maintenance overhead we did not want to incur. AWS API Gateway had a good solution for that, allowing Cognito authentication (we were using Cognito as our IDP) for API endpoints it exposes.
Everything was perfect, until we realized how much it costs to maintain a NAT gateway, the API gateway, the PrivateLink and the Fargate services. This was a PoC so we wanted to keep the costs to a minimum, and most of these managed services weren’t under the free tier AWS offered. It also meant that the same level of costs would incur in case we go to production with the same architecture.
It was time to think beyond the managed services and an alternative to achieve the same level of cloud native cohesive but decoupled architecture.
Deploying the Services
We decided to go back to the basics, and start with a good old EC2 instance. We kept the managed RDS instances as they were, but got rid of the Fargate services and deployed an EC2 instead, in the same private App subnet. Our goal was to have a functioning end to end solution, which meant working services with an API gateway in place which takes care of authentication.
Each service was spun off as docker containers and were ready to be consumed.
Deploying NGINX as the API Gateway
When looking at API gateways we could use in EC2, NGINX came as a possible candidate. It had the self-managed element we were looking for, and a quick skim through the documentation made us realize that it had built-in authentication features as well, which was perfect for our use case.
We launched another EC2 instance running Ubuntu in our public subnet, and installed NGINX in that. The architecture looked like this.
NGINX has quite a good documentation on how an API gateway could be configured for a microservices environment here. I found this conference talk quite useful as well. This was the base for us when we started configuring the API gateway.
It suggested a modular, easy-to-maintain approach to structure the config files. We took the same approach and had the following config structure.
The main config file nginx.conf simply had a include directive to api_gateway.conf where all the API configuration was put into.
http {
# Logging and other configuration goes here
# Default configuration
include /etc/nginx/conf.d/*.conf; # API Gateway configuration
include api_gateway.conf;
}
The api_gateway.conf is the entry point for all things related to the API Gateway we’re configuring.
Inside api_gateway.conf, it imports two conf files, one at the top of the file, which includes upstream definitions for each of the APIs, and the other inside server definition which points to all .conf files inside api_conf directory.
# file with upstream definitions.
include api_backends.conf;server {
listen 443 ssl;
server_name example.com; # TLS config goes here location / {
} # API definitions, one per file
include api_conf/*.conf; # Error Handling goes here.
}
The api_backends.conf file, which defines the upstreams looks like this.
upstream identity_api {
server ec2_app_host:8010;
}
upstream catalog_api {
server ec2_app_host:8020;
}
upstream cart_api {
server ec2_app_host:8030;
}Each API had its own config file named after the API name in api_conf directory. A sample api.conf file looked like this.
# API definition
location /api/identity {
set $upstream identity_api;
rewrite ^ /_identity last;
}location /api/identity/heartbeat {
set $upstream identity_api;
rewrite ^ /_identity last;
}location = /_identity {
internal;
proxy_pass http://$upstream$request_uri;
}
The location directive takes the incoming requests on /api/identity and /api/identity/heartbeat paths and routes to the internal location _identity. As nginx matches the longest path, all routes which are not /api/identity/heartbeat will be routed through /api/identity.
A simple curl returned the API content as expected.
Setting up OAuth 2.0 Authorization
When going through the NGINX documentation on how to validate OAuth 2.0 access tokens, two challenges came up.
- For NGINX open source, an introspection endpoint is required to perform the validation — AWS Cognito doesn’t have one.
- Native JWT support is only available in NGINX Plus.
What NGINX open source does have though, is a cool module called auth_request. According to the documentation,
“The ngx_http_auth_request_module module (1.5.4+) implements client authorization based on the result of a subrequest. If the subrequest returns a 2xx response code, the access is allowed. If it returns 401 or 403, the access is denied with the corresponding error code. Any other response code returned by the subrequest is considered an error.”
So we decided to implement a small service which validates the access token and return a 200 response code if the validation succeeds, or a 401 response if it fails.
The service was implemented in Go. Two Go libraries — github.com/dgrijalva/jwt-go and github.com/lestrrat-go/jwx/jwk were used for JWT processing and the code looked something like this.
// Validate is the HandlerFunc that receives the JWT Validation request.
func Validate(w http.ResponseWriter, r *http.Request) {
err := auth.TokenValid(r)
if err != nil {
compileHTTPResponse(w, http.StatusUnauthorized, "Unauthorized")
return
}
compileHTTPResponse(w, http.OK, "Authorized")
}func extractBearerToken(r *http.Request) string {
bt := r.Header.Get("Authorization")
if len(strings.Split(bt, " ")) == 2 {
return strings.Split(bt, " ")[1]
}
return ""
}func validateToken(r *http.Request) error {
// fetch the JWK URL for Cognito from Env file
ks, err := jwk.Fetch(os.Getenv("JWK_URL"))
ts := extractBearerToken(r)
t, err := jwt.Parse(ts, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodRSA); !ok {
return nil, fmt.Errorf("Signing method unknown: %v", token.Header["alg"])
}
kid, ok := token.Header["kid"].(string)
fmt.Printf("kid: %s\n", kid)
if !ok {
return nil, errors.New("kid header not found")
}
keys := ks.LookupKeyID(kid)
if len(keys) == 0 {
return nil, fmt.Errorf("key %v not found", kid)
}
var raw interface{}
return raw, keys[0].Raw(&raw)
})
if err != nil {
return err
}
return nil
}func compileHTTPResponse(w http.ResponseWriter, statusCode int, msg string) {
w.WriteHeader(statusCode)
json.NewEncoder(w).Encode(msg)
}
After this was in place in our EC2 instance, on its own container, it was only a matter of proxying the requests received by NGINX through the validator service to check if the header contains a valid Bearer Token. Following were the changes made to the api.conf file.
# API definition
location /api/identity {
set $upstream identity_api;
rewrite ^ /_identity/protected last;
}
location /api/identity/heartbeat {
set $upstream identity_api;
rewrite ^ /_identity last;
}
# Policy section
#
location = /_identity {
internal;
proxy_pass http://$upstream$request_uri;
}
location = /_identity/protected {
internal;
auth_request /_oauth2_token_validation;
proxy_pass http://$upstream$request_uri;
}
location = /_oauth2_token_validation {
internal;
proxy_method POST;
proxy_set_header Authorization $http_authorization;
proxy_pass http://ec2_app_host:8040/validate;
}Two new internal locations were added,
- /_identity_protected
- /_oauth2_token_validation
Since we had some endpoints which did not require authorization, all requests with the URI pattern /api/identity were passed into the /_identity_protected location which made use of the auth_request module. Requests to /api/identity/heartbeat endpoint, which shouldn’t be secured with OAuth 2.0 access tokens were passed to /_identity location as before, which simply proxied the request to the relevant service.
In /_oauth2_token_validation location, the subrequest to the access token validator service is proxied, taking the request http headers as they are. $http_authorization variable in NGINX gives access to the headers and it’s set through proxy_set_header directive.
With this setup, we had secured all endpoints in our API except for heartbeat.
Conclusion
This post is a walk-through of how we approached the OAuth 2.0 token validation challenge we faced with NGINX Open Source. There’s a lot more configuration that could be done, in terms of caching for performance optimization and having more control over the type of HTTP methods that are allowed in each endpoint. Both can be achieved easily with NGINX and the documentation is quite comprehensive.
Some more challenges we need to figure out after this is how the API gateway can be configured to automatically discover the service hosts when the instances are set with auto-scaling rules, as well as to consider the health status in each service before forwarding requests. etcd, Consul and Apache ZooKeeper all are candidates for that as of now. Hopefully I’ll write another walk-through about our approach for service discovery in a later date.
Shout out to Thusith who’s my partner in crime for this. ;)