After gathering further requirements such as expected load etc., we decided to launch on AWS Elastic Container Service (ECS) via AWS Copilot (https://aws.github.io/copilot-cli/). This would allow us to keep the containerisation and scalability, but with a dramatically reduced amount of code and complexity to maintain.
Copilot describes itself as: “an open source command line interface that makes it easy for developers to build, release, and operate production ready containerized applications on AWS App Runner, Amazon ECS, and AWS Fargate”
From our perspective, it’s a tool that allows us to create multi-environment infrastructures via low maintenance manifest files and a few commands. Abstracting away the more complicated CloudFormation templates, while still giving us plenty of flexibility and the ability to drop to raw CloudFormation when required.
We’re not going to dive into explaining Copilot, how to install, create a project, setup, exec to a container etc. as the Copilot docs already do a great job of that (https://aws.github.io/copilot-cli/docs/overview/).
Instead, we are going to cover the Django specifics of how our manifest file came together. Below is an example of a manifest for a load-balanced web service (https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/), abbreviated in a couple of places in regards to secrets and variables. It defines a load balancer, 3 containers, environment secrets, and variables per environment for stage and prod.
This manifest file consists of 5 key sections that are worth explaining in more detail, Each category below relates to a key in the above yaml.
Our load balancer config, the key thing here is that we point the target of the load balancer to the nginx sidecar (see below) rather than the Django application itself, which is the default in CoPilot. We will also later customise this to be environment specific.
Our main container i.e. the Django application. In this case, a Python Dockerfile that executes uwsgi. Note that collectstatic and migrations are not run in this docker file, more on that in sidecars.
AWS Copilot has the concept of sidecars (https://aws.github.io/copilot-cli/docs/developing/sidecars/), these are containers that run alongside the main container to perform additional work:
Startup - A container that runs before the main container (see main container
depends on it) to trigger migrations and static collection on each new deployment i.e all the parts of Django we don’t want to be tied to the base application which will be scaled. This container is
essential: false meaning once it’s completed it will stop, saving resources. This runs from the same Dockerfile, with an alternate entry point defined as the command. *
Nginx - Runs nginx via an alternate Dockerfile and proxies to the main container in conjunction with the load balancer, it’s also dependent on the main container running. The Dockerfile and related nginx conf can be quite simple, noting that we allow the health check from any host for the load balancers to correctly assign the targets: *
Various configurations regarding the size of the containers and the auto scaling options.
Here we set environment-specific config, variables, and secrets. Note that currently variables and secrets need to be repeated across each container.
variables - environment variables read into the Django settings that don’t need to be secret.
Secrets - secret environment variables read from AWS secrets manager (remember to assign the CoPilot tags in SSM as noted in the docs - https://aws.github.io/copilot-cli/docs/developing/secrets/). On the server these will be files rather than environment variables. To handle this we wrote a small wrapper that will attempt to read the variable from environ or file to handle this e.g.
Following that we have a largely working example of a Copilot deployable Django project. There are lots of other features in Copilot you can also leverage:
We created an additional service (a separate manifest) for a “Scheduled Job” (https://aws.github.io/copilot-cli/docs/manifest/scheduled-job/) which spins up containers as required (from the same Dockerfile) to run out cron tasks.
addon manifests (https://aws.github.io/copilot-cli/docs/developing/addons/workload/) are additional CloudFormation templates that can also be deployed, for example, we added a SES-IAM role to allow the containers to send emails via SES. In our project, we created the RDS and S3 for staticfiles (django-storages) separately but you could also add that via addons.
Pipelines - You can also add pipelines (https://aws.github.io/copilot-cli/docs/manifest/pipeline/) to deploy your project. In our case we already had CI so we added the CD directly into bitbucket using the copilot cli directly.
Hosted zones and SSL - we briefly covered this above, in our setup we used existing certificates and hosted zones and linked to them in our manifest, you can find out more on this at https://aws.github.io/copilot-cli/docs/developing/domain/
All in all our first experience with CoPilot has been very positive, the autoscaling and deployments work well. It also reduced the time needed to provision the architecture, giving us more time to deliver features. It’s a nice cost-effective approach for our clients who don’t require a more complicated Kubernetes setup and certainly better than provisioning ECS or EC2s manually.
In most scenarios, we like to deploy our infrastructure to two environments, stage and production, both a replication of each other. This gives us and our clients confidence when it comes to going live i.e. there are no hidden gotchas in the production environment.
MangoPay presents itself as "Payments for Marketplaces", but it's far from being just a payment provider...
Sometimes you want to show a user a different feature. Or you want to test it in the production environment without affecting the other users. Or you have a group of beta testers for which you rely on early feedback to improve your app. Below is how to do this...
A lot of websites display notifications in one form or another these days. Facebook shows the new message notification, Gmail have the new mail notifications (with the support of browser notifications). I was asked today how hard it would be to implement a real-time notification system. The answer, as it...