Implementation:Mage ai Mage ai Buildkite ECS Deploy

Knowledge Sources	Mage AI Mage Deployment Docs
Domains	Deployment, CI_CD, AWS
Last Updated	2026-02-09 00:00 GMT

Overview

Buildkite CI/CD pipeline script template for building Docker images and deploying Mage to AWS ECS.

Description

This template shell script provides a Buildkite pipeline step that automates the full deployment lifecycle to AWS Elastic Container Service (ECS). It performs four sequential operations: (1) authenticates to Amazon ECR using aws ecr get-login in the us-east-1 region, (2) builds a Docker image and tags it with both an environment tag ($ENV) and the Buildkite commit SHA ($BUILDKITE_COMMIT), then pushes both tags to ECR, (3) iterates over a comma-separated list of ECS service names ($SERVICES_NAMES), and for each service retrieves the current task definition, updates the container image to the commit-tagged image, injects a DD_VERSION environment variable (for Datadog version tracking) set to the commit SHA, registers a new task definition revision, and updates the ECS service to use it, and (4) waits for all services to reach a stable state using aws ecs wait services-stable. The script uses set -eu for strict error handling, exiting immediately on any failure or undefined variable.

Usage

Copy this template into your Buildkite pipeline configuration to automate Mage deployments to AWS ECS. Configure the required environment variables for your AWS account, ECR repository, ECS cluster, and service names. The script supports deploying to multiple ECS services in a single run.

Code Reference

Source Location

• Repository: mage-ai
• File: templates/cicd/buildkite/deploy-ecs.sh
• Lines: 1-44

Signature

#!/bin/bash
set -eu

# Step 1: ECR Login
$(aws ecr get-login --region us-east-1 --no-include-email)

# Step 2: Build, tag, and push Docker images
docker build --tag $REPOSITORY_URL:$ENV --tag $REPOSITORY_URL:$BUILDKITE_COMMIT .
docker push $REPOSITORY_URL:$ENV
docker push $REPOSITORY_URL:$BUILDKITE_COMMIT

# Step 3: Update ECS services (iterate over SERVICES_NAMES)
IFS=', ' read -r -a SERVICES <<< "$SERVICES_NAMES"
for SERVICE in "${SERVICES[@]}"; do
    # Fetch current task definition, update image and DD_VERSION, register new revision, update service
done

# Step 4: Wait for deployment stability
aws ecs wait services-stable --cluster ${CLUSTER_NAME} --services ...

Import

# Template file - copy to your project's Buildkite pipeline directory
# cp templates/cicd/buildkite/deploy-ecs.sh .buildkite/deploy.sh

I/O Contract

Inputs

Name	Type	Required	Description
REPOSITORY_URL	env var	Yes	Full ECR repository URL (e.g., 123456789012.dkr.ecr.us-east-1.amazonaws.com/mage)
ENV	env var	Yes	Environment name used as a Docker image tag (e.g., production, staging)
BUILDKITE_COMMIT	env var	Yes	Git commit SHA provided automatically by Buildkite, used as image tag and DD_VERSION
SERVICES_NAMES	env var	Yes	Comma-separated list of ECS service names to deploy (e.g., mage-web,mage-worker)
CLUSTER_NAME	env var	Yes	Name of the ECS cluster containing the target services
AWS credentials	env var / IAM role	Yes	AWS credentials with permissions for ECR, ECS describe/register/update operations

Outputs

Name	Type	Description
Docker images	ECR images	Two tagged images pushed to ECR: one tagged with $ENV and one with $BUILDKITE_COMMIT
Task definitions	ECS task definition revisions	New task definition revisions registered for each service with updated image and DD_VERSION
Service updates	ECS service deployments	Each listed ECS service updated to the new task definition revision and confirmed stable

Deployment Flow

1. Script authenticates to Amazon ECR in us-east-1
2. Docker image is built from the current directory's Dockerfile
3. Image is tagged with both the environment name and the commit SHA
4. Both tagged images are pushed to ECR
5. For each service in SERVICES_NAMES:
   1. Current task definition is fetched from the ECS service
   2. Container image is updated to the commit-tagged image
   3. DD_VERSION environment variable is injected with the commit SHA (for Datadog tracking)
   4. Unnecessary fields are stripped (taskDefinitionArn, revision, status, requiresAttributes, registeredAt, registeredBy, compatibilities)
   5. New task definition revision is registered
   6. ECS service is updated to use the new revision
6. Script waits for all services to reach stable state using aws ecs wait services-stable

Usage Examples

# Set required environment variables in your Buildkite pipeline
export REPOSITORY_URL=123456789012.dkr.ecr.us-east-1.amazonaws.com/mage-ai
export ENV=production
export CLUSTER_NAME=mage-cluster
export SERVICES_NAMES="mage-web,mage-worker"
# BUILDKITE_COMMIT is set automatically by Buildkite

# Run the deployment script
bash deploy-ecs.sh

# Example Buildkite pipeline.yml step
steps:
  - label: ":rocket: Deploy to ECS"
    command: ".buildkite/deploy-ecs.sh"
    env:
      REPOSITORY_URL: "123456789012.dkr.ecr.us-east-1.amazonaws.com/mage-ai"
      ENV: "production"
      CLUSTER_NAME: "mage-cluster"
      SERVICES_NAMES: "mage-web,mage-worker"

Related Pages

Implements Principle

• Principle:Mage_ai_Mage_ai_Source_Connector_Scaffolding

Requires Environment

• Environment:Mage_ai_Mage_ai_Python_3_9_Runtime