Needed resources:

• AWS account where we will auto-spawn EC2 instance. We will use t3a.small instance (2 vCPUs, 2GB RAM) which costs ~14$ per month in us-west-2 region (cheapest region). Also it will take $2 per month for EBS gp2 storage (20GB) for EC2 instance.
• Also AWS ECR will charge for $0.09 per GB of data egress traffic (from EC2 to the internet) - this needed to load docker build cache.

The setup shape:

• Build is done using IaaC approach with HashiCorp Terraform, so almoast no manual actions are needed from you. Every resource including EC2 server instance is described in code which is commited to repo.
• Docker images and build cache are stored on Amazon ECR
• Total build time for average commit to AdminForth app (with Vite rebuilds) is around 3 minutes.

Why exactly K3s?​

Previously, our blog featured posts about different types of application deployment, but without the use of Kubernetes. This post will look at the cheapest option for deploying an application using k3s (a lightweight version of k8s Kubernetes). This option is more interesting than most of the alternatives, primarily because of its automation and scalability (it is, of course, inferior to the “older” K8s, but it also requires significantly fewer resources).

How we will store containers?​

The ECR repository will be used for storage. Since we are working with AWS, this is the most reliable option. The image must be assembled from local files on the machine and then sent to the Amazon server. All instructions for performing these actions will be provided below.

Prerequisites

I will assume you run Ubuntu (Native or WSL2).

You should have terraform, here is official repository:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

AWS CLI:

sudo snap install aws-cli --classic

HELM:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh

Also you need Doker Daemon running. We recommend Docker Desktop running. ON WSL2 make sure you have Docker Desktop WSL2 integration enabled.

docker version

It is also worth having kubectl locally on your machine for more convenient interaction with nodes and pods, but this is not mandatory.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

Last step is download ansible:

sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible -y

Practice - deploy setup

Assume you have your AdminForth project in myadmin.

Step 1 - create a SSH keypair​

Make sure you are still in deploy folder, run next command:

mkdir .keys && ssh-keygen -f .keys/id_rsa -N ""

Now it should create deploy/.keys/id_rsa and deploy/.keys/id_rsa.pub files with your SSH keypair. Terraform script will put the public key to the EC2 instance and will use private key to connect to the instance. Also you will be able to use it to connect to the instance manually.

Step 2 - .gitignore file​

Create deploy/.gitignore file with next content:

.terraform/
.keys/
*.tfstate
*.tfstate.*
*.tfvars
tfplan
session-manager-plugin.deb
.terraform.lock.hcl

Step 3 - Terraform folder​

First of all install Terraform as described here terraform installation.

After this create folder ../deploy/terraform

Create file main.tf in deploy/terraform folder:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

}

locals {
  aws_region           = "us-west-2"
  vpc_cidr             = "10.0.0.0/16"
  subnet_a_cidr        = "10.0.10.0/24"
  subnet_b_cidr        = "10.0.11.0/24"
  az_a                 = "us-west-2a"
  az_b                 = "us-west-2b"
  app_name             = <your_app_name>
  app_source_code_path = "../../"
  ansible_dir          = "../ansible/playbooks"
  app_files            = fileset(local.app_source_code_path, "**")

  image_tag = sha256(join("", [
    for f in local.app_files :
    try(filesha256("${local.app_source_code_path}/${f}"), "")
    if length(regexall("^deploy/", f)) == 0
    && length(regexall("^\\.vscode/", f)) == 0
    && length(regexall("^node_modules/", f)) == 0
    && length(regexall("^\\.gitignore", f)) == 0
  ]))

  ingress_ports = [
    { from = 22, to = 22, protocol = "tcp", desc = "SSH" },
    { from = 80, to = 80, protocol = "tcp", desc = "App HTTP (Traefik)" },
    { from = 443, to = 443, protocol = "tcp", desc = "App HTTPS (Traefik)" },
    { from = 6443, to = 6443, protocol = "tcp", desc = "Kubernetes API" }
  ]
}

provider "aws" {
  region  = local.aws_region
  profile = "myaws"
}

data "aws_ami" "ubuntu_22_04" {
  most_recent = true
  owners      = ["099720109477"] # Canonical ubuntu account ID

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }
}

resource "aws_key_pair" "app_deployer" {
  key_name   = "terraform-deploy_${local.app_name}-key"
  public_key = file("../.keys/id_rsa.pub") # Path to your public SSH key
}

resource "aws_instance" "ec2_instance" {
  instance_type = "t3a.small"
  ami           = data.aws_ami.ubuntu_22_04.id

  iam_instance_profile = aws_iam_instance_profile.instance_profile.name

  subnet_id                   = aws_subnet.public_a.id
  vpc_security_group_ids      = [aws_security_group.app_sg.id]
  associate_public_ip_address = true
  key_name                    = aws_key_pair.app_deployer.key_name

  tags = {
    Name = local.app_name
  }

  depends_on = [
    null_resource.docker_build_and_push
  ]

  # prevent accidental termination of ec2 instance and data loss
  lifecycle {
    create_before_destroy = true #uncomment in production
    #prevent_destroy       = true       #uncomment in production
    ignore_changes = [ami]
    replace_triggered_by = [
      null_resource.docker_build_and_push
    ]
  }

  root_block_device {
    volume_size = 10 // Size in GB for root partition
    volume_type = "gp2"

    # Even if the instance is terminated, the volume will not be deleted, delete it manually if needed
    delete_on_termination = true #change to false in production if data persistence is needed
  }

}


resource "local_file" "ansible_inventory" {
  content = <<EOF
[k3s_nodes]
${aws_instance.ec2_instance.public_ip} ansible_user=ubuntu ansible_ssh_private_key_file=../.keys/id_rsa
EOF

  filename = "../ansible/inventory.ini"
}

resource "null_resource" "wait_ssh" {
  depends_on = [aws_instance.ec2_instance]

  triggers = (
    {
      instance_id = aws_instance.ec2_instance.id
    }
  )

  provisioner "local-exec" {
    command = <<EOT
    bash -c '
    for i in {1..10}; do
      nc -zv ${aws_instance.ec2_instance.public_ip} 22 && echo "SSH is ready!" && exit 0
      sleep 5
    done
    exit 1
    '
    EOT
  }
}

resource "null_resource" "ansible_provision" {
  depends_on = [
    aws_instance.ec2_instance,
    local_file.ansible_inventory,
    null_resource.wait_ssh,
    local_file.image_tag
  ]

  triggers = {
    instance_id = aws_instance.ec2_instance.id
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    command = <<-EOT
      set -e
      ANSIBLE_HOST_KEY_CHECKING=False ansible-galaxy collection install community.kubernetes
      ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ${path.module}/../ansible/inventory.ini ${local.ansible_dir}/playbook.yaml 
    EOT
  }
}

👆 Replace <your_app_name> with your app name (no spaces, only underscores or letters)

We will also need a file container.tf

resource "aws_ecr_repository" "app_repo" {
  name = local.app_name

  image_tag_mutability = "MUTABLE"
  image_scanning_configuration {
    scan_on_push = true
  }
  force_delete = true
}

data "aws_caller_identity" "current" {}

resource "null_resource" "docker_build_and_push" {
  depends_on = [aws_ecr_repository.app_repo]

  triggers = {
    image_tag = local.image_tag
  }

  provisioner "local-exec" {
    command = <<-EOT
      set -e
      unset DOCKER_HOST

      REPO_URL="${aws_ecr_repository.app_repo.repository_url}"
      ACCOUNT_ID="${data.aws_caller_identity.current.account_id}"
      REGION="${local.aws_region}"
      TAG="${local.image_tag}"

      echo "LOG: Logging in to ECR..."
      aws ecr get-login-password --region $${REGION} | docker login --username AWS --password-stdin $${ACCOUNT_ID}.dkr.ecr.$${REGION}.amazonaws.com
      
      echo "LOG: Building Docker image..."
      docker build --pull -t $${REPO_URL}:$${TAG} ${local.app_source_code_path}

      echo "LOG: Pushing image to ECR..."
      docker push $${REPO_URL}:$${TAG}

      echo "LOG: Build and push complete. TAG=$${TAG}"
    EOT

    interpreter = ["/bin/bash", "-c"]
  }
}

resource "local_file" "image_tag" {
  depends_on = [null_resource.docker_build_and_push]
  content    = local.image_tag
  filename   = "${path.module}/image_tag.txt"
}

This file contains a script that builds the Docker image locally. This is done for more flexible deployment. When changing the program code, there is no need to manually update the image on EC2 or in the repository. It is updated automatically with each terraform apply. Below is a table showing the time it takes to build this image from scratch and with minimal changes.

Also, resvpc.tf

resource "aws_vpc" "main" {
  cidr_block = local.vpc_cidr

  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = { Name = "main-vpc" }
}

resource "aws_subnet" "public_a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = local.subnet_a_cidr
  map_public_ip_on_launch = true
  availability_zone       = local.az_a
  tags = {
    Name = "public-a"
  }
}

resource "aws_subnet" "public_b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = local.subnet_b_cidr
  map_public_ip_on_launch = true
  availability_zone       = local.az_b
  tags = {
    Name = "public-b"
  }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "main-igw" }
}

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
  tags = { Name = "public-rt" }
}

resource "aws_route_table_association" "public_a_assoc" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_route_table_association" "public_b_assoc" {
  subnet_id      = aws_subnet.public_b.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_security_group" "app_sg" {
  name   = "${local.app_name}-SecurityGroup"
  vpc_id = aws_vpc.main.id

  dynamic "ingress" {
    for_each = local.ingress_ports
    content {
      from_port   = ingress.value.from
      to_port     = ingress.value.to
      protocol    = ingress.value.protocol
      cidr_blocks = ["0.0.0.0/0"]
      description = ingress.value.desc
    }
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_iam_role" "node_role" {
  name = "${local.app_name}node-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ecr_read_only_attach" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.node_role.name
}

resource "aws_iam_role_policy_attachment" "ssm_core_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.node_role.name
}

resource "aws_iam_instance_profile" "instance_profile" {
  name = "${local.app_name}-instance-profile"
  role = aws_iam_role.node_role.name
}

And outputs.tf

output "app_endpoint" {
  value = "http://${aws_instance.ec2_instance.public_dns}"
}

output "ssh_connect_command" {
  value = "ssh -i .keys/id_rsa ubuntu@${aws_instance.ec2_instance.public_dns}"
}

Step 4 - Helm​

Helm is a command-line tool and a set of libraries that helps manage applications in Kubernetes.

Helm Chart (Chart) is a package containing everything needed to run an application in Kubernetes. It's the equivalent of apt or yum packages in Linux.

A chart has this structure:

helm_charts/
├── Chart.yaml        # Metadata about the chart (name, version)
├── values.yaml       # Default values (configuration)
└── templates/        # Folder with Kubernetes templates (YAML files)
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    └── ...

Step 5 - Provider Helm​

Now we need to create .../deploy/helm and .../deploy/helm/helm_charts folders

you need to create a file Chart.yaml in it

apiVersion: v2
name: myadmink3s # SET YOUR APP NAME
description: Helm chart for myadmin app
version: 0.1.0
appVersion: "1.0.0"

And values.yaml

appName: myadmink3s # SET YOUR APP NAME LIKE IN Chart.yaml
appNameSpace: myadmin # SET YOUR APP NAMESPACE
containerPort: 3500
servicePort: 80
adminSecret: "your_secret"
ecrImageFull: ""

After this create .../deploy/helm/helm_charts/templates folder

And create files here:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.appName }}-deployment
  namespace: {{ .Values.appNameSpace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{ .Values.appName }}
  template:
    metadata:
      labels:
        app: {{ .Values.appName }}
    spec:
      containers:
      - name: {{ .Values.appName }}
        image: "{{ .Values.ecrImageFull }}" 
        ports:
        - containerPort: {{ .Values.containerPort }}
        env:
        - name: "ADMINFORTH_SECRET"
          value: "{{ .Values.adminSecret }}"

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Values.appName }}-ingress
  namespace: {{ .Values.appNameSpace }}
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Values.appName }}-service
            port:
              number: {{ .Values.servicePort }}

And service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.appName }}-service
  namespace: {{ .Values.appNameSpace }}
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.appName }}
  ports:
  - port: {{ .Values.servicePort }}
    targetPort: {{ .Values.containerPort }}

The comments in the values.yaml and Chart.yaml files indicate the names of the variables that need to be replaced. They must correspond to the variables in Ansible, which will be discussed later.

Step 6 - Ansible​

If we explain the logic of deployment, Ansible plays a very important role here. If Terraform is used exclusively to configure cloud infrastructure in AWS, Ansible prepares it for the deployment of a Kubernetes cluster. Ansible Playbooks, in simple terms, are templates for configuring the system (of course, their functionality is much broader, but in this deployment method and in most cases, this is how they are used). That is, after preparing the instance with Terraform, it launches Ansible, which prepares it for the deployment of the Kubernetes cluster, after which it launches Helm. This method allows for the highest quality deployment, because each stage is handled by software specialized for that particular stage.

So, create /deploy/ansible/playbooks folder

Then the file playbook.yaml

---
- name: Deploy application
  hosts: k3s_nodes
  become: true
  vars:
    k3s_version: "v1.27.3+k3s1"
    helm_version: "v3.12.3"
    kubeconfig_path: /etc/rancher/k3s/k3s.yaml
    helm_url: https://get.helm.sh/helm-v3.12.3-linux-amd64.tar.gz
    helm_dest: /usr/local/bin/helm
    app_name: myadmink3s    # <-- CHANGE TO YOUR APP NAME LIKE IN main.tf local.app_name
    app_namespace: myadmin  # <-- CHANGE TO YOUR APP NAMESPACE
    aws_account_id: "735356255780" # <-- CHANGE TO YOUR AWS ACCOUNT ID
    chart_path: /home/ubuntu/{{app_name}}/helm_charts

  tasks:

    - name: Read Docker image tag (local)
      ansible.builtin.set_fact:
        image_tag: "{{ lookup('file', '../../terraform/image_tag.txt') }}"
      delegate_to: localhost

    - name: Install unzip
      ansible.builtin.apt:
        name: unzip
        state: present
        update_cache: true

    - name: Download AWS CLI v2
      ansible.builtin.get_url:
        url: "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"
        dest: /tmp/awscliv2.zip
        mode: '0644'

    - name: Unzip AWS CLI
      ansible.builtin.unarchive:
        src: /tmp/awscliv2.zip
        dest: /tmp
        remote_src: true

    - name: Install AWS CLI
      ansible.builtin.command: /tmp/aws/install --update
      args:
        creates: /usr/local/bin/aws

    - name: Update apt cache
      ansible.builtin.apt:
        update_cache: true

    - name: Install required packages
      ansible.builtin.apt:
        name:
          - curl
          - sudo
          - software-properties-common
          - apt-transport-https
          - ca-certificates
        state: present

    - name: Download k3s installation script
      ansible.builtin.get_url:
        url: https://get.k3s.io
        dest: /tmp/install_k3s.sh
        mode: '0700'

    - name: Install k3s
      ansible.builtin.command: /tmp/install_k3s.sh
      environment:
        INSTALL_K3S_VERSION: "{{ k3s_version }}"
      args:
        creates: /usr/local/bin/k3s

    - name: Get ECR token
      ansible.builtin.command: aws ecr get-login-password --region us-west-2
      register: ecr_token
      changed_when: false

    - name: Configure K3s registry for ECR
      ansible.builtin.copy:
        dest: /etc/rancher/k3s/registries.yaml
        content: |
          configs:
            "735356255780.dkr.ecr.us-west-2.amazonaws.com":
              auth:
                username: AWS
                password: "{{ ecr_token.stdout }}"
        mode: '0600'

    - name: Restart k3s to apply registry changes
      ansible.builtin.systemd:
        name: k3s
        state: restarted
        enabled: true

    - name: Wait for k3s node to be ready
      ansible.builtin.wait_for:
        path: /usr/local/bin/k3s
        state: present
        timeout: 300

    - name: Ensure ~/.kube directory exists
      ansible.builtin.file:
        path: /root/.kube
        state: directory
        mode: '0700'

    - name: Copy k3s kubeconfig
      ansible.builtin.copy:
        remote_src: true
        src: /etc/rancher/k3s/k3s.yaml
        dest: /root/.kube/config
        owner: root
        group: root
        mode: '0600'

    - name: Replace localhost with public IP in kubeconfig
      ansible.builtin.replace:
        path: /root/.kube/config
        regexp: "server: https://127.0.0.1:6443"
        replace: "server: https://{{ inventory_hostname }}:6443"

    - name: Download Helm tarball
      ansible.builtin.get_url:
        url: "https://get.helm.sh/helm-{{ helm_version }}-linux-amd64.tar.gz"
        dest: "/tmp/helm.tar.gz"
        mode: '0644'

    - name: Extract Helm tarball
      ansible.builtin.unarchive:
        src: "/tmp/helm.tar.gz"
        dest: "/tmp/"
        remote_src: true

    - name: Move Helm binary to /usr/local/bin
      ansible.builtin.command:
        cmd: mv /tmp/linux-amd64/helm /usr/local/bin/helm
        creates: /usr/local/bin/helm

    - name: Ensure python3-pip is installed
      ansible.builtin.apt:
        name: python3-pip
        state: present
        update_cache: true

    - name: Install Python kubernetes library
      ansible.builtin.pip:
        name: kubernetes
        executable: pip3

    - name: Extract Helm binary
      ansible.builtin.unarchive:
        src: /tmp/helm.tar.gz
        dest: /tmp
        remote_src: true
        extra_opts: [--strip-components=1]
        creates: /tmp/helm

    - name: Move Helm binary to /usr/local/bin
      ansible.builtin.command:
        cmd: mv /tmp/helm /usr/local/bin/helm
        creates: /usr/local/bin/helm

    - name: Copy Helm chart to server
      ansible.builtin.copy:
        src: "../../helm/helm_charts"
        dest: /home/ubuntu/{{ app_name }}
        owner: ubuntu
        group: ubuntu
        mode: '0755'
        force: true

    - name: Ensure namespace exists - {{ app_namespace }}
      kubernetes.core.k8s:
        api_version: v1
        kind: Namespace
        name: "{{ app_namespace }}"
        kubeconfig: "{{ kubeconfig_path }}"

    - name: Deploy stack via Helm - {{ app_name }}
      kubernetes.core.helm:
        name: "{{ app_namespace }}"
        chart_ref: /home/ubuntu/{{ app_name }}/helm_charts
        release_namespace: "{{ app_namespace }}"
        kubeconfig: "{{ kubeconfig_path }}"
        create_namespace: false
        values_files:
          - /home/ubuntu/{{ app_name }}/helm_charts/values.yaml
        values:
          ecrImageFull: "{{ aws_account_id }}.dkr.ecr.us-west-2.amazonaws.com/{{ app_name }}:{{ image_tag }}"
        force: true
        atomic: false

Step 7 - Configure AWS Profile​

Open or create file ~/.aws/credentials and add (if not already there):

[myaws]
aws_access_key_id = <your_access_key>
aws_secret_access_key = <your_secret_key>

Then use

aws configure

And configure your AWS credentials

Step 8 - Run deployment​

All deployment-related actions are automated, so no additional actions are required. To deploy the application, you only need to enter a few commands listed below and wait a few minutes. After that, you will be able to connect to the web application using the link you will receive in terraform_output. Next, if you wish, you can add GitHub Actions. To do this, follow the instructions in our other post.

In ../deploy/terraform folder

terraform init

terraform apply -auto-approve

All done!​

Your application is now deployed on Amazon EC2 and available on the Internet.

Preparation​

First of all, you'll need to update VS Code to version 1.105.1 or higher. To check your version, go to Help -> About.

If you used a .deb file to install VS Code, just run these commands in the terminal to install updates:

sudo apt update 
sudo apt install code

Before opening VS Code, we need to get the Context7 API key. For this, go to https://context7.com/dashboard, create a new account, and copy the API key.

Setup​

Now we can proceed with the setup:

1. After opening VS Code, go to the extensions tab, where you'll see the new MCP SERVERS tab. Click on Enable MCP Servers Marketplace.

2. Find Context7 in the list and press Install:

3. Go to Show Configuration JSON:

4. Click Edit:

5. Insert your API key:

The installation is complete. You can now use it.

Example: Installation of the Bulk-ai-flow Plugin​

Here is an example prompt you can use to add the adminforth bulk-ai-flow plugin:

Prompt:
Add adminforth bulk-ai-flow plugin to this file using Context7 MCP

Generated Code:

Result:

As we can see, the generation works really well.

Tips​

If you don’t want to add use context7 to every prompt, you can define a simple rule in your MCP client's rule section.

If you're using github copilot, you can:

1. In the root of your repository, create the .github directory if it does not already exist.
2. create a file named .github/copilot-instructions.md
3. Inside of new file add:

Always use context7 when I need code generation, setup or configuration steps, or
library/API documentation. This means you should automatically use the Context7 MCP
tools to resolve library id and get library docs without me having to explicitly ask.

Let's consider simple example where we have a Page resource

import { AdminForthResourceInput } from "adminforth";

export default {
  dataSource: "maindb",
  table: "pages",
  resourceId: "pages",
  label: "Pages",
  columns: [
    {
      name: "url",
      primaryKey: true,
      showIn: { all: false },
    },
    {
      name: "meta_title",
      label: "Meta Title",
      type: "string",
      showIn: { all: true },
    },
    {
      name: "meta_desc",
      label: "Meta Description",
      type: "string",
      showIn: { all: true },
    },
  ]
} as AdminForthResourceInput;

You might have this page and return it in your API for nuxt:

import * as z from "zod";

app.get(`${admin.config.baseUrl}/api/get_page`,
  admin.express.withSchema(
    {
      description: 'Returns translated SEO metadata for the page specified by the pageUrl query parameter.',
      response: z.object({
        meta_title: z.string(),
        meta_desc: z.string(),
      }),
    },
    async (req:any, res: Response): Promise<void> => {
      const pageUrl = req.query.pageUrl;
      if (!pageUrl) {
        res.status(400).json({ error: "pageUrl is required" });
        return;
      }
      const page = await admin.resource("pages").get([Filters.EQ("url", pageUrl)]);
      if (!page) {
        res.status(404).json({ error: `Page not found ${pageUrl}` });
        return;
      }
      res.json({
        meta_title: page.meta_title,
        meta_desc: page.meta_desc,
      });
    }
  )
);

Install and import Zod before using this pattern: pnpm add zod or npm install zod, then import * as z from 'zod';. admin.express.withSchema(...) will convert the Zod schema to OpenAPI for you.

Now you want to translate page meta title and meta description. You can do this by using i18n plugin for AdminForth.

import { AdminForth } from "adminforth";
import * as z from "zod";

export const SEO_PAGE_CATEGORY = "seo_page_config";

app.get(`${admin.config.baseUrl}/api/get_page`,\
 admin.express.withSchema(
  {
    description: 'Returns translated SEO metadata for the page specified by the pageUrl query parameter.',
    response: z.object({
      meta_title: z.string(),
      meta_desc: z.string(),
    }),
  },
 admin.express.translatable(
    async (req:any, res: Response): Promise<void> => {
      const pageUrl = req.query.pageUrl;
      if (!pageUrl) {
        res.status(400).json({ error: "pageUrl is required" });
        return;
      }
      const page = await admin.resource("pages").get([Filters.EQ("url", pageUrl)]);
      if (!page) {
        res.status(404).json({ error: `Page not found ${pageUrl}` });
        return;
      }

      const translateKeys = [ "meta_title", "meta_desc", ];
      const [meta_title, meta_desc] = 
          await Promise.all(translateKeys.map((key: string) => req.tr(page[key], SEO_PAGE_CATEGORY)));
        

      res.json({
        meta_title: page.meta_title,
        meta_title,
        meta_desc: page.meta_desc,
        meta_desc,
      });
    }
  )
  ) 
);

Looks straightforward, but here are 2 issues:

Since translation strings are created only when first time you call req.tr function, you need to ensure you will call this API for all your pages (in all envs e.g. local, dev, staging, prod ), this might be not convinient - you have to call this API, then go to translation page and translate it with bulk action LLM or manually, but you might forget one page or so.

To fix this we suggest next, go to Page resource and add next function on top level:

import { AdminForthResourceInput } from "adminforth";
import AdminForth, { AdminForthResourceInput, Filters, IAdminForth } from "adminforth";
import I18nPlugin from "@adminforth/i18n/index.js";

import { SEO_PAGE_CATEGORY } from "../api.ts";

export async function feedAllPageTranslations(adminforth: IAdminForth) {
  const i18nPlugin = adminforth.getPluginByClassName<I18nPlugin>('I18nPlugin');
  const pages = await adminforth.resource('pages').list([]);
  await Promise.all(
    pages.map(async (page: any) => {
      await Promise.all(
      ['meta_title', 'meta_desc'].map(async (key) => {
        if (page[key]) {
          await i18nPlugin.feedCategoryTranslations(
            [{ en_string: page[key], source: `pages.${page.url}.${key}` }], SEO_PAGE_CATEGORY
          );
        }
      })
    );
  })
  );
}

export default {
  dataSource: "maindb",
  table: "pages",
  resourceId: "pages",
  ...

This function will iterate all pages and call feedCategoryTranslations function for each page and each key. This will create translation strings in your database for each page and each key. If translation objects already exist, it will skip them (will not create duplicates and will not overwrite translations).

Now we need to call this function in hooks:

export default {
  dataSource: "maindb",
  table: "pages",
  resourceId: "pages",
  label: "Pages",
  hooks: {
    create: {
      afterSave: async ({ record, adminforth }: { record: Record<string, string>, adminforth: IAdminForth }) => {
        feedAllPageTranslations(adminforth);
        return { ok: true };
      },
    },
    edit: {
      afterSave: async ({ oldRecord, updates, adminforth }: { oldRecord: Record<string, string>, updates: Record<string, string>, adminforth: IAdminForth }) => {
        feedAllPageTranslations(adminforth);
        return { ok: true }; 
      },
    },
  },  
  columns: [
    ...

Please note that we run this function without await, so it will not block your API. SO function will be called in background when hook will already return and user will not wait for it.

Now every time you will create or edit page, it will call feedAllPageTranslations function and create translation strings for each page and each key. You can also import this function into index script and run after database discover, if you already have pages in your database and you want to create translation strings for them even without clicking on create or edit button.

Issue 2 - after modification of page attributes old translation strings will not be removed

You can mitigate this by adding couple of lines into edit hook:

    edit: {
      afterSave: async ({ oldRecord, updates, adminforth }: { oldRecord: Record<string, string>, updates: Record<string, string>, adminforth: IAdminForth }) => {
         if (Object.keys(updates).length) {
          // find old strings which were edited and which are not used anymore
        const oldStrings = await adminforth.resource('translations').list([
            Filters.AND(
              Filters.EQ('category', 'seo_page_config'),
              Filters.IN('en_string', Object.keys(updates).map((key: string) => oldRecord[key]))
            )
          ]);
          // delete them
          await Promise.all(
            oldStrings.map((oldString: any) => {
              return adminforth.resource('translations').delete(oldString.id);
            })
          );
        }
        feedAllPageTranslations(adminforth);
        return { ok: true }; 
      },
    },

This will delete all old translation strings which are not used anymore.

If your have ability to delete pages, you can also add delete hook, you can do this as a homework.

Conclusion

In this article we have shown how to translate dynamic strings in AdminForth API. We have also shown how to create translation strings for each page and each key in your database. We have also shown how to delete old translation strings which are not used anymore. This will help you to keep your translation strings clean and up to date.

In this guide, we will walk you through the process of setting up AdminForth authorization via Keycloak. Most important we will show you how to set up Keycloak in a Docker container and configure it to work with AdminForth.

Prerequisites​

• Docker installed on your machine
• Basic knowledge of Docker and Docker Compose

Step 1: Create a Docker Compose File​

Create a docker-compose.yml file in your project directory. This file will define the Keycloak service and its configuration.

services:
  pg:
    image: postgres:latest
    environment:
      POSTGRES_USER: demo
      POSTGRES_PASSWORD: demo
      POSTGRES_DB: demo
    ports:
      - "5432:5432"
    volumes:
      - pg-data:/var/lib/postgresql/data
  keycloak:
    image: quay.io/keycloak/keycloak:latest
    command: start-dev
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin
      - DB_VENDOR=postgres
      - DB_ADDR=pg
      - DB_DATABASE=demo
      - DB_USER=demo
      - DB_PASSWORD=demo
    ports:
      - "8080:8080"
    depends_on:
      - pg
    volumes:
      - keycloak-data:/opt/keycloak/data

volumes:
  keycloak-data:

Run service:

docker compose -p af-dev-demo up -d --build --remove-orphans --wait

Step 2: Singn in to Keycloak and Create a Keycloak Realm​

1. Open the Keycloak UI at http://localhost:8080/.
2. Sign in with the credentials admin and admin.

3. Select the "Realms" tab and click Create Realm.

4. Enter a name for your realm and click Create.

Step 3: Create a Keycloak Client​

1. Go to Clients tab and click Create Client.

2. Choose OpenID Connect, enter a client ID for your client and click Next.

3. Swith Client authentication to On and click Next.

4. Enter a Valid redirect URI and click Save.

5. In the Client details go to Credentials tab and copy the Client secret.

6. Add the credentials to your .env file:

KEYCLOAK_CLIENT_ID=your_keycloak_client_id
KEYCLOAK_CLIENT_SECRET=your_keycloak_client_secret
KEYCLOAK_REALM=your_keycloak_realm
KEYCLOAK_URL=http://localhost:8080

Step 4: Create a Keycloak User​

1. Go to Users tab and click Create new user.

2. Enter a Username, Email, First name and Last name and click Create.

3. In the User details go to Credentials tab and click Set password.

4. Enter a Password, turn Temporary to Off and click Save.

Finally, you can sign in to AdminForth with your Keycloak credentials. (if user with the same email exists in AdminForth)

This guide shows how to deploy own Docker apps (with AdminForth as example) to Amazon EC2 instance with Docker and Terraform involving pushing images into Amazon ECR.

Needed resources:

• GitHub actions Free plan which includes 2000 minutes per month (1000 of 2-minute builds per month - more then enough for many projects, if you are not running tests). Extra builds would cost 0.008$ per minute.
• AWS account where we will auto-spawn EC2 instance. We will use t3a.small instance (2 vCPUs, 2GB RAM) which costs ~14$ per month in us-east-1 region (cheapest region). Also it will take $2 per month for EBS gp2 storage (20GB) for EC2 instance.
• Also AWS ECR will charge for $0.09 per GB of data egress traffic (from EC2 to the internet) - this needed to load docker build cache.

The setup shape:

• Build is done using IaaC approach with HashiCorp Terraform, so almoast no manual actions are needed from you. Every resource including EC2 server instance is described in code which is commited to repo.
• Docker build process is done on GitHub actions server, so EC2 server is not overloaded with builds
• Changes in infrastructure including changing server type, adding S3 Bucket, changing size of sever disk is also can be done by commiting code to repo.
• Docker images and build cache are stored on Amazon ECR
• Total build time for average commit to AdminForth app (with Vite rebuilds) is around 2 minutes.

Previously we had a blog post about deploying AdminForth to EC2 with Terraform without registry. That method might work well but has a significant disadvantage - build process happens on EC2 itself and uses EC2 RAM and CPU. This can be a problem if your EC2 instance is well-loaded without extra free resources. Moreover, low-end EC2 instances have a small amount of RAM and CPU, so build process which involves vite/tsc/etc can be slow or even fail / cause OOM killer to crash EC2 instance.

So obviously to solve this problem we need to move the build process to CI, however it introduces new chellenges and we will solve them in this post.

Quick difference between approaches from previous post and current post:

Chellenges when you build on CI​

A little bit of theory.

When you move build process to CI you have to solve next chellenges:

1. We need to deliver built docker images to EC2 somehow (and only we)
2. We need to persist cache between builds

Delivering images​

Exporing images to tar files​

Simplest option which you can find is save docker images to tar files and deliver them to EC2. We can easily do it in terraform (using docker save -o ... command on CI and docker load ... command on EC2). However this option has a significant disadvantage - it is slow. Docker images are big (always include all layers, without any options), so it takes infinity to do save/load and another infinity to transfer them to EC2 (via relatively slow rsync/SSH and relatively slow GitHub actions outbound connection).

Docker registry​

Faster, right option which we will use here - involve Docker registry. Registry is a repository which stores docker images. It does it in a smart way - it saves each image as several layers, so if you will update last layer, then only last layer will be pushed to registry and then only last will be pulled to EC2. To give you row compare - whole-layers image might take 1GB, but last layer created by npm run build command might take 50MB. And most builds you will do only last layer changes, so it will be 20 times faster to push/pull last layer than whole image. And this is not all, registry uses TLS HTTP protocol so it is faster then SSH/rsync encrypted connection.

Of course you have to care about a way of registry authentication (so only you and your CI/EC2 can push/pull images).

What docker registry can you use? Pretty known options:

1. Docker Hub - most famous. It is free for public images, so literally every opensource project uses it. However it is not free for private images, and you have to pay for it. Payment model is pretty strange - you pay for user who can login, like 11$ per month, you might pay for your devops only but all this sounds strange.
2. GHCR - Registry from GitHub. Has free plan but allows to store only 500MB and allows to transfer 1GB of traffic per month. Then you pay for every extra GB in storage ($0.0008 per GB/day or $0.24 per GB/month) and for every extra GB in traffic ($0.09 per GB). Probably small images will fit in free plan, but generally even alpine-based docker images are bigger than 500MB, so it is non-free option.
3. Amazon ECR - Same as GHCR but from Amazon. Price is $0.10 per GB of storage per month and $0.09 per GB of data transfer from Amazon (as all Amazon egress traffic). So it is cheaper than GHCR.
4. Self-hosted registry web system. In our software development company, we use Harbor. It is a powerful free open-source registry that can be installed to own server. It allows pushing and pulling without limit. Also, it has internal life-cycle rules that cleanup unnecessary images and layers. The main drawbacks of it are that it is not so fast to install and configure, plus you have to get a domain and another powerfull server to run it. So unless you are a software development company, it is not worth using it.
5. Self-hosted minimal CNCF Distribution registry on EC2 itself. So since we already have EC2, we can run registry on it directly. The registry container is pretty light-weight and it will not consume a lot of extra CPU/RAM on server. Plus images will be stored close to application so pull will be fast, however securing this right is a bit tricky. If you want to try it we have special EC2 with CNCF registry post.

In the post we will use Amazon ECR as registry (3rd way).

Persisting cache​

Docker builds without layer cache persistence are possible but very slow. Most builds only change a couple of layers, and having no ability to cache them will cause the Docker builder to regenerate all layers from scratch. This can, for example, increase the Docker build time from a minute to ten minutes or even more.

Out of the box, GitHub Actions can't save Docker layers between builds, so you have to use external storage.

Though some CI systems can persist docker build cache, e.g. open-source self-hosted Woodpecker CI allows it out of the box. However GitHub actions which is pretty popular, reasonably can't allow such free storage to anyone

So when build-in Docker cache can't be used, there is one alternative - Docker BuildKit external cache. So BuildKit allows you to connect external storage. There are several options, but most sweet for us is using Docker registry as cache storage (not only as images storage to deliver them to application server).

Drawback is that buildx which is running on GitHub action server will download cache from registry which is inside of AWS. And all AWS egress traffic is charged. So you will pay for every build which uses cache. However cache is comnpressed. To give you idea basic alpine image with AdminForth cache is 180MB. One one commercial project we did full release within 2 months and 400 builds so it took 400 * 180MB * $0.09 / GB = $6.48 for cache transfer for whole project.

BuildKit cache in Compose issue Previously we used docker compose to build & run our app, it can be used to both build, push and pull images, but has issues with external cache connection. While they are not solved we have to use docker buildx bake command to build images. It is not so bad, but is another point of configuration which we will cover in this post.

Prerequisites

I will assume you run Ubuntu (Native or WSL2).

You should have terraform, here is official repository:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

AWS CLI:

sudo snap install aws-cli --classic

Also you need Doker Daemon running. We recommend Docker Desktop running. ON WSL2 make sure you have Docker Desktop WSL2 integration enabled.

docker version

Practice - deploy setup

Assume you have your AdminForth project in myadmin.

Step 1 - Dockerfile and .dockerignore​

This guide assumes you have created your AdminForth application with latest version of adminforth create-app command. This command already creates a Dockerfile and .dockerignore for you, so you can use them as is.

Step 2 - compose.yml​

create folder deploy and create file compose.yml inside:

services:
  traefik:
    image: "traefik:v2.5"
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  myadmin:
    image: ${MYADMIN_REPO}:latest
    build:
      context: ../adminforth-app
      tags:
        - ${MYADMIN_REPO}:latest
      cache_from:
        - type=registry,ref=${MYADMIN_REPO}:cache
      cache_to:
        - type=registry,ref=${MYADMIN_REPO}:cache,mode=max,compression=zstd,image-manifest=true,oci-mediatypes=true
      
    pull_policy: always
    restart: always
    env_file:
      - .env.secrets.prod

    volumes:
      - myadmin-db:/code/db
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myadmin.rule=PathPrefix(`/`)"
      - "traefik.http.services.myadmin.loadbalancer.server.port=3500"
      - "traefik.http.routers.myadmin.priority=2"

volumes:
  myadmin-db:

Step 3 - create a SSH keypair​

Make sure you are still in deploy folder, run next command:

mkdir .keys && ssh-keygen -f .keys/id_rsa -N ""

Now it should create deploy/.keys/id_rsa and deploy/.keys/id_rsa.pub files with your SSH keypair. Terraform script will put the public key to the EC2 instance and will use private key to connect to the instance. Also you will be able to use it to connect to the instance manually.

Step 4 - create TLS certificates to encrypt traffic between CI and registry​

Make sure you are still in deploy folder, run next command:

Run next command to create TLS certificates:

openssl req -new -x509 -days 3650 -newkey rsa:4096 -nodes -keyout .keys/ca.key -subj "/CN=My Custom CA" -out .keys/ca.pem

This will create deploy/.keys/ca.key and deploy/.keys/ca.pem files.

Step 5 - .gitignore file​

Create deploy/.gitignore file with next content:

.terraform/
.keys/
*.tfstate
*.tfstate.*
*.tfvars
tfplan
.env.secrets.prod

Step 6 - file with secrets for local deploy​

Create file deploy/.env.secrets.prod

ADMINFORTH_SECRET=<your_secret>

Step 7 - main terraform file main.tf​

First of all install Terraform as described here terraform installation.

Create file main.tf in deploy folder:

locals {
  app_name = "testtf"
  aws_region = "us-east-1"
}

provider "aws" {
  region = local.aws_region
  profile = "myaws"
}

data "aws_ami" "ubuntu_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
  }
}

data "aws_vpc" "default" {
  default = true
}

resource "aws_eip" "eip" {
 domain = "vpc"
}

resource "aws_eip_association" "eip_assoc" {
 instance_id   = aws_instance.app_instance.id
 allocation_id = aws_eip.eip.id
}

data "aws_subnet" "default_subnet" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }

  filter {
    name   = "default-for-az"
    values = ["true"]
  }

  filter {
    name   = "availability-zone"
    values = ["${local.aws_region}a"]
  }
}

resource "aws_security_group" "instance_sg" {
  name   = "${local.app_name}-instance-sg"
  vpc_id = data.aws_vpc.default.id

  ingress {
    description = "Allow HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # SSH
  ingress {
    description = "Allow SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_key_pair" "app_deployer" {
  key_name   = "terraform-deploy_${local.app_name}-key"
  public_key = file("./.keys/id_rsa.pub") # Path to your public SSH key
}

resource "aws_instance" "app_instance" {
  ami                    = data.aws_ami.ubuntu_linux.id
  instance_type          = "t3a.small"  # just change it to another type if you need, check https://instances.vantage.sh/
  subnet_id              = data.aws_subnet.default_subnet.id
  vpc_security_group_ids = [aws_security_group.instance_sg.id]
  key_name               = aws_key_pair.app_deployer.key_name
  iam_instance_profile = aws_iam_instance_profile.ec2_profile.name

  # prevent accidental termination of ec2 instance and data loss
  # if you will need to recreate the instance still (not sure why it can be?), you will need to remove this block manually by next command:
  # > terraform taint aws_instance.app_instance
  lifecycle {
    prevent_destroy = true
    ignore_changes = [ami]
  }

  root_block_device {
    volume_size = 20 // Size in GB for root partition
    volume_type = "gp2"
    
    # Even if the instance is terminated, the volume will not be deleted, delete it manually if needed
    delete_on_termination = false
  }

  user_data = <<-EOF
    #!/bin/bash
    set -euo pipefail

    LOG_FILE="/home/ubuntu/user_data.log"
    exec > >(tee -a "$LOG_FILE") 2>&1
    chown ubuntu:ubuntu "$LOG_FILE" || true

    touch /home/ubuntu/user_data_started

    on_error() {
      echo "failed" > /home/ubuntu/user_data_failed
    }
    trap on_error ERR

    export DEBIAN_FRONTEND=noninteractive

    apt-get update
    apt-get install -y --no-install-recommends \
      ca-certificates \
      curl

    # Install the latest Docker Engine from Docker's official Ubuntu repo
    install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    chmod a+r /etc/apt/keyrings/docker.asc

    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
      $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
      > /etc/apt/sources.list.d/docker.list

    apt-get update
    apt-get install -y --no-install-recommends \
      docker-ce \
      docker-ce-cli \
      containerd.io \
      docker-buildx-plugin \
      docker-compose-plugin

    # Ensure Docker daemon is up
    systemctl enable --now docker

    # Allow ubuntu user to run docker without sudo (new SSH session required)
    usermod -aG docker ubuntu || true

    docker --version
    docker compose version

    echo "done" > /home/ubuntu/user_data_done

  EOF

  tags = {
    Name = "${local.app_name}-instance"
  }
}

resource "null_resource" "wait_for_user_data" {
  provisioner "remote-exec" {
    inline = [
      "echo 'Waiting for EC2 software install to finish...'",
      "while [ ! -f /home/ubuntu/user_data_done ]; do echo '...'; sleep 2; done",
      "echo 'EC2 software install finished.'"
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("./.keys/id_rsa")
      host        = aws_eip_association.eip_assoc.public_ip
    }
  }

  depends_on = [aws_instance.app_instance]
}

resource "aws_ecr_repository" "myadmin_repo" {
  name = "${local.app_name}-myadmin"
  force_delete = true
}

resource "aws_ecr_lifecycle_policy" "safe_cleanup" {
  repository = aws_ecr_repository.myadmin_repo.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Delete untagged images older than 7 days"
        selection = {
          tagStatus     = "untagged"
          countType     = "sinceImagePushed"
          countUnit     = "days"
          countNumber   = 7
        }
        action = {
          type = "expire"
        }
      }
    ]
  })
}

resource "local_file" "compose_env" {
  content  = "MYADMIN_REPO=${aws_ecr_repository.myadmin_repo.repository_url}"
  filename = "${path.module}/.env.ecr"
}

// allow ec2 instance to login to ECR too pull images
resource "aws_iam_role" "ec2_role" {
  name = "${local.app_name}-ec2-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = {
        Service = "ec2.amazonaws.com"
      },
      Action = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ecr_access" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser"
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "${local.app_name}-instance-profile"
  role = aws_iam_role.ec2_role.name
}

resource "null_resource" "sync_files_and_run" {

  provisioner "local-exec" {
    command = <<-EOF
      aws ecr get-login-password --region ${local.aws_region} --profile myaws | docker login --username AWS --password-stdin ${aws_ecr_repository.myadmin_repo.repository_url}

      echo "Running build"
      env $(cat .env.ecr | grep -v "#" | xargs) docker buildx bake --progress=plain --push --allow=fs.read=.. -f compose.yml

      # if you will change host, pleasee add -o StrictHostKeyChecking=no
      echo "Copy files to the instance" 
      rsync -t -avz --mkpath -e "ssh -i ./.keys/id_rsa -o StrictHostKeyChecking=no" \
        --delete \
        --exclude '.terraform' \
        --exclude '.keys' \
        --exclude 'tfplan' \
        . ubuntu@${aws_eip_association.eip_assoc.public_ip}:/home/ubuntu/app/deploy/

      EOF
  }

  # Run docker compose after files have been copied
  provisioner "remote-exec" {
    inline = [<<-EOF
      aws ecr get-login-password --region ${local.aws_region} | docker login --username AWS --password-stdin ${aws_ecr_repository.myadmin_repo.repository_url}

      cd /home/ubuntu/app/deploy

      echo "Spinning up the app"
      docker compose --progress=plain -p app  --env-file .env.ecr -f compose.yml up -d --remove-orphans

      # cleanup unused cache (run in background to not block terraform)
      screen -dm docker system prune -f
    EOF
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("./.keys/id_rsa")
      host        = aws_eip_association.eip_assoc.public_ip
    }


  }

  # Ensure the resource is triggered every time based on timestamp or file hash
  triggers = {
    always_run = timestamp()
  }

  depends_on = [aws_eip_association.eip_assoc, null_resource.wait_for_user_data]
}


output "instance_public_ip" {
  value = aws_eip_association.eip_assoc.public_ip
}


######### META, tf state ##############


# S3 bucket for storing Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket = "${local.app_name}-terraform-state"
}

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  rule {
    status = "Enabled"
    id = "Keep only the latest version of the state file"

    filter {
      prefix = ""
    }

    noncurrent_version_expiration {
      noncurrent_days = 30
    }
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "AES256"
    }
  }
}

👆 Replace <your_app_name> with your app name (no spaces, only underscores or letters)

Step 7.1 - Configure AWS Profile​

Open or create file ~/.aws/credentials and add (if not already there):

[myaws]
aws_access_key_id = <your_access_key>
aws_secret_access_key = <your_secret_key>

Step 7.2 - Run deployment​

We will run first deployment from local machine to create S3 bucket for storing Terraform state. In other words this deployment will create resources needed for storing Terraform state in the cloud and runnign deployment from GitHub actions.

In deploy folder run:

terraform init

Now run deployement:

terraform apply -auto-approve

👆 Please note that this command might block ask you your sudo password to append appserver.local to /etc/hosts file.

👆 Please note that command might show errors about pushing images, this is fine because current deployment is done here only to setup S3 bucket for state migration before migrating to cloud.

Step 8 - Migrate state to the cloud​

First deployment had to create S3 bucket for storing Terraform state. Now we need to migrate the state to the cloud.

Add to the end of main.tf:

# Configure the backend to use the S3 bucket
terraform {
 backend "s3" {
   bucket         = "<your_app_name>-terraform-state"
   key            = "state.tfstate"  # Define a specific path for the state file
   region         = "us-east-1"
   profile        = "myaws"
   use_lockfile   = true
 }
}

👆 Replace <your_app_name> with your app name (no spaces, only underscores or letters). Unfortunately we can't use variables, HashiCorp thinks it is too dangerous 😥

Now run:

terraform init -migrate-state

Now run test deployment:

terraform apply -auto-approve

Now you can delete local terraform.tfstate file and terraform.tfstate.backup file as they are in the cloud now.

Step 9 - CI/CD - Github Actions​

Create file .github/workflows/deploy.yml:

name: Deploy myadmin
run-name: ${{ github.actor }} builds myadmin 🚀
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest

    concurrency:
      group: build-group
      cancel-in-progress: false

    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server"
      - run: echo "🔎 The name of your branch is ${{ github.ref }}"
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.10.1 
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Import SSH keys
        run: |
          mkdir -p deploy/.keys
          echo "$VAULT_SSH_PRIVATE_KEY" > deploy/.keys/id_rsa
          echo "$VAULT_SSH_PUBLIC_KEY" > deploy/.keys/id_rsa.pub
          chmod 600 deploy/.keys/id_rsa*
        env:
          VAULT_SSH_PRIVATE_KEY: ${{ secrets.VAULT_SSH_PRIVATE_KEY }}
          VAULT_SSH_PUBLIC_KEY: ${{ secrets.VAULT_SSH_PUBLIC_KEY }}

      - name: Setup AWS credentials
        run: |
          mkdir -p ~/.aws
          cat <<EOL > ~/.aws/credentials
          [myaws]
          aws_access_key_id=${VAULT_AWS_ACCESS_KEY_ID}
          aws_secret_access_key=${VAULT_AWS_SECRET_ACCESS_KEY}
          EOL
        env:
          VAULT_AWS_ACCESS_KEY_ID: ${{ secrets.VAULT_AWS_ACCESS_KEY_ID }}
          VAULT_AWS_SECRET_ACCESS_KEY: ${{ secrets.VAULT_AWS_SECRET_ACCESS_KEY }}

      - name: Prepare env
        run: |
          echo "ADMINFORTH_SECRET=$VAULT_ADMINFORTH_SECRET" > deploy/.env.secrets.prod
        env:
          VAULT_ADMINFORTH_SECRET: ${{ secrets.VAULT_ADMINFORTH_SECRET }}

      - name: Terraform build
        run: |
          cd deploy
          terraform init -reconfigure
          # example of unlocking tf state if needed
          # terraform force-unlock fb397548-8697-ea93-ab80-128a4f508fdf --force
          terraform plan -out=tfplan 
          terraform apply tfplan 
                
          
      - run: echo "🍏 This job's status is ${{ job.status }}."

Step 10 - Add secrets to GitHub​

Go to your GitHub repository, then Settings -> Secrets -> New repository secret and add:

• VAULT_AWS_ACCESS_KEY_ID - your AWS access key
• VAULT_AWS_SECRET_ACCESS_KEY - your AWS secret key
• VAULT_SSH_PRIVATE_KEY - execute cat deploy/.keys/id_rsa and paste to GitHub secrets
• VAULT_SSH_PUBLIC_KEY - execute cat deploy/.keys/id_rsa.pub and paste to GitHub secrets
• VAULT_ADMINFORTH_SECRET - generate some random string and paste to GitHub secrets, e.g. openssl rand -base64 32 | tr -d '\n'

Now you can push your changes to GitHub and see how it will be deployed automatically.

Adding secrets​

Once you will have sensitive tokens/passwords in your apps you have to store them in a secure way.

Simplest way is to use GitHub secrets.

Let's imagine you have OPENAI_API_KEY which will be used one of AI-powered plugins of adminforth. We can't put this key to the code, so we have to store it in GitHub secrets.

Open your GitHub repository, then Settings -> Secrets -> New repository secret and add VAULT_OPENAI_API_KEY with your key.

Now open GitHub actions file and add it to the env section:

      - name: Prepare env
        run: |
          echo "ADMINFORTH_SECRET=$VAULT_ADMINFORTH_SECRET" > deploy/.env.secrets.prod
          echo "OPENAI_API_KEY=$VAULT_OPENAI_API_KEY" >> deploy/.env.secrets.prod
        env:
          VAULT_ADMINFORTH_SECRET: ${{ secrets.VAULT_ADMINFORTH_SECRET }}
          VAULT_OPENAI_API_KEY: ${{ secrets.VAULT_OPENAI_API_KEY }}

In the same way you can add any other secrets to your GitHub actions.

How to connect to EC2 instance?​

To connect to EC2 instance you can use SSH.

cd deploy
ssh -i ./.keys/id_rsa ubuntu@<your_ec2_ip>

IP address can be found in terminal output after terraform apply.

Out of space on EC2 instance? Extend EBS volume​

To upgrade EBS volume size you have to do next steps:

In main.tf file:

  root_block_device {
    volume_size = 20 // Size in GB for root partition
    volume_size = 40 // Size in GB for root partition
    volume_type = "gp2"
  }

And run build.

This will increase physical size of EBS volume, but you have to increase filesystem size too.

Login to EC2 instance:

ssh -i ./.keys/id_rsa ubuntu@<your_ec2_ip>

You can find your EC2 IP in AWS console by visiting EC2 -> Instances -> Your instance -> IPv4 Public IP

Now run next commands:

lsblk

This would show something like this:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0     7:0    0 99.4M  1 loop /snap/core/10908
nvme0n1 259:0    0   40G  0 disk
└─nvme0n1p1 259:1    0   20G  0 part /

Here we see that nvme0n1 is our disk and nvme0n1p1 is our partition.

Now to extend partition run:

sudo growpart /dev/nvme0n1 1
sudo resize2fs /dev/nvme0n1p1

This will extend partition to the full disk size. No reboot is needed.

Want slack notifications about build?​

Create Slack channel and add Slack app to it.

Then create webhook URL and add it to GitHub secrets as SLACK_WEBHOOK_URL.

Add this steps to the end of your GitHub actions file:

      - name: Notify Slack on success
        if: success()
        run: |
          curl -X POST -H 'Content-type: application/json' --data \
          "{\"text\": \"✅ *${{ github.actor }}* successfully built *${{ github.ref_name }}* with commit \\\"${{ github.event.head_commit.message }}\\\".\n:link: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Build> | :link: <${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}|View Commit>\"}" \
          $SLACK_WEBHOOK_URL 
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

      - name: Notify Slack on failure
        if: failure()
        run: |
          curl -X POST -H 'Content-type: application/json' --data \
          "{\"text\": \"❌ *${{ github.actor }}* failed to build *${{ github.ref_name }}* with commit \\\"${{ github.event.head_commit.message }}\\\".\n:link: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Build> | :link: <${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}|View Commit>\"}" \
          $SLACK_WEBHOOK_URL 
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

This guide is a hackers extended addition of Deploying AdminForth to EC2 with Amazon ECR. The key difference in this post that we will not use Amazon ECR but self-host registry on EC2 itself. Automatically from terraform. And will see whether we will win something in terms of build time.

Costs for Amazon ECR vs consts for self-hosted registry on EC2​

Most of AWS services are formed from EC2 prices plus some extra overhead for own cost. In same way, Amazon ECR pricing is pretty same.

So as you can see there is still no difference in terms of cost. However the approach in this system allows to replace Amazon EC2 with any other cloud provider which does not charge for egress traffic.

Bechnmarking build time

When I implmented this solution, I was curious whether it will be faster to build images on EC2 or on CI. So I did a little bit of testing. First I used results form deploying AdminForth to EC2 with Terraform without registry where we built images on EC2. Then I did the same test but with self-hosted registry on EC2 and compared to deploying AdminForth to EC2 with Amazon ECR where we built images on CI and pushed to Amazon ECR.

So it indeed own self-hosted registry is faster then ECR and overall build time of pure AdminForth is faster then building on EC2. When ECR is slower then self-hosted registry, it is because of network speed.

Chellenges when you build on CI

Registry authorization and traffic encryption

Hosting custom CNCF registry, from other hand is a security responsibility.

If you don't protect it right, someone will be able to push any image to your registry and then pull it to your EC2 instance. This is a big security issue, so we have to protect our registry.

First of all we need to set some authorization to our registry so everyone who will push/pull images will be authorized. Here we have 2 options: HTTP basic auth and Client certificate auth. We will use first one as it is easier to setup. We will generate basic login and password automatically in terraform so no extra actions are needed from you.

But this is not enough. Basic auth is not encrypted, so someone can perform MITM attack and get your credentials. So we need to encrypt traffic between CI and registry. We can do it by using TLS certificates. So we will generate self-signed TLS certificates, and attach them to our registry.

Though the challenge is that we need to provide CA certificate to every daemon which will work with our registry. So we need to provide CA certificate to buildx daemon on CI, also if we want to do it from local machine, we need to provide CA certificate to local docker daemon.

Practice - deploy setup

Assume you have your AdminForth project in myadmin.

Step 1 - Dockerfile and .dockerignore​

This guide assumes you have created your AdminForth application with latest version of adminforth create-app command. This command already creates a Dockerfile and .dockerignore for you, so you can use them as is.

Step 2 - compose.yml​

create folder deploy and create file compose.yml inside:

services:
  traefik:
    image: "traefik:v2.5"
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  myadmin:
    image: localhost:5000/myadmin:latest
    build:
      context: ../adminforth-app
      tags:
        - localhost:5000/myadmin:latest
      cache_from:
        - type=registry,ref=localhost:5000/myadmin:cache
      cache_to:
        - type=registry,ref=localhost:5000/myadmin:cache,mode=max,compression=zstd,image-manifest=true,oci-mediatypes=true
      
    pull_policy: always
    restart: always
    env_file:
      - .env.secrets.prod

    volumes:
      - myadmin-db:/code/db
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myadmin.rule=PathPrefix(`/`)"
      - "traefik.http.services.myadmin.loadbalancer.server.port=3500"
      - "traefik.http.routers.myadmin.priority=2"

volumes:
  myadmin-db:

Step 3 - create a SSH keypair​

Make sure you are still in deploy folder, run next command:

mkdir .keys && ssh-keygen -f .keys/id_rsa -N ""

Now it should create deploy/.keys/id_rsa and deploy/.keys/id_rsa.pub files with your SSH keypair. Terraform script will put the public key to the EC2 instance and will use private key to connect to the instance. Also you will be able to use it to connect to the instance manually.

Step 4 - create TLS certificates to encrypt traffic between CI and registry​

Make sure you are still in deploy folder, run next command:

Run next command to create TLS certificates:

openssl req -new -x509 -days 3650 -newkey rsa:4096 -nodes -keyout .keys/ca.key -subj "/CN=My Custom CA" -out .keys/ca.pem

This will create deploy/.keys/ca.key and deploy/.keys/ca.pem files.

Step 5 - .gitignore file​

Create deploy/.gitignore file with next content:

.terraform/
.keys/
*.tfstate
*.tfstate.*
*.tfvars
tfplan
.env.secrets.prod

Step 6 - file with secrets for local deploy​

Create file deploy/.env.secrets.prod

ADMINFORTH_SECRET=<your_secret>

Step 7 - main terraform file main.tf​

First of all install Terraform as described here terraform installation.

Create file main.tf in deploy folder:

locals {
  app_name = "<your_app_name>"
  aws_region = "us-east-1"
}

provider "aws" {
  region = local.aws_region
  profile = "myaws"
}

data "aws_ami" "ubuntu_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
  }
}

data "aws_vpc" "default" {
  default = true
}

resource "aws_eip" "eip" {
 domain = "vpc"
}
resource "aws_eip_association" "eip_assoc" {
 instance_id   = aws_instance.app_instance.id
 allocation_id = aws_eip.eip.id
}

data "aws_subnet" "default_subnet" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }

  filter {
    name   = "default-for-az"
    values = ["true"]
  }

  filter {
    name   = "availability-zone"
    values = ["${local.aws_region}a"]
  }
}

resource "aws_security_group" "instance_sg" {
  name   = "${local.app_name}-instance-sg"
  vpc_id = data.aws_vpc.default.id

  ingress {
    description = "Allow HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "Allow Docker registry"
    from_port   = 5000
    to_port     = 5000
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # SSH
  ingress {
    description = "Allow SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_key_pair" "app_deployer" {
  key_name   = "terraform-deploy_${local.app_name}-key"
  public_key = file("./.keys/id_rsa.pub") # Path to your public SSH key
}

resource "aws_instance" "app_instance" {
  ami                    = data.aws_ami.ubuntu_linux.id
  instance_type          = "t3a.small"  # just change it to another type if you need, check https://instances.vantage.sh/
  subnet_id              = data.aws_subnet.default_subnet.id
  vpc_security_group_ids = [aws_security_group.instance_sg.id]
  key_name               = aws_key_pair.app_deployer.key_name

  # prevent accidental termination of ec2 instance and data loss
  # if you will need to recreate the instance still (not sure why it can be?), you will need to remove this block manually by next command:
  # > terraform taint aws_instance.app_instance
  lifecycle {
    prevent_destroy = true
    ignore_changes = [ami]
  }

  root_block_device {
    volume_size = 20 // Size in GB for root partition
    volume_type = "gp2"
    
    # Even if the instance is terminated, the volume will not be deleted, delete it manually if needed
    delete_on_termination = false
  }

  user_data = <<-EOF
    #!/bin/bash
    sudo apt-get update
    sudo apt-get install ca-certificates curl
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc

    # Add the repository to Apt sources:
    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
      $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
      sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update

    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin screen

    systemctl start docker
    systemctl enable docker
    usermod -a -G docker ubuntu

    echo "done" > /home/ubuntu/user_data_done

  EOF

  tags = {
    Name = "${local.app_name}-instance"
  }
}

resource "null_resource" "wait_for_user_data" {
  provisioner "remote-exec" {
    inline = [
      "echo 'Waiting for EC2 software install to finish...'",
      "while [ ! -f /home/ubuntu/user_data_done ]; do echo '...'; sleep 2; done",
      "echo 'EC2 software install finished.'"
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("./.keys/id_rsa")
      host        = aws_eip_association.eip_assoc.public_ip
    }
  }

  depends_on = [aws_instance.app_instance]
}

resource "null_resource" "setup_registry" {
  provisioner "local-exec" {
    command = <<-EOF
      echo "Generating secret for local registry"
      sha256sum ./.keys/id_rsa | cut -d ' ' -f1 | tr -d '\n' > ./.keys/registry.pure

      echo "Creating htpasswd file for local registry"
      docker run --rm --entrypoint htpasswd httpd:2 -Bbn ci-user $(cat ./.keys/registry.pure) > ./.keys/registry.htpasswd

      echo "Generating server certificate for registry"
      openssl genrsa -out ./.keys/registry.key 4096
      echo "subjectAltName=DNS:appserver.local,DNS:localhost,IP:127.0.0.1" > san.ext
      openssl req -new -key ./.keys/registry.key -subj "/CN=appserver.local" -addext "$(cat san.ext)" -out ./.keys/registry.csr

      openssl x509 -req -days 365 -CA ./.keys/ca.pem -CAkey ./.keys/ca.key -set_serial 01 -in ./.keys/registry.csr -extfile san.ext -out ./.keys/registry.crt 

      echo "Copying registry secret files to the instance"
      rsync -t -avz -e "ssh -i ./.keys/id_rsa -o StrictHostKeyChecking=no" \
        ./.keys/registry.* ubuntu@${aws_eip_association.eip_assoc.public_ip}:/home/ubuntu/registry-auth
    EOF
  }

  provisioner "remote-exec" {
    inline = [<<-EOF
      # remove old registry if exists
      docker rm -f registry
      # run new registry
      docker run -d --network host \
        --name registry \
        --restart always \
        -v /home/ubuntu/registry-data:/var/lib/registry \
        -v /home/ubuntu/registry-auth:/auth\
        -e "REGISTRY_AUTH=htpasswd" \
        -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
        -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.htpasswd" \
        -e "REGISTRY_HTTP_TLS_CERTIFICATE=/auth/registry.crt" \
        -e "REGISTRY_HTTP_TLS_KEY=/auth/registry.key" \
        registry:2

      EOF
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("./.keys/id_rsa")
      host        = aws_eip_association.eip_assoc.public_ip
    }
  }

  triggers = {
    always_run = 1 # change number to redeploy registry (if for some reason it was removed)
  }

  depends_on = [null_resource.wait_for_user_data]
}


resource "null_resource" "sync_files_and_run" {

  provisioner "local-exec" {
    command = <<-EOF

      # map appserver.local to the instance (in CI we don't know the IP, so have to use this mapping)
      # so then in GA pipeline we will use 
      #  - name: Set up Docker Buildx
      #   uses: docker/setup-buildx-action@v3
      #   with:
      #     buildkitd-config-inline: |
      #       [registry."appserver.local:5000"]
      #         ca=["deploy/.keys/ca.pem"]

      grep -q "appserver.local" /etc/hosts || echo "${aws_eip_association.eip_assoc.public_ip} appserver.local" | sudo tee -a /etc/hosts

      # hosts modification may take some time to apply
      sleep 5

      # generate buildx authorization
      sha256sum ./.keys/id_rsa | cut -d ' ' -f1 | tr -d '\n' > ./.keys/registry.pure
      echo '{"auths":{"appserver.local:5000":{"auth":"'$(echo -n "ci-user:$(cat ./.keys/registry.pure)" | base64 -w 0)'"}}}' > ~/.docker/config.json

      echo "Running build"
      docker buildx bake --progress=plain --push --allow=fs.read=.. -f compose.yml

      # compose temporarily it is not working https://github.com/docker/compose/issues/11072#issuecomment-1848974315
      # docker compose --progress=plain -p app -f ./compose.yml build --push

      # if you will change host, pleasee add -o StrictHostKeyChecking=no
      echo "Copy files to the instance" 
      rsync -t -avz --mkpath -e "ssh -i ./.keys/id_rsa -o StrictHostKeyChecking=no" \
        --delete \
        --exclude '.terraform' \
        --exclude '.keys' \
        --exclude 'tfplan' \
        . ubuntu@${aws_eip_association.eip_assoc.public_ip}:/home/ubuntu/app/deploy/

      EOF
  }

  # Run docker compose after files have been copied
  provisioner "remote-exec" {
    inline = [<<-EOF
      # login to docker registry
      cat /home/ubuntu/registry-auth/registry.pure | docker login localhost:5000 -u ci-user --password-stdin
        
      cd /home/ubuntu/app/deploy

      echo "Spinning up the app"
      docker compose --progress=plain -p app -f compose.yml up -d --remove-orphans

      # cleanup unused cache (run in background to not block terraform)
      screen -dm docker system prune -f
      screen -dm docker exec registry registry garbage-collect /etc/docker/registry/config.yml --delete-untagged=true 
    EOF
    ]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file("./.keys/id_rsa")
      host        = aws_eip_association.eip_assoc.public_ip
    }


  }

  # Ensure the resource is triggered every time based on timestamp or file hash
  triggers = {
    always_run = timestamp()
  }

  depends_on = [aws_eip_association.eip_assoc, null_resource.setup_registry]
}


output "instance_public_ip" {
  value = aws_eip_association.eip_assoc.public_ip
}


######### META, tf state ##############


# S3 bucket for storing Terraform state
resource "aws_s3_bucket" "terraform_state" {
  bucket = "${local.app_name}-terraform-state"
}

resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  rule {
    status = "Enabled"
    id = "Keep only the latest version of the state file"

    filter {
      prefix = ""
    }

    noncurrent_version_expiration {
      noncurrent_days = 30
    }
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "AES256"
    }
  }
}