DevOps Project 11

By /

End-to-End CI/CD Pipeline for Flask Application Based on GitOps

we are going to build an End-to-End Pipeline based on the GitOps approach. You can easily implement this Pipeline by following this Blog.

We are going to start from scratch to deploy our application into a K8S Cluster. First, we start by looking at our application. It is a Simple Flask application that just prints some message on a web page. But I recommend trying your own source code. You can also try out different programming languages. I just want to show you the approach. You can try your own source code.

I will also share my Application repository as well as my Manifest repository link in this Blog. You can also try it out for learning purposes by just forking my repositories.

Introduction:

Before diving into Our Pipeline, Let’s explore GitOps first,

Why GitOps?

Let’s say we have a Kubernetes cluster and Someone Modify some configurations in the cluster. How do we know what has changed? How do we know who made the change? How do we know when the change was made?. So the Idea is like If we can have Version Control System like Git to store Application Source Code, Why not use Git to store the configuration of the Kubernetes Cluster. So GitOps is a way to do Continuous Delivery for Applications and Infrastructure using Git.

What is GitOps?

GitOps is a way to do Continuous Delivery. It uses Git as a single source of truth for declarative infrastructure and applications. This means that by defining our Kubernetes resources in a GitHub Repository, we can utilize tools like Argo CD to orchestrate the deployment of these resources into our Kubernetes cluster. Argo CD ensures that the state of resources aligns with our declared state consistently.

Architecture Overview:

We are going to build a CI/CD pipeline that will be triggered by a push to the main branch of the repository. The pipeline will build the application, run the tests, and deploy the application to a Kubernetes cluster based on GitOps. The pipeline will be implemented using GitHub Actions and ArgoCD.

The Architecture of the project is explained below:

First, We have a GitHub repository that contains the source code of the application. The repository is connected to GitHub Actions, which will be used to build and deploy the application. The GitHub Actions workflow will be triggered by a push to the main branch of the repository. The workflow will build the application, run the tests, and Push the Docker image to Docker Hub. And then It will Update the Image tag in the Manifest Repository. The Manifest Repository is a separate GitHub repository that contains the Kubernetes manifests for the application. And our Argo CD will be watching this repository for changes. Once the image tag is updated in the Manifest Repository, Argo CD will automatically deploy the new version of the application to the Kubernetes cluster.

Architecture Diagram:

Prerequistes:

  • A GitHub account
  • A Docker Hub account
  • A Kubernetes cluster
  • Argo CD installed on the Kubernetes cluster

Steps:

Step 1: Launching an Ubuntu Instance:

In my case, I don’t have a Linux machine. So I am going to launch an Ubuntu instance on AWS.

If you also don’t have one, try to launch a Linux machine using cloud providers, VirtualBox, or any other method of your choice. However a Kubernetes cluster requires some system requirements, so try to create a VM with those specific requirements.

Don’t forget to open necessary Ports on your EC2 Instance Security Group.

System Requirements for Kubeadm:
– Linux operating system (Ubuntu, CentOS, etc.)
– Minimum 2 GB of RAM
– Minimum 2 CPU cores
– 40 GB of disk space

System Requirements for Minikube:
– Linux, macOS, or Windows operating system
– Virtualization enabled
– Minimum 2 CPU cores
– Minimum 2 GB of RAM
– 20 GB of disk space

Step 2: Setting up a K8S Cluster:

There are several ways to set up our K8S cluster. You can choose any method you prefer. I’m going to use the Minikube way for ease of implementation. However if you prefer Kubeadm or manual setup, that’s also okay, it’s not a problem.

I will provide commands for Ubuntu. If you are using any other Linux distribution, please refer to the official documentation for installations.

Let’s start by installing Docker first on Ubuntu:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable docker
sudo usermod -aG docker ubuntu

Once Docker is installed, we can proceed with setting up Minikube and Kubernetes.

Install minikube using Below commands:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
minikube start

Install kubectl using below commands:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Step 3: Install Argo CD using kubectl

There are many options for installing Argo CD, such as HA and Non-HA. However, for learning purposes, we are going to use the non-HA method below

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Now You check all the resources reated by Argo D using the command kubectl -n argocd get all ,

Expose the Argo CD Server Edit the argocd-server service to change the type to NodePort.

kubectl edit svc argocd-server -n argocd

Change the type to NodePort and save the file. Now, use kubectl -n argocd get svc to get the Service’s name. Find out the node Port of your argocd-server service. Then, access the web UI using http://node-ip/node-port

In case if you configured your cluster using Minikube, follow the approach below:

  • We have to do some port-forwarding to access the Argo CD UI.
  • In my case, I used Minikube to set up a Kubernetes Cluster. So, I am going to install socat to do port-forwarding
sudo apt-get install socat

Run the following command to do port-forwarding.

socat TCP4-LISTEN:8080,fork,reuseaddr TCP4:192.168.49.2:NodePort &

Now, we can access the Argo CD UI using http://NodeIP:8080. To get the password for the admin user, use the command below in the Cluster.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Now, we can log in to the Argo CD UI using the admin user and the password we got from the above command.

Now We can create Argo CD Applications using the New App Option.

Step 4: Creating Application Repository with Workflows

Now, we need to create a repository that will contain our application source code, and we also have to define our CI Pipeline. I am going to use GitHub Actions as my CI tool since it is easy for me to set up because I am using GitHub as my Source Code Repository. If you are comfortable with Jenkins or any other CI tools, you can implement the CI Pipeline on that tool. I just want everyone to be clear about my approach, that’s all.

We are going to have the following directory structure:

├── .github
│   └── workflows
│       ├── test.yaml
│       └── docker-image.yaml
├── Dockerfile
├── app.py
├── requirements.txt
├── test_app.py
├── README.md
├── LICENSE

In the above listed files, README.md and LIENSE files are Optional,

Let’s Start Creating our Application Configuration Files:

  • First We are going to start with app.py , Create a file named app.py and paste the below code:

from flask import Flask 
from urllib.parse import quote 

def create_app():
    app = Flask(__name__)

    @app.route('/')
    def home():
        return 'Well done! You Successfully deployed your flask app, Now you can start building your app.'

    return app
if __name__ == '__main__':
    app = create_app()
    app.run(host='0.0.0.0', port=80, debug=True)
  • Now Create a file named requirements.txt and paste the below code:
flask==2.0.1
pytest
Werkzeug==2.0.0
  • Then Create a test_app.py file and Use the below Code:

import pytest
from app import create_app
from urllib.parse import quote

@pytest.fixture
def app():
    return create_app()

@pytest.fixture
def client(app):
    return app.test_client()

def test_home(client):
    response = client.get('/')
    assert response.status_code == 200
    expected_text = 'Well done! You Successfully deployed your flask app, Now you can start building your app.'
    assert expected_text.encode() in response.data

Now We are going to Create Dockerfile for creating a Docker Image. This file will be Consumed by our workflows to build a Docker Image,


FROM python:3.9-slim

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

EXPOSE 80

ENTRYPOINT ["python" , "app.py"]

Then Let’s create our Workflows files, For that we need to create a .github/workflows/ folder,

  • Now create a file named test.yaml , It will test our Application code using Various Python Versions:
name: Upload Python Package

on:
   push:
     branches: ["main"]
   pull_request:
     branches: ["main"]

jobs:
  build-and-test-python-app:

    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        pytest
  • Then create a file named docker-image.yaml , It will Build our Docker image with tag as commit-id and Push it to our Docker Hub . And after that it will also Update our Manifest Repository with that new tag:
name: Publish Docker image and Update Manifest Repo

env:
  CONFIG_REPO_NAME: helm-python-flask

on:
  push:
    branches: [ "main" ]
jobs:

  image-push:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - name: Build the Docker image
      run: docker build --tag memathesh/pythonflaskapp:${{ github.sha }} .

    - name: Login to Docker Hub
      uses: docker/login-action@v3
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_PASSWORD }}

    -
        name: Build and push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: memathesh/pythonflaskapp:${{ github.sha }}

  Update-Config-Repo:
    runs-on: ubuntu-latest
    needs: image-push
    steps:
      - run: |
          echo "promoting into dev environment!"
          git config --global user.email actions@github.com && git config --global user.name GitHub Actions
          echo "cloning config repo $CONFIG_REPO_NAME"
          git clone https://oauth2:${{ secrets.CI_TOKEN }}@github.com/${{ github.repository_owner }}/$CONFIG_REPO_NAME.git
          cd $CONFIG_REPO_NAME
          echo "checkout main branch"
          git checkout main
          echo "Updating Image tag in values.yaml file"
          sed -i "s,tag:.*,tag:\ ${{ github.sha }}," pythonflaskapp/values.yaml
          git add . && git commit -m "Updated Image tag to ${{ github.sha }}"
          git push

If you look into the above File, You can see I defined three Secrets named DOCKERHUB_USERNAME, DOCKERHUB_PASSWORD and CI_TOKEN. So you need to Create GitHub Repository Secrets, For Creating Secrets follow the below steps:

  • Go to your GitHub repository.
  • Click on the Settingstab
  • Click on the Secrets link in the left sidebar.
  • Click on the New repository secretbutton.
  • Add the secret name and value and click on the Add secretbutton.

Remember to Keep Our Secret names as DOCKERHUB_USERNAME, DOCKERHUB_PASSWORD and CI_TOKEN.

You can also Fork my Repository in case you don’t want to create files by Yourself. But you still need to create Secrets by Yourself.

Repository Link: Click here

Step 5: Creating Manifest Repository

Now, we are going to create a Manifest Repository. This repository will contain our K8S resource files. We can store our Manifest files in any format like Helm, Kustomize, or as Plain Manifest. I am going to use the Helm format. You can also change it as per your preference.

Structure of our Manifest Repository:

├── pythonflaskapp
    ├──template
    │    ├── NOTES.txt
    │    ├── _helpers.
    │    ├── deployment.yaml
    │    ├── service.yaml
    │
    ├── Chart.yaml
    ├── values.yaml
    ├── LICENSE
    ├── README.md

In the above listed files, README.md and LICENSE files are optional. We need to create Chart.yaml, values.yaml, and a templates/ folder only.

  • Let’s create the Chart,yaml file. Use the following code:
apiVersion: v2
name: pythonflaskapp
version: 0.0.1
description: A Simple Helm Chart for Python Flask Application
type: application
keywords:
  - python
  - flask
# sources:
#   - https://github.com/mathesh-me/helm-python-flask-app
maintainers:
  - name: Mathesh M
    email: itzmathesh@gmail.com
    url: 
icon: https://helm.sh/img/helm.svg
appVersion: 0.0.1
  • Please use the below link to get the templates/ folder files, as it contains many files and including all of them here would make this blog too long. So try to create files as per our above directory structure and copy the code from the link below.

Repository Link: Click here

  • Then Create a file named values.yaml file, It is used for Providing values to variables we define in our K8S resource manifest Files,

deployment:
  replicas: 2
  image:
    owner: <dockerhub-username>
    repository: <image-name>
    tag: 9fc652bfc6d4cbc0a9d388b28e624e4012dab81c
  container:
    name: pythonflaskapp
    port: 80
service:
  type: NodePort
  port: 80
  targetPort: 80
  nodePort: 30112

Step-6: Creating Argo CD Application

Now we are going to Create our Argo CD Application, We have three ways to create an application in Argo CD:

  1. Web UI: We can create an application using the Argo CD Web UI.
  2. CLI: We can create an application using the Argo CD CLI.
  3. Git Repository: We can define the application in a Git repository and Argo CD will automatically create the application. It is a Declarative way of defining the application. Most Recommended way.

I will show you Declarative approach, Let’s create our Argo CD App using Kubernetes Manifest File,

  • Create a new file with named application.yaml and Paste the below code inside this file.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pythonflaskapp
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: pythonflaskapp
    repoURL: https://github.com/mathesh-me/helm-python-flask
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  • Now just apply kubectl apply -f application.yaml Command, Now It will create our Argo CD application and our application will automatically sync our Cluster with resources we declared in our Manifest Repo.
  • Now try to push some Changes to Application Repository. It will trigger the workflow and update the Image tag automatically in our Manifest Repository. Since our Argo CD Application watching, Argo CD will automatically deploy our application into K8S Cluster
  • You can Check our application using the kubectl -n argocd get app Command. And you can see our application is synced automatically,
  • Now go to Argo CD Web UI And Check our Application
  • If you look into the above Picture you can see the Commit Updated by our Actions User to Update Image tag to 37e7d... that means our commit id is 37e7d and our Workflow updated our Image tag to Commit ID. You can see the same commit ID below,
  • You can also view your Image tag in your Docker Hub Account
  • Now Let’s view our K8S resources deployed by our Application:

Step-7: Accessing our Flask Application

Now we can Access our application, Using the http://node-ip/node-port URL. If you are using minikube cluster Don’t forget to do Port Forwarding using the below command:

socat TCP4-LISTEN:8081,fork,reuseaddr TCP4:192.168.49.2:30112 &

Step-8: Let’s Play with our Application

Now try to make some changes in Application configuration repository and Look at the workflow by using Actions tab in Application Repository,

Now if you look into Manifest Repository, You can see the commit done by GitHub Actions user to Update our Image tag,

Now We Successfully Built a End-to-End CI/CD Pipeline based on GitOps Approach.

Application Repository Link: Click here

Manifest Repository Link: Click here

👏 If you find this helpful, don’t forget to give claps and follow my profile. I’ll be sharing more projects and ideas about Cloud and DevOps. If you have any doubts, feel free to comment or message me on LinkedIn.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top