Building a Complete CI/CD Pipeline for Deploying Python Microservices with GitHub, Jenkins, Docker, and Kubernetes
Streamline Your Python Microservices Deployment: A Hands-On Guide to Building an Automated CI/CD Pipeline with GitHub, Jenkins, Docker, and Kubernetes
Introduction
Continuous Integration and Continuous Deployment (CI/CD) pipelines are essential for modern software development, enabling teams to automate their development and deployment processes. This article walks you through an end-to-end example of deploying a Python-based microservice using GitHub, Jenkins, Docker, and Kubernetes/OpenShift. We will detail each step, from writing the code to exposing the deployed service, ensuring a seamless development-to-production cycle.
Step 1: Writing the Python Microservice
We’ll create a simple Flask-based microservice as our application.
Code for the Microservice
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/api', methods=['GET'])
def home():
return jsonify({'message': 'Hello, Kubernetes!'})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
Save the file as app.py
.
Requirements File
Create a requirements.txt
to specify dependencies:
flask
Git Repository
Push this code to a Git repository on GitHub or Bitbucket:
git init
git add .
git commit -m "Initial commit for Flask microservice"
git remote add origin https://github.com/your-repo-url.git
git push -u origin main
Step 2: Dockerizing the Microservice
To containerize the Python application, we will create a Dockerfile
.
Purpose of a Dockerfile
A Dockerfile
is a text document that contains instructions to build a Docker image. It automates the process of creating containerized environments with all dependencies and configurations needed for the application.
Dockerfile
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]
Code Explanation:
FROM
: Specifies the base image to use. In this case, a lightweight version of Python 3.9.Why? It provides the runtime for Python applications with minimal overhead.
WORKDIR
: Sets the working directory inside the container to/app
.Why? All subsequent commands in the Dockerfile will execute relative to this directory.
COPY
: Copies therequirements.txt
file from the local machine into the container.RUN
: Executes the command to install dependencies listed inrequirements.txt
.Why? This ensures that all necessary Python libraries (like Flask) are installed in the container.
COPY . .
: Copies all files and directories from the local working directory to the container's/app
directory.Why? This includes your Python application files.
CMD
: Specifies the command to run the application when the container starts.Why? It launches the Flask application.
Building the Docker Image
A Docker Image is a static snapshot of your application, dependencies, and configurations. It is created by running the docker build
command with the Dockerfile
.
Purpose of a Docker Image
A Docker image is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, runtime, libraries, and system settings. It serves as a blueprint for deploying containers.
docker build -t flask-microservice:latest .
docker run -p 5000:5000 flask-microservice
Key Steps in the Image Creation
The
python:3.9-slim
base image is pulled from Docker Hub.Dependencies from
requirements.txt
are installed.Application files are copied into the container.
The
CMD
command is set to run the Python application.
Result
The resulting Docker Image is a portable, versioned package that you can deploy to any environment supporting Docker, ensuring consistency across development, testing, and production.
Step 3: Setting Up Jenkins for CI/CD
Jenkins automates the building, testing, and deployment process.
Install Jenkins and Required Plugins
Install Jenkins.
Install plugins:
Git Plugin
Pipeline Plugin
Docker Pipeline Plugin
Kubernetes Plugin
Connect Jenkins to GitHub
Add a webhook in the GitHub repository settings:
URL:
http://<your-jenkins-url>/github-webhook/
Events: Push events.
Purpose of a Jenkins Pipeline Script
The Jenkins Pipeline Script defines the CI/CD process as code. It automates stages like fetching the repository, building the application, pushing the Docker image, and deploying to Kubernetes.
Jenkins Pipeline Script
Create a Jenkinsfile
in your repository to define the pipeline:
pipeline {
agent any
environment {
DOCKER_IMAGE = "your-dockerhub-username/flask-microservice:latest"
}
stages {
stage('Clone Repository') {
steps {
git branch: 'main', url: 'https://github.com/your-repo-url.git'
}
}
stage('Build Docker Image') {
steps {
script {
docker.build(env.DOCKER_IMAGE)
}
}
}
stage('Push Docker Image') {
steps {
withDockerRegistry([credentialsId: 'dockerhub-credentials', url: 'https://index.docker.io/v1/']) {
script {
docker.image(env.DOCKER_IMAGE).push()
}
}
}
}
stage('Deploy to Kubernetes') {
steps {
script {
sh """
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
"""
}
}
}
}
post {
always {
echo 'Pipeline execution completed!'
}
}
}
Code Explanation:
pipeline
: Defines the structure of the CI/CD pipeline.agent any
: Runs the pipeline on any available Jenkins node.
environment
: Declares environment variables. Here,DOCKER_IMAGE
stores the name and tag of the Docker image.The pipeline is broken into logical stages.
git
: Fetches the code from the specified branch and repository.docker.build()
: Builds a Docker image using theDockerfile
in the repository.Why? Converts the source code into a deployable Docker image.
withDockerRegistry
: Logs into Docker Hub using saved credentials (dockerhub-credentials
).docker.image().push()
: Pushes the Docker image to Docker Hub.Why? Makes the image available for deployment.
sh
: Runs shell commands to deploy the application using Kubernetes manifests.kubectl apply
: Applies the deployment and service YAML files to create pods and expose the service.Why? Deploys the microservice to Kubernetes/OpenShift.
post
: Defines actions to perform after the pipeline completes.always
: Ensures the message is logged regardless of the pipeline outcome.
Key Outputs
Dockerfile: Prepares the environment for creating a Docker Image.
Docker Image: A deployable package containing your Python microservice.
Jenkins Pipeline: Automates CI/CD tasks, ensuring consistent and error-free deployment.
Step 4: Configuring Kubernetes for Deployment
Purpose of Deployment Configuration
The Deployment Configuration specifies how the application is deployed, including the number of replicas (pods), container image details, and runtime configurations.
Deployment YAML
Create a k8s/deployment.yaml
file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: flask-microservice
spec:
replicas: 2
selector:
matchLabels:
app: flask-microservice
template:
metadata:
labels:
app: flask-microservice
spec:
containers:
- name: flask-microservice
image: your-dockerhub-username/flask-microservice:latest
ports:
- containerPort: 5000
Code Explanation
apiVersion: apps/v1
: Specifies the Kubernetes API version for deployments.Why? Ensures compatibility with the Kubernetes cluster.
kind: Deployment
: Declares this resource as a deployment.Why? Indicates that this YAML file manages the desired state of application instances.
metadata
: Contains metadata about the deployment.name
: Assigns a unique name (flask-microservice
) to the deployment.
spec
: Specifies the desired state of the deployment.replicas: 2
: Defines the number of replicas (pods) to run.Why? Ensures high availability by running two instances of the application.
selector
: Identifies which pods are managed by this deployment using labels.Why? Ensures that the deployment targets the correct pods.
template
: Describes the pod specification for the deployment.metadata.labels
: Labels the pods withapp: flask-microservice
.Why? Labels help identify and manage pods.
spec.containers
: Defines the container(s) inside each pod.name
: Assigns a name to the container (flask-microservice
).image
: Specifies the Docker image to use (your-dockerhub-username/flask-microservice:latest
).Why? Deploys the application using the pre-built Docker image.
ports
: Exposes container port5000
for communication.
Purpose of Service Configuration
The Service Configuration defines how the application is exposed to the outside world, specifying load balancing, port mapping, and external IP configurations.
Service YAML
Create a k8s/service.yaml
file:
apiVersion: v1
kind: Service
metadata:
name: flask-service
spec:
selector:
app: flask-microservice
ports:
- protocol: TCP
port: 80
targetPort: 5000
type: LoadBalancer
Code Explanation
apiVersion: v1
: Specifies the Kubernetes API version for services.Why? Ensures compatibility with the cluster's service APIs.
kind: Service
: Declares this resource as a service.Why? Indicates that this YAML file exposes the pods as a network service.
metadata
: Contains metadata about the service.name
: Assigns a unique name (flask-service
) to the service.
spec
: Specifies the desired state of the service.selector
: Matches pods with the labelapp: flask-microservice
.Why? Ensures that traffic is directed to the correct pods.
ports
: Defines the port configuration for the service.protocol: TCP
: Specifies the protocol for communication.port: 80
: Exposes the service on port80
(external access point).targetPort: 5000
: Routes traffic to port5000
inside the container.Why? Maps external traffic to the application’s internal port.
type: LoadBalancer
: Specifies the service type.Why? Creates an external IP to access the service, allowing it to be publicly accessible.
How They Work Together
Deployment YAML:
Defines how many instances of the application should run.
Ensures that each instance uses the specified Docker image and listens on port
5000
.
Service YAML:
Exposes the pods created by the deployment to the external world.
Maps traffic from port
80
(external) to port5000
(internal) and balances traffic among the pods.
Lifecycle Example
Deployment Creation:
kubectl apply -f k8s/deployment.yaml
Kubernetes creates two pods running the application using the specified Docker image.
Service Creation:
kubectl apply -f k8s/service.yaml
Kubernetes sets up a LoadBalancer service to expose the pods at an external IP.
Accessing the Application:
Use the external IP of the service to access the application:
http://<external-ip>/api
By combining the Deployment YAML and Service YAML, Kubernetes ensures that your application is both deployed and accessible, enabling scalability, high availability, and seamless user access.
Step 5: Executing the CI/CD Pipeline
Trigger the Pipeline with Webhooks
A webhook is a mechanism that triggers the Jenkins pipeline automatically whenever changes are pushed to the GitHub repository.
Purpose of Webhook
Webhooks enable automation by notifying Jenkins about code changes in the repository. This eliminates manual intervention to start the pipeline.
Pipeline Execution:
Clone the repository.
Build and push the Docker image.
Deploy the application to Kubernetes.
Step 6: Verify the Deployment
Check Pods:
kubectl get pods
Check Services:
kubectl get svc
Access the Application:
Note the external IP of the
flask-service
.Open in a browser or use
curl
:
curl http://<external-ip>/api
Expected Response:
{"message": "Hello, Kubernetes!"}
Conclusion
This step-by-step guide demonstrates how to set up a CI/CD pipeline for deploying Python-based microservices using GitHub, Jenkins, Docker, and Kubernetes/OpenShift. By automating the build and deployment process, you ensure scalability, consistency, and faster delivery cycles, which are critical for modern DevOps workflows.
Stay Connected: Subscribe to my ebasiq for Girish Substack for regular updates, technical insights, and step-by-step guides.
Next Steps
Now that you’ve learned how to build an end-to-end CI/CD pipeline for deploying Python microservices, it's time to take the next steps in your DevOps journey:
Try it Yourself: Implement this pipeline for your own microservices and experience the benefits of automation firsthand.
Collaborate with Me: If you have questions or need assistance, connect with me on LinkedIn or check out my GitHub repositories.
Deepen Your Knowledge: Explore related topics like advanced Kubernetes deployment strategies, integrating security in CI/CD pipelines, or scaling microservices effectively.
Share Your Experience: Let’s discuss your thoughts, challenges, or improvements in the comments below.
Start implementing these concepts today to elevate your DevOps expertise and streamline your deployment process! 🚀
For more in-depth technical insights and articles, feel free to explore:
Girish Central
LinkTree: GirishHub – A single hub for all my content, resources, and online presence.
LinkedIn: Girish LinkedIn – Connect with me for professional insights, updates, and networking.
Ebasiq
Substack: ebasiq by Girish – In-depth articles on AI, Python, and technology trends.
Technical Blog: Ebasiq Blog – Dive into technical guides and coding tutorials.
GitHub Code Repository: Girish GitHub Repos – Access practical Python, AI/ML, Full Stack and coding examples.
YouTube Channel: Ebasiq YouTube Channel – Watch tutorials and tech videos to enhance your skills.
Instagram: Ebasiq Instagram – Follow for quick tips, updates, and engaging tech content.
GirishBlogBox
Substack: Girish BlogBlox – Thought-provoking articles and personal reflections.
Personal Blog: Girish - BlogBox – A mix of personal stories, experiences, and insights.
Ganitham Guru
Substack: Ganitham Guru – Explore the beauty of Vedic mathematics, Ancient Mathematics, Modern Mathematics and beyond.
Mathematics Blog: Ganitham Guru – Simplified mathematics concepts and tips for learners.