Merge pull request #29 from devdattakulkarni/cicd

CICD example with Jenkins and Bitbucket

Author: devdattakulkarni

CICD/build-steps.txt

@@ -0,0 +1,7 @@
+echo "This is a build step"
+
+cd Containers/Kubernetes-examples/GCP/greetings
+docker build -t gcr.io/cloudark-test-gke/greetings .
+gcloud docker -- push gcr.io/cloudark-test-gke/greetings
+
+echo "SUCCESS"

CICD/cicd-steps.txt

@@ -0,0 +1,201 @@
+CI/CD workflow using Jenkins and Bitbucket
+-------------------------------------------
+
+Setup:
+------
+Conceptually you need two different machines for this exercise.
+
+Machine 1: This machine/VM will be your developer workspace where you are developing your application.
+Machine 2: This VM will be your Jenkins server (CI/CD server)
+
+Machine 1 can be your personal Laptop or a cloud VM.
+
+Machine 2 *has* to be a cloud VM (such as a EC2 instance).
+Feel free to choose either a EC2 instance or Google Compute Engine instance
+Make sure you allow TCP traffic to port 8080 on your VM instance.
+
+Note: It is fine to use a single VM as Machine 1 and Machine 2. It is up to you.
+
+
+A] Setup your developer workspace/Machine 1:
+--------------------------------------------
+1) Install Google Cloud CLI
+   - sudo apt-get install python
+   - curl https://sdk.cloud.google.com | bash
+
+2) Configure authentication for gcloud CLI
+   - gcloud auth login --> Follow the prompts (you will have to open browser window and paste the generated link,
+       then paste the generated code in the verification field in your console.)
+   - Create Project in Google Cloud Console --> Note down the Project ID. Remember that Project ID is different than the Project's Name.
+       You will need Project ID in subsequent steps.
+   - export PROJECT_ID=<Project-ID-from-previous-step>
+   - export CLOUDSDK_COMPUTE_ZONE=us-central1-b
+   - gcloud config set project ${PROJECT_ID}
+
+3) Create a Google Cloud SQL instance
+   - gcloud sql instances create instance1 --tier=db-f1-micro --authorized-networks=0.0.0.0/0
+
+4) Create testdb database on the Cloud SQL instance
+   - gcloud sql users set-password root % --instance instance1 --password 'testpass123!@#'
+   - sudo apt-get install mysql-client
+   - gcloud sql instances list
+   - mysql -h <IP-address-of-SQL-Instance> --user=root --password='testpass123!@#'
+   - mysql>create database testdb;
+
+5) Create a static IP address for our application
+   - gcloud compute addresses create greetings-ip --global
+
+6) Create your Bitbucket repository named "assignment6"
+   - Make it Public
+   - Generate a SSH Key pair and add it to your Bitbucket account
+     - Account -> Settings -> Add Key
+      - You can either generate an SSH key or add an existing key if you have one
+
+7) Download Sample CI/CD project
+   - git clone https://github.com/devdattakulkarni/CloudComputing
+   - cd CloudComputing/CICD/greetings
+   - Edit greetings-deployment.yaml and set the environment variables corresponding to
+     the Google Cloud SQL instance
+     (Note: In the image tag don't include any version.
+      i.e.: Define your image tag as: "gcr.io/<project-id>/greetings" and NOT as
+      "gcr.io/<project-id>/greetings:v1")
+
+   - Add a remote for your Bitbucket repository and Push the sample CI/CD project to that remote
+     - cd ../../
+     - git remote add origin1 git@bitbucket.org:<your-bitbucket-username>/assignment6.git
+     - git push origin1 master
+
+
+
+B] Setup your Jenkins Server/Machine 2:
+--------------------------------------
+1) Jenkins Server Installation preparation:
+   - Login to your Cloud VM instance (Machine2)
+   - Install Java (JRE and JDK)
+     - sudo apt-add-repository ppa:openjdk-r/ppa
+     - sudo apt-get install -y openjdk-8-jre openjdk-8-jdk wget
+   - Install Google Cloud CLI
+     - sudo apt-get install python
+     - curl https://sdk.cloud.google.com | bash
+   - Install Kubectl
+     - gcloud components install kubectl
+   - Install Docker
+     - git clone https://github.com/devdattakulkarni/CloudComputing.git
+     - cd Containers/Kubernetes-examples/GCP/
+     - ./install-docker-ubuntu.sh 
+     - exit from Cloud VM instance
+     - SSH back into Cloud VM instance
+
+2) Download and Start Jenkins Server:
+   - wget http://mirrors.jenkins.io/war-stable/latest/jenkins.war
+   - nohup java -jar jenkins.war --httpPort=8080 &
+
+3) Configure Jenkins Server
+   - Point your browser to the DNS name of the Jenkins Server at port 8080
+     https://<DNS-of-Jenkins-Server>:8080/
+   - Keep the user as 'admin'
+   - Install recommended plugins
+
+4) Once Jenkins starts up do following:
+   4.1) Login as admin user
+        - Password is stored in: $HOME/.jenkins/secrets/initialAdminPassword
+   4.2) Set the path in Jenkins for Git
+        Jenkins -> Manage Jenkins -> Configure -> Global Properties
+        - Check Tool Locations
+        - List of tool locations (Choose Git)
+        - Set the path of Git from your VM
+          (You can find the path by running 'which git' on VM)
+   4.3) Install the Post build task Plugin
+        Jenkins -> Manage Jenkins -> Manage Plugins -> Post build task plugin
+   4.4) Install Bitbucket Plugin
+        Jenkins -> Manage Jenkins -> Manage Plugins -> Bitbucket
+
+5) Add a WebHook to your Bitbucket repository
+   5.1) Go to your Bitbucket repository -> Settings -> URL. E
+        - http://<DNS-of-Jenkins-server>:8080/bitbucket-hook/
+          (Note: The trailing slash '/' is important. Don't forget that!!)
+        - Make Status as "Active"
+        - Check "Skip certificate verification"
+        - Triggers -> Choose from a full list of triggers
+          - Select following: 
+            - Repository->Push, Pull Request->Created, Updated, Comment created, Comment updated, Comment deleted
+              Issue->Created, Issue->Updated, Issue->Comment Created
+         
+6) Set up a Jenkins Job
+   6.1) Make sure your Bitbucket repository ("assignment6") is Public
+        - This will make it possible to be cloned by Jenkins.
+   6.2) Jenkins -> New Item -> Give name -> Select Freestyle project
+        -> Source Code Management -> Give Bitbucket Repository URL (in git)
+   6.3) Build Triggers
+        -> Choose "Build when a change is pushed to BitBucket"
+   6.4) Buid Section -> Choose Execute Shell script
+        -> In the Execute shell Command section:
+           - Add contents of build-steps.txt 
+             - Notes:
+               - First modify build-steps.txt to use your GCP Project ID
+               - Make sure that the image tag that use here is what you defined 
+                 in deployment.yaml (Part A, step 7)
+   6.5) In Post-build Actions
+        -> Tasks -> Log text "SUCCESS" -> Operation "-- OR --"
+           -> In the Tasks -> Script section:
+	      - Add contents of deploy-steps.txt (modify the steps with your assignment's name)
+
+
+
+CI/CD Experiment:
+-----------------
+1) Open the Jenkins URL in your browser and navigate to the job
+
+2) On Machine 1 (developer workspace) make changes to your code and push the code:
+   - Modify index.html
+   - git add
+   - git commit
+   - git push origin1 master
+
+3) In the Jenkins Job you should see a new build triggered (visible under "Build History")
+   - Select the Build
+   - Go to Console Output
+   - Verify that the build and deploy steps have been executed
+
+4) On Machine 2 (Jenkins server):
+   - kubectl get ingress
+   - Navigate to the IP address and verify that your changes
+     (Note: It may take 6/7 minutes before the ingress filtering routes will take effect
+      and your changes are visible.)
+
+5) Repeat steps 2-3-4
+
+
+
+Clean up:
+---------
+From Google Cloud Console:
+- Delete Kubernetes Cluster
+- Delete Google Cloud SQL Instance
+- Delete the Static IP address
+
+
+
+
+
+
+Troubleshooting:
+----------------
+1) Restarting Jenkins:
+   - From EC2 instance find out Jenkins process
+     - ps -eaf | grep java
+     - kill <process-id>
+   - Re-run Jenkins using the java command
+
+2) Jenkins URL location
+
+3) 
+
+
+
+
+Reference:
+-----------
+- https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
+- https://blog.terrenceryan.com/index.php/making-kubernetes-ip-addresses-static-on-google-container-engine/
+- http://ec2-34-212-126-125.us-west-2.compute.amazonaws.com:8080/bitbucket-hook/

CICD/cicd-steps.txt~

@@ -0,0 +1,163 @@
+Steps to set up CI/CD workflow using Jenkins with your Bitbucket repository.
+----------------------------------------------------------------------------
+
+Setup:
+------
+Conceptually you need two different machines for this exercise.
+
+Machine 1: This machine/VM will be your developer workspace where you are developing your application.
+Machine 2: This VM will be your Jenkins server
+
+Machine 1 can be your personal Laptop or a cloud VM.
+
+Machine 2 *has* to be a cloud VM (such as a EC2 instance).
+You will need to use a publicly available VM instance for Machine2
+Feel free to choose either a EC2 instance or / Google Compute Engine instance
+Make sure you allow TCP traffic to port 8080 on your VM instance.
+
+
+Setup of your developer workspace/Machine 1:
+---------------------------------------------
+1) Install Google Cloud CLI
+   - sudo apt-get install python
+   - curl https://sdk.cloud.google.com | bash
+
+2) Configure authentication for gcloud CLI
+   - gcloud auth login --> Follow the prompts (you will have to open browser window and paste the generated link,
+       then paste the generated code in the verification field in your console.)
+   - Create Project in Google Cloud Console --> Note down the Project ID. Remember that Project ID is different than the Project's Name.
+       You will need Project ID in subsequent steps.
+   - export PROJECT_ID=<Project-ID-from-previous-step>
+   - export CLOUDSDK_COMPUTE_ZONE=us-central1-b
+   - gcloud config set project ${PROJECT_ID}
+
+3) Create a Google Cloud SQL instance
+   - gcloud sql instances create instance1 --tier=db-f1-micro --authorized-networks=0.0.0.0/0
+
+4) Create testdb database on the Cloud SQL instance
+   - gcloud sql users set-password root % --instance instance1 --password 'testpass123!@#'
+   - sudo apt-get install mysql-client
+   - gcloud sql instances list
+   - mysql -h <IP-address-of-SQL-Instance> --user=root --password='testpass123!@#'
+   - mysql>create database testdb;
+
+5) Create your Bitbucket repository named "assignment6"
+   - Make it Public
+
+6) Download Sample CI/CD project
+   - git clone https://github.com/devdattakulkarni/CloudComputing
+   - cd CICD/greetings
+   - Edit greetings-deployment.yaml and set the environment variables corresponding to
+     the Google Cloud SQL instance
+     (Note: You don't need to change the image URI)
+   - Add a remote for your Bitbucket repository and Push the sample CI/CD project to that remote
+     - cd ..
+     - git remote add origin1 git@bitbucket.org:<your-bitbucket-username>/assignment6.git
+     - git push origin1 master
+
+
+
+Setup of Jenkins Server/Machine 2:
+-----------------------------------
+1) Jenkins Server Installation preparation:
+   - Login to your Cloud VM instance (Machine2)
+   - Install Java (JRE and JDK)
+     - sudo apt-add-repository ppa:openjdk-r/ppa
+     - sudo apt-get install -y openjdk-8-jre openjdk-8-jdk wget
+   - Install Google Cloud CLI
+     - sudo apt-get install python
+     - curl https://sdk.cloud.google.com | bash
+   - Install Kubectl
+     - gcloud components install kubectl
+   - Install Docker
+     - git clone https://github.com/devdattakulkarni/CloudComputing.git
+     - cd Containers/Kubernetes-examples/GCP/
+     - ./install-docker-ubuntu.sh 
+     - exit from Cloud VM instance
+     - SSH back into Cloud VM EC2 instance
+
+2) Download and Start Jenkins Server:
+   - wget http://mirrors.jenkins.io/war-stable/latest/jenkins.war
+   - nohup java -jar jenkins.war --httpPort=8080 &
+
+3) Configure Jenkins Server
+   - Point your browser to the DNS name of the Jenkins Server at port 8080
+     https://<DNS-of-Jenkins-Server>:8080/
+   - Keep the user as 'admin'
+   - Install recommended plugins
+
+4) Once Jenkins starts up do following:
+   4.1) Login as admin user
+        - Password is stored in: $HOME/.jenkins/secrets/initialAdminPassword
+   4.2) Set the path in Jenkins for Git
+        Jenkins -> Manage Jenkins -> Configure -> Global Properties
+        - Check Tool Locations
+        - List of tool locations (Choose Git)
+        - Set the path of Git from your VM
+          (You can find the path by running 'which git' on VM)
+   4.3) Install the Post build task Plugin
+        Jenkins -> Manage Jenkins -> Manage Plugins -> Post build task plugin
+   4.4) Install Bitbucket Plugin
+        Jenkins -> Manage Jenkins -> Manage Plugins -> Bitbucket
+
+5) Add a WebHook to your Bitbucket repository
+   5.1) Go to your Bitbucket repository -> Settings -> URL
+         - http://ec2-34-212-126-125.us-west-2.compute.amazonaws.com:8080/bitbucket-hook/
+        - Check Status to Active
+        - Check "Skip certificate verification"
+        - Triggers -> Choose from a full list of triggers
+          - Select following: 
+            - Repository->Push, Pull Request->Created, Updated, Comment created, Comment updated, Comment deleted
+              Issue->Created, Issue->Updated, Issue->Comment Created
+   5.2) Generate a SSH Key pair and add it to your Bitbucket account
+        - Account -> Settings -> Add Key
+          - You can either generate an SSH key or add an existing key if you have one
+         
+6) Set up a Jenkins Job
+   6.1) Make sure your Bitbucket repository ("assignment6") is Public
+        - This will make it possible to be cloned by Jenkins.
+   6.2) Jenkins -> New Item -> Give name -> Select Freestyle project
+        -> Source Code Management -> Give Bitbucket Repository URL (in git)
+   6.3) Build Triggers
+        -> Choose "Build when a change is pushed to BitBucket"
+   6.4) Buid Section -> Choose Execute Shell script
+        -> In the Execute shell Command section:
+           - Add contents of build-steps.txt (first modify build-steps.txt to use your GCP Project ID)
+   6.5) In Post-build Actions
+        -> Tasks -> Log text "SUCCESS" -> Operation "-- OR --"
+           -> In the Tasks -> Script section:
+	      - Add contents of deploy-steps.txt (modify the steps with your assignment's name)
+
+
+
+CI/CD Experiment:
+-----------------
+1) Make changes to your code and push the code:
+   - Modify index.html
+   - git add
+   - git commit
+   - git push origin1 master
+
+2) Open 
+
+
+
+
+
+
+
+Help Commands:
+---------------
+1) Restarting Jenkins:
+   - From EC2 instance find out Jenkins process
+     - ps -eaf | grep java
+     - kill <process-id>
+   - Re-run Jenkins using the java command
+
+
+Reference:
+-----------
+- https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
+- https://blog.terrenceryan.com/index.php/making-kubernetes-ip-addresses-static-on-google-container-engine/
+
+

CICD/deploy-steps.txt

@@ -0,0 +1,12 @@
+echo "This is the deploy step"
+cd Containers/Kubernetes-examples/GCP/greetings
+kubectl delete deployment greetings-deployment || echo "greetings-deployment deployment does not exist"
+kubectl delete service greetings-deployment || echo "greetings-deployment service does not exist"
+kubectl delete ingress greetings-ingress || echo "greetings-ingress does not exist"
+
+kubectl create -f greetings-deployment.yaml
+kubectl expose deployment greetings-deployment --target-port=5000 --type=NodePort
+
+kubectl apply -f greetings-ingress.yaml
+
+echo "Done deploying"

CICD/deploy-steps.txt~

@@ -0,0 +1,12 @@
+echo "This is the deploy step"
+cd Containers/Kubernetes-examples/GCP/greetings
+kubectl delete deployment greetings-deployment || echo "greetings-deployment deployment does not exist"
+kubectl delete service greetings-deployment || echo "greetings-deployment service does not exist"
+kubectl delete ingress greetings-ingress-1 || echo "greetings-ingress does not exist"
+
+kubectl create -f greetings-deployment.yaml
+kubectl expose deployment greetings-deployment --target-port=5000 --type=NodePort
+
+kubectl apply -f greetings-ingress.yaml
+
+echo "Done deploying"