Github integration with Jenkins behind firewall

Posted on | Edited on |

TL;DR

This post will explain how to trigger a job on Jenkins that will test changes on Github repository when Jenkins is behind a firewall. Github has service integration with Amazon SQS and Jenkins has a plugin that triggers a job on SQS event. To make it work you will need to do the following:

  1. Create Amazon SQS for your Github repository and configure it
  2. Add SQS to Github and configure it to notify on push and pull requests
  3. Install SQS plugin and configure it in Jenkins
  4. Write pipeline job that will handle the different events
  5. Post job results to Github branch or pull request

Amazon SQS

“Amazon Simple Queue Service (SQS) is a fully managed message queuing service”, in other words SQS it will be our message bus, where we send messages from Github and listen to them on Jenkins. Jenkins will have to connect to SQS (outbond connection) and not listen to inbound connections. To get started, you need an AWS account and then you can create a new SQS service like this:

  1. Go to AWS SQS console
  2. Click on Create New Queue, and then:
    • What do you want to name your queue? Repository name1
    • What type of queue do you need?2 Standard Queue
  3. Click on Quick-Create-Queue3
  4. Select the queue you’ve just created and at the bottom copy for later:
    • ARN
    • URL

Next we will create a new IAM user for Jenkins:

  1. Go to IAM console
  2. Click on Add user, and then:
    • User name: jenkins4
    • Access type: Programmatic access
  3. Click on Next:Permissions, and then:
    1. Select Attach existing policies directly
    2. In the Filter search for AmazonSQSFullAccess
  4. Click on Next:Preview and then Click on Create user
  5. Copy the Access Key ID and Secret access key5

That’s it, you have SQS configured.

Github

Github has a great support for webhooks and services, and there is even a one for Jenkins. But all of them assume Github can reach your Jenkins. Fortunately Github also has a service for Amazon SQS:

  1. Go to your repository services: https://github.com/<user>/<repo>/settings/installations
  2. Click on Add service, then find and select Amazon SQS
  3. Click on Amazon SQS service to configure it, and then:
    • Aws access key - IAM user Access Key ID
    • Aws sqs arn - SQS queue ARN
    • Aws secret key - IAM user Secret access key
  4. Update service

  5. Open your terminal:

    1. Find the service id: curl -u <user> https://api.github.com/repos/<user>/<repo>/hooks

    2. Modify on which events the service will trigger: curl -X PATCH --data '{ "events": ["push", "pull_request"] }' -u <user> 'https://api.github.com/repos/<user>/<root>/hooks/<id>'

Github will now send messages on push and pull request to your Amazon SQS queue. You can find what other events can be added and the messages content at Github documentation on web-hooks and events.

Jenkins

Jenkins has a vast collection of open source plugins and even two for Amazon SQS. You will need only one of them and a pipeline support:

  1. Go to Jenkins Plugin manager: http://<jenkins FQDN>/pluginManager/available
  2. Install AWS SQS Build Trigger Plugin6
  3. Go to Jenkins configuration: http://<jenkins FQDN>/configure

  4. In section Configuration of Amazon SQS queues click on Add, then:

    1. Credentials: Click on Add and select Jenkins, then:
      • Kind: Secret Text
      • Scope: Global
      • Secret: IAM user’s Secret access key
      • ID: IAM user’s Access Key ID
    2. Queue name: Queue URL
    3. Click on Test access, you should see: “Access to <queue> successful”

  5. Click on Save

Create a new Pipeline job7 with the following setting:

  • In section Build Triggers:
    • mark: ‘Trigger build when a message is published to an Amazon SQS queue’
    • SQS queue to monitor: <queue>
  • In section Pipeline, use Pipeline script and you can use this as template for your job:
    • Lines 11-36 function for commenting on Github:
      • Install python library PyGithub on the executing system
      • Create Access token in Github for your user (or a bot user)
      • Set global parameter in Jenkins GITHUB_ACCESS_TOKEN
    • Lines 38-61 function for cloning and checking out the right repository based on the SQS message from Github.
    • Lines 69-73 will handle parameters from SQS trigger, the most important is sqs_body
    • Lines 76-87 will prepare everything you need for the job to work

  1. Whatever you want↩

  2. I think it will work well for most cases↩

  3. We will configure it later↩

  4. Whatever you want↩

  5. You will need it for Jenkins and Github integration↩

  6. version 2.0.1↩

  7. should work with other too↩