Job DSL Part I

Jenkins CI is a great tool for automating your build and deployment pipeline. You set up jobs for build, test, deployment and whatever, and let Jenkins do the work. But there’s a catch. In the recent blog post Bringing in the herd I already talked a bit about the difficulties you have to tackle if you are dealing with microservices: they are like rabbits! When you start with a project, there may be only a couple of microservices, but soon there will be a few dozens or even hundreds.Setting up jobs for these herds is a growing pain you have to the master, and that’s where the Job DSL comes to the rescue. This post is the start of a small series on the Job DSL.

One lesson we already learned about microservices is that you have to automate everything. Even – or especially – the configuration of the tooling used to build, deploy, and monitor your application. Not to mention the things you have to do to run the application like distributing, load balancing etc. But let’s start with the build pipeline for the moment. Setting up a Jenkins job is an easy task. When you create a job using the UI, you just have to select the things Jenkins is supposed to do, like check out source code from GIT, run the maven or gradle build, and publish the test results. Once the job does what you want, it is easy to set up this job for another project: Jenkins allows to make copies. Just adapt some data like names and paths, and that’s it. So there’s no challenge in creating jobs for new microservices. If you have to create multiple jobs for each microservice – let’s say integration and acceptance tests, release builds, deployment to various environments – things start to get annoying. But one day you recognize that you – just for example - forgot to publish the checkstyle results, and you will have to change all your existing jobs… manually :-0

Don’t do it. Not even once! What does developers do in order to avoid repetitive, boring, annoying, error-prone tasks? They write a script, yep.We are lazy bones, so instead of doing stuff, we’re telling the machine what to do, and have cup of coffee while the work is being done. Jenkins job definitions are nothing but a little XML, so we could easily write a little script - Groovy has great built-in support for processing XML - and generate that. We could even invent a DSL using Groovy, so our script will be more readable.And since all that is so obvious, somebody already had this idea: The Jenkins Job DSL Plugin.

The development of this plugin was driven by one of the protagonists of microservices: Netflix. They currently have about 600 microservices, so they really need to automate everything. And that’s why they invented the Job DSL: it allows you to describe your Jenkins job using a predefined DSL. It is implemented as a Jenkins plugin, so the creation of the Jenkins job is performed as a Jenkins job itself: Let’s start with a basic example. At first we create a seed job. That’s the common lingual for the job that generates other jobs using the Job DSL:



We will need only a single build step: the Job DSL:



Now copy the following sample DSL to the editor field:

freeStyleJob('order-build') { 
  scm { 
    git { 
      remote { 
        url('https://github.com/ralfstuckert/jobdsl-sample.git') 
      } 
      branch('order')
      createTag(false) 
    } 
  } 
  triggers { 
     scm('H/15 * * * *') 
  } 

  steps { 
    maven { 
      mavenInstallation('3.1.1') 
      goals('clean install') 
    } 
  } 
}

Let’s go through it step by step. We define a freestyle build job named order-build. Next is a source control block with a GIT repository. I don’t wanted to set up a dozen repositories for the projects used in this example, so I used different branches. So to check out the order project, select the branch named order. We don’t want Jenkins to create a tag (with the build-number) after the checkout, so we set this property to false. In the trigger block we watch the source control system for changes every 15 minutes. In the following (build-) steps block, we define just one step: maven. A maven installation is selected (as predefined in Jenkins) and the goals clean and install are executed. Save and run. Now we have generated a new job order-build:



Looks good, so let run the order-build: Yep, it builds :-)



Ok, so we generated a build job. But we could have done the same thing using the Jenkins UI, so where is the big deal? The benefit of generating jobs pays off when generate the same class of job for multiple projects. Let’s say we have a some projects named customer, order, datastore etc. Now we will extend our DSL with a little Groovy code that iterates over these projects, and create a build job for each. So (re-)configure your seed build, and replace the DSL part with the following stuff:

def microservices = '''
microservices {
  ad {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'ad'
  }
  billing {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'billing'
  }
  cart {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'cart'
  }
  config {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'config'
  }
  controlling {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'controlling'
  }
  customer {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'customer'
  }
  datastore {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'datastore'
  }
  help {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'help'
  }
  logon {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'logon'
  }
  order {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'order'
  }
  preview {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'preview'
  }
  security {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'security'
  }
  shipping {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shipping'
  }
  shop {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'shop'
  }
  statistics {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'statistics'
  }
  warrenty {
    url = 'https://github.com/ralfstuckert/jobdsl-sample.git'
    branch = 'warrenty'
  }
}
'''

def slurper = new ConfigSlurper()
// fix classloader problem using ConfigSlurper in job dsl
slurper.classLoader = this.class.classLoader
def config = slurper.parse(microservices)

// create job for every microservice
config.microservices.each { name, data ->
  createBuildJob(name,data)
}


def createBuildJob(name,data) {
  
  freeStyleJob("${name}-build") {
  
    scm {
      git {
        remote {
          url(data.url)
        }
        branch(data.branch)
        createTag(false)
      }
    }
  
    triggers {
       scm('H/15 * * * *')
    }

    steps {
      maven {
        mavenInstallation('3.1.1')
        goals('clean install')
      }
    }

  }

}

Ok, let’s go through this again step by step. At firs, we define al little DSL describing our microservices. After that, we use the Groovy ConfigSlurper to parse the DSL (ignore this class loader stuff for the moment, that’s a bug). Than we iterate over the microservices and pass the name and the data of each service to the method createBuildJob(). This method contains the Job DSL we used in the first example. Well, almost. We parameterized some things like the name, Git URL and branch, so we can reuse the DSL for creating all the build jobs.

Let the seed job run again and watch the output:



Looks good. Now let’s see the dashboard:




Ta-da. It’s done. We generated a dozen build jobs using a single script. That’s it for the first installment. In the next part, we will alter our DSL, automate the Job creation itself, and create some views.

When you got a job to do, 
you gotta do it well.
Paul McCartney – Live and let die