diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index 7d6ddf6..134ef60 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -12,11 +12,11 @@ This `audit` command operates by fetching all of the pipelines defined in a Jenk ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials) to configure credentials for Valet to use. +2. Completed the [configure lab](./1-configure.md#configuring-credentials). ## Perform an audit -We will be performing an audit against a preconfigured Jenkins instance. We will need to answer the following questions before running this command: +We will be performing an audit against a preconfigured Jenkins server. We will need to answer the following questions before running this command: 1. Do we want to audit the entire Jenkins instance or just a single folder? - In this example we will be auditing the entire Jenkins instance, but in the future if you wanted to configure a specific folder to be audited add the `-f ` flag to the audit command. @@ -28,7 +28,7 @@ We will be performing an audit against a preconfigured Jenkins instance. We will 1. Navigate to the codespace terminal. 2. Run the following command from the root directory: - + ```bash gh valet audit jenkins --output-dir tmp/audit ``` diff --git a/jenkins/3-dry-run.md b/jenkins/3-dry-run.md index 548b141..08d9e94 100644 --- a/jenkins/3-dry-run.md +++ b/jenkins/3-dry-run.md @@ -1,53 +1,57 @@ -# Dry-run the migration of a Jenkins pipeline to GitHub Actions +# Perform a dry-run of a Jenkins pipeline In this lab, you will use the Valet `dry-run` command to convert a Jenkins pipeline to its equivalent GitHub Actions workflow. The end result of this command will be the actions workflow written to your local filesystem. - [Prerequisites](#prerequisites) - [Perform a dry-run](#perform-a-dry-run) -- [Review dry-run output](#review-dry-run-output) -- [Next Lab](#next-lab) +- [Inspect the output files](#inspect-the-output-files) +- [Next lab](#next-lab) ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. -3. Completed the [audit lab](../Jenkins/valet-audit-lab.md#audit-jenkins-pipelines-using-the-valet-audit-command). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [audit lab](./1-audit.md). ## Perform a dry-run -We will be performing a dry-run against a preconfigured pipeline in the Jenkins instance. Before running the command we need to collect some information: +We will be performing a dry-run against a pipeline in the preconfigured Jenkins server. We will need to answer the following questions before running this command: - 1. What is the name of the pipeline we want to convert? __test_pipeline__ - 2. What is the source URL of the pipeline we want to convert? ____ - 3. Where do we want to store the result? __./tmp/dry-run-lab. This can be any valid path on the system. In the case of codespaces it is generally best to use `./tmp/SOME_DIRECTORY_HERE` so the files show in explorer__ +1. What is the name of the pipeline we want to convert? + - __test_pipeline__ + +2. What is the URL of the pipeline we want to convert? + - ____ + +3. Where do we want to store the result? + - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. ### Steps 1. Navigate to the codespace terminal -2. Run the dry-run command using the values determined above +2. Run the following command from the root directory: - ``` - gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run - ``` + ```bash + gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run + ``` -3. When the command finishes the output files should be printed to the terminal. - Screen Shot 2022-08-16 at 9 54 26 AM -4. Open generated actions workflow - - Find `./tmp/dry-run-lab/valet` in the file explorer pane in codespaces. - - Click `test_pipeline.yml` to open +3. The command will list all the files written to disk when the command succeeds. - Screen Shot 2022-08-16 at 9 55 44 AM + ![img](https://user-images.githubusercontent.com/19557880/184935603-5c2d4dfe-66ef-4cb1-9398-e96954ca72e3.png) -## Review dry-run output +4. View the converted workflow: + - Find `./tmp/dry-run` in the file explorer pane in codespaces. + - Click `test_pipeline.yml` to open -The dry-run output will show you the GitHub Actions yaml that would be migrated to GitHub with the `migrate` command. We will now take a quick look at what was generated. +## Inspect the output files + +The files generated from the `dry-run` command represent the equivalent Actions workflow for the given Jenkins pipeline. The Jenkins pipeline and converted workflow can be seen below: -__Click to Expand__
- Jenkins Pipeline + Jenkins Pipeline 👇 -```yaml +```groovy pipeline { agent { label 'TeamARunner' @@ -78,7 +82,7 @@ pipeline {
- Actions Workflow + Converted workflow 👇 ```yaml name: test_pipeline @@ -125,35 +129,7 @@ jobs:
-In the Jenkins pipeline we have 2 stages and 4 steps that run on the self-hosted runner labeled `TeamARunner`. - -In the Actions workflow we have the same steps and the stages are now being enforced using the `needs` keyword. We can see this if we examine the `test` job, it has `needs: build`, which makes it depend on the `build` job. - -```diff -- stages: test -+ needs: build -``` - -The `agent` in the Jenkins pipeline has been transformed to `runs-on` on each of the jobs. - -```diff -- agent { -- label 'TeamARunner' -- } -+ runs-on: -+ - self-hosted -+ - TeamARunner -``` - -And the `echo` commands remain mostly the same - -```diff -- echo "Database engine is ${DB_ENGINE}" -+ - name: echo message -+ run: echo "DISABLE_AUTH is ${{ env.DISABLE_AUTH }}" -``` - -Note how Valet was not able to find a suitable conversion for the `sleep` command, and it added a comment to the yaml so this information was not lost, and it could be addressed manually later if needed. +These 2 pipelines function equivantly despite using different syntax. In this case, the pipeline conversion was “partially successful” (i.e. there were item(s) not automatically converted) and the unconverted item was placed as comment in the location the Jenkins pipeline used it. For example: ```diff - sleep 80 @@ -165,19 +141,10 @@ Note how Valet was not able to find a suitable conversion for the `sleep` comman + # value: 80 ``` -Lastly, the `junit` command was transformed using a third party action `EnricoMi/publish-unit-test-result-action@v1.7` +In the next lab, we'll learn how to override Valet's default behavior and customize the converted workflow that is generate. -```diff -- junit '**/target/*.xml' -+ - name: Publish test results -+ uses: EnricoMi/publish-unit-test-result-action@v1.7 -+ if: always() -+ with: -+ files: "**/target/*.xml" -``` +Try running the `dry-run` command for different pipelines in the Jenkins server. As a hint, you just have to change the `--source-url` CLI option. -Try constructing and running the `dry-run` command yourself. Hint, you should just have to change the project name. +## Next lab -## Next Lab - -[Using Custom Transformers in a dry-run](4-custom-transformers.md#using-custom-transformers-in-a-dry-run) +[Use custom transformers to customize Valet's behavior](4-custom-transformers.md) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 802e428..1cf4658 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -1,13 +1,13 @@ -# Using Custom Transformers in a `dry-run` +# Use custom transformers to customize Valet's behavior -In this lab we want to do a `dry-run` of the `test_pipeline` pipeline, however, after a closer inspection, we discovered that we will need to customize how Valet transforms: +In this lab we will build upon the `dry-run` command to override Valet's default behavior and customize the converted workflow using "custom transformers". Custom transformers can be used to: -1. Steps that are unsupported by Valet, but are essential to our devops process. -2. Steps in which the GitHub action chosen by Valet does not meet our current needs. -3. All pipelines that have the environment variable `DB_ENGINE` must be changed to a different value: `mongodb`. -4. The self-hosted runners before used the `TeamARunner` label in Jenkins but will run on the `ubuntu-latest` runner in Actions. +1. Convert items that are not automatically converted. +2. Convert items that were automatically converted using different actions. +3. Convert environment variable values differently. +4. Convert references to runners to use a different runner name in Actions. -These customizations will be present in many of our company's pipelines and an automated way to apply these changes would be ideal. In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language. +In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language. - [Prerequisites](#prerequisites) - [Perform a dry-run](#perform-a-dry-run) @@ -19,23 +19,22 @@ These customizations will be present in many of our company's pipelines and an a ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. -3. Completed the [dry-run lab](../jenkins/valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [dry-run lab](./1-dry-run.md). ## Perform a dry-run -- Let’s run the `dry-run` command to see what information we can get from the generated action yaml. +We will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal: - ```bash - gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run - ``` + ```bash + gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o tmp/jenkins/dry-run + ``` -- Open the resulting GitHub Actions workflow by navigating to `tmp/valet/test_pipeline.yml` from the explorer +The converted workflow that is generated by the above command can be seen below -__Click to Expand__
- Actions Workflow + Converted workflow 👇 ```yaml name: test_pipeline @@ -98,47 +97,55 @@ jobs:
-Go back to the previous lab [dry-run Jenkins pipeline](../jenkins/valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions) if you need a review of the details of the dry-run. + + +_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step -We can see in the center of the transformed workflow that the `sleep` step was not transformed. In order for us to write a custom transformer for this we need to know the identifier. In general, the identifier will be the key of a key/value pair within the step of a Jenkinsfile, which in this case is `sleep`. This is how our custom transformer will target the correct step. +The converted workflow above contains a `sleep` step was not automatically converted. We will need to answer the following questions before writing a custom transformer: -After some research we have decided that a simple bash script will meet our needs: +1. What is the "identifier" of the step to customize? + - __sleep__. The identifier will be the key of a key value pair within the step of a Jenkinsfile. - ```yaml - - name: Sleep for 80 seconds - run: sleep 80s - shell: bash - ``` +2. What is the desired Actions syntax to use instead? + - After some research, we have determined that the following bash script will provide similar functionality: -Now that we know the final yaml needed for the transformer, we can start to write the ruby file.The custom transformers file can have any name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting. + ```yaml + - name: Sleep for 80 seconds + run: sleep 80s + shell: bash + ``` -- Create a new file where you will put the custom transformers logic. It can have any file name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting. For this example we will call it `transformers.rb`. - -In the custom transformers file we will add a `transform` method. This is a special method that Valet exposes, that takes the identifier we determined earlier and returns a ruby hash of the final YAML for the pipeline. - -The ruby hash can be thought of as the JSON representation of the YAML we want. Valet will call that method when it encounters the identifier and pass in an `item`. The `item` is the key of that key/value pair defined for that step in Jenkins. In this case the item is the settings of the sleep command. - - ```ruby - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - ``` - -Let’s run the `dry-run` command again with `--custom-transformer transformers.rb` to see if our custom transformer worked. +Now we can begin to write the custom transformer. Customer transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb +code transformers.rb ``` -When you open the file you should see the following changes in the GitHub Actions Workflow, note how the sleep timer step is no longer commented out! +Next, we will define a `transform` method for the `sleep` identifier by adding the following code to `transformers.rb`: + +```ruby +transform "sleep" do |item| + wait_time = item["arguments"][0]["value"]["value"] + + { + "name": "Sleep for #{wait_time} seconds", + "run": "sleep #{wait_time}s", + "shell": "bash" + } +end +``` + +This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Jenkins. + +Now, we can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: + +```bash +gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o tmp/jenkins/dry-run --custom-transformers transformers.rb +``` + +Open the workflow that is generated and inspect the contents. Now, the `sleep` step is converted and uses the customized behavior! ```diff - # # This item has no matching transformer @@ -154,22 +161,24 @@ When you open the file you should see the following changes in the GitHub Action ## Custom transformers for a known step -Next lets address the third-party GitHub action `EnricoMi/publish-unit-test-result-action@v1.7` that Valet chose for the `junit` step. This action does not meet our needs because our CTO has dictated that our pipelines take no dependencies on third-party actions. We can customize the action that we would like Valet to use instead. +We can also override Valet's default behavior. In this scenario, we may not desire to use the third-party action for publishing junit test results that is used by default. Again, we will need to answer the following questions before writing a custom transformer: -After some research we find that the following action will upload an artifact with the junit test results without depending on a third party action. +1. What is the "identifier" of the step to customize? + - __junit__ -```yaml -- uses: actions/upload-artifact@v3 - with: - name: junit-artifact - path: path/to/artifact/world.txt -``` +2. What is the desired Actions syntax to use instead? + - After some research, we have determined that the uploading test results as an artifact will be suitable: -We will build this custom transformer similar to how we built the previous custom transformer. This time, we need to add some additional logic. We would like to programmatically assign the file path to junit's test results to our upload action. + ```yaml + - uses: actions/upload-artifact@v3 + with: + name: junit-artifact + path: path/to/artifact/world.txt + ``` -The challenge is, that we are unsure what `item` represents (what the incoming JSON object looks like from Jenkins). In order to troubleshoot this, you could use some basic ruby to print `item` to the terminal. You can achieve this by adding the following line in the transform method: `puts "This is the item: #{item}"`. +We will build this custom transformer similar to the previous custom transformer, however, we first need to inspect the `item` keyword to programmatically use the file path to junit's test results in the `actions/upload-artifact@v3` step. -You are able to add multiple custom transformers to the same file, so you can add this one right below the previous custom transformer. +To do this, we will print `item` to the console. You can achieve this by adding the following custom transformer to `transformers.rb`: ```ruby transform "junit" do |item| @@ -177,77 +186,30 @@ transform "junit" do |item| end ``` -Let’s run the `dry-run` command to see what information is outputted in the terminal. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. The output of the `dry-run` command should look similar to this: -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` +![img](https://user-images.githubusercontent.com/19557880/186782050-65ece0c4-52a3-4f88-818f-0f860c50c2b7.png) -You should see something like the following outputted into the terminal: - -Screen Shot 2022-08-17 at 10 25 53 AM - -Now that we know the structure of the incoming JSON blob for `item` we can access the file path programmatically. +Now that we know the data structure of `item` we can access the file path programmatically by editing the custom transformer to following: ```ruby transform "junit" do |item| test_results = item["arguments"].find{ |a| a["key"] == "testResults" } file_path = test_results.dig("value", "value") - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "junit-artifact", - "path": file_path - } - } - ] + { + "uses" => "actions/upload-artifact@v3", + "with" => { + "name" => "junit-artifact", + "path" => file_path + } + } end ``` -Your file should include both custom transformers. Ensure it matches the file below: +_Note_: `transformers.rb` should contain a `transform` method for both `sleep` and `junit`. -__Click to Expand__ -
- Custom Transformer file - -```ruby - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - - transform "junit" do |item| - test_results = item["arguments"].find{ |a| a["key"] == "testResults" } - file_path = test_results.dig("value", "value") - - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "junit-artifact", - "path": file_path - } - } - ] - end -``` - -
- -Let’s run the `dry-run` command again to see if our custom transformer worked. - -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` - -When you open the file you should see the`EnricoMi/publish-unit-test-result-action@v1.7` action has been replaced with the customized steps. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `EnricoMi/publish-unit-test-result-action@v1.7` action will have been replaced with the customized steps. ```diff - - name: Publish test results @@ -263,58 +225,17 @@ When you open the file you should see the`EnricoMi/publish-unit-test-result-acti ## Custom transformers for environment variables -But wait! There's more we need to customize to make these pipelines fit our needs. Several of our pipelines will need to have the value of the `DB_ENGINE` environment variable changed from `sqlite` to `mongodb`. +We can also use custom transformers to edit the values of environment variables in converted workflows. In our example, we will be updating the `DB_ENGINE` environment variable to be `mongodb` instead of `sqlite`. -Valet allows you to replace values of `variables` by using the `env` method. Let’s replace the value for `DB_ENGINE` by using the below line. The first value of the `env` method is the target variable name and the second is the new value to be used. - - ```ruby - env "DB_ENGINE", "mongodb" - ``` - -You can keep adding these transformer values to your `transformer.rb` file. Your transformer file should look like the one below: - -__Click to Expand__ -
- Custom Transformer file +To do this, add the following code to the `transformers.rb` file. ```ruby - env "DB_ENGINE", "mongodb" - - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - - transform "junit" do |item| - test_results = item["arguments"].find{ |a| a["key"] == "testResults" } - file_path = test_results.dig("value", "value") - - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "my-artifact", - "path": file_path - } - } - ] - end +env "DB_ENGINE", "mongodb" ``` -
+In this example, the first parameter to the `env` method is the environment variable name and the second is the updated value. -Let’s run the `dry-run` command again to see if our custom transformer worked. - -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` - -When you open the outputed file you should see that the value for `DB_ENGINE` has been changed from `sqlite` to `mongodb`. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `DB_ENGINE` environment variable will be set to `mongodb`: ```diff env: @@ -325,21 +246,35 @@ env: ## Custom transformers for runners -Lastly, several of our pipelines will need to have the runner label updated from `TeamARunner` to `ubuntu-latest`. +Finally, we can use custom transformers to dictate which runners converted workflows should use. To do this we will need to answer the following questions: -Valet offers the ability to provide customized mapping between build agents in the source CI instance and their equivalent GitHub Actions runner labels by using the `runner` method. The syntax for this is similar to the `env` method. +1. What is label of the runner in Jenkins to update? + - __TeamARunner__ -The `runner` methods first parameter is the Jenkins agent label and the second parameter is the GitHub Actions runner label(s) to map too. As an example, the following could be used: +2. What is the label of the runner in Actions to use instead? + - __ubuntu-latest__ - ```ruby - runner "TeamARunner", "ubuntu-latest" - ``` +With these questions answered, we can add the following code to the `transformers.rb` file: -You can keep adding these transformer methods to your `transformer.rb` file. Your transformer file should look like the one below: +```ruby +runner "TeamARunner", "ubuntu-latest" +``` + +In this example, the first parameter to the `runner` method is the Jenkins label and the second is the Actions runner label. + +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` statement will use the customized runner label: + +```diff +runs-on: +- - self-hosted +- - TeamARunner ++ - ubuntu-latest +``` + +At this point of the lab the file contents of `transformers.rb` should match this: -__Click to Expand__
- Custom Transformer file + Custom transformers 👇 ```ruby runner "TeamARunner", "ubuntu-latest" @@ -374,27 +309,13 @@ __Click to Expand__
-Let’s run the `dry-run` command again to see if our custom transformer worked. +Thats it! Congratulations you have overridden Valet's default behavior by customizing the conversion of: -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` +- Unknown steps +- Known steps +- Environment variables +- Runners -When you open the file you should see that the value for `TeamARunner` has been changed to `ubuntu-latest`. +## Next lab -```diff -runs-on: -- - self-hosted -- - TeamARunner -+ - ubuntu-latest -``` - -Thats it! Congratulations you have customized transforming: - -- unsupported steps -- steps that are supported but don't meet your requirements -- environment variables -- runners - -## Next Lab [Forecast Jenkins Usage](../jenkins/5-forecast.md#forecast-the-runner-usage-of-a-jenkins-instance)