From 85f6a51fbbfe4dd264184c423c686b519f66be59 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Wed, 31 Aug 2022 13:14:02 -0700 Subject: [PATCH 01/20] Rename lab files --- jenkins/{valet-configure-lab.md => 1-configure.md} | 0 jenkins/{valet-audit-lab.md => 2-audit.md} | 0 jenkins/{valet-dry-run-lab.md => 3-dry-run.md} | 0 ...{valet-custom-transformers-lab.md => 4-custom-transformers.md} | 0 jenkins/{valet-forecast-lab.md => 6-forecast.md} | 0 jenkins/{valet-migrate-lab.md => 7-migrate.md} | 0 6 files changed, 0 insertions(+), 0 deletions(-) rename jenkins/{valet-configure-lab.md => 1-configure.md} (100%) rename jenkins/{valet-audit-lab.md => 2-audit.md} (100%) rename jenkins/{valet-dry-run-lab.md => 3-dry-run.md} (100%) rename jenkins/{valet-custom-transformers-lab.md => 4-custom-transformers.md} (100%) rename jenkins/{valet-forecast-lab.md => 6-forecast.md} (100%) rename jenkins/{valet-migrate-lab.md => 7-migrate.md} (100%) diff --git a/jenkins/valet-configure-lab.md b/jenkins/1-configure.md similarity index 100% rename from jenkins/valet-configure-lab.md rename to jenkins/1-configure.md diff --git a/jenkins/valet-audit-lab.md b/jenkins/2-audit.md similarity index 100% rename from jenkins/valet-audit-lab.md rename to jenkins/2-audit.md diff --git a/jenkins/valet-dry-run-lab.md b/jenkins/3-dry-run.md similarity index 100% rename from jenkins/valet-dry-run-lab.md rename to jenkins/3-dry-run.md diff --git a/jenkins/valet-custom-transformers-lab.md b/jenkins/4-custom-transformers.md similarity index 100% rename from jenkins/valet-custom-transformers-lab.md rename to jenkins/4-custom-transformers.md diff --git a/jenkins/valet-forecast-lab.md b/jenkins/6-forecast.md similarity index 100% rename from jenkins/valet-forecast-lab.md rename to jenkins/6-forecast.md diff --git a/jenkins/valet-migrate-lab.md b/jenkins/7-migrate.md similarity index 100% rename from jenkins/valet-migrate-lab.md rename to jenkins/7-migrate.md From f7573c22ad82a6dc705ed42ae9623c222ef09d09 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Wed, 31 Aug 2022 21:00:10 -0700 Subject: [PATCH 02/20] Copy edit jenkins readme --- jenkins/readme.md | 128 +++++++++++++++++++++++++--------------------- 1 file changed, 70 insertions(+), 58 deletions(-) diff --git a/jenkins/readme.md b/jenkins/readme.md index 6ac095f..d2c05cb 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -1,88 +1,100 @@ -# Valet labs for Jenkins +# Migrating from Jenkins to Actions with Valet -This lab bootstraps a Valet environment using GitHub Codespaces and enables you to spin up a Jenkins instance against which to run the Valet CI/CD migration tool. +The instructions below will guide you through configuring a GitHub Codespace environment that will be used in subsequent labs that demonstrate how to use Valet to migrate Jenkins pipelines to GitHub Actions. -- [Use this repo as a template](#repo-template) -- [Use Valet with a codespace](#use-valet-with-a-codespace) -- [Bootstrap Jenkins](#bootstrap-jenkins) -- [Labs for Jenkins](#labs-for-jenkins) -- [Troubleshoot GH Valet extension](#troubleshoot-gh-valet-extension) -- [Troubleshooting Jenkins](#troubleshooting-jenkins) +These steps **must** be completed prior to starting other labs. -## Repo template +## Create your own repository for these labs -1. Verify you are in your own Repository created from the landing page [Valet Labs](https://github.com/valet-customers/labs). +1. Ensure that you have created a repository using the [valet-customers/labs](https://github.com/valet-customers/labs) as a template. -## Use Valet with a codespace +## Configure your Codespace -1. Start the codespace - - Click the `Code` with button down arrow above repository on the repository's landing page. - - Click the `Codespaces` tab - - Click `Create codespaces on main` to create the codespace. If you are in another branch then the `main` branch, the codespace will button will have the current branch specified. - - Wait a couple minutes, then verify that the codespace starts up. Once it is fully booted up, the terminal should be present. -2. Verify Valet CLI is installed and working. More information on the [GitHub Valet CLI extension](https://github.com/github/gh-valet) - - Verify Valet CLI is installed and working - - Run the following command in the Visual Studio Code terminal: +1. Start a new Codespace. - ``` - gh valet version - ``` +- Click the `Code` with button down arrow above repository on the repository's landing page. +- Click the `Codespaces` tab +- Click `Create codespaces on main` to create the codespace. +- After the Codespace has initialized there will be a terminal present. - - Verify the output looks like the below image. Note the valet version will be different than below as the latest version gets pulled down. - - If `gh valet version` did not produce a similar image with a version please follow these instructions [Troubleshoot GH Valet extension](#troubleshoot-gh-valet-extension) +2. Verify the Valet CLI is installed and working. More information on the Valet extension for the official GitHub CLI can be found [here](https://github.com/github/gh-valet). -Screen Shot 2022-08-10 at 1 45 20 PM +- Run the following command in the codespace's terminal: -## Bootstrap Jenkins + ```bash + gh valet version + ``` - 1. Run the Jenkins setup script. This script will setup Jenkins and ensure it is ready to use. In general, this script should be run first if you are starting a new codespace or restarting an existing one. +- Verify the output is similar to the image below. The version information may differ than what is shown below. + - If `gh valet version` did not produce similar output then please follow the troubleshooting [guide](#troubleshoot-the-valet-cli). -- Navigate to the terminal within your Codespace. -- Run the following command to kick off the creation of your Jenkins instance: + ![img](https://user-images.githubusercontent.com/19557880/186771327-631e8839-3614-4ab7-8108-818b5a0c6e93.png) - ``` +## Bootstrap a Jenkins server + + 1. Execute the Jenkins setup script that will start a container with a Jenkins server running inside of it. This script should be executed when starting a new Codespace or restarting an existing one. + +- Run the following command from the codespace's terminal to start a Jenkins server: + + ```bash ./jenkins/bootstrap/setup.sh ``` -- After a couple seconds, a pop-up box should appear with a link to the forwarded URL for your Jenkins instance. -- You can also access the forwarded URL by going to the `Ports` tab in your terminal. Right click on the URL listed under the `Local Address` and clicking the `Open in Browser` tab. -- Once you have navigated to the url, the following default credentials have been assigned: +- After some time, a pop-up box should appear with a link to the URL for your Jenkins server. + - You can also access the URL by going to the `Ports` tab in your terminal. Right click on the URL listed under the `Local Address` and click the `Open in Browser` tab. - - username: `admin` - - password: `password` +2. Open the Jenkins server in your browser and use the following credentials to authenticate: -2. Click the `Sign in` button and you should now see your new Jenkins instance with a few pre-populated pipelines. + - Username: `admin` + - Password: `password` + +- Once authenticated, you should see a Jenkins server with a few predefined pipelines. ## Labs for Jenkins Perform the following labs to test-drive Valet -- [Configure Valet to work with Jenkins](valet-configure-lab.md#configure-valet-to-work-with-jenkins) -- [Audit Jenkins using the Valet audit command](valet-audit-lab.md#audit-jenkins-pipelines-using-the-valet-audit-command) -- [Dry-run the migration of an Jenkins pipeline to GitHub Actions](valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions) -- [Using Custom Transformers in a dry-run](valet-custom-transformers-lab.md#using-custom-transformers-in-a-dry-run) -- [Migrate an Jenkins Project to GitHub Actions](valet-migrate-lab.md#migrate-a-jenkins-project-to-github-actions) -- [Forecast the usage of a Jenkins namespace](valet-forecast-lab.md#forecast-the-runner-usage-of-a-jenkins-instance) +- [Configure credentials for Valet](1-configure.md) +- [Perform an audit of a Jenkins server](2-audit.md) +- [Perform a dry-run of a Jenkins pipeline](3-dry-run.md) +- [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) +- [Perform a production migration of a Jenkins pipeline](5-migrate.md) +- [Forecast potential build runner usage](6-forecast.md) -## Troubleshoot GH Valet extension +## Troubleshoot the Valet CLI -Manually Install the GitHub CLI Valet extension. More information on the [GitHub Valet CLI extension](https://github.com/github/gh-valet) +The CLI extension for Valet can be manually installed by following these steps: - Verify you are in the codespace terminal -- Run this command to install the GitHub Valet extension -- `gh extension install github/gh-valet` -- Verify the result of the install is: `✓ Installed extension github/gh-valet` -- If you get a similar error to the following, click the link to authorize the token - - Restart Codespace after clicking the link -- Verify Valet CLI is installed and working by running `gh valet version` +- Run this command from within the codespace's terminal: - ![linktolcickauth](https://user-images.githubusercontent.com/26442605/169588015-9414404f-82b6-4d0f-89d4-5f0e6941b029.png) - -## Troubleshooting Jenkins + ```bash + gh extension install github/gh-valet + ``` -1. Navigate to the Docker tab on your left hand side. -2. Under the `Containers` tab you should see a Docker container `jenkins:valet` listed with a green play button ▶ - - If you see the `jenkins:valet` container, but it has a red stopped symbol next to it ▢, right click on the container and click on `start`, the container should begin running again. - - If the container does not start even after trying to manually start it, right click on the `jenkins:valet` container and click the `remove` button. Next continue by following all the #bootstrap-jenkins instructions again. +- Verify the result of the install contains: - Screen Shot 2022-08-09 at 3 06 46 PM + ```bash + ✓ Installed extension github/gh-valet + ``` + +- If you get an error similar to the image below, then click the link in the terminal output to authorize the token. + - Restart the codespace after clicking the link. + ![img](https://user-images.githubusercontent.com/26442605/169588015-9414404f-82b6-4d0f-89d4-5f0e6941b029.png) +- Verify Valet CLI extension is installed and working by running the following command from the codespace's terminal: + + ```bash + gh valet version + ``` + +## Troubleshooting the Jenkins server + +Follow these steps if the Jenkins server does not start correctly after running the setup script: + +1. Navigate to the `Docker` tab on the left side of the codespace. +2. Under the `Containers` tab you should see a docker container named `jenkins:valet` listed with a green play button ▶ + +- If you see the `jenkins:valet` container, but it has a red stopped symbol next to it ▢, right click on the container and click on `start`, the container should begin running again. +- If the container does not start even after trying to manually start it, right click on the `jenkins:valet` container and click the `remove` button. Then, attempt to start the Jenkins server again by following the steps [here](#bootstrap-a-jenkins-server). + +![img](https://user-images.githubusercontent.com/19557880/183770210-c0386616-656e-4fe9-9324-b410ad62c406.png) From 28edcee29c3d3f5dc7b751d81c0138c887c5fe77 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Wed, 31 Aug 2022 21:02:08 -0700 Subject: [PATCH 03/20] Update jenkins/readme.md --- jenkins/readme.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/readme.md b/jenkins/readme.md index d2c05cb..2777a5e 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -1,4 +1,4 @@ -# Migrating from Jenkins to Actions with Valet +# Jenkins to Actions migrations powered by Valet The instructions below will guide you through configuring a GitHub Codespace environment that will be used in subsequent labs that demonstrate how to use Valet to migrate Jenkins pipelines to GitHub Actions. From efd7837a7bdee5c9ef441f2d2065b2fd4384ed32 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 09:05:51 -0400 Subject: [PATCH 04/20] Update 1-configure.md --- jenkins/1-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 1f7018e..20f45f8 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -75,4 +75,4 @@ To verify Valet works we are going to run a `update` and `dry-run` command. We ### Next Lab -[Audit Jenkins using the Valet audit command](valet-audit-lab.md#audit-jenkins-pipelines-using-the-valet-audit-command) +[Audit Jenkins using the Valet audit command](2-audit.md#audit-jenkins-pipelines-using-the-valet-audit-command) From e65e7b5ffcaa243db558af9e4ac58812fa8e5148 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 09:06:28 -0400 Subject: [PATCH 05/20] Update 2-audit.md --- jenkins/2-audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index d1b2f51..d0f1d8f 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -121,4 +121,4 @@ In addition, you’ll see a file that shows the raw JSON data that we pull from ### Next Lab -[Dry-run the migration of a Jenkins pipeline to GitHub Actions](valet-dry-run-lab.md) +[Dry-run the migration of a Jenkins pipeline to GitHub Actions](3-dry-run.md) From c138ad69f9070c7be1f660b3b031dd82b33a9526 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 09:07:26 -0400 Subject: [PATCH 06/20] Update 3-dry-run.md --- jenkins/3-dry-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/3-dry-run.md b/jenkins/3-dry-run.md index 7229d6f..548b141 100644 --- a/jenkins/3-dry-run.md +++ b/jenkins/3-dry-run.md @@ -180,4 +180,4 @@ Try constructing and running the `dry-run` command yourself. Hint, you should ju ## Next Lab -[Using Custom Transformers in a dry-run](valet-custom-transformers-lab.md#using-custom-transformers-in-a-dry-run) +[Using Custom Transformers in a dry-run](4-custom-transformers.md#using-custom-transformers-in-a-dry-run) From 950727899c65f9cfe415c722f4c7e3dd171c7f18 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Thu, 1 Sep 2022 07:40:00 -0700 Subject: [PATCH 07/20] Rename file and make ordered list --- jenkins/{6-forecast.md => 5-forecast.md} | 0 jenkins/{7-migrate.md => 6-migrate.md} | 0 jenkins/readme.md | 12 ++++++------ 3 files changed, 6 insertions(+), 6 deletions(-) rename jenkins/{6-forecast.md => 5-forecast.md} (100%) rename jenkins/{7-migrate.md => 6-migrate.md} (100%) diff --git a/jenkins/6-forecast.md b/jenkins/5-forecast.md similarity index 100% rename from jenkins/6-forecast.md rename to jenkins/5-forecast.md diff --git a/jenkins/7-migrate.md b/jenkins/6-migrate.md similarity index 100% rename from jenkins/7-migrate.md rename to jenkins/6-migrate.md diff --git a/jenkins/readme.md b/jenkins/readme.md index 2777a5e..852a113 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -54,12 +54,12 @@ These steps **must** be completed prior to starting other labs. Perform the following labs to test-drive Valet -- [Configure credentials for Valet](1-configure.md) -- [Perform an audit of a Jenkins server](2-audit.md) -- [Perform a dry-run of a Jenkins pipeline](3-dry-run.md) -- [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -- [Perform a production migration of a Jenkins pipeline](5-migrate.md) -- [Forecast potential build runner usage](6-forecast.md) +1. [Configure credentials for Valet](1-configure.md) +2. [Perform an audit of a Jenkins server](2-audit.md) +3. [Perform a dry-run of a Jenkins pipeline](3-dry-run.md) +4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) +5. [Perform a production migration of a Jenkins pipeline](5-migrate.md) +6.[Forecast potential build runner usage](6-forecast.md) ## Troubleshoot the Valet CLI From 55ae48f1aacdf526a1c3d37eca6b993a7ee685c5 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 11:37:33 -0400 Subject: [PATCH 08/20] Update 4-custom-transformers.md --- jenkins/4-custom-transformers.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 8b41b9c..802e428 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -397,5 +397,4 @@ Thats it! Congratulations you have customized transforming: - runners ## Next Lab - -[Migrating a Jenkins Pipeline](../jenkins/valet-migrate-lab.md#migrate-a-jenkins-project-to-github-actions) +[Forecast Jenkins Usage](../jenkins/5-forecast.md#forecast-the-runner-usage-of-a-jenkins-instance) From 51c7cfb51b1e73ae3c6198b1693d679624488a1d Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 11:39:34 -0400 Subject: [PATCH 09/20] Update 5-forecast.md --- jenkins/5-forecast.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jenkins/5-forecast.md b/jenkins/5-forecast.md index e7aa300..2595015 100644 --- a/jenkins/5-forecast.md +++ b/jenkins/5-forecast.md @@ -177,5 +177,4 @@ Using `--source-file-path` we can combine data from multiple forecast runs into ![combined-report](https://user-images.githubusercontent.com/19557880/186264213-b3201710-8093-4ae5-9aef-5c7f95cc3951.png) ## Next steps - -This concludes the Valet labs for Jenkins! If you are interested exploring the power of Valet more, you can leverage the demo Jenkins Instance and modify and add new projects that more closely match your needs and try out the commands again! +[Migrating a Jenkins Pipeline](../jenkins/6-migrate.md#migrate-a-jenkins-project-to-github-actions) From 7fe198d8aeef6065c2d5d65501558cb21776f9a1 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 1 Sep 2022 11:40:04 -0400 Subject: [PATCH 10/20] Update 6-migrate.md --- jenkins/6-migrate.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jenkins/6-migrate.md b/jenkins/6-migrate.md index 62d55c7..3764467 100644 --- a/jenkins/6-migrate.md +++ b/jenkins/6-migrate.md @@ -54,5 +54,4 @@ Before running the command we need to collect some information: - The migration is complete and we have successfully transformed a pipeline from Jenkins into a GitHub Actions workflow, added it to a GitHub Repository, and run the workflow in Actions. ### Next Lab - -[Forecast Jenkins Usage](../jenkins/valet-forecast-lab.md#forecast-the-runner-usage-of-a-jenkins-instance) +This concludes the Valet labs for Jenkins! If you are interested exploring the power of Valet more, you can leverage the demo Jenkins Instance and modify and add new projects that more closely match your needs and try out the commands again! From a414ae978c977903cb2453793c0817ef7e8ba44a Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Thu, 1 Sep 2022 09:57:41 -0700 Subject: [PATCH 11/20] More edits --- jenkins/1-configure.md | 74 ++++++++++++++++++++---------------------- 1 file changed, 35 insertions(+), 39 deletions(-) diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 20f45f8..1197cd6 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -1,55 +1,51 @@ -# Configure Valet to work with Jenkins +# Configure credentials for Valet -In this lab, you will use the Valet `configure` command to set up the required information to communicate with the Jenkins and GitHub instances. The `configure` command can be used for all of the supported providers, in this lab we will be focusing on Jenkins. +In this lab, you will use the `configure` CLI command to set the required credentials and information for Valet to use when working with Jenkins and GitHub. -- [Prerequisites](#prerequisites) -- [Configuring Valet](#configuring-valet) +You will need to complete all of the setup instructions [here](./readme.md#configure-your-codespace) prior to performing this lab. + +- [Configuring credentials](#configuring-credentials) - [Verify Valet Works](#verify-valet-works) - [Next Lab](#next-lab) -## Prerequisites +## Configuring credentials -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. - -## Configuring Valet - -1. Login to the Jenkins instance to generate a personal access token: +1. Open the Jenkins server in a new browser tab: 1. Click the `PORTS` tab in the codespace terminal window. 2. In the `PORTS` tab find the row for port 8080. - 3. Hover over the address under the `Local Address` column, and click the globe to "open in browser". - 4. Login to the Jenkins server and generate a Jenkins API token. - - Click the `admin` button located within the top right menu. - - Click on the `configure` gear located on the left hand panel. - - Under the `API token` section, click `Add new Token`. - - Add a default name to your token, then click `Generate`. - - Copy the token that was generated and record token for a later step. + 3. Hover over the address under the `Local Address` column and click the globe to "open in browser". - ![configure-result](https://user-images.githubusercontent.com/19557880/184041667-d06cb7f2-a885-474e-b728-7567314aeaf3.png) +2. Create a Jenkins API token: + 1. Click the `admin` button in the top right menu bar. + 2. Click on the `Configure` gear located on the left hand panel. + 3. Click the `Add new Token` button in the `API token` section and click `Generate`. + 4. Copy the generated API token and save in a safe location. -2. Create a GitHub personal access token (PAT). - - Navigate to your GitHub `Settings` - click your profile photo and then click `Settings`. - - Go to `Developer Settings` - - Go to `Personal Access Tokens` -> `Legacy tokens (if present)` - - Click `Generate new token` -> `Legacy tokens (if present)`. If required, provide your password. - - Select at least these scopes: `read packages` and `workflow`. Optionally, provide a text in the **Note** field and change the expiration. - - Click `Generate token` - - Copy the token somewhere safe and temporary. -3. Run Valet configure commands - - In the codespace terminal window click back to the `TERMINAL` tab. - - Within the terminal, ensure you are in the root directory. - - Run `gh valet configure`. - - Use the down arrow key to highlight `Jenkins`, press the spacebar to select, then hit enter to accept. + ![img](https://user-images.githubusercontent.com/19557880/184041667-d06cb7f2-a885-474e-b728-7567314aeaf3.png) + +3. Create a GitHub Personal Access Token (PAT): + 1. Open github.com in a new browser tab. + 2. Click your profile photo in the top right of the UI and click `Settings`. + 3. Click on `Developer Settings` in the left hand panel. + 4. Click `Personal Access Tokens` and then `Legacy tokens` (if present). + 5. Click `Generate new token` and then `Legacy tokens`. You may be required to authenticate with GitHub during this step. + 6. Select the following scopes: `read:packages` and `workflow`. + 7. Click `Generate token`. + 8. Copy the generated PAT and save in a safe location. +4. Run the `configure` CLI command: + - Select the `TERMINAL` tab from within the codespace terminal window. + - Run the following command: `gh valet configure`. + - Using the down arrow key to highlight `Jenkins`, press the spacebar to select, and then hit enter to continue. - At the prompt enter your GitHub Username and press enter. - - At the GitHub Container Registry prompt enter the GitHub PAT generated in step 2 and press enter - - At the GitHub PAT prompt enter the GitHub PAT generated in step 2 and press enter. - - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default, if you are using github.com then the default is the right choice. - - At the Jenkins token prompt enter the Jenkins access token from step 1 and press enter. + - At the GitHub Container Registry prompt enter the GitHub PAT generated in step 3 and press enter. + - At the GitHub PAT prompt enter the GitHub PAT generated in step 3 and press enter. + - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). + - At the Jenkins token prompt enter the Jenkins access token from step 2 and press enter. - At the Jenkins url prompt enter `http://localhost:8080/` and press enter. - - At the Personal access token to fetch source code in GitHub prompt, if any of your Jenkins pipelines have source code in a GitHub repository enter the GitHub PAT that would have access to these files. -4. If all went well you should see a similar output in your terminal: - ![configure-result](https://user-images.githubusercontent.com/19557880/184041328-ce54ea22-b0cd-4c84-b02c-10ad7b09ad89.png) + - At the Personal access token to fetch source code in GitHub prompt hit enter to accept the default value. + ![img](https://user-images.githubusercontent.com/19557880/184041328-ce54ea22-b0cd-4c84-b02c-10ad7b09ad89.png) -## Verify Valet Works +## Verify your environment To verify Valet works we are going to run a `update` and `dry-run` command. We will go further into details about the `dry-run` command in a later lab, but for now we want to get the latest version of Valet and confirm that Valet can perform a dry-run with no errors. From 324359e6609e15dd2a7392ef40164c7b35d1e680 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Thu, 1 Sep 2022 10:08:10 -0700 Subject: [PATCH 12/20] Moar edits --- jenkins/1-configure.md | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 1197cd6..4444a92 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -47,28 +47,24 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ## Verify your environment -To verify Valet works we are going to run a `update` and `dry-run` command. We will go further into details about the `dry-run` command in a later lab, but for now we want to get the latest version of Valet and confirm that Valet can perform a dry-run with no errors. +To verify our environment is configured correctly, we are going to run the `update` CLI command. The `update` CLI command will download the latest version of Valet to your codespace. -1. In the codespace terminal update Valet by running `gh valet update` -2. In the terminal you should see a confirmation that it logged into the GitHub Container Registry and pulled the latest version. +1. In the codespace terminal run the following command: - ``` - Login Succeeded - latest: Pulling from valet-customers/valet-cli - Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb - Status: Image is up to date for ghcr.io/valet-customers/valet-cli:latest - ghcr.io/valet-customers/valet-cli:latest - ``` + ```bash + gh valet update + ``` -3. Next, lets run the dry-run command in the codespaces terminal, to verify we can talk to Jenkins +2. You should see a confirmation that you were logged into the GitHub Container Registry and Valet was updated to the latest version. - ``` - gh valet dry-run jenkins --source-url https://localhost:8080/job/test_pipeline/ --output-dir ./tmp/dry-run-lab - ``` - -4. In the terminal you should see the command was successful, if not it is a good time to practice the configure command again and make sure the access tokens values are correct and were generated with the correct permissions. - ![configure-dry-run](https://user-images.githubusercontent.com/19557880/184255620-8e9b120e-5de0-41df-9cb6-c52028de3b0f.png) + ```bash + Login Succeeded + latest: Pulling from valet-customers/valet-cli + Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb + Status: Image is up to date for ghcr.io/valet-customers/valet-cli:latest + ghcr.io/valet-customers/valet-cli:latest + ``` ### Next Lab -[Audit Jenkins using the Valet audit command](2-audit.md#audit-jenkins-pipelines-using-the-valet-audit-command) +[Perform an audit of a Jenkins server](2-audit.md#perform-an-audit-of-a-jenkins-server) From ad6e0726eb239e10b78c182c5a75181dd35b6a4d Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Thu, 1 Sep 2022 11:56:55 -0700 Subject: [PATCH 13/20] Update link --- jenkins/1-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 4444a92..745cbf9 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -5,7 +5,7 @@ In this lab, you will use the `configure` CLI command to set the required creden You will need to complete all of the setup instructions [here](./readme.md#configure-your-codespace) prior to performing this lab. - [Configuring credentials](#configuring-credentials) -- [Verify Valet Works](#verify-valet-works) +- [Verify your environment](#verify-your-environment) - [Next Lab](#next-lab) ## Configuring credentials From 9b3936d34b484ec12daf5a696bbb9fca76a5be5a Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 08:14:26 -0700 Subject: [PATCH 14/20] Update jenkins/readme.md Co-authored-by: j-dunham --- jenkins/readme.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jenkins/readme.md b/jenkins/readme.md index 852a113..94414af 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -58,8 +58,8 @@ Perform the following labs to test-drive Valet 2. [Perform an audit of a Jenkins server](2-audit.md) 3. [Perform a dry-run of a Jenkins pipeline](3-dry-run.md) 4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Perform a production migration of a Jenkins pipeline](5-migrate.md) -6.[Forecast potential build runner usage](6-forecast.md) +5. [Forecast potential build runner usage](5-forecast.md) +6. [Perform a production migration of a Jenkins pipeline](6-migrate.md) ## Troubleshoot the Valet CLI From 7d9bff8e667cfdb9ceb15725ebe404852097f836 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 09:23:04 -0700 Subject: [PATCH 15/20] Copyedit audit lab --- jenkins/2-audit.md | 150 +++++++++++++++++++++------------------------ 1 file changed, 71 insertions(+), 79 deletions(-) diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index d0f1d8f..7d6ddf6 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -1,124 +1,116 @@ -# Audit Jenkins pipelines using the Valet audit command +# Perform an audit of a Jenkins server -In this lab, you will use Valet to `audit` a Jenkins organization. The `audit` command can be used to scan a CI server and output a summary of the current pipelines. +In this lab, you will use the `audit` command to get a high-level view of all pipelines in a Jenkins server. -What happens behind the scenes is that Valet will perform a `dry-run` on each of the Jenkins pipelines. Once that is complete, Valet will perform an aggregation of all of the transformed workflows. This aggregate summary can be used as a planning tool and help understand how complete of a migration is possible with Valet. +This `audit` command operates by fetching all of the pipelines defined in a Jenkins server, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. -By the end of this lab you will have performed an audit on the demo Jenkins instance, and have a good understanding of the components that make up an audit. - -- [Prerequisites](#prerequisites) -- [Perform an audit](#perform-an-audit) -- [View audit output](#view-audit-output) -- [Review the pipelines](#review-the-pipelines) -- [Next Lab](#next-lab) +1. [Prerequisites](#prerequisites) +1. [Perform an audit](#perform-an-audit) +1. [Inspect the output files](#inspect-the-output-files) +1. [Next lab](#next-lab) ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials) to configure credentials for Valet to use. ## Perform an audit -We will be performing an audit against a preconfigured Jenkins instance. Before running the command we need to collect some information: +We will be performing an audit against a preconfigured Jenkins instance. We will need to answer the following questions before running this command: - 1. Do we want to audit the entire Jenkins instance, or just a single folder? __In this example we will be auditing the entire Jenkins instance, but in the future if you wanted to configure a specific folder to be audited add the `-f ` flag to the audit command__ - 2. Where do we want to store the result? __./tmp/audit. This can be any valid path on the system. In the case of codespaces it is generally best to use `./tmp/SOME_DIRECTORY_HERE` so the files show in explorer__ +1. Do we want to audit the entire Jenkins instance or just a single folder? + - In this example we will be auditing the entire Jenkins instance, but in the future if you wanted to configure a specific folder to be audited add the `-f ` flag to the audit command. + +2. Where do we want to store the result? + - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. ### Steps 1. Navigate to the codespace terminal. -2. Now, from root directory, run the following Valet audit command: +2. Run the following command from the root directory: -``` -gh valet audit jenkins --output-dir tmp/audit -``` + ```bash + gh valet audit jenkins --output-dir tmp/audit + ``` -3. Valet will log the output files in green when the audit is successful +3. The command will list all the files written to disk in green when the command succeeds. -### Example + ![img](https://user-images.githubusercontent.com/19557880/184682347-b19760fa-36a6-423e-a445-bb30eda5ac59.png) -valet-audit-1 +## Inspect the output files -## View audit output +The audit summary, logs, config files, jenkinsfiles, and transformed workflows will be located within the `tmp/audit` folder. -The audit summary, logs, config files, jenkinsfiles, and transformed Actions Workflows should all be located within the `tmp/audit` folder. +1. Find the `audit_summary.md` file in the file explorer. +2. Right-click the `audit_summary.md` file and select `Open Preview`. +3. This file contains details that summarizes what percentage of your pipelines were converted automatically. -1. Under the `audit` folder find the `audit_summary.md` -2. Right-click the `audit_summary.md` file and select `Open Preview` -3. The file contains details about your current pipelines and what can be migrated 100% automatically vs. what will need some manual intervention or aren't supported by GitHub Actions. -4. Review the file, it should look like the image below: +### Review the audit summary -### Example +#### Pipelines -valet-audit-2 +The pipeline summary section contains high level statistics regarding the conversion rate done by Valet: -## Review the pipelines + ![img](https://user-images.githubusercontent.com/19557880/184683664-81985baf-5c03-4765-a067-f4023416e3ea.png) -### Pipelines +Here are some key terms in the “Pipelines” section in the above example: -The audit summary starts by giving a summary of the types of pipelines that were extracted from Jenkins. +- __Successful__ pipelines had 100% of the pipeline constructs and individual items converted automatically to their GitHub Actions equivalent. +- __Partially successful__ pipelines had all of the pipeline constructs converted, however, there were some individual items (e.g. build tasks or build triggers) that were not converted automatically to their GitHub Actions equivalent. +- __Unsupported__ pipelines are definition types that are not supported by Valet. The following Jenkins pipeline types are supported: + - Flow Definition + - Project (declarative Jenkinsfile pipelines) + - Multibranch Project +- __Failed pipelines__ encountered a fatal error when being converted. This can occur for one of three reasons: + - The pipeline was misconfigured and not valid in Jenkins. + - Valet encountered an internal error when converting it. + - There was an unsuccessful network response, often due to invalid credentials, that caused the pipeline to be inaccessible. -- It shows that there are a total of 7 pipelines extracted. +The “Job types” section will summarize which types of pipelines are being used and which are supported or unsupported by Valet. -- 42% pipelines were successful. This means that Valet knew how to map all the constructs of the Jenkins pipeline to a GitHub Actions equivalent. All of the build plugins and triggers that are referenced were all successfully converted into a GitHub Actions equivalent. +#### Build steps -- 42% pipelines were partially successful. This means that Valet knew how to map all the constructs of the Jenkins pipeline but there may be a plugin that was referenced that Valet wasn't able to automatically map to a Github Actions equivalent. +The build steps summary section presents an overview of the individual build steps that are used across all pipelines and how many were automatically converted by Valet. -- 1% of these pipelines were unsupported. This means that the pipeline type is fundamentally unsupported by Valet. This is most likely a Jenkins scripted pipeline. + ![img](https://user-images.githubusercontent.com/19557880/184684062-69ab0bde-5e32-45f8-a7dd-ed4655872975.png) -- 0% of these fail altogether. If there were any pipelines that would fall under this category, that would mean that those pipelines were misconfigured or there was an issue with Valet. +Here are some key terms in the “Build steps” section in the above example: -Under the `Job types` section, we can see that the `audit` command is able to support the conversion of project, freestyle (flow-definition), and multibranch pipelines from Jenkins and convert them to a GitHub Actions workflow. Valet does not support converting [scripted pipelines](https://www.jenkins.io/doc/book/pipeline/syntax/#scripted-pipeline) (e.g. pure Groovy). +- A __known__ build step is a step that was automatically converted to an equivalent action. +- An __unknown__ build step is a step that was not automatically converted to an equivalent action. +- An __unsupported__ build step is a step that is either: + - A step that is fundamentally not supported by GitHub Actions. + - A step that is configured in a way that is incompatible with GitHub Actions. +- An __action__ is a list of the actions that were used in the converted workflows. This is important for the following scenarios: + - Gathering the list of actions to sync to your appliance if you use GitHub Enterprise Server. + - Defining an organization-level allowlist of actions that can be used. This list of actions is a comprehensive list of which actions their security and/or compliance teams will need to review. -#### Example +There is an equivalent breakdown of build triggers, environment variables, and other uncategorized items displayed in the audit summary file. -valet-audit-3 +#### Manual Tasks -### Build steps +The manual tasks summary section presents an overview of the manual tasks that you will need to perform that Valet is not able to complete automatically. -Under the `Build steps` section we can see a breakdown of the build steps that were used in these pipelines. + ![img](https://user-images.githubusercontent.com/19557880/184684249-9accfd94-c2df-4891-af56-dcff66beb557.png) -- Supported: 12/16 discrete build steps are considered known by Valet. When Valet encounters a build step of this type, it knows exactly how to map that into a GitHub Actions equivalent. -- Unknown: 3/16 discrete build steps are considered unknown by Valet. When Valet encounters a build step of this type, it does not yet know to map this automatically to a GitHub Action equivalent. -- Unsupported: 1/16 discrete build steps are considered unsupported by Valet. This could mean one of three things: - 1. The way that plugin was configured for a given job is unsupported. - 2. The plugin itself is fundamentally not supported in GitHub Actions. - 3. It's supported by default in GitHub Actions. +Here are some key terms in the “Manual tasks” section in the above example: -Under the `Actions` section we have the list of the Actions that were used in order to implement the transformation of all of these build steps. Valet is a planning tool that can help in facilitating the migration into GitHub Actions and this list of Actions is a great place to understand what dependencies you would be taking on third-party Actions after this migration. +- A __secret__ refers to a repository or organization level secret that is used by the converted pipelines. These secrets will need to be created manually in Actions in order for these pipelines to function properly. +- A __self-hosted runner__ refers to a label of a runner that is referenced by a converted pipeline that is not a GitHub-hosted runner. You will need to manually define these runners in order for these pipelines to function properly. -For example, if you are doing things like setting up the allow list of third-party Actions in a GitHub Enterprise server instance this list of Actions is a fantastic place to begin security reviews and audits of what third-party actions to depend on. +#### Files -#### Example +The final section of the audit report provides a manifest of all of the files that are written to disk during the audit. These files include: -valet-audit-4 + ![img](https://user-images.githubusercontent.com/19557880/184684416-b3db774e-4ab8-46e0-91ad-e503632df5cb.png) -### Trigger, Environment, Other +Each pipeline will have a variety of files written that include: -Similar to `Build steps`, there are `Trigger`, `Environment`, and a catch all `Other` section that breakdown each of their uses across the audited pipelines. +- The original pipeline as it was defined in Jenkins. +- Any network responses used to convert a pipeline. +- The converted workflow. +- Stack traces that can used to troubleshoot a failed pipeline conversion -### Example +## Next lab -valet-audit-5 - -### Manual Tasks - -Under the Manual task section you will find a list of all the manual tasks that the pipelines would surface in a migration. Manual tasks are Valet's way of indicating tasks a user needs to do in order for a pipeline to be functional, such as adding `secrets`, or setting up a `self-hosted` runner. We will see how these manual tasks appear on a pull request when we do a migration in a lab later on. - -### Example - -valet-audit-5 - -### Files - -At the end of the Audit Summary page you will find a list of all of the files that were written to disk. Generally, for any given pipeline, you’ll find 2 or 3 associated files. In these files are the actual converted GitHub Actions workflows. - -In addition, you’ll see a file that shows the raw JSON data that we pull from Jenkins as well as any associated Jenkinsfiles for a given job. These files are really useful for engineering teams to help debug any issues and to understand what may have gone on in a transformation. - -#### Example - -valet-audit-6 - -### Next Lab - -[Dry-run the migration of a Jenkins pipeline to GitHub Actions](3-dry-run.md) +[Perform a dry-run of a Jenkins pipeline](3-dry-run.md) From 058b33e146893ed90bbfc51e2ef832631a22a682 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 11:15:07 -0700 Subject: [PATCH 16/20] Edit dry run and custom transformers labs --- jenkins/2-audit.md | 6 +- jenkins/3-dry-run.md | 103 ++++------- jenkins/4-custom-transformers.md | 305 ++++++++++++------------------- 3 files changed, 151 insertions(+), 263 deletions(-) diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index 7d6ddf6..134ef60 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -12,11 +12,11 @@ This `audit` command operates by fetching all of the pipelines defined in a Jenk ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials) to configure credentials for Valet to use. +2. Completed the [configure lab](./1-configure.md#configuring-credentials). ## Perform an audit -We will be performing an audit against a preconfigured Jenkins instance. We will need to answer the following questions before running this command: +We will be performing an audit against a preconfigured Jenkins server. We will need to answer the following questions before running this command: 1. Do we want to audit the entire Jenkins instance or just a single folder? - In this example we will be auditing the entire Jenkins instance, but in the future if you wanted to configure a specific folder to be audited add the `-f ` flag to the audit command. @@ -28,7 +28,7 @@ We will be performing an audit against a preconfigured Jenkins instance. We will 1. Navigate to the codespace terminal. 2. Run the following command from the root directory: - + ```bash gh valet audit jenkins --output-dir tmp/audit ``` diff --git a/jenkins/3-dry-run.md b/jenkins/3-dry-run.md index 548b141..08d9e94 100644 --- a/jenkins/3-dry-run.md +++ b/jenkins/3-dry-run.md @@ -1,53 +1,57 @@ -# Dry-run the migration of a Jenkins pipeline to GitHub Actions +# Perform a dry-run of a Jenkins pipeline In this lab, you will use the Valet `dry-run` command to convert a Jenkins pipeline to its equivalent GitHub Actions workflow. The end result of this command will be the actions workflow written to your local filesystem. - [Prerequisites](#prerequisites) - [Perform a dry-run](#perform-a-dry-run) -- [Review dry-run output](#review-dry-run-output) -- [Next Lab](#next-lab) +- [Inspect the output files](#inspect-the-output-files) +- [Next lab](#next-lab) ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. -3. Completed the [audit lab](../Jenkins/valet-audit-lab.md#audit-jenkins-pipelines-using-the-valet-audit-command). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [audit lab](./1-audit.md). ## Perform a dry-run -We will be performing a dry-run against a preconfigured pipeline in the Jenkins instance. Before running the command we need to collect some information: +We will be performing a dry-run against a pipeline in the preconfigured Jenkins server. We will need to answer the following questions before running this command: - 1. What is the name of the pipeline we want to convert? __test_pipeline__ - 2. What is the source URL of the pipeline we want to convert? ____ - 3. Where do we want to store the result? __./tmp/dry-run-lab. This can be any valid path on the system. In the case of codespaces it is generally best to use `./tmp/SOME_DIRECTORY_HERE` so the files show in explorer__ +1. What is the name of the pipeline we want to convert? + - __test_pipeline__ + +2. What is the URL of the pipeline we want to convert? + - ____ + +3. Where do we want to store the result? + - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. ### Steps 1. Navigate to the codespace terminal -2. Run the dry-run command using the values determined above +2. Run the following command from the root directory: - ``` - gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run - ``` + ```bash + gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run + ``` -3. When the command finishes the output files should be printed to the terminal. - Screen Shot 2022-08-16 at 9 54 26 AM -4. Open generated actions workflow - - Find `./tmp/dry-run-lab/valet` in the file explorer pane in codespaces. - - Click `test_pipeline.yml` to open +3. The command will list all the files written to disk when the command succeeds. - Screen Shot 2022-08-16 at 9 55 44 AM + ![img](https://user-images.githubusercontent.com/19557880/184935603-5c2d4dfe-66ef-4cb1-9398-e96954ca72e3.png) -## Review dry-run output +4. View the converted workflow: + - Find `./tmp/dry-run` in the file explorer pane in codespaces. + - Click `test_pipeline.yml` to open -The dry-run output will show you the GitHub Actions yaml that would be migrated to GitHub with the `migrate` command. We will now take a quick look at what was generated. +## Inspect the output files + +The files generated from the `dry-run` command represent the equivalent Actions workflow for the given Jenkins pipeline. The Jenkins pipeline and converted workflow can be seen below: -__Click to Expand__
- Jenkins Pipeline + Jenkins Pipeline 👇 -```yaml +```groovy pipeline { agent { label 'TeamARunner' @@ -78,7 +82,7 @@ pipeline {
- Actions Workflow + Converted workflow 👇 ```yaml name: test_pipeline @@ -125,35 +129,7 @@ jobs:
-In the Jenkins pipeline we have 2 stages and 4 steps that run on the self-hosted runner labeled `TeamARunner`. - -In the Actions workflow we have the same steps and the stages are now being enforced using the `needs` keyword. We can see this if we examine the `test` job, it has `needs: build`, which makes it depend on the `build` job. - -```diff -- stages: test -+ needs: build -``` - -The `agent` in the Jenkins pipeline has been transformed to `runs-on` on each of the jobs. - -```diff -- agent { -- label 'TeamARunner' -- } -+ runs-on: -+ - self-hosted -+ - TeamARunner -``` - -And the `echo` commands remain mostly the same - -```diff -- echo "Database engine is ${DB_ENGINE}" -+ - name: echo message -+ run: echo "DISABLE_AUTH is ${{ env.DISABLE_AUTH }}" -``` - -Note how Valet was not able to find a suitable conversion for the `sleep` command, and it added a comment to the yaml so this information was not lost, and it could be addressed manually later if needed. +These 2 pipelines function equivantly despite using different syntax. In this case, the pipeline conversion was “partially successful” (i.e. there were item(s) not automatically converted) and the unconverted item was placed as comment in the location the Jenkins pipeline used it. For example: ```diff - sleep 80 @@ -165,19 +141,10 @@ Note how Valet was not able to find a suitable conversion for the `sleep` comman + # value: 80 ``` -Lastly, the `junit` command was transformed using a third party action `EnricoMi/publish-unit-test-result-action@v1.7` +In the next lab, we'll learn how to override Valet's default behavior and customize the converted workflow that is generate. -```diff -- junit '**/target/*.xml' -+ - name: Publish test results -+ uses: EnricoMi/publish-unit-test-result-action@v1.7 -+ if: always() -+ with: -+ files: "**/target/*.xml" -``` +Try running the `dry-run` command for different pipelines in the Jenkins server. As a hint, you just have to change the `--source-url` CLI option. -Try constructing and running the `dry-run` command yourself. Hint, you should just have to change the project name. +## Next lab -## Next Lab - -[Using Custom Transformers in a dry-run](4-custom-transformers.md#using-custom-transformers-in-a-dry-run) +[Use custom transformers to customize Valet's behavior](4-custom-transformers.md) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 802e428..1cf4658 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -1,13 +1,13 @@ -# Using Custom Transformers in a `dry-run` +# Use custom transformers to customize Valet's behavior -In this lab we want to do a `dry-run` of the `test_pipeline` pipeline, however, after a closer inspection, we discovered that we will need to customize how Valet transforms: +In this lab we will build upon the `dry-run` command to override Valet's default behavior and customize the converted workflow using "custom transformers". Custom transformers can be used to: -1. Steps that are unsupported by Valet, but are essential to our devops process. -2. Steps in which the GitHub action chosen by Valet does not meet our current needs. -3. All pipelines that have the environment variable `DB_ENGINE` must be changed to a different value: `mongodb`. -4. The self-hosted runners before used the `TeamARunner` label in Jenkins but will run on the `ubuntu-latest` runner in Actions. +1. Convert items that are not automatically converted. +2. Convert items that were automatically converted using different actions. +3. Convert environment variable values differently. +4. Convert references to runners to use a different runner name in Actions. -These customizations will be present in many of our company's pipelines and an automated way to apply these changes would be ideal. In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language. +In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language. - [Prerequisites](#prerequisites) - [Perform a dry-run](#perform-a-dry-run) @@ -19,23 +19,22 @@ These customizations will be present in many of our company's pipelines and an a ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. -3. Completed the [dry-run lab](../jenkins/valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [dry-run lab](./1-dry-run.md). ## Perform a dry-run -- Let’s run the `dry-run` command to see what information we can get from the generated action yaml. +We will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal: - ```bash - gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run - ``` + ```bash + gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o tmp/jenkins/dry-run + ``` -- Open the resulting GitHub Actions workflow by navigating to `tmp/valet/test_pipeline.yml` from the explorer +The converted workflow that is generated by the above command can be seen below -__Click to Expand__
- Actions Workflow + Converted workflow 👇 ```yaml name: test_pipeline @@ -98,47 +97,55 @@ jobs:
-Go back to the previous lab [dry-run Jenkins pipeline](../jenkins/valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions) if you need a review of the details of the dry-run. + + +_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step -We can see in the center of the transformed workflow that the `sleep` step was not transformed. In order for us to write a custom transformer for this we need to know the identifier. In general, the identifier will be the key of a key/value pair within the step of a Jenkinsfile, which in this case is `sleep`. This is how our custom transformer will target the correct step. +The converted workflow above contains a `sleep` step was not automatically converted. We will need to answer the following questions before writing a custom transformer: -After some research we have decided that a simple bash script will meet our needs: +1. What is the "identifier" of the step to customize? + - __sleep__. The identifier will be the key of a key value pair within the step of a Jenkinsfile. - ```yaml - - name: Sleep for 80 seconds - run: sleep 80s - shell: bash - ``` +2. What is the desired Actions syntax to use instead? + - After some research, we have determined that the following bash script will provide similar functionality: -Now that we know the final yaml needed for the transformer, we can start to write the ruby file.The custom transformers file can have any name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting. + ```yaml + - name: Sleep for 80 seconds + run: sleep 80s + shell: bash + ``` -- Create a new file where you will put the custom transformers logic. It can have any file name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting. For this example we will call it `transformers.rb`. - -In the custom transformers file we will add a `transform` method. This is a special method that Valet exposes, that takes the identifier we determined earlier and returns a ruby hash of the final YAML for the pipeline. - -The ruby hash can be thought of as the JSON representation of the YAML we want. Valet will call that method when it encounters the identifier and pass in an `item`. The `item` is the key of that key/value pair defined for that step in Jenkins. In this case the item is the settings of the sleep command. - - ```ruby - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - ``` - -Let’s run the `dry-run` command again with `--custom-transformer transformers.rb` to see if our custom transformer worked. +Now we can begin to write the custom transformer. Customer transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb +code transformers.rb ``` -When you open the file you should see the following changes in the GitHub Actions Workflow, note how the sleep timer step is no longer commented out! +Next, we will define a `transform` method for the `sleep` identifier by adding the following code to `transformers.rb`: + +```ruby +transform "sleep" do |item| + wait_time = item["arguments"][0]["value"]["value"] + + { + "name": "Sleep for #{wait_time} seconds", + "run": "sleep #{wait_time}s", + "shell": "bash" + } +end +``` + +This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Jenkins. + +Now, we can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: + +```bash +gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o tmp/jenkins/dry-run --custom-transformers transformers.rb +``` + +Open the workflow that is generated and inspect the contents. Now, the `sleep` step is converted and uses the customized behavior! ```diff - # # This item has no matching transformer @@ -154,22 +161,24 @@ When you open the file you should see the following changes in the GitHub Action ## Custom transformers for a known step -Next lets address the third-party GitHub action `EnricoMi/publish-unit-test-result-action@v1.7` that Valet chose for the `junit` step. This action does not meet our needs because our CTO has dictated that our pipelines take no dependencies on third-party actions. We can customize the action that we would like Valet to use instead. +We can also override Valet's default behavior. In this scenario, we may not desire to use the third-party action for publishing junit test results that is used by default. Again, we will need to answer the following questions before writing a custom transformer: -After some research we find that the following action will upload an artifact with the junit test results without depending on a third party action. +1. What is the "identifier" of the step to customize? + - __junit__ -```yaml -- uses: actions/upload-artifact@v3 - with: - name: junit-artifact - path: path/to/artifact/world.txt -``` +2. What is the desired Actions syntax to use instead? + - After some research, we have determined that the uploading test results as an artifact will be suitable: -We will build this custom transformer similar to how we built the previous custom transformer. This time, we need to add some additional logic. We would like to programmatically assign the file path to junit's test results to our upload action. + ```yaml + - uses: actions/upload-artifact@v3 + with: + name: junit-artifact + path: path/to/artifact/world.txt + ``` -The challenge is, that we are unsure what `item` represents (what the incoming JSON object looks like from Jenkins). In order to troubleshoot this, you could use some basic ruby to print `item` to the terminal. You can achieve this by adding the following line in the transform method: `puts "This is the item: #{item}"`. +We will build this custom transformer similar to the previous custom transformer, however, we first need to inspect the `item` keyword to programmatically use the file path to junit's test results in the `actions/upload-artifact@v3` step. -You are able to add multiple custom transformers to the same file, so you can add this one right below the previous custom transformer. +To do this, we will print `item` to the console. You can achieve this by adding the following custom transformer to `transformers.rb`: ```ruby transform "junit" do |item| @@ -177,77 +186,30 @@ transform "junit" do |item| end ``` -Let’s run the `dry-run` command to see what information is outputted in the terminal. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. The output of the `dry-run` command should look similar to this: -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` +![img](https://user-images.githubusercontent.com/19557880/186782050-65ece0c4-52a3-4f88-818f-0f860c50c2b7.png) -You should see something like the following outputted into the terminal: - -Screen Shot 2022-08-17 at 10 25 53 AM - -Now that we know the structure of the incoming JSON blob for `item` we can access the file path programmatically. +Now that we know the data structure of `item` we can access the file path programmatically by editing the custom transformer to following: ```ruby transform "junit" do |item| test_results = item["arguments"].find{ |a| a["key"] == "testResults" } file_path = test_results.dig("value", "value") - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "junit-artifact", - "path": file_path - } - } - ] + { + "uses" => "actions/upload-artifact@v3", + "with" => { + "name" => "junit-artifact", + "path" => file_path + } + } end ``` -Your file should include both custom transformers. Ensure it matches the file below: +_Note_: `transformers.rb` should contain a `transform` method for both `sleep` and `junit`. -__Click to Expand__ -
- Custom Transformer file - -```ruby - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - - transform "junit" do |item| - test_results = item["arguments"].find{ |a| a["key"] == "testResults" } - file_path = test_results.dig("value", "value") - - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "junit-artifact", - "path": file_path - } - } - ] - end -``` - -
- -Let’s run the `dry-run` command again to see if our custom transformer worked. - -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` - -When you open the file you should see the`EnricoMi/publish-unit-test-result-action@v1.7` action has been replaced with the customized steps. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `EnricoMi/publish-unit-test-result-action@v1.7` action will have been replaced with the customized steps. ```diff - - name: Publish test results @@ -263,58 +225,17 @@ When you open the file you should see the`EnricoMi/publish-unit-test-result-acti ## Custom transformers for environment variables -But wait! There's more we need to customize to make these pipelines fit our needs. Several of our pipelines will need to have the value of the `DB_ENGINE` environment variable changed from `sqlite` to `mongodb`. +We can also use custom transformers to edit the values of environment variables in converted workflows. In our example, we will be updating the `DB_ENGINE` environment variable to be `mongodb` instead of `sqlite`. -Valet allows you to replace values of `variables` by using the `env` method. Let’s replace the value for `DB_ENGINE` by using the below line. The first value of the `env` method is the target variable name and the second is the new value to be used. - - ```ruby - env "DB_ENGINE", "mongodb" - ``` - -You can keep adding these transformer values to your `transformer.rb` file. Your transformer file should look like the one below: - -__Click to Expand__ -
- Custom Transformer file +To do this, add the following code to the `transformers.rb` file. ```ruby - env "DB_ENGINE", "mongodb" - - transform "sleep" do |item| - wait_time = item["arguments"][0]["value"]["value"] - - { - "name": "Sleep for #{wait_time} seconds", - "run": "sleep #{wait_time}s", - "shell": "bash" - } - end - - transform "junit" do |item| - test_results = item["arguments"].find{ |a| a["key"] == "testResults" } - file_path = test_results.dig("value", "value") - - [ - { - "uses": "actions/upload-artifact@v3", - "with": { - "name": "my-artifact", - "path": file_path - } - } - ] - end +env "DB_ENGINE", "mongodb" ``` -
+In this example, the first parameter to the `env` method is the environment variable name and the second is the updated value. -Let’s run the `dry-run` command again to see if our custom transformer worked. - -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` - -When you open the outputed file you should see that the value for `DB_ENGINE` has been changed from `sqlite` to `mongodb`. +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `DB_ENGINE` environment variable will be set to `mongodb`: ```diff env: @@ -325,21 +246,35 @@ env: ## Custom transformers for runners -Lastly, several of our pipelines will need to have the runner label updated from `TeamARunner` to `ubuntu-latest`. +Finally, we can use custom transformers to dictate which runners converted workflows should use. To do this we will need to answer the following questions: -Valet offers the ability to provide customized mapping between build agents in the source CI instance and their equivalent GitHub Actions runner labels by using the `runner` method. The syntax for this is similar to the `env` method. +1. What is label of the runner in Jenkins to update? + - __TeamARunner__ -The `runner` methods first parameter is the Jenkins agent label and the second parameter is the GitHub Actions runner label(s) to map too. As an example, the following could be used: +2. What is the label of the runner in Actions to use instead? + - __ubuntu-latest__ - ```ruby - runner "TeamARunner", "ubuntu-latest" - ``` +With these questions answered, we can add the following code to the `transformers.rb` file: -You can keep adding these transformer methods to your `transformer.rb` file. Your transformer file should look like the one below: +```ruby +runner "TeamARunner", "ubuntu-latest" +``` + +In this example, the first parameter to the `runner` method is the Jenkins label and the second is the Actions runner label. + +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` statement will use the customized runner label: + +```diff +runs-on: +- - self-hosted +- - TeamARunner ++ - ubuntu-latest +``` + +At this point of the lab the file contents of `transformers.rb` should match this: -__Click to Expand__
- Custom Transformer file + Custom transformers 👇 ```ruby runner "TeamARunner", "ubuntu-latest" @@ -374,27 +309,13 @@ __Click to Expand__
-Let’s run the `dry-run` command again to see if our custom transformer worked. +Thats it! Congratulations you have overridden Valet's default behavior by customizing the conversion of: -```bash -gh valet dry-run jenkins --source-url http://localhost:8080/job/test_pipeline -o .tmp/jenkins/dry-run --custom-transformers transformers.rb -``` +- Unknown steps +- Known steps +- Environment variables +- Runners -When you open the file you should see that the value for `TeamARunner` has been changed to `ubuntu-latest`. +## Next lab -```diff -runs-on: -- - self-hosted -- - TeamARunner -+ - ubuntu-latest -``` - -Thats it! Congratulations you have customized transforming: - -- unsupported steps -- steps that are supported but don't meet your requirements -- environment variables -- runners - -## Next Lab [Forecast Jenkins Usage](../jenkins/5-forecast.md#forecast-the-runner-usage-of-a-jenkins-instance) From 3af1b573069fa43d0f7b73e89e1c0e68843ae0a8 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 11:16:22 -0700 Subject: [PATCH 17/20] Reorder the migrate and forecast labs --- jenkins/4-custom-transformers.md | 2 +- jenkins/{6-migrate.md => 5-migrate.md} | 0 jenkins/{5-forecast.md => 6-forecast.md} | 0 jenkins/readme.md | 4 ++-- 4 files changed, 3 insertions(+), 3 deletions(-) rename jenkins/{6-migrate.md => 5-migrate.md} (100%) rename jenkins/{5-forecast.md => 6-forecast.md} (100%) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 1cf4658..612d274 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -318,4 +318,4 @@ Thats it! Congratulations you have overridden Valet's default behavior by custom ## Next lab -[Forecast Jenkins Usage](../jenkins/5-forecast.md#forecast-the-runner-usage-of-a-jenkins-instance) +[Perform a production migration of a Jenkins pipeline](5-migrate.md) diff --git a/jenkins/6-migrate.md b/jenkins/5-migrate.md similarity index 100% rename from jenkins/6-migrate.md rename to jenkins/5-migrate.md diff --git a/jenkins/5-forecast.md b/jenkins/6-forecast.md similarity index 100% rename from jenkins/5-forecast.md rename to jenkins/6-forecast.md diff --git a/jenkins/readme.md b/jenkins/readme.md index 94414af..9b76a0f 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -58,8 +58,8 @@ Perform the following labs to test-drive Valet 2. [Perform an audit of a Jenkins server](2-audit.md) 3. [Perform a dry-run of a Jenkins pipeline](3-dry-run.md) 4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Forecast potential build runner usage](5-forecast.md) -6. [Perform a production migration of a Jenkins pipeline](6-migrate.md) +5. [Perform a production migration of a Jenkins pipeline](5-migrate.md) +6. [Forecast potential build runner usage](6-forecast.md) ## Troubleshoot the Valet CLI From c70c92f8942ab8aecea55eef7fee3d206e97fa40 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 12:20:34 -0700 Subject: [PATCH 18/20] Edit migrate lab --- jenkins/2-audit.md | 5 --- jenkins/3-dry-run.md | 10 +---- jenkins/4-custom-transformers.md | 12 +----- jenkins/5-migrate.md | 67 +++++++++++++++----------------- jenkins/6-forecast.md | 2 + 5 files changed, 37 insertions(+), 59 deletions(-) diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index 134ef60..5cfb27c 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -4,11 +4,6 @@ In this lab, you will use the `audit` command to get a high-level view of all pi This `audit` command operates by fetching all of the pipelines defined in a Jenkins server, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. -1. [Prerequisites](#prerequisites) -1. [Perform an audit](#perform-an-audit) -1. [Inspect the output files](#inspect-the-output-files) -1. [Next lab](#next-lab) - ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. diff --git a/jenkins/3-dry-run.md b/jenkins/3-dry-run.md index 08d9e94..cd8795b 100644 --- a/jenkins/3-dry-run.md +++ b/jenkins/3-dry-run.md @@ -1,18 +1,12 @@ # Perform a dry-run of a Jenkins pipeline -In this lab, you will use the Valet `dry-run` command to convert a Jenkins pipeline to its equivalent GitHub Actions workflow. -The end result of this command will be the actions workflow written to your local filesystem. - -- [Prerequisites](#prerequisites) -- [Perform a dry-run](#perform-a-dry-run) -- [Inspect the output files](#inspect-the-output-files) -- [Next lab](#next-lab) +In this lab you will use the `dry-run` command to convert a Jenkins pipeline to its equivalent GitHub Actions workflow. ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [audit lab](./1-audit.md). +3. Completed the [audit lab](./2-audit.md). ## Perform a dry-run diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 612d274..87d48de 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -7,21 +7,11 @@ In this lab we will build upon the `dry-run` command to override Valet's default 3. Convert environment variable values differently. 4. Convert references to runners to use a different runner name in Actions. -In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language. - -- [Prerequisites](#prerequisites) -- [Perform a dry-run](#perform-a-dry-run) -- [Custom transformers for an unknown step](#custom-transformers-for-an-unknown-step) -- [Custom transformers for an known step](#custom-transformers-for-a-known-step) -- [Custom transformers for environment variables](#custom-transformers-for-environment-variables) -- [Custom transformers for runners](#custom-transformers-for-runners) -- [Next Lab](#next-lab) - ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./1-dry-run.md). +3. Completed the [dry-run lab](./3-dry-run.md). ## Perform a dry-run diff --git a/jenkins/5-migrate.md b/jenkins/5-migrate.md index 3764467..9b69d3e 100644 --- a/jenkins/5-migrate.md +++ b/jenkins/5-migrate.md @@ -1,57 +1,54 @@ -# Migrate a Jenkins Project to GitHub Actions +# Perform a production migration of a Jenkins pipeline -In this lab, we will use the Valet `migrate` command to migrate a Jenkins pipeline to GitHub Actions. -The previous commands used in the labs, such as `audit` and `dry-run` have prepared us to run a migration. -The `migrate` command will transform the Jenkins pipeline into a GitHub Actions workflow like the `dry-run` command did, but instead of writing these files locally, it will open a pull request with the files. -The pull request will contain a checklist of `Manual Tasks` if required. These tasks are changes that Valet could not do on our behalf, like creating a runner or adding a secret to a repository. - -- [Prerequisites](#prerequisites) -- [Preparing for migration](#preparing-for-migration) -- [Performing the migration](#performing-a-migration) -- [Reviewing the pull request](#reviewing-the-pull-request) -- [Next Lab](#next-lab) +In this lab, you will use the `migrate` command to convert a Jenkins pipeline and open a pull request with the equivalent Actions workflow. ## Prerequisites -1. Followed the steps [here](../jenkins/readme.md#valet-labs-for-jenkins) to set up your Codespace environment and start a Jenkins server. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. -3. Completed the [dry-run lab](../jenkins/valet-dry-run-lab.md#dry-run-the-migration-of-a-jenkins-pipeline-to-github-actions). -4. Completed the [dry-run lab with custom transformer](../jenkins/valet-custom-transformers-lab.md#using-custom-transformers-in-a-dry-run). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [dry-run lab](./3-dry-run.md). +4. Completed the [custom transformers lab](./4-custom-transformers.md). ## Preparing for migration -Before running the command we need to collect some information: +We need to answer the following questions before running a `migrate` command: - 1. What is the name of the pipeline we want to convert? __monas_dev_work/job/monas_freestyle__ - 2. What is the source URL of the pipeline we want to convert? ____ - 3. Where do we want to store the result? __./tmp/migrate__. - 4. What is the URL for the GitHub repository we want to add the workflow too? __this repo!__. *When constructing the value for the migrate command it should match this url with `GITHUB-ORG-USERNAME` and `GITHUB-REPO-NAME` substitued with your values* +1. What is the source URL of the pipeline we want to convert? + - ____ +2. Where do we want to store the result? + - __./tmp/migrate__ +3. What is the URL for the GitHub repository to add the workflow to? + - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. ## Performing a migration -1. Run `migrate` command using the information collected above, make sure to update the `--target-url` value with the information from step 4. +1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate jenkins --target-url https://github.com/GITHUB-ORG-USERNAME-HERE/GITHUB-REPO-NAME-HERE --output-dir ./tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle + gh valet migrate jenkins --target-urlhttps://github.com/:owner/:repo --output-dir ./tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle ``` -2. Valet will create a pull request directly in the target GitHub repository. -3. Open the pull request by clicking the green pull request link in the output of the migrate command, if you have trouble clicking it you can always copy and paste the url in your browser. - ![pr-screen-shot](https://user-images.githubusercontent.com/19557880/185509412-ab64d92d-2a56-4d5a-bbb4-35a41a2ca48c.png) +2. The command will write the URL to the pull request that was created when the command succeeds. + ![img](https://user-images.githubusercontent.com/19557880/185509412-ab64d92d-2a56-4d5a-bbb4-35a41a2ca48c.png) -## Reviewing the pull request +3. Open the generated pull request in a new browser tab. -- Let's first look at the `Conversation` tab of the PR. It tells us we have a manual task to perform before the GitHub Actions workflow is functional. We need to add a secret. We can use the GitHub [documentation](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) for secrets and add a `actions` secret for `EXPRESSION_FIRST_VAR` with any value. +### Inspect the pull request - ![Manual steps](https://user-images.githubusercontent.com/19557880/186784161-b7882ac4-ac99-4462-b69f-f49b9202527b.png) +The first thing we should notice about the PR is that there is a list of manual steps for us to complete: -- Next, let's review the workflow we are adding by clicking on `Files changed` tab. This is where you would double check everything looks good. If it didn't you could push commits with the required changes, prior to merging. +![img](https://user-images.githubusercontent.com/19557880/186784161-b7882ac4-ac99-4462-b69f-f49b9202527b.png) -- Now our review is completed we want to go back to the `Conversation` tab and click `Merge pull request` +Next, let's review the workflow we are adding by clicking on `Files changed` tab. This is where you would double check everything looks good. If it didn't you could push commits with the required changes, prior to merging. -- Once the PR is merged the new workflow should start and we can view it by clicking `Actions` in the top navigation - actions-screen-shot -- The migration is complete and we have successfully transformed a pipeline from Jenkins into a GitHub Actions workflow, added it to a GitHub Repository, and run the workflow in Actions. +Next, you can inspect the "Files changed" in this PR and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this PR. -### Next Lab -This concludes the Valet labs for Jenkins! If you are interested exploring the power of Valet more, you can leverage the demo Jenkins Instance and modify and add new projects that more closely match your needs and try out the commands again! +Finally, you can merge the PR once your review has completed. We can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. + +![img](https://user-images.githubusercontent.com/19557880/185509704-90243ec5-e77f-4baf-a9b2-d9a4d9fda199.png) + +At this point, the migration has completed and you have successfully migrated a Jenkins pipeline to Actions! + +### Next lab + +[Forecast potential build runner usage](6-forecast.md) diff --git a/jenkins/6-forecast.md b/jenkins/6-forecast.md index 2595015..d531ece 100644 --- a/jenkins/6-forecast.md +++ b/jenkins/6-forecast.md @@ -178,3 +178,5 @@ Using `--source-file-path` we can combine data from multiple forecast runs into ## Next steps [Migrating a Jenkins Pipeline](../jenkins/6-migrate.md#migrate-a-jenkins-project-to-github-actions) + +This concludes the Valet labs for Jenkins! If you are interested exploring the power of Valet more, you can leverage the demo Jenkins Instance and modify and add new projects that more closely match your needs and try out the commands again! \ No newline at end of file From 17de43ecac3aeeb87fea52b4077d37d7e62a1ef7 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 12:47:23 -0700 Subject: [PATCH 19/20] Edit forecast labs --- jenkins/5-migrate.md | 5 +- jenkins/6-forecast.md | 143 ++++++++++++++++++++---------------------- 2 files changed, 71 insertions(+), 77 deletions(-) diff --git a/jenkins/5-migrate.md b/jenkins/5-migrate.md index 9b69d3e..a8ebba2 100644 --- a/jenkins/5-migrate.md +++ b/jenkins/5-migrate.md @@ -9,7 +9,7 @@ In this lab, you will use the `migrate` command to convert a Jenkins pipeline an 3. Completed the [dry-run lab](./3-dry-run.md). 4. Completed the [custom transformers lab](./4-custom-transformers.md). -## Preparing for migration +## Performing a migration We need to answer the following questions before running a `migrate` command: @@ -20,7 +20,7 @@ We need to answer the following questions before running a `migrate` command: 3. What is the URL for the GitHub repository to add the workflow to? - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. -## Performing a migration +### Steps 1. Run the following `migrate` command in the codespace terminal: @@ -29,6 +29,7 @@ We need to answer the following questions before running a `migrate` command: ``` 2. The command will write the URL to the pull request that was created when the command succeeds. + ![img](https://user-images.githubusercontent.com/19557880/185509412-ab64d92d-2a56-4d5a-bbb4-35a41a2ca48c.png) 3. Open the generated pull request in a new browser tab. diff --git a/jenkins/6-forecast.md b/jenkins/6-forecast.md index d531ece..67a6294 100644 --- a/jenkins/6-forecast.md +++ b/jenkins/6-forecast.md @@ -1,58 +1,48 @@ # Forecast the runner usage of a Jenkins instance -In this lab we will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from the historical pipeline data in our Jenkins instance. The metrics will be stored on disk in a markdown file and include job metrics for execution time, queue time, and concurrency. We will look at each of these metrics in more depth later in this lab. - -- [Prerequisites](#prerequisites) -- [Prepare for forecast](#prepare-for-forecast) -- [Perform a forecast](#perform-a-forecast) -- [Review forecast report](#review-forecast-report) -- [Forecasting multiple providers](#forecasting-multiple-providers) -- [Next steps](#next-steps) +In this lab you will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from completed pipeline runs in your Jenkins server. ## Prerequisites -1. Followed [steps](../jenkins/readme.md#valet-labs-for-jenkins) to set up your codespace environment and start your Jenkins instance. -2. Completed the [configure lab](../jenkins/valet-configure-lab.md#configure-valet-to-work-with-jenkins) to configure the Valet CLI. - -## Prepare for forecast - -Before we can run the forecast we need to answer a few questions so we can construct the correct command. - -1. Do we want to forecast the entire Jenkins instance, or just a single folder? - -- In this example we will be auditing the entire Jenkins instance, but in the future if you wanted to configure a specific folder to be audited add the `-f ` flag to the forecast command. - -2. What is the date we want to start forecasting from? - -- __2022-08-02__. This date is before the time the data was populated in the Jenkins server running in these labs. This value defaults to the date one week ago, however, you should ensure a date is used that will capture enough data to get a representative view of typical usage. - -3. Where do we want to store the results? - -- `./tmp/forecast_reports`. This can be any valid path on the system, but for simplicity it is recommend to use a directory in the root of the workspace. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a Jenkins server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). ## Perform a forecast -- Using the answers above we get the following `forecast` command: +We will need to answer the following questions before running the `forecast` command: -``` -gh valet forecast jenkins --output-dir ./tmp/forecast_reports --start-date 2022-08-02 -``` +1. Do we want to forecast the entire Jenkins server or a single folder? + - We will be forecasting the entire Jenkins server but we could optionally limit this by using the `-f ` CLI option. -- Run the command in the codespace terminal. -- Verify that the command output is similar to this. - ![forecast_output](https://user-images.githubusercontent.com/19557880/186223037-18556c82-5a29-4434-bc17-4b906d704967.png) +2. What is the date we want to start forecasting from? + - __2022-08-02__. This date is needed as it is prior to when the data was seeded in Jenkins for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. -## Review forecast report +3. Where do we want to store the results? + - `./tmp/forecast_reports` -Open the forecast report and review the calculated metrics. +### Steps -- From the codespace explorer pane find `./tmp/forecast_reports/forecast_report.md` and right-click, and select __Open Preview__. +1. Navigate to the codespace terminal +2. Run the following command from the root directory: - ![forecast_explorer](https://user-images.githubusercontent.com/18723510/185234641-948a551b-316f-4cce-9e7d-4c078ae11a04.png) + ```bash + gh valet forecast jenkins --output-dir ./tmp/forecast_reports --start-date 2022-08-02 + ``` -- The file should be similar to this. +3. The command will list all the files written to disk when the command succeeds. -
+ ![img](https://user-images.githubusercontent.com/19557880/186223037-18556c82-5a29-4434-bc17-4b906d704967.png) + +## Review the forecast report + +The forecast report, logs, and completed job data will be located within the `tmp/forecast_reports` folder. + +1. Find the `forecast_report.md` file in the file explorer. +2. Right-click the `forecast_report.md` file and select `Open Preview`. +3. This file contains metrics used to forecast potential GitHub Actions usage. + + + -### Metric Definitions -| Name | Description | -| ----- | ----------- | -| Median | The __middle__ value | -| P90 | 90% of the values are less than or equal to | -| Min | The lowest value | -| Max | The highest value | +### Total -### Total Section - -- This section shows the metrics for all of the jobs run within the Jenkins instance from 08/02/2022 to the time the command was executed. - -## Total +The "Total" section of the forecast report contains high level statistics related to all the jobs completed after the `--start-date` CLI option: +```md - Job count: __73__ - Pipeline count: __6__ - --- +- Execution time -We can see there were 73 completed jobs across 6 unique pipelines. A pipeline can have one or more jobs and a pipeline may be executed multiple times in the date range included in the forecast. - - For example `monas_freestyle` contains 1 job. - ![demo_pipeline](https://user-images.githubusercontent.com/19557880/186261368-d4dbbe8d-71e0-4084-bbbb-7557e9dbbb86.png) + - Total: __27,057 minutes__ + - Median: __2 minutes__ + - P90: __19 minutes__ + - Min: __0 minutes__ + - Max: __15,625 minutes__ + +- Queue time + + - Median: __0 minutes__ + - P90: __0 minutes__ + - Min: __0 minutes__ + - Max: __0 minutes__ + +- Concurrent jobs + + - Median: __1__ + - P90: __3__ + - Min: __0__ + - Max: __29__ +``` Here are some key terms of items defined in the forecast report: - The `job count` is the total number of completed jobs. - The `pipeline count` is the number of unique pipelines used. -- `Execution time` describes the amount of time a runner spent on a job. This metric can be used for a customer to help set expectations for the cost of GitHub hosted runners. - - This metric is correlated to the amount of spend a customer should expect in GitHub Actions. This will vary greatly depending on the hardware the customer uses for these minutes and the Actions pricing calculator should be used to get an estimate of the approximate spend the customer should expect. - - Looking closer we can see during our forecast timeframe the total job run time was 27,057 minutes with 90% of the jobs finishing under 20 minutes, and the longest job taking 15,625 minutes. The `min` is 0 because the quickest job took less than a minute and was rounded down to 0. - +- `Execution time` describes the amount of time a runner spent on a job. This metric can be used to help plan for the cost of GitHub hosted runners. + - This metric is correlated to how much you should expect to spend in GitHub Actions. This will vary depending on the hardware used for these minutes and the [Actions pricing calculator](https://github.com/pricing/calculator) should be used to estimate a dollar amount. - `Queue time` metrics describe the amount of time a job spent waiting for a runner to be available to execute it. - `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure. -Additionally, these metrics are defined for each queue of runners that a customer has defined in the CI/CD platform. This is especially useful for customers that use a mix of hosted and self-hosted runners to see runner utilization metrics that are specific to different types of runners. - -### Runner Group Sections - -- The preceding section shows the same metrics as the `Total` section, but are grouped by runner group. A runner group is a machine (or group of machines) that each job runs on -- In this case we do not have any runner groups, so the metrics match under `N/A` match the `Total` section. If there were different groups we could possibly identify runner types that needed to be increased or decreased when moving to GitHub Actions. +Additionally, these metrics are defined for each queue of runners defined in Jenkins. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. ## Forecasting multiple providers -If we examine the help for the `forecast` command by running `gh valet forecast --help` we can see the option: `--source-file-path` +We can examine the available options for the `forecast` command by running `gh valet forecast --help`. When you do this you will see the `--source-file-path` option: -![forecast-help](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) +![img](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) -Using `--source-file-path` we can combine data from multiple forecast runs into a single report. This becomes useful if we are using multiple CI/CD providers, such as Azure DevOps and Jenkins, and wanted to get a holistic view of the runner usage across the providers. The way this works is the forecast command creates a `.json` file in a `jobs` directory for each command execution. The `--source-file-path` takes a space-delimited list of data file paths or a glob pattern that will match all of the data files we want to include and combine into a single report. We will use a glob pattern, which in general should match `OUTPUT_DIR/**/jobs/*.json` where the `OUTPUT_DIR` is the previous value used for `--output-dir`, which in this lab was `./tmp/forecast_reports`. We do not have multiple providers but we can still try it out because we have a data file at `tmp/forecast_reports/jobs/`! +The `--source-file-path` CLI option can be used to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and wanted to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). -- run `gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast` -- Now we have a new report that was generated from all the data files that matched the glob pattern. Note this command does not introspect the CI/CD provider, it only operates on the data files it finds. -![combined-report](https://user-images.githubusercontent.com/19557880/186264213-b3201710-8093-4ae5-9aef-5c7f95cc3951.png) +Run the following command from within the codespace terminal: + +```bash +gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast +``` + +You can now inspect the output of the command to see a forecast report using all of the files matching the `tmp/**/jobs/*.json` pattern. ## Next steps -[Migrating a Jenkins Pipeline](../jenkins/6-migrate.md#migrate-a-jenkins-project-to-github-actions) -This concludes the Valet labs for Jenkins! If you are interested exploring the power of Valet more, you can leverage the demo Jenkins Instance and modify and add new projects that more closely match your needs and try out the commands again! \ No newline at end of file +This concludes all labs for migrating Jenkins pipelines to Actions with Valet! \ No newline at end of file From 0b3f64ae5da51a290209912e878f447017fea7ac Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 6 Sep 2022 13:38:24 -0700 Subject: [PATCH 20/20] Update inline md --- jenkins/4-custom-transformers.md | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index 87d48de..244e7a7 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -37,7 +37,7 @@ env: DISABLE_AUTH: 'true' DB_ENGINE: sqlite jobs: - check_env_variables: + build: runs-on: - self-hosted - TeamARunner @@ -54,22 +54,6 @@ jobs: # value: 80 - name: echo message run: echo "DISABLE_AUTH is ${{ env.DISABLE_AUTH }}" - build: - runs-on: - - self-hosted - - TeamARunner - needs: check_env_variables - steps: - - name: checkout - uses: actions/checkout@v2 - - name: Set up JDK 1.11 - uses: actions/setup-java@v1 - with: - java-version: '1.11' - settings-path: "${{ github.workspace }}" - - name: sh - shell: bash - run: mvn clean package test: runs-on: - self-hosted @@ -87,8 +71,6 @@ jobs:
- - _Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step