From 5f6ce0003e39d2d6cb6c8d16ef78a4be653662f0 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Wed, 7 Sep 2022 09:36:46 -0400 Subject: [PATCH 01/59] add readme for circleCI setup --- circle_ci/readme.md | 75 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 circle_ci/readme.md diff --git a/circle_ci/readme.md b/circle_ci/readme.md new file mode 100644 index 0000000..d99995a --- /dev/null +++ b/circle_ci/readme.md @@ -0,0 +1,75 @@ +# CircleCI to Actions migrations powered by Valet + +The instructions below will guide you through configuring a GitHub Codespace environment that will be used in subsequent labs that demonstrate how to use Valet to migrate CircleCI pipelines to GitHub Actions. + +These steps **must** be completed prior to starting other labs. + +## Create your own repository for these labs + +1. Ensure that you have created a repository using the [valet-customers/labs](https://github.com/valet-customers/labs) as a template. + +## Configure your Codespace + +1. Start a new Codespace. + +- Click the `Code` with button down arrow above repository on the repository's landing page. +- Click the `Codespaces` tab +- Click `Create codespaces on main` to create the codespace. +- After the Codespace has initialized there will be a terminal present. + +2. Verify the Valet CLI is installed and working. More information on the Valet extension for the official GitHub CLI can be found [here](https://github.com/github/gh-valet). + +- Run the following command in the codespace's terminal: + + ```bash + gh valet version + ``` + +- Verify the output is similar to below. The version information may differ than what is shown below, it is expected that the `valet-cli` version is unknown. This will be corrected in the [configure credentials for Valet](1-configure.md) lab. + + ```bash + gh version 2.14.3 (2022-07-26) + gh valet github/gh-valet v0.1.12 + valet-cli unknown + ``` + + - If `gh valet version` did not produce similar output then please follow the troubleshooting [guide](#troubleshoot-the-valet-cli). + + + +## Labs for CircleCI + +Perform the following labs to test-drive Valet + +1. [Configure credentials for Valet](1-configure.md) +2. [Perform an audit of CircleCI](2-audit.md) +3. [Perform a dry-run of a CircleCI pipeline](3-dry-run.md) +4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) +5. [Perform a production migration of a CircleCI pipeline](5-migrate.md) +6. [Forecast potential build runner usage](6-forecast.md) + +## Troubleshoot the Valet CLI + +The CLI extension for Valet can be manually installed by following these steps: + +- Verify you are in the codespace terminal +- Run this command from within the codespace's terminal: + + ```bash + gh extension install github/gh-valet + ``` + +- Verify the result of the install contains: + + ```bash + ✓ Installed extension github/gh-valet + ``` + +- If you get an error similar to the image below, then click the link in the terminal output to authorize the token. + - Restart the codespace after clicking the link. + ![img](https://user-images.githubusercontent.com/26442605/169588015-9414404f-82b6-4d0f-89d4-5f0e6941b029.png) +- Verify Valet CLI extension is installed and working by running the following command from the codespace's terminal: + + ```bash + gh valet version + ``` From 146283ce12bc168e7ee99ab51906dd7394da153a Mon Sep 17 00:00:00 2001 From: j-dunham Date: Wed, 7 Sep 2022 11:12:37 -0400 Subject: [PATCH 02/59] Update circle_ci/readme.md Co-authored-by: Ethan Dennis --- circle_ci/readme.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/circle_ci/readme.md b/circle_ci/readme.md index d99995a..e2cc56a 100644 --- a/circle_ci/readme.md +++ b/circle_ci/readme.md @@ -35,8 +35,6 @@ These steps **must** be completed prior to starting other labs. - If `gh valet version` did not produce similar output then please follow the troubleshooting [guide](#troubleshoot-the-valet-cli). - - ## Labs for CircleCI Perform the following labs to test-drive Valet From 6008a7a272d5ef3fc1765d6bda3473329d933a0f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Wed, 7 Sep 2022 11:26:38 -0400 Subject: [PATCH 03/59] Create 1-configure.md --- circle-ci/1-configure.md | 48 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 circle-ci/1-configure.md diff --git a/circle-ci/1-configure.md b/circle-ci/1-configure.md new file mode 100644 index 0000000..ac1515c --- /dev/null +++ b/circle-ci/1-configure.md @@ -0,0 +1,48 @@ +# Configure credentials for Valet + +In this lab, you will use the `configure` CLI command to set the required credentials and information for Valet to use when working with CircleCI and GitHub. + +You will need to complete all of the setup instructions [here](./readme.md#configure-your-codespace) prior to performing this lab. + +## Configuring credentials + +1. Create a GitHub Personal Access Token (PAT): + - Open github.com in a new browser tab. + - Click your profile photo in the top right of the UI and click `Settings`. + - Click on `Developer Settings` in the left hand panel. + - Click `Personal Access Tokens` and then `Legacy tokens` (if present). + - Click `Generate new token` and then `Legacy tokens`. You may be required to authenticate with GitHub during this step. + - Select the following scopes: `read:packages` and `workflow`. + - Click `Generate token`. + - Copy the generated PAT and save in a safe location. + +3. Create a CircleCI personal API token using CircleCI's [documentation](https://circleci.com/docs/managing-api-tokens#creating-a-personal-api-token) + +2. Run the `configure` CLI command: + ADD-DETAILS-HERE + + ADD-IMAGE-HERE + +## Verify your environment + +To verify our environment is configured correctly, we are going to run the `update` CLI command. The `update` CLI command will download the latest version of Valet to your codespace. + +1. In the codespace terminal run the following command: + + ```bash + gh valet update + ``` + +2. You should see a confirmation that you were logged into the GitHub Container Registry and Valet was updated to the latest version. + + ```bash + Login Succeeded + latest: Pulling from valet-customers/valet-cli + Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb + Status: Image is up to date for ghcr.io/valet-customers/valet-cli:latest + ghcr.io/valet-customers/valet-cli:latest + ``` + +### Next lab + +[Perform an audit of CircleCI](./2-audit.md) From 21d5161220a5eaa4f6fce017d593b366ad411833 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Wed, 7 Sep 2022 12:59:55 -0400 Subject: [PATCH 04/59] Update 1-configure.md --- circle-ci/1-configure.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/circle-ci/1-configure.md b/circle-ci/1-configure.md index ac1515c..d95acc7 100644 --- a/circle-ci/1-configure.md +++ b/circle-ci/1-configure.md @@ -19,9 +19,20 @@ You will need to complete all of the setup instructions [here](./readme.md#confi 3. Create a CircleCI personal API token using CircleCI's [documentation](https://circleci.com/docs/managing-api-tokens#creating-a-personal-api-token) 2. Run the `configure` CLI command: - ADD-DETAILS-HERE + - Select the `TERMINAL` tab from within the codespace terminal window. + - Run the following command: `gh valet configure`. + - Using the down arrow key to highlight `CircleCI`, press the spacebar to select, and then hit enter to continue. + - At the prompt enter your GitHub Username and press enter. + - At the GitHub Container Registry prompt enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub PAT prompt enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). + - At the CircleCI token prompt enter the CircleCI access token from step 2 and press enter. + - At the CircleCI base url prompt hit enter to accept the default value (`https://circleci.com`). + - At the access token to fetch source code in GitHub prompt enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub instance url containing source code prompt press enter to accept the default value (`https://github.com`). + + ![configure-screenshot](https://user-images.githubusercontent.com/18723510/188935436-308fa6d3-6eb7-48b9-bbf0-20ab0cdae411.png) - ADD-IMAGE-HERE ## Verify your environment From e179dc7fc58f0b1154d34b7078961c34f2847886 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Wed, 7 Sep 2022 19:53:17 -0400 Subject: [PATCH 05/59] Update circle-ci/1-configure.md Co-authored-by: Ethan Dennis --- circle-ci/1-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle-ci/1-configure.md b/circle-ci/1-configure.md index d95acc7..e2bf147 100644 --- a/circle-ci/1-configure.md +++ b/circle-ci/1-configure.md @@ -16,7 +16,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Click `Generate token`. - Copy the generated PAT and save in a safe location. -3. Create a CircleCI personal API token using CircleCI's [documentation](https://circleci.com/docs/managing-api-tokens#creating-a-personal-api-token) +3. Create a CircleCI personal API token using CircleCI's [documentation](https://circleci.com/docs/managing-api-tokens#creating-a-personal-api-token) and store the token in a safe location. 2. Run the `configure` CLI command: - Select the `TERMINAL` tab from within the codespace terminal window. From c546aca77811f81f1cebd9c0ea73d9025fb49bc5 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 8 Sep 2022 10:04:16 -0400 Subject: [PATCH 06/59] Rename circle-ci/1-configure.md to circle_ci/1-configure.md --- {circle-ci => circle_ci}/1-configure.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {circle-ci => circle_ci}/1-configure.md (100%) diff --git a/circle-ci/1-configure.md b/circle_ci/1-configure.md similarity index 100% rename from circle-ci/1-configure.md rename to circle_ci/1-configure.md From 7e8b3d2d52c8c25726b49bc0ee333d9bd12eccd5 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 8 Sep 2022 10:42:28 -0400 Subject: [PATCH 07/59] Create 2-audit.md --- circle_ci/2-audit.md | 108 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 circle_ci/2-audit.md diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md new file mode 100644 index 0000000..3e36a17 --- /dev/null +++ b/circle_ci/2-audit.md @@ -0,0 +1,108 @@ +# Perform an audit of CircleCI + +In this lab, you will use the `audit` command to get a high-level view of all projects in CircleCI. + +The `audit` command operates by fetching all of the projects defined in CircleCI , converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment. +2. Completed the [configure lab](./1-configure.md#configure-credentials-for-valet). + +## Perform an audit + +We will be performing an audit against a organization in CircleCI created for the purpose of this lab. The name of this organization was already set during the configure lab. The only question that remains to be answered is: + +1. Where do we want to store the result? + - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. + +### Steps + +1. Navigate to the codespace terminal. +2. Run the following command from the root directory: + + ```bash + gh valet audit circle-ci --output-dir tmp/audit + ``` + +3. The command will list all the files written to disk in green when the command succeeds. + +## Inspect the output files + +1. Find the `audit_summary.md` file in the file explorer. +2. Right-click the `audit_summary.md` file and select `Open Preview`. +3. This file contains details that summarizes what percentage of your pipelines were converted automatically. + +### Review audit summary + +#### Pipelines + +The pipeline summary section contains high level statistics regarding the conversion rate done by Valet: + +```md +UPDATE_ME +``` + +Here are some key terms in the “Pipelines” section in the above example: + +- __Successful__ pipelines had 100% of the pipeline constructs and individual items converted automatically to their GitHub Actions equivalent. +- __Partially successful__ pipelines had all of the pipeline constructs converted, however, there were some individual items that were not converted automatically to their GitHub Actions equivalent. +- __Failed pipelines__ encountered a fatal error when being converted. This can occur for one of three reasons: + - The pipeline was misconfigured and not valid in CircleCI. + - Valet encountered an internal error when converting it. + - There was an unsuccessful network response, often due to invalid credentials, that caused the pipeline to be inaccessible. + +The "Job types" section will summarize which types of pipelines are being used and which are supported or unsupported by Valet. + +#### Build steps + +The build steps summary section presents an overview of the individual build steps that are used across all pipelines and how many were automatically converted by Valet. + +```md +UPDATE_ME +``` + +Here are some key terms in the "Build steps" section in the above example: + +- A __known__ build step is a step that was automatically converted to an equivalent action. +- An __unknown__ build step is a step that was not automatically converted to an equivalent action. +- An __unsupported__ build step is a step that is either: + - A step that is fundamentally not supported by GitHub Actions. + - A step that is configured in a way that is incompatible with GitHub Actions. +- An __action__ is a list of the actions that were used in the converted workflows. This is important for the following scenarios: + - Gathering the list of actions to sync to your appliance if you use GitHub Enterprise Server. + - Defining an organization-level allowlist of actions that can be used. This list of actions is a comprehensive list of which actions their security and/or compliance teams will need to review. + +There is an equivalent breakdown of build triggers, environment variables, and other uncategorized items displayed in the audit summary file. + +#### Manual Tasks + +The manual tasks summary section presents an overview of the manual tasks that you will need to perform that Valet is not able to complete automatically. + +```md +UPDATE_ME and update pipeline to add this section +``` + +Here are some key terms in the “Manual tasks” section in the above example: + +- A __secret__ refers to a repository or organization level secret that is used by the converted pipelines. These secrets will need to be created manually in Actions in order for these pipelines to function properly. +- A __self-hosted runner__ refers to a label of a runner that is referenced by a converted pipeline that is not a GitHub-hosted runner. You will need to manually define these runners in order for these pipelines to function properly. + +#### Files + +The final section of the audit report provides a manifest of all of the files that are written to disk during the audit. These files include: + +```md +UPDATE_ME +``` + +Each pipeline will have a variety of files written that include: + +- The original pipeline as it was defined in GitHub. +- Any network responses used to convert a pipeline. +- The converted workflow. +- Stack traces that can used to troubleshoot a failed pipeline conversion + +### Next lab + +[Perform a dry-run of a GitLab pipeline](3-dry-run.md) From d4645af5ae5c19707772fc3e05dbc51951093269 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 8 Sep 2022 11:03:07 -0400 Subject: [PATCH 08/59] Update 1-configure.md --- circle_ci/1-configure.md | 1 + 1 file changed, 1 insertion(+) diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index e2bf147..6f7b768 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -28,6 +28,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). - At the CircleCI token prompt enter the CircleCI access token from step 2 and press enter. - At the CircleCI base url prompt hit enter to accept the default value (`https://circleci.com`). + - At the CircleCI organization name prompt enter `labs-data`. This is a preconfigured organization created just for the purpose of these labs. - At the access token to fetch source code in GitHub prompt enter the GitHub PAT generated in step 1 and press enter. - At the GitHub instance url containing source code prompt press enter to accept the default value (`https://github.com`). From 1cfb227fb25ab80e3bb387c4d47c209b94ff45df Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 8 Sep 2022 16:44:19 -0400 Subject: [PATCH 09/59] Update 2-audit.md --- circle_ci/2-audit.md | 99 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 7 deletions(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 3e36a17..9e8d7b4 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -1,8 +1,8 @@ # Perform an audit of CircleCI -In this lab, you will use the `audit` command to get a high-level view of all projects in CircleCI. +In this lab, you will use the `audit` command to get a high-level view of all projects in a CircleCI organization. -The `audit` command operates by fetching all of the projects defined in CircleCI , converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command operates by fetching all of the projects defined in CircleCI organization, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites @@ -11,7 +11,7 @@ The `audit` command operates by fetching all of the projects defined in CircleCI ## Perform an audit -We will be performing an audit against a organization in CircleCI created for the purpose of this lab. The name of this organization was already set during the configure lab. The only question that remains to be answered is: +We will be performing an audit against a organization in CircleCI that was created for the purposes of this lab *labs-data*. Valet was already set to use this organization during the configure lab. The only remaining information needed for the `audit` command is: 1. Where do we want to store the result? - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. @@ -40,7 +40,21 @@ We will be performing an audit against a organization in CircleCI created for th The pipeline summary section contains high level statistics regarding the conversion rate done by Valet: ```md -UPDATE_ME +## Pipelines + +Total: **6** + +- Successful: **6 (100%)** +- Partially successful: **0 (0%)** +- Unsupported: **0 (0%)** +- Failed: **0 (0%)** + +### Job types + +Supported: **6 (100%)** + +- 2.1: **5** +- 2: **1** ``` Here are some key terms in the “Pipelines” section in the above example: @@ -59,7 +73,35 @@ The "Job types" section will summarize which types of pipelines are being used a The build steps summary section presents an overview of the individual build steps that are used across all pipelines and how many were automatically converted by Valet. ```md -UPDATE_ME +### Build steps + +Total: **42** + +Known: **42 (100%)** + +- run: **10** +- checkout: **6** +- circleci_node_install_packages: **6** +- circleci_node_job_test: **5** +- store_artifacts: **3** +- circleci_python_install_packages: **3** +- circleci_ruby_install_deps: **2** +- restore_cache: **2** +- attach_workspace: **1** +- persist_to_workspace: **1** +- store_test_results: **1** +- circleci_ruby_rspec_test: **1** +- circleci_ruby_rubocop_check: **1** + +Actions: **45** + +- run: **23** +- actions/checkout@v2: **7** +- actions/cache@v2: **6** +- actions/upload-artifact@v2: **5** +- ruby/setup-ruby@v1: **2** +- actions/download-artifact@v2: **1** +- ./.github/actions/greeting: **1** ``` Here are some key terms in the "Build steps" section in the above example: @@ -80,7 +122,13 @@ There is an equivalent breakdown of build triggers, environment variables, and o The manual tasks summary section presents an overview of the manual tasks that you will need to perform that Valet is not able to complete automatically. ```md -UPDATE_ME and update pipeline to add this section +### Manual tasks + +Total: **1** + +Self hosted runners: **1** + +- `medium+`: **1** ``` Here are some key terms in the “Manual tasks” section in the above example: @@ -93,7 +141,44 @@ Here are some key terms in the “Manual tasks” section in the above example: The final section of the audit report provides a manifest of all of the files that are written to disk during the audit. These files include: ```md -UPDATE_ME +### Successful + +#### labs-data/circleci-hello-world + +- [labs-data/circleci-hello-world/say-hello-workflow.yml](labs-data/circleci-hello-world/say-hello-workflow.yml) +- [labs-data/circleci-hello-world.config.json](labs-data/circleci-hello-world.config.json) +- [labs-data/circleci-hello-world.source.yml](labs-data/circleci-hello-world.source.yml) + +#### labs-data/circleci-command-example + +- [labs-data/circleci-command-example/my-workflow.yml](labs-data/circleci-command-example/my-workflow.yml) +- [.github/actions/greeting/action.yml](.github/actions/greeting/action.yml) +- [labs-data/circleci-command-example.config.json](labs-data/circleci-command-example.config.json) +- [labs-data/circleci-command-example.source.yml](labs-data/circleci-command-example.source.yml) + +#### labs-data/circleci-node-example + +- [labs-data/circleci-node-example/sample.yml](labs-data/circleci-node-example/sample.yml) +- [labs-data/circleci-node-example.config.json](labs-data/circleci-node-example.config.json) +- [labs-data/circleci-node-example.source.yml](labs-data/circleci-node-example.source.yml) + +#### labs-data/circleci-python-example + +- [labs-data/circleci-python-example/sample.yml](labs-data/circleci-python-example/sample.yml) +- [labs-data/circleci-python-example.config.json](labs-data/circleci-python-example.config.json) +- [labs-data/circleci-python-example.source.yml](labs-data/circleci-python-example.source.yml) + +#### labs-data/circleci-demo-java-spring + +- [labs-data/circleci-demo-java-spring/workflow.yml](labs-data/circleci-demo-java-spring/workflow.yml) +- [labs-data/circleci-demo-java-spring.config.json](labs-data/circleci-demo-java-spring.config.json) +- [labs-data/circleci-demo-java-spring.source.yml](labs-data/circleci-demo-java-spring.source.yml) + +#### labs-data/circleci-demo-ruby-rails + +- [labs-data/circleci-demo-ruby-rails/build_and_test.yml](labs-data/circleci-demo-ruby-rails/build_and_test.yml) +- [labs-data/circleci-demo-ruby-rails.config.json](labs-data/circleci-demo-ruby-rails.config.json) +- [labs-data/circleci-demo-ruby-rails.source.yml](labs-data/circleci-demo-ruby-rails.source.yml) ``` Each pipeline will have a variety of files written that include: From c2e584abead6e10c0e77dd139f852cd6237b0b0b Mon Sep 17 00:00:00 2001 From: j-dunham Date: Thu, 8 Sep 2022 16:56:39 -0400 Subject: [PATCH 10/59] Update 2-audit.md --- circle_ci/2-audit.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 9e8d7b4..9fb1ee7 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -2,7 +2,7 @@ In this lab, you will use the `audit` command to get a high-level view of all projects in a CircleCI organization. -The `audit` command operates by fetching all of the projects defined in CircleCI organization, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command operates by fetching all of the projects defined in a CircleCI organization, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites @@ -11,7 +11,7 @@ The `audit` command operates by fetching all of the projects defined in CircleCI ## Perform an audit -We will be performing an audit against a organization in CircleCI that was created for the purposes of this lab *labs-data*. Valet was already set to use this organization during the configure lab. The only remaining information needed for the `audit` command is: +We will be performing an audit against the **labs-data** organization in CircleCI, which was created for the purposes of this lab. Valet was already been set to use this organization during the configure lab. The only remaining information needed for the `audit` command is: 1. Where do we want to store the result? - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. @@ -57,7 +57,7 @@ Supported: **6 (100%)** - 2: **1** ``` -Here are some key terms in the “Pipelines” section in the above example: +Here are some key terms that can appear in the “Pipelines” section: - __Successful__ pipelines had 100% of the pipeline constructs and individual items converted automatically to their GitHub Actions equivalent. - __Partially successful__ pipelines had all of the pipeline constructs converted, however, there were some individual items that were not converted automatically to their GitHub Actions equivalent. @@ -104,7 +104,7 @@ Actions: **45** - ./.github/actions/greeting: **1** ``` -Here are some key terms in the "Build steps" section in the above example: +Here are some key terms that can appear in "Build steps" section: - A __known__ build step is a step that was automatically converted to an equivalent action. - An __unknown__ build step is a step that was not automatically converted to an equivalent action. @@ -131,7 +131,7 @@ Self hosted runners: **1** - `medium+`: **1** ``` -Here are some key terms in the “Manual tasks” section in the above example: +Here are some key terms that can appear in “Manual tasks” section: - A __secret__ refers to a repository or organization level secret that is used by the converted pipelines. These secrets will need to be created manually in Actions in order for these pipelines to function properly. - A __self-hosted runner__ refers to a label of a runner that is referenced by a converted pipeline that is not a GitHub-hosted runner. You will need to manually define these runners in order for these pipelines to function properly. From dfd2b68fe97535aebe145acf3aa929be40307564 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Fri, 9 Sep 2022 12:29:56 -0400 Subject: [PATCH 11/59] Update circle_ci/1-configure.md Co-authored-by: Ethan Dennis --- circle_ci/1-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index 6f7b768..8cdc30b 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -28,7 +28,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). - At the CircleCI token prompt enter the CircleCI access token from step 2 and press enter. - At the CircleCI base url prompt hit enter to accept the default value (`https://circleci.com`). - - At the CircleCI organization name prompt enter `labs-data`. This is a preconfigured organization created just for the purpose of these labs. + - At the CircleCI organization name prompt enter `labs-data`. This is the organization we'll be using throughout these labs. - At the access token to fetch source code in GitHub prompt enter the GitHub PAT generated in step 1 and press enter. - At the GitHub instance url containing source code prompt press enter to accept the default value (`https://github.com`). From 885c5665746dc94fdfa8a096cd94cb5a2c89673f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:37:57 -0400 Subject: [PATCH 12/59] Create 3-dry-run.md --- circle_ci/3-dry-run.md | 63 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 circle_ci/3-dry-run.md diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md new file mode 100644 index 0000000..8d3c410 --- /dev/null +++ b/circle_ci/3-dry-run.md @@ -0,0 +1,63 @@ +# Perform a dry-run of a CircleCI pipeline + +In this lab you will use the `dry-run` command to convert a CircleCI pipeline to its equivalent GitHub Actions workflow. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [audit lab](./2-audit.md). + +## Perform a dry run + +We will be performing a dry-run against a CircleCI project. We will need to answer the following questions before running this command: + +1. What is the name of the project we want to convert? + - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. + +2. Where do we want to store the result? + - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. + +### Steps + +1. Navigate to the codespace terminal +2. Run the following command from the root directory: + + ```bash + gh valet dry-run circle-ci --output-dir ./tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails + ``` + +3. The command will list all the files written to disk when the command succeeds. + + ADD_IMAGE_HERE + +4. View the converted workflow: + - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in codespaces. + - Click `build_and_test.yml` to open. + +## Inspect the output files + +The files generated from the `dry-run` command represent the equivalent Actions workflow for the CircleCI project. The CircleCI configuration and converted workflow can be seen below: + +
+ CircleCI configuration 👇 + +```yaml +# ADD YAML +``` + +
+ +
+ Converted workflow 👇 + +```yaml +# ADD YAML +``` +
+ +Despite these 2 pipelines using different syntax they will function equivalently. + +## Next lab + +[Use custom transformers to customize Valet's behavior](./4-custom-transformers.md) From a3ac50c62535bf16612159aa19e745fb0204f252 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:38:16 -0400 Subject: [PATCH 13/59] Update 2-audit.md --- circle_ci/2-audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 9fb1ee7..a102e6c 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -190,4 +190,4 @@ Each pipeline will have a variety of files written that include: ### Next lab -[Perform a dry-run of a GitLab pipeline](3-dry-run.md) +[Perform a dry-run of a CircleCI pipeline](3-dry-run.md) From 070a6d691020ab8b608f1271285aa2edeb5b8dd0 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:48:13 -0400 Subject: [PATCH 14/59] Update 3-dry-run.md --- circle_ci/3-dry-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md index 8d3c410..efb6423 100644 --- a/circle_ci/3-dry-run.md +++ b/circle_ci/3-dry-run.md @@ -5,7 +5,7 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [audit lab](./2-audit.md). ## Perform a dry run From 3a6c88ba20bc27e36016e2ca7bfda655aa387f83 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 09:22:22 -0400 Subject: [PATCH 15/59] Update 3-dry-run.md --- circle_ci/3-dry-run.md | 150 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 140 insertions(+), 10 deletions(-) diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md index efb6423..4aa0dc3 100644 --- a/circle_ci/3-dry-run.md +++ b/circle_ci/3-dry-run.md @@ -4,23 +4,23 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to ## Prerequisites -1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [audit lab](./2-audit.md). ## Perform a dry run -We will be performing a dry-run against a CircleCI project. We will need to answer the following questions before running this command: +You will be performing a dry-run against a CircleCI project. Answer the following questions before running this command: -1. What is the name of the project we want to convert? +1. What project do you want to convert? - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. -2. Where do we want to store the result? +2. Where do you want to store the result? - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. ### Steps -1. Navigate to the codespace terminal +1. Navigate to your codespace terminal 2. Run the following command from the root directory: ```bash @@ -29,10 +29,12 @@ We will be performing a dry-run against a CircleCI project. We will need to answ 3. The command will list all the files written to disk when the command succeeds. - ADD_IMAGE_HERE + ![command-result](https://user-images.githubusercontent.com/18723510/189911131-bf6bfd6f-2b5e-4e49-8d14-95ef9c312117.png) + + 4. View the converted workflow: - - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in codespaces. + - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. ## Inspect the output files @@ -43,7 +45,64 @@ The files generated from the `dry-run` command represent the equivalent Actions CircleCI configuration 👇 ```yaml -# ADD YAML +version: 2.1 + +orbs: + ruby: circleci/ruby@1.1.0 + node: circleci/node@2 + +jobs: + build: + docker: + - image: cimg/ruby:2.7.5-node + steps: + - checkout + - ruby/install-deps + # Store bundle cache + - node/install-packages: + pkg-manager: yarn + cache-key: "yarn.lock" + test: + parallelism: 3 + docker: + - image: cimg/ruby:2.7.5-node + - image: circleci/postgres:9.5-alpine + environment: + POSTGRES_USER: circleci-demo-ruby + POSTGRES_DB: rails_blog_test + POSTGRES_PASSWORD: "" + environment: + BUNDLE_JOBS: "3" + BUNDLE_RETRY: "3" + PGHOST: 127.0.0.1 + PGUSER: circleci-demo-ruby + PGPASSWORD: "" + RAILS_ENV: test + steps: + - checkout + - ruby/install-deps + - node/install-packages: + pkg-manager: yarn + cache-key: "yarn.lock" + - run: + name: Wait for DB + command: dockerize -wait tcp://localhost:5432 -timeout 1m + - run: + name: Database setup + command: bundle exec rails db:schema:load --trace + # Run rspec in parallel + - ruby/rspec-test + - ruby/rubocop-check + +workflows: + version: 2 + build_and_test: + jobs: + - build + - test: + requires: + - build + ``` @@ -52,11 +111,82 @@ The files generated from the `dry-run` command represent the equivalent Actions Converted workflow 👇 ```yaml -# ADD YAML +name: labs-data/circleci-demo-ruby-rails/build_and_test +on: + push: + branches: + - master +jobs: + build: + runs-on: ubuntu-latest + container: + image: cimg/ruby:2.7.5-node + steps: + - name: Set up bundler cache + uses: ruby/setup-ruby@v1 + with: + ruby-version: 3.0.2 + bundler-cache: true + - uses: actions/checkout@v2 + - run: bundle check || bundle install + env: + BUNDLE_DEPLOYMENT: true + - id: yarn-cache-dir-path + run: echo "::set-output name=dir::$(yarn config get cacheFolder)" + - uses: actions/cache@v2 + with: + path: "${{ steps.yarn-cache-dir-path.outputs.dir }}" + key: "${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}" + restore-keys: "${{ runner.os }}-yarn-" + - run: yarn install --frozen-lockfile + test: + runs-on: ubuntu-latest + container: + image: cimg/ruby:2.7.5-node + services: + postgres: + image: postgres:9.5-alpine + env: + POSTGRES_USER: circleci-demo-ruby + POSTGRES_DB: rails_blog_test + POSTGRES_PASSWORD: '' + needs: + - build + env: + BUNDLE_JOBS: '3' + BUNDLE_RETRY: '3' + PGHOST: 127.0.0.1 + PGUSER: circleci-demo-ruby + PGPASSWORD: '' + RAILS_ENV: test + steps: + - name: Set up bundler cache + uses: ruby/setup-ruby@v1 + with: + ruby-version: 3.0.2 + bundler-cache: true + - uses: actions/checkout@v2 + - run: bundle check || bundle install + env: + BUNDLE_DEPLOYMENT: true + - id: yarn-cache-dir-path + run: echo "::set-output name=dir::$(yarn config get cacheFolder)" + - uses: actions/cache@v2 + with: + path: "${{ steps.yarn-cache-dir-path.outputs.dir }}" + key: "${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}" + restore-keys: "${{ runner.os }}-yarn-" + - run: yarn install --frozen-lockfile + - name: Wait for DB + run: dockerize -wait tcp://localhost:5432 -timeout 1m + - name: Database setup + run: bundle exec rails db:schema:load --trace + - run: bundle exec rspec spec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec/results.xml --format progress + - run: bundle exec rubocop --format progress ``` -Despite these 2 pipelines using different syntax they will function equivalently. +Despite these two pipelines using different syntax they will function equivalently. ## Next lab From 070bd720d240921453cf19ff268b5190c5c51d1f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 09:48:15 -0400 Subject: [PATCH 16/59] Create 5-migrate.md --- circle_ci/5-migrate.md | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 circle_ci/5-migrate.md diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md new file mode 100644 index 0000000..e1b159a --- /dev/null +++ b/circle_ci/5-migrate.md @@ -0,0 +1,53 @@ +# Perform a production migration of a CircleCI pipeline + +In this lab, you will use the `migrate` command to convert a CircleCI pipeline and open a pull request with the equivalent Actions workflow. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [dry-run lab](./3-dry-run.md). +4. Completed the [custom transformers lab](./4-custom-transformers.md). + +## Performing a migration + +Answer the following questions before running a `migrate` command: + +1. What project do you want to migrate? + - __circleci-hello-world__ +2. What is the namespace for that project? + - __Valet__ +3. Where do you want to store the logs? + - __./tmp/migrate__ +4. What is the URL for the GitHub repository to add the workflow to? + - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. + +### Steps + +1. Run the following `migrate` command in the codespace terminal: + + ```bash + gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world + ``` + +2. The command will write the URL to the pull request that was created when the command succeeds. + + ADD_SCREENSHOT_HERE + +3. Open the generated pull request in a new browser tab. + +### Inspect the pull request + +The first thing to notice about the pull request is that there is a list of manual steps to complete. + +Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request. + +Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. + +ADD_SCREENSHOT_HERE + +At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! + +### Next Lab + +[Forecast potential build runner usage](6-forecast.md) From 7469d8565ce21bf63c08fe412bb5c7d81c0115b0 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:13:10 -0400 Subject: [PATCH 17/59] Update 5-migrate.md --- circle_ci/5-migrate.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md index e1b159a..d9ad85f 100644 --- a/circle_ci/5-migrate.md +++ b/circle_ci/5-migrate.md @@ -5,7 +5,7 @@ In this lab, you will use the `migrate` command to convert a CircleCI pipeline a ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./3-dry-run.md). 4. Completed the [custom transformers lab](./4-custom-transformers.md). @@ -15,11 +15,9 @@ Answer the following questions before running a `migrate` command: 1. What project do you want to migrate? - __circleci-hello-world__ -2. What is the namespace for that project? - - __Valet__ -3. Where do you want to store the logs? +2. Where do you want to store the logs? - __./tmp/migrate__ -4. What is the URL for the GitHub repository to add the workflow to? +3. What is the URL for the GitHub repository to add the workflow to? - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps @@ -32,7 +30,7 @@ Answer the following questions before running a `migrate` command: 2. The command will write the URL to the pull request that was created when the command succeeds. - ADD_SCREENSHOT_HERE + ![pr](https://user-images.githubusercontent.com/18723510/189923353-1166bd94-8d64-4924-ac55-3e06fa5d1734.png) 3. Open the generated pull request in a new browser tab. @@ -42,9 +40,12 @@ The first thing to notice about the pull request is that there is a list of manu Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request. +![action-run](https://user-images.githubusercontent.com/18723510/189924238-9f6799c7-e029-4695-a1de-a23666171992.png) + Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. - -ADD_SCREENSHOT_HERE + + + At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! From b228150fffea19c922787e730c384a3c279c39f7 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:24:20 -0400 Subject: [PATCH 18/59] Update 5-migrate.md --- circle_ci/5-migrate.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md index d9ad85f..e8c1f09 100644 --- a/circle_ci/5-migrate.md +++ b/circle_ci/5-migrate.md @@ -18,14 +18,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __./tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps -1. Run the following `migrate` command in the codespace terminal: +1. Run the following `migrate` command in the codespace terminal. Ensure the values in `--target-url` for `:owner` and `:repo` are replaced with your values: ```bash - gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world + gh valet migrate circle-ci --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world ``` 2. The command will write the URL to the pull request that was created when the command succeeds. @@ -44,9 +44,6 @@ Next, you can inspect the "Files changed" in this pull request and see the conve Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. - - - At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! ### Next Lab From 9dab504ed333246b87d03f2486c15e659f6eb574 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:50:43 -0400 Subject: [PATCH 19/59] Create 6-forecast.md --- circle_ci/6-forecast.md | 99 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 circle_ci/6-forecast.md diff --git a/circle_ci/6-forecast.md b/circle_ci/6-forecast.md new file mode 100644 index 0000000..356d771 --- /dev/null +++ b/circle_ci/6-forecast.md @@ -0,0 +1,99 @@ +# Forecast potential build runner usage + +In this lab you will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from completed pipeline runs in CircleCI. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. +2. Completed the [configure lab](./1-configure.md#configuring-credentials). + +## Perform a forecast + +Answer the following questions before running the `forecast` command: +1. What is the date you want to start forecasting from? + - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. +2. Where do you want to store the results? + - **./tmp/forecast_reports** + +### Steps + +1. Navigate to your codespace terminal +2. Run the following command from the root directory: + + ```bash + gh valet forecast circle-ci --output-dir ./tmp/forecast_reports --start-date 2022-09-02 + ``` + +3. The command will list all the files written to disk when the command succeeds. + + ADD_SCREENSHOT_HERE + +## Review the forecast report + +The forecast report, logs, and completed job data will be located within the `tmp/forecast_reports` folder. + +1. Find the `forecast_report.md` file in the file explorer. +2. Right-click the `forecast_report.md` file and select `Open Preview`. +3. This file contains metrics used to forecast potential GitHub Actions usage. + +### Total + +The "Total" section of the forecast report contains high level statistics related to all the jobs completed after the `--start-date` CLI option: + +```md +- Job count: **18** +- Pipeline count: **13** + +- Execution time + + - Total: **46 minutes** + - Median: **1 minutes** + - P90: **6 minutes** + - Min: **0 minutes** + - Max: **10 minutes** + +- Queue time + + - Median: **0 minutes** + - P90: **0 minutes** + - Min: **0 minutes** + - Max: **0 minutes** + +- Concurrent jobs + + - Median: **0** + - P90: **0** + - Min: **0** + - Max: **4** +``` + +Here are some key terms of items defined in the forecast report: + +- The `job count` is the total number of completed jobs. +- The `pipeline count` is the number of unique pipelines used. +- `Execution time` describes the amount of time a runner spent on a job. This metric can be used to help plan for the cost of GitHub-hosted runners. + - This metric is correlated to how much you should expect to spend in GitHub Actions. This will vary depending on the hardware used for these minutes. You can use the [Actions pricing calculator](https://github.com/pricing/calculator) to estimate a dollar amount. +- `Queue time` metrics describe the amount of time a job spent waiting for a runner to be available to execute it. +- `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure. + +Additionally, these metrics are defined for each queue of runners defined in GitLab. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. + +## Forecasting multiple providers + +You can examine the available options for the `forecast` command by running `gh valet forecast --help`. When you do this you will see the `--source-file-path` option: + +![img](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) + +You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and want to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). + +Run the following command from within the codespace terminal: + +```bash +gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast +``` + +You can now inspect the output of the command to see a forecast report using all of the files matching the `tmp/**/jobs/*.json` pattern. + +## Next steps + +This concludes all labs for migrating GitLab pipelines to Actions with Valet! From 6a205884d88b81bcbfe4d395b2ea3a1b6de32570 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:56:54 -0400 Subject: [PATCH 20/59] Update 6-forecast.md --- circle_ci/6-forecast.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/circle_ci/6-forecast.md b/circle_ci/6-forecast.md index 356d771..84be464 100644 --- a/circle_ci/6-forecast.md +++ b/circle_ci/6-forecast.md @@ -26,7 +26,7 @@ Answer the following questions before running the `forecast` command: 3. The command will list all the files written to disk when the command succeeds. - ADD_SCREENSHOT_HERE + ![command-output](https://user-images.githubusercontent.com/18723510/189935161-dfd66c9c-e4f8-4833-ae7d-445c83cf75ba.png) ## Review the forecast report @@ -76,7 +76,7 @@ Here are some key terms of items defined in the forecast report: - `Queue time` metrics describe the amount of time a job spent waiting for a runner to be available to execute it. - `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure. -Additionally, these metrics are defined for each queue of runners defined in GitLab. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. +Additionally, these metrics are defined for each queue of runners defined in CircleCI. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. ## Forecasting multiple providers @@ -96,4 +96,4 @@ You can now inspect the output of the command to see a forecast report using all ## Next steps -This concludes all labs for migrating GitLab pipelines to Actions with Valet! +This concludes all labs for migrating CircleCI pipelines to Actions with Valet! From 708c4fdc92a2e9d14de0e8b774895dae3d6aba14 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 12:05:34 -0400 Subject: [PATCH 21/59] Create 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 52 ++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 circle_ci/4-custom-transformers.md diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md new file mode 100644 index 0000000..caed142 --- /dev/null +++ b/circle_ci/4-custom-transformers.md @@ -0,0 +1,52 @@ +# Using custom transformers to customize Valet's behavior + +In this lab you will build upon the `dry-run` command to override Valet's default behavior and customize the converted workflow using "custom transformers". Custom transformers can be used to: + +1. Convert items that are not automatically converted. +2. Convert items that were automatically converted using different actions. +3. Convert environment variable values differently. +4. Convert references to runners to use a different runner name in Actions. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start you GitLab server. +2. Completed the [configure lab](./1-configure.md#configuring-credentials). +3. Completed the [dry-run lab](./3-dry-run.md). + +## Perform a dry-run + +You will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal: + +```bash +# ADD_COMMAND +``` + +The converted workflow that is generated by the above command can be seen below: + +
+ Converted workflow 👇 + +```yaml +# ADD_YAML +``` + +
+ +_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. + +## Custom transformers for an unknown step + +ADD_CONTENT_OR_REMOVE + +## Custom transformers for environment variables + +ADD_CONTENT + +That's it! Congratulations, you have overridden Valet's default behavior by customizing the conversion of: + +- UPDATE_LIST_HERE +- Environment variables + +## Next lab + +[Perform a production migration of a CircleCI pipeline](5-migrate.md) From 6074f73ffe9f039d462b10f9ecf304e35d9d1260 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 15:48:39 -0400 Subject: [PATCH 22/59] Update 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 94 +++++++++++++++++++++++++++++- 1 file changed, 91 insertions(+), 3 deletions(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index caed142..fe2bdea 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -18,7 +18,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul You will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal: ```bash -# ADD_COMMAND +gh valet dry-run circle-ci -o tmp/custom_transformers --circle-ci-project circleci-node-example ``` The converted workflow that is generated by the above command can be seen below: @@ -27,7 +27,35 @@ The converted workflow that is generated by the above command can be seen below: Converted workflow 👇 ```yaml -# ADD_YAML +name: labs-data/circleci-node-example/sample +on: + push: + branches: + - main +jobs: + setup: + runs-on: + - self-hosted + - large + env: + COVERAGE_DIR: "./tmp/cov" + steps: + - run: mkdir -p $COVERAGE_DIR + node_test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - id: npm-cache-dir + run: echo "::set-output name=dir::$(npm config get cache)" + - uses: actions/cache@v2 + with: + path: "${{ steps.npm-cache-dir.outputs.dir }}" + key: "${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}" + restore-keys: "${{ runner.os }}-node-" + - run: npm ci + - run: npm run test +# # This item has no matching transformer +# - codecov_codecov_upload: ``` @@ -36,7 +64,67 @@ _Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the f ## Custom transformers for an unknown step -ADD_CONTENT_OR_REMOVE +The converted workflow above contains an `codecov_codecov_upload` step that was not automatically converted. Answer the following questions before writing a custom transformer: + +1. What is the "identifier" of the step to customize? + - __codecov_codecov_upload__ + +2. What is the desired Actions syntax to use instead? + - After some research, you have determined that the [codecov action](https://github.com/marketplace/actions/codecov) in the marketplace will provide similar functionality: + + ```yaml + - uses: codecov/codecov-action@v3 + with: + token: ${{ secrets.CODECOV_TOKEN }} + ``` + +Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: + +```bash +code transformers.rb +``` + +Next, you will define a `transform` method for the `codecov_codecov_upload` identifier by adding the following code to `transformers.rb`: + +```ruby +transform "codecov_codecov_upload" do |_item| + { + name: "Upload coverage to Codecov", + uses: "codecov/codecov-action@v3", + with: { + token: "${{ secrets.CODECOV_TOKEN }}" + } + } +end +``` + +This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in CircleCI. + +Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: + +```bash +gh valet dry-run circle-ci -o tmp/custom_transformers --circle-ci-project circleci-node-example --custom-transformers transformers.rb +``` + +The converted workflow that is generated by the above command will now use the custom logic for the `codecov_codecov_upload` step. + +```diff +- # # This item has no matching transformer +- # - codecov_codecov_upload: ++ - name: upload coverage to Codecov ++ uses: codecov/codecov-action@v3 ++ with: ++ token: "${{ secrets.CODECOV_TOKEN }}" +``` + +_Note_: If you were unsure what the data structure of `item` was, you could use the following code in the custom transformer to print `item` to the console: + +```ruby +transform "codecov_codecov_upload" do |item| + puts item +end +``` + ## Custom transformers for environment variables From ad436877ac5ffc1c1a5acf122948bc3b72ce86a1 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 15:54:38 -0400 Subject: [PATCH 23/59] Update 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index fe2bdea..9e1ad70 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -128,7 +128,26 @@ end ## Custom transformers for environment variables -ADD_CONTENT +You can also use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `tmp/cov`. + +To do this, add the following code to the `transformers.rb` file. + +```ruby +env "COVERAGE_DIR", "$RUNNER_TEMP/cov" +``` + +In this example, the first parameter to the `env` method is the environment variable name and the second is the updated value. + +Now you can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `COVERAGE_DIR` environment variable will be set to `$RUNNER_TEMP/cov`: + +```diff + env: +- COVERAGE_DIR: "./tmp/cov" ++. COVERAGE_DIR: "$RUNNER_TEMP/cov" +``` + +## Custom transformers for resource class: +ADD_CONTENT_HERE That's it! Congratulations, you have overridden Valet's default behavior by customizing the conversion of: From 7e6747eb5977ec9bf485f98979e1e438015e092b Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 16:19:19 -0400 Subject: [PATCH 24/59] Update 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 44 ++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index 9e1ad70..6f4dc19 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -128,7 +128,7 @@ end ## Custom transformers for environment variables -You can also use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `tmp/cov`. +You can use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `tmp/cov`. To do this, add the following code to the `transformers.rb` file. @@ -147,12 +147,50 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C ``` ## Custom transformers for resource class: -ADD_CONTENT_HERE +You can use custom transformers to change the runner associated with a `resource_class`. In the example pipeline the setup job is using a `large` resource class runner. You want to change that to the correct runner in GitHub Actions with label 'some-large-runner'. +To do this, add the following code to the `transformers.rb` file. + +```ruby +runner "large", "some-large-runner" +``` + +In this example, the first parameter to the `runner` method is the resource class and the second is the runner labels to use in Actions. + +Now you can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` will be set to `some-large-runner`: + +```diff + setup: + runs-on: + - - self-hosted + - - large + + - some-large-runner +``` + +At this point, the file contents of `transformers.rb` should match this: + +
+ Custom transformers 👇 + +```ruby +env "COVERAGE_DIR", "$RUNNER_TEMP/cov" +runner "large", "some-large-runner" + +transform "codecov_codecov_upload" do |_item| + { + name: "Upload coverage to Codecov", + uses: "codecov/codecov-action@v3", + with: { token: "${{ secrets.CODECOV_TOKEN }}" } + } +end +``` + +
That's it! Congratulations, you have overridden Valet's default behavior by customizing the conversion of: -- UPDATE_LIST_HERE +- Unknown steps - Environment variables +- Runner ## Next lab From b54e0538109b0de088fd31421d779c7f76de044f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 16:25:31 -0400 Subject: [PATCH 25/59] Update 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index 6f4dc19..194845e 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -70,7 +70,7 @@ The converted workflow above contains an `codecov_codecov_upload` step that was - __codecov_codecov_upload__ 2. What is the desired Actions syntax to use instead? - - After some research, you have determined that the [codecov action](https://github.com/marketplace/actions/codecov) in the marketplace will provide similar functionality: + - After some research, you have determined that the [Codecov action](https://github.com/marketplace/actions/codecov) in the marketplace will provide similar functionality: ```yaml - uses: codecov/codecov-action@v3 @@ -147,7 +147,7 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C ``` ## Custom transformers for resource class: -You can use custom transformers to change the runner associated with a `resource_class`. In the example pipeline the setup job is using a `large` resource class runner. You want to change that to the correct runner in GitHub Actions with label 'some-large-runner'. +You can use custom transformers to change the runner associated with a `resource_class`. In the example pipeline the setup job is using a `large` resource class runner. You want to change that to the correct runner name in GitHub Actions, which is some-large-runner. To do this, add the following code to the `transformers.rb` file. From 195b2c97615aff5b3d0c7d417d22f75c609ee116 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 16:26:06 -0400 Subject: [PATCH 26/59] Update 4-custom-transformers.md --- circle_ci/4-custom-transformers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index 194845e..eb1e903 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -9,7 +9,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul ## Prerequisites -1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start you GitLab server. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./3-dry-run.md). From f42a0a87486f30e4f223c30c0a6d7660d518931e Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 16:27:00 -0400 Subject: [PATCH 27/59] Update 2-audit.md --- circle_ci/2-audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 9fb1ee7..a102e6c 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -190,4 +190,4 @@ Each pipeline will have a variety of files written that include: ### Next lab -[Perform a dry-run of a GitLab pipeline](3-dry-run.md) +[Perform a dry-run of a CircleCI pipeline](3-dry-run.md) From 011945bab917cc7b6e377de070be8e0be908a38e Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 18:43:57 -0400 Subject: [PATCH 28/59] Update circle_ci/4-custom-transformers.md Co-authored-by: Ethan Dennis --- circle_ci/4-custom-transformers.md | 1 - 1 file changed, 1 deletion(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index eb1e903..f550398 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -125,7 +125,6 @@ transform "codecov_codecov_upload" do |item| end ``` - ## Custom transformers for environment variables You can use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `tmp/cov`. From 83bb60496fddfd720e8cb195c1d9f4bd91bfd28c Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 18:45:01 -0400 Subject: [PATCH 29/59] Update circle_ci/4-custom-transformers.md Co-authored-by: Ethan Dennis --- circle_ci/4-custom-transformers.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index f550398..5f9eb8e 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -146,7 +146,8 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C ``` ## Custom transformers for resource class: -You can use custom transformers to change the runner associated with a `resource_class`. In the example pipeline the setup job is using a `large` resource class runner. You want to change that to the correct runner name in GitHub Actions, which is some-large-runner. + +You can also use custom transformers to change the runner for a job that defines a `resource_class` attribute. In the example pipeline, the `setup` job uses a `resource_class` of `large` to dictate the machine used to execute the job. In this scenario, you may want to use this value to perform the `setup` job on a runner with the label of `some-large-runner` in Actions. To do this, add the following code to the `transformers.rb` file. From 44d60e9c37175cab304c1f0d10b8ef3ae460847c Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 18:45:12 -0400 Subject: [PATCH 30/59] Update circle_ci/4-custom-transformers.md Co-authored-by: Ethan Dennis --- circle_ci/4-custom-transformers.md | 1 + 1 file changed, 1 insertion(+) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index 5f9eb8e..9f2012f 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -186,6 +186,7 @@ end ``` + That's it! Congratulations, you have overridden Valet's default behavior by customizing the conversion of: - Unknown steps From 7e84d06d75a658f39579b063d1a4346c1d74b0b1 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 18:45:24 -0400 Subject: [PATCH 31/59] Update circle_ci/4-custom-transformers.md Co-authored-by: Ethan Dennis --- circle_ci/4-custom-transformers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/4-custom-transformers.md index 9f2012f..58bda71 100644 --- a/circle_ci/4-custom-transformers.md +++ b/circle_ci/4-custom-transformers.md @@ -155,7 +155,7 @@ To do this, add the following code to the `transformers.rb` file. runner "large", "some-large-runner" ``` -In this example, the first parameter to the `runner` method is the resource class and the second is the runner labels to use in Actions. +In this example, the first parameter to the `runner` method is the resource class value and the second is the label(s) of the runner to use in Actions. Now you can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` will be set to `some-large-runner`: From 98e87f371877448305d47780d52090aba78f787c Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Fri, 16 Sep 2022 09:25:43 -0700 Subject: [PATCH 32/59] Reorder cci labs --- circle_ci/{6-forecast.md => 3-forecast.md} | 0 circle_ci/{3-dry-run.md => 4-dry-run.md} | 0 circle_ci/{4-custom-transformers.md => 5-custom-transformers.md} | 0 circle_ci/{5-migrate.md => 6-migrate.md} | 0 readme.md | 1 + 5 files changed, 1 insertion(+) rename circle_ci/{6-forecast.md => 3-forecast.md} (100%) rename circle_ci/{3-dry-run.md => 4-dry-run.md} (100%) rename circle_ci/{4-custom-transformers.md => 5-custom-transformers.md} (100%) rename circle_ci/{5-migrate.md => 6-migrate.md} (100%) diff --git a/circle_ci/6-forecast.md b/circle_ci/3-forecast.md similarity index 100% rename from circle_ci/6-forecast.md rename to circle_ci/3-forecast.md diff --git a/circle_ci/3-dry-run.md b/circle_ci/4-dry-run.md similarity index 100% rename from circle_ci/3-dry-run.md rename to circle_ci/4-dry-run.md diff --git a/circle_ci/4-custom-transformers.md b/circle_ci/5-custom-transformers.md similarity index 100% rename from circle_ci/4-custom-transformers.md rename to circle_ci/5-custom-transformers.md diff --git a/circle_ci/5-migrate.md b/circle_ci/6-migrate.md similarity index 100% rename from circle_ci/5-migrate.md rename to circle_ci/6-migrate.md diff --git a/readme.md b/readme.md index f2c1371..c27d48f 100644 --- a/readme.md +++ b/readme.md @@ -9,6 +9,7 @@ To get started: 1. Use the `valet-customers/labs` repository as a template to [generate](https://github.com/valet-customers/labs/generate) a new GitHub repository. 2. Select which learning path to begin with. There are currently learning paths for: - [Migrations from Azure DevOps to GitHub Actions](/azure_devops/readme.md) + - [Migrations from CircleCI to GitHub Actions](/circle_ci/readme.md) - [Migrations from GitLab CI to GitHub Actions](/gitlab/readme.md) - [Migrations from Jenkins to GitHub Actions](/jenkins/readme.md) 3. Each learning path describes how to configure your codespace, bootstrap a CI/CD environment, and troubleshoot the Valet CLI. From f5bb8a469159aac77277e313ca2b5afd8c93ea1f Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Fri, 16 Sep 2022 09:27:18 -0700 Subject: [PATCH 33/59] Update links --- circle_ci/2-audit.md | 2 +- circle_ci/3-forecast.md | 2 +- circle_ci/4-dry-run.md | 3 ++- circle_ci/5-custom-transformers.md | 2 +- circle_ci/6-migrate.md | 2 +- 5 files changed, 6 insertions(+), 5 deletions(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index a102e6c..eaa864d 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -190,4 +190,4 @@ Each pipeline will have a variety of files written that include: ### Next lab -[Perform a dry-run of a CircleCI pipeline](3-dry-run.md) +[Forecast potential build runner usage](3-forecast.md) diff --git a/circle_ci/3-forecast.md b/circle_ci/3-forecast.md index 84be464..6d57bb5 100644 --- a/circle_ci/3-forecast.md +++ b/circle_ci/3-forecast.md @@ -96,4 +96,4 @@ You can now inspect the output of the command to see a forecast report using all ## Next steps -This concludes all labs for migrating CircleCI pipelines to Actions with Valet! +[Perform a dry-run of a CircleCI pipeline](4-dry-run.md) diff --git a/circle_ci/4-dry-run.md b/circle_ci/4-dry-run.md index 4aa0dc3..6be32a6 100644 --- a/circle_ci/4-dry-run.md +++ b/circle_ci/4-dry-run.md @@ -184,10 +184,11 @@ jobs: - run: bundle exec rspec spec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec/results.xml --format progress - run: bundle exec rubocop --format progress ``` + Despite these two pipelines using different syntax they will function equivalently. ## Next lab -[Use custom transformers to customize Valet's behavior](./4-custom-transformers.md) +[Use custom transformers to customize Valet's behavior](./5-custom-transformers.md) diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index 58bda71..e1abef7 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -195,4 +195,4 @@ That's it! Congratulations, you have overridden Valet's default behavior by cust ## Next lab -[Perform a production migration of a CircleCI pipeline](5-migrate.md) +[Perform a production migration of a CircleCI pipeline](6-migrate.md) diff --git a/circle_ci/6-migrate.md b/circle_ci/6-migrate.md index e8c1f09..d31d3bc 100644 --- a/circle_ci/6-migrate.md +++ b/circle_ci/6-migrate.md @@ -48,4 +48,4 @@ At this point, the migration has completed and you have successfully migrated a ### Next Lab -[Forecast potential build runner usage](6-forecast.md) +This concludes all labs for migrating CircleCI pipelines to Actions with Valet! From bbe9385689a2082367b0ce2a6b39382ff2e49e1b Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Fri, 16 Sep 2022 09:37:46 -0700 Subject: [PATCH 34/59] Add some more revisions of cci labs --- azure_devops/5-custom-transformers.md | 4 ++-- azure_devops/6-migrate.md | 4 ++-- azure_devops/readme.md | 8 ++++---- circle_ci/1-configure.md | 1 - circle_ci/2-audit.md | 2 +- circle_ci/3-forecast.md | 1 + circle_ci/4-dry-run.md | 8 +++----- circle_ci/5-custom-transformers.md | 6 +++--- circle_ci/6-migrate.md | 4 ++-- circle_ci/readme.md | 8 ++++---- gitlab/5-custom-transformers.md | 4 ++-- gitlab/6-migrate.md | 4 ++-- gitlab/readme.md | 8 ++++---- jenkins/5-custom-transformers.md | 4 ++-- jenkins/6-migrate.md | 4 ++-- jenkins/readme.md | 8 ++++---- 16 files changed, 38 insertions(+), 40 deletions(-) diff --git a/azure_devops/5-custom-transformers.md b/azure_devops/5-custom-transformers.md index c46a28b..0c20c4a 100644 --- a/azure_devops/5-custom-transformers.md +++ b/azure_devops/5-custom-transformers.md @@ -12,7 +12,7 @@ In this lab we will build upon the `dry-run` command to override Valet's default 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and bootstrap an Azure DevOps project. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). 3. Completed the [audit lab](./2-audit.md). -4. Completed the [dry-run lab](./3-dry-run.md). +4. Completed the [dry-run lab](./4-dry-run.md). ## Perform a dry run @@ -77,7 +77,7 @@ jobs: -_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. +_Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for build steps diff --git a/azure_devops/6-migrate.md b/azure_devops/6-migrate.md index f735447..7288461 100644 --- a/azure_devops/6-migrate.md +++ b/azure_devops/6-migrate.md @@ -6,8 +6,8 @@ In this lab, you will use the `migrate` command to convert an Azure DevOps pipel 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and bootstrap an Azure DevOps project. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). -4. Completed the [custom transformers lab](./4-custom-transformers.md). +3. Completed the [dry-run lab](./4-dry-run.md). +4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration diff --git a/azure_devops/readme.md b/azure_devops/readme.md index a924ca6..c5f5e16 100644 --- a/azure_devops/readme.md +++ b/azure_devops/readme.md @@ -77,10 +77,10 @@ Perform the following labs to learn how to migrate Azure DevOps pipelines to Git 1. [Configure credentials for Valet](1-configure.md) 2. [Perform an audit of an Azure DevOps project](2-audit.md) -3. [Perform a dry-run migration of an Azure DevOps pipeline](3-dry-run.md) -4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Perform a production migration of a Azure DevOps pipeline](5-migrate.md) -6. [Forecast potential build runner usage](6-forecast.md) +3. [Forecast potential build runner usage](3-forecast.md) +4. [Perform a dry-run migration of an Azure DevOps pipeline](4-dry-run.md) +5. [Use custom transformers to customize Valet's behavior](5-custom-transformers.md) +6. [Perform a production migration of a Azure DevOps pipeline](6-migrate.md) ## Troubleshoot the Valet CLI diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index 8cdc30b..d80ed60 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -34,7 +34,6 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ![configure-screenshot](https://user-images.githubusercontent.com/18723510/188935436-308fa6d3-6eb7-48b9-bbf0-20ab0cdae411.png) - ## Verify your environment To verify our environment is configured correctly, we are going to run the `update` CLI command. The `update` CLI command will download the latest version of Valet to your codespace. diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index eaa864d..b48f72b 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -11,7 +11,7 @@ The `audit` command operates by fetching all of the projects defined in a Circle ## Perform an audit -We will be performing an audit against the **labs-data** organization in CircleCI, which was created for the purposes of this lab. Valet was already been set to use this organization during the configure lab. The only remaining information needed for the `audit` command is: +You will be performing an `audit` for the __labs-data__ CircleCI organization that was created for the purposes of these labs. Your environment was configured to use this organization during the [configure lab](./1-configure.md). The remaining information needed to perform an `audit` is: 1. Where do we want to store the result? - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. diff --git a/circle_ci/3-forecast.md b/circle_ci/3-forecast.md index 6d57bb5..3b86b48 100644 --- a/circle_ci/3-forecast.md +++ b/circle_ci/3-forecast.md @@ -10,6 +10,7 @@ In this lab you will use the `forecast` command to forecast potential GitHub Act ## Perform a forecast Answer the following questions before running the `forecast` command: + 1. What is the date you want to start forecasting from? - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. 2. Where do you want to store the results? diff --git a/circle_ci/4-dry-run.md b/circle_ci/4-dry-run.md index 6be32a6..6860d68 100644 --- a/circle_ci/4-dry-run.md +++ b/circle_ci/4-dry-run.md @@ -1,4 +1,4 @@ -# Perform a dry-run of a CircleCI pipeline +# Perform a dry-run migration of a CircleCI pipeline In this lab you will use the `dry-run` command to convert a CircleCI pipeline to its equivalent GitHub Actions workflow. @@ -10,7 +10,7 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to ## Perform a dry run -You will be performing a dry-run against a CircleCI project. Answer the following questions before running this command: +You will be performing a dry run migration against a CircleCI project. Answer the following questions before running this command: 1. What project do you want to convert? - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. @@ -31,12 +31,10 @@ You will be performing a dry-run against a CircleCI project. Answer the followin ![command-result](https://user-images.githubusercontent.com/18723510/189911131-bf6bfd6f-2b5e-4e49-8d14-95ef9c312117.png) - - 4. View the converted workflow: - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. - + ## Inspect the output files The files generated from the `dry-run` command represent the equivalent Actions workflow for the CircleCI project. The CircleCI configuration and converted workflow can be seen below: diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index e1abef7..f6da2f0 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -11,7 +11,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). +3. Completed the [dry-run lab](./4-dry-run.md). ## Perform a dry-run @@ -60,7 +60,7 @@ jobs: -_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. +_Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step @@ -145,7 +145,7 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C +. COVERAGE_DIR: "$RUNNER_TEMP/cov" ``` -## Custom transformers for resource class: +## Custom transformers for resource class You can also use custom transformers to change the runner for a job that defines a `resource_class` attribute. In the example pipeline, the `setup` job uses a `resource_class` of `large` to dictate the machine used to execute the job. In this scenario, you may want to use this value to perform the `setup` job on a runner with the label of `some-large-runner` in Actions. diff --git a/circle_ci/6-migrate.md b/circle_ci/6-migrate.md index d31d3bc..9892f1e 100644 --- a/circle_ci/6-migrate.md +++ b/circle_ci/6-migrate.md @@ -6,8 +6,8 @@ In this lab, you will use the `migrate` command to convert a CircleCI pipeline a 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). -4. Completed the [custom transformers lab](./4-custom-transformers.md). +3. Completed the [dry-run lab](./4-dry-run.md). +4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration diff --git a/circle_ci/readme.md b/circle_ci/readme.md index e2cc56a..dbaa3bc 100644 --- a/circle_ci/readme.md +++ b/circle_ci/readme.md @@ -41,10 +41,10 @@ Perform the following labs to test-drive Valet 1. [Configure credentials for Valet](1-configure.md) 2. [Perform an audit of CircleCI](2-audit.md) -3. [Perform a dry-run of a CircleCI pipeline](3-dry-run.md) -4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Perform a production migration of a CircleCI pipeline](5-migrate.md) -6. [Forecast potential build runner usage](6-forecast.md) +3. [Forecast potential build runner usage](3-forecast.md) +4. [Perform a dry-run of a CircleCI pipeline](4-dry-run.md) +5. [Use custom transformers to customize Valet's behavior](5-custom-transformers.md) +6. [Perform a production migration of a CircleCI pipeline](6-migrate.md) ## Troubleshoot the Valet CLI diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 0333f19..adb089d 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -11,7 +11,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start you GitLab server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). +3. Completed the [dry-run lab](./4-dry-run.md). ## Perform a dry-run @@ -53,7 +53,7 @@ jobs: -_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. +_Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step diff --git a/gitlab/6-migrate.md b/gitlab/6-migrate.md index 0e4543d..655f8e7 100644 --- a/gitlab/6-migrate.md +++ b/gitlab/6-migrate.md @@ -6,8 +6,8 @@ In this lab, you will use the `migrate` command to convert a GitLab pipeline and 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start a GitLab server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). -4. Completed the [custom transformers lab](./4-custom-transformers.md). +3. Completed the [dry-run lab](./4-dry-run.md). +4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration diff --git a/gitlab/readme.md b/gitlab/readme.md index 96cd78f..cf15e48 100644 --- a/gitlab/readme.md +++ b/gitlab/readme.md @@ -61,10 +61,10 @@ Perform the following labs to learn more about Actions migrations with Valet: 1. [Configure credentials for Valet](1-configure.md) 2. [Perform an audit on GitLab pipelines](2-audit.md) -3. [Perform a dry-run migration of a GitLab pipeline](3-dry-run.md) -4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Perform a production migration of a GitLab pipeline](5-migrate.md) -6. [Forecast potential build runner usage](6-forecast.md) +3. [Forecast potential build runner usage](3-forecast.md) +4. [Perform a dry-run migration of a GitLab pipeline](4-dry-run.md) +5. [Use custom transformers to customize Valet's behavior](5-custom-transformers.md) +6. [Perform a production migration of a GitLab pipeline](6-migrate.md) ## Troubleshoot the Valet CLI diff --git a/jenkins/5-custom-transformers.md b/jenkins/5-custom-transformers.md index 129509f..e631050 100644 --- a/jenkins/5-custom-transformers.md +++ b/jenkins/5-custom-transformers.md @@ -11,7 +11,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start a Jenkins server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). +3. Completed the [dry-run lab](./4-dry-run.md). ## Perform a dry-run @@ -71,7 +71,7 @@ jobs: -_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. +_Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the fundamentals of the `dry-run` command. ## Custom transformers for an unknown step diff --git a/jenkins/6-migrate.md b/jenkins/6-migrate.md index 325a620..f8b992b 100644 --- a/jenkins/6-migrate.md +++ b/jenkins/6-migrate.md @@ -6,8 +6,8 @@ In this lab, you will use the `migrate` command to convert a Jenkins pipeline an 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start a Jenkins server. 2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). -3. Completed the [dry-run lab](./3-dry-run.md). -4. Completed the [custom transformers lab](./4-custom-transformers.md). +3. Completed the [dry-run lab](./4-dry-run.md). +4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration diff --git a/jenkins/readme.md b/jenkins/readme.md index a8e78c7..8d55d31 100644 --- a/jenkins/readme.md +++ b/jenkins/readme.md @@ -61,10 +61,10 @@ Perform the following labs to learn more about Actions migrations with Valet: 1. [Configure credentials for Valet](1-configure.md) 2. [Perform an audit of a Jenkins server](2-audit.md) -3. [Perform a dry-run migration of a Jenkins pipeline](3-dry-run.md) -4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md) -5. [Perform a production migration of a Jenkins pipeline](5-migrate.md) -6. [Forecast potential build runner usage](6-forecast.md) +3. [Forecast potential build runner usage](3-forecast.md) +4. [Perform a dry-run migration of a Jenkins pipeline](4-dry-run.md) +5. [Use custom transformers to customize Valet's behavior](5-custom-transformers.md) +6. [Perform a production migration of a Jenkins pipeline](6-migrate.md) ## Troubleshoot the Valet CLI From c8887a6bcac625a32e3904abe316dd9bb1c9f0c1 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Fri, 16 Sep 2022 11:59:02 -0700 Subject: [PATCH 35/59] Convert audit summary to list --- azure_devops/2-audit.md | 5 ++++- circle_ci/2-audit.md | 5 ++++- gitlab/2-audit.md | 5 ++++- jenkins/2-audit.md | 5 ++++- 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/azure_devops/2-audit.md b/azure_devops/2-audit.md index d9d58f9..a721074 100644 --- a/azure_devops/2-audit.md +++ b/azure_devops/2-audit.md @@ -2,7 +2,10 @@ In this lab, you will use the `audit` command to get a high-level view of all pipelines in an Azure DevOps organization or project. -The `audit` command operates by fetching all of the pipelines defined in an Azure DevOps organization or project, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command will perform the following steps: +1. Fetch all of the projects defined in an Azure DevOps organization. +2. Convert each pipeline to their equivalent GitHub Actions workflow. +3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index b48f72b..30e0663 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -2,7 +2,10 @@ In this lab, you will use the `audit` command to get a high-level view of all projects in a CircleCI organization. -The `audit` command operates by fetching all of the projects defined in a CircleCI organization, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command will perform the following steps: +1. Fetch all of the projects defined in an CircleCI organization. +2. Convert each pipeline to their equivalent GitHub Actions workflow. +3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites diff --git a/gitlab/2-audit.md b/gitlab/2-audit.md index 657726b..d5a3c0a 100644 --- a/gitlab/2-audit.md +++ b/gitlab/2-audit.md @@ -2,7 +2,10 @@ In this lab, you will use the `audit` command to get a high-level view of all pipelines in a GitLab server. -The `audit` command operates by fetching all of the pipelines defined in a GitLab server, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command will perform the following steps: +1. Fetch all of the projects defined in a GitLab organization. +2. Convert each pipeline to their equivalent GitHub Actions workflow. +3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index e2c8b35..f31f02f 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -2,7 +2,10 @@ In this lab, you will use the `audit` command to get a high-level view of all pipelines in a Jenkins server. -The `audit` command operates by fetching all of the pipelines defined in a Jenkins server, converting each to their equivalent GitHub Actions workflow, and writing a report that summarizes how complete and complex of a migration is possible with Valet. +The `audit` command will perform the following steps: +1. Fetch all of the projects defined in a Jenkins organization. +2. Convert each pipeline to their equivalent GitHub Actions workflow. +3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. ## Prerequisites From a9f9e4cd20c2f35ebe2269d5c9fef0acb32fdc61 Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Fri, 16 Sep 2022 12:52:26 -0700 Subject: [PATCH 36/59] Apply suggestions from code review Co-authored-by: Ethan Dennis --- gitlab/2-audit.md | 2 +- jenkins/2-audit.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/gitlab/2-audit.md b/gitlab/2-audit.md index d5a3c0a..88c698c 100644 --- a/gitlab/2-audit.md +++ b/gitlab/2-audit.md @@ -3,7 +3,7 @@ In this lab, you will use the `audit` command to get a high-level view of all pipelines in a GitLab server. The `audit` command will perform the following steps: -1. Fetch all of the projects defined in a GitLab organization. +1. Fetch all of the projects defined in a GitLab group. 2. Convert each pipeline to their equivalent GitHub Actions workflow. 3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index f31f02f..ae1d621 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -3,7 +3,7 @@ In this lab, you will use the `audit` command to get a high-level view of all pipelines in a Jenkins server. The `audit` command will perform the following steps: -1. Fetch all of the projects defined in a Jenkins organization. +1. Fetch all of the projects defined in a Jenkins server. 2. Convert each pipeline to their equivalent GitHub Actions workflow. 3. Generate a report that summarizes how complete and complex of a migration is possible with Valet. From 26302692eea47193c2fa82a5935748dca8e21900 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Fri, 16 Sep 2022 13:08:06 -0700 Subject: [PATCH 37/59] Update circle_ci/1-configure.md Co-authored-by: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> --- circle_ci/1-configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index d80ed60..0526cbb 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -22,7 +22,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Select the `TERMINAL` tab from within the codespace terminal window. - Run the following command: `gh valet configure`. - Using the down arrow key to highlight `CircleCI`, press the spacebar to select, and then hit enter to continue. - - At the prompt enter your GitHub Username and press enter. + - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 2 and press enter. - At the GitHub Container Registry prompt enter the GitHub PAT generated in step 1 and press enter. - At the GitHub PAT prompt enter the GitHub PAT generated in step 1 and press enter. - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). From b25b37292310b7d8948545e3e7a57c053585d66f Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Mon, 19 Sep 2022 12:40:37 -0700 Subject: [PATCH 38/59] Update references to use valet-labs org --- circle_ci/1-configure.md | 19 ++++++----- circle_ci/2-audit.md | 54 +++++++++++++++--------------- circle_ci/4-dry-run.md | 6 ++-- circle_ci/5-custom-transformers.md | 2 +- 4 files changed, 41 insertions(+), 40 deletions(-) diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index 0526cbb..5214d16 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -23,14 +23,14 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Run the following command: `gh valet configure`. - Using the down arrow key to highlight `CircleCI`, press the spacebar to select, and then hit enter to continue. - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 2 and press enter. - - At the GitHub Container Registry prompt enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub PAT prompt enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub url prompt enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). - - At the CircleCI token prompt enter the CircleCI access token from step 2 and press enter. - - At the CircleCI base url prompt hit enter to accept the default value (`https://circleci.com`). - - At the CircleCI organization name prompt enter `labs-data`. This is the organization we'll be using throughout these labs. - - At the access token to fetch source code in GitHub prompt enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub instance url containing source code prompt press enter to accept the default value (`https://github.com`). + - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub PAT prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub url prompt, enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). + - At the CircleCI token prompt, enter the CircleCI access token from step 2 and press enter. + - At the CircleCI base url prompt, hit enter to accept the default value (`https://circleci.com`). + - At the CircleCI organization name prompt, enter `valet-labs`. This is the organization we'll be using throughout these labs. + - At the access token to fetch source code in GitHub prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub instance url containing source code prompt, press enter to accept the default value (`https://github.com`). ![configure-screenshot](https://user-images.githubusercontent.com/18723510/188935436-308fa6d3-6eb7-48b9-bbf0-20ab0cdae411.png) @@ -46,7 +46,8 @@ To verify our environment is configured correctly, we are going to run the `upda 2. You should see a confirmation that you were logged into the GitHub Container Registry and Valet was updated to the latest version. - ```bash + ```console + $ gh valet version Login Succeeded latest: Pulling from valet-customers/valet-cli Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 30e0663..1e954ef 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -9,15 +9,15 @@ The `audit` command will perform the following steps: ## Prerequisites -1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configure-credentials-for-valet). ## Perform an audit -You will be performing an `audit` for the __labs-data__ CircleCI organization that was created for the purposes of these labs. Your environment was configured to use this organization during the [configure lab](./1-configure.md). The remaining information needed to perform an `audit` is: +You will be performing an `audit` for the __valet-labs__ CircleCI organization that was created for the purposes of these labs. Your environment was configured to use this organization during the [configure lab](./1-configure.md). The remaining information needed to perform an `audit` is: 1. Where do we want to store the result? - - __./tmp/audit__. This can be any path within the working directory that Valet commands are executed from. + - __tmp/audit__. This can be any path within the working directory that Valet commands are executed from. ### Steps @@ -146,42 +146,42 @@ The final section of the audit report provides a manifest of all of the files th ```md ### Successful -#### labs-data/circleci-hello-world +#### valet-labs/circleci-hello-world -- [labs-data/circleci-hello-world/say-hello-workflow.yml](labs-data/circleci-hello-world/say-hello-workflow.yml) -- [labs-data/circleci-hello-world.config.json](labs-data/circleci-hello-world.config.json) -- [labs-data/circleci-hello-world.source.yml](labs-data/circleci-hello-world.source.yml) +- [valet-valet-labs/circleci-hello-world/say-hello-workflow.yml](valet-labs/circleci-hello-world/say-hello-workflow.yml) +- [valet-labs/circleci-hello-world.config.json](valet-labs/circleci-hello-world.config.json) +- [valet-labs/circleci-hello-world.source.yml](valet-labs/circleci-hello-world.source.yml) -#### labs-data/circleci-command-example +#### valet-labs/circleci-command-example -- [labs-data/circleci-command-example/my-workflow.yml](labs-data/circleci-command-example/my-workflow.yml) +- [valet-labs/circleci-command-example/my-workflow.yml](valet-labs/circleci-command-example/my-workflow.yml) - [.github/actions/greeting/action.yml](.github/actions/greeting/action.yml) -- [labs-data/circleci-command-example.config.json](labs-data/circleci-command-example.config.json) -- [labs-data/circleci-command-example.source.yml](labs-data/circleci-command-example.source.yml) +- [valet-labs/circleci-command-example.config.json](valet-labs/circleci-command-example.config.json) +- [valet-labs/circleci-command-example.source.yml](valet-labs/circleci-command-example.source.yml) -#### labs-data/circleci-node-example +#### valet-labs/circleci-node-example -- [labs-data/circleci-node-example/sample.yml](labs-data/circleci-node-example/sample.yml) -- [labs-data/circleci-node-example.config.json](labs-data/circleci-node-example.config.json) -- [labs-data/circleci-node-example.source.yml](labs-data/circleci-node-example.source.yml) +- [valet-labs/circleci-node-example/sample.yml](valet-labs/circleci-node-example/sample.yml) +- [valet-labs/circleci-node-example.config.json](valet-labs/circleci-node-example.config.json) +- [valet-labs/circleci-node-example.source.yml](valet-labs/circleci-node-example.source.yml) -#### labs-data/circleci-python-example +#### valet-labs/circleci-python-example -- [labs-data/circleci-python-example/sample.yml](labs-data/circleci-python-example/sample.yml) -- [labs-data/circleci-python-example.config.json](labs-data/circleci-python-example.config.json) -- [labs-data/circleci-python-example.source.yml](labs-data/circleci-python-example.source.yml) +- [valet-labs/circleci-python-example/sample.yml](valet-labs/circleci-python-example/sample.yml) +- [valet-labs/circleci-python-example.config.json](valet-labs/circleci-python-example.config.json) +- [valet-labs/circleci-python-example.source.yml](valet-labs/circleci-python-example.source.yml) -#### labs-data/circleci-demo-java-spring +#### valet-labs/circleci-demo-java-spring -- [labs-data/circleci-demo-java-spring/workflow.yml](labs-data/circleci-demo-java-spring/workflow.yml) -- [labs-data/circleci-demo-java-spring.config.json](labs-data/circleci-demo-java-spring.config.json) -- [labs-data/circleci-demo-java-spring.source.yml](labs-data/circleci-demo-java-spring.source.yml) +- [valet-labs/circleci-demo-java-spring/workflow.yml](valet-labs/circleci-demo-java-spring/workflow.yml) +- [valet-labs/circleci-demo-java-spring.config.json](valet-labs/circleci-demo-java-spring.config.json) +- [valet-labs/circleci-demo-java-spring.source.yml](valet-labs/circleci-demo-java-spring.source.yml) -#### labs-data/circleci-demo-ruby-rails +#### valet-labs/circleci-demo-ruby-rails -- [labs-data/circleci-demo-ruby-rails/build_and_test.yml](labs-data/circleci-demo-ruby-rails/build_and_test.yml) -- [labs-data/circleci-demo-ruby-rails.config.json](labs-data/circleci-demo-ruby-rails.config.json) -- [labs-data/circleci-demo-ruby-rails.source.yml](labs-data/circleci-demo-ruby-rails.source.yml) +- [valet-labs/circleci-demo-ruby-rails/build_and_test.yml](valet-labs/circleci-demo-ruby-rails/build_and_test.yml) +- [valet-labs/circleci-demo-ruby-rails.config.json](valet-labs/circleci-demo-ruby-rails.config.json) +- [valet-labs/circleci-demo-ruby-rails.source.yml](valet-labs/circleci-demo-ruby-rails.source.yml) ``` Each pipeline will have a variety of files written that include: diff --git a/circle_ci/4-dry-run.md b/circle_ci/4-dry-run.md index 6860d68..e2f07aa 100644 --- a/circle_ci/4-dry-run.md +++ b/circle_ci/4-dry-run.md @@ -13,7 +13,7 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to You will be performing a dry run migration against a CircleCI project. Answer the following questions before running this command: 1. What project do you want to convert? - - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. + - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI valet-labs organization. 2. Where do you want to store the result? - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. @@ -32,7 +32,7 @@ You will be performing a dry run migration against a CircleCI project. Answer th ![command-result](https://user-images.githubusercontent.com/18723510/189911131-bf6bfd6f-2b5e-4e49-8d14-95ef9c312117.png) 4. View the converted workflow: - - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in your codespace. + - Find `./tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. ## Inspect the output files @@ -109,7 +109,7 @@ workflows: Converted workflow 👇 ```yaml -name: labs-data/circleci-demo-ruby-rails/build_and_test +name: valet-labs/circleci-demo-ruby-rails/build_and_test on: push: branches: diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index f6da2f0..d864d71 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -27,7 +27,7 @@ The converted workflow that is generated by the above command can be seen below: Converted workflow 👇 ```yaml -name: labs-data/circleci-node-example/sample +name: valet-labs/circleci-node-example/sample on: push: branches: From fc0f71f88dfb2df19324ea4ec5eaa2cef6488544 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Mon, 19 Sep 2022 12:49:08 -0700 Subject: [PATCH 39/59] Update labs to use text instead of images --- circle_ci/3-forecast.md | 20 ++++++++++++++++---- circle_ci/4-dry-run.md | 13 +++++++++---- circle_ci/5-custom-transformers.md | 4 ++-- circle_ci/6-migrate.md | 10 +++++++--- gitlab/3-forecast.md | 14 +++++++++++++- jenkins/3-forecast.md | 24 ++++++++++-------------- 6 files changed, 57 insertions(+), 28 deletions(-) diff --git a/circle_ci/3-forecast.md b/circle_ci/3-forecast.md index 3b86b48..980ca02 100644 --- a/circle_ci/3-forecast.md +++ b/circle_ci/3-forecast.md @@ -14,7 +14,7 @@ Answer the following questions before running the `forecast` command: 1. What is the date you want to start forecasting from? - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. 2. Where do you want to store the results? - - **./tmp/forecast_reports** + - **tmp/forecast_reports** ### Steps @@ -22,7 +22,7 @@ Answer the following questions before running the `forecast` command: 2. Run the following command from the root directory: ```bash - gh valet forecast circle-ci --output-dir ./tmp/forecast_reports --start-date 2022-09-02 + gh valet forecast circle-ci --output-dir tmp/forecast_reports --start-date 2022-09-02 ``` 3. The command will list all the files written to disk when the command succeeds. @@ -83,14 +83,26 @@ Additionally, these metrics are defined for each queue of runners defined in Cir You can examine the available options for the `forecast` command by running `gh valet forecast --help`. When you do this you will see the `--source-file-path` option: -![img](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) +```console +$ gh valet forecast -h +Options: + --source-file-path (REQUIRED) The file path(s) to existing jobs data. + -o, --output-dir (REQUIRED) The location for any output files. + --start-date The start date of the forecast analysis in YYYY-MM-DD format. [default: 9/12/2022 12:42:39 PM] + --time-slice The time slice in seconds to use for computing concurrency metrics. [default: 60] + --credentials-file The file containing the credentials to use. + --no-telemetry Boolean value to disallow telemetry. + --no-ssl-verify Disable ssl certificate verification. + --no-http-cache Disable caching of http responses. + -?, -h, --help Show help and usage information +``` You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and want to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). Run the following command from within the codespace terminal: ```bash -gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast +gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/forecast-combined ``` You can now inspect the output of the command to see a forecast report using all of the files matching the `tmp/**/jobs/*.json` pattern. diff --git a/circle_ci/4-dry-run.md b/circle_ci/4-dry-run.md index e2f07aa..84ff5fe 100644 --- a/circle_ci/4-dry-run.md +++ b/circle_ci/4-dry-run.md @@ -16,7 +16,7 @@ You will be performing a dry run migration against a CircleCI project. Answer th - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI valet-labs organization. 2. Where do you want to store the result? - - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. + - __tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. ### Steps @@ -24,15 +24,20 @@ You will be performing a dry run migration against a CircleCI project. Answer th 2. Run the following command from the root directory: ```bash - gh valet dry-run circle-ci --output-dir ./tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails + gh valet dry-run circle-ci --output-dir tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails ``` 3. The command will list all the files written to disk when the command succeeds. - ![command-result](https://user-images.githubusercontent.com/18723510/189911131-bf6bfd6f-2b5e-4e49-8d14-95ef9c312117.png) + ```console + $ gh valet dry-run circle-ci --output-dir tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails --circle-ci-organization valet-labs + [2022-09-19 19:46:03] Logs: 'tmp/dry-run-lab/log/valet-20220919-194603.log' + [2022-09-19 19:46:05] Output file(s): + [2022-09-19 19:46:05] tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails/build_and_test.yml + ``` 4. View the converted workflow: - - Find `./tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails` in the file explorer pane in your codespace. + - Find `tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. ## Inspect the output files diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index d864d71..3fc5189 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -18,7 +18,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul You will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal: ```bash -gh valet dry-run circle-ci -o tmp/custom_transformers --circle-ci-project circleci-node-example +gh valet dry-run circle-ci --output-dir tmp/dry-run --circle-ci-project circleci-node-example ``` The converted workflow that is generated by the above command can be seen below: @@ -103,7 +103,7 @@ This method can use any valid ruby syntax and should return a `Hash` that repres Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: ```bash -gh valet dry-run circle-ci -o tmp/custom_transformers --circle-ci-project circleci-node-example --custom-transformers transformers.rb +gh valet dry-run circle-ci --output-dir tmp/dry-run --circle-ci-project circleci-node-example --custom-transformers transformers.rb ``` The converted workflow that is generated by the above command will now use the custom logic for the `codecov_codecov_upload` step. diff --git a/circle_ci/6-migrate.md b/circle_ci/6-migrate.md index 9892f1e..ee136a9 100644 --- a/circle_ci/6-migrate.md +++ b/circle_ci/6-migrate.md @@ -16,7 +16,7 @@ Answer the following questions before running a `migrate` command: 1. What project do you want to migrate? - __circleci-hello-world__ 2. Where do you want to store the logs? - - __./tmp/migrate__ + - __tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. @@ -25,12 +25,16 @@ Answer the following questions before running a `migrate` command: 1. Run the following `migrate` command in the codespace terminal. Ensure the values in `--target-url` for `:owner` and `:repo` are replaced with your values: ```bash - gh valet migrate circle-ci --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world + gh valet migrate circle-ci --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --circle-ci-project circleci-hello-world ``` 2. The command will write the URL to the pull request that was created when the command succeeds. - ![pr](https://user-images.githubusercontent.com/18723510/189923353-1166bd94-8d64-4924-ac55-3e06fa5d1734.png) + ```console + $ gh valet migrate circle-ci --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --circle-ci-project circleci-hello-world + [2022-08-20 22:08:20] Logs: 'tmp/migrate/log/valet-20220916-014033.log' + [2022-08-20 22:08:20] Pull request: 'https://github.com/:owner/:repo/pull/1' + ``` 3. Open the generated pull request in a new browser tab. diff --git a/gitlab/3-forecast.md b/gitlab/3-forecast.md index 172ad98..708ab96 100644 --- a/gitlab/3-forecast.md +++ b/gitlab/3-forecast.md @@ -85,7 +85,19 @@ Additionally, these metrics are defined for each queue of runners defined in Git You can examine the available options for the `forecast` command by running `gh valet forecast --help`. When you do this you will see the `--source-file-path` option: -![img](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) +```console +$ gh valet forecast -h +Options: + --source-file-path (REQUIRED) The file path(s) to existing jobs data. + -o, --output-dir (REQUIRED) The location for any output files. + --start-date The start date of the forecast analysis in YYYY-MM-DD format. [default: 9/12/2022 12:42:39 PM] + --time-slice The time slice in seconds to use for computing concurrency metrics. [default: 60] + --credentials-file The file containing the credentials to use. + --no-telemetry Boolean value to disallow telemetry. + --no-ssl-verify Disable ssl certificate verification. + --no-http-cache Disable caching of http responses. + -?, -h, --help Show help and usage information +``` You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and want to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). diff --git a/jenkins/3-forecast.md b/jenkins/3-forecast.md index 7196536..d2aa4fd 100644 --- a/jenkins/3-forecast.md +++ b/jenkins/3-forecast.md @@ -95,21 +95,17 @@ Additionally, these metrics are defined for each queue of runners defined in Jen You can examine the available options for the `forecast` command by running `gh valet forecast jenkins --help`. When you do this you will see the `--source-file-path` option: ```console -$ gh valet forecast jenkins --help +$ gh valet forecast -h Options: - -u, --jenkins-instance-url The URL of the Jenkins CI instance. - -n, --jenkins-username Username for the Jenkins instance. - -t, --jenkins-access-token Access token for the Jenkins instance. - -f, --folders Folders to forecast in the instance - --source-file-path The file path(s) to existing jobs data. - -o, --output-dir (REQUIRED) The location for any output files. - --start-date The start date of the forecast analysis in YYYY-MM-DD format. [default: 9/9/2022 2:14:15 AM] - --time-slice The time slice in seconds to use for computing concurrency metrics. [default: 60] - --credentials-file The file containing the credentials to use. - --no-telemetry Boolean value to disallow telemetry. - --no-ssl-verify Disable ssl certificate verification. - --no-http-cache Disable caching of http responses. - -?, -h, --help Show help and usage information + --source-file-path (REQUIRED) The file path(s) to existing jobs data. + -o, --output-dir (REQUIRED) The location for any output files. + --start-date The start date of the forecast analysis in YYYY-MM-DD format. [default: 9/12/2022 12:42:39 PM] + --time-slice The time slice in seconds to use for computing concurrency metrics. [default: 60] + --credentials-file The file containing the credentials to use. + --no-telemetry Boolean value to disallow telemetry. + --no-ssl-verify Disable ssl certificate verification. + --no-http-cache Disable caching of http responses. + -?, -h, --help Show help and usage information ``` You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and wanted to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). From f0acd3135ba36477d3e41c1551531c1456a5e3c7 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Mon, 19 Sep 2022 12:51:07 -0700 Subject: [PATCH 40/59] More text --- circle_ci/3-forecast.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/circle_ci/3-forecast.md b/circle_ci/3-forecast.md index 980ca02..a798adc 100644 --- a/circle_ci/3-forecast.md +++ b/circle_ci/3-forecast.md @@ -27,7 +27,14 @@ Answer the following questions before running the `forecast` command: 3. The command will list all the files written to disk when the command succeeds. - ![command-output](https://user-images.githubusercontent.com/18723510/189935161-dfd66c9c-e4f8-4833-ae7d-445c83cf75ba.png) + ```console + $ gh valet forecast circle-ci --output-dir tmp/forecast_reports --start-date 2022-09-02 + [2022-08-20 22:08:20] Logs: 'tmp/forecast/log/valet-20220916-021004.log' + [2022-08-20 22:08:20] Forecasting 'http://app.circleci.com/pipelines/github/valet-labs' + [2022-08-20 22:08:20] Output file(s): + [2022-08-20 22:08:20] tmp/forecast/jobs/09-16-2022-02-10_jobs_0.json + [2022-08-20 22:08:20] tmp/forecast/forecast_report.md + ``` ## Review the forecast report From 12fcd5afb506724b894b465e0d10f6ef5baaa44c Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Mon, 19 Sep 2022 12:53:05 -0700 Subject: [PATCH 41/59] More updates --- circle_ci/4-dry-run.md | 12 ++++++------ circle_ci/readme.md | 34 ++++++++++++++++++---------------- 2 files changed, 24 insertions(+), 22 deletions(-) diff --git a/circle_ci/4-dry-run.md b/circle_ci/4-dry-run.md index 84ff5fe..0d745f5 100644 --- a/circle_ci/4-dry-run.md +++ b/circle_ci/4-dry-run.md @@ -16,7 +16,7 @@ You will be performing a dry run migration against a CircleCI project. Answer th - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI valet-labs organization. 2. Where do you want to store the result? - - __tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. + - __tmp/dry-run__. This can be any path within the working directory that Valet commands are executed from. ### Steps @@ -24,20 +24,20 @@ You will be performing a dry run migration against a CircleCI project. Answer th 2. Run the following command from the root directory: ```bash - gh valet dry-run circle-ci --output-dir tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails + gh valet dry-run circle-ci --output-dir tmp/dry-run --circle-ci-project circleci-demo-ruby-rails ``` 3. The command will list all the files written to disk when the command succeeds. ```console - $ gh valet dry-run circle-ci --output-dir tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails --circle-ci-organization valet-labs - [2022-09-19 19:46:03] Logs: 'tmp/dry-run-lab/log/valet-20220919-194603.log' + $ gh valet dry-run circle-ci --output-dir tmp/dry-run --circle-ci-project circleci-demo-ruby-rails --circle-ci-organization valet-labs + [2022-09-19 19:46:03] Logs: 'tmp/dry-run/log/valet-20220919-194603.log' [2022-09-19 19:46:05] Output file(s): - [2022-09-19 19:46:05] tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails/build_and_test.yml + [2022-09-19 19:46:05] tmp/dry-run/valet-labs/circleci-demo-ruby-rails/build_and_test.yml ``` 4. View the converted workflow: - - Find `tmp/dry-run-lab/valet-labs/circleci-demo-ruby-rails` in the file explorer pane in your codespace. + - Find `tmp/dry-run/valet-labs/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. ## Inspect the output files diff --git a/circle_ci/readme.md b/circle_ci/readme.md index dbaa3bc..3131b01 100644 --- a/circle_ci/readme.md +++ b/circle_ci/readme.md @@ -12,28 +12,29 @@ These steps **must** be completed prior to starting other labs. 1. Start a new Codespace. -- Click the `Code` with button down arrow above repository on the repository's landing page. -- Click the `Codespaces` tab -- Click `Create codespaces on main` to create the codespace. -- After the Codespace has initialized there will be a terminal present. + - Click the `Code` with button down arrow above repository on the repository's landing page. + - Click the `Codespaces` tab + - Click `Create codespaces on main` to create the codespace. + - After the Codespace has initialized there will be a terminal present. 2. Verify the Valet CLI is installed and working. More information on the Valet extension for the official GitHub CLI can be found [here](https://github.com/github/gh-valet). -- Run the following command in the codespace's terminal: + - Run the following command in the codespace's terminal: - ```bash - gh valet version - ``` + ```bash + gh valet version + ``` -- Verify the output is similar to below. The version information may differ than what is shown below, it is expected that the `valet-cli` version is unknown. This will be corrected in the [configure credentials for Valet](1-configure.md) lab. + - Verify the output is similar to below. - ```bash - gh version 2.14.3 (2022-07-26) - gh valet github/gh-valet v0.1.12 - valet-cli unknown - ``` + ```console + $ gh valet version + gh version 2.14.3 (2022-07-26) + gh valet github/gh-valet v0.1.12 + valet-cli unknown + ``` - - If `gh valet version` did not produce similar output then please follow the troubleshooting [guide](#troubleshoot-the-valet-cli). + - If `gh valet version` did not produce similar output, refer to the troubleshooting [guide](#troubleshoot-the-valet-cli). ## Labs for CircleCI @@ -59,7 +60,8 @@ The CLI extension for Valet can be manually installed by following these steps: - Verify the result of the install contains: - ```bash + ```console + $ gh extension install github/gh-valet ✓ Installed extension github/gh-valet ``` From aa0bdc7607ac0e3ab420f86983ef99e36a8fca16 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Mon, 19 Sep 2022 16:01:07 -0700 Subject: [PATCH 42/59] Create file before opening it --- azure_devops/5-custom-transformers.md | 4 ++-- circle_ci/5-custom-transformers.md | 2 +- gitlab/5-custom-transformers.md | 2 +- jenkins/5-custom-transformers.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/azure_devops/5-custom-transformers.md b/azure_devops/5-custom-transformers.md index 8b2658a..6d2d0d5 100644 --- a/azure_devops/5-custom-transformers.md +++ b/azure_devops/5-custom-transformers.md @@ -94,10 +94,10 @@ You can use custom transformers to override Valet's default behavior. In this sc shell: bash ``` -Now you can begin to write the custom transformer. Customer transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: +Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -code transformers.rb +touch transformers.rb && code transformers.rb ``` Next, you will define a `transform` method for the `DotnetCoreCLI@2` identifier by adding the following code to `transformers.rb`: diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index 3fc5189..04a7d05 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -81,7 +81,7 @@ The converted workflow above contains an `codecov_codecov_upload` step that was Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -code transformers.rb +touch transformers.rb && code transformers.rb ``` Next, you will define a `transform` method for the `codecov_codecov_upload` identifier by adding the following code to `transformers.rb`: diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 7466dc9..5413ed4 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -74,7 +74,7 @@ The converted workflow above contains an `artifacts.terraform` step that was not Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -code transformers.rb +touch transformers.rb && code transformers.rb ``` Next, you will define a `transform` method for the `artifacts.terraform` identifier by adding the following code to `transformers.rb`: diff --git a/jenkins/5-custom-transformers.md b/jenkins/5-custom-transformers.md index 91a9f6e..281c4a2 100644 --- a/jenkins/5-custom-transformers.md +++ b/jenkins/5-custom-transformers.md @@ -92,7 +92,7 @@ The converted workflow above contains a `sleep` step was not automatically conve Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: ```bash -code transformers.rb +touch transformers.rb && code transformers.rb ``` Next, you will define a `transform` method for the `sleep` identifier by adding the following code to `transformers.rb`: From b321b60c85832516208b6207a60ad5f1e8cf63d3 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Mon, 19 Sep 2022 17:18:05 -0700 Subject: [PATCH 43/59] Initial edits to ADO/gitlab/jenkins --- azure_devops/1-configure.md | 10 ++++++---- azure_devops/3-forecast.md | 2 +- azure_devops/5-custom-transformers.md | 2 +- azure_devops/6-migrate.md | 5 ++--- azure_devops/readme.md | 8 ++++---- gitlab/1-configure.md | 3 ++- gitlab/4-dry-run.md | 10 ++++++++-- gitlab/5-custom-transformers.md | 6 +++--- gitlab/6-migrate.md | 5 ++--- jenkins/1-configure.md | 3 ++- jenkins/5-custom-transformers.md | 4 ++-- jenkins/6-migrate.md | 7 ++----- 12 files changed, 35 insertions(+), 30 deletions(-) diff --git a/azure_devops/1-configure.md b/azure_devops/1-configure.md index 0f0fc00..d7645c7 100644 --- a/azure_devops/1-configure.md +++ b/azure_devops/1-configure.md @@ -14,11 +14,11 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Click `Personal access tokens`. - Select `+ New Token` - Name your token, select the organization where you want to use the token, and set your token to automatically expire after a set number of days. - - Select the following scopes (you may need to `Show more scopes` to reveal all scopes): + - Select the following scopes (you may need to `Show all scopes` to reveal all scopes): - Agents Pool: `Read` - - Build: `Read & Execute` - - Code: `Read & Write` - - Project and Team: `Read, Write, & Manage` + - Build: `Read & execute` + - Code: `Read & write` + - Project and Team: `Read, write, & manage` - Release: `Read` - Service Connections: `Read` - Task Groups: `Read` @@ -32,6 +32,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - In the left panel, click `Developer Settings`. - Click `Personal access tokens` and then `Legacy tokens` (if present). - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Name your token in the `Note` field. - Select the following scopes: `workflow` and `read:packages`. - Click `Generate token`. - Copy the generated PAT and save it in a safe location. @@ -40,6 +41,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Select the `TERMINAL` tab from within the codespace terminal. - Run the following command: `gh valet configure`. - Use the down arrow key to highlight `Azure DevOps`, press the spacebar to select, and then press enter to continue. + - At the GitHub handle prompt, enter the GitHub handle used to generate the GitHub PAT in step 2 and press enter. - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 2 and press enter. - At the GitHub PAT prompt, enter the GitHub PAT generated in step 2 and press enter. - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). diff --git a/azure_devops/3-forecast.md b/azure_devops/3-forecast.md index 6d0bd7d..8e9ef35 100644 --- a/azure_devops/3-forecast.md +++ b/azure_devops/3-forecast.md @@ -35,7 +35,7 @@ Answer the following questions before running the `forecast` command: ![img](https://user-images.githubusercontent.com/18723510/187690315-6312088d-9888-4c55-9bbf-c6f2687fa547.png) -4. If you inspect the help menu using the `gh valet forecast --help` command, you will see a `--source-file-path` option. You can use this option to perform a `forecast` command using json files that are already present on the filesystem. These labs come bundled with sample json files located [here](./bootstrap/jobs.json). +4. If you inspect the help menu using the `gh valet forecast --help` command, you will see a `--source-file-path` option. You can use this option to perform a `forecast` using json files that are already present on the filesystem. These labs come bundled with sample json files located [here](./bootstrap/jobs.json). ![img](https://user-images.githubusercontent.com/18723510/187692843-623d4bdc-8970-4348-a632-73c8b00a40f8.png) diff --git a/azure_devops/5-custom-transformers.md b/azure_devops/5-custom-transformers.md index 6d2d0d5..24a3c4d 100644 --- a/azure_devops/5-custom-transformers.md +++ b/azure_devops/5-custom-transformers.md @@ -87,7 +87,7 @@ You can use custom transformers to override Valet's default behavior. In this sc - __DotnetCoreCLI@2__ 2. What is the desired Actions syntax to use instead? - - After some research, you have determined that the uploading test results as an artifact will be suitable: + - After some research, you have determined that the following script will provide the desired functionality: ```yaml - run: shopt -s globstar; for f in ./**/*.csproj; do dotnet build $f --configuration ${{ env.BUILDCONFIGURATION }} ; done diff --git a/azure_devops/6-migrate.md b/azure_devops/6-migrate.md index 551f5af..98e4ed2 100644 --- a/azure_devops/6-migrate.md +++ b/azure_devops/6-migrate.md @@ -7,7 +7,6 @@ In this lab, you will use the `migrate` command to convert an Azure DevOps pipel 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and bootstrap an Azure DevOps project. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./4-dry-run.md). -4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration @@ -21,14 +20,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. ### Steps 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url https://github.com/:owner/:repo --output-dir tmp/migrate + gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url $(git remote get-url origin) --output-dir ./tmp/migrate ``` 2. The command will write the URL to the pull request that was created when the command succeeds. diff --git a/azure_devops/readme.md b/azure_devops/readme.md index ca3805e..6b30624 100644 --- a/azure_devops/readme.md +++ b/azure_devops/readme.md @@ -45,11 +45,11 @@ These steps **must** be completed prior to starting other labs. - Click `Personal access tokens`. - Select `+ New Token` - Name your token, select the organization where you want to use the token, and set your token to automatically expire after a set number of days. - - Select the following scopes (you may need to `Show more scopes` to reveal all scopes): + - Select the following scopes (you may need to `Show all scopes` at the bottom of the page to reveal all scopes): - Agents Pool: `Read` - - Build: `Read & Execute` - - Code: `Read & Write` - - Project and Team: `Read, Write, & Manage` + - Build: `Read & execute` + - Code: `Read & write` + - Project and Team: `Read, write, & manage` - Release: `Read` - Service Connections: `Read` - Task Groups: `Read` diff --git a/gitlab/1-configure.md b/gitlab/1-configure.md index 6a00435..4df7fd6 100644 --- a/gitlab/1-configure.md +++ b/gitlab/1-configure.md @@ -31,6 +31,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - In the left panel, click `Developer Settings`. - Click `Personal access tokens` and then `Legacy tokens` (if present). - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Name your token in the `Note` field. - Select the following scopes: `workflow` and `read:packages`. - Click `Generate token`. - Copy the generated PAT and save it in a safe location. @@ -39,7 +40,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Select the `TERMINAL` tab from within the codespace terminal window. - Run the following command: `gh valet configure`. - Use the down arrow key to highlight `GitLab CI`, press the spacebar to select, and then press enter to continue. - - At the prompt, enter your GitHub username and press enter. + - At the GitHub handle prompt, enter the GitHub handle used to generate the GitHub PAT in step 2 and press enter. - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 4 and press enter. - At the GitHub PAT prompt, enter the GitHub PAT generated in step 4 and press enter. - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). diff --git a/gitlab/4-dry-run.md b/gitlab/4-dry-run.md index 169e925..597e9a3 100644 --- a/gitlab/4-dry-run.md +++ b/gitlab/4-dry-run.md @@ -211,7 +211,13 @@ include: - local: /config/test.gitlab-ci.yml ``` -The output of the `dry-run` command can be seen below: +Run the following command from the root directory: + +```bash +gh valet dry-run gitlab --output-dir tmp/dry-run --namespace valet --project included-files-example +``` + +The output of the command above can be seen below: ```yaml name: valet/included-files-example @@ -246,7 +252,7 @@ jobs: - run: echo "this is from a local file" ``` -It's important to note that Valet converted this into a single workflow without templates. This is because of fundamental differences in how GitLab templates and GitHub Actions templates (i.e. Reusable Workflows and Composite Actions) function in regards to job ordering. Unfortunately, elements of reusability will be sacrificed in order for the converted pipelines to function the same. It is likely that the output of Valet could be refactored to use [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows) at a later date. +It's important to note that Valet converted this into a single workflow without templates. This is because of fundamental differences in how GitLab templates and GitHub Actions templates (i.e. Reusable Workflows and Composite Actions) function in regards to job ordering. Unfortunately, elements of reusability will be sacrificed in order for the converted pipelines to function the same. It is likely that the output of Valet could be refactored to use [reusable workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows) at a later date. As an added challenge, try constructing and running the `dry-run` command yourself. Hint, you should only have to change the project name. diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 5413ed4..6d34709 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -9,7 +9,7 @@ In this lab you will build upon the `dry-run` command to override Valet's defaul ## Prerequisites -1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start you GitLab server. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start your GitLab server. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./4-dry-run.md). @@ -63,7 +63,7 @@ The converted workflow above contains an `artifacts.terraform` step that was not - __artifacts.terraform__ 2. What is the desired Actions syntax to use instead? - - After some research, you have determined that the following bash script will provide similar functionality: + - After some research, you have determined that the following GitHub Action will provide similar functionality: ```yaml - uses: actions/upload-artifact@v3 @@ -82,7 +82,7 @@ Next, you will define a `transform` method for the `artifacts.terraform` identif ```ruby transform "artifacts.terraform" do |item| { - uses: "actions/upload-artifact@v2", + uses: "actions/upload-artifact@v3", with: { path: item } diff --git a/gitlab/6-migrate.md b/gitlab/6-migrate.md index 3dcd734..3bac8ff 100644 --- a/gitlab/6-migrate.md +++ b/gitlab/6-migrate.md @@ -7,7 +7,6 @@ In this lab, you will use the `migrate` command to convert a GitLab pipeline and 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start a GitLab server. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./4-dry-run.md). -4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration @@ -20,14 +19,14 @@ Answer the following questions before running a `migrate` command: 3. Where do you want to store the logs? - __tmp/migrate__ 4. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. + - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. ### Steps 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --namespace valet --project rails-example + gh valet migrate gitlab --target-url $(git remote get-url origin) --output-dir ./tmp/migrate --namespace valet --project rails-example ``` 2. The command will write the URL to the pull request that was created when the command succeeds. diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 8128140..71ede17 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -32,11 +32,12 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - Select the `TERMINAL` tab from within the codespace terminal window. - Run the following command: `gh valet configure`. - Use the down arrow key to highlight `Jenkins`, press the spacebar to select, and then press enter to continue. - - At the prompt, enter your GitHub username and press enter. + - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 3 and press enter. - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 3 and press enter. - At the GitHub PAT prompt, enter the GitHub PAT generated in step 3 and press enter. - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). - At the Jenkins token prompt, enter the Jenkins access token from step 2 and press enter. + - At the Jenkins username prompt, enter `admin` and press enter. - At the Jenkins URL prompt, enter `http://localhost:8080/` and press enter. - At the Personal access token to fetch source code in GitHub prompt, press enter to accept the default value. diff --git a/jenkins/5-custom-transformers.md b/jenkins/5-custom-transformers.md index 281c4a2..c4628e0 100644 --- a/jenkins/5-custom-transformers.md +++ b/jenkins/5-custom-transformers.md @@ -75,7 +75,7 @@ _Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the f ## Custom transformers for an unknown step -The converted workflow above contains a `sleep` step was not automatically converted. Answer the following questions before writing a custom transformer: +The converted workflow above contains a `sleep` step that was not automatically converted. Answer the following questions before writing a custom transformer: 1. What is the "identifier" of the step to customize? - __sleep__. The identifier will be the key of a key value pair within the step of a Jenkinsfile. @@ -198,7 +198,7 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C + - uses: actions/upload-artifact@v3 + with: + name: junit-artifact -+ path: path/to/artifact/world.txt ++ path: "**/target/*.xml" ``` ## Custom transformers for environment variables diff --git a/jenkins/6-migrate.md b/jenkins/6-migrate.md index 773b929..04b8348 100644 --- a/jenkins/6-migrate.md +++ b/jenkins/6-migrate.md @@ -7,7 +7,6 @@ In this lab, you will use the `migrate` command to convert a Jenkins pipeline an 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment and start a Jenkins server. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./4-dry-run.md). -4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration @@ -18,14 +17,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. ### Steps 1. Run the following `migrate` command in your codespace terminal: ```bash - gh valet migrate jenkins --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle + gh valet migrate jenkins --target-url $(git remote get-url origin) --output-dir ./tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle ``` 2. The command will write the URL to the pull request that was created when the command succeeds. @@ -44,8 +43,6 @@ The first thing we should notice about the pull request is that there is a list ![img](https://user-images.githubusercontent.com/19557880/186784161-b7882ac4-ac99-4462-b69f-f49b9202527b.png) -Next, review the workflow you are adding by clicking on the `Files changed` tab. This is where you double check that everything looks good. If it didn't, you could push commits with the required changes, prior to merging. - Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request. Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. From 71786f95274b9a6651049a62eec8f92bf5f8bc97 Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Mon, 19 Sep 2022 17:29:02 -0700 Subject: [PATCH 44/59] Update 6-migrate.md --- azure_devops/6-migrate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/azure_devops/6-migrate.md b/azure_devops/6-migrate.md index 98e4ed2..f64ea2b 100644 --- a/azure_devops/6-migrate.md +++ b/azure_devops/6-migrate.md @@ -27,7 +27,7 @@ Answer the following questions before running a `migrate` command: 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url $(git remote get-url origin) --output-dir ./tmp/migrate + gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url $(git remote get-url origin) --output-dir tmp/migrate ``` 2. The command will write the URL to the pull request that was created when the command succeeds. From ea795e56b470d476f13b795e70d11b4349437807 Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Mon, 19 Sep 2022 17:32:01 -0700 Subject: [PATCH 45/59] Update 6-migrate.md --- gitlab/6-migrate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gitlab/6-migrate.md b/gitlab/6-migrate.md index 3bac8ff..cb96850 100644 --- a/gitlab/6-migrate.md +++ b/gitlab/6-migrate.md @@ -26,7 +26,7 @@ Answer the following questions before running a `migrate` command: 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate gitlab --target-url $(git remote get-url origin) --output-dir ./tmp/migrate --namespace valet --project rails-example + gh valet migrate gitlab --target-url $(git remote get-url origin) --output-dir tmp/migrate --namespace valet --project rails-example ``` 2. The command will write the URL to the pull request that was created when the command succeeds. From 50130ff591aa43285d6b2b3e55aff8a67a79c9de Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Mon, 19 Sep 2022 17:32:26 -0700 Subject: [PATCH 46/59] Update 6-migrate.md --- jenkins/6-migrate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jenkins/6-migrate.md b/jenkins/6-migrate.md index 04b8348..76907de 100644 --- a/jenkins/6-migrate.md +++ b/jenkins/6-migrate.md @@ -24,7 +24,7 @@ Answer the following questions before running a `migrate` command: 1. Run the following `migrate` command in your codespace terminal: ```bash - gh valet migrate jenkins --target-url $(git remote get-url origin) --output-dir ./tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle + gh valet migrate jenkins --target-url $(git remote get-url origin) --output-dir tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle ``` 2. The command will write the URL to the pull request that was created when the command succeeds. From dd9609b42c55484ee8ea1c41b637422edd535a98 Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Tue, 20 Sep 2022 09:33:10 -0700 Subject: [PATCH 47/59] Change 'GitHub Action' to 'action Co-authored-by: Ethan Dennis --- gitlab/5-custom-transformers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 6d34709..98f533a 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -63,7 +63,7 @@ The converted workflow above contains an `artifacts.terraform` step that was not - __artifacts.terraform__ 2. What is the desired Actions syntax to use instead? - - After some research, you have determined that the following GitHub Action will provide similar functionality: + - After some research, you have determined that the following action will provide similar functionality: ```yaml - uses: actions/upload-artifact@v3 From ef4b59aea4c6f504a26d253f2a4cb690950a3afb Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 09:44:41 -0700 Subject: [PATCH 48/59] Remove git remote get-url --- azure_devops/6-migrate.md | 4 ++-- gitlab/6-migrate.md | 4 ++-- jenkins/6-migrate.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/azure_devops/6-migrate.md b/azure_devops/6-migrate.md index f64ea2b..4466b6a 100644 --- a/azure_devops/6-migrate.md +++ b/azure_devops/6-migrate.md @@ -20,14 +20,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url $(git remote get-url origin) --output-dir tmp/migrate + gh valet migrate azure-devops pipeline --pipeline-id :pipeline_id --target-url https://github.com/:owner/:repo --output-dir tmp/migrate ``` 2. The command will write the URL to the pull request that was created when the command succeeds. diff --git a/gitlab/6-migrate.md b/gitlab/6-migrate.md index cb96850..ba561fb 100644 --- a/gitlab/6-migrate.md +++ b/gitlab/6-migrate.md @@ -19,14 +19,14 @@ Answer the following questions before running a `migrate` command: 3. Where do you want to store the logs? - __tmp/migrate__ 4. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. + - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps 1. Run the following `migrate` command in the codespace terminal: ```bash - gh valet migrate gitlab --target-url $(git remote get-url origin) --output-dir tmp/migrate --namespace valet --project rails-example + gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --namespace valet --project rails-example ``` 2. The command will write the URL to the pull request that was created when the command succeeds. diff --git a/jenkins/6-migrate.md b/jenkins/6-migrate.md index 76907de..85a45b7 100644 --- a/jenkins/6-migrate.md +++ b/jenkins/6-migrate.md @@ -17,14 +17,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. The command below uses `git remote get-url` to automatically populate these values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps 1. Run the following `migrate` command in your codespace terminal: ```bash - gh valet migrate jenkins --target-url $(git remote get-url origin) --output-dir tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle + gh valet migrate jenkins --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --source-url http://localhost:8080/job/monas_dev_work/job/monas_freestyle ``` 2. The command will write the URL to the pull request that was created when the command succeeds. From d58cd99cf15137c4ef0085895da03e4584648b38 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 11:49:00 -0700 Subject: [PATCH 49/59] Add more thorough custom transformer instructions --- azure_devops/5-custom-transformers.md | 29 ++++++++++++++++++++--- gitlab/5-custom-transformers.md | 34 ++++++++++++++++++--------- 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/azure_devops/5-custom-transformers.md b/azure_devops/5-custom-transformers.md index 24a3c4d..2a9a3a8 100644 --- a/azure_devops/5-custom-transformers.md +++ b/azure_devops/5-custom-transformers.md @@ -100,7 +100,32 @@ Now you can begin to write the custom transformer. Custom transformers use a DSL touch transformers.rb && code transformers.rb ``` -Next, you will define a `transform` method for the `DotnetCoreCLI@2` identifier by adding the following code to `transformers.rb`: +To build this custom transformer, you first need to inspect the `item` keyword to programmatically obtain the projects, command, and arguments to use in the `DotNetCoreCLI@2` step. + +To do this, you will print `item` to the console. You can achieve this by adding the following custom transformer to `transformers.rb`: + +```ruby +transform "DotNetCoreCLI@2" do |item| + puts "This is the item: #{item}" +end +``` + +The `transform` method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Azure DevOps. + +Now, we can perform a `dry-run` command with the `--custom-transformers` CLI option. The output of the `dry-run` command should look similar to this: + +```console +$ gh valet dry-run azure-devops pipeline --pipeline-id 6 --output-dir tmp/dry-run --custom-transformers transformers.rb +[2022-09-20 18:39:50] Logs: 'tmp/dry-run/log/valet-20220920-183950.log' +This is the item: {"command"=>"restore", "projects"=>"$(BuildParameters.RESTOREBUILDPROJECTS)"} +This is the item: {"projects"=>"$(BuildParameters.RESTOREBUILDPROJECTS)", "arguments"=>"--configuration $(BUILDCONFIGURATION)"} +[2022-09-20 18:39:51] Output file(s): +[2022-09-20 18:39:51] tmp/dry-run/lab-test/pipelines/valet-custom-transformer-example.yml +``` + +In the above command you will see two instances of `item` printed to the console. This is because there are two `DotNetCoreCLI@2` steps in the pipeline. Each item listed above represents each `DotNetCoreCLI@2` step in the order that they are defined in the pipeline. + +Now that you know the data structure of `item`, you can access the dotnet projects, command, and arguments programmatically by editing the custom transformer to the following: ```ruby transform "DotNetCoreCLI@2" do |item| @@ -122,8 +147,6 @@ transform "DotNetCoreCLI@2" do |item| end ``` -This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Azure DevOps. - Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: ```bash diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 98f533a..4ac3624 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -77,7 +77,29 @@ Now you can begin to write the custom transformer. Custom transformers use a DSL touch transformers.rb && code transformers.rb ``` -Next, you will define a `transform` method for the `artifacts.terraform` identifier by adding the following code to `transformers.rb`: +To build this custom transformer, you first need to inspect the `item` keyword to programmatically use the file path of the terraform json in the `actions/upload-artifact@v3` step. + +To do this, you will print `item` to the console. You can achieve this by adding the following custom transformer to `transformers.rb`: + +```ruby +transform "artifacts.terraform" do |item| + puts "This is the item: #{item}" +end +``` + +The `transform` method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in GitLab. + +Now, we can perform a `dry-run` command with the `--custom-transformers` CLI option. The output of the `dry-run` command should look similar to this: + +```console +$ gh valet dry-run gitlab --output-dir tmp --namespace valet --project terraform-example --custom-transformers transformers.rb +[2022-09-20 17:47:55] Logs: 'tmp/log/valet-20220920-174755.log' +This is the item: $PLAN_JSON +[2022-09-20 17:47:56] Output file(s): +[2022-09-20 17:47:56] tmp/valet/terraform-example.yml +``` + +Now that you know the data structure of `item`, you can access the file path programmatically by editing the custom transformer to the following: ```ruby transform "artifacts.terraform" do |item| @@ -90,8 +112,6 @@ transform "artifacts.terraform" do |item| end ``` -This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in GitLab. - Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: ```bash @@ -107,14 +127,6 @@ The converted workflow that is generated by the above command will now use the c + path: "$PLAN_JSON" ``` -_Note_: If you were unsure what the data structure of `item` was, you could use the following code in the custom transformer to print `item` to the console: - -```ruby -transform "artifacts.terraform" do |item| - puts item -end -``` - ## Custom transformers for environment variables You can also use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `PLAN_JSON` environment variable to be `custom_plan.json` instead of `plan.json`. From a11044361ed349f22c0d686fa59f6de597be5311 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 20 Sep 2022 12:05:36 -0700 Subject: [PATCH 50/59] More img -> text --- azure_devops/1-configure.md | 86 +++++++++++++++++++++---------------- circle_ci/1-configure.md | 54 ++++++++++++++--------- gitlab/1-configure.md | 65 ++++++++++++++++------------ jenkins/1-configure.md | 68 +++++++++++++++++------------ 4 files changed, 160 insertions(+), 113 deletions(-) diff --git a/azure_devops/1-configure.md b/azure_devops/1-configure.md index 0f0fc00..e9e4e57 100644 --- a/azure_devops/1-configure.md +++ b/azure_devops/1-configure.md @@ -7,48 +7,60 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ## Configuring credentials 1. Create an Azure DevOps personal access token (PAT). - - __Note__: you can skip this step if you still have the PAT created during the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). - - Navigate to your existing organization () in your browser. - - In the top right corner of your screen, click `User settings`. - - Click `Personal access tokens`. - - Select `+ New Token` - - Name your token, select the organization where you want to use the token, and set your token to automatically expire after a set number of days. - - Select the following scopes (you may need to `Show more scopes` to reveal all scopes): - - Agents Pool: `Read` - - Build: `Read & Execute` - - Code: `Read & Write` - - Project and Team: `Read, Write, & Manage` - - Release: `Read` - - Service Connections: `Read` - - Task Groups: `Read` - - Variable Groups: `Read` - - Click `Create`. - - Copy the generated API token and save it in a safe location. + __Note__: you can skip this step if you still have the PAT created during the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). + - Navigate to your existing organization () in your browser. + - In the top right corner of your screen, click `User settings`. + - Click `Personal access tokens`. + - Select `+ New Token` + - Name your token, select the organization where you want to use the token, and set your token to automatically expire after a set number of days. + - Select the following scopes (you may need to `Show more scopes` to reveal all scopes): + - Agents Pool: `Read` + - Build: `Read & Execute` + - Code: `Read & Write` + - Project and Team: `Read, Write, & Manage` + - Release: `Read` + - Service Connections: `Read` + - Task Groups: `Read` + - Variable Groups: `Read` + - Click `Create`. + - Copy the generated API token and save it in a safe location. 2. Create a GitHub personal access token (PAT): - - Open github.com in a new browser tab. - - In the top right corner of the UI, click your profile photo and then click `Settings`. - - In the left panel, click `Developer Settings`. - - Click `Personal access tokens` and then `Legacy tokens` (if present). - - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. - - Select the following scopes: `workflow` and `read:packages`. - - Click `Generate token`. - - Copy the generated PAT and save it in a safe location. + - Open github.com in a new browser tab. + - In the top right corner of the UI, click your profile photo and then click `Settings`. + - In the left panel, click `Developer Settings`. + - Click `Personal access tokens` and then `Legacy tokens` (if present). + - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Select the following scopes: `workflow` and `read:packages`. + - Click `Generate token`. + - Copy the generated PAT and save it in a safe location. 3. Run the `configure` CLI command: - - Select the `TERMINAL` tab from within the codespace terminal. - - Run the following command: `gh valet configure`. - - Use the down arrow key to highlight `Azure DevOps`, press the spacebar to select, and then press enter to continue. - - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 2 and press enter. - - At the GitHub PAT prompt, enter the GitHub PAT generated in step 2 and press enter. - - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). - - At the Azure DevOps token prompt, enter the access token from step 1 and press enter. - - At the Azure DevOps URL prompt, enter your Azure DevOps URL or press enter to accept the default value (`https://dev.azure.com`). - - At the prompt, enter your Azure DevOps organization name. This should be the same organization used in the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). - - At the prompt, enter your Azure DevOps project name. This should be the same project name used in the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). + - Select the `TERMINAL` tab from within the codespace terminal. + - Run the following command: `gh valet configure`. + - Use the down arrow key to highlight `Azure DevOps`, press the spacebar to select, and then press enter to continue. + - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 2 and press enter. + - At the GitHub PAT prompt, enter the GitHub PAT generated in step 2 and press enter. + - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). + - At the Azure DevOps token prompt, enter the access token from step 1 and press enter. + - At the Azure DevOps URL prompt, enter your Azure DevOps URL or press enter to accept the default value (`https://dev.azure.com`). + - At the prompt, enter your Azure DevOps organization name. This should be the same organization used in the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). + - At the prompt, enter your Azure DevOps project name. This should be the same project name used in the setup steps [here](./readme.md#bootstrap-your-azure-devops-organization). - ![img](https://user-images.githubusercontent.com/18723510/187771230-27c97889-d98a-48f7-af01-c1a9f8df6423.png) + ```console + $ gh valet configure + ✔ Which CI providers are you configuring?: Azure DevOps + Enter the following values (leave empty to omit): + ✔ GitHub handle used to authenticate with the GitHub Container Registry: mona + ✔ Personal access token to authenticate with the GitHub Container Registry: *************** + ✔ Personal access token for GitHub: *************** + ✔ Base url of the GitHub instance: https://github.com + ✔ Personal access token for Azure DevOps: *************** + ✔ Base url of the Azure DevOps instance: https://dev.azure.com + ✔ Azure DevOps organization name: :organization + ✔ Azure DevOps project name: :project + Environment variables successfully updated. + ``` ## Verify your environment diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index 5214d16..16c05de 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -7,32 +7,44 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ## Configuring credentials 1. Create a GitHub Personal Access Token (PAT): - - Open github.com in a new browser tab. - - Click your profile photo in the top right of the UI and click `Settings`. - - Click on `Developer Settings` in the left hand panel. - - Click `Personal Access Tokens` and then `Legacy tokens` (if present). - - Click `Generate new token` and then `Legacy tokens`. You may be required to authenticate with GitHub during this step. - - Select the following scopes: `read:packages` and `workflow`. - - Click `Generate token`. - - Copy the generated PAT and save in a safe location. + - Open github.com in a new browser tab. + - Click your profile photo in the top right of the UI and click `Settings`. + - Click on `Developer Settings` in the left hand panel. + - Click `Personal Access Tokens` and then `Legacy tokens` (if present). + - Click `Generate new token` and then `Legacy tokens`. You may be required to authenticate with GitHub during this step. + - Select the following scopes: `read:packages` and `workflow`. + - Click `Generate token`. + - Copy the generated PAT and save in a safe location. 3. Create a CircleCI personal API token using CircleCI's [documentation](https://circleci.com/docs/managing-api-tokens#creating-a-personal-api-token) and store the token in a safe location. 2. Run the `configure` CLI command: - - Select the `TERMINAL` tab from within the codespace terminal window. - - Run the following command: `gh valet configure`. - - Using the down arrow key to highlight `CircleCI`, press the spacebar to select, and then hit enter to continue. - - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 2 and press enter. - - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub PAT prompt, enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub url prompt, enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). - - At the CircleCI token prompt, enter the CircleCI access token from step 2 and press enter. - - At the CircleCI base url prompt, hit enter to accept the default value (`https://circleci.com`). - - At the CircleCI organization name prompt, enter `valet-labs`. This is the organization we'll be using throughout these labs. - - At the access token to fetch source code in GitHub prompt, enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub instance url containing source code prompt, press enter to accept the default value (`https://github.com`). + - Select the `TERMINAL` tab from within the codespace terminal window. + - Run the following command: `gh valet configure`. + - Using the down arrow key to highlight `CircleCI`, press the spacebar to select, and then hit enter to continue. + - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 2 and press enter. + - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub PAT prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub url prompt, enter the GitHub instance url or hit enter to accept the default value (`https://github.com`). + - At the CircleCI token prompt, enter the CircleCI access token from step 2 and press enter. + - At the CircleCI base url prompt, hit enter to accept the default value (`https://circleci.com`). + - At the CircleCI organization name prompt, enter `valet-labs`. This is the organization we'll be using throughout these labs. + - At the access token to fetch source code in GitHub prompt, enter the GitHub PAT generated in step 1 and press enter. + - At the GitHub instance url containing source code prompt, press enter to accept the default value (`https://github.com`). - ![configure-screenshot](https://user-images.githubusercontent.com/18723510/188935436-308fa6d3-6eb7-48b9-bbf0-20ab0cdae411.png) + ```console + $ gh valet configure + ✔ Which CI providers are you configuring?: CircleCI + Enter the following values (leave empty to omit): + ✔ GitHub handle used to authenticate with the GitHub Container Registry: mona + ✔ Personal access token to authenticate with the GitHub Container Registry: *************** + ✔ Personal access token for GitHub: *************** + ✔ Base url of the GitHub instance: https://github.com + ✔ Personal access token for CircleCI: *************** + ✔ Base url of the CircleCI instance: https://circleci.com + ✔ CircleCI organization name: valet-labs + Environment variables successfully updated. + ``` ## Verify your environment diff --git a/gitlab/1-configure.md b/gitlab/1-configure.md index 6a00435..759c950 100644 --- a/gitlab/1-configure.md +++ b/gitlab/1-configure.md @@ -13,40 +13,51 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ``` 2. Open the GitLab server in a new browser tab: - - Click the `PORTS` tab in the codespace terminal window. - - In the `PORTS` tab find the row for port 80. - - Hover over the address under the `Local Address` column and click the globe to "open in browser". + - Click the `PORTS` tab in the codespace terminal window. + - In the `PORTS` tab find the row for port 80. + - Hover over the address under the `Local Address` column and click the globe to "open in browser". 3. Create a GitLab personal access token (PAT): - - Authenticate with the GitLab server using the following credentials: - - Username: `root` - - Password: `valet-labs!` - - Follow the GitLab [instructions](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token) to generate a PAT. - - Ensure the token has the `read_api` scope. - - Copy the generated token and save it in a safe location. + - Authenticate with the GitLab server using the following credentials: + - Username: `root` + - Password: `valet-labs!` + - Follow the GitLab [instructions](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token) to generate a PAT. + - Ensure the token has the `read_api` scope. + - Copy the generated token and save it in a safe location. 4. Create a GitHub personal access token (PAT): - - Open github.com in a new browser tab. - - In the top right corner of the UI, click your profile photo and click `Settings`. - - In the left panel, click `Developer Settings`. - - Click `Personal access tokens` and then `Legacy tokens` (if present). - - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. - - Select the following scopes: `workflow` and `read:packages`. - - Click `Generate token`. - - Copy the generated PAT and save it in a safe location. + - Open github.com in a new browser tab. + - In the top right corner of the UI, click your profile photo and click `Settings`. + - In the left panel, click `Developer Settings`. + - Click `Personal access tokens` and then `Legacy tokens` (if present). + - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Select the following scopes: `workflow` and `read:packages`. + - Click `Generate token`. + - Copy the generated PAT and save it in a safe location. 5. Run the `configure` CLI command: - - Select the `TERMINAL` tab from within the codespace terminal window. - - Run the following command: `gh valet configure`. - - Use the down arrow key to highlight `GitLab CI`, press the spacebar to select, and then press enter to continue. - - At the prompt, enter your GitHub username and press enter. - - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 4 and press enter. - - At the GitHub PAT prompt, enter the GitHub PAT generated in step 4 and press enter. - - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). - - At the GitLab CI token prompt, enter the GitLab CI access token from step 3 and press enter. - - At the GitLab CI URL prompt, enter `http://localhost` and press enter. + - Select the `TERMINAL` tab from within the codespace terminal window. + - Run the following command: `gh valet configure`. + - Use the down arrow key to highlight `GitLab CI`, press the spacebar to select, and then press enter to continue. + - At the prompt, enter your GitHub username and press enter. + - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 4 and press enter. + - At the GitHub PAT prompt, enter the GitHub PAT generated in step 4 and press enter. + - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). + - At the GitLab CI token prompt, enter the GitLab CI access token from step 3 and press enter. + - At the GitLab CI URL prompt, enter `http://localhost` and press enter. - ![img](https://user-images.githubusercontent.com/18723510/183990474-d0b2559c-d2bf-40d9-ac43-19af53e45329.png) + ```console + $ gh valet configure + ✔ Which CI providers are you configuring?: GitLab CI + Enter the following values (leave empty to omit): + ✔ GitHub handle used to authenticate with the GitHub Container Registry: mona + ✔ Personal access token to authenticate with the GitHub Container Registry: *************** + ✔ Personal access token for GitHub: *************** + ✔ Base url of the GitHub instance: https://github.com + ✔ Private token for GitLab: *************** + ✔ Base url of the GitLab instance: https://gitlab.com + Environment variables successfully updated. + ``` ## Verify your environment diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 8128140..b798761 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -7,40 +7,52 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ## Configuring credentials 1. Open the Jenkins server in a new browser tab: - - Click the `PORTS` tab in the codespace terminal window. - - In the `PORTS` tab, find the row for port 8080. - - Hover over the address under the `Local Address` column and click the globe to "open in browser". + - Click the `PORTS` tab in the codespace terminal window. + - In the `PORTS` tab, find the row for port 8080. + - Hover over the address under the `Local Address` column and click the globe to "open in browser". 2. Create a Jenkins API token: - - In the top right menu bar, click the `admin` button. - - In the left panel, click the `Configure` gear. - - In the `API token` section, click the `Add new Token` button and click `Generate`. - - Copy the generated API token and save it in a safe location. + - In the top right menu bar, click the `admin` button. + - In the left panel, click the `Configure` gear. + - In the `API token` section, click the `Add new Token` button and click `Generate`. + - Copy the generated API token and save it in a safe location. - ![img](https://user-images.githubusercontent.com/19557880/184041667-d06cb7f2-a885-474e-b728-7567314aeaf3.png) + ![img](https://user-images.githubusercontent.com/19557880/184041667-d06cb7f2-a885-474e-b728-7567314aeaf3.png) 3. Create a GitHub personal access token (PAT): - - Open github.com in a new browser tab. - - In the top right corner of the UI, click your profile photo and click `Settings`. - - In the left panel, click `Developer Settings`. - - Click `Personal access tokens` and then `Legacy tokens` (if present). - - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. - - Select the following scopes: `workflow` and `read:packages`. - - Click `Generate token`. - - Copy the generated PAT and save it in a safe location. -4. Run the `configure` CLI command: - - Select the `TERMINAL` tab from within the codespace terminal window. - - Run the following command: `gh valet configure`. - - Use the down arrow key to highlight `Jenkins`, press the spacebar to select, and then press enter to continue. - - At the prompt, enter your GitHub username and press enter. - - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 3 and press enter. - - At the GitHub PAT prompt, enter the GitHub PAT generated in step 3 and press enter. - - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). - - At the Jenkins token prompt, enter the Jenkins access token from step 2 and press enter. - - At the Jenkins URL prompt, enter `http://localhost:8080/` and press enter. - - At the Personal access token to fetch source code in GitHub prompt, press enter to accept the default value. + - Open github.com in a new browser tab. + - In the top right corner of the UI, click your profile photo and click `Settings`. + - In the left panel, click `Developer Settings`. + - Click `Personal access tokens` and then `Legacy tokens` (if present). + - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Select the following scopes: `workflow` and `read:packages`. + - Click `Generate token`. + - Copy the generated PAT and save it in a safe location. - ![img](https://user-images.githubusercontent.com/19557880/184041328-ce54ea22-b0cd-4c84-b02c-10ad7b09ad89.png) +4. Run the `configure` CLI command: + - Select the `TERMINAL` tab from within the codespace terminal window. + - Run the following command: `gh valet configure`. + - Use the down arrow key to highlight `Jenkins`, press the spacebar to select, and then press enter to continue. + - At the prompt, enter your GitHub username and press enter. + - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 3 and press enter. + - At the GitHub PAT prompt, enter the GitHub PAT generated in step 3 and press enter. + - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). + - At the Jenkins token prompt, enter the Jenkins access token from step 2 and press enter. + - At the Jenkins URL prompt, enter `http://localhost:8080/` and press enter. + + ```console + $ gh valet configure + ✔ Which CI providers are you configuring?: Jenkins + Enter the following values (leave empty to omit): + ✔ GitHub handle used to authenticate with the GitHub Container Registry: mona + ✔ Personal access token to authenticate with the GitHub Container Registry: *************** + ✔ Personal access token for GitHub: *************** + ✔ Base url of the GitHub instance: https://github.com + ✔ Personal access token for Jenkins: *************** + ✔ Username of Jenkins user: admin + ✔ Base url of the Jenkins instance: https://localhost + Environment variables successfully updated. + ``` ## Verify your environment From 640df0c0891698297730fe3fbfc8e8ba3088600b Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 12:12:05 -0700 Subject: [PATCH 51/59] Add custom runner example to gitlab --- gitlab/5-custom-transformers.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 4ac3624..215a2cf 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -148,12 +148,42 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C + PLAN_JSON: custom_plan.json ``` +## Custom transformers for runners + +Finally, you can use custom transformers to dictate which runners the converted workflows should use. To do this, answer the following questions: + +1. What is label of the runner in GitLab to update? + - Since there is no `tags` stanza defined in the `terraform-example` project in GitLab, you will need to use the Ruby symbol for the default runner: + + __:default__ + +2. What is the label of the runner in Actions to use instead? + - __custom-runner__ + +With these questions answered, you can add the following code to the `transformers.rb` file: + +```ruby +runner :default, "custom-runner" +``` + +In this example, the first parameter to the `runner` method is the GitLab label and the second is the Actions runner label. + +Now you can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` statement will use the customized runner label: + +```diff +runs-on: +- - ubuntu-latest ++ - custom-runner +``` + At this point, the file contents of `transformers.rb` should match this:
Custom transformers 👇 ```ruby + runner :default, "custom-runner" + env "PLAN_JSON", "custom_plan.json" transform "artifacts.terraform" do |item| From d27d8c45a1025168472f1de752bf03f4b439c558 Mon Sep 17 00:00:00 2001 From: Ethan Dennis Date: Tue, 20 Sep 2022 12:13:40 -0700 Subject: [PATCH 52/59] Update readme.md --- readme.md | 1 + 1 file changed, 1 insertion(+) diff --git a/readme.md b/readme.md index c27d48f..775d800 100644 --- a/readme.md +++ b/readme.md @@ -12,4 +12,5 @@ To get started: - [Migrations from CircleCI to GitHub Actions](/circle_ci/readme.md) - [Migrations from GitLab CI to GitHub Actions](/gitlab/readme.md) - [Migrations from Jenkins to GitHub Actions](/jenkins/readme.md) + - [Migrations from Travis CI to GitHub Actions](/travis/readme.md) 3. Each learning path describes how to configure your codespace, bootstrap a CI/CD environment, and troubleshoot the Valet CLI. From 40bdb2d5474c3afb5fe725f8ae9f7f9f81bd742e Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 13:26:17 -0700 Subject: [PATCH 53/59] Circle edits --- circle_ci/1-configure.md | 4 +--- circle_ci/2-audit.md | 2 +- circle_ci/3-forecast.md | 2 +- circle_ci/5-custom-transformers.md | 6 +++--- circle_ci/6-migrate.md | 1 - 5 files changed, 6 insertions(+), 9 deletions(-) diff --git a/circle_ci/1-configure.md b/circle_ci/1-configure.md index 16c05de..45ff5b3 100644 --- a/circle_ci/1-configure.md +++ b/circle_ci/1-configure.md @@ -29,8 +29,6 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - At the CircleCI token prompt, enter the CircleCI access token from step 2 and press enter. - At the CircleCI base url prompt, hit enter to accept the default value (`https://circleci.com`). - At the CircleCI organization name prompt, enter `valet-labs`. This is the organization we'll be using throughout these labs. - - At the access token to fetch source code in GitHub prompt, enter the GitHub PAT generated in step 1 and press enter. - - At the GitHub instance url containing source code prompt, press enter to accept the default value (`https://github.com`). ```console $ gh valet configure @@ -59,7 +57,7 @@ To verify our environment is configured correctly, we are going to run the `upda 2. You should see a confirmation that you were logged into the GitHub Container Registry and Valet was updated to the latest version. ```console - $ gh valet version + $ gh valet update Login Succeeded latest: Pulling from valet-customers/valet-cli Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 1e954ef..30e842b 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -34,7 +34,7 @@ You will be performing an `audit` for the __valet-labs__ CircleCI organization t 1. Find the `audit_summary.md` file in the file explorer. 2. Right-click the `audit_summary.md` file and select `Open Preview`. -3. This file contains details that summarizes what percentage of your pipelines were converted automatically. +3. This file contains details that summarize what percentage of your pipelines were converted automatically. ### Review audit summary diff --git a/circle_ci/3-forecast.md b/circle_ci/3-forecast.md index a798adc..a4f5c53 100644 --- a/circle_ci/3-forecast.md +++ b/circle_ci/3-forecast.md @@ -12,7 +12,7 @@ In this lab you will use the `forecast` command to forecast potential GitHub Act Answer the following questions before running the `forecast` command: 1. What is the date you want to start forecasting from? - - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. + - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were triggered in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. 2. Where do you want to store the results? - **tmp/forecast_reports** diff --git a/circle_ci/5-custom-transformers.md b/circle_ci/5-custom-transformers.md index 04a7d05..21523a4 100644 --- a/circle_ci/5-custom-transformers.md +++ b/circle_ci/5-custom-transformers.md @@ -64,7 +64,7 @@ _Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the f ## Custom transformers for an unknown step -The converted workflow above contains an `codecov_codecov_upload` step that was not automatically converted. Answer the following questions before writing a custom transformer: +The converted workflow above contains a `codecov_codecov_upload` step that was not automatically converted. Answer the following questions before writing a custom transformer: 1. What is the "identifier" of the step to customize? - __codecov_codecov_upload__ @@ -127,7 +127,7 @@ end ## Custom transformers for environment variables -You can use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `tmp/cov`. +You can use custom transformers to edit the values of environment variables in converted workflows. In this example, you will update the `COVERAGE_DIR` environment variable to be `$RUNNER_TEMP/cov` instead of `./tmp/cov`. To do this, add the following code to the `transformers.rb` file. @@ -191,7 +191,7 @@ That's it! Congratulations, you have overridden Valet's default behavior by cust - Unknown steps - Environment variables -- Runner +- Runners ## Next lab diff --git a/circle_ci/6-migrate.md b/circle_ci/6-migrate.md index ee136a9..915e6bc 100644 --- a/circle_ci/6-migrate.md +++ b/circle_ci/6-migrate.md @@ -7,7 +7,6 @@ In this lab, you will use the `migrate` command to convert a CircleCI pipeline a 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./4-dry-run.md). -4. Completed the [custom transformers lab](./5-custom-transformers.md). ## Performing a migration From 285053ac1edba680d7cd83ae04bbe160ab80b122 Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Tue, 20 Sep 2022 13:32:02 -0700 Subject: [PATCH 54/59] Update gitlab/5-custom-transformers.md Co-authored-by: Ethan Dennis --- gitlab/5-custom-transformers.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 15b30e5..9b29f96 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -153,9 +153,7 @@ Now you can perform another `dry-run` command with the `--custom-transformers` C Finally, you can use custom transformers to dictate which runners the converted workflows should use. To do this, answer the following questions: 1. What is label of the runner in GitLab to update? - - Since there is no `tags` stanza defined in the `terraform-example` project in GitLab, you will need to use the Ruby symbol for the default runner: - - __:default__ + - __:default__. This is a special keyword to define the default runner to use. You can optional target specific `tags` in a job. 2. What is the label of the runner in Actions to use instead? - __custom-runner__ From a4c13b43f52888b079404b962c89accfe91c4b84 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 13:44:12 -0700 Subject: [PATCH 55/59] Update circle upload-artifact version --- gitlab/5-custom-transformers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gitlab/5-custom-transformers.md b/gitlab/5-custom-transformers.md index 9b29f96..dde61e0 100644 --- a/gitlab/5-custom-transformers.md +++ b/gitlab/5-custom-transformers.md @@ -122,7 +122,7 @@ The converted workflow that is generated by the above command will now use the c ```diff - # # 'artifacts.terraform' was not transformed because there is no suitable equivalent in GitHub Actions -+ uses: actions/upload-artifact@v2 ++ uses: actions/upload-artifact@v3 + with: + path: "$PLAN_JSON" ``` From 830c9084ef586ed69be210a8a058021fd077fd00 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 14:26:28 -0700 Subject: [PATCH 56/59] Add workflow usage csv overview --- azure_devops/2-audit.md | 30 +++++++++++++++++++++++++++ circle_ci/2-audit.md | 37 +++++++++++++++++++++++++++++++++ gitlab/1-configure.md | 2 +- gitlab/2-audit.md | 45 +++++++++++++++++++++++++++++++++++++++++ jenkins/2-audit.md | 36 +++++++++++++++++++++++++++++++++ 5 files changed, 149 insertions(+), 1 deletion(-) diff --git a/azure_devops/2-audit.md b/azure_devops/2-audit.md index d59e0d9..4ad5f81 100644 --- a/azure_devops/2-audit.md +++ b/azure_devops/2-audit.md @@ -173,6 +173,36 @@ Each pipeline will have a variety of files written that include: - The converted workflow. - Stack traces that can used to troubleshoot a failed pipeline conversion +## Inspect the workflow usage csv file + +1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer. +2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline: + + ```csv + Pipeline,Action,File path + lab-test/pipelines/valet-pipeline2,actions/checkout@v2,/data/tmp/adoa/lab-test/pipelines/valet-pipeline2.yml + lab-test/pipelines/valet-pipeline1,actions/checkout@v2,/data/tmp/adoa/lab-test/pipelines/valet-pipeline1.yml + lab-test/pipelines/valet-custom-transformer-example,actions/checkout@v2,/data/tmp/adoa/lab-test/pipelines/valet-custom-transformer-example.yml + lab-test/pipelines/valet-custom-transformer-example,actions/setup-node@v2,/data/tmp/adoa/lab-test/pipelines/valet-custom-transformer-example.yml + + Pipeline,Secret,File path + + + Pipeline,Runner,File path + lab-test/pipelines/valet-pipeline2,mechamachine,/data/tmp/adoa/lab-test/pipelines/valet-pipeline2.yml + lab-test/pipelines/valet-custom-transformer-example,mechamachine,/data/tmp/adoa/lab-test/pipelines/valet-custom-transformer-example.yml + ``` + +### Review workflow usage csv file + +#### Actions section + +The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. + +#### Secrets and Runners sections + +The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. + ### Next lab [Forecast potential build runner usage](3-forecast.md) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 30e842b..367c84b 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -191,6 +191,43 @@ Each pipeline will have a variety of files written that include: - The converted workflow. - Stack traces that can used to troubleshoot a failed pipeline conversion +## Inspect the workflow usage csv file + +1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer. +2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline: + + ```csv + Pipeline,Action,File path + valet-labs/circleci-hello-world,actions/checkout@v2,/data/tmp/audit/valet-labs/circleci-hello-world/say-hello-workflow.yml + valet-labs/circleci-command-example,./.github/actions/greeting,/data/tmp/audit/valet-labs/circleci-command-example/my-workflow.yml + valet-labs/circleci-node-example,actions/checkout@v2,/data/tmp/audit/valet-labs/circleci-node-example/sample.yml + valet-labs/circleci-node-example,actions/cache@v2,/data/tmp/audit/valet-labs/circleci-node-example/sample.yml + valet-labs/circleci-python-example,actions/checkout@v2,/data/tmp/audit/valet-labs/circleci-python-example/sample.yml + valet-labs/circleci-python-example,actions/cache@v2,/data/tmp/audit/valet-labs/circleci-python-example/sample.yml + valet-labs/circleci-demo-java-spring,actions/checkout@v2,/data/tmp/audit/valet-labs/circleci-demo-java-spring/workflow.yml + valet-labs/circleci-demo-java-spring,actions/cache@v2,/data/tmp/audit/valet-labs/circleci-demo-java-spring/workflow.yml + valet-labs/circleci-demo-java-spring,actions/upload-artifact@v2,/data/tmp/audit/valet-labs/circleci-demo-java-spring/workflow.yml + valet-labs/circleci-demo-java-spring,actions/download-artifact@v2,/data/tmp/audit/valet-labs/circleci-demo-java-spring/workflow.yml + valet-labs/circleci-demo-ruby-rails,ruby/setup-ruby@v1,/data/tmp/audit/valet-labs/circleci-demo-ruby-rails/build_and_test.yml + valet-labs/circleci-demo-ruby-rails,actions/checkout@v2,/data/tmp/audit/valet-labs/circleci-demo-ruby-rails/build_and_test.yml + valet-labs/circleci-demo-ruby-rails,actions/cache@v2,/data/tmp/audit/valet-labs/circleci-demo-ruby-rails/build_and_test.yml + + Pipeline,Secret,File path + + + Pipeline,Runner,File path + ``` + +### Review workflow usage csv file + +#### Actions section + +The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. + +#### Secrets and Runners sections + +The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. + ### Next lab [Forecast potential build runner usage](3-forecast.md) diff --git a/gitlab/1-configure.md b/gitlab/1-configure.md index b573429..091ac34 100644 --- a/gitlab/1-configure.md +++ b/gitlab/1-configure.md @@ -56,7 +56,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi ✔ Personal access token for GitHub: *************** ✔ Base url of the GitHub instance: https://github.com ✔ Private token for GitLab: *************** - ✔ Base url of the GitLab instance: https://gitlab.com + ✔ Base url of the GitLab instance: http://localhost Environment variables successfully updated. ``` diff --git a/gitlab/2-audit.md b/gitlab/2-audit.md index 3527bf9..09db3b8 100644 --- a/gitlab/2-audit.md +++ b/gitlab/2-audit.md @@ -181,6 +181,51 @@ Each pipeline will have a variety of files written that include: - The converted workflow. - Stack traces that can be used to troubleshoot a failed pipeline conversion +## Inspect the workflow usage csv file + +1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer. +2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline: + + ```csv + Pipeline,Action,File path + valet/included-files-example,actions/checkout@v2,/data/tmp/audit/valet/included-files-example.yml + valet/terraform-example,actions/checkout@v2,/data/tmp/audit/valet/terraform-example.yml + valet/child-parent-example,actions/checkout@v2,/data/tmp/audit/valet/child-parent-example.yml + valet/child-parent-example,./.github/workflows/a-.gitlab-ci.yml,/data/tmp/audit/valet/child-parent-example.yml + valet/child-parent-example,./.github/workflows/b-.gitlab-ci.yml,/data/tmp/audit/valet/child-parent-example.yml + valet/include-file-example,actions/checkout@v2,/data/tmp/audit/valet/include-file-example.yml + valet/basic-pipeline-example,actions/checkout@v2,/data/tmp/audit/valet/basic-pipeline-example.yml + valet/gatsby-example,actions/checkout@v2,/data/tmp/audit/valet/gatsby-example.yml + valet/gatsby-example,actions/cache@v2,/data/tmp/audit/valet/gatsby-example.yml + valet/gatsby-example,actions/upload-artifact@v2,/data/tmp/audit/valet/gatsby-example.yml + valet/gatsby-example,actions/download-artifact@v2,/data/tmp/audit/valet/gatsby-example.yml + valet/gatsby-example,JamesIves/github-pages-deploy-action@4.1.5,/data/tmp/audit/valet/gatsby-example.yml + valet/android-example,actions/checkout@v2,/data/tmp/audit/valet/android-example.yml + valet/android-example,actions/upload-artifact@v2,/data/tmp/audit/valet/android-example.yml + valet/android-example,actions/download-artifact@v2,/data/tmp/audit/valet/android-example.yml + valet/dotnet-example,actions/checkout@v2,/data/tmp/audit/valet/dotnet-example.yml + valet/dotnet-example,actions/upload-artifact@v2,/data/tmp/audit/valet/dotnet-example.yml + valet/dotnet-example,actions/download-artifact@v2,/data/tmp/audit/valet/dotnet-example.yml + valet/node-example,actions/checkout@v2,/data/tmp/audit/valet/node-example.yml + valet/node-example,actions/cache@v2,/data/tmp/audit/valet/node-example.yml + valet/rails-example,actions/checkout@v2,/data/tmp/audit/valet/rails-example.yml + + Pipeline,Secret,File path + valet/rails-example,${{ secrets.PASSWORD }},/data/tmp/audit/valet/rails-example.yml + + Pipeline,Runner,File path + ``` + +### Review workflow usage csv file + +#### Actions section + +The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. + +#### Secrets and Runners sections + +The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. + ### Next lab [Forecast potential build runner usage](3-forecast.md) diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index 8436d9b..050833a 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -238,6 +238,42 @@ Each pipeline will have a variety of files written that include: - The converted workflow. - Stack traces that can used to troubleshoot a failed pipeline conversion +## Inspect the workflow usage csv file + +1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer. +2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline: + + ```csv + Pipeline,Action,File path + demo_pipeline,actions/checkout@v2,/data/tmp/audit/demo_pipeline.yml + demo_pipeline,actions/upload-artifact@v2,/data/tmp/audit/demo_pipeline.yml + demo_pipeline,EnricoMi/publish-unit-test-result-action@v1.7,/data/tmp/audit/demo_pipeline.yml + monas_dev_work/monas_freestyle,actions/checkout@v2,/data/tmp/audit/monas_dev_work/monas_freestyle.yml + monas_dev_work/monas_pipeline,actions/checkout@v2,/data/tmp/audit/monas_dev_work/monas_pipeline.yml + test_freestyle_project,actions/checkout@v2,/data/tmp/audit/test_freestyle_project.yml + test_pipeline,actions/checkout@v2,/data/tmp/audit/test_pipeline.yml + test_pipeline,EnricoMi/publish-unit-test-result-action@v1.7,/data/tmp/audit/test_pipeline.yml + + Pipeline,Secret,File path + monas_dev_work/monas_freestyle,${{ secrets.SECRET_TEST_EXPRESSION_VAR }},/data/tmp/audit/monas_dev_work/monas_freestyle.yml + test_freestyle_project,${{ secrets.EXPRESSION_FIRST_VAR }},/data/tmp/audit/test_freestyle_project.yml + + Pipeline,Runner,File path + demo_pipeline,TeamARunner,/data/tmp/audit/demo_pipeline.yml + test_freestyle_project,DemoRunner,/data/tmp/audit/test_freestyle_project.yml + test_pipeline,TeamARunner,/data/tmp/audit/test_pipeline.yml + ``` + +### Review workflow usage csv file + +#### Actions section + +The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. + +#### Secrets and Runners sections + +The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your successfully converted pipelines. + ## Next lab [Forecast potential build runner usage](3-forecast.md) From 197fcd7d1c64e78995722108d59212a8dae85e69 Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 15:01:34 -0700 Subject: [PATCH 57/59] Add travis edits --- jenkins/1-configure.md | 1 + travis/1-configure.md | 5 ++++- travis/2-audit.md | 35 ++++++++++++++++++++++++++++++++- travis/3-forecast.md | 2 +- travis/4-dry-run.md | 2 +- travis/5-custom-transformers.md | 6 +++--- travis/6-migrate.md | 1 - 7 files changed, 44 insertions(+), 8 deletions(-) diff --git a/jenkins/1-configure.md b/jenkins/1-configure.md index 35264ca..738e6d5 100644 --- a/jenkins/1-configure.md +++ b/jenkins/1-configure.md @@ -25,6 +25,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - In the left panel, click `Developer Settings`. - Click `Personal access tokens` and then `Legacy tokens` (if present). - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Name your token in the `Note` field. - Select the following scopes: `workflow` and `read:packages`. - Click `Generate token`. - Copy the generated PAT and save it in a safe location. diff --git a/travis/1-configure.md b/travis/1-configure.md index c0d6dfb..3685491 100644 --- a/travis/1-configure.md +++ b/travis/1-configure.md @@ -12,6 +12,7 @@ You will need to complete all of the setup instructions [here](./readme.md#confi - In the left panel, click `Developer Settings`. - Click `Personal access tokens` and then `Legacy tokens` (if present). - Click `Generate new token` and then `Generate new legacy token`. You may be required to authenticate with GitHub during this step. + - Name your token in the `Note` field. - Select the following scopes: `workflow` and `read:packages`. - Click `Generate token`. - Copy the generated PAT and save it in a safe location. @@ -19,13 +20,15 @@ You will need to complete all of the setup instructions [here](./readme.md#confi 3. Create a Travis CI personal access token (PAT): - Open app.travis-ci.com in a new browser tab. - Click on your profile icon in the top right hand corner to reveal a dropdown menu. + - Click `Settings`. - Click on the `Settings` tab. - - Click on the `COPY TOKEN` button under "API authentication header" and save it in a safe location. + - Click on the `COPY TOKEN` button under the "API authentication" header and save it in a safe location. 2. Run the `configure` CLI command: - Select the `TERMINAL` tab from within the codespace terminal. - Run the following command: `gh valet configure`. - Use the down arrow key to highlight `Travis CI`, press the spacebar to select, and then press enter to continue. + - At the GitHub handle prompt, enter the GitHub username used to generate the GitHub PAT in step 3 and press enter. - At the GitHub Container Registry prompt, enter the GitHub PAT generated in step 2 and press enter. - At the GitHub PAT prompt, enter the GitHub PAT generated in step 2 and press enter. - At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`). diff --git a/travis/2-audit.md b/travis/2-audit.md index a7bf042..1c288b0 100644 --- a/travis/2-audit.md +++ b/travis/2-audit.md @@ -15,7 +15,7 @@ The `audit` command operates by performing the following: ## Perform an audit -You will be performing an audit against the **labs-data** Travis CI organization that was created for the purposes of this lab. Your environment was configured to use this organization during the [configure lab](./1-configure.md). The remaining information needed to perform an `audit` is: +You will be performing an audit against the **valet-labs** Travis CI organization that was created for the purposes of this lab. Your environment was configured to use this organization during the [configure lab](./1-configure.md). The remaining information needed to perform an `audit` is: 1. Where do we want to store the result? - **tmp/audit**. This can be any path within the working directory that Valet commands are executed from. @@ -316,6 +316,39 @@ Each pipeline will have a variety of files written that include: - The converted workflow. - Stack traces that can used to troubleshoot a failed pipeline conversion +## Inspect the workflow usage csv file + +1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer. +2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline: + + ```csv + Pipeline,Action,File path + valet-labs/travisci-php-example,actions/checkout@v2,/data/tmp/audit/valet-labs/travisci-php-example.yml + valet-labs/travisci-php-example,shivammathur/setup-php@v2,/data/tmp/audit/valet-labs/travisci-php-example.yml + valet-labs/travisci-php-example,rectalogic/notify-irc@v1,/data/tmp/audit/valet-labs/travisci-php-example.yml + valet-labs/travisci-nodejs-example,actions/checkout@v2,/data/tmp/audit/valet-labs/travisci-nodejs-example.yml + valet-labs/travisci-ruby-example,actions/checkout@v2,/data/tmp/audit/valet-labs/travisci-ruby-example.yml + valet-labs/travisci-ruby-example,ruby/setup-ruby@v1,/data/tmp/audit/valet-labs/travisci-ruby-example.yml + valet-labs/travisci-python-example,actions/checkout@v2,/data/tmp/audit/valet-labs/travisci-python-example.yml + valet-labs/travisci-deploy-example,actions/checkout@v2,/data/tmp/audit/valet-labs/travisci-deploy-example.yml + valet-labs/travisci-deploy-example,desiderati/github-action-pushover@v1,/data/tmp/audit/valet-labs/travisci-deploy-example.yml + + Pipeline,Secret,File path + valet-labs/travisci-deploy-example,${{ secrets.PUSHOVER_USER_KEY }},/data/tmp/audit/valet-labs/travisci-deploy-example.yml + + Pipeline,Runner,File path + ``` + +### Review workflow usage csv file + +#### Actions section + +The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. + +#### Secrets and Runners sections + +The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your successfully converted pipelines. + ### Next lab [Forecast potential build runner usage](3-forecast.md) diff --git a/travis/3-forecast.md b/travis/3-forecast.md index f62f883..3efeeb8 100644 --- a/travis/3-forecast.md +++ b/travis/3-forecast.md @@ -12,7 +12,7 @@ In this lab you will use the `forecast` command to forecast potential GitHub Act Answer the following questions before running the `forecast` command: 1. What is the date you want to start forecasting from? - - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo Travis CI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. + - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were triggered in the demo Travis CI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. 2. Where do you want to store the results? - **tmp/forecast** diff --git a/travis/4-dry-run.md b/travis/4-dry-run.md index f30a77f..982e907 100644 --- a/travis/4-dry-run.md +++ b/travis/4-dry-run.md @@ -37,7 +37,7 @@ You will be performing a dry-run against a TravisCI project. Answer the followin ``` 4. View the converted workflow: - - Find `tmp/dry-run` in the file explorer pane in your codespace. + - Find `tmp/dry-run/valet-labs` in the file explorer pane in your codespace. - Click `travisci-ruby-example.yml` to open. ## Inspect the output files diff --git a/travis/5-custom-transformers.md b/travis/5-custom-transformers.md index 6531102..f2b7101 100644 --- a/travis/5-custom-transformers.md +++ b/travis/5-custom-transformers.md @@ -126,7 +126,7 @@ This method can use any valid ruby syntax and should return a `Hash`, or an arra Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: ```bash - gh valet dry-run travis-ci --travis-ci-repository "deploy-example" --output-dir tmp/travis/dry-run --custom-transformers transformers.rb + gh valet dry-run travis-ci --travis-ci-repository "travisci-deploy-example" --output-dir tmp/travis/dry-run --custom-transformers transformers.rb ``` The converted workflow that is generated by the above command will now use the custom logic for the `codedeploy` step. @@ -188,8 +188,8 @@ Finally, you can use custom transformers to dictate which runners converted work 1. What is the label of the runner in Travis to update? - __linux__ -2. What is the label of the runner in Actions to use instead? - - __new-runner__ +2. What is/are the label(s) of the runner(s) in Actions to use instead? + - __new-runner__, __self-hosted__ With these questions answered, you can add the following code to the `transformers.rb` file: diff --git a/travis/6-migrate.md b/travis/6-migrate.md index c701903..f6219b0 100644 --- a/travis/6-migrate.md +++ b/travis/6-migrate.md @@ -7,7 +7,6 @@ In this lab, you will use the `migrate` command to convert a Travis CI pipeline 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./3-dry-run.md). -4. Completed the [custom transformers lab](./4-custom-transformers.md). ## Performing a migration From a7a010c125fdfb65a6cf875005ab12208a74cb9c Mon Sep 17 00:00:00 2001 From: Luke Cheung Engle <99493186+luke-engle@users.noreply.github.com> Date: Tue, 20 Sep 2022 15:23:49 -0700 Subject: [PATCH 58/59] Update circle_ci/2-audit.md Co-authored-by: Ethan Dennis --- circle_ci/2-audit.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 367c84b..f19fa9f 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -217,16 +217,12 @@ Each pipeline will have a variety of files written that include: Pipeline,Runner,File path ``` +The contents of this file can be useful in answering questions similar to the following: -### Review workflow usage csv file - -#### Actions section - -The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. - -#### Secrets and Runners sections - -The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. +- What workflows will depend on which actions? +- What workflows use an action that must go through a security review? +- What workflows use specific secrets? +- What workflows use specific runners? ### Next lab From 8ab9e6c5de115dff3f8a3c8333f136f647922b0b Mon Sep 17 00:00:00 2001 From: Luke Engle Date: Tue, 20 Sep 2022 15:26:39 -0700 Subject: [PATCH 59/59] Update workflow usage structure --- azure_devops/2-audit.md | 14 +++++--------- circle_ci/2-audit.md | 1 + gitlab/2-audit.md | 14 +++++--------- jenkins/2-audit.md | 14 +++++--------- travis/2-audit.md | 14 +++++--------- 5 files changed, 21 insertions(+), 36 deletions(-) diff --git a/azure_devops/2-audit.md b/azure_devops/2-audit.md index 4ad5f81..aa2fc5e 100644 --- a/azure_devops/2-audit.md +++ b/azure_devops/2-audit.md @@ -193,15 +193,11 @@ Each pipeline will have a variety of files written that include: lab-test/pipelines/valet-custom-transformer-example,mechamachine,/data/tmp/adoa/lab-test/pipelines/valet-custom-transformer-example.yml ``` -### Review workflow usage csv file - -#### Actions section - -The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. - -#### Secrets and Runners sections - -The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. +The contents of this file can be useful in answering questions similar to the following: +- What workflows will depend on which actions? +- What workflows use an action that must go through a security review? +- What workflows use specific secrets? +- What workflows use specific runners? ### Next lab diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index f19fa9f..9bb9b64 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -217,6 +217,7 @@ Each pipeline will have a variety of files written that include: Pipeline,Runner,File path ``` + The contents of this file can be useful in answering questions similar to the following: - What workflows will depend on which actions? diff --git a/gitlab/2-audit.md b/gitlab/2-audit.md index 09db3b8..a5e6b14 100644 --- a/gitlab/2-audit.md +++ b/gitlab/2-audit.md @@ -216,15 +216,11 @@ Each pipeline will have a variety of files written that include: Pipeline,Runner,File path ``` -### Review workflow usage csv file - -#### Actions section - -The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. - -#### Secrets and Runners sections - -The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your pipelines. +The contents of this file can be useful in answering questions similar to the following: +- What workflows will depend on which actions? +- What workflows use an action that must go through a security review? +- What workflows use specific secrets? +- What workflows use specific runners? ### Next lab diff --git a/jenkins/2-audit.md b/jenkins/2-audit.md index 050833a..56e3e7a 100644 --- a/jenkins/2-audit.md +++ b/jenkins/2-audit.md @@ -264,15 +264,11 @@ Each pipeline will have a variety of files written that include: test_pipeline,TeamARunner,/data/tmp/audit/test_pipeline.yml ``` -### Review workflow usage csv file - -#### Actions section - -The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. - -#### Secrets and Runners sections - -The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your successfully converted pipelines. +The contents of this file can be useful in answering questions similar to the following: +- What workflows will depend on which actions? +- What workflows use an action that must go through a security review? +- What workflows use specific secrets? +- What workflows use specific runners? ## Next lab diff --git a/travis/2-audit.md b/travis/2-audit.md index 1c288b0..4251bae 100644 --- a/travis/2-audit.md +++ b/travis/2-audit.md @@ -339,15 +339,11 @@ Each pipeline will have a variety of files written that include: Pipeline,Runner,File path ``` -### Review workflow usage csv file - -#### Actions section - -The actions section contains a list of all actions that are used by each converted pipeline. This will enable you to perform strict security reviews to know exactly which repo will need access to a third-party action. - -#### Secrets and Runners sections - -The secrets and runners sections contain a list of all secrets and runners that are used by each converted pipeline. This will enable you to have a holistic view of what secrets and runners are used across all of your successfully converted pipelines. +The contents of this file can be useful in answering questions similar to the following: +- What workflows will depend on which actions? +- What workflows use an action that must go through a security review? +- What workflows use specific secrets? +- What workflows use specific runners? ### Next lab