From 885c5665746dc94fdfa8a096cd94cb5a2c89673f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:37:57 -0400 Subject: [PATCH 1/9] Create 3-dry-run.md --- circle_ci/3-dry-run.md | 63 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 circle_ci/3-dry-run.md diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md new file mode 100644 index 0000000..8d3c410 --- /dev/null +++ b/circle_ci/3-dry-run.md @@ -0,0 +1,63 @@ +# Perform a dry-run of a CircleCI pipeline + +In this lab you will use the `dry-run` command to convert a CircleCI pipeline to its equivalent GitHub Actions workflow. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [audit lab](./2-audit.md). + +## Perform a dry run + +We will be performing a dry-run against a CircleCI project. We will need to answer the following questions before running this command: + +1. What is the name of the project we want to convert? + - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. + +2. Where do we want to store the result? + - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. + +### Steps + +1. Navigate to the codespace terminal +2. Run the following command from the root directory: + + ```bash + gh valet dry-run circle-ci --output-dir ./tmp/dry-run-lab --circle-ci-project circleci-demo-ruby-rails + ``` + +3. The command will list all the files written to disk when the command succeeds. + + ADD_IMAGE_HERE + +4. View the converted workflow: + - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in codespaces. + - Click `build_and_test.yml` to open. + +## Inspect the output files + +The files generated from the `dry-run` command represent the equivalent Actions workflow for the CircleCI project. The CircleCI configuration and converted workflow can be seen below: + +
+ CircleCI configuration 👇 + +```yaml +# ADD YAML +``` + +
+ +
+ Converted workflow 👇 + +```yaml +# ADD YAML +``` +
+ +Despite these 2 pipelines using different syntax they will function equivalently. + +## Next lab + +[Use custom transformers to customize Valet's behavior](./4-custom-transformers.md) From a3ac50c62535bf16612159aa19e745fb0204f252 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:38:16 -0400 Subject: [PATCH 2/9] Update 2-audit.md --- circle_ci/2-audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/2-audit.md b/circle_ci/2-audit.md index 9fb1ee7..a102e6c 100644 --- a/circle_ci/2-audit.md +++ b/circle_ci/2-audit.md @@ -190,4 +190,4 @@ Each pipeline will have a variety of files written that include: ### Next lab -[Perform a dry-run of a GitLab pipeline](3-dry-run.md) +[Perform a dry-run of a CircleCI pipeline](3-dry-run.md) From 070a6d691020ab8b608f1271285aa2edeb5b8dd0 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Mon, 12 Sep 2022 10:48:13 -0400 Subject: [PATCH 3/9] Update 3-dry-run.md --- circle_ci/3-dry-run.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md index 8d3c410..efb6423 100644 --- a/circle_ci/3-dry-run.md +++ b/circle_ci/3-dry-run.md @@ -5,7 +5,7 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [audit lab](./2-audit.md). ## Perform a dry run From 3a6c88ba20bc27e36016e2ca7bfda655aa387f83 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 09:22:22 -0400 Subject: [PATCH 4/9] Update 3-dry-run.md --- circle_ci/3-dry-run.md | 150 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 140 insertions(+), 10 deletions(-) diff --git a/circle_ci/3-dry-run.md b/circle_ci/3-dry-run.md index efb6423..4aa0dc3 100644 --- a/circle_ci/3-dry-run.md +++ b/circle_ci/3-dry-run.md @@ -4,23 +4,23 @@ In this lab you will use the `dry-run` command to convert a CircleCI pipeline to ## Prerequisites -1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and start a GitLab server. +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment. 2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [audit lab](./2-audit.md). ## Perform a dry run -We will be performing a dry-run against a CircleCI project. We will need to answer the following questions before running this command: +You will be performing a dry-run against a CircleCI project. Answer the following questions before running this command: -1. What is the name of the project we want to convert? +1. What project do you want to convert? - __circleci-demo-ruby-rails__. This is one of the sample projects avaiable in the CircleCI labs-data organization. -2. Where do we want to store the result? +2. Where do you want to store the result? - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. ### Steps -1. Navigate to the codespace terminal +1. Navigate to your codespace terminal 2. Run the following command from the root directory: ```bash @@ -29,10 +29,12 @@ We will be performing a dry-run against a CircleCI project. We will need to answ 3. The command will list all the files written to disk when the command succeeds. - ADD_IMAGE_HERE + ![command-result](https://user-images.githubusercontent.com/18723510/189911131-bf6bfd6f-2b5e-4e49-8d14-95ef9c312117.png) + + 4. View the converted workflow: - - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in codespaces. + - Find `./tmp/dry-run-lab/labs-data/circleci-demo-ruby-rails` in the file explorer pane in your codespace. - Click `build_and_test.yml` to open. ## Inspect the output files @@ -43,7 +45,64 @@ The files generated from the `dry-run` command represent the equivalent Actions CircleCI configuration 👇 ```yaml -# ADD YAML +version: 2.1 + +orbs: + ruby: circleci/ruby@1.1.0 + node: circleci/node@2 + +jobs: + build: + docker: + - image: cimg/ruby:2.7.5-node + steps: + - checkout + - ruby/install-deps + # Store bundle cache + - node/install-packages: + pkg-manager: yarn + cache-key: "yarn.lock" + test: + parallelism: 3 + docker: + - image: cimg/ruby:2.7.5-node + - image: circleci/postgres:9.5-alpine + environment: + POSTGRES_USER: circleci-demo-ruby + POSTGRES_DB: rails_blog_test + POSTGRES_PASSWORD: "" + environment: + BUNDLE_JOBS: "3" + BUNDLE_RETRY: "3" + PGHOST: 127.0.0.1 + PGUSER: circleci-demo-ruby + PGPASSWORD: "" + RAILS_ENV: test + steps: + - checkout + - ruby/install-deps + - node/install-packages: + pkg-manager: yarn + cache-key: "yarn.lock" + - run: + name: Wait for DB + command: dockerize -wait tcp://localhost:5432 -timeout 1m + - run: + name: Database setup + command: bundle exec rails db:schema:load --trace + # Run rspec in parallel + - ruby/rspec-test + - ruby/rubocop-check + +workflows: + version: 2 + build_and_test: + jobs: + - build + - test: + requires: + - build + ``` @@ -52,11 +111,82 @@ The files generated from the `dry-run` command represent the equivalent Actions Converted workflow 👇 ```yaml -# ADD YAML +name: labs-data/circleci-demo-ruby-rails/build_and_test +on: + push: + branches: + - master +jobs: + build: + runs-on: ubuntu-latest + container: + image: cimg/ruby:2.7.5-node + steps: + - name: Set up bundler cache + uses: ruby/setup-ruby@v1 + with: + ruby-version: 3.0.2 + bundler-cache: true + - uses: actions/checkout@v2 + - run: bundle check || bundle install + env: + BUNDLE_DEPLOYMENT: true + - id: yarn-cache-dir-path + run: echo "::set-output name=dir::$(yarn config get cacheFolder)" + - uses: actions/cache@v2 + with: + path: "${{ steps.yarn-cache-dir-path.outputs.dir }}" + key: "${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}" + restore-keys: "${{ runner.os }}-yarn-" + - run: yarn install --frozen-lockfile + test: + runs-on: ubuntu-latest + container: + image: cimg/ruby:2.7.5-node + services: + postgres: + image: postgres:9.5-alpine + env: + POSTGRES_USER: circleci-demo-ruby + POSTGRES_DB: rails_blog_test + POSTGRES_PASSWORD: '' + needs: + - build + env: + BUNDLE_JOBS: '3' + BUNDLE_RETRY: '3' + PGHOST: 127.0.0.1 + PGUSER: circleci-demo-ruby + PGPASSWORD: '' + RAILS_ENV: test + steps: + - name: Set up bundler cache + uses: ruby/setup-ruby@v1 + with: + ruby-version: 3.0.2 + bundler-cache: true + - uses: actions/checkout@v2 + - run: bundle check || bundle install + env: + BUNDLE_DEPLOYMENT: true + - id: yarn-cache-dir-path + run: echo "::set-output name=dir::$(yarn config get cacheFolder)" + - uses: actions/cache@v2 + with: + path: "${{ steps.yarn-cache-dir-path.outputs.dir }}" + key: "${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}" + restore-keys: "${{ runner.os }}-yarn-" + - run: yarn install --frozen-lockfile + - name: Wait for DB + run: dockerize -wait tcp://localhost:5432 -timeout 1m + - name: Database setup + run: bundle exec rails db:schema:load --trace + - run: bundle exec rspec spec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec/results.xml --format progress + - run: bundle exec rubocop --format progress ``` -Despite these 2 pipelines using different syntax they will function equivalently. +Despite these two pipelines using different syntax they will function equivalently. ## Next lab From 070bd720d240921453cf19ff268b5190c5c51d1f Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 09:48:15 -0400 Subject: [PATCH 5/9] Create 5-migrate.md --- circle_ci/5-migrate.md | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 circle_ci/5-migrate.md diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md new file mode 100644 index 0000000..e1b159a --- /dev/null +++ b/circle_ci/5-migrate.md @@ -0,0 +1,53 @@ +# Perform a production migration of a CircleCI pipeline + +In this lab, you will use the `migrate` command to convert a CircleCI pipeline and open a pull request with the equivalent Actions workflow. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [dry-run lab](./3-dry-run.md). +4. Completed the [custom transformers lab](./4-custom-transformers.md). + +## Performing a migration + +Answer the following questions before running a `migrate` command: + +1. What project do you want to migrate? + - __circleci-hello-world__ +2. What is the namespace for that project? + - __Valet__ +3. Where do you want to store the logs? + - __./tmp/migrate__ +4. What is the URL for the GitHub repository to add the workflow to? + - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. + +### Steps + +1. Run the following `migrate` command in the codespace terminal: + + ```bash + gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world + ``` + +2. The command will write the URL to the pull request that was created when the command succeeds. + + ADD_SCREENSHOT_HERE + +3. Open the generated pull request in a new browser tab. + +### Inspect the pull request + +The first thing to notice about the pull request is that there is a list of manual steps to complete. + +Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request. + +Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. + +ADD_SCREENSHOT_HERE + +At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! + +### Next Lab + +[Forecast potential build runner usage](6-forecast.md) From 7469d8565ce21bf63c08fe412bb5c7d81c0115b0 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:13:10 -0400 Subject: [PATCH 6/9] Update 5-migrate.md --- circle_ci/5-migrate.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md index e1b159a..d9ad85f 100644 --- a/circle_ci/5-migrate.md +++ b/circle_ci/5-migrate.md @@ -5,7 +5,7 @@ In this lab, you will use the `migrate` command to convert a CircleCI pipeline a ## Prerequisites 1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. -2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +2. Completed the [configure lab](./1-configure.md#configuring-credentials). 3. Completed the [dry-run lab](./3-dry-run.md). 4. Completed the [custom transformers lab](./4-custom-transformers.md). @@ -15,11 +15,9 @@ Answer the following questions before running a `migrate` command: 1. What project do you want to migrate? - __circleci-hello-world__ -2. What is the namespace for that project? - - __Valet__ -3. Where do you want to store the logs? +2. Where do you want to store the logs? - __./tmp/migrate__ -4. What is the URL for the GitHub repository to add the workflow to? +3. What is the URL for the GitHub repository to add the workflow to? - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps @@ -32,7 +30,7 @@ Answer the following questions before running a `migrate` command: 2. The command will write the URL to the pull request that was created when the command succeeds. - ADD_SCREENSHOT_HERE + ![pr](https://user-images.githubusercontent.com/18723510/189923353-1166bd94-8d64-4924-ac55-3e06fa5d1734.png) 3. Open the generated pull request in a new browser tab. @@ -42,9 +40,12 @@ The first thing to notice about the pull request is that there is a list of manu Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request. +![action-run](https://user-images.githubusercontent.com/18723510/189924238-9f6799c7-e029-4695-a1de-a23666171992.png) + Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. - -ADD_SCREENSHOT_HERE + + + At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! From b228150fffea19c922787e730c384a3c279c39f7 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:24:20 -0400 Subject: [PATCH 7/9] Update 5-migrate.md --- circle_ci/5-migrate.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/circle_ci/5-migrate.md b/circle_ci/5-migrate.md index d9ad85f..e8c1f09 100644 --- a/circle_ci/5-migrate.md +++ b/circle_ci/5-migrate.md @@ -18,14 +18,14 @@ Answer the following questions before running a `migrate` command: 2. Where do you want to store the logs? - __./tmp/migrate__ 3. What is the URL for the GitHub repository to add the workflow to? - - __this repository__. The URL should should follow the pattern with `:owner` and `:repo` replaced with your values. + - __this repository__. The URL should follow the pattern with `:owner` and `:repo` replaced with your values. ### Steps -1. Run the following `migrate` command in the codespace terminal: +1. Run the following `migrate` command in the codespace terminal. Ensure the values in `--target-url` for `:owner` and `:repo` are replaced with your values: ```bash - gh valet migrate gitlab --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world + gh valet migrate circle-ci --target-url https://github.com/:owner/:repo --output-dir ./tmp/migrate --circle-ci-project circleci-hello-world ``` 2. The command will write the URL to the pull request that was created when the command succeeds. @@ -44,9 +44,6 @@ Next, you can inspect the "Files changed" in this pull request and see the conve Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub. - - - At this point, the migration has completed and you have successfully migrated a CircleCI pipeline to Actions! ### Next Lab From 9dab504ed333246b87d03f2486c15e659f6eb574 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:50:43 -0400 Subject: [PATCH 8/9] Create 6-forecast.md --- circle_ci/6-forecast.md | 99 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 circle_ci/6-forecast.md diff --git a/circle_ci/6-forecast.md b/circle_ci/6-forecast.md new file mode 100644 index 0000000..356d771 --- /dev/null +++ b/circle_ci/6-forecast.md @@ -0,0 +1,99 @@ +# Forecast potential build runner usage + +In this lab you will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from completed pipeline runs in CircleCI. + +## Prerequisites + +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment. +2. Completed the [configure lab](./1-configure.md#configuring-credentials). + +## Perform a forecast + +Answer the following questions before running the `forecast` command: +1. What is the date you want to start forecasting from? + - **2022-09-02**. This date is needed as it is prior to when the bulk of builds were trigger in the demo CircleCI organization for these labs. This value defaults to the date one week ago, however, you should use a start date that will show a representative view of typical usage. +2. Where do you want to store the results? + - **./tmp/forecast_reports** + +### Steps + +1. Navigate to your codespace terminal +2. Run the following command from the root directory: + + ```bash + gh valet forecast circle-ci --output-dir ./tmp/forecast_reports --start-date 2022-09-02 + ``` + +3. The command will list all the files written to disk when the command succeeds. + + ADD_SCREENSHOT_HERE + +## Review the forecast report + +The forecast report, logs, and completed job data will be located within the `tmp/forecast_reports` folder. + +1. Find the `forecast_report.md` file in the file explorer. +2. Right-click the `forecast_report.md` file and select `Open Preview`. +3. This file contains metrics used to forecast potential GitHub Actions usage. + +### Total + +The "Total" section of the forecast report contains high level statistics related to all the jobs completed after the `--start-date` CLI option: + +```md +- Job count: **18** +- Pipeline count: **13** + +- Execution time + + - Total: **46 minutes** + - Median: **1 minutes** + - P90: **6 minutes** + - Min: **0 minutes** + - Max: **10 minutes** + +- Queue time + + - Median: **0 minutes** + - P90: **0 minutes** + - Min: **0 minutes** + - Max: **0 minutes** + +- Concurrent jobs + + - Median: **0** + - P90: **0** + - Min: **0** + - Max: **4** +``` + +Here are some key terms of items defined in the forecast report: + +- The `job count` is the total number of completed jobs. +- The `pipeline count` is the number of unique pipelines used. +- `Execution time` describes the amount of time a runner spent on a job. This metric can be used to help plan for the cost of GitHub-hosted runners. + - This metric is correlated to how much you should expect to spend in GitHub Actions. This will vary depending on the hardware used for these minutes. You can use the [Actions pricing calculator](https://github.com/pricing/calculator) to estimate a dollar amount. +- `Queue time` metrics describe the amount of time a job spent waiting for a runner to be available to execute it. +- `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure. + +Additionally, these metrics are defined for each queue of runners defined in GitLab. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. + +## Forecasting multiple providers + +You can examine the available options for the `forecast` command by running `gh valet forecast --help`. When you do this you will see the `--source-file-path` option: + +![img](https://user-images.githubusercontent.com/19557880/186263140-f02c6cab-7979-417c-bdfe-b9590e9c5597.png) + +You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and want to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`). + +Run the following command from within the codespace terminal: + +```bash +gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast +``` + +You can now inspect the output of the command to see a forecast report using all of the files matching the `tmp/**/jobs/*.json` pattern. + +## Next steps + +This concludes all labs for migrating GitLab pipelines to Actions with Valet! From 6a205884d88b81bcbfe4d395b2ea3a1b6de32570 Mon Sep 17 00:00:00 2001 From: j-dunham Date: Tue, 13 Sep 2022 10:56:54 -0400 Subject: [PATCH 9/9] Update 6-forecast.md --- circle_ci/6-forecast.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/circle_ci/6-forecast.md b/circle_ci/6-forecast.md index 356d771..84be464 100644 --- a/circle_ci/6-forecast.md +++ b/circle_ci/6-forecast.md @@ -26,7 +26,7 @@ Answer the following questions before running the `forecast` command: 3. The command will list all the files written to disk when the command succeeds. - ADD_SCREENSHOT_HERE + ![command-output](https://user-images.githubusercontent.com/18723510/189935161-dfd66c9c-e4f8-4833-ae7d-445c83cf75ba.png) ## Review the forecast report @@ -76,7 +76,7 @@ Here are some key terms of items defined in the forecast report: - `Queue time` metrics describe the amount of time a job spent waiting for a runner to be available to execute it. - `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure. -Additionally, these metrics are defined for each queue of runners defined in GitLab. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. +Additionally, these metrics are defined for each queue of runners defined in CircleCI. This is especially useful if there are a mix of hosted/self-hosted runners or high/low spec machines to see metrics specific to different types of runners. ## Forecasting multiple providers @@ -96,4 +96,4 @@ You can now inspect the output of the command to see a forecast report using all ## Next steps -This concludes all labs for migrating GitLab pipelines to Actions with Valet! +This concludes all labs for migrating CircleCI pipelines to Actions with Valet!