diff --git a/azure_devops/4-custom-transformers.md b/azure_devops/4-custom-transformers.md index ffe7725..7d04a12 100644 --- a/azure_devops/4-custom-transformers.md +++ b/azure_devops/4-custom-transformers.md @@ -1,54 +1,106 @@ -# Migrate an Azure DevOps pipeline to GitHub Actions with a custom transformer -In this lab, you will create a custom plugin that transforms some of the existing migration mapping and replaces it with your own mapping. Then use the `migrate` subcommand to migrate the pipeline. The `migrate` subcommand can be used to convert a pipeline to its GitHub Actions equivalent and then create a pull request with the contents. +# Using custom transformers to customize Valet's behavior -- [Prerequisites](#prerequisites) -- [Identify the Azure DevOps pipeline ID to use](#identify-the-azure-devops-pipeline-id-to-use) -- [Create a custom transformer](#create-a-custom-transformer) -- [Migrate with a custom transformer](#migrate-with-a-custom-transformer) -- [View the pull request](#view-the-pull-request) +In this lab we will build upon the `dry-run` command to override Valet's default behavior and customize the converted workflow using "custom transformers". Custom transformers can be used to: + +1. Convert items that are not automatically converted. +2. Convert items that were automatically converted using different actions. +3. Convert environment variable values differently. +4. Convert references to runners to use a different runner name in Actions. ## Prerequisites -1. Follow all steps [here](../azure_devops#readme) to set up your environment -2. Create or start a codespace in this repository (if not started) -3. Complete the [Valet audit lab](valet-audit-lab.md). -4. Complete the [Valet migrate lab](valet-migrate-lab.md). +1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment and bootstrap an Azure DevOps project. +2. Completed the [configure lab](./1-configure-lab.md#configuring-credentials). +3. Completed the [audit lab](./2-audit.md). +4. Completed the [dry-run lab](./3-dry-run.md). -## Identify the Azure DevOps pipeline ID to use -You will need the `valet-custom-transformer-example` Azure DevOps pipeline ID to perform the migration -1. Go to the `valet/ValetBootstrap/pipelines` folder -2. Open the `valet/ValetBootstrap/pipelines/valet-custom-transformer-example.config.json` file -3. Look for the `web - href` link -4. At the end of the link is the pipeline ID. Copy or note the ID. +## Perform a dry run -### Example -![mapperprops](https://user-images.githubusercontent.com/26442605/175090567-525b97a7-60d2-41b7-9dcd-d559ca1c5bd7.png) +We will be performing a dry-run for a pipeline in the bootstrapped Azure DevOps project. We will need to answer the following questions before running this command: -## Create a custom transformer +1. What is the id of the pipeline to convert? + - __:pipeline_id__. This id can be found by: + - Navigating to the build pipelines in the bootstrapped Azure DevOps project + - Selecting the pipeline with the name "valet-custom-transformer-example" + - Inspecting the URL to locate the pipeline id -To create a transformer, you need to create a Ruby file that looks as follows: -``` ruby -transform "azuredevopstaskname" do |item| - # your ruby code here that produces output -end -``` +2. Where do we want to store the result? + - __./tmp/dry-run-lab__. This can be any path within the working directory that Valet commands are executed from. -We start by creating a new folder called `plugin` under the `valet` folder in your repository. In there create a file called `DotNetCoreCLI.rb`. +### Steps -Next change the function name to match the Azure DevOps task name `DotNetCoreCLI@2`. -The way you find this name is by clicking the **view yaml** button at a task in the pipeline: +1. Navigate to the codespace terminal +2. Run the following command from the root directory: -This results in the following code: -``` ruby -transform "DotNetCoreCLI@2" do |item| - # your ruby code here that produces output -end -``` -The parameter item is a collection of items than contain the properties of the original task that was retrieved from Azure DevOps. -In this case we can see in the yaml that the properties that are set are `command` and `projects`. - -Add the following code to the ruby file: -``` Ruby + ```bash + gh valet dry-run azure-devops pipeline --pipeline-id :pipeline_id -o tmp/dry-run-lab + ``` + +3. The command will list all the files written to disk when the command succeeds. +4. View the converted workflow: + - Find `./tmp/dry-run-lab` in the file explorer pane in codespaces. + - Click `valet-custom-transformer-example.yml` to open. + +The converted workflow that is generated can be seen below: + +
+ Converted workflow 👇 + +```yaml +name: valet-bootstrap/pipelines/valet-custom-transformer-example +on: + push: + branches: + - "*" +env: + BUILDCONFIGURATION: Release + BuildParameters_RESTOREBUILDPROJECTS: "**/*.csproj" +jobs: + Job_1: + name: Agent job 1 + runs-on: windows-latest + steps: + - name: checkout + uses: actions/checkout@v2 + - uses: actions/checkout@v2 + - name: Use Node 10.16.3 + uses: actions/setup-node@v2 + with: + node-version: 10.16.3 + - name: Restore + run: dotnet restore ${{ env.BuildParameters_RESTOREBUILDPROJECTS }} + - name: Build + run: dotnet build ${{ env.BuildParameters_RESTOREBUILDPROJECTS }} --configuration ${{ env.BUILDCONFIGURATION }} +``` + +
+ +_Note_: You can refer to the previous [lab](./3-dry-run.md) to learn about the fundamentals of the `dry-run` command. + +## Custom transformers for build steps + +We can use custom transformers override Valet's default behavior. In this scenario, we may want to override the behavior for converting `DotnetCoreCLI@2` tasks to support parameters that are glob patterns. We will need to answer the following questions before writing a custom transformer: + +1. What is the "identifier" of the step to customize? + - __DotnetCoreCLI@2__ + +2. What is the desired Actions syntax to use instead? + - After some research, we have determined that the uploading test results as an artifact will be suitable: + + ```yaml + - run: shopt -s globstar; for f in ./**/*.csproj; do dotnet build $f --configuration ${{ env.BUILDCONFIGURATION }} ; done + shell: bash + ``` + +Now we can begin to write the custom transformer. Customer transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal: + +```bash +code transformers.rb +``` + +Next, we will define a `transform` method for the `DotnetCoreCLI@2` identifier by adding the following code to `transformers.rb`: + +```ruby transform "DotNetCoreCLI@2" do |item| projects = item["projects"] command = item["command"] @@ -60,28 +112,121 @@ transform "DotNetCoreCLI@2" do |item| else run_command << "dotnet #{command} #{item['projects']} #{item['arguments']}" end - + + { + run: run_command.join("\n"), + shell: "bash", + } +end +``` + +This method can use any valid ruby syntax and should return a `Hash` that represents the YAML that should be generated for a given step. Valet will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Azure DevOps. + +Now, we can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal: + +```bash +gh valet dry-run azure-devops pipeline --pipeline-id :pipeline_id -o tmp/dry-run-lab --custom-transformers transformers.rb +``` + +Open the workflow that is generated and inspect the contents. Now, the `DotnetCoreCLI@2` steps are converted using the customized behavior! + +```diff +- - name: Restore +- run: dotnet restore ${{ env.BuildParameters_RESTOREBUILDPROJECTS }} +- - name: Build +- run: dotnet build ${{ env.BuildParameters_RESTOREBUILDPROJECTS }} --configuration ${{ env.BUILDCONFIGURATION }} ++ - name: Restore ++ run: shopt -s globstar; for f in ./**/*.csproj; do dotnet restore $f ; done ++ shell: bash ++ - name: Build ++ run: shopt -s globstar; for f in ./**/*.csproj; do dotnet build $f --configuration ${{ env.BUILDCONFIGURATION }} ; done ++ shell: bash +``` + +## Custom transformers for environment variables + +We can also use custom transformers to edit the values of environment variables in converted workflows. In our example, we will be updating the `BUILDCONFIGURATION` environment variable to be `Debug` instead of `Release`. + +To do this, add the following code to the `transformers.rb` file. + +```ruby +env "BUILDCONFIGURATION", "Debug" +``` + +In this example, the first parameter to the `env` method is the environment variable name and the second is the updated value. + +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `DB_ENGINE` environment variable will be set to `mongodb`: + +```diff +env: +- BUILDCONFIGURATION: Release ++ BUILDCONFIGURATION: Debug + BuildParameters_RESTOREBUILDPROJECTS: "**/*.csproj" +``` + +## Custom transformers for runners + +Finally, we can use custom transformers to dictate which runners converted workflows should use. To do this we will need to answer the following questions: + +1. What is label of the runner in Azure DevOps to update? + - __windows-latest__ + +2. What is the label of the runner in Actions to use instead? + - __ubuntu-latest__ + +With these questions answered, we can add the following code to the `transformers.rb` file: + +```ruby +runner "windows-latest", "ubuntu-latest" +``` + +In this example, the first parameter to the `runner` method is the Azure DevOps label and the second is the Actions runner label. + +Now, we can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` statement will use the customized runner label: + +```diff +runs-on: +- - windows-latest ++ - ubuntu-latest +``` + +At this point of the lab the file contents of `transformers.rb` should match this: + +
+ Custom transformers 👇 + +```ruby +transform "DotNetCoreCLI@2" do |item| + projects = item["projects"] + command = item["command"] + run_command = [] + + if projects.include?("$") + command = "build" if command.nil? + run_command << "shopt -s globstar; for f in ./**/*.csproj; do dotnet #{command} $f #{item['arguments']} ; done" + else + run_command << "dotnet #{command} #{item['projects']} #{item['arguments']}" + end + { shell: "bash", run: run_command.join("\n") } end + +env "BUILDCONFIGURATION", "Debug" + +runner "windows-latest", "ubuntu-latest" ``` -## Migrate with a custom transformer +
-Run the `gh valet migrate` command from the `valet` directory with the transformer again and pass it the custom plugin. Look at the result and see if it results in a successful build. +Thats it! At this point you have overridden Valet's default behavior by customizing the conversion of: -``` -gh valet migrate azure-devops pipeline --target-url https://github.com/GITHUB-ORG/GITHUB-REPO --pipeline-id PIPELINE-ID --custom-transformers plugin/DotNetCoreCLI.rb --output-dir migrate -``` +- Build steps +- Environment variables +- Runners -### Example -![valet-cm-1](https://user-images.githubusercontent.com/26442605/169618556-7c79b34b-6d4c-48d5-98e5-7f8d771117a5.png) - -## View the pull request -The migrate output will show you the pull request on GitHub. Note here that the checks on the pull request all passed! - -### Example -![mapper-ex3](https://user-images.githubusercontent.com/26442605/161117488-93e38847-3034-4f04-a768-e74e16dba4ae.png) +## Next lab +[Perform a production migration of an Azure DevOps pipeline](5-migrate.md) diff --git a/jenkins/4-custom-transformers.md b/jenkins/4-custom-transformers.md index b9d28ba..e97ee48 100644 --- a/jenkins/4-custom-transformers.md +++ b/jenkins/4-custom-transformers.md @@ -139,7 +139,7 @@ We can also override Valet's default behavior. In this scenario, we may not desi - __junit__ 2. What is the desired Actions syntax to use instead? - - After some research, we have determined that the uploading test results as an artifact will be suitable: + - After some research, we have determined that uploading test results as an artifact will be suitable: ```yaml - uses: actions/upload-artifact@v3