actions/importer-labs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #48 from actions/bitbucket-labs

Browse Source 
Add Bitbucket Labs
This commit is contained in:
j-dunham
2023-09-15 11:34:26 -04:00
committed by GitHub
parent 5aee14b780 f98963a7f0
commit d06e69b75d
16 changed files with 1285 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bitbucket/1-configure.md
+61
View File
@@ -0,0 +1,61 @@
# Configure credentials for GitHub Actions Importer
In this lab, you will use the `configure` CLI command to set the required credentials and information for GitHub Actions Importer to use when working with Bitbucket and GitHub.
You will need to complete all of the setup instructions [here](./readme.md#configure-your-codespace) prior to performing this lab.
## Configuring credentials
1. Create a GitHub personal access token (PAT):
- Open github.com in a new browser tab.
- In the top right of the UI, click your profile photo and then click `Settings`.
- In the left panel, click `Developer Settings`.
- Click `Personal access tokens` and then `Tokens (classic)` (if present).
- Click `Generate new token` and then `Generate new token (classic)`. You may be required to authenticate with GitHub during this step.
- Select the following scope: `workflow`.
- Click `Generate token`.
- Copy the generated PAT and save in a safe location.
2. Follow Bitbucket's [documentation](https://support.atlassian.com/bitbucket-cloud/docs/create-a-workspace-access-token/) to generate a workspace access token with read scopes for pipeline, project, and repository, and ensure that you store the token securely. This step is optional since we are using source files in this lab.
3. Run the `configure` CLI command:
- Select the `TERMINAL` tab from within the codespace terminal window.
- Run the following command: `gh actions-importer configure`.
- Using the down arrow key to highlight `Bitbucket`, press the spacebar to select, and then press enter to continue.
- At the GitHub PAT prompt, enter the GitHub PAT generated in step 1 and press enter.
- At the GitHub URL prompt, enter the GitHub instance URL or press enter to accept the default value (`https://github.com`).
- At the Bitbucket token prompt, enter the Bitbucket access token from step 2 or any random string if no token was generated, and press enter.
```console
$ gh actions-importer configure
✔ Which CI providers are you configuring?: Bitbucket
Enter the following values (leave empty to omit):
✔ Personal access token for GitHub: ***************
✔ Base url of the GitHub instance: https://github.com
✔ Personal access token for Bitbucket: ***************
Environment variables successfully updated.
```
## Verify your environment
To verify your environment is configured correctly, you are going to run the `update` CLI command. The `update` CLI command will download the latest version of GitHub Actions Importer to your codespace.
1. In the codespace terminal run the following command:
```bash
gh actions-importer update
```
2. You should see a confirmation that you were logged into the GitHub Container Registry and the image was updated to the latest version.
```console
$ gh actions-importer update
Login Succeeded
latest: Pulling from actions-importer/cli
Digest: sha256:a7d00dee8a37e25da59daeed44b1543f476b00fa2c41c47f48deeaf34a215bbb
Status: Image is up to date for ghcr.io/actions-importer/cli:latest
ghcr.io/actions-importer/cli:latest
```
### Next lab
[Perform an audit of Bitbucket](./2-audit.md)
bitbucket/2-audit.md
+217
View File
@@ -0,0 +1,217 @@
# Perform an audit of Bitbucket
In this lab, you will use the `audit` command to get a high-level view of all pipelines in your Bitbucket workspace.
The `audit` command will perform the following steps:
1. Fetch all of the pipelines in the Bitbucket workspace.
2. Convert each pipeline to its equivalent GitHub Actions workflow(s).
3. Generate a report that summarizes how complete and complex of a migration is possible with GitHub Actions Importer.
## Prerequisites
1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment.
2. Completed the [configure lab](./1-configure.md#configuring-credentials).
## Perform an audit
Answer the following questions before running this command:
1. What workspace do you want to audit?
- __actions-importer__. In this example we will be instructing Actions Importer to audit the `actions-importer` workspace.
2. Where do you want to store the result?
- __tmp/audit__. This can be any path within the working directory from which GitHub Actions Importer commands are executed.
### Steps
1. Navigate to the codespace terminal.
2. Run the following command from the root directory:
```bash
gh actions-importer audit bitbucket --output-dir tmp/audit --workspace actions-importer --config-file-path bitbucket/bootstrap/config.yml
```
> Note: The `--config-file-path` option is not required and is used throughout these labs to convert files that are stored locally for this lab. When performing a migration on your own workspace, it can be omitted and GitHub Actions Importer will programmatically fetch pipelines using the Bitbucket REST APIs.
3. The command will list all the files written to disk in green when the command succeeds.
## Inspect the output files
1. Find the `audit_summary.md` file in the file explorer.
2. Right-click the `audit_summary.md` file and select `Open Preview`.
3. This file contains details that summarize what percentage of your pipelines were converted automatically.
### Review audit summary
#### Pipelines
The pipeline summary section contains high level statistics regarding the conversion rate done by GitHub Actions Importer:
```md
## Pipelines
Total: **4**
- Successful: **4 (100%)**
- Partially successful: **0 (0%)**
- Unsupported: **0 (0%)**
- Failed: **0 (0%)**
### Job types
Supported: **4 (100%)**
- YAML: **4**
```
Here are some key terms in the “Pipelines” section:
- __Successful__ pipelines had 100% of the pipeline constructs and individual items converted automatically to their GitHub Actions equivalent.
- __Partially successful__ pipelines had all of the pipeline constructs converted, however, there were some individual items (e.g. build tasks or build triggers) that were not converted automatically to their GitHub Actions equivalent.
- __Unsupported__ pipelines are definition types that are not supported by GitHub Actions Importer.
- __Failed__ pipelines encountered a fatal error when being converted. This can occur for one of three reasons:
- The pipeline was misconfigured and not valid in Bitbucket.
- GitHub Actions Importer encountered an internal error when converting it.
- There was an unsuccessful network response, often due to invalid credentials, that caused the pipeline to be inaccessible.
The "Job types" section will summarize which types of pipelines are being used and which are supported or unsupported by GitHub Actions Importer.
#### Build steps
The build steps summary section presents an overview of the individual build steps that are used across all pipelines and how many were automatically converted by GitHub Actions Importer.
```md
### Build steps
Total: **22**
Known: **22 (100%)**
- script: **11**
- atlassian/git-secrets-scan: **3**
- cache-node: **3**
- cache-docker: **2**
- atlassian/aws-cloudfront-invalidate: **1**
- atlassian/aws-s3-deploy: **1**
- cache-pip: **1**
Actions: **40**
- run: **15**
- actions/checkout@v3.6.0: **14**
- actions/cache@v3.3.1: **6**
- actions/upload-artifact@v3.1.1: **2**
- actions/download-artifact@v3.0.1: **2**
- aws-actions/configure-aws-credentials@v3.0.1: **1**
```
Here are some key terms in the "Build steps" section:
- A __known__ build step is a step that was automatically converted to an equivalent action.
- An __unknown__ build step is a step that was not automatically converted to an equivalent action.
- An __unsupported__ build step is a step that is either:
- A step that is fundamentally not supported by GitHub Actions.
- A step that is configured in a way that is incompatible with GitHub Actions.
- An __action__ is a list of the actions that were used in the converted workflows. This is important for the following scenarios:
- Gathering the list of actions to sync to your appliance if you use GitHub Enterprise Server.
- Defining an organization-level allowlist of actions that can be used. This list of actions is a comprehensive list of which actions their security and/or compliance teams will need to review.
There is an equivalent breakdown of build triggers, environment variables, and other uncategorized items displayed in the audit summary file.
#### Manual Tasks
The manual tasks summary section presents an overview of the manual tasks that you will need to perform that GitHub Actions Importer is not able to complete automatically.
```md
### Manual tasks
Total: **2**
Self hosted runners: **2**
- `my.custom.label`: **2**
```
Here are some key terms in the “Manual tasks” section:
- A __secret__ refers to a repository or organization level secret that is used by the converted pipelines. These secrets will need to be created manually in Actions in order for these pipelines to function properly.
- A __self-hosted runner__ refers to a label of a runner that is referenced by a converted pipeline that is not a GitHub-hosted runner. You will need to manually define these runners in order for these pipelines to function properly.
#### Files
The final section of the audit report provides a manifest of all of the files that are written to disk during the audit. These files include:
```md
### Successful
#### actions-importer/python
- [actions-importer/python/.github/workflows/default.yml](actions-importer/python/.github/workflows/default.yml)
- [actions-importer/python/config.json](actions-importer/python/config.json)
- [actions-importer/python/source.yml](actions-importer/python/source.yml)
#### actions-importer/ruby
- [actions-importer/ruby/.github/workflows/default.yml](actions-importer/ruby/.github/workflows/default.yml)
- [actions-importer/ruby/config.json](actions-importer/ruby/config.json)
- [actions-importer/ruby/source.yml](actions-importer/ruby/source.yml)
#### actions-importer/docker
- [actions-importer/docker/.github/workflows/default.yml](actions-importer/docker/.github/workflows/default.yml)
- [actions-importer/docker/.github/workflows/branches-master.yml](actions-importer/docker/.github/workflows/branches-master.yml)
- [actions-importer/docker/config.json](actions-importer/docker/config.json)
- [actions-importer/docker/source.yml](actions-importer/docker/source.yml)
#### actions-importer/react-deploy
- [actions-importer/react-deploy/.github/workflows/default.yml](actions-importer/react-deploy/.github/workflows/default.yml)
- [actions-importer/react-deploy/.github/workflows/branches-master.yml](actions-importer/react-deploy/.github/workflows/branches-master.yml)
- [actions-importer/react-deploy/config.json](actions-importer/react-deploy/config.json)
- [actions-importer/react-deploy/source.yml](actions-importer/react-deploy/source.yml)
```
Each pipeline will have a variety of files written that include:
- The original pipeline as it was defined in Bitbucket.
- Any network responses used to convert a pipeline.
- The converted workflow.
- Stack traces that can be used to troubleshoot a failed pipeline conversion
## Inspect the workflow usage .csv file
1. Open the `tmp/audit/workflow_usage.csv` file in the file explorer.
2. This file contains a comma-separated list of all actions, secrets, and runners that are used by each successfully converted pipeline:
```csv
Pipeline,Action,File path
actions-importer/python,actions/checkout@v3.6.0,tmp/audit/actions-importer/python/.github/workflows/default.yml
actions-importer/python,actions/cache@v3.3.1,tmp/audit/actions-importer/python/.github/workflows/default.yml
actions-importer/ruby,actions/checkout@v3.6.0,tmp/audit/actions-importer/ruby/.github/workflows/default.yml
actions-importer/docker,actions/checkout@v3.6.0,tmp/audit/actions-importer/docker/.github/workflows/default.yml
actions-importer/docker,actions/cache@v3.3.1,tmp/audit/actions-importer/docker/.github/workflows/default.yml
actions-importer/docker,actions/upload-artifact@v3.1.1,tmp/audit/actions-importer/docker/.github/workflows/default.yml
actions-importer/docker,actions/download-artifact@v3.0.1,tmp/audit/actions-importer/docker/.github/workflows/default.yml
actions-importer/react-deploy,actions/checkout@v3.6.0,tmp/audit/actions-importer/react-deploy/.github/workflows/default.yml
actions-importer/react-deploy,actions/cache@v3.3.1,tmp/audit/actions-importer/react-deploy/.github/workflows/default.yml
actions-importer/react-deploy,actions/upload-artifact@v3.1.1,tmp/audit/actions-importer/react-deploy/.github/workflows/default.yml
actions-importer/react-deploy,aws-actions/configure-aws-credentials@v3.0.1,tmp/audit/actions-importer/react-deploy/.github/workflows/default.yml
actions-importer/react-deploy,actions/download-artifact@v3.0.1,tmp/audit/actions-importer/react-deploy/.github/workflows/default.yml
Pipeline,Secret,File path
Pipeline,Runner,File path
actions-importer/python,my.custom.label,tmp/audit/actions-importer/python/.github/workflows/default.yml
```
The contents of this file can be useful in answering questions similar to the following:
- What workflows will depend on which actions?
- What workflows use an action that must go through a security review?
- What workflows use specific secrets?
- What workflows use specific runners?
### Next lab
[Forecast potential build runner usage](3-forecast.md)
bitbucket/3-forecast.md
+108
View File
@@ -0,0 +1,108 @@
# Forecast potential build runner usage
In this lab you will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from completed pipeline runs in Bitbucket.
## Prerequisites
1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment.
2. Completed the [configure lab](./1-configure.md#configuring-credentials).
## Perform a forecast
Answer the following questions before running the `forecast` command:
1. What workspace do you want to run the forecast for?
- **actions-importer**
2. What is the date you want to start forecasting from?
- **2023-08-03**. By default, the value is set to one week ago, but it's recommended that you choose a start date that provides a representative view of typical usage.
3. Where do you want to store the results?
- **tmp/forecast**
### Steps
1. Navigate to your codespace terminal
2. Run the following command from the root directory:
```bash
gh actions-importer forecast bitbucket --workspace actions-importer --start-date 2023-08-03 --output-dir tmp/forecast --source-file-path bitbucket/**/source_files/*.json
```
> Note: The `--source-file-path` option is not required and is used throughout this lab to convert files that are stored locally. This can be omitted and GitHub Actions Importer will programmatically fetch historical pipeline runs using the Bitbucket REST APIs.
3. The command will list all the files written to disk when the command succeeds.
```console
[2023-09-07 18:26:22] Logs: 'tmp/audit/log/valet-20230907-182622.log'
[2023-09-07 18:26:22] Forecasting 'https://bitbucket.org/actions-importer'
[2023-09-07 18:26:22] Output file(s):
[2023-09-07 18:26:22] tmp/forecast/forecast_report.md
```
## Review the forecast report
The forecast report, logs, and completed job data will be located within the `tmp/forecast` folder.
1. Find the `forecast_report.md` file in the file explorer.
2. Right-click the `forecast_report.md` file and select `Open Preview`.
3. This file contains metrics used to forecast potential GitHub Actions usage.
### Total
The "Total" section of the forecast report contains high level statistics related to all the jobs completed after the `--start-date` CLI option:
```md
- Job count: **8**
- Pipeline count: **8**
- Execution time
- Total: **11 minutes**
- Median: **1 minutes**
- P90: **4 minutes**
- Min: **1 minutes**
- Max: **4 minutes**
- Concurrent jobs
- Median: **0**
- P90: **0**
- Min: **0**
- Max: **1**
```
Here are some key terms of items defined in the forecast report:
- The `job count` is the total number of completed jobs.
- The `pipeline count` is the number of unique pipelines used.
- `Execution time` describes the amount of time a runner spent on a job. This metric can be used to help plan for the cost of GitHub-hosted runners.
- This metric is correlated to how much you should expect to spend in GitHub Actions. This will vary depending on the hardware used for these minutes. You can use the [Actions pricing calculator](https://github.com/pricing/calculator) to estimate a dollar amount.
- `Concurrent jobs` metrics describe the amount of jobs running at any given time. This metric can be used to define the number of runners a customer should configure.
Additionally, these metrics are defined by hosted and self-hosted runners.
## Forecasting multiple providers
You can examine the available options for the `forecast` command by running `gh actions-importer forecast --help`. When you do this you will see the `--source-file-path` option:
```console
$ gh actions-importer forecast -h
Options:
--source-file-path <source-file-path> (REQUIRED) The file path(s) to existing jobs data.
-o, --output-dir <output-dir> (REQUIRED) The location for any output files.
--start-date <start-date> The start date of the forecast analysis in YYYY-MM-DD format. [default: 9/12/2022 12:42:39 PM]
--time-slice <time-slice> The time slice in seconds to use for computing concurrency metrics. [default: 60]
--credentials-file <credentials-file> The file containing the credentials to use.
--no-telemetry Boolean value to disallow telemetry.
--no-ssl-verify Disable ssl certificate verification.
--no-http-cache Disable caching of http responses.
-?, -h, --help Show help and usage information
```
You can use the `--source-file-path` CLI option to combine data from multiple reports into a single report. This becomes useful if you use multiple CI/CD providers and want to get a holistic view of the runner usage. This works by using the `.json` files generated by `forecast` commands as space-delimited values for the `--source-file-path` CLI option. Optionally, this value could be a glob pattern to dynamically specify the list of files (e.g. `**/*.json`).
Below is a example command that would generate a report for all files matching `tmp/**/jobs/*.json`:
```bash
gh actions-importer forecast --source-file-path tmp/**/jobs/*.json --output-dir tmp/forecast-combined
```
## Next steps
[Perform a dry-run migration of a Bitbucket pipeline](4-dry-run.md)
bitbucket/4-dry-run.md
+265
View File
@@ -0,0 +1,265 @@
# Perform a dry-run migration of a Bitbucket pipeline
In this lab you will use the `dry-run` command to convert a Bitbucket pipeline to its equivalent GitHub Actions workflow.
## Prerequisites
1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your Codespace environment.
2. Completed the [configure lab](./1-configure.md#configuring-credentials).
3. Completed the [audit lab](./2-audit.md).
## Perform a dry run
Before executing the dry-run command we will need to answer the following questions.
1. What repository is that pipeline in?
- __react-deploy__
2. What is the workspace for that repository?
- __actions-importer__
3. Where do you want to store the result?
- __tmp/dry-run__. This can be any path within the working directory from which GitHub Actions Importer commands are executed.
### Steps
1. Navigate to your codespace terminal
2. Run the following command from the root directory:
```bash
gh actions-importer dry-run bitbucket --output-dir tmp/dry-run --workspace actions-importer --repository react-deploy --source-file-path ./bitbucket/bootstrap/source_files/react_deploy.yml
```
> Note: The `--source-file-path` option is not required and is used throughout this lab to convert a pipeline that is stored locally. This can be omitted and GitHub Actions Importer will programmatically fetch pipelines using the Bitbucket REST APIs
3. The command will list all the files written to disk when the command succeeds.
```console
❯ gh actions-importer dry-run bitbucket --output-dir tmp/dry-run --workspace actions-importer --repository react-deploy --source-file-path ./bitbucket/bootstrap/source_files/react_deploy.yml
[2023-09-07 19:00:29] Logs: 'tmp/dry-run/log/valet-20230907-190029.log'
[2023-09-07 19:00:29] Output file(s):
[2023-09-07 19:00:29] tmp/dry-run/actions-importer/react-deploy/.github/workflows/default.yml
[2023-09-07 19:00:29] tmp/dry-run/actions-importer/react-deploy/.github/workflows/branches-master.yml
```
4. View the converted workflows:
- Find `tmp/dry-run/actions-importer/react-deploy/.github/workflows` in the file explorer pane in your codespace.
- Click `default.yml` to open the workflow that was generated for the __default__ start condition.
- Click `branches-master.yml` to open workflow that was generated for the __master branch__ start condition.
## Inspect the output files
The files generated from the `dry-run` command represent the equivalent Actions workflows for the given Bitbucket pipeline. The Bitbucket pipeline and converted workflows can be seen below:
<details>
<summary><em>Bitbucket pipelines 👇</em></summary>
```yaml
pipelines:
default:
- parallel:
- step:
name: Build and Test
caches:
- node
script:
- npm install
- npm test
- step:
name: Lint the node package
script:
- npm install eslint
- npx eslint src
caches:
- node
branches:
master:
- parallel:
- step:
name: Build and Test
caches:
- node
script:
- npm install
- npm test
- npm run build
artifacts:
- build/**
- step:
name: Security Scan
script:
- pipe: atlassian/git-secrets-scan:0.5.1
- step:
name: Deploy to Production
deployment: Production
trigger: manual
clone:
enabled: false
script:
- pipe: atlassian/aws-s3-deploy:1.1.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
S3_BUCKET: 'my-bucket-name'
LOCAL_PATH: 'build'
- pipe: atlassian/aws-cloudfront-invalidate:0.6.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
DISTRIBUTION_ID: '123xyz'
```
</details>
<details>
<summary><em>Converted default workflow 👇</em></summary>
default.yml
```yaml
name: default
on:
push:
branches:
- "!master"
jobs:
parallel_job_1:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- name: Cache node
uses: actions/cache@v3.3.1
with:
key: "${{ runner.os }}-node-${{ hashFiles('node_modules') }}"
path: node_modules
- name: Build and Test
run: |-
npm install
npm test
parallel_job_2:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- name: Cache node
uses: actions/cache@v3.3.1
with:
key: "${{ runner.os }}-node-${{ hashFiles('node_modules') }}"
path: node_modules
- name: Lint the node package
run: |-
npm install eslint
npx eslint src
```
</details>
<details>
<summary><em>Converted master branch workflow 👇</em></summary>
default.yml
```yaml
name: branches-master
on:
push:
branches: master
jobs:
parallel_job_1:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- name: Cache node
uses: actions/cache@v3.3.1
with:
key: "${{ runner.os }}-node-${{ hashFiles('node_modules') }}"
path: node_modules
- name: Build and Test
run: |-
npm install
npm test
npm run build
- uses: actions/upload-artifact@v3.1.1
with:
name: parallel_job_1
path: build/**
parallel_job_2:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- uses: actions/checkout@v3.6.0
with:
path: git-secrets
repository: awslabs/git-secrets
- run: |
cd git-secrets
sudo make install
git secrets --register-aws --global
cd ..
# This transformed result does custom secret scanning using AWS, however the recommended way
# to do this is to use the GitHub secret scanning feature.
# See https://docs.github.com/en/code-security/secret-scanning/protecting-pushes-with-secret-scanning for more information.
- run: git secrets --scan --recursive .
step_job_3:
runs-on: ubuntu-latest
environment:
name: Production
needs:
- parallel_job_2
- parallel_job_1
steps:
- uses: actions/checkout@v3.6.0
- uses: aws-actions/configure-aws-credentials@v3.0.1
with:
aws-access-key-id: "$AWS_ACCESS_KEY_ID"
aws-secret-access-key: "$AWS_SECRET_ACCESS_KEY"
aws-region: "$AWS_DEFAULT_REGION"
- uses: actions/download-artifact@v3.0.1
with:
name: parallel_job_1
- run: aws s3 sync build s3://my-bucket-name
- name: Invalidate Cloudfront Distribution
run: aws cloudfront create-invalidation --distribution-id 123xyz
```
</details>
Let's compare the Bitbucket pipeline with the generated workflows and identify some key differences:
- Two workflows were created for a single Bitbucket pipeline. This is because GitHub Actions defines triggers at the workflow level, and the Bitbucket pipeline had two triggers (or start conditions).
- The default start condition's parallel steps have been split into two jobs, `parallel_job_1` and `parallel_job_2`. This modification was made to align more closely with GitHub Actions, where steps cannot run in parallel but jobs can.
```yaml
jobs:
parallel_job_1:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- name: Cache node
uses: actions/cache@v3.3.1
with:
key: "${{ runner.os }}-node-${{ hashFiles('node_modules') }}"
path: node_modules
- name: Build and Test
run: |-
npm install
npm test
parallel_job_2:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3.6.0
- name: Cache node
uses: actions/cache@v3.3.1
with:
key: "${{ runner.os }}-node-${{ hashFiles('node_modules') }}"
path: node_modules
- name: Lint the node package
run: |-
npm install eslint
npx eslint src
```
Despite these differences they will function equivalently.
## Next lab
[Use custom transformers to customize GitHub Actions Importer's behavior](./5-custom-transformers.md)
bitbucket/5-custom-transformers.md
+215
View File
@@ -0,0 +1,215 @@
# Use custom transformers to customize GitHub Actions Importer's behavior
In this lab, you will build upon the `dry-run` command to override GitHub Actions Importer's default behavior and customize the converted workflow using "custom transformers." Custom transformers can be used to:
1. Convert items that are not automatically converted.
2. Convert items that were automatically converted using different actions.
3. Convert environment variable values differently.
4. Convert references to runners to use a different runner name in GitHub Actions.
## Prerequisites
1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment.
2. Completed the [configure lab](./1-configure.md#configuring-credentials).
3. Completed the [dry-run lab](./4-dry-run.md).
## Perform a dry run
You will be performing a `dry-run` command to inspect the workflow that is converted by default. Run the following command within the codespace terminal:
```bash
gh actions-importer dry-run bitbucket --output-dir tmp/dry-run --workspace actions-importer --repository node-deploy --source-file-path ./bitbucket/bootstrap/source_files/node_deploy.yml
```
The converted workflow that is generated by the above command can be seen below:
<details>
<summary><em>Converted workflow 👇</em></summary>
```yaml
name: default
on:
push:
jobs:
step_job_1:
runs-on: ubuntu-latest
container:
image: node:16
steps:
- uses: actions/checkout@v3.6.0
- name: Build and Test
run: |-
npm install
npm test
apt update && apt install zip
zip -r app-${{ github.run_number }}.zip . -x *.git* bitbucket-pipelines.yml
- name: Code linting
run: |-
npm install eslint
npx eslint .
# # This item has no matching transformer
# - identifier: atlassian/unknown-azure-deploy:1.1.0
# name: Deploy to Production
# variables:
# AZURE_APP_ID: "$AZURE_APP_ID"
# AZURE_PASSWORD: "$AZURE_PASSWORD"
# AZURE_TENANT_ID: "$AZURE_TENANT_ID"
# AZURE_RESOURCE_GROUP: "$AZURE_RESOURCE_GROUP"
# AZURE_APP_NAME: my-site
# ZIP_FILE: my-package.zip
- uses: actions/upload-artifact@v3.1.1
with:
name: step_job_1
path: "*.zip"
```
</details>
_Note_: You can refer to the previous [lab](./4-dry-run.md) to learn about the fundamentals of the `dry-run` command.
## Custom transformers for an unknown step
The converted workflow above contains a `atlassian/unknown-azure-deploy` step that was not automatically converted. Let's write a custom transformer to handle this unknown pipe!
Let's answer the following questions before proceeding to write a custom transformer.
1) What is the "identifier" of the step to customize? This should be the identifier from the comment in the workflow without the version, or in other words the name of the pipe.
- __atlassian/unknown-azure-deploy__
2) What is the desired Actions syntax to use instead?
- Upon conducting some research, you've discovered that the [Azure Web App](https://github.com/marketplace/actions/azure-webapp) and [Azure Login](https://github.com/marketplace/actions/azure-login) actions available in the marketplace offer comparable functionality.
```yaml
- uses: azure/login@v1.4.6
with:
creds: "${{ secrets.AZURE_CREDENTIALS }}"
- uses: azure/webapps-deploy@v2.2.5
with:
app-name: my-site
package: my-package.zip
resource-group-name: "$AZURE_RESOURCE_GROUP"
```
Now you can begin to write the custom transformer. Custom transformers use a DSL built on top of Ruby and should be defined in a file with the `.rb` file extension. You can create this file by running the following command in your codespace terminal:
```bash
touch transformers.rb && code transformers.rb
```
Next, you will define a `transform` method for the `atlassian/unknown-azure-deploy` identifier by adding the following code to `transformers.rb`:
```ruby
transform "atlassian/unknown-azure-deploy" do |item|
[
{
"uses" => "azure/login@v1.4.6",
"with" => {
"creds" => "${{ secrets.AZURE_CREDENTIALS }}"
}
},
{
"name" => "Upload coverage to Codecov",
"uses" => "azure/webapps-deploy@v2.2.5",
"with" => {
"app-name" => "my-site",
"package" => item["variables"]["ZIP_FILE"],
"resource-group-name" => item["variables"]["AZURE_RESOURCE_GROUP"]
}
}
]
end
```
This method can use any valid Ruby syntax and should return a `Hash` or `Array` that represents the YAML that should be generated for a given step. GitHub Actions Importer will use this method to convert a step with the provided identifier and will use the `item` parameter for the original values configured in Bitbucket. The Bitbucket variables can be accessed in the `item` parameter via `item["variables"]["<BITBUCKET_VARIABLE_NAME>"]`.
```yaml
identifier: atlassian/unknown-azure-deploy:1.1.0
name: Deploy to Production
variables:
AZURE_APP_ID: "$AZURE_APP_ID"
AZURE_PASSWORD: "$AZURE_PASSWORD"
AZURE_TENANT_ID: "$AZURE_TENANT_ID"
AZURE_RESOURCE_GROUP: "$AZURE_RESOURCE_GROUP"
AZURE_APP_NAME: my-site
ZIP_FILE: my-package.zip
```
To access the values of the zip file and resource group information in the transformer, we are using `item["variables"]["ZIP_FILE"]` and `item["variables"]["AZURE_RESOURCE_GROUP"]`.
Now you can perform another `dry-run` command and use the `--custom-transformers` CLI option to provide this custom transformer. Run the following command within your codespace terminal:
```bash
gh actions-importer dry-run bitbucket --output-dir tmp/dry-run --workspace actions-importer --repository node-deploy --source-file-path ./bitbucket/bootstrap/source_files/node_deploy.yml --custom-transformers transformers.rb
```
The converted workflow that is generated by the above command will now use the custom logic for the `atlassian/unknown-azure-deploy` step.
## Custom transformers for runners
Next, we will use a custom transformers to dictate which runners the converted workflows should use. To do this, answer the following questions:
1. What is the label of the runner in Bitbucket to change?
- __my.custom.label__
2. What is the label of the runner in GitHub Actions to use instead?
- __some-other-runner__
With these questions answered, you can add the following code to the `transformers.rb` file:
```ruby
runner "my.custom.label", "some.other.label"
```
In this example, the first parameter to the `runner` method is the runner label in Bitbucket and the second is the new runner label to use in GitHub Actions.
Now you can perform another `dry-run` command with the `--custom-transformers` CLI option. When you open the converted workflow the `runs-on` statement will use the customized runner label:
```bash
gh actions-importer dry-run bitbucket --output-dir tmp/dry-run --workspace actions-importer --repository python --source-file-path ./bitbucket/bootstrap/source_files/python.yml --custom-transformers transformers.rb
```
> Note: we are using a different pipeline than before that defines a `runs-on` tag with the target label.
```diff
runs-on:
- - my.custom.label
+ - some.other.label
```
At this point the file contents of `transformers.rb` should match this:
<details>
<summary><em>Custom transformers 👇</em></summary>
```ruby
runner "my.custom.label", "some-other-runner"
transform "atlassian/unknown-azure-deploy" do |item|
variables = item["variables"]
[
{
"uses" => "azure/login@v1.4.6",
"with" => {
"creds" => "${{ secrets.AZURE_CREDENTIALS }}"
}
},
{
"name" => "Upload coverage to Codecov",
"uses" => "azure/webapps-deploy@v2.2.5",
"with" => {
"app-name" => "my-site",
"package" => variables["ZIP_FILE"],
"resource-group-name" => variables["AZURE_RESOURCE_GROUP"]
}
}
]
end
```
</details>
That's it. Congratulations, you have overridden GitHub Actions Importer's default behavior by customizing the conversion of:
- Unknown steps
- Runners
## Next lab
[Perform a production migration of a Bitbucket pipeline](6-migrate.md)
bitbucket/6-migrate.md
+54
View File
@@ -0,0 +1,54 @@
# Perform a production migration of a Bitbucket pipeline
In this lab, you will use the `migrate` command to convert a Bitbucket pipeline and open a pull request with the equivalent Actions workflow.
## Prerequisites
1. Followed the steps [here](./readme.md#configure-your-codespace) to set up your GitHub Codespaces environment.
2. Completed the [configure lab](./1-configure.md#configuring-credentials).
3. Completed the [dry-run lab](./4-dry-run.md).
## Performing a migration
Answer the following questions before running a `migrate` command:
1. What repository do you want to migrate?
- __basic-pipeline__
2. What is the workspace for that repository?
- __actions-importer__
3. Where do you want to store the logs?
- __tmp/migrate__
4. What is the URL for the GitHub repository to add the workflow to?
- __this repository__. The URL should should follow the pattern <https://github.com/:owner/:repo> with `:owner` and `:repo` replaced with your values.
### Steps
1. Run the below `migrate` command in the codespace terminal, remember to update the `--target-url` before executing:
```bash
gh actions-importer migrate bitbucket --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --workspace actions-importer --repository basic-pipeline --source-file-path ./bitbucket/bootstrap/source_files/basic_pipeline.yml
```
Note: The `--source-file-path` option is not required and is used throughout this lab to convert files that are stored locally. This can be omitted and GitHub Actions Importer will programmatically fetch pipeline definitions using the Bitbucket REST APIs.
2. The command will write the URL to the pull request that was created when the command succeeds.
```console
$ gh actions-importer migrate bitbucket --target-url https://github.com/:owner/:repo --output-dir tmp/migrate --workspace actions-importer --repository basic-pipeline --source-file-path ./bitbucket/bootstrap/source_files/basic_pipeline.yml
[2022-08-20 22:08:20] Logs: 'tmp/migrate/log/actions-importer-20220916-014033.log'
[2022-08-20 22:08:20] Pull request: 'https://github.com/:owner/:repo/pull/1'
```
3. Open the generated pull request in a new browser tab.
### Inspect the pull request
The first thing to notice about the pull request is that there is a list of manual steps to complete.
Next, you can inspect the "Files changed" in this pull request and see the converted workflow that is being added. Any additional changes or code reviews that were needed should be done in this pull request.
Finally, you can merge the pull request once your review has completed. You can then view the workflow running by selecting the "Actions" menu in the top navigation bar in GitHub.
At this point, the migration has completed and you have successfully migrated a Bitbucket pipeline to Actions!
### Next Lab
This concludes all labs for migrating Bitbucket pipelines to Actions with GitHub Actions Importer!
bitbucket/bootstrap/config.yml
+9
View File
@@ -0,0 +1,9 @@
source_files:
- repository_slug: python
path: bitbucket/bootstrap/source_files/python.yml
- repository_slug: ruby
path: bitbucket/bootstrap/source_files/ruby.yml
- repository_slug: docker
path: bitbucket/bootstrap/source_files/docker.yml
- repository_slug: react-deploy
path: bitbucket/bootstrap/source_files/react_deploy.yml
bitbucket/bootstrap/source_files/basic_pipeline.yml
+26
View File
@@ -0,0 +1,26 @@
image: atlassian/default-image:3
pipelines:
default:
- parallel:
- step:
name: 'Build and Test'
script:
- echo "Your build and test goes here..."
- step:
name: 'Lint'
script:
- echo "Your linting goes here..."
- step:
name: 'Security scan'
script:
- echo "Your security scan goes here..."
- step:
name: 'Deployment to Staging'
script:
- echo "Your deployment to staging script goes here..."
- step:
name: 'Deployment to Production'
trigger: 'manual'
script:
- echo "Your deployment to production script goes here..."
bitbucket/bootstrap/source_files/docker.yml
+46
View File
@@ -0,0 +1,46 @@
image: atlassian/default-image:3
pipelines:
default:
- parallel:
- step:
name: Build and Test
script:
- IMAGE_NAME=$BITBUCKET_REPO_SLUG
- docker build . --file Dockerfile --tag ${IMAGE_NAME}
services:
- docker
caches:
- docker
- step:
name: Lint the Dockerfile
image: hadolint/hadolint:latest-debian
script:
- hadolint Dockerfile
branches:
master:
- step:
name: Build and Test
script:
- IMAGE_NAME=$BITBUCKET_REPO_SLUG
- docker build . --file Dockerfile --tag ${IMAGE_NAME}
- docker save ${IMAGE_NAME} --output "${IMAGE_NAME}.tar"
services:
- docker
caches:
- docker
artifacts:
- "*.tar"
- step:
name: Deploy to Production
deployment: Production
script:
- echo ${DOCKERHUB_PASSWORD} | docker login --username "$DOCKERHUB_USERNAME" --password-stdin
- IMAGE_NAME=$BITBUCKET_REPO_SLUG
- docker load --input "${IMAGE_NAME}.tar"
- VERSION="prod-0.1.${BITBUCKET_BUILD_NUMBER}"
- IMAGE=${DOCKERHUB_NAMESPACE}/${IMAGE_NAME}
- docker tag "${IMAGE_NAME}" "${IMAGE}:${VERSION}"
- docker push "${IMAGE}:${VERSION}"
services:
- docker
bitbucket/bootstrap/source_files/forecast_jobs.json
+90
View File
@@ -0,0 +1,90 @@
[
{
"id": "{d47f79cd-9cb6-4ac8-8e85-ba448edab7df}",
"build_number": "{d47f79cd-9cb6-4ac8-8e85-ba448edab7df}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-07T08:00:46+00:00",
"finish_time": "2023-09-07T08:00:49+00:00",
"definition_id": "{a990d84b-2229-4fb5-addf-9d85a20568bb}",
"runner_name": null,
"runner_group": "hosted"
},
{
"id": "{91fa4c59-31e6-48e8-9db2-24133dd820c6}",
"build_number": "{91fa4c59-31e6-48e8-9db2-24133dd820c6}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-06T08:00:37+00:00",
"finish_time": "2023-09-06T08:00:43+00:00",
"definition_id": "{fa347742-00ea-4a28-9866-c5257b15d719}",
"runner_name": null,
"runner_group": "self-hosted"
},
{
"id": "{dd548f45-405b-4efe-9413-15a2cd585f53}",
"build_number": "{dd548f45-405b-4efe-9413-15a2cd585f53}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-05T08:00:56+00:00",
"finish_time": "2023-09-05T08:03:59+00:00",
"definition_id": "{6112d63f-6e28-4464-b9aa-7f24eb34e00f}",
"runner_name": null,
"runner_group": "self-hosted"
},
{
"id": "{4eb4521a-9159-4ff6-b328-c62cac52131f}",
"build_number": "{4eb4521a-9159-4ff6-b328-c62cac52131f}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-04T08:00:19+00:00",
"finish_time": "2023-09-04T08:00:23+00:00",
"definition_id": "{c7e80d72-f205-4603-8569-f074191187e4}",
"runner_name": null,
"runner_group": "hosted"
},
{
"id": "{12c99c31-d12d-49e5-9d38-83c5e226b0a5}",
"build_number": "{12c99c31-d12d-49e5-9d38-83c5e226b0a5}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-03T08:00:10+00:00",
"finish_time": "2023-09-03T08:00:13+00:00",
"definition_id": "{fc61a473-8718-4ddd-a89b-3343db294726}",
"runner_name": null,
"runner_group": "hosted"
},
{
"id": "{190e777d-3ff3-4f3c-9f77-cbf03e69415b}",
"build_number": "{190e777d-3ff3-4f3c-9f77-cbf03e69415b}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-02T08:00:35+00:00",
"finish_time": "2023-09-02T08:00:38+00:00",
"definition_id": "{fc5bae20-556d-43db-9be4-80bbe1eb1995}",
"runner_name": null,
"runner_group": "hosted"
},
{
"id": "{a597affa-c2a2-441b-b91b-e80751ba3ecc}",
"build_number": "{a597affa-c2a2-441b-b91b-e80751ba3ecc}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-09-01T08:00:48+00:00",
"finish_time": "2023-09-01T08:00:52+00:00",
"definition_id": "{0117b0f0-dcd3-4680-812f-6162cb919d90}",
"runner_name": null,
"runner_group": "hosted"
},
{
"id": "{ea0e658d-86d8-40cc-90e5-b69ceff1bf4e}",
"build_number": "{ea0e658d-86d8-40cc-90e5-b69ceff1bf4e}",
"result": "pipeline_step_state_completed_successful",
"queue_time": null,
"start_time": "2023-08-31T08:00:16+00:00",
"finish_time": "2023-08-31T08:00:20+00:00",
"definition_id": "{32a1db46-05b4-405f-aeac-6a34eb04f587}",
"runner_name": null,
"runner_group": "hosted"
}
]
bitbucket/bootstrap/source_files/node_deploy.yml
+29
View File
@@ -0,0 +1,29 @@
image: node:16
pipelines:
default:
- step:
name: Build and Test
script:
- npm install
- npm test
- apt update && apt install zip
- zip -r app-$BITBUCKET_BUILD_NUMBER.zip . -x *.git* bitbucket-pipelines.yml
artifacts:
- "*.zip"
- step:
name: Code linting
script:
- npm install eslint
- npx eslint .
- step:
name: Deploy to Production
script:
- pipe: atlassian/unknown-azure-deploy:1.1.0
variables:
AZURE_APP_ID: $AZURE_APP_ID
AZURE_PASSWORD: $AZURE_PASSWORD
AZURE_TENANT_ID: $AZURE_TENANT_ID
AZURE_RESOURCE_GROUP: $AZURE_RESOURCE_GROUP
AZURE_APP_NAME: "my-site"
ZIP_FILE: "my-package.zip"
bitbucket/bootstrap/source_files/python.yml
+23
View File
@@ -0,0 +1,23 @@
pipelines:
default:
- parallel:
- step:
name: Test
runs-on:
- 'self.hosted'
- 'my.custom.label'
caches:
- pip
script:
- if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- pip install pytest
- pytest -v tests/* --junitxml=test-reports/report.xml
- step:
name: Lint code
runs-on:
- 'self.hosted'
- 'my.custom.label'
script:
- pip install flake8
- flake8 . --extend-exclude=dist,build --show-source --statistics
bitbucket/bootstrap/source_files/react_deploy.yml
+55
View File
@@ -0,0 +1,55 @@
pipelines:
default:
- parallel:
- step:
name: Build and Test
caches:
- node
script:
- npm install
- npm test
- step:
name: Lint the node package
script:
- npm install eslint
- npx eslint src
caches:
- node
branches:
master:
- parallel:
- step:
name: Build and Test
caches:
- node
script:
- npm install
- npm test
- npm run build
artifacts:
- build/**
- step:
name: Security Scan
script:
- pipe: atlassian/git-secrets-scan:0.5.1
- step:
name: Deploy to Production
deployment: Production
trigger: manual
clone:
enabled: false
script:
- pipe: atlassian/aws-s3-deploy:1.1.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
S3_BUCKET: 'my-bucket-name'
LOCAL_PATH: 'build'
- pipe: atlassian/aws-cloudfront-invalidate:0.6.0
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
DISTRIBUTION_ID: '123xyz'
bitbucket/bootstrap/source_files/ruby.yml
+14
View File
@@ -0,0 +1,14 @@
image: ruby:2.7
pipelines:
default:
- parallel:
- step:
name: Test
script:
- bundle install
- bundle exec rake
- step:
name: Lint code
script:
- ruby -wc **/*.rb
bitbucket/readme.md
+72
View File
@@ -0,0 +1,72 @@
# Bitbucket Pipelines migrations powered by GitHub Actions Importer
These instructions will guide you through configuring the GitHub Codespaces environment that you will use in these labs to learn how to use GitHub Actions Importer to migrate from Bitbucket Pipelines to GitHub Actions.
These steps **must** be completed prior to starting other labs.
## Create your own repository for these labs
- Ensure that you have created a repository using [actions/importer-labs](https://github.com/actions/importer-labs) as a template.
## Configure your codespace
1. Start a new codespace
- Click the `Code` button on your repository's landing page.
- Click the `Codespaces` tab.
- Click `Create codespaces on main` to create the codespace.
- After the codespace has initialized there will be a terminal present.
2. Verify the GitHub Actions Importer CLI is installed and working. More information on the GitHub Actions Importer extension for the official GitHub CLI can be found [here](https://github.com/github/gh-actions-importer).
- Run the following command in the codespace terminal:
```bash
gh actions-importer version
```
- Verify the output is similar to below.
```console
$ gh actions-importer version
gh version 2.31.0 (2023-06-20)
gh actions-importer github/gh-actions-importer v1.3.3
actions-importer/cli unknown
```
- If `gh actions-importer version` did not produce similar output, please refer to the [troubleshooting section](#troubleshoot-the-github-actions-importer-cli).
## Labs for Bitbucket
Perform the following labs to learn more about Actions migrations with GitHub Actions Importer:
1. [Configure credentials for GitHub Actions Importer](1-configure.md)
2. [Perform an audit on Bitbucket](2-audit.md)
3. [Forecast potential build runner usage](3-forecast.md)
4. [Perform a dry-run migration of a Bitbucket pipeline](4-dry-run.md)
5. [Use custom transformers to customize GitHub Actions Importer's behavior](5-custom-transformers.md)
6. [Perform a production migration of a Bitbucket pipeline](6-migrate.md)
## Troubleshoot the GitHub Actions Importer CLI
The CLI extension for GitHub Actions Importer can be manually installed by following these steps:
- Verify you are in the codespace terminal
- Run this command from within the codespace terminal:
```bash
gh extension install github/gh-actions-importer
```
- Verify the result of the install contains:
```console
$ gh extension install github/gh-actions-importer
✓ Installed extension github/gh-actions-importer
```
- Verify GitHub Actions Importer CLI extension is installed and working by running the following command from the codespace terminal:
```bash
gh actions-importer version
```
readme.md
+1
View File
@@ -9,6 +9,7 @@ To get started:
1. Use the `actions/importer-labs` repository as a template to [generate](https://github.com/actions/importer-labs/generate) a new GitHub repository.
2. Choose where to start. There are currently learning paths for:
- [Migrating from Azure DevOps to GitHub Actions](/azure_devops/readme.md)
- [Migrating from Bitbucket to GitHub Actions](/bitbucket/readme.md)
- [Migrating from CircleCI to GitHub Actions](/circle_ci/readme.md)
- [Migrating from GitLab to GitHub Actions](/gitlab/readme.md)
- [Migrating from Jenkins to GitHub Actions](/jenkins/readme.md)