Merge pull request #41 from valet-customers/j-dunham/gitlab-labs-edits
GitLab: Rename Labs and Formatting changes
This commit is contained in:
@@ -2,15 +2,15 @@
|
||||
In this lab, you will use the Valet `configure` command to set up the required information to communicate with the GitLab and GitHub instances. The `configure` command can be used for all of the supported providers, in this lab we will be focusing on GitLab.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Configuring Valet](#configuring-valet)
|
||||
- [Verify Valet Works](#verify-valet-works)
|
||||
- [Configuring credentials](#configuring-credentials)
|
||||
- [Verify your environment](#verify-your-environment)
|
||||
- [Next Lab](#next-lab)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Followed [steps](../gitlab#readme) to set up your codespace environment.
|
||||
|
||||
## Configuring Valet
|
||||
## Configuring credentials
|
||||
1. Run the setup script in the codespace terminal `source ./gitlab/bootstrap/setup.sh` to ensure the GitLab instance is ready.
|
||||
2. Login to the GitLab instance to generate a personal access token:
|
||||
- Click the `PORTS` tab in the codespace terminal window.
|
||||
@@ -27,7 +27,7 @@ In this lab, you will use the Valet `configure` command to set up the required i
|
||||
- Select at least these scopes: `read packages` and `workflow`. Optionally, provide a text in the **Note** field and change the expiration.
|
||||
- Click `Generate token`
|
||||
- Copy the token somewhere safe and temporary.
|
||||
3. Run Valet configure commands
|
||||
4. Run Valet configure commands
|
||||
- In the codespace terminal window click back to the `TERMINAL` tab.
|
||||
- Run `gh valet configure`
|
||||
- Use the down arrow key to highlight `GitLab CI`, press the spacebar to select, then hit enter to accept.
|
||||
@@ -38,12 +38,12 @@ In this lab, you will use the Valet `configure` command to set up the required i
|
||||
- At the GitLab CI token prompt enter the GitLab CI access token from step 2 and press enter.
|
||||
- At the GitLab CI url prompt enter `http://localhost` and press enter.
|
||||
- At the Personal access token to fetch source code in GitHub prompt, if any of your Jenkins pipelines have source code in a GitHub repository enter the GitHub PAT that would have acess to these files.
|
||||
4. If all went well you should see a similar output in your terminal and a new file (.env.local) should have been created in the root of the project.
|
||||
5. If all went well you should see a similar output in your terminal and a new file (.env.local) should have been created in the root of the project.
|
||||
|
||||

|
||||
|
||||
## Verify Valet Works
|
||||
To verify Valet works we are going to run a `update` and `dry-run` command. We will go further into details about the `dry-run` command in a later lab, but for now we want to get the latest version of Valet and confirm that Valet can perform a dry-run with no errors.
|
||||
## Verify your environment
|
||||
To verify our environment is configured correctly, we are going to run a `update` and `dry-run` command. The `update` command will download the latest version of Valet to your codespace. We will go further into details about the `dry-run` command in a later lab, but for now we will just verify it finishes with no errors.
|
||||
|
||||
1. In the codespace terminal update Valet by running `gh valet update`
|
||||
2. In the terminal you should see a confirmation that it logged into the GitHub Container Registry and pulled the latest version.
|
||||
@@ -54,13 +54,14 @@ To verify Valet works we are going to run a `update` and `dry-run` command. We
|
||||
Status: Image is up to date for ghcr.io/valet-customers/valet-cli:latest
|
||||
ghcr.io/valet-customers/valet-cli:latest
|
||||
```
|
||||
3. Next, lets run the dry-run command in the codespaces terminal, to verify we can talk to GitLab
|
||||
3. Run the dry-run command in the codespaces terminal to verify Valet can communicate with the GitLab server
|
||||
```
|
||||
gh valet dry-run gitlab --output-dir ./tmp/dry-run-lab --namespace valet --project basic-pipeline-example
|
||||
```
|
||||
4. In the terminal you should see the command was successful, if not it is a good time to practice the configure command again and make sure the access tokens values are correct and were generated with the correct permissions.
|
||||
|
||||

|
||||
|
||||
### Next Lab
|
||||
[Dry run the migration of an GitLab pipeline to GitHub Actions](../gitlab/valet-audit-lab.md)
|
||||
[Perform and a audit of GitLab](../gitlab/2-audit.md)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Audit GitLab using the Valet audit command
|
||||
# Perform an audit on GitLab pipelines
|
||||
|
||||
In this lab, you will use Valet to `audit` a GitLab namespace. The `audit` command will scan the GitLab namespace for all projects with pipelines that have run at least once, perform a `dry-run` on each of those pipelines, and finally perform an aggregation of all of the transformed workflows. This aggregate summary can be used as a planning tool and help understand how complete of a migration is possible with Valet.
|
||||
The goal of this lab is to performed an audit on the demo GitLab instance, and gain a good understanding of the components that make up an audit output.
|
||||
@@ -16,24 +16,27 @@ The goal of this lab is to performed an audit on the demo GitLab instance, and g
|
||||
2. Follow all steps [here](../gitlab#valet-configure-lab) to configure Valet.
|
||||
|
||||
## Perform an audit
|
||||
Before running the command we need to collect some information:
|
||||
|
||||
1. What namespace or group do we want to audit? __valet__
|
||||
2. Where do we want to store the result? __tmp/audit__
|
||||
1. Collect information for constructing `audit` command:
|
||||
- What namespace or group do we want to audit? __valet__
|
||||
- Where do we want to store the result? __tmp/audit__
|
||||
|
||||
### Steps
|
||||
2. Navigate to the codespace terminal.
|
||||
|
||||
1. Navigate to the codespace terminal.
|
||||
2. Run the following Valet audit command with the provided answers above:
|
||||
3. Run the Valet audit command using the answers from step 1:
|
||||
|
||||
```
|
||||
gh valet audit gitlab --output-dir tmp/audit --namespace valet
|
||||
```
|
||||
|
||||
3. Valet will print the locations of the audit results in the terminal when complete
|
||||
<img width="360" alt="Screen Shot 2022-08-16 at 9 57 36 AM" src="https://user-images.githubusercontent.com/18723510/184898037-1212493b-25e2-44b8-a3dc-2b6ef703daf3.png">
|
||||
4. Valet will print the locations of the audit results in the terminal when complete.
|
||||
|
||||
5. Verify the audit results are present in the explorer.
|
||||
|
||||
<img width="360" alt="Screen Shot 2022-08-16 at 9 57 36 AM" src="https://user-images.githubusercontent.com/18723510/184898037-1212493b-25e2-44b8-a3dc-2b6ef703daf3.png">
|
||||
|
||||
## Audit Files
|
||||
|
||||
The `audit` command outputs the following files:
|
||||
|
||||
| File Type | Naming Convention | Description |
|
||||
@@ -44,12 +47,17 @@ The `audit` command outputs the following files:
|
||||
| GitHub Actions workflows | valet/PROJECT_NAME.yml | GitHub Actions workflow that would be migrated to GitHub |
|
||||
| Error file | valet/PROJECT_NAME.error.txt | Created when there is a error transforming the pipeline |
|
||||
| Audit summary | audit_summary.md | Contains a summary of audit results and the main file of interest to understand the how complete of a migration can be done with Valet |
|
||||
| GitHub Actions Reusable Workflows & Composite Actions | .github/workflows/NAME_HERE.yml | These files will only appear if Valet encountered a pipeline that would generated them. These files would be part of the migration to GitHub if they existed |
|
||||
| GitHub Actions Reusable Workflows | .github/workflows/NAME_HERE.yml | Will only appear if Valet encountered a pipeline that would generated them. |
|
||||
| GitHub Composite Actions | .github/actions/NAME_HERE.yml | Will only appear if Valet encountered a pipeline that would generated them. |
|
||||
|
||||
## Review audit summary
|
||||
|
||||
1. Under the `audit` folder find the `audit_summary.md`.
|
||||
|
||||
2. Right-click the `audit_summary.md` file and select `Open Preview`.
|
||||
|
||||
3. This file contains details about what can be migrated 100% automatically vs. what will need some manual intervention or aren't supported by GitHub Actions.
|
||||
|
||||
4. Review the file, it should match the `audit_summary` below:
|
||||
<details>
|
||||
<summary> Click to expand <code>audit_summary.md</code></summary>
|
||||
@@ -240,7 +248,9 @@ Secrets: **1**
|
||||
</details>
|
||||
|
||||
## Review the Pipelines Section
|
||||
|
||||
The audit summary starts by giving a summary of the types of pipelines that were extracted.
|
||||
|
||||
```yaml
|
||||
## Pipelines
|
||||
|
||||
@@ -271,7 +281,7 @@ Unsupported: **1 (9%)**
|
||||
- Auto DevOps: **1**
|
||||
```
|
||||
|
||||
Under the `Build steps` section we can see a breakdown of the build steps that are used in the pipelines and what was `Known` and `Unsupported` by Valet. In a later lab we will address the unsupported step `artifacts.terraform`.
|
||||
- Under the `Build steps` section we can see a breakdown of the build steps that are used in the pipelines and what was `Known` and `Unsupported` by Valet. In a later lab we will address the unsupported step `artifacts.terraform`.
|
||||
|
||||
```yaml
|
||||
### Build steps
|
||||
@@ -294,7 +304,7 @@ Unsupported: **1 (0%)**
|
||||
- artifacts.terraform: **1**
|
||||
```
|
||||
|
||||
Under the `Actions` section in `Build Steps` we have the list of the Actions that were used in order to implement the transformation of all of these build steps.
|
||||
Under the `Actions` section in `Build Steps` we have the list of the Actions that were used in order to implement the transformation of the of the `known` build steps.
|
||||
|
||||
```yaml
|
||||
Actions: **135**
|
||||
@@ -312,8 +322,8 @@ Actions: **135**
|
||||
Valet is a planning tool that can help in facilitating the migration into GitHub Actions and this list of Actions is a great place to understand what dependencies you would be taking on third-party Actions after this migration. For example, if you are doing things like setting up the allow list of third-party Actions in a GitHub Enterprise server instance this list of Actions is a fantastic place to begin security reviews and audits of what third-party actions to depend on.
|
||||
|
||||
Valet breaks down the pipeline components further into `Triggers`, `Environment`, `Other`, and `Manual tasks`.
|
||||
- Triggers are a list of pipeline trigger found
|
||||
- Environment are a list of project variables found
|
||||
- Triggers are a list of pipeline trigger found.
|
||||
- Environment are a list of project variables found.
|
||||
- Manual tasks are a list of user tasks that needs to be done in order for a pipeline to be functional when migrating to GitHub, such as adding `secrets` for a masked project variable, like we see here for the variable `PASSWORD`. In a later lab we will see how these manual tasks appear on a pull request when we do a migration.
|
||||
```
|
||||
Secrets: **1**
|
||||
@@ -336,4 +346,4 @@ The remaining sections `Successful` and `Failed` are groupings of the generated
|
||||
Note: this has files under the `.github` directory. This tells us that this pipeline generated reusable workflows from the `include` statements used in the source pipeline.
|
||||
|
||||
### Next Lab
|
||||
[Dry run the migration of a GitLab pipeline to GitHub Actions](valet-dry-run-lab.md)
|
||||
[Perform a dry-run of a GitLab pipeline](3-dry-run.md)
|
||||
@@ -1,4 +1,4 @@
|
||||
# Dry run the migration of an GitLab pipeline to GitHub Actions
|
||||
# Perform a Dry-run of a GitLab pipeline
|
||||
In this lab, you will use the Valet `dry-run` command to convert a GitLab pipeline to its equivalent GitHub Actions workflow.
|
||||
The end result of this command will be the actions workflow written to your local filesystem.
|
||||
|
||||
@@ -15,24 +15,31 @@ The end result of this command will be the actions workflow written to your loca
|
||||
|
||||
|
||||
## Perform a dry run
|
||||
We will be performing a dry-run against a preconfigured project in the GitLab instance. Before running the command we need to collect some information:
|
||||
1. What is the project we want to convert? __basic-pipeline-example__
|
||||
2. What is the namespace for that project? __Valet. In this case the namespace is the same as the group the project is in__
|
||||
3. Where do we want to store the result? __./tmp/dry-run-lab. This can be any valid path on the system. In the case of codespaces it is generally best to use `./tmp/SOME_DIRECTORY_HERE` so the files show in explorer__
|
||||
|
||||
### Steps
|
||||
1. Navigate to the codespace terminal
|
||||
2. Run the dry-run command using the values determined above
|
||||
1) Collect information to construct the `dry-run` command:
|
||||
- What is the project we want to convert?
|
||||
- basic-pipeline-example
|
||||
- What is the namespace for that project?
|
||||
- Valet. In this case the namespace is the same as the group the project is in.
|
||||
- Where do we want to store the result?
|
||||
- ./tmp/dry-run-lab. This can be any valid path on the system. In the case of codespaces it is generally best to use `./tmp/SOME_DIRECTORY_HERE` so the files show in explorer.
|
||||
|
||||
2. Run the dry-run command in the terminal using the values from step 1.
|
||||
|
||||
```
|
||||
gh valet dry-run gitlab --output-dir ./tmp/dry-run-lab --namespace valet --project basic-pipeline-example
|
||||
```
|
||||
3. When the command finishes the output files should be printed to the terminal.
|
||||
<img width="1112" alt="dry-run-terminal" src="https://user-images.githubusercontent.com/18723510/184173635-aec28d1c-8c61-4dcf-a743-f86cbdc836c5.png">
|
||||
4. Open generated actions workflow
|
||||
- Find `./tmp/dry-run-lab/valet` in the file explorer pane in codespaces.
|
||||
- Click `basic-pipeline-example.yml` to open
|
||||
<img width="231" alt="dry-run-explorer" src="https://user-images.githubusercontent.com/18723510/184177477-747905a8-32f3-4c15-8955-32079844a509.png">
|
||||
|
||||
3. Verify the command printed the result files to the terminal.
|
||||
|
||||
<img width="1112" alt="dry-run-terminal" src="https://user-images.githubusercontent.com/18723510/184173635-aec28d1c-8c61-4dcf-a743-f86cbdc836c5.png">
|
||||
|
||||
4. Open generated actions workflow
|
||||
|
||||
- Find `./tmp/dry-run-lab/valet` in the file explorer pane in codespaces.
|
||||
- Click `basic-pipeline-example.yml` to open.
|
||||
|
||||
<img width="231" alt="dry-run-explorer" src="https://user-images.githubusercontent.com/18723510/184177477-747905a8-32f3-4c15-8955-32079844a509.png">
|
||||
|
||||
## Review dry-run output
|
||||
The dry-run output will show you the GitHub Actions yaml that would be migrated to GitHub with the `migrate` command. We will now take a quick look at what was generated.
|
||||
@@ -261,6 +268,6 @@ jobs:
|
||||
Try constructing and running the `dry-run` command yourself. Hint, you should just have to change the project name.
|
||||
|
||||
## Next Lab
|
||||
[Audit GitLab Pipelines to GitHub Actions](../gitlab/valet-audit-lab.md)
|
||||
[Use custom transformers to customize Valet's behavior](../gitlab/4-custom-transformers.md)
|
||||
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Using custom transformers to customize Valet's behavior
|
||||
In this lab we want to do a `dry-run` of the `terraform-example` project. We can now perform a `dry-run` of this project, however, we have discovered that the output will need to be customized to transform the Terraform artifact report. After some research, we have determined that the `actions/upload-artifact` action will be an adequate substitute for it. Additionally, we will need to change the environment variable `PLAN_JSON` to use a different value: `custom_plan.json`. This customization will be present in many pipelines and an automated way to apply this would be ideal. In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Write Custom Transformers](#write-custom-transformers)
|
||||
- [Run with Custom Transformers](#run-with-custom-transformers)
|
||||
- [Next Lab](#next-lab)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Followed [steps](../gitlab#readme) to set up your codespace environment.
|
||||
2. Completed the [configure lab](../gitlab/valet-configure-lab.md)
|
||||
3. Completed the [dry-run lab](../gitlab/valet-dry-run-lab.md)
|
||||
|
||||
## Write Custom Transformers
|
||||
1. Run the `dry-run` command to see what information we can get from the generated action yaml.
|
||||
|
||||
```bash
|
||||
gh valet dry-run gitlab --output-dir tmp --namespace valet --project terraform-example
|
||||
```
|
||||
|
||||
2. Open the resulting GitHub Actions workflow by navigating to `tmp/valet/custom-transformer.yml` from the explorer
|
||||
|
||||
```yaml
|
||||
name: valet/custom-transformer
|
||||
on:
|
||||
push:
|
||||
workflow_dispatch:
|
||||
concurrency:
|
||||
group: "${{ github.ref }}"
|
||||
cancel-in-progress: true
|
||||
jobs:
|
||||
plan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
PLAN: plan.cache
|
||||
PLAN_JSON: plan.json
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
with:
|
||||
fetch-depth: 20
|
||||
lfs: true
|
||||
- run: terraform plan -out=$PLAN
|
||||
- run: terraform show --json $PLAN | convert_report > $PLAN_JSON
|
||||
# # 'artifacts.terraform' was not transformed because there is no suitable equivalent in GitHub Actions
|
||||
```
|
||||
3. We can see from the last line that `artifacts.terraform` was not transformed. In order for us to write a custom transformer for this we need to know the identifier. In general, the identifier will be the value between the backticks, which in this case is `artifacts.terraform`. This is how our custom transformer will target the correct step.
|
||||
|
||||
4. The custom transformers file can have any name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting.
|
||||
|
||||
5. we have chosen the `actions/upload-artifacts` as our replacement so we should look at the action's [documentation](https://github.com/marketplace/actions/upload-a-build-artifact) to determine the correct final yaml
|
||||
|
||||
```yaml
|
||||
- uses: actions/upload-artifact@v3
|
||||
with:
|
||||
path: VALUE_FROM_GITLAB
|
||||
```
|
||||
|
||||
6. Now that we know the final yaml needed for the transformer, we can start to write the ruby file. In the custom transformers file we will call the `transform` method. This is a special method that Valet exposes, that takes the identifier we determined earlier and returns a ruby Hash of the final YAML for the pipeline. The ruby Hash can be thought of as the JSON representation of the YAML we want. Valet will call that method when it encounters the identifier and pass in an `item`. The `item` is the values defined for that step in GitLab. In this case the item is the path of the terraform report.
|
||||
|
||||
```ruby
|
||||
transform "artifacts.terraform" do |item|
|
||||
{
|
||||
uses: "actions/upload-artifact@v2",
|
||||
with: {
|
||||
path: item
|
||||
}
|
||||
}
|
||||
end
|
||||
```
|
||||
|
||||
- Note: If you were unsure what `item` represents, you could use some basic ruby to print `item` to the terminal. You can achieve this by adding the following line in the transform method:
|
||||
`puts "This is the item: #{item}"`
|
||||
|
||||
7. The custom transformers file also lets you replace values of `variables` by using the `env` method. Let’s replace the value for `PLAN_JSON` by using the below line. The first value of the `env` method is the target variable name and the second is the new value to be used.
|
||||
|
||||
```ruby
|
||||
env "PLAN_JSON", "custom_plan.json"
|
||||
```
|
||||
|
||||
8. Create a new file in the root of the workspace called `transformers.rb` with the following contents
|
||||
|
||||
```ruby
|
||||
env "PLAN_JSON", "custom_plan.json"
|
||||
|
||||
transform "artifacts.terraform" do |item|
|
||||
{
|
||||
uses: "actions/upload-artifact@v2",
|
||||
with: {
|
||||
path: item
|
||||
}
|
||||
}
|
||||
end
|
||||
```
|
||||
|
||||
## Run Again with Customer Transformers
|
||||
1. Run the `dry-run` with our custom transformer by adding the `--custom-transformers` option followed by the path of the custom transformer ruby file
|
||||
|
||||
```bash
|
||||
gh valet dry-run gitlab --output-dir tmp --namespace valet --project terraform-example --custom-transformers transformers.rb
|
||||
```
|
||||
|
||||
2. Verify the custom transformer worked and now you have the `upload-artifact` in the place of the unsupported result.
|
||||
|
||||
```diff
|
||||
- # # 'artifacts.terraform' was not transformed because there is no suitable equivalent in GitHub Actions
|
||||
+ uses: actions/upload-artifact@v2
|
||||
+ with:
|
||||
+ path: "$PLAN_JSON"
|
||||
```
|
||||
|
||||
3. Verify that `PLAN_JSON` env has been updated to `custom_plan.json`
|
||||
|
||||
```diff
|
||||
env:
|
||||
PLAN: plan.cache
|
||||
- PLAN_JSON: plan.json
|
||||
+ PLAN_JSON: custom_plan.json
|
||||
```
|
||||
|
||||
Now that you have a custom transformers file you can add additional `transform`and `env` methods as needed and reuse it while running other `dry-run` and `migrate` commands.
|
||||
|
||||
Note: The custom transformers will only affect the pipeline being transformed if they contain the matching identifiers. If you believe a custom transformer should have altered the output, double check that the identifier is correct.
|
||||
|
||||
## Next Lab
|
||||
[Forecast GitLab Usage](../gitlab/5-forecast.md)
|
||||
@@ -1,11 +1,11 @@
|
||||
# Forecast the usage of a GitLab namespace
|
||||
# Forecast potential build runner usage
|
||||
|
||||
In this lab we will use the `forecast` command to forecast potential GitHub Actions usage by computing metrics from the historical pipeline data in our GitLab instance. The metrics will be stored on disk in a markdown file and include job metrics for execution time, queue time, and concurrency. We will look at each of these metrics in more depth later in this lab.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Prepare for forecast](#prepare-for-forecast)
|
||||
- [Perform a forecast](#perform-a-forecast)
|
||||
- [Review forecast report](#review-forecast-report)
|
||||
- [Review additional files](#review-additional-files)
|
||||
- [Forecasting multiple providers](#forecasting-multiple-providers)
|
||||
- [Next steps](#next-steps)
|
||||
|
||||
@@ -14,29 +14,40 @@ In this lab we will use the `forecast` command to forecast potential GitHub Acti
|
||||
1. Followed [steps](../gitlab#readme) to set up your codespace environment.
|
||||
2. Completed the [configure lab](../gitlab/valet-configure-lab.md).
|
||||
3. Ran the setup script in the terminal to make sure the GitLab instance is ready.
|
||||
```
|
||||
source gitlab/bootstrap/setup.sh
|
||||
```
|
||||
|
||||
```
|
||||
source gitlab/bootstrap/setup.sh
|
||||
```
|
||||
|
||||
## Prepare for forecast
|
||||
Before we can run the forecast we need to answer a few questions so we can construct the correct command.
|
||||
1) What namespace do we want to run the forecast for? __valet. This is the only group in the demo GitLab instance.__
|
||||
2) What is the date we want to start forecasting from? __2022-08-02. This date is before the time the data was populated on our demo GitLab instance. In practice, this should be a date that will give you enough data to get a good understanding of the typical usage. Too little data and the metrics might not give an accurate picture__
|
||||
3) Where do we want to store the results? __./tmp/forecast_reports. This can be any valid path on the system, but for simplicity it is recommend to use a directory in the root of the codespace workspace.__
|
||||
|
||||
1) What namespace do we want to run the forecast for?
|
||||
- **valet**. This is the only group in the demo GitLab instance.
|
||||
2) What is the date we want to start forecasting from?
|
||||
- **2022-08-02**. This date is before the time the data was populated on our demo GitLab instance. In practice, this should be a date that will give you enough data to get a good understanding of the typical usage. Too little data and the metrics might not give an accurate picture
|
||||
3) Where do we want to store the results?
|
||||
- **./tmp/forecast_reports**. This can be any valid path on the system, but for simplicity it is recommend to use a directory in the root of the codespace workspace.
|
||||
|
||||
## Perform a forecast
|
||||
- Using the answers above we get the following `forecast` command:
|
||||
|
||||
1. Run `forecast` command in the terminal using the answers above.
|
||||
|
||||
```
|
||||
gh valet forecast gitlab --output-dir ./tmp/forecast_reports --namespace valet --start-date 2022-08-02
|
||||
```
|
||||
- Run the command in the codespace terminal.
|
||||
- Verify that the command output is similar to this.
|
||||

|
||||
|
||||
2. Verify that the command output is similar to this.
|
||||
|
||||

|
||||
|
||||
## Review forecast report
|
||||
Open the forecast report and review the calculated metrics.
|
||||
|
||||
- From the codespace explorer pane find `./tmp/forecast_reports/forecast_report.md` and right-click, and select __Open Preview__.
|
||||
|
||||

|
||||
|
||||
- The file should be similar to this.
|
||||
<details>
|
||||
<summary>example forecast_report.md</summary>
|
||||
@@ -116,16 +127,20 @@ Open the forecast report and review the calculated metrics.
|
||||
| Max | The highest value |
|
||||
|
||||
### Total Section
|
||||
- This section shows the metrics for all of the jobs run in projects contained in the `valet` namespace, from 08/02/2022 to the time the command was executed.
|
||||
|
||||
- This section shows the metrics for all of the jobs that ran for projects contained in the `valet` namespace, from 08/02/2022 to the time the command was executed.
|
||||
|
||||
## Total
|
||||
|
||||
- Job count: **57**
|
||||
- Pipeline count: **15**
|
||||
---
|
||||
|
||||
We can see we ran 15 pipelines that contained 57 jobs. The number of jobs is expected to be larger than pipelines because a pipeline is typically a collection of jobs. For example `basic-pipeline-example` contains 6 jobs
|
||||

|
||||
|
||||
- `Execution time` shows the metrics for the time a job __took to run__. Looking closer we can see during our forecast timeframe the total job run time was 135 minutes with 90% of the jobs finishing under 7 minutes, and the longest job taking 10 minutes. The `min` is 0 because the quickest job took less than a minute and was rounded down to 0.
|
||||
|
||||
- Execution time
|
||||
- Total: **135 minutes**
|
||||
- Median: **0 minutes**
|
||||
@@ -134,20 +149,27 @@ Open the forecast report and review the calculated metrics.
|
||||
- Max: **10 minutes**
|
||||
|
||||
- `Queue time` shows the metrics for how long jobs __waited__ for a runner to be available.
|
||||
|
||||
- Queue time
|
||||
- Median: **0 minutes**
|
||||
- P90: **5 minutes**
|
||||
- Min: **0 minutes**
|
||||
- Max: **42 minutes**
|
||||
- `Concurrent jobs` show the metrics for how many jobs were run at the __same time__.
|
||||
|
||||
- `Concurrent jobs` show the metrics for how many jobs started or ended within the same 60 second time slice. The time slice window can be changed using the `--time-slice` option.
|
||||
|
||||
- Concurrent jobs
|
||||
- Median: **0**
|
||||
- P90: **0**
|
||||
- Min: **0**
|
||||
- Max: **9**
|
||||
|
||||
### Runner Group Sections
|
||||
- The preceding section shows the same metrics as the `Total` section, but are grouped by runner group. A runner group is a machine (or group of machines) that each job runs on
|
||||
|
||||
- The preceding section shows the same metrics as the `Total` section, but are grouped by runner group. A runner group is a machine (or group of machines) that jobs can run on
|
||||
|
||||
- In this case we only have one runner group `gitlab-runner` so the metrics match the `Total` section. If there were different groups we could possibly identify runner types that needed to be increased or decreased when moving to GitHub Actions
|
||||
|
||||
## gitlab-runner
|
||||
|
||||
- Job count: **57**
|
||||
@@ -173,14 +195,18 @@ Open the forecast report and review the calculated metrics.
|
||||
- Max: **9**
|
||||
|
||||
## Forecasting multiple providers
|
||||
|
||||
If we examine the help for the `forecast` command by running `gh valet forecast --help` we can see a new option `--source-file-path`
|
||||
|
||||

|
||||
|
||||
Using `--source-file-path` we can combine data from multiple forecast runs into a single report. This becomes useful if we are using multiple CI/CD providers, such as GitLab and Jenkins, and wanted to get a holistic view of the runner usage across the providers. The way this works is the forecast command creates a `.json` file in a `jobs` directory for each command execution. The `--source-file-path` takes a space-delimited list of data file paths or a glob pattern that will match all of the data files we want to include and combine into a single report. We will use a glob pattern, which in general should match `OUTPUT_DIR/**/jobs/*.json` where the `OUTPUT_DIR` is the previous value used for `--output-dir`, which in this lab was `./tmp/forecast_reports`. We do not have multiple providers but we can still try it out because we have a data file at `tmp/forecast_reports/jobs/`!
|
||||
|
||||
- run `gh valet forecast --source-file-path tmp/**/jobs/*.json -o tmp/combined-forecast`
|
||||
- Now we have a new report that was generated from all the data files that matched the glob pattern. Note this command does not introspect the CI/CD provider, it only operates on the data files it finds.
|
||||
|
||||
- Now we have a new report that was generated from all the data files that matched the glob pattern. Note this command does not introspect the CI/CD provider, it only operates on the data files it finds.
|
||||
|
||||

|
||||
|
||||
## Next steps
|
||||
This concludes the Valet labs for GitLab, if you are interested exploring the power of Valet more. You can leverage the demo GitLab Instance and modify and add new projects that more closely match your needs and try out the commands again!
|
||||
[Perform a production migration of a GitLab pipeline](../gitlab/6-migrate.md)
|
||||
@@ -1,4 +1,4 @@
|
||||
# Migrate an GitLab Project to GitHub Actions
|
||||
# Perform a production migration of a GitLab pipeline
|
||||
In this lab, we will use the Valet `migrate` command to migrate a GitLab pipeline to GitHub Actions.
|
||||
The previous commands used in the labs, such as `audit` and `dry-run` have prepared us to run a migration.
|
||||
The `migrate` command will transform the Gitlab pipeline into a GitHub Actions workflow like the `dry-run` command did, but instead of writing these files locally, it will open a pull request with the files.
|
||||
@@ -11,12 +11,14 @@ The pull request will contain a checklist of `Manual Tasks` if required. These t
|
||||
- [Next Lab](#next-lab)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Followed [steps](../gitlab#readme) to set up your codespace environment.
|
||||
2. Completed the [configure lab](../gitlab/valet-configure-lab.md)
|
||||
3. Completed the [dry-run lab](../gitlab/valet-dry-run-lab.md)
|
||||
4. Completed the [dry-run lab with custom transformer](../gitlab/valet-custom-transformers-lab.md)
|
||||
|
||||
## Preparing for migration
|
||||
|
||||
Before running the command we need to collect some information:
|
||||
1. What is the project we want to migrate? __rails-example__
|
||||
2. What is the namespace for that project? __Valet. In this case the namespace is the same as the group the project is in__
|
||||
@@ -24,22 +26,33 @@ Before running the command we need to collect some information:
|
||||
4. What is the URL for the GitHub repository we want to add the workflow too? __this repo!__. *When constructing the value for the migrate command it should match this url https://github.com/GITHUB-ORG-USERNAME-HERE/GITHUB-REPO-NAME-HERE with `GITHUB-ORG-USERNAME` and `GITHUB-REPO-NAME` substitued with your values*
|
||||
|
||||
## Performing a migration
|
||||
|
||||
1. Run `migrate` command using the information collected above, make sure to update the `--target-url` value with the information from step 4.
|
||||
|
||||
```
|
||||
gh valet migrate gitlab --target-url https://github.com/GITHUB-ORG-USERNAME-HERE/GITHUB-REPO-NAME-HERE --output-dir ./tmp/migrate --namespace valet --project rails-example
|
||||
```
|
||||
|
||||
2. Valet will create a pull request directly in the target GitHub repository.
|
||||
3. Open the pull request by clicking the green pull request link in the output of the migrate command, if you have trouble clicking it you can always copy and paste the url in your browser.
|
||||

|
||||
|
||||
4. Open the pull request by clicking the green pull request link in the output of the migrate command, if you have trouble clicking it you can always copy and paste the url in your browser.
|
||||
|
||||

|
||||
|
||||
## Reviewing the pull request
|
||||
- Lets first look at the `Conversation` tab of the PR. It tells us we have a manual task to perform before the GitHub Actions workflow is functional. We need to add a secret. We can use the GitHub [documentation](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) for secrets and add a `actions` secret for `PASSWORD` with any value.
|
||||
|
||||
- Next, lets review the workflow we are adding by clicking on `Files changed` tab. This is where you would double check everything looks good. If it didn't you could push commits with the required changes, prior to merging.
|
||||
- Now our review is completed we want to go back to the `Conversation` tab and click `Merge pull request`
|
||||
- Once the PR is merged the new workflow should start and we can view it by clicking `Actions` in the top navigation
|
||||
<img width="1119" alt="actions-screen-shot" src="https://user-images.githubusercontent.com/18723510/184960870-590b1a28-422f-4350-9ec0-0423bf7ad445.png">
|
||||
- The migration is complete and we have successfully transformed a pipeline from GitLab into a GitHub Actions workflow, added it to a GitHub Repository, and run the workflow in Actions.
|
||||
1. Lets first look at the `Conversation` tab of the PR. It tells us we have a manual task to perform before the GitHub Actions workflow is functional. We need to add a secret. We can use the GitHub [documentation](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) for secrets and add a `actions` secret for `PASSWORD` with any value.
|
||||
|
||||
2. Review the workflow we are adding by clicking on `Files changed` tab. This is where you would double check everything looks good. If it didn't you could push commits with the required changes, prior to merging.
|
||||
|
||||
3. Go back to the `Conversation` tab and click `Merge pull request`
|
||||
|
||||
4. Once the PR is merged the new workflow should start and we can view it by clicking `Actions` in the top navigation bar
|
||||
|
||||
<img width="1119" alt="actions-screen-shot" src="https://user-images.githubusercontent.com/18723510/184960870-590b1a28-422f-4350-9ec0-0423bf7ad445.png">
|
||||
|
||||
5. The migration is complete and we have successfully transformed a pipeline from GitLab into a GitHub Actions workflow, added it to a GitHub Repository, and run the workflow in Actions.
|
||||
|
||||
### Next Lab
|
||||
[Forecast GitLab Usage](../gitlab/valet-forecast-lab.md)
|
||||
|
||||
This concludes the Valet labs for GitLab, if you are interested exploring the power of Valet more. You can leverage the demo GitLab Instance and modify and add new projects that more closely match your needs and try out the commands again!
|
||||
+82
-50
@@ -1,59 +1,91 @@
|
||||
# Valet labs for GitLab
|
||||
# GitLab to Actions migrations powered by Valet
|
||||
The instructions below will guide you through configuring a GitHub Codespace environment that will be used in subsequent labs that demonstrate how to use Valet to migrate GitLab pipelines to GitHub Actions.
|
||||
|
||||
This lab bootstraps a Valet environment using GitHub Codespaces and enables you to run the Valet CI/CD migration tool against an isolated GitLab instance running in Docker.
|
||||
These steps **must** be completed prior to starting other labs.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Setup Codespace for Labs](#setup-codespace-for-labs)
|
||||
- [Labs for GitLab](#labs-for-gitlab)
|
||||
## Create your own repository for these labs
|
||||
1. Ensure that you have created a repository using the [valet-customers/labs](https://github.com/valet-customers/labs) as a template.
|
||||
|
||||
|
||||
## Prerequisites
|
||||
1. [Valet Labs](https://github.com/valet-customers/labs) template has been used to generate this repository.
|
||||
|
||||
## Setup Codespace for Labs
|
||||
## Configure your Codespace
|
||||
|
||||
1. Start the codespace
|
||||
- Click the `Code` button down arrow above the repository on the repository's landing page.
|
||||
- Click the `Codespaces` tab.
|
||||
- Click `Create codespaces on main` to create the codespace. If you are in another branch then the `main` branch, the codespace button will have the current branch specified.
|
||||
- Wait a couple minutes, then verify that the codespace starts up. Once it is fully booted up, the terminal should be present.
|
||||
2. Verify Valet CLI is installed and working. More information on the [GitHub Valet CLI extension](https://github.com/github/gh-valet)
|
||||
- Run `gh valet version` in the codespace terminal and verify it returns results like below, actual versions will vary.
|
||||
```
|
||||
gh version 2.14.3 (2022-07-26)
|
||||
gh valet github/gh-valet v0.1.12
|
||||
valet-cli unknown
|
||||
```
|
||||
if results do not match try manually installing the valet extension, using the [troubleshooting](#troubleshooting) section.
|
||||
|
||||
3. Run the GitLab setup script. This script will set up GitLab and ensure it is ready to use. In general, this script should be run first if you are starting a new codespace or restarting an existing one.
|
||||
- from the codespace terminal run `source gitlab/bootstrap/setup.sh`.
|
||||
|
||||
4. Verify you can login to the GitLab Server.
|
||||
- Click the `PORTS` tab in the codespace terminal window.
|
||||
- In the row that starts with `80` mouse over the address under the column heading `Local Address`, and click the globe icon.
|
||||
- Verify a new browser tab opened to the GitLab login screen.
|
||||
- Login with the username: `root` and password: `valet-labs!`.
|
||||
- Verify that you can see the auto populated projects in GitLab, under the `valet` namespace by clicking the `Menu` icon in GitLab, then `Your projects`.
|
||||
- Click the `Code` button down arrow above the repository on the repository's landing page.
|
||||
- Click the `Codespaces` tab.
|
||||
- Click `Create codespaces on main` to create the codespace.
|
||||
- After the Codespace has initialized there will be a terminal present.
|
||||
|
||||
2. Verify the Valet CLI is installed and working. More information on the Valet extension for the official GitHub CLI can be found [here](https://github.com/github/gh-valet).
|
||||
|
||||
- Run the following command in the codespace's terminal:
|
||||
|
||||
```bash
|
||||
gh valet version
|
||||
```
|
||||
|
||||
- Verify the output is similar to below.
|
||||
|
||||
```bash
|
||||
gh version 2.14.3 (2022-07-26)
|
||||
gh valet github/gh-valet v0.1.12
|
||||
valet-cli unknown
|
||||
```
|
||||
|
||||
- If `gh valet version` did not produce similar output then please follow the troubleshooting [guide](#troubleshoot-the-valet-cli).
|
||||
|
||||
## Bootstrap a GitLab Server
|
||||
|
||||
1. Execute the GitLab setup script that will start a container with a GitLab server running inside of it. The script should be executed when starting a new Codespace or restarting an existing one.
|
||||
|
||||
- Run the following command from the codespace's terminal
|
||||
|
||||
```bash
|
||||
source gitlab/bootstrap/setup.sh
|
||||
```
|
||||
|
||||
2. Wait for the script to finish and then verify you can login to the GitLab Server.
|
||||
|
||||
- Click the `PORTS` tab in the codespace terminal window.
|
||||
- In the row that starts with `80` mouse over the address under the column heading `Local Address`, and click the globe icon.
|
||||
- Verify a new browser tab opened to the GitLab login screen.
|
||||
- Login using the following credentials:
|
||||
- Username: `root`
|
||||
- Password: `valet-labs!`
|
||||
- Verify that you can see the auto populated projects in GitLab, under the `valet` namespace by clicking the `Menu` icon in GitLab, then `Your projects`.
|
||||
|
||||
## Labs for GitLab
|
||||
Perform the following labs to test-drive Valet
|
||||
- [Configure Valet to work with GitLab](valet-configure-lab.md)
|
||||
- [Audit GitLab using the Valet audit command](valet-audit-lab.md)
|
||||
- [Dry run the migration of an GitLab pipeline to GitHub Actions](valet-dry-run-lab.md)
|
||||
- [Using Custom Transformers in a dry-run](valet-custom-transformers-lab.md)
|
||||
- [Migrate an GitLab Project to GitHub Actions](valet-migrate-lab.md)
|
||||
- [Forecast the usage of a GitLab namespace](valet-forecast-lab.md)
|
||||
|
||||
## Troubleshooting
|
||||
- **GitHub CLI Valet extension is not installed**. More information on the [GitHub Valet CLI extension](https://github.com/github/gh-valet)
|
||||
- Verify you are in the codespace terminal
|
||||
- Run `gh extension install github/gh-valet`
|
||||
- Verify the result of the command is: `✓ Installed extension github/gh-valet`
|
||||
- If you get a similar error to the following, click the link to authorize the token
|
||||

|
||||
- Restart the codespace after clicking the link
|
||||
- **Port 80 is not being forwarded for GitLab server**. This should be auto detected by codespaces, but can be manually setup by:
|
||||
- In the codespace terminal click the PORTS tab
|
||||
- Click the "Add Port" button
|
||||
- Enter 80 and hit enter
|
||||
Perform the following labs to test-drive Valet
|
||||
|
||||
1. [Configure credentials for Valet](1-configure.md)
|
||||
2. [Perform an audit on GitLab pipelines](2-audit.md)
|
||||
3. [Perform a dry-run of a GitLab pipeline](3-dry-run.md)
|
||||
4. [Use custom transformers to customize Valet's behavior](4-custom-transformers.md)
|
||||
5. [Forecast potential build runner usage](5-forecast.md)
|
||||
6. [Perform a production migration of a GitLab pipeline](6-migrate.md)
|
||||
|
||||
## Troubleshoot the Valet CLI
|
||||
|
||||
The CLI extension for Valet can be manually installed by following these steps:
|
||||
|
||||
- Verify you are in the codespace terminal
|
||||
- Run this command from within the codespace's terminal:
|
||||
|
||||
```bash
|
||||
gh extension install github/gh-valet
|
||||
```
|
||||
|
||||
- Verify the result of the install contains:
|
||||
|
||||
```bash
|
||||
✓ Installed extension github/gh-valet
|
||||
```
|
||||
|
||||
- If you get an error similar to the image below, then click the link in the terminal output to authorize the token.
|
||||
- Restart the codespace after clicking the link.
|
||||

|
||||
- Verify Valet CLI extension is installed and working by running the following command from the codespace's terminal:
|
||||
|
||||
```bash
|
||||
gh valet version
|
||||
```
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
# Using Custom Transformers in a `dry-run`
|
||||
In this lab we want to do a `dry-run` of the `terraform-example` project. We can now perform a `dry-run` of this project, however, we have discovered that the output will need to be customized to transform the Terraform artifact report. After some research, we have determined that the `actions/upload-artifact` action will be an adequate substitute for it. Additionally, we will need to change the environment variable `PLAN_JSON` to use a different value: `custom_plan.json`. This customization will be present in many pipelines and an automated way to apply this would be ideal. In this lab, we will use the `--custom-transformers` flag to change the behavior of Valet using its DSL built on top of the Ruby language.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Write Custom Transformers](#write-custom-transformers)
|
||||
- [Run with Custom Transformers](#run-with-custom-transformers)
|
||||
- [Next Lab](#next-lab)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Followed [steps](../gitlab#readme) to set up your codespace environment.
|
||||
2. Completed the [configure lab](../gitlab/valet-configure-lab.md)
|
||||
3. Completed the [dry-run lab](../gitlab/valet-dry-run-lab.md)
|
||||
|
||||
## Write Custom Transformers
|
||||
- Let’s run the `dry-run` command to see what information we can get from the generated action yaml.
|
||||
```bash
|
||||
gh valet dry-run gitlab --output-dir tmp --namespace valet --project terraform-example
|
||||
```
|
||||
- Open the resulting GitHub Actions workflow by navigating to `tmp/valet/custom-transformer.yml` from the explorer
|
||||
```yaml
|
||||
name: valet/custom-transformer
|
||||
on:
|
||||
push:
|
||||
workflow_dispatch:
|
||||
concurrency:
|
||||
group: "${{ github.ref }}"
|
||||
cancel-in-progress: true
|
||||
jobs:
|
||||
plan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
PLAN: plan.cache
|
||||
PLAN_JSON: plan.json
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
with:
|
||||
fetch-depth: 20
|
||||
lfs: true
|
||||
- run: terraform plan -out=$PLAN
|
||||
- run: terraform show --json $PLAN | convert_report > $PLAN_JSON
|
||||
# # 'artifacts.terraform' was not transformed because there is no suitable equivalent in GitHub Actions
|
||||
```
|
||||
- We can see from the last line that `artifacts.terraform` was not transformed. In order for us to write a custom transformer for this we need to know the identifier. In general, the identifier will be the value between the backticks, which in this case is `artifacts.terraform`. This is how our custom transformer will target the correct step.
|
||||
- The custom transformers file can have any name, but it is recommended that you use an `.rb` extension so the codespaces editor knows it is a ruby file and can provide syntax highlighting.
|
||||
- we have chosen the `actions/upload-artifacts` as our replacement so we should look at the action's [documentation](https://github.com/marketplace/actions/upload-a-build-artifact) to determine the correct final yaml
|
||||
```yaml
|
||||
- uses: actions/upload-artifact@v3
|
||||
with:
|
||||
path: VALUE_FROM_GITLAB
|
||||
```
|
||||
- Now that we know the final yaml needed for the transformer, we can start to write the ruby file. In the custom transformers file we will call the `transform` method. This is a special method that Valet exposes, that takes the identifier we determined earlier and returns a ruby Hash of the final YAML for the pipeline. The ruby Hash can be thought of as the JSON representation of the YAML we want. Valet will call that method when it encounters the identifier and pass in an `item`. The `item` is the values defined for that step in GitLab. In this case the item is the path of the terraform report.
|
||||
|
||||
Note: If you were unsure what `item` represents, you could use some basic ruby to print `item` to the terminal. You can achieve this by adding the following line in the transform method:
|
||||
`puts "This is the item: #{item}"`
|
||||
```ruby
|
||||
transform "artifacts.terraform" do |item|
|
||||
{
|
||||
uses: "actions/upload-artifact@v2",
|
||||
with: {
|
||||
path: item
|
||||
}
|
||||
}
|
||||
end
|
||||
```
|
||||
|
||||
- The custom transformers file also lets you replace values of `variables` by using the `env` method. Let’s replace the value for `PLAN_JSON` by using the below line. The first value of the `env` method is the target variable name and the second is the new value to be used.
|
||||
```ruby
|
||||
env "PLAN_JSON", "custom_plan.json"
|
||||
```
|
||||
- create a new file in the root of the workspace called `transformers.rb` with the following contents
|
||||
```ruby
|
||||
env "PLAN_JSON", "custom_plan.json"
|
||||
|
||||
transform "artifacts.terraform" do |item|
|
||||
{
|
||||
uses: "actions/upload-artifact@v2",
|
||||
with: {
|
||||
path: item
|
||||
}
|
||||
}
|
||||
end
|
||||
```
|
||||
## Run Again with Customer Transformers
|
||||
To run the `dry-run` with our custom transformer we will add the `--custom-transformers` flag followed by the path of the custom transformer ruby file
|
||||
```bash
|
||||
gh valet dry-run gitlab --output-dir tmp --namespace valet --project terraform-example --custom-transformers transformers.rb
|
||||
```
|
||||
|
||||
The custom transformer worked and now we have the `upload-artifact` in the place of the unsupported result.
|
||||
```diff
|
||||
- # # 'artifacts.terraform' was not transformed because there is no suitable equivalent in GitHub Actions
|
||||
+ uses: actions/upload-artifact@v2
|
||||
+ with:
|
||||
+ path: "$PLAN_JSON"
|
||||
```
|
||||
Also we can see the `PLAN_JSON` env has been updated to `custom_plan.json`
|
||||
```diff
|
||||
env:
|
||||
PLAN: plan.cache
|
||||
- PLAN_JSON: plan.json
|
||||
+ PLAN_JSON: custom_plan.json
|
||||
```
|
||||
|
||||
Now that we have this custom transformers file we can add additional `transform`and `env` methods as needed and reuse it while running other `dry-run` and `migrate` commands.
|
||||
|
||||
Note: The custom transformers will only affect the pipeline being transformed if they contain the matching identifiers. If you believe a custom transformer should have altered the output, double check that the identifier is correct.
|
||||
|
||||
## Next Lab
|
||||
[Migrating a GitLab Pipeline](../gitlab/valet-migrate-lab.md)
|
||||
Reference in New Issue
Block a user