Compare commits

...

308 Commits

Author SHA1 Message Date
Luke Tomlinson 73d8e12de2 Fix package-lock.json 2022-03-31 11:10:17 -04:00
Luke Tomlinson 31a98126a0 Update @actions/github to 5.0.1 2022-03-31 10:44:15 -04:00
Luke Tomlinson 7e7e8d4206 Update octokit dependencies (#1037) 2022-03-31 10:36:40 -04:00
Shubham Tiwari b463992869 Adding support in cache package to check if Artifact Cache service is enabled or not (#1028)
* Added support to check if Artifact cache service is enabled or not.

* enablingForGHES

* added ACTIONS_CACHE_URL in fixtures

* Fix CI

* CI fix

* changed function name

* Function rename

* Updated release

* added test case

* Update RELEASES.md

* Lint errors

* lint

* linting

* lint

* update name to actions service

* Update packages/cache/src/internal/cacheUtils.ts

Co-authored-by: Brian Cristante <33549821+brcrista@users.noreply.github.com>

* review comments

* linting

* linting

* push to start CI

* Update RELEASES.md

* remove extra spaces

* reverting version update

* Revert "reverting version update"

This reverts commit af84eba61e.

* Update RELEASES.md

Co-authored-by: Brian Cristante <33549821+brcrista@users.noreply.github.com>
2022-03-25 14:40:02 +05:30
Ashwin Sangem 39b9640642 Merge pull request #1030 from actions/users/ashwinsangem/fix_download_chunk_cap
Cap the cache download chunk to 2 GB
2022-03-24 19:11:52 +05:30
Ashwin Sangem f0a876ab8b bumped up the @action/cache version. 2022-03-24 11:19:52 +00:00
Ashwin Sangem 58406447b5 Fixed toolkit audit by running npm audit fix. 2022-03-23 11:24:15 +00:00
Ashwin Sangem 087191dabd Update downloadUtils.ts 2022-03-23 16:46:52 +05:30
Ashwin Sangem 862c4e9db4 Cap the cache download chunk to 2 GB 2022-03-23 16:44:40 +05:30
Jonathan Tamsut d1abf7dc74 Update lockfileVersion in package-lock.json in tool-cache package (#1025)
* update packages

* update RELEASE

* update RELEASE

* remove extra README line
2022-03-16 11:27:30 -07:00
Jonathan Tamsut 475192a0c3 Update lockfileVersion in package-lock.json in cache package (#1022)
* update versions

* update release notes
2022-03-16 11:25:03 -07:00
Jonathan Tamsut c07c5fc410 Update lockfileVersion in package-lock.json in glob package (#1023)
* update versions

* update RELEASE file
2022-03-16 11:24:24 -07:00
Jonathan Tamsut b820a0ff59 Update lockfileVersion in package-lock.json in exec package (#1024)
* update packages

* update RELEASE
2022-03-16 11:24:01 -07:00
Jonathan Tamsut 72dfadb0c3 Update lockfileVersion in package-lock.json in io package (#1020)
* update lockfileVersion

* Update package
2022-03-16 11:23:44 -07:00
Jonathan Tamsut a502af8759 Merge pull request #1009 from actions/jtamsut/update-artifact-file-version
Update `lockfileVersion` for artifact package
2022-03-01 12:47:16 -08:00
Jonathan Tamsut 5905c6b5c1 Bump major version 2022-03-01 12:36:05 -08:00
Jonathan Tamsut 5e37db2c2b update lockfileVersion for artifact 2022-03-01 12:10:10 -08:00
Vipul fcb8c4ca79 Merge pull request #991 from actions/fix-dep-cache
Update ms-rest-js and storage-blog dependencies for cache
2022-02-08 09:58:05 +05:30
vsvipul 4a793fd385 Update RELEASES.md 2022-02-04 14:11:29 +05:30
Brian Cristante 15e2399826 Update CODEOWNERS with new teams (#990)
* Use the actions-cache team as owner of the cache package

* Update CODEOWNERS
2022-02-02 12:43:38 -05:00
vsvipul 39a1ec60b2 Bump up patch version 2022-02-01 17:15:42 +05:30
vsvipul eafa9d39d3 Update ms-rest-js and storage-blog dependencies for cache 2022-02-01 16:24:23 +05:30
Konrad Pabjan daf8bb0060 0.6.1 release (#964) 2021-12-14 16:01:55 -05:00
Zoran Regvart 37f5a85219 fix: drop support for named pipes on Windows (#962)
Seems that folk are having issues with uploading 0-byte files from
Windows agents. This effectively removes the support for Windows for
uploading from named files that, due to `isFIFO` returning `false` on
Windows for named pipes created using MSYS2's `mkfifo` command, resorted
to checking if the file size is 0 - a common trait of named pipes.

See https://github.com/actions/upload-artifact/issues/281
2021-12-14 15:50:50 -05:00
Konrad Pabjan d1a6612b14 Update releases.yml (#960) 2021-12-07 10:38:25 -05:00
Konrad Pabjan 6fcdd6ab0d [Artifacts] Prep for @actions/artifact 0.6.0 release (#958)
* actions-artifact-0.6.0 release

* Fix lint issue

* Update RELEASES.md
2021-12-06 18:39:23 -05:00
Konrad Pabjan 45a3c7bf81 [Artifacts] More detailed information for chunked uploads (#957)
* More detailed information for chunked uploads

* Run npm format
2021-12-06 16:48:14 -05:00
Konrad Pabjan cdd4e107a6 [Artifacts] Exempt certain types of files from gzip compression (#956)
* Exempt certain types of files from gzip compression

* Fix lint issue
2021-12-06 16:47:44 -05:00
Konrad Pabjan 88062ec473 Check for newlines and carriage return in artifact paths and name (#951)
* Check for newlines and carriage return in artifact paths and name

* Fix linting issue

* Update comments

* Add comment about spacing

* Remove extra space
2021-12-01 16:31:37 -05:00
Konrad Pabjan 4df5abb3ee Updates to logging for artifact uploads (#949)
* More details logs during artifact upload

* extra logging

* Updates to artifact logging + clarifications around upload size

* Fix linting errors

* Update packages/artifact/src/internal/artifact-client.ts

Co-authored-by: campersau <buchholz.bastian@googlemail.com>

Co-authored-by: campersau <buchholz.bastian@googlemail.com>
2021-11-30 12:53:24 -05:00
campersau e19e4261da Reset processedCount when downloading all artifacts (#889) 2021-11-29 17:28:03 -05:00
Ichinose Shogo e9b0746ee3 artifact: @types/tmp should be devDependencies (#860) 2021-11-29 17:22:33 -05:00
Zoran Regvart 7932c147a0 Support upload from named pipes (#748)
Named pipes report file size as 0, which leads to reading the whole
content into memory (0 is less than 64K). This adds additional check to
make sure that the passed in path is not a named pipe, and in that case
opts for the create-temp-file-to-gzip code path.

When running on GitHub Actions infrastructure on `windows` node, named
pipes can be created using `mkfifo` from MSYS2. In that case `fs.Stats`s
`isFIFO()` returns `false`, and not `true` as expected. This case is
detected by `process.platform` being `win32` and the passed file having
length of 0.

As a side note, when MSYS2's `mkfifo` is run, a pipe file is created:

```
prw-rw-rw- 1 User None  0 Mar 31 12:58 pipe
```

If `fs.stat` is invoked at this point `ENOENT` error will be thrown. As
soon as the pipe is written to, this pipe file is replaced by two same-
named files:

```
-rw-r--r-- 1 User None  0 Mar 31 13:00 pipe
-rw-r--r-- 1 User None  0 Mar 31 13:00 pipe
```

And at this point `fs.stat` `isFIFO()` returns `false`. Even though the
file acts as a named pipe.
2021-11-29 17:19:02 -05:00
Aparna Ravindra 45d2019161 Cache: Increasing client validation to 10GB (#934)
* increasing client validation limit in cache package to 10gb
2021-11-19 16:34:33 +05:30
Luke Tomlinson e2eeb0a784 Fix high sev in github package (#924) 2021-10-15 15:26:30 -04:00
Luke Tomlinson 6ce349e08c Update High Severity Dev Dependencies (#923)
* Update deps

* More Updates

* Use npm 7

* Update package-lock.json
2021-10-14 09:20:09 -04:00
Thomas Boop 27f76dfe1a Full release of actions/core 1.6.0 with oidc behavior (#919)
* OIDC Client for actions/core

Co-authored-by: Sourav Chanduka <souravchanduka37@gmail.com>
Co-authored-by: Sourav Chanduka <souravchanduka@users.noreply.github.com>
Co-authored-by: Tingluo Huang <tingluohuang@github.com>
2021-09-28 12:55:21 -04:00
Marcono1234 60145e408c Add file property to AnnotationProperties (#896) 2021-09-28 09:47:06 -04:00
Luke Tomlinson ea81280a4d Update release for core 1.5.0 (#873)
* Update release for core 1.5.0

* Update RELEASES.md

* Run npm audit fix
2021-08-18 09:26:19 -04:00
Luke Tomlinson f0b00fd201 Add notice annotation and support more annotation fields (#855)
* Add support for notice annotation and additional properties

* Add additional tests

* Update readme

* Change casing for endLine and endColumn

* Update utils.ts

* Update README.md

* Rename files to have internal- nomenclature

* Revert "Rename files to have internal- nomenclature"

This reverts commit 7911689f29.

* Update utils.ts
2021-07-28 17:34:31 -04:00
Rob Cowsill 4564768940 Delete temporary archive after cache upload (#792)
This is to avoid filling the SSD while saving multiple large caches
2021-06-28 17:27:09 +02:00
Brian Cristante a31b7eca9e Bump artifact package version to v0.5.2 (#845)
* bump version in package*.json

* changelog
2021-06-16 09:37:06 -04:00
Brian Cristante 9167ce1f3a Resolve vulnerabilities found by npm audit (#846) 2021-06-16 09:20:08 -04:00
Thomas Boop 11601c0d2d Release new version of the tool-cache (#838)
* update to latest version of @actions/io

* Release new version and update dependencies

* add pr number
2021-06-07 15:50:05 -04:00
Thomas Boop b9414eecb3 we really shouldn't warn on these errors, action author can decide what to do (#837) 2021-06-07 15:31:03 -04:00
Thomas Boop 243a8bba07 New versions of toolkit packages (#835) 2021-06-07 15:09:34 -04:00
Thomas Boop c5e1af5dc3 Add HashFiles to the toolkit (#830)
* add hash files to the toolkit
2021-06-07 14:26:00 -04:00
Thomas Boop c9af6bb1b3 Update escaping rules in io's rmRF (#828)
* Better Handling of escaping in rmrf
2021-06-07 14:16:16 -04:00
Luke Tomlinson bf4ce74a0f Update @actions/exec to 1.1.0 (#834) 2021-06-07 10:09:34 -04:00
Brian Cristante db21627995 Retry artifact uploads on HTTP 500 (#833)
* Retry on 500

* bump package version

* fix tests

* Remove spurious change

* fix another test

* Roll back package version
2021-06-04 17:09:30 -04:00
Thomas Boop bb2f39337d Sarpik/get input list support (#829)
* feat(core): Create `getInputList` utility

Signed-off-by: Kipras Melnikovas <kipras@kipras.org>

* chore(core): Document usage of '\n' instead of [] @ `getInputList`

Signed-off-by: Kipras Melnikovas <kipras@kipras.org>

* test(core): Create a very simple test for `getInputList`

Signed-off-by: Kipras Melnikovas <kipras@kipras.org>

* run linter

* update commands/readme

Co-authored-by: Kipras Melnikovas <kipras@kipras.org>
2021-06-04 09:28:49 -04:00
Thomas Boop dc4b4dab1d add the lint-fix script (#831) 2021-06-04 09:25:13 -04:00
Luke Tomlinson 8df94d9879 Add test for large stdline output (#827)
* Add test for large stdline output

* Format/Lint

* Update stdlineoutput.js

* Update stdlineoutput.js
2021-06-03 09:31:48 -04:00
Andrey Savitsky c5035362ab Fix broken line buffers (#773)
* Fix broken line buffers

* Code style
2021-06-02 16:29:46 -04:00
Matisse Hack 439eaced07 Add directory filtering to globber (#728)
Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
2021-06-01 15:57:03 -04:00
Thomas Boop 51dc07a106 Only run codeql on main branch pushes (#826) 2021-06-01 10:11:52 -04:00
dependabot[bot] 36b8c66aec Bump ws from 7.4.5 to 7.4.6 in /packages/github (#824)
Bumps [ws](https://github.com/websockets/ws) from 7.4.5 to 7.4.6.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/7.4.5...7.4.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-01 09:50:37 -04:00
dependabot[bot] aa29345ae8 Bump ws from 7.2.3 to 7.4.6 (#823)
Bumps [ws](https://github.com/websockets/ws) from 7.2.3 to 7.4.6.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/7.2.3...7.4.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-01 09:50:25 -04:00
Sergey Ukustov e1a7863be6 feat: get linux version from os-release file if available (#594) 2021-05-28 15:40:45 -04:00
Thomas Boop c507914181 Fix debug logging link (#820) (#822)
Co-authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2021-05-27 14:15:37 -04:00
Thomas Boop a65bca60a1 Tool Cache 1.7.0 release (#821)
* tc 1.7.0 release

* update verbiage
2021-05-27 11:44:59 -04:00
Luke Tomlinson a1b068ec31 Bugfix: Fix issue with interactive unzip on Linux (#807)
* Add new powershell commands for windows unzip

* Test fails to overwrite file

* Add new windows commands for unzip

* Add Test for failing case for both pwsh and powershell

* Modify test to confirm overwrite behavior for xar

* Delete ._test.txt

* Add fallback case for older windows systems

* Remove try

* Run Tests on windows-2016

* Update tar tests to handle existing files

* Lint

* Update tool-cache.test.ts

* Update tool-cache.test.ts

* Update tool-cache.test.ts

* Update tool-cache.test.ts

* Update from PR feedback
2021-05-21 17:01:42 -04:00
Luke Tomlinson 6e33c78c3d Update @actions/glob to 0.1.2 (#818) 2021-05-21 15:50:31 -04:00
Luke Tomlinson 9ac66375a0 Fix flakey test (#817) 2021-05-21 15:32:09 -04:00
Luke Tomlinson ddd04b6997 Add getExecOutput function (#814)
* Add getExecOutput function

* Add tests for exec output

* Modify tests to not rely on buffer size, but only test larger output

* Handle split multi-byte characters + PR feedback

* Fix tests

* Lint

* Update how split byte are sent for tests
2021-05-21 12:12:16 -04:00
Thomas Boop 566ea66979 prep for actions core 1.3.0 release (#816) 2021-05-21 09:19:53 -04:00
Thomas Boop 0d74e9080a Re-enable the audit tools step and update dependencies (#815)
* update package versions

* run audit

* fix eslint config

* linter updates

* re-enable audit

* update timeouts test

* pass done into callback

* fix format
2021-05-21 09:19:40 -04:00
Chris Mc 8dc2d6eb6a Update location of typescript definitions (#743)
https://github.com/octokit/webhooks.js#typescript
2021-05-20 16:49:57 -04:00
rethab 3bd746139f Describe behaviour of getInput (#808) 2021-05-19 11:08:51 -04:00
Thomas Boop f915ace085 add release notes (#809) 2021-05-14 14:43:08 -04:00
dependabot[bot] 1bafbed467 Bump lodash from 4.17.15 to 4.17.21 (#801)
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.15...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-14 14:39:47 -04:00
Luke Tomlinson 98549fbf21 Prevent accidental file matching when glob contains trailing slash (#805)
* Draft Solution

* Update internal-globber.ts

* Cleanup

* Fix Test

* Cleanup
2021-05-14 14:12:26 -04:00
Luke Tomlinson b33912b7cc Core: Add trimWhitespace to getInput (#802)
* Add option to not trim whitespace from inputs

* Fix typos

* Add doc clarification

* Rename options
2021-05-11 13:51:36 -04:00
dependabot[bot] cac7db2d19 Bump handlebars from 4.5.3 to 4.7.7 (#799)
Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.5.3 to 4.7.7.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.5.3...v4.7.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-07 16:37:28 -04:00
Luke Tomlinson 1c367e0a26 Export isExplicitVersion and evaluateVersions (#796)
* Export isExplicitVersion and evaluateVersions

* Lint

* Add docs
2021-05-07 16:13:26 -04:00
Luke Tomlinson 09e59b9a5c Exec: throw error when cwd option does not exist (#793)
* Exec: throw error when cwd option does not exist

* Simplify promise rejection
2021-05-07 16:12:40 -04:00
dependabot[bot] fecf6cdd59 Bump hosted-git-info from 2.7.1 to 2.8.9 (#800)
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.7.1 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.7.1...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-05-07 14:17:30 -04:00
Thomas Boop 2b97eb3192 Include urls in @actions/github context (#794)
* include urls in github context

* fix format
2021-05-07 14:05:08 -04:00
Thomas Boop ed490dc20d Update dependencies of tool-cache to fix npm audit (#795)
* Update dependencies to resolve security issue

* run npm audit fix in `actions/github`

* update jest as well to newest version
2021-05-07 14:04:38 -04:00
Luke Tomlinson 3491e2eeea Add option to cp to only copy contents of directory (#788)
* Add option to not copy source directory

* Cleanup

* Update condition to be consistent
2021-05-05 09:40:12 -04:00
Thomas Boop 208fa83feb Release @actions/github v.5.0.0 (#783)
* update latest octokit definitions

* update package versions

* update link in release notes

* update tsc version
2021-05-04 16:20:38 -04:00
Thomas Boop 393feda10a Runtime is gonna take tool-cache (#787) 2021-05-04 16:19:54 -04:00
Minh Nguyen d972090333 Fix typo in function name (#590) 2021-05-03 11:09:44 -04:00
yi_Xu fbdf27470c feat(core): add getBooleanInput function (#725)
* feat(core): add getBooleanInput function

* docs(core): update readme

* test(core): update the core.test.ts
2021-04-28 16:32:25 -04:00
Chris Gavin ff45a53422 Allow specifying arbitrary headers when downloading tools to the tool cache. (#530) 2021-04-28 14:39:15 -04:00
Thomas Boop 15fef78171 Simplify mkdirP implementation (#781)
Co-authored-by: Linus Unnebäck <linus@folkdatorn.se>
2021-04-28 14:38:41 -04:00
Simran dd046652c3 docs: Fix two typos and add some blank lines to commands.md (#607)
* docs: Fix two typos and add some blank lines to commands.md

* Update docs/commands.md

Co-authored-by: Simran <Simran-B@users.noreply.github.com>

Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
2021-04-27 10:41:51 -04:00
Scott Moeller cf3d93512b Update problem-matchers.md (#735) 2021-04-27 10:38:54 -04:00
dependabot[bot] 3512925c1c Bump ssri from 6.0.1 to 6.0.2 (#776)
Bumps [ssri](https://github.com/npm/ssri) from 6.0.1 to 6.0.2.
- [Release notes](https://github.com/npm/ssri/releases)
- [Changelog](https://github.com/npm/ssri/blob/v6.0.2/CHANGELOG.md)
- [Commits](https://github.com/npm/ssri/compare/v6.0.1...v6.0.2)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-04-27 10:06:31 -04:00
eric sciple e76decaf8a Prepend newline for set-output (#772) 2021-04-13 12:01:19 -05:00
Vikas Kedia 8afb976445 Update README.md (#531)
more logical to read save cache before restore cache
2021-04-12 10:18:07 -04:00
Tom Jenkinson b05573d945 fix formatting in core package readme (#563) 2021-04-12 09:50:56 -04:00
David Hadka 634dc61da2 Merge pull request #767 from actions/dhadka/fix-permissions
Fix permissions issues using gtar on mac
2021-04-09 14:34:51 -05:00
David Hadka aad34ab0bc Update RELEASES.md 2021-04-09 14:23:05 -05:00
Dave Hadka fac664b5d0 Fix lint issue 2021-04-09 14:16:19 -05:00
Dave Hadka 74236358e6 Update tests 2021-04-09 14:10:36 -05:00
David Hadka ac7b0e436e Update package-lock.json 2021-04-09 13:58:53 -05:00
David Hadka 440a06ef56 Update package.json 2021-04-09 13:58:35 -05:00
David Hadka 547a77cf75 Fix permissions issues using gtar on mac 2021-04-09 13:38:55 -05:00
Thomas Boop a6966e3148 fix deploy pipeline (#763) 2021-04-06 14:41:33 -04:00
Thomas Boop 92488b8ab2 Update releases.yml (#762) 2021-04-06 14:08:29 -04:00
Konrad Pabjan f628f161c4 Bump artifact package version (#761) 2021-04-06 13:49:15 -04:00
eric sciple ea2465fe63 Update and rename deno.yml to rename.yml (#721)
* Create process to release packages via actions


Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
Co-authored-by: Konrad Pabjan <konradpabjan@github.com>
2021-04-06 13:37:17 -04:00
Konrad Pabjan 770dc3a982 @actions/artifact 0.5.1 release (#760)
* Bump @actions/http-client to latest for artifact package

* Remove typescript libs to fix failing imports
2021-04-06 12:44:56 -04:00
Scott Moeller 418978c2e0 Note annotation limits in Problem Matchers documentation (#734)
* Update problem-matchers.md

* Update docs/problem-matchers.md

Fixup text

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>
2021-04-06 11:11:05 -04:00
dependabot[bot] fc00528337 Bump y18n from 4.0.0 to 4.0.1 (#753)
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-04-02 12:41:01 -04:00
Sora Morimoto bd9017e99f Add findInPath method to locate all matching executables in the system path (#609)
Signed-off-by: Sora Morimoto <sora@morimoto.io>
2021-04-02 12:22:30 -04:00
Thomas Boop de122731f3 Run update octokit on workflow_dispatch 2021-04-02 12:08:46 -04:00
Sora Morimoto 4dd900dde0 Replace find with which in io package description (#633)
Signed-off-by: Sora Morimoto <sora@morimoto.io>
2021-03-11 09:57:31 -05:00
Konrad Pabjan 383ec9fb03 Update deno.yml 2021-02-18 17:18:59 +01:00
Konrad Pabjan c3478210af Create deno.yml 2021-02-18 17:14:38 +01:00
dependabot[bot] 228a9534d1 Bump ini from 1.3.5 to 1.3.7 (#672)
Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.7.
- [Release notes](https://github.com/isaacs/ini/releases)
- [Commits](https://github.com/isaacs/ini/compare/v1.3.5...v1.3.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-02-05 13:38:54 -05:00
Konrad Pabjan 2202465c69 actions/cache 1.0.6 release (#705) 2021-02-02 20:48:46 +01:00
Benedek Kozma ccd1dd298f Use GNU tar on macOS if available (#701)
* use GNU tar on macOS

* remove unnecessary code

* formatting fix

* fix tests

* fix more tests

* typo fix

* fix test
2021-02-02 20:09:10 +01:00
Yang Cao 825204968b Merge pull request #650 from rosik/main
Make caching more verbose
2021-01-19 15:43:18 -05:00
Robin Neatherway 85f6235ca9 Add on: pull_request trigger to CodeQL workflow (#689)
From February 2021, in order to provide feedback on pull requests, Code Scanning workflows must be configured with both `push` and `pull_request` triggers. This is because Code Scanning compares the results from a pull request against the results for the base branch to tell you only what has changed between the two.

Early in the beta period we supported displaying results on pull requests for workflows with only `push` triggers, but have discontinued support as this proved to be less robust.

See https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#scanning-pull-requests for more information on how best to configure your Code Scanning workflows.
2021-01-15 12:22:00 +01:00
Yaroslav Dynnikov bfdba95ece Make caching more verbose
- Print cache size when saving cache similarly to restoring
- Print restore success similarly to saving
- Print cached file list if debug logging is enabled

See also: https://github.com/actions/cache/issues/471
2021-01-07 23:51:43 +03:00
Konrad Pabjan c861dd8859 Retry all http calls for artifact upload and download (#675)
* Retry all http calls for artifact upload and download

* Extra debug information

* Fix lint

* Always read response body

* PR Feedback

* Change error message if patch call fails

* Add exponential backoff when retrying

* Rework tests and add diagnostic info if exception thrown

* Fix lint

* fix lint error for real this time

* PR cleanup

* 0.5.0 @actions/artifact release

* Display diagnostic info if non-retryable code is hit
2020-12-18 21:40:50 +01:00
Konrad Pabjan 73d5917a6b actions/artifact 0.4.2 release (#673)
* actions/artifact 0.4.2

* Update releases note
2020-12-11 17:43:12 +01:00
Aiqiao Yan 4e1ffd548b Merge pull request #670 from actions/aiqiaoy/path
Fix windows cache path
2020-12-10 16:00:39 -05:00
Aiqiao Yan 42b3ff04b2 Fix windows cache path 2020-12-10 10:59:04 -05:00
Josh Gross 8d11ee5a8c Add CODEOWNERS (#662)
* Add CODEOWNERS

* Update owners of tool-cache
2020-12-07 15:48:38 -05:00
Chris Sidi 7ad5004f87 Merge pull request #661 from yacaovsnc/main
Retry artifact download when response stream is truncated (continued)
2020-12-04 13:10:40 -05:00
Yang Cao b593d1deb4 Close destination steam before reject
Co-authored-by: Chris Sidi <hashtagchris@github.com>
2020-12-04 11:03:17 -05:00
Yang Cao 5be846b72d style change, refactor arg sequence 2020-12-04 09:41:00 -05:00
Yang Cao dc491a61ca pipeResponseToFile already throws when reading truncated stream 2020-12-04 09:19:58 -05:00
Yang Cao a600dd34a5 Try close stream before unlink 2020-12-03 13:34:59 -05:00
Yang Cao d4e990d92f Always print skip validation log 2020-12-03 13:09:43 -05:00
Yang Cao 05b1692026 Add some debugging statements 2020-12-03 12:40:37 -05:00
Yang Cao 8ed9455d68 More styling fix 2020-12-03 10:57:04 -05:00
Yang Cao b55731c11b Fix styling 2020-12-03 10:33:10 -05:00
Yang Cao 6b83f0554a Adding retry when we received 200
If we received 200, we will attempt to download the stream. If the
stream is gzipped, gzip should throw an error when trying to decompress
the stream; or if the stream is truncated, the received bytes should be
different from the value set in content-length header.
2020-12-03 10:15:12 -05:00
Chris Sidi 990647a104 More error handling 2020-12-03 10:12:44 -05:00
Chris Sidi 520206f818 Retry artifact download when response stream is truncated 2020-12-03 10:12:44 -05:00
Chris Sidi ff4308098f Remove spyon, use readable stream to test pipeResponseToFile (#651)
* Remove spyon, use readable stream to test pipeResponseToFile

* Check file contents
2020-11-30 12:12:12 -05:00
Thomas Boop 2bf7365352 Update @actions/core in remaining toolkit packages (#636)
* Update to latest @actions/core version

* Bump release notes for new versions

* Add correct pr #
2020-11-13 16:51:15 -05:00
David Hadka e7eb2c7418 Merge pull request #626 from edburns/patch-1
Explain to the ignorant reader where this `STATE_` prefix comes from.
2020-11-05 08:15:00 -06:00
Ed Burns 0b69311011 Explain to the ignorant reader where this STATE_ prefix comes from.
Explain to the ignorant reader where this `STATE_` prefix comes from.
2020-11-04 13:50:18 -05:00
David Hadka 5e5e1b7aac Merge pull request #624 from actions/dhadka/upload-chunk-error
Handle non-successful responses from uploadChunk
2020-11-03 14:12:06 -06:00
Dave Hadka 5e8657cf12 Bump @actions/cache to 1.0.4 2020-11-03 13:12:34 -06:00
Dave Hadka 678f278caa Handle non-successful responses from uploadChunk 2020-11-03 09:53:44 -06:00
David Hadka f1b118b2a9 Merge pull request #558 from dhadka/dhadka/fix-error-handling
Handle errors representing non-successful http responses in retry logic
2020-10-20 09:54:21 -05:00
Dave Hadka 464aebd43b Update package version and release notes 2020-10-12 13:47:24 -05:00
Dave Hadka c3c81d44c1 Clean up test code 2020-10-12 13:28:50 -05:00
Thomas Boop af82147423 Add -Append flag to powershell workflow commands 2020-10-08 16:43:28 -04:00
Thomas Boop 4f7fb6513a swap to file commands (#587)
* swap to file commands

* swap to require
2020-09-30 15:46:54 +02:00
Konrad Pabjan 2178f0baee Documentation around colored output in logs (#583) 2020-09-24 16:38:19 +02:00
Thomas Boop a5e05630d8 Actions core 1.2.6 release (#582) 2020-09-23 12:10:51 -04:00
Thomas Boop 0759cdc230 Add File Commands (#571)
* Add File Commands

* pr updates w/ feedback

* run format

* fix lint/format

* slight update with an example in the docs

* pr feedback
2020-09-23 11:19:20 -04:00
dependabot[bot] da34bfb74d Bump node-fetch from 2.6.0 to 2.6.1 in /packages/github (#570)
Bumps [node-fetch](https://github.com/bitinn/node-fetch) from 2.6.0 to 2.6.1.
- [Release notes](https://github.com/bitinn/node-fetch/releases)
- [Changelog](https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md)
- [Commits](https://github.com/bitinn/node-fetch/compare/v2.6.0...v2.6.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-09-21 09:57:23 -04:00
Yang Cao bb290885a3 Release preparation of artifact 0.4.0 (#578) 2020-09-18 19:06:53 +02:00
Yang Cao ace7a82469 Add an option to specify retention days for artifacts (#575)
* Add an option to specify retention days for artifacts

* Validate against settings exposed as env var to give early feedback

* Fix lint

* Add tests and addressing feedback

* Update packages/artifact/__tests__/upload.test.ts

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

* Update packages/artifact/README.md

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

* Update packages/artifact/src/internal/utils.ts

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

* Update packages/artifact/__tests__/util.test.ts

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>

Co-authored-by: Konrad Pabjan <konradpabjan@github.com>
2020-09-18 11:30:00 -04:00
dependabot[bot] 71b19c1d65 Bump yargs-parser from 18.1.0 to 18.1.3 in /packages/github (#566)
Bumps [yargs-parser](https://github.com/yargs/yargs-parser) from 18.1.0 to 18.1.3.
- [Release notes](https://github.com/yargs/yargs-parser/releases)
- [Changelog](https://github.com/yargs/yargs-parser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/yargs-parser/compare/v18.1.0...v18.1.3)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-09-10 23:36:25 -04:00
Thomas Boop c9819f79d2 Actions/core v1.2.5 release notes (#562) 2020-08-26 15:09:39 -04:00
Thomas Boop c2bc747506 Add License.md to all npm packages (#548)
* Add License.md to all npm packages
2020-08-25 16:26:50 -04:00
github-actions[bot] ea69f13737 Update Dependencies (#561)
Co-authored-by: github-actions[bot] <github-actions@github.com>
2020-08-25 13:44:55 -04:00
Ross Brodbeck 7f7e22a940 zeit renamed to vercel 2020-08-21 07:32:09 -04:00
Dave Hadka de52c861c1 Add retry delay 2020-08-17 20:28:35 -05:00
Dave Hadka 1ef26b2390 Handle errors representing non-successful http responses in retry logic 2020-08-17 15:10:51 -05:00
Sora Morimoto 9ad01e4fd3 Use posix archive format (#533)
* Use posix archive format

Signed-off-by: Sora Morimoto <sora@morimoto.io>

* Update package.json and RELEASES.md

Signed-off-by: Sora Morimoto <sora@morimoto.io>
2020-08-05 17:17:23 +02:00
Konrad Pabjan 6c5508d1fb @actions/artifact 0.3.5 2020-08-04 17:11:50 +02:00
Konrad Pabjan b2cba168a2 Retry on 413 (#540) 2020-08-04 16:57:38 +02:00
Thomas Boop e3c6237940 Update audit.yml 2020-08-03 16:55:53 -04:00
github-actions[bot] fc28be88e4 Update Dependencies (#536)
Co-authored-by: github-actions[bot] <github-actions@github.com>
2020-08-03 11:40:36 -04:00
Konrad Pabjan 905b2c7b06 @actions/artifact 0.3.3 update 2020-07-30 17:30:41 +02:00
Konrad Pabjan 3a9dc00629 Better Artifact Telemetry + Increase Upload chunk size (#535)
* Differentiate user-agents for better internal telemetry

* Bump chunk size from 4 to 8 MB

* Update User-Agent Strings
2020-07-30 17:23:28 +02:00
Thomas Boop ccad19055e Set main as the default branch (#527)
* set main as the default branch

* revert change to tool-cache default branch

* use versions where applicable
2020-07-21 11:33:05 -04:00
Aiqiao Yan cb18a3df6b Merge pull request #528 from dhadka/dhadka/fix-download-calc
Fix bug downloading large files with the Azure SDK
2020-07-21 10:32:24 -04:00
Dave Hadka 781092b1d1 Fix bug downloading large files with the Azure SDK 2020-07-20 15:36:34 -05:00
github-actions[bot] 1cc56db0ff Update Dependencies (#526)
Co-authored-by: github-actions[bot] <github-actions@github.com>
2020-07-20 10:38:14 -04:00
dependabot[bot] 0bf9897205 Bump lodash from 4.17.15 to 4.17.19 (#524)
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.15...4.17.19)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-07-16 16:56:22 -04:00
dependabot[bot] b152907c1f Bump lodash from 4.17.15 to 4.17.19 in /packages/github (#522)
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.15...4.17.19)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-07-16 15:54:18 -04:00
Thomas Boop 8f6ddeb087 Tool cache 1.6.0 Release Notes (#519)
* Tool-cache 1.6.0 release notes

* Adjust spacing
2020-07-16 15:46:02 -04:00
Thomas Boop 32f15666bd Fix Issue where we can no longer create zip files on windows during cli tests. (#520)
* Try using pwsh instead of powershell
* Fallback to powershell
* Format files correctly
2020-07-16 11:51:00 -04:00
Frederik Wallner 2710592b73 tool-cache: Support for extracting xar compatible archives (#207)
* Test xar extraction

* Support for extracting xar compatible archives

* Only allow extractXar on mac

* Create xar during test instead of using prebuilt

* Update lockfiles

* Add verbose flag if debug

* Add extractXar example to readme

* Revert "Update lockfiles"

This reverts commit a6cbddccf6.

* Use node pkg in example

* Remove and ignore prebuilt xar

* Tests for non-existing dir and without flags

* Better arguments handling

* Make sure that target directory exists

Co-authored-by: Thomas Boop <52323235+thboop@users.noreply.github.com>
2020-07-15 14:49:23 -04:00
Thomas Boop 7e1c59c51e Update email in octokit upgrade job (#516) 2020-07-14 16:46:07 -04:00
Thomas Boop 95a10d23fa Pipe audit results to a json file so lerna does not overflow (#515)
* Pipe audit results to a json file so lerna does not overflow

* reorder flags and args
2020-07-14 16:05:53 -04:00
Aiqiao Yan 8fdeff41f3 Merge pull request #513 from dhadka/dhadka/download-progress
Display download progress when using Azure Storage SDK
2020-07-14 10:26:43 -04:00
Dave Hadka c4a92b0b60 Update with reviewer feedback 2020-07-13 16:09:37 -05:00
Dave Hadka cc474239c9 Display download progress 2020-07-13 12:46:34 -05:00
David Hadka 4964b0cc7c Use Azure storage SDK to download cache (#497)
* Adds option to download using AzCopy

* Bump version number and add release notes

* Ensure we use at least v10

* Negate env var so it disables AzCopy

* Use Azure storage SDK to download cache

* Use same level of parallelism as AzCopy

* Fix naming of variable

* React to feedback

* Bump Node types to Node 12

* Make linter happy

* Pass options into restoreCache method

* Fix tests

* Restructure files and add tests

* Add method to get the default download and upload options

* Include breaking changes in RELEASES.md

Co-authored-by: Josh Gross <joshmgross@github.com>
2020-07-10 17:09:32 +02:00
github-actions[bot] cee7d92d1d Update Dependencies (#509)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2020-07-06 16:36:08 -04:00
Thomas Boop ea503a235e Add release notes for actions/github 4.0.0 (#507)
* Add release notes for 4.0.0
2020-06-26 13:42:57 -04:00
839 2d47f7b7f6 Add execution state information (#499)
* Add execution state information (#371)

* Type conformance to REST API (#371)

* Changed to get the job name (#371)
2020-06-25 23:25:13 -04:00
Thomas Boop 7a1a0dd6fc Add core.info to Logging section 2020-06-24 10:48:13 -04:00
David Hadka 9e2d61e548 Delete cache folders prior to restore in tests (#486)
* Delete cache folders prior to restore in tests

* Update cache-tests.yml
2020-06-19 14:43:38 -04:00
Aiqiao Yan 66931ff481 Update cache readme to include a link to cache action (#478)
Co-authored-by: Aiqiao Yan <aiqiaoy@aiqiaos-mbp.attlocal.net>
2020-06-19 14:43:10 -04:00
Thomas Boop 8e14ff9f0a Setup Weekly Automation to Update @actions/github (#498)
* create automation to update Octokit for actions toolkit

Co-authored-by: Dependency Update Bot <bot@github.com>
Co-authored-by: Shohei Ueda <30958501+peaceiris@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2020-06-19 14:35:19 -04:00
Thomas Boop 3e40dd39cc Explicitly exclude DS store files (#492) 2020-06-04 15:14:19 -04:00
Thomas Boop c65fe87e33 Minor readme clarification 2020-06-03 13:51:19 -04:00
Ryo Ota d4340966b7 Add type of context.payload.comment (#375) 2020-06-03 11:54:01 -04:00
Thomas Boop d69e699ab9 Update release information for @actions/github 3.0.0 (#489) 2020-06-03 10:12:22 -04:00
Thomas Boop 4a89cf72de @actions/github v3 using Octokit/core (#453)
* Rebuild to use @Octokit/Core
2020-06-02 21:39:46 -04:00
Thomas Boop 9ba7c679ad Update Save State Docs (#467)
* Update Save State Docs
2020-05-27 10:40:32 -04:00
Sora Morimoto c94bc40d84 Remove unnecessary trailing spaces (#474) 2020-05-27 10:33:23 -04:00
Sora Morimoto 8ae8acce72 Update unit-tests.yml (#477) 2020-05-27 10:32:54 -04:00
Aiqiao Yan 30e0a77337 Merge pull request #471 from actions/aiyan/minor-edit
Fix a bug with getCompressionMethod and minor edit to release note
2020-05-19 16:42:40 -04:00
Aiqiao Yan 44a99f6c43 Add section for 0.2.1 2020-05-19 16:36:18 -04:00
Aiqiao Yan f84d1a2ae2 Fix a bug with getCompressionMethod 2020-05-19 16:12:28 -04:00
Thomas Boop 2c693984a8 Update Bug Report Template (#466) 2020-05-19 13:43:20 -04:00
Bryan MacFarlane f4aa824135 bump tool-cache version 2020-05-19 13:38:19 -04:00
Bryan MacFarlane 4e9375da09 Tool cache install from a manifest file (#382) 2020-05-19 13:25:57 -04:00
Aiqiao Yan e3a666f5b7 Minor edit to cache release note 2020-05-19 13:20:27 -04:00
Aiqiao Yan dcf5c88bb3 Merge pull request #469 from actions/users/aiyan/zstd-fixes
Fix two issues related to using zstd compression
2020-05-19 13:05:21 -04:00
Aiqiao Yan 0dea61a3a8 Bump cache package version 2020-05-19 12:59:38 -04:00
Aiqiao Yan fde6221228 React to feedback 2020-05-19 12:38:45 -04:00
Aiqiao Yan 9a3466a094 Fix windows tests 2020-05-19 11:46:50 -04:00
Aiqiao Yan 5112dc231e Only use zstd on windows when gnu tar is installed, otherwise use gzip due to bug #301 2020-05-18 16:35:13 -04:00
Aiqiao Yan 77761a4dc9 Fix issue with using zstd long range mode on ubuntu-16.04 2020-05-18 16:33:15 -04:00
Aiqiao Yan a67b91ea15 Merge pull request #448 from actions/users/aiyan/cache-package
Initial commit to create @actions/cache package
2020-05-15 13:41:32 -04:00
Konrad Pabjan 2fdf3b71f8 @actions/artifact 0.3.2 2020-05-15 18:34:09 +02:00
Aiqiao Yan d2b2399bd2 React to feedback 2020-05-15 12:26:42 -04:00
Konrad Pabjan 628f82f221 Correctly reset chunk during artifact upload on retry (#458)
* Correctly reset chunk during artifact upload on retry

* Update workflow

* Implementation details around the passthrough stream
2020-05-14 22:18:21 +02:00
Konrad Pabjan 6d83c79964 Remove --save in README (#455) 2020-05-14 17:58:46 +02:00
Thomas Boop 36e732155e tool-cache 1.3.5 release (#454)
* tc 1.3.5 release
2020-05-13 14:39:10 -04:00
Aiqiao Yan b3c8e19a7a Attempt to fix the test 2020-05-13 13:43:13 -04:00
Aiqiao Yan 1413cd0e32 Add cache upload options and pull from latest actions/cache master 2020-05-12 12:53:45 -04:00
Aiqiao Yan c534ad2cbd Add docs and tests 2020-05-12 12:02:19 -04:00
Aiqiao Yan 15fefd9336 Fix tests 2020-05-12 12:02:18 -04:00
Aiqiao Yan 7409ad5fae Change variable path to a list 2020-05-12 12:02:18 -04:00
Aiqiao Yan 932779cf58 Initial commit to create @actions/cache package 2020-05-12 12:02:18 -04:00
Konrad Pabjan 0471ed4ad7 artifact header cleanup (#441)
* Update NPM packages for @actions/artifact

* Clarifications around headers

* Revert NPM updates

* Apply suggestions from code review

Co-authored-by: Josh Gross <joshmgross@github.com>

Co-authored-by: Josh Gross <joshmgross@github.com>
2020-05-12 17:48:36 +02:00
Kevin Sawicki d1b52e7168 Update homepage to be tool-cache instead of exec (#452) 2020-05-12 10:22:22 -04:00
eric sciple 83dd3ef0f1 separate audit workflow (#450) 2020-05-07 11:39:38 -04:00
Thomas Boop a5ff692285 Redirect general feedback to the Community Forums (#447) 2020-05-07 10:24:11 -04:00
eric sciple 264e52add6 set base URL for GHES (#449) 2020-05-07 09:41:18 -04:00
Justin Hutchings 11dcc8b313 Add CodeQL Analysis workflow (#434)
* Add CodeQL Analysis workflow

* Rename .github/workflows/workflows/codeql.yml to .github/workflows/codeql.yml

* Remove autobuilder

* Add back autobuilder

* Disable c# analysis
2020-05-06 12:58:36 -04:00
dependabot[bot] 6a744be7ed Bump @actions/http-client from 1.0.3 to 1.0.8 in /packages/github (#438)
Bumps [@actions/http-client](https://github.com/actions/http-client) from 1.0.3 to 1.0.8.
- [Release notes](https://github.com/actions/http-client/releases)
- [Changelog](https://github.com/actions/http-client/blob/master/RELEASES.md)
- [Commits](https://github.com/actions/http-client/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-05-01 09:46:19 -04:00
Thomas Boop 94cd8efa47 Ignore DS_Store files. (#440)
* Ignore DSStore Files

* Simplify Ignore for DS_Store
2020-05-01 09:42:38 -04:00
dependabot[bot] 97cabf0eb9 Bump @actions/http-client from 1.0.7 to 1.0.8 in /packages/artifact (#437)
Bumps [@actions/http-client](https://github.com/actions/http-client) from 1.0.7 to 1.0.8.
- [Release notes](https://github.com/actions/http-client/releases)
- [Changelog](https://github.com/actions/http-client/blob/master/RELEASES.md)
- [Commits](https://github.com/actions/http-client/commits)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-05-01 11:32:41 +02:00
Thomas Boop 187d4aa625 @actions/core 1.2.4 + release notes (#439) 2020-04-30 09:48:16 -04:00
Arthur Baars 57d20b4db4 tool-cache: make extract functions quiet by default and more verbose if core.isDebug is set (#206)
* tool-cache: make unzip and 7z extract quiet by default

This avoids spamming the log when unzipping large archives.

* tool-cache: make tar, unzip and 7z verbose when `core.isDebug`

Make the extract function print the list of extracted file if
the action is run in debug mode.
2020-04-29 11:33:01 -04:00
Bryan MacFarlane 4e734dc4c1 bump exec version 2020-04-28 11:59:04 -04:00
Reinier Timmer 34f71e80ce Check if tool path exists before executing (#385) 2020-04-28 10:36:49 -04:00
Konrad Pabjan 1e88dec883 Update ts-jest to latest versions (#419) 2020-04-27 17:17:31 +02:00
Josh Soref 7257597d73 Spelling (#431)
* spelling: absolute

* spelling: content

* spelling: received

* spelling: sanitizes
2020-04-27 09:13:56 -04:00
per1234 df7e2c13c8 Fix broken links in documentation (#425) 2020-04-24 11:55:31 +02:00
Bryan MacFarlane eec6689a61 bump tool-cache's http-client to 1.0.8 (#429) 2020-04-23 21:12:00 -04:00
Konrad Pabjan 1688b117e1 E2E tests for the @actions/artifact package (#421)
* End-to-end artifact tests

* E2E tests for artifact package
2020-04-23 20:52:53 +02:00
Konrad Pabjan bb1053a8a7 @actions/artifact 0.3.1 update (#420)
* Updates to 0.3.1 package update
2020-04-20 22:58:53 +02:00
Pierre Grimaud a28977e977 Fix typos (#417) 2020-04-20 12:56:03 +02:00
Seth Vargo c4b6011310 Allow specifying stdin (#360)
* Allow specifying stdin
2020-04-13 13:39:42 -04:00
Thomas Boop 05e39f551d Add docs and wrapper for "echo" command (#411)
* Add docs and wrapper for "echo" command

* Update parameter to enabled
2020-04-13 13:25:54 -04:00
Thomas Boop 3c125ce4e0 Update eslint to 2.2.7 (#410) 2020-04-13 10:19:49 -04:00
Thomas Boop 5b940ebda7 Be more lenient in accepting command inputs (#405)
* Be more lenient in accepting command inputs
2020-04-09 17:00:53 -04:00
Linda_pp a6a5d98be8 Expose ExecOptions interface (#322) 2020-04-09 12:52:24 -04:00
Konrad Pabjan c010a271d9 @actions/artifact package updates (#408)
* Clear error message when storage quota has been hit

* Improved download of empty files

* Extra info to RELEASES.md

* PR Feedback
2020-04-09 17:14:12 +02:00
Konrad Pabjan 1b521c4778 Updates to @actions/artifact (#396)
* Add support for 429s and Exponential backoff

* Refactor status-reporter so it can be used with download and upload

* Extra logs

* Fixes around download & gzip

* Cleanup headers and add extra tests

* Improved Docs

* Spelling bloopers

* Improved error messages

* User http client version 1.0.7
2020-04-08 16:55:18 +02:00
Brian Cristante 36a4b7df61 Add screenshot to release instructions (#406) 2020-04-07 09:42:56 -04:00
Konrad Pabjan 905c2aa216 Contribution guidelines for artifact package (#388)
* Contribution guidelines
* PR Feedback
2020-03-24 10:13:06 +01:00
Konrad Pabjan cb7022ea2c Update implementation-details.md 2020-03-20 18:42:18 +01:00
Konrad Pabjan b94f6a1340 Update additional-information.md 2020-03-19 15:56:27 +01:00
Thomas Boop 12f30111a0 Update Contributing.md and add information about ADR's (#383)
* Updating Contributing.md + add adr details
2020-03-17 11:57:32 -04:00
eric sciple 17b97eceec adr: glob (#381) 2020-03-16 16:08:48 -04:00
Konrad Pabjan b0e01b71c0 Update package-lock.json 2020-03-12 20:39:10 +01:00
Konrad Pabjan 3748609c73 Update package.json 2020-03-12 20:36:46 +01:00
Konrad Pabjan 37590cb3ee Update RELEASES.md 2020-03-12 20:36:09 +01:00
Konrad Pabjan dffb5572a9 Updates to @actions/artifact package (#367)
* GZip implementation

* Optimizations and cleanup

* Update tests

* More test updates

* Update packages/artifact/src/internal-utils.ts

Co-Authored-By: Josh Gross <joshmgross@github.com>

* Clarification around Upload Paths

* Refactor to make http clients classes

* GZip fixes

* Documentation around compression

* More detailed status information during large uploads

* Pretty format

* Percentage updates without rounding

* Fix edge cases with formatting numbers

* Update packages/artifact/src/internal-utils.ts

Co-Authored-By: Josh Gross <joshmgross@github.com>

* Cleanup

* Small reorg with status reporter

* PR Feedback

* Cleanup + Simplification

* Test Cleanup

* Mock updates

* More cleanup

* Format fixes

* Overhaul to the http-manager

* Fix tests

* Promisify stats

* Documentation around implementation

* Improvements to documentation

* PR Feedback

* Remove Downloading multiple artifacts concurrently

Co-authored-by: Josh Gross <joshmgross@github.com>
2020-03-12 14:50:27 +01:00
eric sciple 5859d7172e only retry downloadtool on 500s and 408 and 429 (#373) 2020-03-09 14:35:53 -04:00
Thomas Boop 82fbe5da0f Update jest to 25.1 (#374)
* Update jest to 25.1

* Update acorn to 6.4.1

* Update dependencies, run audit on all packages, update packagelock

* Remove package-lock dependencies
2020-03-09 14:17:29 -04:00
eric sciple df0aa9077a generated 2020-03-05 12:07:25 -05:00
eric sciple 259743ae13 update downloadTool to handle errors from response stream and retry (#369) 2020-03-05 12:05:27 -05:00
Bryan MacFarlane 6459481e98 full sha binding in versioning doc 2020-03-03 11:16:15 -05:00
Bryan MacFarlane 3261dd9883 core 1.2.3 release (#366) 2020-03-02 08:02:40 -05:00
Tingluo Huang a649207792 Add core.isDebug() to check whether actions_step_debug is on or not. (#278) 2020-03-02 07:45:27 -05:00
Josh Gross 54bcb7c4f1 Update tool cache docs (#347) 2020-02-26 11:43:55 -05:00
Konrad Pabjan d17d4a9163 Add info about new @actions/artifact package 2020-02-20 15:05:07 -05:00
Thomas Boop 41157b23c7 Release 2.1.1 of @actions/github (#357) 2020-02-20 14:02:42 -05:00
Ryo Ota fa03eb4d22 Use import {Octokit} (#332)
* Use import {Octokit}

* Update @octokit/rest to 16.43.1
2020-02-18 15:43:07 -05:00
Konrad Pabjan f383109dc3 @actions/artifact download artifacts (#340)
* Download Artifacts using @actions/artifact
2020-02-13 18:24:11 -05:00
eric sciple 84f1e31b69 send tar --version to debug log (#342) 2020-02-13 09:54:56 -05:00
eric sciple f90a2dcfd4 temporarily revert guidance for adding matcher from container action 2020-02-12 21:42:14 -05:00
eric sciple 9cfb1604bd troubleshooting problem matchers (#341) 2020-02-12 14:59:01 -05:00
Bryan MacFarlane 2dcdf049a8 proxy guidance 2020-02-12 09:26:59 -05:00
Konrad Pabjan 6cbb8e9bc8 @actions/artifact package (#304)
* Initial commit for @actions/artifact package
2020-02-11 09:49:46 -05:00
Fabio Niephaus 0ecc141d4e await tc.downloadTool (#337) 2020-02-09 22:15:26 -05:00
eric sciple 432a78c48c check proxy bypass before setting proxy agent (#320) 2020-01-23 14:35:41 -05:00
Thomas Boop 80e6ba7033 Update readme with type assertion information (#310)
* Update readme with type assertion information

* PR updates
2020-01-22 11:53:39 -05:00
eric sciple 6072c249ee release notes (#317) 2020-01-21 13:25:05 -05:00
Thomas Boop a9175f3986 Correctly pull issue number for pull request review events (#311) 2020-01-21 10:44:38 -05:00
eric sciple 8b0300129f fix command escaping (#302) 2020-01-18 23:52:44 -05:00
eric sciple ab5bd9d696 octokit client should follow proxy settings (#314) 2020-01-18 14:28:37 -05:00
eric sciple e69833ed16 release notes (#308) 2020-01-14 11:58:44 -05:00
eric sciple 461fc2b9c9 bump checkout to v2, pin setup-node to v1 (#277) 2020-01-14 11:19:06 -05:00
sullis c514e7481a GitHub Actions checkout v2 (#303) 2020-01-14 10:48:34 -05:00
Bryan MacFarlane 2e88402d19 audit fix and update http-client (#298) 2020-01-10 16:37:48 -05:00
eric sciple 058ad6937d remove todo comment (#297) 2020-01-10 12:09:48 -05:00
eric sciple dd64d8c7c9 glob readme (#296) 2020-01-10 12:00:22 -05:00
eric sciple 947ba5b559 bump versions (#295) 2020-01-09 17:31:27 -05:00
eric sciple 03ebc5b885 generated file (#294) 2020-01-09 15:26:22 -05:00
eric sciple 1a2c592903 multiple glob patterns (#287) 2020-01-09 15:05:31 -05:00
Bryan MacFarlane 683245ad5e bump default test timeout 2020-01-07 01:14:32 -05:00
Bryan MacFarlane bfd29dcef8 only audit on ubuntu-latest (#283) 2020-01-04 14:08:05 -05:00
Bryan MacFarlane 803934eca0 audit security vulnerabilities as part of ci (#280) 2020-01-03 17:54:10 -05:00
francisfuzz 4e69ce10e9 package-lock.json: update handlebars & uglify-js (#279) 2020-01-03 15:26:02 -05:00
eric sciple a11539e1db glob (#268) 2019-12-31 10:16:18 -05:00
dependabot[bot] a94e2440cb Bump handlebars from 4.1.2 to 4.5.3 in /packages/github (#276)
Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.1.2 to 4.5.3.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/wycats/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.1.2...v4.5.3)

Signed-off-by: dependabot[bot] <support@github.com>
2019-12-27 19:48:20 -05:00
Josh Gross 8a4134761f Update to latest typescript version (#274) 2019-12-27 19:42:30 -05:00
Josh Gross 60d3096c71 Only run CI on PRs and pushes to master (#275) 2019-12-26 17:00:18 -05:00
Josh Gross 5feb835dff Fix broken doc links (#273) 2019-12-23 15:52:20 -05:00
eric sciple e7cbd693eb fix extractTar on Windows (#264) 2019-12-19 10:44:08 -05:00
eric sciple 81bdf00982 fix Buffer deprecation warnings (#265) 2019-12-18 13:24:01 -05:00
eric sciple 568f12cee6 remove trailing comma from commands (#263) 2019-12-18 13:23:16 -05:00
eric sciple f79897266e fix errors during npm install (#262) 2019-12-17 12:48:13 -05:00
Yusuke Sakurai 606e1f27ac add: "types" to each package.json (#221)
fix es #148
2019-12-17 11:03:58 -05:00
eric sciple 895bdd6dd5 remove misleading verbiage (#258) 2019-12-16 12:43:21 -05:00
eric sciple 61d502068b overload downloadTool to accept destination path (#257) 2019-12-16 11:59:48 -05:00
eric sciple 17acd9c66f update problem matchers doc for fromPath and default severity (#256) 2019-12-14 10:11:59 -05:00
191 changed files with 30886 additions and 15450 deletions
+2 -1
View File
@@ -1,3 +1,4 @@
node_modules/
packages/*/node_modules/
packages/*/lib/
packages/*/lib/
packages/glob/__tests__/_temp
+20 -8
View File
@@ -1,28 +1,42 @@
{
"plugins": ["jest", "@typescript-eslint"],
"extends": ["plugin:github/es6"],
"extends": ["plugin:github/recommended"],
"parser": "@typescript-eslint/parser",
"parserOptions": {
"ecmaVersion": 9,
"sourceType": "module",
"project": "./tsconfig.json"
"project": "./tsconfig.eslint.json"
},
"rules": {
"eslint-comments/no-use": "off",
"github/no-then": "off",
"import/no-namespace": "off",
"no-shadow": "off",
"no-unused-vars": "off",
"no-undef": "off",
"@typescript-eslint/no-unused-vars": "error",
"@typescript-eslint/explicit-member-accessibility": ["error", {"accessibility": "no-public"}],
"@typescript-eslint/no-require-imports": "error",
"@typescript-eslint/array-type": "error",
"@typescript-eslint/await-thenable": "error",
"@typescript-eslint/ban-ts-ignore": "error",
"@typescript-eslint/ban-ts-comment": "error",
"camelcase": "off",
"@typescript-eslint/camelcase": "error",
"@typescript-eslint/class-name-casing": "error",
"@typescript-eslint/camelcase": "off",
"@typescript-eslint/consistent-type-assertions": "off",
"@typescript-eslint/explicit-function-return-type": ["error", {"allowExpressions": true}],
"@typescript-eslint/func-call-spacing": ["error", "never"],
"@typescript-eslint/generic-type-naming": ["error", "^[A-Z][A-Za-z]*$"],
"@typescript-eslint/naming-convention": [
"error",
{
"format": null,
"filter": {
// you can expand this regex as you find more cases that require quoting that you want to allow
"regex": "^[A-Z][A-Za-z]*$",
"match": true
},
"selector": "memberLike"
}
],
"@typescript-eslint/no-array-constructor": "error",
"@typescript-eslint/no-empty-interface": "error",
"@typescript-eslint/no-explicit-any": "error",
@@ -32,7 +46,6 @@
"@typescript-eslint/no-misused-new": "error",
"@typescript-eslint/no-namespace": "error",
"@typescript-eslint/no-non-null-assertion": "warn",
"@typescript-eslint/no-object-literal-type-assertion": "error",
"@typescript-eslint/no-unnecessary-qualifier": "error",
"@typescript-eslint/no-unnecessary-type-assertion": "error",
"@typescript-eslint/no-useless-constructor": "error",
@@ -40,7 +53,6 @@
"@typescript-eslint/prefer-for-of": "warn",
"@typescript-eslint/prefer-function-type": "warn",
"@typescript-eslint/prefer-includes": "error",
"@typescript-eslint/prefer-interface": "error",
"@typescript-eslint/prefer-string-starts-ends-with": "error",
"@typescript-eslint/promise-function-async": "error",
"@typescript-eslint/require-array-sort-compare": "error",
+21 -2
View File
@@ -1,4 +1,23 @@
## Development
# Contributions
We welcome contributions in the form of issues and pull requests. We view the contributions and process as the same for internal and external contributors.
## Issues
Log issues for both bugs and enhancement requests. Logging issues are important for the open community.
Issues in this repository should be for the toolkit packages. General feedback for GitHub Actions should be filed in the [community forums.](https://github.community/t5/GitHub-Actions/bd-p/actions) Runner specific issues can be filed [in the runner repository](https://github.com/actions/runner).
## Enhancements and Feature Requests
We ask that before significant effort is put into code changes, that we have agreement on taking the change before time is invested in code changes.
1. Create a feature request.
2. When we agree to take the enhancement, create an ADR to agree on the details of the change.
An ADR is an Architectural Decision Record. This allows consensus on the direction forward and also serves as a record of the change and motivation. [Read more here](../docs/adrs/README.md).
## Development Life Cycle
This repository uses [Lerna](https://github.com/lerna/lerna#readme) to manage multiple packages. Read the documentation there to begin contributing.
@@ -37,4 +56,4 @@ This will ask you some questions about the new package. Start with `0.0.0` as th
}
```
3. Start developing 😄 and open a pull request.
3. Start developing 😄.
+8
View File
@@ -7,6 +7,14 @@ assignees: ''
---
Thank you 🙇‍♀ for wanting to create an issue in this repository. Before you do, please ensure you are filing the issue in the right place. Issues should only be opened on if the issue **relates to code in this repository**.
* If you have found a security issue [please submit it here](https://hackerone.com/github)
* If you have questions about writing workflows or action files, then please [visit the GitHub Community Forum's Actions Board](https://github.community/t5/GitHub-Actions/bd-p/actions)
* If you are having an issue or question about GitHub Actions then please [contact customer support](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-github-actions#contacting-support)
If your issue is relevant to this repository, please include the information below:
**Describe the bug**
A clear and concise description of what the bug is.
+95
View File
@@ -0,0 +1,95 @@
name: artifact-unit-tests
on:
push:
branches:
- main
paths-ignore:
- '**.md'
pull_request:
paths-ignore:
- '**.md'
jobs:
build:
name: Build
strategy:
matrix:
runs-on: [ubuntu-latest, windows-latest, macos-latest]
fail-fast: false
runs-on: ${{ matrix.runs-on }}
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set Node.js 12.x
uses: actions/setup-node@v1
with:
node-version: 12.x
# In order to upload & download artifacts from a shell script, certain env variables need to be set that are only available in the
# node context. This runs a local action that gets and sets the necessary env variables that are needed
- name: Set env variables
uses: ./packages/artifact/__tests__/ci-test-action/
# Need root node_modules because certain npm packages like jest are configured for the entire repository and it won't be possible
# without these to just compile the artifacts package
- name: Install root npm packages
run: npm ci
- name: Compile artifact package
run: |
npm ci
npm run tsc
working-directory: packages/artifact
- name: Set artifact file contents
shell: bash
run: |
echo "non-gzip-artifact-content=hello" >> $GITHUB_ENV
echo "gzip-artifact-content=Some large amount of text that has a compression ratio that is greater than 100%. If greater than 100%, gzip is used to upload the file" >> $GITHUB_ENV
echo "empty-artifact-content=_EMPTY_" >> $GITHUB_ENV
- name: Create files that will be uploaded
run: |
mkdir artifact-path
echo ${{ env.non-gzip-artifact-content }} > artifact-path/world.txt
echo ${{ env.gzip-artifact-content }} > artifact-path/gzip.txt
touch artifact-path/empty.txt
# We're using node -e to call the functions directly available in the @actions/artifact package
- name: Upload artifacts using uploadArtifact()
run: |
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().uploadArtifact('my-artifact-1',['artifact-path/world.txt'], '${{ github.workspace }}'))"
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().uploadArtifact('my-artifact-2',['artifact-path/gzip.txt'], '${{ github.workspace }}'))"
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().uploadArtifact('my-artifact-3',['artifact-path/empty.txt'], '${{ github.workspace }}'))"
- name: Download artifacts using downloadArtifact()
run: |
mkdir artifact-1-directory
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().downloadArtifact('my-artifact-1','artifact-1-directory'))"
mkdir artifact-2-directory
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().downloadArtifact('my-artifact-2','artifact-2-directory'))"
mkdir artifact-3-directory
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().downloadArtifact('my-artifact-3','artifact-3-directory'))"
- name: Verify downloadArtifact()
shell: bash
run: |
packages/artifact/__tests__/test-artifact-file.sh "artifact-1-directory/artifact-path/world.txt" "${{ env.non-gzip-artifact-content }}"
packages/artifact/__tests__/test-artifact-file.sh "artifact-2-directory/artifact-path/gzip.txt" "${{ env.gzip-artifact-content }}"
packages/artifact/__tests__/test-artifact-file.sh "artifact-3-directory/artifact-path/empty.txt" "${{ env.empty-artifact-content }}"
- name: Download artifacts using downloadAllArtifacts()
run: |
mkdir multi-artifact-directory
node -e "Promise.resolve(require('./packages/artifact/lib/artifact-client').create().downloadAllArtifacts('multi-artifact-directory'))"
- name: Verify downloadAllArtifacts()
shell: bash
run: |
packages/artifact/__tests__/test-artifact-file.sh "multi-artifact-directory/my-artifact-1/artifact-path/world.txt" "${{ env.non-gzip-artifact-content }}"
packages/artifact/__tests__/test-artifact-file.sh "multi-artifact-directory/my-artifact-2/artifact-path/gzip.txt" "${{ env.gzip-artifact-content }}"
packages/artifact/__tests__/test-artifact-file.sh "multi-artifact-directory/my-artifact-3/artifact-path/empty.txt" "${{ env.empty-artifact-content }}"
+39
View File
@@ -0,0 +1,39 @@
name: toolkit-audit
on:
push:
branches:
- main
paths-ignore:
- '**.md'
pull_request:
paths-ignore:
- '**.md'
jobs:
build:
name: Audit
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set Node.js 12.x
uses: actions/setup-node@v1
with:
node-version: 12.x
- name: npm install
run: npm install
- name: Bootstrap
run: npm run bootstrap
- name: audit tools
# `|| npm audit` to pretty-print the output if vulnerabilies are found after filtering.
run: npm audit --audit-level=moderate --json | scripts/audit-allow-list || npm audit --audit-level=moderate
- name: audit packages
run: npm run audit-all
+91
View File
@@ -0,0 +1,91 @@
name: cache-unit-tests
on:
push:
branches:
- main
paths-ignore:
- '**.md'
pull_request:
paths-ignore:
- '**.md'
jobs:
build:
name: Build
strategy:
matrix:
runs-on: [ubuntu-latest, windows-latest, macOS-latest]
fail-fast: false
runs-on: ${{ matrix.runs-on }}
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set Node.js 12.x
uses: actions/setup-node@v1
with:
node-version: 12.x
# In order to save & restore cache from a shell script, certain env variables need to be set that are only available in the
# node context. This runs a local action that gets and sets the necessary env variables that are needed
- name: Set env variables
uses: ./packages/cache/__tests__/__fixtures__/
# Need root node_modules because certain npm packages like jest are configured for the entire repository and it won't be possible
# without these to just compile the cache package
- name: Install root npm packages
run: npm ci
- name: Compile cache package
run: |
npm ci
npm run tsc
working-directory: packages/cache
- name: Generate files in working directory
shell: bash
run: packages/cache/__tests__/create-cache-files.sh ${{ runner.os }} test-cache
- name: Generate files outside working directory
shell: bash
run: packages/cache/__tests__/create-cache-files.sh ${{ runner.os }} ~/test-cache
# We're using node -e to call the functions directly available in the @actions/cache package
- name: Save cache using saveCache()
run: |
node -e "Promise.resolve(require('./packages/cache/lib/cache').saveCache(['test-cache','~/test-cache'],'test-${{ runner.os }}-${{ github.run_id }}'))"
- name: Delete cache folders before restoring
shell: bash
run: |
rm -rf test-cache
rm -rf ~/test-cache
- name: Restore cache using restoreCache() with http-client
run: |
node -e "Promise.resolve(require('./packages/cache/lib/cache').restoreCache(['test-cache','~/test-cache'],'test-${{ runner.os }}-${{ github.run_id }}',[],{useAzureSdk: false}))"
- name: Verify cache restored with http-client
shell: bash
run: |
packages/cache/__tests__/verify-cache-files.sh ${{ runner.os }} test-cache
packages/cache/__tests__/verify-cache-files.sh ${{ runner.os }} ~/test-cache
- name: Delete cache folders before restoring
shell: bash
run: |
rm -rf test-cache
rm -rf ~/test-cache
- name: Restore cache using restoreCache() with Azure SDK
run: |
node -e "Promise.resolve(require('./packages/cache/lib/cache').restoreCache(['test-cache','~/test-cache'],'test-${{ runner.os }}-${{ github.run_id }}'))"
- name: Verify cache restored with Azure SDK
shell: bash
run: |
packages/cache/__tests__/verify-cache-files.sh ${{ runner.os }} test-cache
packages/cache/__tests__/verify-cache-files.sh ${{ runner.os }} ~/test-cache
+37
View File
@@ -0,0 +1,37 @@
name: "Code Scanning - Action"
on:
push:
branches:
- main
pull_request:
schedule:
- cron: '0 0 * * 0'
jobs:
CodeQL-Build:
strategy:
fail-fast: false
# CodeQL runs on ubuntu-latest, windows-latest, and macos-latest
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
with:
languages: javascript
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
+80
View File
@@ -0,0 +1,80 @@
name: Publish NPM
on:
workflow_dispatch:
inputs:
package:
required: true
description: 'core, artifact, cache, exec, github, glob, io, tool-cache'
jobs:
test:
runs-on: macos-latest
steps:
- name: setup repo
uses: actions/checkout@v2
- name: verify package exists
run: ls packages/${{ github.event.inputs.package }}
- name: Set Node.js 12.x
uses: actions/setup-node@v1
with:
node-version: 12.x
- name: npm install
run: npm install
- name: bootstrap
run: npm run bootstrap
- name: build
run: npm run build
- name: test
run: npm run test
- name: pack
run: npm pack
working-directory: packages/${{ github.event.inputs.package }}
- name: upload artifact
uses: actions/upload-artifact@v2
with:
name: ${{ github.event.inputs.package }}
path: packages/${{ github.event.inputs.package }}/*.tgz
publish:
runs-on: macos-latest
needs: test
environment: npm-publish
steps:
- name: download artifact
uses: actions/download-artifact@v2
with:
name: ${{ github.event.inputs.package }}
- name: setup authentication
run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" >> .npmrc
env:
NPM_TOKEN: ${{ secrets.TOKEN }}
- name: publish
run: npm publish *.tgz
- name: notify slack on failure
if: failure()
run: |
curl -X POST -H 'Content-type: application/json' --data '{"text":":pb__failed: Failed to publish a new version of ${{ github.event.inputs.package }}"}' $SLACK_WEBHOOK
env:
SLACK_WEBHOOK: ${{ secrets.SLACK }}
- name: notify slack on success
if: success()
run: |
curl -X POST -H 'Content-type: application/json' --data '{"text":":dance: Successfully published a new version of ${{ github.event.inputs.package }}"}' $SLACK_WEBHOOK
env:
SLACK_WEBHOOK: ${{ secrets.SLACK }}
+9 -5
View File
@@ -1,11 +1,13 @@
name: toolkit-unit-tests
on:
on:
push:
branches:
- main
paths-ignore:
- '**.md'
pull_request:
paths-ignore:
- '**.md'
- '**.md'
jobs:
@@ -14,17 +16,17 @@ jobs:
strategy:
matrix:
runs-on: [ubuntu-latest, macOS-latest, windows-latest]
runs-on: [ubuntu-latest, macos-latest, windows-latest]
fail-fast: false
runs-on: ${{ matrix.runs-on }}
steps:
- name: Checkout
uses: actions/checkout@v1
uses: actions/checkout@v2
- name: Set Node.js 12.x
uses: actions/setup-node@master
uses: actions/setup-node@v1
with:
node-version: 12.x
@@ -39,6 +41,8 @@ jobs:
- name: npm test
run: npm test
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Lint
run: npm run lint
+45
View File
@@ -0,0 +1,45 @@
name: "UpdateOctokit"
on:
workflow_dispatch:
jobs:
UpdateOctokit:
runs-on: ubuntu-latest
if: ${{ github.repository_owner == 'actions' }}
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Update Octokit
working-directory: packages/github
run: |
npx npm-check-updates -u --dep prod
npm install
- name: Check Status
id: status
working-directory: packages/github
run: |
if [[ "$(git status --porcelain)" != "" ]]; then
echo "::set-output name=createPR::true"
git config --global user.email "github-actions@github.com"
git config --global user.name "github-actions[bot]"
git checkout -b bots/updateGitHubDependencies-${{github.run_number}}
git add .
git commit -m "Update Dependencies"
git push --set-upstream origin bots/updateGitHubDependencies-${{github.run_number}}
fi
- name: Create PR
if: ${{steps.status.outputs.createPR}}
uses: actions/github-script@v2
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
github.pulls.create(
{
base: "main",
owner: "${{github.repository_owner}}",
repo: "toolkit",
title: "Update Octokit dependencies",
body: "Update Octokit dependencies",
head: "bots/updateGitHubDependencies-${{github.run_number}}"
})
+4 -1
View File
@@ -1,4 +1,7 @@
node_modules/
packages/*/node_modules/
packages/*/lib/
packages/*/__tests__/_temp/
packages/*/__tests__/_temp/
.DS_Store
*.xar
packages/*/audit.json
+4
View File
@@ -0,0 +1,4 @@
* @actions/actions-runtime
/packages/artifact/ @actions/artifacts-actions
/packages/cache/ @actions/actions-cache
+2
View File
@@ -1,3 +1,5 @@
The MIT License (MIT)
Copyright 2019 GitHub
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+58 -22
View File
@@ -4,7 +4,8 @@
</p>
<p align="center">
<a href="https://github.com/actions/toolkit"><img alt="GitHub Actions status" src="https://github.com/actions/toolkit/workflows/toolkit-unit-tests/badge.svg"></a>
<a href="https://github.com/actions/toolkit/actions?query=workflow%3Atoolkit-unit-tests"><img alt="Toolkit unit tests status" src="https://github.com/actions/toolkit/workflows/toolkit-unit-tests/badge.svg"></a>
<a href="https://github.com/actions/toolkit/actions?query=workflow%3Atoolkit-audit"><img alt="Toolkit audit status" src="https://github.com/actions/toolkit/workflows/toolkit-audit/badge.svg"></a>
</p>
@@ -18,48 +19,77 @@ The GitHub Actions ToolKit provides a set of packages to make creating actions e
## Packages
:heavy_check_mark: [@actions/core](packages/core)
:heavy_check_mark: [@actions/core](packages/core)
Provides functions for inputs, outputs, results, logging, secrets and variables. Read more [here](packages/core)
```bash
$ npm install @actions/core --save
$ npm install @actions/core
```
<br/>
:runner: [@actions/exec](packages/exec)
:runner: [@actions/exec](packages/exec)
Provides functions to exec cli tools and process output. Read more [here](packages/exec)
```bash
$ npm install @actions/exec --save
$ npm install @actions/exec
```
<br/>
:pencil2: [@actions/io](packages/io)
:ice_cream: [@actions/glob](packages/glob)
Provides disk i/o functions like cp, mv, rmRF, find etc. Read more [here](packages/io)
Provides functions to search for files matching glob patterns. Read more [here](packages/glob)
```bash
$ npm install @actions/io --save
$ npm install @actions/glob
```
<br/>
:hammer: [@actions/tool-cache](packages/tool-cache)
:pencil2: [@actions/io](packages/io)
Provides disk i/o functions like cp, mv, rmRF, which etc. Read more [here](packages/io)
```bash
$ npm install @actions/io
```
<br/>
:hammer: [@actions/tool-cache](packages/tool-cache)
Provides functions for downloading and caching tools. e.g. setup-* actions. Read more [here](packages/tool-cache)
See @actions/cache for caching workflow dependencies.
```bash
$ npm install @actions/tool-cache --save
$ npm install @actions/tool-cache
```
<br/>
:octocat: [@actions/github](packages/github)
:octocat: [@actions/github](packages/github)
Provides an Octokit client hydrated with the context that the current action is being run in. Read more [here](packages/github)
```bash
$ npm install @actions/github --save
$ npm install @actions/github
```
<br/>
:floppy_disk: [@actions/artifact](packages/artifact)
Provides functions to interact with actions artifacts. Read more [here](packages/artifact)
```bash
$ npm install @actions/artifact
```
<br/>
:dart: [@actions/cache](packages/cache)
Provides functions to cache dependencies and build outputs to improve workflow execution time. Read more [here](packages/cache)
```bash
$ npm install @actions/cache
```
<br/>
@@ -83,6 +113,12 @@ Problem Matchers are a way to scan the output of actions for a specified regex p
<br/>
<br/>
:warning: [Proxy Server Support](docs/proxy-support.md)
Self-hosted runners can be configured to run behind proxy servers.
<br/>
<br/>
<h3><a href="https://github.com/actions/hello-world-javascript-action">Hello World JavaScript Action</a></h3>
Illustrates how to create a simple hello world javascript action.
@@ -96,23 +132,23 @@ Illustrates how to create a simple hello world javascript action.
<br/>
<h3><a href="https://github.com/actions/javascript-action">JavaScript Action Walkthrough</a></h3>
Walkthrough and template for creating a JavaScript Action with tests, linting, workflow, publishing, and versioning.
```javascript
async function run() {
try {
try {
const ms = core.getInput('milliseconds');
console.log(`Waiting ${ms} milliseconds ...`)
...
```
```javascript
PASS ./index.test.js
throws invalid number
wait 500 ms
throws invalid number
wait 500 ms
test runs
Test Suites: 1 passed, 1 total
Test Suites: 1 passed, 1 total
Tests: 3 passed, 3 total
```
<br/>
@@ -132,11 +168,11 @@ async function run() {
```
```javascript
PASS ./index.test.js
throws invalid number
wait 500 ms
throws invalid number
wait 500 ms
test runs
Test Suites: 1 passed, 1 total
Test Suites: 1 passed, 1 total
Tests: 3 passed, 3 total
```
<br/>
@@ -169,13 +205,13 @@ const myInput = core.getInput('myInput');
core.debug(`Hello ${myInput} from inside a container`);
const context = github.context;
console.log(`We can even get context data, like the repo: ${context.repo.repo}`)
console.log(`We can even get context data, like the repo: ${context.repo.repo}`)
```
<br/>
## Contributing
We welcome contributions. See [how to contribute](docs/contribute.md).
We welcome contributions. See [how to contribute](.github/CONTRIBUTING.md).
## Code of Conduct
+16 -11
View File
@@ -1,9 +1,23 @@
# Debugging
If the build logs do not provide enough detail on why a build may be failing, some other options exist to assist with troubleshooting.
If the job logs do not provide enough detail on why a job may be failing, some other options exist to assist with troubleshooting.
## Step Debug Logs
This is the primary way for customers to debug job failures caused by failed steps.
Step debug logs increase the verbosity of a job's logs during and after a job's execution to assist with troubleshooting.
Additional log events with the prefix `::debug::` will now also appear in the job's logs, these log events are provided by the Action's author and the runner process.
### How to Access Step Debug Logs
This flag can be enabled by [setting the secret](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets#creating-encrypted-secrets) `ACTIONS_STEP_DEBUG` to `true`.
All actions ran while this secret is enabled will show debug events in the [Downloaded Logs](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#downloading-logs) and [Web Logs](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#viewing-logs-to-diagnose-failures).
## Runner Diagnostic Logs
Runner Diagnostic Logs provide additional log files detailing how the Runner is executing an action.
You need the runner diagnostic logs only if you think there is an infrastructure problem with GitHub Actions and you want the product team to check the logs.
Each file contains different logging information that corresponds to that process:
* The Runner process coordinates setting up workers to execute jobs.
* The Worker process executes the job.
@@ -13,14 +27,5 @@ These files contain the prefix `Runner_` or `Worker_` to indicate the log source
### How to Access Runner Diagnostic Logs
These log files are enabled by [setting the secret](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets#creating-encrypted-secrets) `ACTIONS_RUNNER_DEBUG` to `true`.
All actions ran while this secret is enabled contain additional diagnostic log files in the `runner-diagnostic-logs` folder of the [log archive](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#downloading-logs-and-artifacts).
All actions ran while this secret is enabled contain additional diagnostic log files in the `runner-diagnostic-logs` folder of the [log archive](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#downloading-logs).
## Step Debug Logs
Step debug logs increase the verbosity of a job's logs during and after a job's execution to assist with troubleshooting.
Additional log events with the prefix `::debug::` will now also appear in the job's logs.
### How to Access Step Debug Logs
This flag can be enabled by [setting the secret](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets#creating-encrypted-secrets) `ACTIONS_STEP_DEBUG` to `true`.
All actions ran while this secret is enabled will show debug events in the [Downloaded Logs](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#downloading-logs-and-artifacts) and [Web Logs](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#viewing-logs-to-diagnose-failures).
+2 -2
View File
@@ -32,14 +32,14 @@ jobs:
os: [ubuntu-16.04, windows-2019]
runs-on: ${{matrix.os}}
actions:
- uses: actions/setup-node@master
- uses: actions/setup-node@v1
with:
version: ${{matrix.node}}
- run: |
npm install
- run: |
npm test
- uses: actions/custom-action@master
- uses: actions/custom-action@v1
```
JavaScript actions work on any environment that host action runtime is supported on which is currently node 12. However, a host action that runs a toolset expects the environment that it's running on to have that toolset in its PATH or using a setup-* action to acquire it on demand.
+8 -6
View File
@@ -8,7 +8,7 @@ Examples:
steps:
- uses: actions/javascript-action@v1 # recommended. starter workflows use this
- uses: actions/javascript-action@v1.0.0 # if an action offers specific releases
- uses: actions/javascript-action@41775a4 # binding to a specific sha
- uses: actions/javascript-action@41775a4da8ffae865553a738ab8ac1cd5a3c0044 # sha
```
# Compatibility
@@ -17,16 +17,16 @@ Binding to a major version is the latest of that major version ( e.g. `v1` == "1
Major versions should guarantee compatibility. A major version can add net new capabilities but should not break existing input compatibility or break existing workflows.
Major version binding allows you to take advantage of bug fixes and critical functionality and security fixes. The `master` branch has the latest code and is unstable to bind to since changes get committed to master and released to the market place by creating a tag. In addition, a new major version carrying breaking changes will get implemented in master after branching off the previous major version.
Major version binding allows you to take advantage of bug fixes and critical functionality and security fixes. The `main` branch has the latest code and is unstable to bind to since changes get committed to main and released to the market place by creating a tag. In addition, a new major version carrying breaking changes will get implemented in main after branching off the previous major version.
> Warning: do not reference `master` since that is the latest code and can be carrying breaking changes of the next major version.
> Warning: do not reference `main` since that is the latest code and can be carrying breaking changes of the next major version.
```yaml
steps:
- uses: actions/javascript-action@master # do not do this
- uses: actions/javascript-action@main # do not do this
```
Binding to the immutable sha1 may offer more reliability. However, note that the hosted images toolsets (e.g. ubuntu-latest) move forward and if there is a tool breaking issue, actions may react with a patch to a major version to compensate so binding to a specific SHA may prevent you from getting fixes.
Binding to the immutable full sha1 may offer more reliability. However, note that the hosted images toolsets (e.g. ubuntu-latest) move forward and if there is a tool breaking issue, actions may react with a patch to a major version to compensate so binding to a specific SHA may prevent you from getting fixes.
> Recommendation: bind to major versions to get functionality and fixes but reserve binding to a specific release or SHA as a mitigation strategy for unforeseen breaks.
@@ -34,7 +34,9 @@ Binding to the immutable sha1 may offer more reliability. However, note that th
1. **Create a GitHub release for each specific version**: Creating a release like [ v1.0.0 ](https://github.com/actions/javascript-action/releases/tag/v1.0.0) allows users to bind back to a specific version if an issue is encountered with the latest major version.
2. **Publish the specific version to the marketplace**: When you release a specific version, choose the option to "Publish this release to the GitHub Marketplace".
2. **Publish the specific version to the marketplace**: When you release a specific version, choose the option to "Publish this Action to the GitHub Marketplace".
<img src="https://user-images.githubusercontent.com/33549821/78670739-36f5ae00-78ac-11ea-9660-57d5687ce520.png" alt="screenshot" height="250"/>
3. **Make the new release available to those binding to the major version tag**: Move the major version tag (v1, v2, etc.) to point to the ref of the current release. This will act as the stable release for that major version. You should keep this tag updated to the most recent stable minor/patch release.
+216
View File
@@ -0,0 +1,216 @@
# ADR 381: `glob` module
**Date**: 2019-12-05
**Status**: Accepted
## Context
This ADR proposes adding a `glob` function to the toolkit.
First party actions should have a consistent glob experience.
Related to artifact upload/download v2.
## Decision
### New module
Create a new module `@actions/glob` that can be versioned at it's own pace - not tied to `@actions/io`.
### Signature
```js
/**
* Constructs a globber from patterns
*
* @param patterns Patterns separated by newlines
* @param options Glob options
*/
export function create(
patterns: string,
options?: GlobOptions
): Promise<Globber> {}
/**
* Used to match files and directories
*/
export interface Globber {
/**
* Returns the search path preceding the first glob segment, from each pattern.
* Duplicates and descendants of other paths are filtered out.
*
* Example 1: The patterns `/foo/*` and `/bar/*` returns `/foo` and `/bar`.
*
* Example 2: The patterns `/foo/*` and `/foo/bar/*` returns `/foo`.
*/
getSearchPaths(): string[]
/**
* Returns files and directories matching the glob patterns.
*
* Order of the results is not guaranteed.
*/
glob(): Promise<string[]>
/**
* Returns files and directories matching the glob patterns.
*
* Order of the results is not guaranteed.
*/
globGenerator(): AsyncGenerator<string, void>
}
/**
* Options to control globbing behavior
*/
export interface GlobOptions {
/**
* Indicates whether to follow symbolic links. Generally should set to false
* when deleting files.
*
* @default true
*/
followSymbolicLinks?: boolean
/**
* Indicates whether directories that match a glob pattern, should implicitly
* cause all descendant paths to be matched.
*
* For example, given the directory `my-dir`, the following glob patterns
* would produce the same results: `my-dir/**`, `my-dir/`, `my-dir`
*
* @default true
*/
implicitDescendants?: boolean
/**
* Indicates whether broken symbolic should be ignored and omitted from the
* result set. Otherwise an error will be thrown.
*
* @default true
*/
omitBrokenSymbolicLinks?: boolean
}
```
### Toolkit usage
Example, do not follow symbolic links:
```js
const patterns = core.getInput('path')
const globber = glob.create(patterns, {followSymbolicLinks: false})
const files = globber.glob()
```
Example, iterator:
```js
const patterns = core.getInput('path')
const globber = glob.create(patterns)
for await (const file of this.globGenerator()) {
console.log(file)
}
```
### Action usage
Actions should follow symbolic links by default.
Users can opt-out.
Example:
```yaml
jobs:
build:
steps:
- uses: actions/upload-artifact@v1
with:
path: |
**/*.tar.gz
**/*.pkg
follow-symbolic-links: false # opt out, should default to true
```
### HashFiles function
Hash files should not follow symbolic links by default.
User can opt-in by specifying flag `--follow-symbolic-links`.
Example:
```yaml
jobs:
build:
steps:
- uses: actions/cache@v1
with:
hash: ${{ hashFiles('--follow-symbolic-links', '**/package-lock.json') }}
```
### Glob behavior
Patterns `*`, `?`, `[...]`, `**` (globstar) are supported.
With the following behaviors:
- File names that begin with `.` may be included in the results
- Case insensitive on Windows
- Directory separator `/` and `\` both supported on Windows
Note:
- Refer [here](https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html#Pattern-Matching) for more information about Bash glob patterns.
- Refer [here](https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html) for more information about Bash glob options.
### Tilde expansion
Support basic tilde expansion, for current user HOME replacement only.
For example, on macOS:
- `~` may expand to `/Users/johndoe`
- `~/foo` may expand to `/Users/johndoe/foo`
Note:
- Refer [here](https://www.gnu.org/software/bash/manual/html_node/Tilde-Expansion.html) for more information about Bash tilde expansion.
- All other forms of tilde expansion are not supported.
- Use `os.homedir()` to resolve the HOME path
### Root and normalize paths
An unrooted pattern will be rooted using the current working directory, prior to searching. Additionally the search path will be normalized prior to searching (relative pathing removed, slashes normalized on Windows, extra slashes removed).
The two side effects are:
1. Rooted and normalized paths are always returned
2. The pattern `**` will include the working directory in the results
These side effects diverge from Bash behavior. Whereas Bash is designed to be a shell, we are designing an API. This decision is intended to improve predictability of the API results.
Note:
- In Bash, the results are not rooted when the pattern is relative.
- In Bash, the results are not normalized. For example, the results from `./*` may look like: `./foo ./bar`
- In Bash, the results from the pattern `**` does not include the working directory. However the results from `/foo/**` would include the directory `/foo`. Also the results from `foo/**` would include the directory `foo`.
## Comments
Patterns that begin with `#` are treated as comments.
## Exclude patterns
Leading `!` changes the meaning of an include pattern to exclude.
Note:
- Multiple leading `!` flips the meaning.
## Escaping
Wrapping special characters in `[]` can be used to escape literal glob characters in a file name. For example the literal file name `hello[a-z]` can be escaped as `hello[[]a-z]`.
On Linux/macOS `\` is also treated as an escape character.
## Consequences
- Publish new module `@actions/glob`
- Publish docs for the module (add link from `./README.md` to new doc `./packages/glob/README.md`)
+19
View File
@@ -0,0 +1,19 @@
# ADRs
ADR, short for "Architecture Decision Record" is a way of capturing important architectural decisions, along with their context and consequences.
This folder includes ADRs for the actions toolkit. ADRs are proposed in the form of a pull request, and they commonly follow this format:
* **Title**: short present tense imperative phrase, less than 50 characters, like a git commit message.
* **Status**: proposed, accepted, rejected, deprecated, superseded, etc.
* **Context**: what is the issue that we're seeing that is motivating this decision or change.
* **Decision**: what is the change that we're actually proposing or doing.
* **Consequences**: what becomes easier or more difficult to do because of this change.
---
- More information about ADRs can be found [here](https://github.com/joelparkerhenderson/architecture_decision_record).
Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

+102 -36
View File
@@ -1,43 +1,12 @@
# :: Commands
The [core toolkit package](https://github.com/actions/toolkit/tree/master/packages/core) offers a number of convenience functions for
The [core toolkit package](https://github.com/actions/toolkit/tree/main/packages/core) offers a number of convenience functions for
setting results, logging, registering secrets and exporting variables across actions. Sometimes, however, its useful to be able to do
these things in a script or other tool.
To allow this, we provide a special `::` syntax which, if logged to `stdout` on a new line, will allow the runner to perform special behavior on
your commands. The following commands are all supported:
### Set an environment variable
To set an environment variable for future out of process steps, use `::set-env`:
```sh
echo "::set-env name=FOO::BAR"
```
Running `$FOO` in a future step will now return `BAR`
This is wrapped by the core exportVariable method which sets for future steps but also updates the variable for this step
```javascript
export function exportVariable(name: string, val: string): void {}
```
### PATH Manipulation
To prepend a string to PATH, use `::addPath`:
```sh
echo "::add-path::BAR"
```
Running `$PATH` in a future step will now return `BAR:{Previous Path}`;
This is wrapped by the core addPath method:
```javascript
export function addPath(inputPath: string): void {}
```
### Set outputs
To set an output for the step, use `::set-output`:
@@ -88,7 +57,7 @@ For example, if you mask the letter `l`, running `echo "Hello FOO BAR World"` wi
### Group and Ungroup Log Lines
Emitting a group with a title will instruct the logs to create a collapsable region up to the next ungroup command.
Emitting a group with a title will instruct the logs to create a collapsible region up to the next endgroup command.
```bash
echo "::group::my title"
@@ -103,6 +72,7 @@ function endGroup(): void {}
```
### Problem Matchers
Problems matchers can be used to scan a build's output to automatically surface lines to the user that matches the provided pattern. A file path to a .json Problem Matcher must be provided. See [Problem Matchers](problem-matchers.md) for more information on how to define a Problem Matcher.
```bash
@@ -112,17 +82,20 @@ echo "::remove-matcher owner=eslint-compact::"
`add-matcher` takes a path to a Problem Matcher file
`remove-matcher` removes a Problem Matcher by owner
### Save State
Save state to be used in the corresponding wrapper (finally) post job entry point.
Save a state to an environmental variable that can later be used in the main or post action.
```bash
echo "::save-state name=FOO::foovalue"
```
Because `save-state` prepends the string `STATE_` to the name, the environment variable `STATE_FOO` will be available to use in the post or main action. See [Sending Values to the pre and post actions](https://help.github.com/en/actions/reference/workflow-commands-for-github-actions#sending-values-to-the-pre-and-post-actions) for more information.
### Log Level
Finally, there are several commands to emit different levels of log output:
There are several commands to emit different levels of log output:
| log level | example usage |
|---|---|
@@ -130,8 +103,101 @@ Finally, there are several commands to emit different levels of log output:
| warning | `echo "::warning::My warning message"` |
| error | `echo "::error::My error message"` |
### Command Prompt
### Command Echoing
By default, the echoing of commands to stdout only occurs if [Step Debugging is enabled](./action-debugging.md#How-to-Access-Step-Debug-Logs)
You can enable or disable this for the current step by using the `echo` command.
```bash
echo "::echo::on"
```
You can also disable echoing.
```bash
echo "::echo::off"
```
This is wrapped by the core method:
```javascript
function setCommandEcho(enabled: boolean): void {}
```
The `add-mask`, `debug`, `warning` and `error` commands do not support echoing.
### Command Prompt
CMD processes the `"` character differently from other shells when echoing. In CMD, the above snippets should have the `"` characters removed in order to correctly process. For example, the set output command would be:
```cmd
echo ::set-output name=FOO::BAR
```
## Environment files
During the execution of a workflow, the runner generates temporary files that can be used to perform certain actions. The path to these files are exposed via environment variables. You will need to use the `utf-8` encoding when writing to these files to ensure proper processing of the commands. Multiple commands can be written to the same file, separated by newlines.
### Set an environment variable
To set an environment variable for future out of process steps, write to the file located at `GITHUB_ENV` or use the equivalent `actions/core` function
```sh
echo "FOO=BAR" >> $GITHUB_ENV
```
Running `$FOO` in a future step will now return `BAR`
For multiline strings, you may use a heredoc style syntax with your choice of delimeter. In the below example, we use `EOF`.
```
steps:
- name: Set the value
id: step_one
run: |
echo 'JSON_RESPONSE<<EOF' >> $GITHUB_ENV
curl https://httpbin.org/json >> $GITHUB_ENV
echo 'EOF' >> $GITHUB_ENV
```
This would set the value of the `JSON_RESPONSE` env variable to the value of the curl response.
The expected syntax for the heredoc style is:
```
{VARIABLE_NAME}<<{DELIMETER}
{VARIABLE_VALUE}
{DELIMETER}
```
This is wrapped by the core `exportVariable` method which sets for future steps but also updates the variable for this step.
```javascript
export function exportVariable(name: string, val: string): void {}
```
### PATH Manipulation
To prepend a string to PATH write to the file located at `GITHUB_PATH` or use the equivalent `actions/core` function
```sh
echo "/Users/test/.nvm/versions/node/v12.18.3/bin" >> $GITHUB_PATH
```
Running `$PATH` in a future step will now return `/Users/test/.nvm/versions/node/v12.18.3/bin:{Previous Path}`;
This is wrapped by the core addPath method:
```javascript
export function addPath(inputPath: string): void {}
```
### Powershell
Powershell does not use UTF8 by default. You will want to make sure you write in the correct encoding. For example, to set the path:
```
steps:
- run: echo "mypath" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
```
+1 -1
View File
@@ -18,7 +18,7 @@ e.g. To use https://github.com/actions/setup-node, users will author:
```yaml
steps:
using: actions/setup-node@master
using: actions/setup-node@v1
```
# Define Metadata
+2 -2
View File
@@ -8,7 +8,7 @@ Note that a complete version of this action can be found at https://github.com/d
## Prerequisites
This walkthrough assumes that you have gone through the basic [javascript action walkthrough](./javascript-action.md) and have a basic action set up. If not, we recommend you go through that first.
This walkthrough assumes that you have gone through the basic [javascript action walkthrough](https://github.com/actions/javascript-action) and have a basic action set up. If not, we recommend you go through that first.
## Installing dependencies
@@ -159,7 +159,7 @@ run();
## Writing unit tests for your action
Next, we're going to write a basic unit test for our action using jest. If you followed the [javascript walkthrough](./javascript-action.md), you should have a file `__tests__/main.test.ts` that runs tests when `npm test` is called. We're going to start by populating that with one test:
Next, we're going to write a basic unit test for our action using jest. If you followed the [javascript walkthrough](https://github.com/actions/javascript-action), you should have a file `__tests__/main.test.ts` that runs tests when `npm test` is called. We're going to start by populating that with one test:
```ts
const nock = require('nock');
+50 -8
View File
@@ -1,13 +1,27 @@
# Problem Matchers
Problem Matchers are a way to scan the output of actions for a specified regex pattern and surface that information prominently in the UI. Both [GitHub Annotations](https://developer.github.com/v3/checks/runs/#annotations-object-1) and log file decorations are created when a match is detected.
## Limitations
Currently, GitHub Actions limit the annotation count in a workflow run.
- 10 warning annotations, 10 error annotations, and 10 notice annotations per step
- 50 annotations per job (sum of annotations from all the steps)
- 50 annotations per run (separate from the job annotations, these annotations arent created by users)
If your workflow may exceed these annotation counts, consider filtering of the log messages which the Problem Matcher is exposed to (e.g. by PR touched files, lines, or other).
## Single Line Matchers
Let's consider the ESLint compact output:
```
badFile.js: line 50, col 11, Error - 'myVar' is defined but never used. (no-unused-vars)
```
We can define a problem matcher in json that detects input in that format:
```json
{
"problemMatcher": [
@@ -33,31 +47,34 @@ The following fields are available for problem matchers:
```
{
owner: An ID field that can be used to remove or replace the problem matcher. **required**
owner: an ID field that can be used to remove or replace the problem matcher. **required**
severity: indicates the default severity, either 'warning' or 'error' case-insensitive. Defaults to 'error'
pattern: [
{
regexp: The regex pattern that provides the groups to match against **required**
regexp: the regex pattern that provides the groups to match against **required**
file: a group number containing the file name
fromPath: a group number containing a filepath used to root the file (e.g. a project file)
line: a group number containing the line number
column: a group number containing the column information
severity: a group number containing either 'warning' or 'error' case-insensitive. Defaults to `error`
code: a group number containing the error code
message: a group number containing the error message. **required** at least one pattern must set the message
loop: loops until a match is not found, only valid on the last pattern of a multipattern matcher
loop: whether to loop until a match is not found, only valid on the last pattern of a multipattern matcher
}
]
}
```
## Multiline Matching
Consider the following output:
```
test.js
1:0 error Missing "use strict" statement strict
5:10 error 'addOne' is defined but never used no-unused-vars
✖ 2 problems (2 errors, 0 warnings)
```
The file name is printed once, yet multiple error lines are printed. The `loop` keyword provides a way to discover multiple errors in outputs.
The eslint-stylish problem matcher defined below catches that output, and creates two annotations from it.
@@ -93,15 +110,40 @@ The eslint-stylish problem matcher defined below catches that output, and create
The first pattern matches the `test.js` line and records the file information. This line is not decorated in the UI.
The second pattern loops through the remaining lines with `loop: true` until it fails to find a match, and surfaces these lines prominently in the UI.
Note that the pattern matches must be on consecutive lines. The following would not result in any match findings.
```
test.js
extraneous log line of no interest
1:0 error Missing "use strict" statement strict
5:10 error 'addOne' is defined but never used no-unused-vars
✖ 2 problems (2 errors, 0 warnings)
```
## Adding and Removing Problem Matchers
Problem Matchers are enabled and removed via the toolkit [commands](commands.md#problem-matchers).
## Duplicate Problem Matchers
Registering two problem-matchers with the same owner will result in only the problem matcher registered last running.
## Examples
Some of the starter actions are already using problem matchers, for example:
- [setup-node](https://github.com/actions/setup-node/tree/master/.github)
- [setup-python](https://github.com/actions/setup-python/tree/master/.github)
- [setup-go](https://github.com/actions/setup-go/tree/master/.github)
- [setup-dotnet](https://github.com/actions/setup-dotnet/tree/master/.github)
- [setup-node](https://github.com/actions/setup-node/tree/main/.github)
- [setup-python](https://github.com/actions/setup-python/tree/main/.github)
- [setup-go](https://github.com/actions/setup-go/tree/main/.github)
- [setup-dotnet](https://github.com/actions/setup-dotnet/tree/main/.github)
## Troubleshooting
### Regular expression not matching
Use ECMAScript regular expression syntax when testing patterns.
### File property getting dropped
[Enable debug logging](https://docs.github.com/en/actions/managing-workflow-runs/enabling-debug-logging) to determine why the file is getting dropped.
This usually happens when the file does not exist or is not under the workflow repo.
+10
View File
@@ -0,0 +1,10 @@
# Proxy Server Support
Self-hosted runners [can be configured](https://help.github.com/en/actions/hosting-your-own-runners/using-a-proxy-server-with-self-hosted-runners) to run behind a proxy server in enterprises.
For actions to **just work** behind a proxy server:
1. Use [tool-cache] version >= 1.3.1
2. Optionally use [actions/http-client](https://github.com/actions/http-client)
If you are using other http clients, refer to the [environment variables set by the runner](https://help.github.com/en/actions/hosting-your-own-runners/using-a-proxy-server-with-self-hosted-runners).
-1
View File
@@ -4,7 +4,6 @@ module.exports = {
roots: ['<rootDir>/packages'],
testEnvironment: 'node',
testMatch: ['**/__tests__/*.test.ts'],
testRunner: 'jest-circus/runner',
transform: {
'^.+\\.ts$': 'ts-jest'
},
+11267 -9299
View File
File diff suppressed because it is too large Load Diff
+17 -15
View File
@@ -2,29 +2,31 @@
"name": "root",
"private": true,
"scripts": {
"audit-all": "lerna run audit-moderate",
"bootstrap": "lerna bootstrap",
"build": "lerna run tsc",
"check-all": "concurrently \"npm:format-check\" \"npm:lint\" \"npm:test\" \"npm:build -- -- --noEmit\"",
"format": "prettier --write packages/**/*.ts",
"format-check": "prettier --check packages/**/*.ts",
"lint": "eslint packages/**/*.ts",
"lint-fix": "eslint packages/**/*.ts --fix",
"new-package": "scripts/create-package",
"test": "jest"
"test": "jest --testTimeout 10000"
},
"devDependencies": {
"@types/jest": "^24.0.11",
"@types/node": "^11.13.5",
"@types/signale": "^1.2.1",
"@typescript-eslint/parser": "^1.9.0",
"concurrently": "^4.1.0",
"eslint": "^5.16.0",
"eslint-plugin-github": "^2.0.0",
"eslint-plugin-jest": "^22.5.1",
"jest": "^24.9.0",
"jest-circus": "^24.7.1",
"lerna": "^3.18.4",
"prettier": "^1.17.0",
"ts-jest": "^24.0.2",
"typescript": "^3.6.2"
"@types/jest": "^27.0.2",
"@types/node": "^12.20.13",
"@types/signale": "^1.4.1",
"@typescript-eslint/parser": "^4.0.0",
"concurrently": "^6.1.0",
"eslint": "^7.23.0",
"eslint-plugin-github": "^4.1.3",
"eslint-plugin-jest": "^22.21.0",
"flow-bin": "^0.115.0",
"jest": "^27.2.5",
"lerna": "^4.0.0",
"prettier": "^1.19.1",
"ts-jest": "^27.0.5",
"typescript": "^3.9.9"
}
}
+30
View File
@@ -0,0 +1,30 @@
# Contributions
This package is used internally by the v2+ versions of [upload-artifact](https://github.com/actions/upload-artifact) and [download-artifact](https://github.com/actions/download-artifact). This package can also be used by other actions to interact with artifacts. Any changes or updates to this package will propagate updates to these actions so it is important that major changes or updates get properly tested.
Any issues or feature requests that are related to the artifact actions should be filled in the appropriate repo.
A limited range of unit tests run as part of each PR when making changes to the artifact packages. For small contributions and fixes, they should be sufficient.
If making large changes, there are a few scenarios that should be tested.
- Uploading very large artifacts (large artifacts get compressed using gzip so compression/decompression must be tested)
- Uploading artifacts with lots of small files (each file is uploaded with its own HTTP call, timeouts and non-success HTTP responses can be expected so they must be properly handled)
- Uploading artifacts using a self-hosted runner (uploads and downloads behave differently due to extra latency)
- Downloading a single artifact (large and small, if lots of small files are part of an artifact, timeouts and non-success HTTP responses can be expected)
- Downloading all artifacts at once
Large architectural changes can impact upload/download performance so it is important to separately run extra tests. We request that any large contributions/changes have extra detailed testing so we can verify performance and possible regressions.
It is not possible to run end-to-end tests for artifacts as part of a PR in this repo because certain env variables such as `ACTIONS_RUNTIME_URL` are only available from the context of an action as opposed to a shell script. These env variables are needed in order to make the necessary API calls.
# Testing
Any easy way to test changes is to fork the artifact actions and to use `npm link` to test your changes.
1. Fork the [upload-artifact](https://github.com/actions/upload-artifact) and [download-artifact](https://github.com/actions/download-artifact) repos
2. Clone the forks locally
3. With your local changes to the toolkit repo, type `npm link` after ensuring there are no errors when running `tsc`
4. In the locally cloned fork, type `npm link @actions/artifact`
4. Create a new release for your local fork using `tsc` and `npm run release` (this will create a new `dist/index.js` file using `@vercel/ncc`)
5. Commit and push your local changes, you will then be able to test your changes with your forked action
+9
View File
@@ -0,0 +1,9 @@
The MIT License (MIT)
Copyright 2019 GitHub
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+213
View File
@@ -0,0 +1,213 @@
# `@actions/artifact`
## Usage
You can use this package to interact with the actions artifacts.
- [Upload an Artifact](#Upload-an-Artifact)
- [Download a Single Artifact](#Download-a-Single-Artifact)
- [Download All Artifacts](#Download-all-Artifacts)
- [Additional Documentation](#Additional-Documentation)
- [Contributions](#Contributions)
Relative paths and absolute paths are both allowed. Relative paths are rooted against the current working directory.
## Upload an Artifact
Method Name: `uploadArtifact`
#### Inputs
- `name`
- The name of the artifact that is being uploaded
- Required
- `files`
- A list of file paths that describe what should be uploaded as part of the artifact
- If a path is provided that does not exist, an error will be thrown
- Can be absolute or relative. Internally everything is normalized and resolved
- Required
- `rootDirectory`
- A file path that denotes the root directory of the files being uploaded. This path is used to strip the paths provided in `files` to control how they are uploaded and structured
- If a file specified in `files` is not in the `rootDirectory`, an error will be thrown
- Required
- `options`
- Extra options that allow for the customization of the upload behavior
- Optional
#### Available Options
- `continueOnError`
- Indicates if the artifact upload should continue in the event a file fails to upload. If there is a error during upload, a partial artifact will always be created and available for download at the end. The `size` reported will be the amount of storage that the user or org will be charged for the partial artifact.
- If set to `false`, and an error is encountered, all other uploads will stop and any files that were queued will not be attempted to be uploaded. The partial artifact available will only include files up until the failure.
- If set to `true` and an error is encountered, the failed file will be skipped and ignored and all other queued files will be attempted to be uploaded. There will be an artifact available for download at the end with everything excluding the file that failed to upload
- Optional, defaults to `true` if not specified
- `retentionDays`
- Duration after which artifact will expire in days
- Minimum value: 1
- Maximum value: 90 unless changed by repository setting
- If this is set to a greater value than the retention settings allowed, the retention on artifacts will be reduced to match the max value allowed on the server, and the upload process will continue. An input of 0 assumes default retention value.
#### Example using Absolute File Paths
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'/home/user/files/plz-upload/file1.txt',
'/home/user/files/plz-upload/file2.txt',
'/home/user/files/plz-upload/dir/file3.txt'
]
const rootDirectory = '/home/user/files/plz-upload'
const options = {
continueOnError: true
}
const uploadResult = await artifactClient.uploadArtifact(artifactName, files, rootDirectory, options)
```
#### Example using Relative File Paths
```js
// Assuming the current working directory is /home/user/files/plz-upload
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'file1.txt',
'file2.txt',
'dir/file3.txt'
]
const rootDirectory = '.' // Also possible to use __dirname
const options = {
continueOnError: false
}
const uploadResponse = await artifactClient.uploadArtifact(artifactName, files, rootDirectory, options)
```
#### Upload Result
The returned `UploadResponse` will contain the following information
- `artifactName`
- The name of the artifact that was uploaded
- `artifactItems`
- A list of all files that describe what is uploaded if there are no errors encountered. Usually this will be equal to the inputted `files` with the exception of empty directories (will not be uploaded)
- `size`
- Total size of the artifact that was uploaded in bytes
- `failedItems`
- A list of items that were not uploaded successfully (this will include queued items that were not uploaded if `continueOnError` is set to false). This is a subset of `artifactItems`
## Download a Single Artifact
Method Name: `downloadArtifact`
#### Inputs
- `name`
- The name of the artifact to download
- Required
- `path`
- Path that denotes where the artifact will be downloaded to
- Optional. Defaults to the GitHub workspace directory(`$GITHUB_WORKSPACE`) if not specified
- `options`
- Extra options that allow for the customization of the download behavior
- Optional
#### Available Options
- `createArtifactFolder`
- Specifies if a folder (the artifact name) is created for the artifact that is downloaded (contents downloaded into this folder),
- Optional. Defaults to false if not specified
#### Example
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const path = 'some/directory'
const options = {
createArtifactFolder: false
}
const downloadResponse = await artifactClient.downloadArtifact(artifactName, path, options)
// Post download, the directory structure will look like this
/some
/directory
/file1.txt
/file2.txt
/dir
/file3.txt
// If createArtifactFolder is set to true, the directory structure will look like this
/some
/directory
/my-artifact
/file1.txt
/file2.txt
/dir
/file3.txt
```
#### Download Response
The returned `DownloadResponse` will contain the following information
- `artifactName`
- The name of the artifact that was downloaded
- `downloadPath`
- The full Path to where the artifact was downloaded
## Download All Artifacts
Method Name: `downloadAllArtifacts`
#### Inputs
- `path`
- Path that denotes where the artifact will be downloaded to
- Optional. Defaults to the GitHub workspace directory(`$GITHUB_WORKSPACE`) if not specified
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create();
const downloadResponse = await artifactClient.downloadAllArtifacts();
// output result
for (response in downloadResponse) {
console.log(response.artifactName);
console.log(response.downloadPath);
}
```
Because there are multiple artifacts, an extra directory (denoted by the name of the artifact) will be created for each artifact in the path. With 2 artifacts(`my-artifact-1` and `my-artifact-2` for example) and the default path, the directory structure will be as follows:
```js
/GITHUB_WORKSPACE
/my-artifact-1
/ .. contents of `my-artifact-1`
/my-artifact-2
/ .. contents of `my-artifact-2`
```
#### Download Result
An array will be returned that describes the results for downloading all artifacts. The number of items in the array indicates the number of artifacts that were downloaded.
Each artifact will have the same `DownloadResponse` as if it was individually downloaded
- `artifactName`
- The name of the artifact that was downloaded
- `downloadPath`
- The full Path to where the artifact was downloaded
## Additional Documentation
Check out [additional-information](docs/additional-information.md) for extra documentation around usage, restrictions and behavior.
Check out [implementation-details](docs/implementation-details.md) for extra information about the implementation of this package.
## Contributions
See [contributor guidelines](https://github.com/actions/toolkit/blob/main/.github/CONTRIBUTING.md) for general guidelines and information about toolkit contributions.
For contributions related to this package, see [artifact contributions](CONTRIBUTIONS.md) for more information.
+80
View File
@@ -0,0 +1,80 @@
# @actions/artifact Releases
### 0.1.0
- Initial release
### 0.2.0
- Fixes to TCP connections not closing
- GZip file compression to speed up downloads
- Improved logging and output
- Extra documentation
### 0.3.0
- Fixes to gzip decompression when downloading artifacts
- Support handling 429 response codes
- Improved download experience when dealing with empty files
- Exponential backoff when retryable status codes are encountered
- Clearer error message if storage quota has been reached
- Improved logging and output during artifact download
### 0.3.1
- Fix to ensure temporary gzip files get correctly deleted during artifact upload
- Remove spaces as a forbidden character during upload
### 0.3.2
- Fix to ensure readstreams get correctly reset in the event of a retry
### 0.3.3
- Increase chunk size during upload from 4MB to 8MB
- Improve user-agent strings during API calls to help internally diagnose issues
### 0.3.5
- Retry in the event of a 413 response
### 0.4.0
- Add option to specify custom retentions on artifacts
### 0.4.1
- Update to latest @actions/core version
### 0.4.2
- Improved retry-ability when a partial artifact download is encountered
### 0.5.0
- Improved retry-ability for all http calls during artifact upload and download if an error is encountered
### 0.5.1
- Bump @actions/http-client to version 1.0.11 to fix proxy related issues during artifact upload and download
### 0.5.2
- Add HTTP 500 as a retryable status code for artifact upload and download.
### 0.6.0
- Support upload from named pipes [#748](https://github.com/actions/toolkit/pull/748)
- Fixes to percentage values being greater than 100% when downloading all artifacts [#889](https://github.com/actions/toolkit/pull/889)
- Improved logging and output during artifact upload [#949](https://github.com/actions/toolkit/pull/949)
- Improvements to client-side validation for certain invalid characters not allowed during upload: [#951](https://github.com/actions/toolkit/pull/951)
- Faster upload speeds for certain types of large files by exempting gzip compression [#956](https://github.com/actions/toolkit/pull/956)
- More detailed logging when dealing with chunked uploads [#957](https://github.com/actions/toolkit/pull/957)
### 0.6.1
- Fix for failing 0 byte file uploads on Windows [#962](https://github.com/actions/toolkit/pull/962)
### 1.0.0
- Update `lockfileVersion` to `v2` in `package-lock.json [#1009](https://github.com/actions/toolkit/pull/1009)
@@ -0,0 +1,5 @@
name: 'Set env variables'
description: 'Sets certain env variables so that e2e artifact upload and download can be tested in a shell'
runs:
using: 'node12'
main: 'index.js'
@@ -0,0 +1,14 @@
// Certain env variables are not set by default in a shell context and are only available in a node context from a running action
// In order to be able to upload and download artifacts e2e in a shell when running CI tests, we need these env variables set
const fs = require('fs');
const os = require('os');
const filePath = process.env[`GITHUB_ENV`]
fs.appendFileSync(filePath, `ACTIONS_RUNTIME_URL=${process.env.ACTIONS_RUNTIME_URL}${os.EOL}`, {
encoding: 'utf8'
})
fs.appendFileSync(filePath, `ACTIONS_RUNTIME_TOKEN=${process.env.ACTIONS_RUNTIME_TOKEN}${os.EOL}`, {
encoding: 'utf8'
})
fs.appendFileSync(filePath, `GITHUB_RUN_ID=${process.env.GITHUB_RUN_ID}${os.EOL}`, {
encoding: 'utf8'
})
@@ -0,0 +1,552 @@
import * as path from 'path'
import * as core from '@actions/core'
import {URL} from 'url'
import {getDownloadSpecification} from '../src/internal/download-specification'
import {ContainerEntry} from '../src/internal/contracts'
const artifact1Name = 'my-artifact'
const artifact2Name = 'my-artifact-extra'
// Populating with only the information that is necessary
function getPartialContainerEntry(): ContainerEntry {
return {
containerId: 10,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'ADD_INFORMATION',
itemType: 'ADD_INFORMATION',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: 'ADD_INFORMATION',
contentLocation: 'ADD_INFORMATION',
contentId: '',
fileLength: 100
}
}
function createFileEntry(entryPath: string): ContainerEntry {
const newFileEntry = getPartialContainerEntry()
newFileEntry.path = entryPath
newFileEntry.itemType = 'file'
newFileEntry.itemLocation = createItemLocation(entryPath)
newFileEntry.contentLocation = createContentLocation(entryPath)
return newFileEntry
}
function createDirectoryEntry(directoryPath: string): ContainerEntry {
const newDirectoryEntry = getPartialContainerEntry()
newDirectoryEntry.path = directoryPath
newDirectoryEntry.itemType = 'folder'
newDirectoryEntry.itemLocation = createItemLocation(directoryPath)
newDirectoryEntry.contentLocation = createContentLocation(directoryPath)
return newDirectoryEntry
}
function createItemLocation(relativePath: string): string {
const itemLocation = new URL(
'https://testing/_apis/resources/Containers/10000'
)
itemLocation.searchParams.append('itemPath', relativePath)
itemLocation.searchParams.append('metadata', 'true')
return itemLocation.toString()
}
function createContentLocation(relativePath: string): string {
const itemLocation = new URL(
'https://testing/_apis/resources/Containers/10000'
)
itemLocation.searchParams.append('itemPath', relativePath)
return itemLocation.toString()
}
/*
Represents a set of container entries for two artifacts with the following directory structure
/my-artifact
/file1.txt
/file2.txt
/dir1
/file3.txt
/dir2
/dir3
/dir4
file4.txt
file5.txt (no length property)
file6.txt (empty file)
/my-artifact-extra
/file1.txt
*/
// main artifact
const file1Path = path.join(artifact1Name, 'file1.txt')
const file2Path = path.join(artifact1Name, 'file2.txt')
const dir1Path = path.join(artifact1Name, 'dir1')
const file3Path = path.join(dir1Path, 'file3.txt')
const dir2Path = path.join(dir1Path, 'dir2')
const dir3Path = path.join(dir2Path, 'dir3')
const dir4Path = path.join(dir3Path, 'dir4')
const file4Path = path.join(dir4Path, 'file4.txt')
const file5Path = path.join(dir4Path, 'file5.txt')
const file6Path = path.join(dir4Path, 'file6.txt')
const rootDirectoryEntry = createDirectoryEntry(artifact1Name)
const directoryEntry1 = createDirectoryEntry(dir1Path)
const directoryEntry2 = createDirectoryEntry(dir2Path)
const directoryEntry3 = createDirectoryEntry(dir3Path)
const directoryEntry4 = createDirectoryEntry(dir4Path)
const fileEntry1 = createFileEntry(file1Path)
const fileEntry2 = createFileEntry(file2Path)
const fileEntry3 = createFileEntry(file3Path)
const fileEntry4 = createFileEntry(file4Path)
const missingLengthFileEntry = createFileEntry(file5Path)
missingLengthFileEntry.fileLength = undefined // one file does not have a fileLength
const emptyLengthFileEntry = createFileEntry(file6Path)
emptyLengthFileEntry.fileLength = 0 // empty file path
// extra artifact
const artifact2File1Path = path.join(artifact2Name, 'file1.txt')
const rootDirectoryEntry2 = createDirectoryEntry(artifact2Name)
const extraFileEntry = createFileEntry(artifact2File1Path)
const artifactContainerEntries: ContainerEntry[] = [
rootDirectoryEntry,
fileEntry1,
fileEntry2,
directoryEntry1,
fileEntry3,
directoryEntry2,
directoryEntry3,
directoryEntry4,
fileEntry4,
missingLengthFileEntry,
emptyLengthFileEntry,
rootDirectoryEntry2,
extraFileEntry
]
describe('Search', () => {
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
})
it('Download Specification - Absolute Path with no root directory', () => {
const testDownloadPath = path.join(
__dirname,
'some',
'destination',
'folder'
)
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
false
)
expect(specification.rootDownloadLocation).toEqual(testDownloadPath)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(testDownloadPath, 'file1.txt')
const item2ExpectedTargetPath = path.join(testDownloadPath, 'file2.txt')
const item3ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const item6ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file6.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(testDownloadPath)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1')
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1', 'dir2', 'dir3', 'dir4')
)
expect(specification.emptyFilesToCreate.length).toEqual(1)
expect(specification.emptyFilesToCreate).toContain(item6ExpectedTargetPath)
})
it('Download Specification - Relative Path with no root directory', () => {
const testDownloadPath = path.join('some', 'destination', 'folder')
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
false
)
expect(specification.rootDownloadLocation).toEqual(testDownloadPath)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(testDownloadPath, 'file1.txt')
const item2ExpectedTargetPath = path.join(testDownloadPath, 'file2.txt')
const item3ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const item6ExpectedTargetPath = path.join(
testDownloadPath,
'dir1',
'dir2',
'dir3',
'dir4',
'file6.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(testDownloadPath)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1')
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, 'dir1', 'dir2', 'dir3', 'dir4')
)
expect(specification.emptyFilesToCreate.length).toEqual(1)
expect(specification.emptyFilesToCreate).toContain(item6ExpectedTargetPath)
})
it('Download Specification - Absolute Path with root directory', () => {
const testDownloadPath = path.join(
__dirname,
'some',
'destination',
'folder'
)
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
true
)
expect(specification.rootDownloadLocation).toEqual(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file1.txt'
)
const item2ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file2.txt'
)
const item3ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const item6ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file6.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir1Path)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir4Path)
)
expect(specification.emptyFilesToCreate.length).toEqual(1)
expect(specification.emptyFilesToCreate).toContain(item6ExpectedTargetPath)
})
it('Download Specification - Relative Path with root directory', () => {
const testDownloadPath = path.join('some', 'destination', 'folder')
const specification = getDownloadSpecification(
artifact1Name,
artifactContainerEntries,
testDownloadPath,
true
)
expect(specification.rootDownloadLocation).toEqual(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.filesToDownload.length).toEqual(5)
const item1ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file1.txt'
)
const item2ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'file2.txt'
)
const item3ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'file3.txt'
)
const item4ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file4.txt'
)
const item5ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file5.txt'
)
const item6ExpectedTargetPath = path.join(
testDownloadPath,
artifact1Name,
'dir1',
'dir2',
'dir3',
'dir4',
'file6.txt'
)
const targetLocations = specification.filesToDownload.map(
item => item.targetPath
)
expect(targetLocations).toContain(item1ExpectedTargetPath)
expect(targetLocations).toContain(item2ExpectedTargetPath)
expect(targetLocations).toContain(item3ExpectedTargetPath)
expect(targetLocations).toContain(item4ExpectedTargetPath)
expect(targetLocations).toContain(item5ExpectedTargetPath)
for (const downloadItem of specification.filesToDownload) {
if (downloadItem.targetPath === item1ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file1Path)
)
} else if (downloadItem.targetPath === item2ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file2Path)
)
} else if (downloadItem.targetPath === item3ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file3Path)
)
} else if (downloadItem.targetPath === item4ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file4Path)
)
} else if (downloadItem.targetPath === item5ExpectedTargetPath) {
expect(downloadItem.sourceLocation).toEqual(
createContentLocation(file5Path)
)
} else {
throw new Error('this should never be reached')
}
}
expect(specification.directoryStructure.length).toEqual(3)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, artifact1Name)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir1Path)
)
expect(specification.directoryStructure).toContain(
path.join(testDownloadPath, dir4Path)
)
expect(specification.emptyFilesToCreate.length).toEqual(1)
expect(specification.emptyFilesToCreate).toContain(item6ExpectedTargetPath)
})
})
@@ -0,0 +1,490 @@
import * as core from '@actions/core'
import * as http from 'http'
import * as io from '../../io/src/io'
import * as net from 'net'
import * as path from 'path'
import * as configVariables from '../src/internal/config-variables'
import {promises as fs} from 'fs'
import {DownloadItem} from '../src/internal/download-specification'
import {HttpClient, HttpClientResponse} from '@actions/http-client'
import {DownloadHttpClient} from '../src/internal/download-http-client'
import {
ListArtifactsResponse,
QueryArtifactResponse
} from '../src/internal/contracts'
import * as stream from 'stream'
import {gzip} from 'zlib'
import {promisify} from 'util'
const root = path.join(__dirname, '_temp', 'artifact-download-tests')
const defaultEncoding = 'utf8'
jest.mock('../src/internal/config-variables')
jest.mock('@actions/http-client')
describe('Download Tests', () => {
beforeAll(async () => {
await io.rmRF(root)
await fs.mkdir(path.join(root), {
recursive: true
})
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
})
/**
* Test Listing Artifacts
*/
it('List Artifacts - Success', async () => {
setupSuccessfulListArtifactsResponse()
const downloadHttpClient = new DownloadHttpClient()
const artifacts = await downloadHttpClient.listArtifacts()
expect(artifacts.count).toEqual(2)
const artifactNames = artifacts.value.map(item => item.name)
expect(artifactNames).toContain('artifact1-name')
expect(artifactNames).toContain('artifact2-name')
for (const artifact of artifacts.value) {
if (artifact.name === 'artifact1-name') {
expect(artifact.url).toEqual(
`${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact1-name`
)
} else if (artifact.name === 'artifact2-name') {
expect(artifact.url).toEqual(
`${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact2-name`
)
} else {
throw new Error(
'Invalid artifact combination. This should never be reached'
)
}
}
})
it('List Artifacts - Failure', async () => {
setupFailedResponse()
const downloadHttpClient = new DownloadHttpClient()
expect(downloadHttpClient.listArtifacts()).rejects.toThrow(
'List Artifacts failed: Artifact service responded with 400'
)
})
/**
* Test Container Items
*/
it('Container Items - Success', async () => {
setupSuccessfulContainerItemsResponse()
const downloadHttpClient = new DownloadHttpClient()
const response = await downloadHttpClient.getContainerItems(
'artifact-name',
configVariables.getRuntimeUrl()
)
expect(response.count).toEqual(2)
const itemPaths = response.value.map(item => item.path)
expect(itemPaths).toContain('artifact-name')
expect(itemPaths).toContain('artifact-name/file1.txt')
for (const containerEntry of response.value) {
if (containerEntry.path === 'artifact-name') {
expect(containerEntry.itemType).toEqual('folder')
} else if (containerEntry.path === 'artifact-name/file1.txt') {
expect(containerEntry.itemType).toEqual('file')
} else {
throw new Error(
'Invalid container combination. This should never be reached'
)
}
}
})
it('Container Items - Failure', async () => {
setupFailedResponse()
const downloadHttpClient = new DownloadHttpClient()
expect(
downloadHttpClient.getContainerItems(
'artifact-name',
configVariables.getRuntimeUrl()
)
).rejects.toThrow(
`Get Container Items failed: Artifact service responded with 400`
)
})
it('Test downloading an individual artifact with gzip', async () => {
const fileContents = Buffer.from(
'gzip worked on the first try\n',
defaultEncoding
)
const targetPath = path.join(root, 'FileA.txt')
setupDownloadItemResponse(fileContents, true, 200, false, false)
const downloadHttpClient = new DownloadHttpClient()
const items: DownloadItem[] = []
items.push({
sourceLocation: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13?itemPath=my-artifact%2FFileA.txt`,
targetPath
})
await expect(
downloadHttpClient.downloadSingleArtifact(items)
).resolves.not.toThrow()
await checkDestinationFile(targetPath, fileContents)
})
it('Test downloading an individual artifact without gzip', async () => {
const fileContents = Buffer.from(
'plaintext worked on the first try\n',
defaultEncoding
)
const targetPath = path.join(root, 'FileB.txt')
setupDownloadItemResponse(fileContents, false, 200, false, false)
const downloadHttpClient = new DownloadHttpClient()
const items: DownloadItem[] = []
items.push({
sourceLocation: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13?itemPath=my-artifact%2FFileB.txt`,
targetPath
})
await expect(
downloadHttpClient.downloadSingleArtifact(items)
).resolves.not.toThrow()
await checkDestinationFile(targetPath, fileContents)
})
it('Test retryable status codes during artifact download', async () => {
// The first http response should return a retryable status call while the subsequent call should return a 200 so
// the download should successfully finish
const retryableStatusCodes = [429, 500, 502, 503, 504]
for (const statusCode of retryableStatusCodes) {
const fileContents = Buffer.from('try, try again\n', defaultEncoding)
const targetPath = path.join(root, `FileC-${statusCode}.txt`)
setupDownloadItemResponse(fileContents, false, statusCode, false, true)
const downloadHttpClient = new DownloadHttpClient()
const items: DownloadItem[] = []
items.push({
sourceLocation: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13?itemPath=my-artifact%2FFileC.txt`,
targetPath
})
await expect(
downloadHttpClient.downloadSingleArtifact(items)
).resolves.not.toThrow()
await checkDestinationFile(targetPath, fileContents)
}
})
it('Test retry on truncated response with gzip', async () => {
const fileContents = Buffer.from(
'Sometimes gunzip fails on the first try\n',
defaultEncoding
)
const targetPath = path.join(root, 'FileD.txt')
setupDownloadItemResponse(fileContents, true, 200, true, true)
const downloadHttpClient = new DownloadHttpClient()
const items: DownloadItem[] = []
items.push({
sourceLocation: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13?itemPath=my-artifact%2FFileD.txt`,
targetPath
})
await expect(
downloadHttpClient.downloadSingleArtifact(items)
).resolves.not.toThrow()
await checkDestinationFile(targetPath, fileContents)
})
it('Test retry on truncated response without gzip', async () => {
const fileContents = Buffer.from(
'You have to inspect the content-length header to know if you got everything\n',
defaultEncoding
)
const targetPath = path.join(root, 'FileE.txt')
setupDownloadItemResponse(fileContents, false, 200, true, true)
const downloadHttpClient = new DownloadHttpClient()
const items: DownloadItem[] = []
items.push({
sourceLocation: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13?itemPath=my-artifact%2FFileD.txt`,
targetPath
})
await expect(
downloadHttpClient.downloadSingleArtifact(items)
).resolves.not.toThrow()
await checkDestinationFile(targetPath, fileContents)
})
/**
* Helper used to setup mocking for the HttpClient
*/
async function emptyMockReadBody(): Promise<string> {
return new Promise(resolve => {
resolve()
})
}
/**
* Setups up HTTP GET response for a successful listArtifacts() call
*/
function setupSuccessfulListArtifactsResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
let mockReadBody = emptyMockReadBody
mockMessage.statusCode = 201
const response: ListArtifactsResponse = {
count: 2,
value: [
{
containerId: '13',
size: -1,
signedContent: 'false',
fileContainerResourceUrl: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13`,
type: 'actions_storage',
name: 'artifact1-name',
url: `${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact1-name`
},
{
containerId: '13',
size: -1,
signedContent: 'false',
fileContainerResourceUrl: `${configVariables.getRuntimeUrl()}_apis/resources/Containers/13`,
type: 'actions_storage',
name: 'artifact2-name',
url: `${configVariables.getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=artifact2-name`
}
]
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
}
/**
* Setups up HTTP GET response for downloading items
* @param isGzip is the downloaded item gzip encoded
* @param firstHttpResponseCode the http response code that should be returned
*/
function setupDownloadItemResponse(
fileContents: Buffer,
isGzip: boolean,
firstHttpResponseCode: number,
truncateFirstResponse: boolean,
retryExpected: boolean
): void {
const spyInstance = jest
.spyOn(HttpClient.prototype, 'get')
.mockImplementationOnce(async () => {
if (firstHttpResponseCode === 200) {
const fullResponse = await constructResponse(isGzip, fileContents)
const actualResponse = truncateFirstResponse
? fullResponse.subarray(0, 3)
: fullResponse
return {
message: getDownloadResponseMessage(
firstHttpResponseCode,
isGzip,
fullResponse.length,
actualResponse
),
readBody: emptyMockReadBody
}
} else {
return {
message: getDownloadResponseMessage(
firstHttpResponseCode,
false,
0,
null
),
readBody: emptyMockReadBody
}
}
})
// set up a second mock only if we expect a retry. Otherwise this mock will affect other tests.
if (retryExpected) {
spyInstance.mockImplementationOnce(async () => {
// chained response, if the HTTP GET function gets called again, return a successful response
const fullResponse = await constructResponse(isGzip, fileContents)
return {
message: getDownloadResponseMessage(
200,
isGzip,
fullResponse.length,
fullResponse
),
readBody: emptyMockReadBody
}
})
}
}
async function constructResponse(
isGzip: boolean,
plaintext: Buffer | string
): Promise<Buffer> {
if (isGzip) {
return await promisify(gzip)(plaintext)
} else if (typeof plaintext === 'string') {
return Buffer.from(plaintext, defaultEncoding)
} else {
return plaintext
}
}
function getDownloadResponseMessage(
httpResponseCode: number,
isGzip: boolean,
contentLength: number,
response: Buffer | null
): http.IncomingMessage {
let readCallCount = 0
const mockMessage = <http.IncomingMessage>new stream.Readable({
read(size) {
switch (readCallCount++) {
case 0:
if (!!response && response.byteLength > size) {
throw new Error(
`test response larger than requested size (${size})`
)
}
this.push(response)
break
default:
// end the stream
this.push(null)
break
}
}
})
mockMessage.statusCode = httpResponseCode
mockMessage.headers = {
'content-length': contentLength.toString()
}
if (isGzip) {
mockMessage.headers['content-encoding'] = 'gzip'
}
return mockMessage
}
/**
* Setups up HTTP GET response when querying for container items
*/
function setupSuccessfulContainerItemsResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
let mockReadBody = emptyMockReadBody
mockMessage.statusCode = 201
const response: QueryArtifactResponse = {
count: 2,
value: [
{
containerId: 10000,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'artifact-name',
itemType: 'folder',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name&metadata=True`,
contentLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name`,
contentId: ''
},
{
containerId: 10000,
scopeIdentifier: '00000000-0000-0000-0000-000000000000',
path: 'artifact-name/file1.txt',
itemType: 'file',
status: 'created',
dateCreated: '2020-02-06T22:13:35.373Z',
dateLastModified: '2020-02-06T22:13:35.453Z',
createdBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
lastModifiedBy: '82f0bf89-6e55-4e5a-b8b6-f75eb992578c',
itemLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name%2Ffile1.txt&metadata=True`,
contentLocation: `${configVariables.getRuntimeUrl()}/_apis/resources/Containers/10000?itemPath=artifact-name%2Ffile1.txt`,
contentId: ''
}
]
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
}
/**
* Setups up HTTP GET response for a generic failed request
*/
function setupFailedResponse(): void {
jest.spyOn(HttpClient.prototype, 'get').mockImplementationOnce(async () => {
const mockMessage = new http.IncomingMessage(new net.Socket())
mockMessage.statusCode = 400
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: emptyMockReadBody
})
})
})
}
async function checkDestinationFile(
targetPath: string,
expectedContents: Buffer
): Promise<void> {
const fileContents = await fs.readFile(targetPath)
expect(fileContents.byteLength).toEqual(expectedContents.byteLength)
expect(fileContents.equals(expectedContents)).toBeTruthy()
}
})
@@ -0,0 +1,78 @@
import {
checkArtifactName,
checkArtifactFilePath
} from '../src/internal/path-and-artifact-name-validation'
import * as core from '@actions/core'
describe('Path and artifact name validation', () => {
beforeAll(() => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
})
it('Check Artifact Name for any invalid characters', () => {
const invalidNames = [
'my\\artifact',
'my/artifact',
'my"artifact',
'my:artifact',
'my<artifact',
'my>artifact',
'my|artifact',
'my*artifact',
'my?artifact',
''
]
for (const invalidName of invalidNames) {
expect(() => {
checkArtifactName(invalidName)
}).toThrow()
}
const validNames = [
'my-normal-artifact',
'myNormalArtifact',
'm¥ñðrmålÄr†ï£å¢†'
]
for (const validName of validNames) {
expect(() => {
checkArtifactName(validName)
}).not.toThrow()
}
})
it('Check Artifact File Path for any invalid characters', () => {
const invalidNames = [
'some/invalid"artifact/path',
'some/invalid:artifact/path',
'some/invalid<artifact/path',
'some/invalid>artifact/path',
'some/invalid|artifact/path',
'some/invalid*artifact/path',
'some/invalid?artifact/path',
'some/invalid\rartifact/path',
'some/invalid\nartifact/path',
'some/invalid\r\nartifact/path',
''
]
for (const invalidName of invalidNames) {
expect(() => {
checkArtifactFilePath(invalidName)
}).toThrow()
}
const validNames = [
'my/perfectly-normal/artifact-path',
'my/perfectly\\Normal/Artifact-path',
'm¥/ñðrmål/Är†ï£å¢†'
]
for (const validName of validNames) {
expect(() => {
checkArtifactFilePath(validName)
}).not.toThrow()
}
})
})
+114
View File
@@ -0,0 +1,114 @@
import * as http from 'http'
import * as net from 'net'
import * as core from '@actions/core'
import * as configVariables from '../src/internal/config-variables'
import {retry} from '../src/internal/requestUtils'
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {HttpClientResponse} from '@actions/http-client'
jest.mock('../src/internal/config-variables')
interface ITestResult {
responseCode: number
errorMessage: string | null
}
async function testRetry(
responseCodes: number[],
expectedResult: ITestResult
): Promise<void> {
const reverse = responseCodes.reverse() // Reverse responses since we pop from end
if (expectedResult.errorMessage) {
// we expect some exception to be thrown
expect(
retry(
'test',
async () => handleResponse(reverse.pop()),
new Map(), // extra error message for any particular http codes
configVariables.getRetryLimit()
)
).rejects.toThrow(expectedResult.errorMessage)
} else {
// we expect a correct status code to be returned
const actualResult = await retry(
'test',
async () => handleResponse(reverse.pop()),
new Map(), // extra error message for any particular http codes
configVariables.getRetryLimit()
)
expect(actualResult.message.statusCode).toEqual(expectedResult.responseCode)
}
}
async function handleResponse(
testResponseCode: number | undefined
): Promise<IHttpClientResponse> {
if (!testResponseCode) {
throw new Error(
'Test incorrectly set up. reverse.pop() was called too many times so not enough test response codes were supplied'
)
}
return setupSingleMockResponse(testResponseCode)
}
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
})
/**
* Helpers used to setup mocking for the HttpClient
*/
async function emptyMockReadBody(): Promise<string> {
return new Promise(resolve => {
resolve()
})
}
async function setupSingleMockResponse(
statusCode: number
): Promise<IHttpClientResponse> {
const mockMessage = new http.IncomingMessage(new net.Socket())
const mockReadBody = emptyMockReadBody
mockMessage.statusCode = statusCode
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
}
test('retry works on successful response', async () => {
await testRetry([200], {
responseCode: 200,
errorMessage: null
})
})
test('retry works after retryable status code', async () => {
await testRetry([503, 200], {
responseCode: 200,
errorMessage: null
})
})
test('retry fails after exhausting retries', async () => {
// __mocks__/config-variables caps the max retry count in tests to 2
await testRetry([503, 503, 200], {
responseCode: 200,
errorMessage: 'test failed: Artifact service responded with 503'
})
})
test('retry fails after non-retryable status code', async () => {
await testRetry([400, 200], {
responseCode: 400,
errorMessage: 'test failed: Artifact service responded with 400'
})
})
+27
View File
@@ -0,0 +1,27 @@
#!/bin/bash
path="$1"
expectedContent="$2"
if [ "$path" == "" ]; then
echo "File path not provided"
exit 1
fi
if [ "$expectedContent" == "" ]; then
echo "Expected file contents not provided"
exit 1
fi
if [ ! -f "$path" ]; then
echo "Expected file $path does not exist"
exit 1
fi
actualContent=$(cat "$path")
if [ "$expectedContent" == "_EMPTY_" ] && [ ! -s "$path" ]; then
exit 0
elif [ "$actualContent" != "$expectedContent" ]; then
echo "File contents are not correct, expected $expectedContent, received $actualContent"
exit 1
fi
@@ -0,0 +1,79 @@
import * as core from '@actions/core'
import * as tmp from 'tmp-promise'
import * as path from 'path'
import * as io from '../../io/src/io'
import {promises as fs} from 'fs'
import {createGZipFileOnDisk} from '../src/internal/upload-gzip'
const root = path.join(__dirname, '_temp', 'upload-gzip')
const tempGzipFilePath = path.join(root, 'file1.gzip')
const tempZipFilePath = path.join(root, 'file2.zip')
const tempTarlzFilePath = path.join(root, 'file3.tar.lz')
const tempGzFilePath = path.join(root, 'file4.tar.gz')
const tempBz2FilePath = path.join(root, 'file5.tar.bz2')
const temp7zFilePath = path.join(root, 'file6.7z')
const tempNormalFilePath = path.join(root, 'file6.txt')
jest.mock('../src/internal/config-variables')
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
// clear temp directory and create files that will be "uploaded"
await io.rmRF(root)
await fs.mkdir(path.join(root))
await fs.writeFile(tempGzipFilePath, 'a file with a .gzip file extension')
await fs.writeFile(tempZipFilePath, 'a file with a .zip file extension')
await fs.writeFile(tempTarlzFilePath, 'a file with a tar.lz file extension')
await fs.writeFile(tempGzFilePath, 'a file with a gz file file extension')
await fs.writeFile(tempBz2FilePath, 'a file with a .bz2 file extension')
await fs.writeFile(temp7zFilePath, 'a file with a .7z file extension')
await fs.writeFile(tempNormalFilePath, 'a file with a .txt file extension')
})
test('Number.MAX_SAFE_INTEGER is returned when an existing compressed file is used', async () => {
// create temporary file
const tempFile = await tmp.file()
expect(await createGZipFileOnDisk(tempGzipFilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(await createGZipFileOnDisk(tempZipFilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(await createGZipFileOnDisk(tempTarlzFilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(await createGZipFileOnDisk(tempGzFilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(await createGZipFileOnDisk(tempBz2FilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(await createGZipFileOnDisk(temp7zFilePath, tempFile.path)).toEqual(
Number.MAX_SAFE_INTEGER
)
expect(
await createGZipFileOnDisk(tempNormalFilePath, tempFile.path)
).not.toEqual(Number.MAX_SAFE_INTEGER)
})
test('gzip file on disk gets successfully created', async () => {
// create temporary file
const tempFile = await tmp.file()
const gzipFileSize = await createGZipFileOnDisk(
tempNormalFilePath,
tempFile.path
)
const fileStat = await fs.stat(tempNormalFilePath)
const totalFileSize = fileStat.size
// original file and gzip file should not be equal in size
expect(gzipFileSize).not.toEqual(totalFileSize)
})
@@ -0,0 +1,353 @@
import * as io from '../../io/src/io'
import * as path from 'path'
import {promises as fs} from 'fs'
import * as core from '@actions/core'
import {getUploadSpecification} from '../src/internal/upload-specification'
const artifactName = 'my-artifact'
const root = path.join(__dirname, '_temp', 'upload-specification')
const goodItem1Path = path.join(
root,
'folder-a',
'folder-b',
'folder-c',
'good-item1.txt'
)
const goodItem2Path = path.join(root, 'folder-d', 'good-item2.txt')
const goodItem3Path = path.join(root, 'folder-d', 'good-item3.txt')
const goodItem4Path = path.join(root, 'folder-d', 'good-item4.txt')
const goodItem5Path = path.join(root, 'good-item5.txt')
const badItem1Path = path.join(
root,
'folder-a',
'folder-b',
'folder-c',
'bad-item1.txt'
)
const badItem2Path = path.join(root, 'folder-d', 'bad-item2.txt')
const badItem3Path = path.join(root, 'folder-f', 'bad-item3.txt')
const badItem4Path = path.join(root, 'folder-h', 'folder-i', 'bad-item4.txt')
const badItem5Path = path.join(root, 'folder-h', 'folder-i', 'bad-item5.txt')
const extraFileInFolderCPath = path.join(
root,
'folder-a',
'folder-b',
'folder-c',
'extra-file-in-folder-c.txt'
)
const amazingFileInFolderHPath = path.join(root, 'folder-h', 'amazing-item.txt')
const artifactFilesToUpload = [
goodItem1Path,
goodItem2Path,
goodItem3Path,
goodItem4Path,
goodItem5Path,
extraFileInFolderCPath,
amazingFileInFolderHPath
]
describe('Search', () => {
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
// clear temp directory
await io.rmRF(root)
await fs.mkdir(path.join(root, 'folder-a', 'folder-b', 'folder-c'), {
recursive: true
})
await fs.mkdir(path.join(root, 'folder-a', 'folder-b', 'folder-e'), {
recursive: true
})
await fs.mkdir(path.join(root, 'folder-d'), {
recursive: true
})
await fs.mkdir(path.join(root, 'folder-f'), {
recursive: true
})
await fs.mkdir(path.join(root, 'folder-g'), {
recursive: true
})
await fs.mkdir(path.join(root, 'folder-h', 'folder-i'), {
recursive: true
})
await fs.writeFile(goodItem1Path, 'good item1 file')
await fs.writeFile(goodItem2Path, 'good item2 file')
await fs.writeFile(goodItem3Path, 'good item3 file')
await fs.writeFile(goodItem4Path, 'good item4 file')
await fs.writeFile(goodItem5Path, 'good item5 file')
await fs.writeFile(badItem1Path, 'bad item1 file')
await fs.writeFile(badItem2Path, 'bad item2 file')
await fs.writeFile(badItem3Path, 'bad item3 file')
await fs.writeFile(badItem4Path, 'bad item4 file')
await fs.writeFile(badItem5Path, 'bad item5 file')
await fs.writeFile(extraFileInFolderCPath, 'extra file')
await fs.writeFile(amazingFileInFolderHPath, 'amazing file')
/*
Directory structure of files that get created:
root/
folder-a/
folder-b/
folder-c/
good-item1.txt
bad-item1.txt
extra-file-in-folder-c.txt
folder-e/
folder-d/
good-item2.txt
good-item3.txt
good-item4.txt
bad-item2.txt
folder-f/
bad-item3.txt
folder-g/
folder-h/
amazing-item.txt
folder-i/
bad-item4.txt
bad-item5.txt
good-item5.txt
*/
})
it('Upload Specification - Fail non-existent rootDirectory', async () => {
const invalidRootDirectory = path.join(
__dirname,
'_temp',
'upload-specification-invalid'
)
expect(() => {
getUploadSpecification(
artifactName,
invalidRootDirectory,
artifactFilesToUpload
)
}).toThrow(`Provided rootDirectory ${invalidRootDirectory} does not exist`)
})
it('Upload Specification - Fail invalid rootDirectory', async () => {
expect(() => {
getUploadSpecification(artifactName, goodItem1Path, artifactFilesToUpload)
}).toThrow(
`Provided rootDirectory ${goodItem1Path} is not a valid directory`
)
})
it('Upload Specification - File does not exist', async () => {
const fakeFilePath = path.join(
artifactName,
'folder-a',
'folder-b',
'non-existent-file.txt'
)
expect(() => {
getUploadSpecification(artifactName, root, [fakeFilePath])
}).toThrow(`File ${fakeFilePath} does not exist`)
})
it('Upload Specification - Non parent directory', async () => {
const folderADirectory = path.join(root, 'folder-a')
const artifactFiles = [
goodItem1Path,
badItem1Path,
extraFileInFolderCPath,
goodItem5Path
]
expect(() => {
getUploadSpecification(artifactName, folderADirectory, artifactFiles)
}).toThrow(
`The rootDirectory: ${folderADirectory} is not a parent directory of the file: ${goodItem5Path}`
)
})
it('Upload Specification - Success', async () => {
const specifications = getUploadSpecification(
artifactName,
root,
artifactFilesToUpload
)
expect(specifications.length).toEqual(7)
const absolutePaths = specifications.map(item => item.absoluteFilePath)
expect(absolutePaths).toContain(goodItem1Path)
expect(absolutePaths).toContain(goodItem2Path)
expect(absolutePaths).toContain(goodItem3Path)
expect(absolutePaths).toContain(goodItem4Path)
expect(absolutePaths).toContain(goodItem5Path)
expect(absolutePaths).toContain(extraFileInFolderCPath)
expect(absolutePaths).toContain(amazingFileInFolderHPath)
for (const specification of specifications) {
if (specification.absoluteFilePath === goodItem1Path) {
expect(specification.uploadFilePath).toEqual(
path.join(
artifactName,
'folder-a',
'folder-b',
'folder-c',
'good-item1.txt'
)
)
} else if (specification.absoluteFilePath === goodItem2Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item2.txt')
)
} else if (specification.absoluteFilePath === goodItem3Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item3.txt')
)
} else if (specification.absoluteFilePath === goodItem4Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item4.txt')
)
} else if (specification.absoluteFilePath === goodItem5Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'good-item5.txt')
)
} else if (specification.absoluteFilePath === extraFileInFolderCPath) {
expect(specification.uploadFilePath).toEqual(
path.join(
artifactName,
'folder-a',
'folder-b',
'folder-c',
'extra-file-in-folder-c.txt'
)
)
} else if (specification.absoluteFilePath === amazingFileInFolderHPath) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-h', 'amazing-item.txt')
)
} else {
throw new Error(
'Invalid specification found. This should never be reached'
)
}
}
})
it('Upload Specification - Success with extra slash', async () => {
const rootWithSlash = `${root}/`
const specifications = getUploadSpecification(
artifactName,
rootWithSlash,
artifactFilesToUpload
)
expect(specifications.length).toEqual(7)
const absolutePaths = specifications.map(item => item.absoluteFilePath)
expect(absolutePaths).toContain(goodItem1Path)
expect(absolutePaths).toContain(goodItem2Path)
expect(absolutePaths).toContain(goodItem3Path)
expect(absolutePaths).toContain(goodItem4Path)
expect(absolutePaths).toContain(goodItem5Path)
expect(absolutePaths).toContain(extraFileInFolderCPath)
expect(absolutePaths).toContain(amazingFileInFolderHPath)
for (const specification of specifications) {
if (specification.absoluteFilePath === goodItem1Path) {
expect(specification.uploadFilePath).toEqual(
path.join(
artifactName,
'folder-a',
'folder-b',
'folder-c',
'good-item1.txt'
)
)
} else if (specification.absoluteFilePath === goodItem2Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item2.txt')
)
} else if (specification.absoluteFilePath === goodItem3Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item3.txt')
)
} else if (specification.absoluteFilePath === goodItem4Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item4.txt')
)
} else if (specification.absoluteFilePath === goodItem5Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'good-item5.txt')
)
} else if (specification.absoluteFilePath === extraFileInFolderCPath) {
expect(specification.uploadFilePath).toEqual(
path.join(
artifactName,
'folder-a',
'folder-b',
'folder-c',
'extra-file-in-folder-c.txt'
)
)
} else if (specification.absoluteFilePath === amazingFileInFolderHPath) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-h', 'amazing-item.txt')
)
} else {
throw new Error(
'Invalid specification found. This should never be reached'
)
}
}
})
it('Upload Specification - Directories should not be included', async () => {
const folderEPath = path.join(root, 'folder-a', 'folder-b', 'folder-e')
const filesWithDirectory = [
goodItem1Path,
goodItem4Path,
folderEPath,
badItem3Path
]
const specifications = getUploadSpecification(
artifactName,
root,
filesWithDirectory
)
expect(specifications.length).toEqual(3)
const absolutePaths = specifications.map(item => item.absoluteFilePath)
expect(absolutePaths).toContain(goodItem1Path)
expect(absolutePaths).toContain(goodItem4Path)
expect(absolutePaths).toContain(badItem3Path)
for (const specification of specifications) {
if (specification.absoluteFilePath === goodItem1Path) {
expect(specification.uploadFilePath).toEqual(
path.join(
artifactName,
'folder-a',
'folder-b',
'folder-c',
'good-item1.txt'
)
)
} else if (specification.absoluteFilePath === goodItem2Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item2.txt')
)
} else if (specification.absoluteFilePath === goodItem4Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-d', 'good-item4.txt')
)
} else if (specification.absoluteFilePath === badItem3Path) {
expect(specification.uploadFilePath).toEqual(
path.join(artifactName, 'folder-f', 'bad-item3.txt')
)
} else {
throw new Error(
'Invalid specification found. This should never be reached'
)
}
}
})
})
+551
View File
@@ -0,0 +1,551 @@
import * as http from 'http'
import * as io from '../../io/src/io'
import * as net from 'net'
import * as path from 'path'
import {mocked} from 'ts-jest/utils'
import {exec, execSync} from 'child_process'
import {createGunzip} from 'zlib'
import {promisify} from 'util'
import {UploadHttpClient} from '../src/internal/upload-http-client'
import * as core from '@actions/core'
import {promises as fs} from 'fs'
import {getRuntimeUrl} from '../src/internal/config-variables'
import {HttpClient, HttpClientResponse} from '@actions/http-client'
import {
ArtifactResponse,
PatchArtifactSizeSuccessResponse
} from '../src/internal/contracts'
import {UploadSpecification} from '../src/internal/upload-specification'
import {getArtifactUrl} from '../src/internal/utils'
import {UploadOptions} from '../src/internal/upload-options'
const root = path.join(__dirname, '_temp', 'artifact-upload')
const file1Path = path.join(root, 'file1.txt')
const file2Path = path.join(root, 'file2.txt')
const file3Path = path.join(root, 'folder1', 'file3.txt')
const file4Path = path.join(root, 'folder1', 'file4.txt')
const file5Path = path.join(root, 'folder1', 'folder2', 'folder3', 'file5.txt')
let file1Size = 0
let file2Size = 0
let file3Size = 0
let file4Size = 0
let file5Size = 0
jest.mock('../src/internal/config-variables')
jest.mock('@actions/http-client')
describe('Upload Tests', () => {
beforeAll(async () => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
// setup mocking for calls that got through the HttpClient
setupHttpClientMock()
// clear temp directory and create files that will be "uploaded"
await io.rmRF(root)
await fs.mkdir(path.join(root, 'folder1', 'folder2', 'folder3'), {
recursive: true
})
await fs.writeFile(file1Path, 'this is file 1')
await fs.writeFile(file2Path, 'this is file 2')
await fs.writeFile(file3Path, 'this is file 3')
await fs.writeFile(file4Path, 'this is file 4')
await fs.writeFile(file5Path, 'this is file 5')
/*
Directory structure for files that get created:
root/
file1.txt
file2.txt
folder1/
file3.txt
file4.txt
folder2/
folder3/
file5.txt
*/
file1Size = (await fs.stat(file1Path)).size
file2Size = (await fs.stat(file2Path)).size
file3Size = (await fs.stat(file3Path)).size
file4Size = (await fs.stat(file4Path)).size
file5Size = (await fs.stat(file5Path)).size
})
/**
* Artifact Creation Tests
*/
it('Create Artifact - Success', async () => {
const artifactName = 'valid-artifact-name'
const uploadHttpClient = new UploadHttpClient()
const response = await uploadHttpClient.createArtifactInFileContainer(
artifactName
)
expect(response.containerId).toEqual('13')
expect(response.size).toEqual(-1)
expect(response.signedContent).toEqual('false')
expect(response.fileContainerResourceUrl).toEqual(
`${getRuntimeUrl()}_apis/resources/Containers/13`
)
expect(response.type).toEqual('actions_storage')
expect(response.name).toEqual(artifactName)
expect(response.url).toEqual(
`${getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=${artifactName}`
)
})
it('Create Artifact - Failure', async () => {
const artifactName = 'invalid-artifact-name'
const uploadHttpClient = new UploadHttpClient()
expect(
uploadHttpClient.createArtifactInFileContainer(artifactName)
).rejects.toEqual(
new Error(
`Create Artifact Container failed: The artifact name invalid-artifact-name is not valid. Request URL ${getArtifactUrl()}`
)
)
})
it('Create Artifact - Retention Less Than Min Value Error', async () => {
const artifactName = 'valid-artifact-name'
const options: UploadOptions = {
retentionDays: -1
}
const uploadHttpClient = new UploadHttpClient()
expect(
uploadHttpClient.createArtifactInFileContainer(artifactName, options)
).rejects.toEqual(new Error('Invalid retention, minimum value is 1.'))
})
it('Create Artifact - Storage Quota Error', async () => {
const artifactName = 'storage-quota-hit'
const uploadHttpClient = new UploadHttpClient()
expect(
uploadHttpClient.createArtifactInFileContainer(artifactName)
).rejects.toEqual(
new Error(
'Create Artifact Container failed: Artifact storage quota has been hit. Unable to upload any new artifacts'
)
)
})
/**
* Artifact Upload Tests
*/
it('Upload Artifact - Success', async () => {
/**
* Normally search.findFilesToUpload() would be used for providing information about what to upload. These tests however
* focuses solely on the upload APIs so searchResult[] will be hard-coded
*/
const artifactName = 'successful-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `${artifactName}/file1.txt`
},
{
absoluteFilePath: file2Path,
uploadFilePath: `${artifactName}/file2.txt`
},
{
absoluteFilePath: file3Path,
uploadFilePath: `${artifactName}/folder1/file3.txt`
},
{
absoluteFilePath: file4Path,
uploadFilePath: `${artifactName}/folder1/file4.txt`
},
{
absoluteFilePath: file5Path,
uploadFilePath: `${artifactName}/folder1/folder2/folder3/file5.txt`
}
]
const expectedTotalSize =
file1Size + file2Size + file3Size + file4Size + file5Size
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification
)
expect(uploadResult.failedItems.length).toEqual(0)
expect(uploadResult.uploadSize).toEqual(expectedTotalSize)
})
function hasMkfifo(): boolean {
try {
// make sure we drain the stdout
return (
process.platform !== 'win32' &&
execSync('which mkfifo').toString().length > 0
)
} catch (e) {
return false
}
}
const withMkfifoIt = hasMkfifo() ? it : it.skip
withMkfifoIt(
'Upload Artifact with content from named pipe - Success',
async () => {
// create a named pipe 'pipe' with content 'hello pipe'
const content = Buffer.from('hello pipe')
const pipeFilePath = path.join(root, 'pipe')
await promisify(exec)('mkfifo pipe', {cwd: root})
// don't want to await here as that would block until read
fs.writeFile(pipeFilePath, content)
const artifactName = 'successful-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: pipeFilePath,
uploadFilePath: `${artifactName}/pipe`
}
]
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification
)
// accesses the ReadableStream that was passed into sendStream
// eslint-disable-next-line @typescript-eslint/unbound-method
const stream = mocked(HttpClient.prototype.sendStream).mock.calls[0][2]
expect(stream).not.toBeNull()
// decompresses the passed stream
const data: Buffer[] = []
for await (const chunk of stream.pipe(createGunzip())) {
data.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk as string))
}
const uploaded = Buffer.concat(data)
expect(uploadResult.failedItems.length).toEqual(0)
expect(uploaded).toEqual(content)
}
)
it('Upload Artifact - Failed Single File Upload', async () => {
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `this-file-upload-will-fail`
}
]
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification
)
expect(uploadResult.failedItems.length).toEqual(1)
expect(uploadResult.uploadSize).toEqual(0)
})
it('Upload Artifact - Partial Upload Continue On Error', async () => {
const artifactName = 'partial-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `${artifactName}/file1.txt`
},
{
absoluteFilePath: file2Path,
uploadFilePath: `${artifactName}/file2.txt`
},
{
absoluteFilePath: file3Path,
uploadFilePath: `${artifactName}/folder1/file3.txt`
},
{
absoluteFilePath: file4Path,
uploadFilePath: `this-file-upload-will-fail`
},
{
absoluteFilePath: file5Path,
uploadFilePath: `${artifactName}/folder1/folder2/folder3/file5.txt`
}
]
const expectedPartialSize = file1Size + file2Size + file4Size + file5Size
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification,
{continueOnError: true}
)
expect(uploadResult.failedItems.length).toEqual(1)
expect(uploadResult.uploadSize).toEqual(expectedPartialSize)
})
it('Upload Artifact - Partial Upload Fail Fast', async () => {
const artifactName = 'partial-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `${artifactName}/file1.txt`
},
{
absoluteFilePath: file2Path,
uploadFilePath: `${artifactName}/file2.txt`
},
{
absoluteFilePath: file3Path,
uploadFilePath: `${artifactName}/folder1/file3.txt`
},
{
absoluteFilePath: file4Path,
uploadFilePath: `this-file-upload-will-fail`
},
{
absoluteFilePath: file5Path,
uploadFilePath: `${artifactName}/folder1/folder2/folder3/file5.txt`
}
]
const expectedPartialSize = file1Size + file2Size + file3Size
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification,
{continueOnError: false}
)
expect(uploadResult.failedItems.length).toEqual(2)
expect(uploadResult.uploadSize).toEqual(expectedPartialSize)
})
it('Upload Artifact - Failed upload with no options', async () => {
const artifactName = 'partial-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `${artifactName}/file1.txt`
},
{
absoluteFilePath: file2Path,
uploadFilePath: `${artifactName}/file2.txt`
},
{
absoluteFilePath: file3Path,
uploadFilePath: `${artifactName}/folder1/file3.txt`
},
{
absoluteFilePath: file4Path,
uploadFilePath: `this-file-upload-will-fail`
},
{
absoluteFilePath: file5Path,
uploadFilePath: `${artifactName}/folder1/folder2/folder3/file5.txt`
}
]
const expectedPartialSize = file1Size + file2Size + file3Size + file5Size
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification
)
expect(uploadResult.failedItems.length).toEqual(1)
expect(uploadResult.uploadSize).toEqual(expectedPartialSize)
})
it('Upload Artifact - Failed upload with empty options', async () => {
const artifactName = 'partial-artifact'
const uploadSpecification: UploadSpecification[] = [
{
absoluteFilePath: file1Path,
uploadFilePath: `${artifactName}/file1.txt`
},
{
absoluteFilePath: file2Path,
uploadFilePath: `${artifactName}/file2.txt`
},
{
absoluteFilePath: file3Path,
uploadFilePath: `${artifactName}/folder1/file3.txt`
},
{
absoluteFilePath: file4Path,
uploadFilePath: `this-file-upload-will-fail`
},
{
absoluteFilePath: file5Path,
uploadFilePath: `${artifactName}/folder1/folder2/folder3/file5.txt`
}
]
const expectedPartialSize = file1Size + file2Size + file3Size + file5Size
const uploadUrl = `${getRuntimeUrl()}_apis/resources/Containers/13`
const uploadHttpClient = new UploadHttpClient()
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
uploadUrl,
uploadSpecification,
{}
)
expect(uploadResult.failedItems.length).toEqual(1)
expect(uploadResult.uploadSize).toEqual(expectedPartialSize)
})
/**
* Artifact Association Tests
*/
it('Associate Artifact - Success', async () => {
const uploadHttpClient = new UploadHttpClient()
expect(async () => {
uploadHttpClient.patchArtifactSize(130, 'my-artifact')
}).not.toThrow()
})
it('Associate Artifact - Not Found', async () => {
const uploadHttpClient = new UploadHttpClient()
expect(
uploadHttpClient.patchArtifactSize(100, 'non-existent-artifact')
).rejects.toThrow(
'An Artifact with the name non-existent-artifact was not found'
)
})
it('Associate Artifact - Error', async () => {
const uploadHttpClient = new UploadHttpClient()
expect(
uploadHttpClient.patchArtifactSize(-2, 'my-artifact')
).rejects.toThrow(
'Finalize artifact upload failed: Artifact service responded with 400'
)
})
/**
* Helpers used to setup mocking for the HttpClient
*/
async function emptyMockReadBody(): Promise<string> {
return new Promise(resolve => {
resolve()
})
}
function setupHttpClientMock(): void {
/**
* Mocks Post calls that are used during Artifact Creation tests
*
* Simulates success and non-success status codes depending on the artifact name along with an appropriate
* payload that represents an expected response
*/
jest
.spyOn(HttpClient.prototype, 'post')
.mockImplementation(async (requestdata, data) => {
// parse the input data and use the provided artifact name as part of the response
const inputData = JSON.parse(data)
const mockMessage = new http.IncomingMessage(new net.Socket())
let mockReadBody = emptyMockReadBody
if (inputData.Name === 'invalid-artifact-name') {
mockMessage.statusCode = 400
} else if (inputData.Name === 'storage-quota-hit') {
mockMessage.statusCode = 403
} else {
mockMessage.statusCode = 201
const response: ArtifactResponse = {
containerId: '13',
size: -1,
signedContent: 'false',
fileContainerResourceUrl: `${getRuntimeUrl()}_apis/resources/Containers/13`,
type: 'actions_storage',
name: inputData.Name,
url: `${getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=${
inputData.Name
}`
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
/**
* Mocks SendStream calls that are made during Artifact Upload tests
*
* A 500 response is used to simulate a failed upload stream. The uploadUrl can be set to
* include 'fail' to specify that the upload should fail
*/
jest
.spyOn(HttpClient.prototype, 'sendStream')
.mockImplementation(async (verb, requestUrl) => {
const mockMessage = new http.IncomingMessage(new net.Socket())
mockMessage.statusCode = 200
if (requestUrl.includes('fail')) {
mockMessage.statusCode = 500
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: emptyMockReadBody
})
})
})
/**
* Mocks Patch calls that are made during Artifact Association tests
*
* Simulates success and non-success status codes depending on the input size along with an appropriate
* payload that represents an expected response
*/
jest
.spyOn(HttpClient.prototype, 'patch')
.mockImplementation(async (requestdata, data) => {
const inputData = JSON.parse(data)
const mockMessage = new http.IncomingMessage(new net.Socket())
// Get the name from the end of requestdata. Will be something like https://www.example.com/_apis/pipelines/workflows/15/artifacts?api-version=6.0-preview&artifactName=my-artifact
const artifactName = requestdata.split('=')[2]
let mockReadBody = emptyMockReadBody
if (inputData.Size < 1) {
mockMessage.statusCode = 400
} else if (artifactName === 'non-existent-artifact') {
mockMessage.statusCode = 404
} else {
mockMessage.statusCode = 200
const response: PatchArtifactSizeSuccessResponse = {
containerId: 13,
size: inputData.Size,
signedContent: 'false',
type: 'actions_storage',
name: artifactName,
url: `${getRuntimeUrl()}_apis/pipelines/1/runs/1/artifacts?artifactName=${artifactName}`,
uploadUrl: `${getRuntimeUrl()}_apis/resources/Containers/13`
}
const returnData: string = JSON.stringify(response, null, 2)
mockReadBody = async function(): Promise<string> {
return new Promise(resolve => {
resolve(returnData)
})
}
}
return new Promise<HttpClientResponse>(resolve => {
resolve({
message: mockMessage,
readBody: mockReadBody
})
})
})
}
})
+222
View File
@@ -0,0 +1,222 @@
import * as fs from 'fs'
import * as io from '../../io/src/io'
import * as path from 'path'
import * as utils from '../src/internal/utils'
import * as core from '@actions/core'
import {HttpCodes} from '@actions/http-client'
import {
getRuntimeUrl,
getWorkFlowRunId,
getInitialRetryIntervalInMilliseconds,
getRetryMultiplier
} from '../src/internal/config-variables'
jest.mock('../src/internal/config-variables')
describe('Utils', () => {
beforeAll(() => {
// mock all output so that there is less noise when running tests
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
})
it('Check exponential retry range', () => {
// No retries should return the initial retry interval
const retryWaitTime0 = utils.getExponentialRetryTimeInMilliseconds(0)
expect(retryWaitTime0).toEqual(getInitialRetryIntervalInMilliseconds())
const testMinMaxRange = (retryCount: number): void => {
const retryWaitTime = utils.getExponentialRetryTimeInMilliseconds(
retryCount
)
const minRange =
getInitialRetryIntervalInMilliseconds() *
getRetryMultiplier() *
retryCount
const maxRange = minRange * getRetryMultiplier()
expect(retryWaitTime).toBeGreaterThanOrEqual(minRange)
expect(retryWaitTime).toBeLessThan(maxRange)
}
for (let i = 1; i < 10; i++) {
testMinMaxRange(i)
}
})
it('Test negative artifact retention throws', () => {
expect(() => {
utils.getProperRetention(-1, undefined)
}).toThrow()
})
it('Test no setting specified takes artifact retention input', () => {
expect(utils.getProperRetention(180, undefined)).toEqual(180)
})
it('Test artifact retention must conform to max allowed', () => {
expect(utils.getProperRetention(180, '45')).toEqual(45)
})
it('Test constructing artifact URL', () => {
const runtimeUrl = getRuntimeUrl()
const runId = getWorkFlowRunId()
const artifactUrl = utils.getArtifactUrl()
expect(artifactUrl).toEqual(
`${runtimeUrl}_apis/pipelines/workflows/${runId}/artifacts?api-version=${utils.getApiVersion()}`
)
})
it('Test constructing upload headers with all optional parameters', () => {
const contentType = 'application/octet-stream'
const size = 24
const uncompressedLength = 100
const range = 'bytes 0-199/200'
const headers = utils.getUploadHeaders(
contentType,
true,
true,
uncompressedLength,
size,
range
)
expect(Object.keys(headers).length).toEqual(8)
expect(headers['Accept']).toEqual(
`application/json;api-version=${utils.getApiVersion()}`
)
expect(headers['Content-Type']).toEqual(contentType)
expect(headers['Connection']).toEqual('Keep-Alive')
expect(headers['Keep-Alive']).toEqual('10')
expect(headers['Content-Encoding']).toEqual('gzip')
expect(headers['x-tfs-filelength']).toEqual(uncompressedLength)
expect(headers['Content-Length']).toEqual(size)
expect(headers['Content-Range']).toEqual(range)
})
it('Test constructing upload headers with only required parameter', () => {
const headers = utils.getUploadHeaders('application/octet-stream')
expect(Object.keys(headers).length).toEqual(2)
expect(headers['Accept']).toEqual(
`application/json;api-version=${utils.getApiVersion()}`
)
expect(headers['Content-Type']).toEqual('application/octet-stream')
})
it('Test constructing download headers with all optional parameters', () => {
const contentType = 'application/json'
const headers = utils.getDownloadHeaders(contentType, true, true)
expect(Object.keys(headers).length).toEqual(5)
expect(headers['Content-Type']).toEqual(contentType)
expect(headers['Connection']).toEqual('Keep-Alive')
expect(headers['Keep-Alive']).toEqual('10')
expect(headers['Accept-Encoding']).toEqual('gzip')
expect(headers['Accept']).toEqual(
`application/octet-stream;api-version=${utils.getApiVersion()}`
)
})
it('Test constructing download headers with only required parameter', () => {
const headers = utils.getDownloadHeaders('application/octet-stream')
expect(Object.keys(headers).length).toEqual(2)
expect(headers['Content-Type']).toEqual('application/octet-stream')
// check for default accept type
expect(headers['Accept']).toEqual(
`application/json;api-version=${utils.getApiVersion()}`
)
})
it('Test Success Status Code', () => {
expect(utils.isSuccessStatusCode(HttpCodes.OK)).toEqual(true)
expect(utils.isSuccessStatusCode(201)).toEqual(true)
expect(utils.isSuccessStatusCode(299)).toEqual(true)
expect(utils.isSuccessStatusCode(HttpCodes.NotFound)).toEqual(false)
expect(utils.isSuccessStatusCode(HttpCodes.BadGateway)).toEqual(false)
expect(utils.isSuccessStatusCode(HttpCodes.Forbidden)).toEqual(false)
})
it('Test Retry Status Code', () => {
expect(utils.isRetryableStatusCode(HttpCodes.BadGateway)).toEqual(true)
expect(utils.isRetryableStatusCode(HttpCodes.ServiceUnavailable)).toEqual(
true
)
expect(utils.isRetryableStatusCode(HttpCodes.GatewayTimeout)).toEqual(true)
expect(utils.isRetryableStatusCode(HttpCodes.TooManyRequests)).toEqual(true)
expect(utils.isRetryableStatusCode(HttpCodes.OK)).toEqual(false)
expect(utils.isRetryableStatusCode(HttpCodes.NotFound)).toEqual(false)
expect(utils.isRetryableStatusCode(HttpCodes.Forbidden)).toEqual(false)
expect(utils.isRetryableStatusCode(413)).toEqual(true) // Payload Too Large
})
it('Test Throttled Status Code', () => {
expect(utils.isThrottledStatusCode(HttpCodes.TooManyRequests)).toEqual(true)
expect(utils.isThrottledStatusCode(HttpCodes.InternalServerError)).toEqual(
false
)
expect(utils.isThrottledStatusCode(HttpCodes.BadGateway)).toEqual(false)
expect(utils.isThrottledStatusCode(HttpCodes.ServiceUnavailable)).toEqual(
false
)
})
it('Test Forbidden Status Code', () => {
expect(utils.isForbiddenStatusCode(HttpCodes.Forbidden)).toEqual(true)
expect(utils.isForbiddenStatusCode(HttpCodes.InternalServerError)).toEqual(
false
)
expect(utils.isForbiddenStatusCode(HttpCodes.TooManyRequests)).toEqual(
false
)
expect(utils.isForbiddenStatusCode(HttpCodes.OK)).toEqual(false)
})
it('Test Creating Artifact Directories', async () => {
const root = path.join(__dirname, '_temp', 'artifact-download')
// remove directory before starting
await io.rmRF(root)
const directory1 = path.join(root, 'folder2', 'folder3')
const directory2 = path.join(directory1, 'folder1')
// Initially should not exist
await expect(fs.promises.access(directory1)).rejects.not.toBeUndefined()
await expect(fs.promises.access(directory2)).rejects.not.toBeUndefined()
const directoryStructure = [directory1, directory2]
await utils.createDirectoriesForArtifact(directoryStructure)
// directories should now be created
await expect(fs.promises.access(directory1)).resolves.toEqual(undefined)
await expect(fs.promises.access(directory2)).resolves.toEqual(undefined)
})
it('Test Creating Empty Files', async () => {
const root = path.join(__dirname, '_temp', 'empty-files')
await io.rmRF(root)
const emptyFile1 = path.join(root, 'emptyFile1')
const directoryToCreate = path.join(root, 'folder1')
const emptyFile2 = path.join(directoryToCreate, 'emptyFile2')
// empty files should only be created after the directory structure is fully setup
// ensure they are first created by using the createDirectoriesForArtifact method
const directoryStructure = [root, directoryToCreate]
await utils.createDirectoriesForArtifact(directoryStructure)
await expect(fs.promises.access(root)).resolves.toEqual(undefined)
await expect(fs.promises.access(directoryToCreate)).resolves.toEqual(
undefined
)
await expect(fs.promises.access(emptyFile1)).rejects.not.toBeUndefined()
await expect(fs.promises.access(emptyFile2)).rejects.not.toBeUndefined()
const emptyFilesToCreate = [emptyFile1, emptyFile2]
await utils.createEmptyFilesForArtifact(emptyFilesToCreate)
await expect(fs.promises.access(emptyFile1)).resolves.toEqual(undefined)
const size1 = (await fs.promises.stat(emptyFile1)).size
expect(size1).toEqual(0)
await expect(fs.promises.access(emptyFile2)).resolves.toEqual(undefined)
const size2 = (await fs.promises.stat(emptyFile2)).size
expect(size2).toEqual(0)
})
})
@@ -0,0 +1,53 @@
# Additional Information
Extra information
- [Non-Supported Characters](#Non-Supported-Characters)
- [Permission loss](#Permission-Loss)
- [Considerations](#Considerations)
- [Compression](#Is-my-artifact-compressed)
## Non-Supported Characters
When uploading an artifact, the inputted `name` parameter along with the files specified in `files` cannot contain any of the following characters. They will be rejected by the server if attempted to be sent over and the upload will fail. These characters are not allowed due to limitations and restrictions with certain file systems such as NTFS. To maintain platform-agnostic behavior, all characters that are not supported by an individual filesystem/platform will not be supported on all filesystems/platforms.
- "
- :
- <
- \>
- |
- \*
- ?
In addition to the aforementioned characters, the inputted `name` also cannot include the following
- \
- /
## Permission Loss
File permissions are not maintained between uploaded and downloaded artifacts. If file permissions are something that need to be maintained (such as an executable), consider archiving all of the files using something like `tar` and then uploading the single archive. After downloading the artifact, you can `un-tar` the individual file and permissions will be preserved.
```js
const artifact = require('@actions/artifact');
const artifactClient = artifact.create()
const artifactName = 'my-artifact';
const files = [
'/home/user/files/plz-upload/my-archive.tgz',
]
const rootDirectory = '/home/user/files/plz-upload'
const uploadResult = await artifactClient.uploadArtifact(artifactName, files, rootDirectory)
```
## Considerations
During upload, each file is uploaded concurrently in 4MB chunks using a separate HTTPS connection per file. Chunked uploads are used so that in the event of a failure (which is entirely possible because the internet is not perfect), the upload can be retried. If there is an error, a retry will be attempted after a certain period of time.
Uploading will be generally be faster if there are fewer files that are larger in size vs if there are lots of smaller files. Depending on the types and quantities of files being uploaded, it might be beneficial to separately compress and archive everything into a single archive (using something like `tar` or `zip`) before starting and artifact upload to speed things up.
## Is my artifact compressed?
GZip is used internally to compress individual files before starting an upload. Compression helps reduce the total amount of data that must be uploaded and stored while helping to speed up uploads (this performance benefit is significant especially on self hosted runners). If GZip does not reduce the size of the file that is being uploaded, the original file is uploaded as-is.
Compression using GZip also helps speed up artifact download as part of a workflow. Header information is used to determine if an individual file was uploaded using GZip and if necessary, decompression is used.
When downloading an artifact from the GitHub UI (this differs from downloading an artifact during a workflow), a single Zip file is dynamically created that contains all of the files uploaded as part of an artifact. Any files that were uploaded using GZip will be decompressed on the server before being added to the Zip file with the remaining files.
@@ -0,0 +1,57 @@
# Implementation Details
Warning: Implementation details may change at any time without notice. This is meant to serve as a reference to help users understand the package.
## Upload/Compression flow
![image](https://user-images.githubusercontent.com/16109154/79765587-19522b00-8327-11ea-9679-410bb10e1b13.png)
During artifact upload, gzip is used to compress individual files that then get uploaded. This is used to minimize the amount of data that gets uploaded which reduces the total amount of HTTP calls (upload happens in 4MB chunks). This results in considerably faster uploads with huge performance implications especially on self-hosted runners.
If a file is less than 64KB in size, a passthrough stream (readable and writable) is used to convert an in-memory buffer into a readable stream without any extra streams or pipping.
## Retry Logic when downloading an individual file
![image](https://user-images.githubusercontent.com/16109154/78555461-5be71400-780d-11ea-9abd-b05b77a95a3f.png)
## Proxy support
This package uses the `@actions/http-client` NPM package internally which supports proxied requests out of the box.
## HttpManager
### `keep-alive` header
When an HTTP call is made to upload or download an individual file, the server will close the HTTP connection after the upload/download is complete and respond with a header indicating `Connection: close`.
[HTTP closed connection header information](https://tools.ietf.org/html/rfc2616#section-14.10)
TCP connections are sometimes not immediately closed by the node client (Windows might hold on to the port for an extra period of time before actually releasing it for example) and a large amount of closed connections can cause port exhaustion before ports get released and are available again.
VMs hosted by GitHub Actions have 1024 available ports so uploading 1000+ files very quickly can cause port exhaustion if connections get closed immediately. This can start to cause strange undefined behavior and timeouts.
In order for connections to not close immediately, the `keep-alive` header is used to indicate to the server that the connection should stay open. If a `keep-alive` header is used, the connection needs to be disposed of by calling `dispose()` in the `HttpClient`.
[`keep-alive` header information](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive)
[@actions/http-client client disposal](https://github.com/actions/http-client/blob/04e5ad73cd3fd1f5610a32116b0759eddf6570d2/index.ts#L292)
### Multiple HTTP clients
During an artifact upload or download, files are concurrently uploaded or downloaded using `async/await`. When an error or retry is encountered, the `HttpClient` that made a call is disposed of and a new one is created. If a single `HttpClient` was used for all HTTP calls and it had to be disposed, it could inadvertently effect any other calls that could be concurrently happening.
Any other concurrent uploads or downloads should be left untouched. Because of this, each concurrent upload or download gets its own `HttpClient`. The `http-manager` is used to manage all available clients and each concurrent upload or download maintains a `httpClientIndex` that keep track of which client should be used (and potentially disposed and recycled if necessary)
### Potential resource leaks
When an HTTP response is received, it consists of two parts
- `message`
- `body`
The `message` contains information such as the response code and header information and it is available immediately. The body however is not available immediately and it can be read by calling `await response.readBody()`.
TCP connections consist of an input and output buffer to manage what is sent and received across a connection. If the body is not read (even if its contents are not needed) the buffers can stay in use even after `dispose()` gets called on the `HttpClient`. The buffers get released automatically after a certain period of time, but in order for them to be explicitly cleared, `readBody()` is always called.
### Non Concurrent calls
Both `upload-http-client` and `download-http-client` do not instantiate or create any HTTP clients (the `HttpManager` has that responsibility). If an HTTP call has to be made that does not require the `keep-alive` header (such as when calling `listArtifacts` or `patchArtifactSize`), the first `HttpClient` in the `HttpManager` is used. The number of available clients is equal to the upload or download concurrency and there will always be at least one available.
+324
View File
@@ -0,0 +1,324 @@
{
"name": "@actions/artifact",
"version": "0.6.1",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "@actions/artifact",
"version": "0.6.1",
"license": "MIT",
"dependencies": {
"@actions/core": "^1.2.6",
"@actions/http-client": "^1.0.11",
"tmp": "^0.2.1",
"tmp-promise": "^3.0.2"
},
"devDependencies": {
"@types/tmp": "^0.2.1",
"typescript": "^3.8.3"
}
},
"node_modules/@actions/core": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/@actions/core/-/core-1.6.0.tgz",
"integrity": "sha512-NB1UAZomZlCV/LmJqkLhNTqtKfFXJZAUPcfl/zqG7EfsQdeUJtaWO98SGbuQ3pydJ3fHl2CvI/51OKYlCYYcaw==",
"dependencies": {
"@actions/http-client": "^1.0.11"
}
},
"node_modules/@actions/http-client": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/@actions/http-client/-/http-client-1.0.11.tgz",
"integrity": "sha512-VRYHGQV1rqnROJqdMvGUbY/Kn8vriQe/F9HR2AlYHzmKuM/p3kjNuXhmdBfcVgsvRWTz5C5XW5xvndZrVBuAYg==",
"dependencies": {
"tunnel": "0.0.6"
}
},
"node_modules/@types/tmp": {
"version": "0.2.3",
"resolved": "https://registry.npmjs.org/@types/tmp/-/tmp-0.2.3.tgz",
"integrity": "sha512-dDZH/tXzwjutnuk4UacGgFRwV+JSLaXL1ikvidfJprkb7L9Nx1njcRHHmi3Dsvt7pgqqTEeucQuOrWHPFgzVHA==",
"dev": true
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="
},
"node_modules/brace-expansion": {
"version": "1.1.11",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
"integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==",
"dependencies": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
},
"node_modules/fs.realpath": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
"integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8="
},
"node_modules/glob": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/glob/-/glob-7.2.0.tgz",
"integrity": "sha512-lmLf6gtyrPq8tTjSmrO94wBeQbFR3HbLHbuyD69wuyQkImp2hWqMGB47OX65FBkPffO641IP9jWa1z4ivqG26Q==",
"dependencies": {
"fs.realpath": "^1.0.0",
"inflight": "^1.0.4",
"inherits": "2",
"minimatch": "^3.0.4",
"once": "^1.3.0",
"path-is-absolute": "^1.0.0"
},
"engines": {
"node": "*"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/inflight": {
"version": "1.0.6",
"resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
"integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
"dependencies": {
"once": "^1.3.0",
"wrappy": "1"
}
},
"node_modules/inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
},
"node_modules/minimatch": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz",
"integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==",
"dependencies": {
"brace-expansion": "^1.1.7"
},
"engines": {
"node": "*"
}
},
"node_modules/once": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=",
"dependencies": {
"wrappy": "1"
}
},
"node_modules/path-is-absolute": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
"integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18=",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/rimraf": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz",
"integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==",
"dependencies": {
"glob": "^7.1.3"
},
"bin": {
"rimraf": "bin.js"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/tmp": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/tmp/-/tmp-0.2.1.tgz",
"integrity": "sha512-76SUhtfqR2Ijn+xllcI5P1oyannHNHByD80W1q447gU3mp9G9PSpGdWmjUOHRDPiHYacIk66W7ubDTuPF3BEtQ==",
"dependencies": {
"rimraf": "^3.0.0"
},
"engines": {
"node": ">=8.17.0"
}
},
"node_modules/tmp-promise": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/tmp-promise/-/tmp-promise-3.0.3.tgz",
"integrity": "sha512-RwM7MoPojPxsOBYnyd2hy0bxtIlVrihNs9pj5SUvY8Zz1sQcQG2tG1hSr8PDxfgEB8RNKDhqbIlroIarSNDNsQ==",
"dependencies": {
"tmp": "^0.2.0"
}
},
"node_modules/tunnel": {
"version": "0.0.6",
"resolved": "https://registry.npmjs.org/tunnel/-/tunnel-0.0.6.tgz",
"integrity": "sha512-1h/Lnq9yajKY2PEbBadPXj3VxsDDu844OnaAo52UVmIzIvwwtBPIuNvkjuzBlTWpfJyUbG3ez0KSBibQkj4ojg==",
"engines": {
"node": ">=0.6.11 <=0.7.0 || >=0.7.3"
}
},
"node_modules/typescript": {
"version": "3.9.10",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.10.tgz",
"integrity": "sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==",
"dev": true,
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=4.2.0"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
}
},
"dependencies": {
"@actions/core": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/@actions/core/-/core-1.6.0.tgz",
"integrity": "sha512-NB1UAZomZlCV/LmJqkLhNTqtKfFXJZAUPcfl/zqG7EfsQdeUJtaWO98SGbuQ3pydJ3fHl2CvI/51OKYlCYYcaw==",
"requires": {
"@actions/http-client": "^1.0.11"
}
},
"@actions/http-client": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/@actions/http-client/-/http-client-1.0.11.tgz",
"integrity": "sha512-VRYHGQV1rqnROJqdMvGUbY/Kn8vriQe/F9HR2AlYHzmKuM/p3kjNuXhmdBfcVgsvRWTz5C5XW5xvndZrVBuAYg==",
"requires": {
"tunnel": "0.0.6"
}
},
"@types/tmp": {
"version": "0.2.3",
"resolved": "https://registry.npmjs.org/@types/tmp/-/tmp-0.2.3.tgz",
"integrity": "sha512-dDZH/tXzwjutnuk4UacGgFRwV+JSLaXL1ikvidfJprkb7L9Nx1njcRHHmi3Dsvt7pgqqTEeucQuOrWHPFgzVHA==",
"dev": true
},
"balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="
},
"brace-expansion": {
"version": "1.1.11",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
"integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==",
"requires": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
}
},
"concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
},
"fs.realpath": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
"integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8="
},
"glob": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/glob/-/glob-7.2.0.tgz",
"integrity": "sha512-lmLf6gtyrPq8tTjSmrO94wBeQbFR3HbLHbuyD69wuyQkImp2hWqMGB47OX65FBkPffO641IP9jWa1z4ivqG26Q==",
"requires": {
"fs.realpath": "^1.0.0",
"inflight": "^1.0.4",
"inherits": "2",
"minimatch": "^3.0.4",
"once": "^1.3.0",
"path-is-absolute": "^1.0.0"
}
},
"inflight": {
"version": "1.0.6",
"resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
"integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
"requires": {
"once": "^1.3.0",
"wrappy": "1"
}
},
"inherits": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
},
"minimatch": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.2.tgz",
"integrity": "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw==",
"requires": {
"brace-expansion": "^1.1.7"
}
},
"once": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=",
"requires": {
"wrappy": "1"
}
},
"path-is-absolute": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
"integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18="
},
"rimraf": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz",
"integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==",
"requires": {
"glob": "^7.1.3"
}
},
"tmp": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/tmp/-/tmp-0.2.1.tgz",
"integrity": "sha512-76SUhtfqR2Ijn+xllcI5P1oyannHNHByD80W1q447gU3mp9G9PSpGdWmjUOHRDPiHYacIk66W7ubDTuPF3BEtQ==",
"requires": {
"rimraf": "^3.0.0"
}
},
"tmp-promise": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/tmp-promise/-/tmp-promise-3.0.3.tgz",
"integrity": "sha512-RwM7MoPojPxsOBYnyd2hy0bxtIlVrihNs9pj5SUvY8Zz1sQcQG2tG1hSr8PDxfgEB8RNKDhqbIlroIarSNDNsQ==",
"requires": {
"tmp": "^0.2.0"
}
},
"tunnel": {
"version": "0.0.6",
"resolved": "https://registry.npmjs.org/tunnel/-/tunnel-0.0.6.tgz",
"integrity": "sha512-1h/Lnq9yajKY2PEbBadPXj3VxsDDu844OnaAo52UVmIzIvwwtBPIuNvkjuzBlTWpfJyUbG3ez0KSBibQkj4ojg=="
},
"typescript": {
"version": "3.9.10",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.10.tgz",
"integrity": "sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==",
"dev": true
},
"wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
}
}
}
+49
View File
@@ -0,0 +1,49 @@
{
"name": "@actions/artifact",
"version": "1.0.0",
"preview": true,
"description": "Actions artifact lib",
"keywords": [
"github",
"actions",
"artifact"
],
"homepage": "https://github.com/actions/toolkit/tree/main/packages/artifact",
"license": "MIT",
"main": "lib/artifact-client.js",
"types": "lib/artifact-client.d.ts",
"directories": {
"lib": "lib",
"test": "__tests__"
},
"files": [
"lib",
"!.DS_Store"
],
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/actions/toolkit.git",
"directory": "packages/artifact"
},
"scripts": {
"audit-moderate": "npm install && npm audit --json --audit-level=moderate > audit.json",
"test": "echo \"Error: run tests from root\" && exit 1",
"tsc": "tsc"
},
"bugs": {
"url": "https://github.com/actions/toolkit/issues"
},
"dependencies": {
"@actions/core": "^1.2.6",
"@actions/http-client": "^1.0.11",
"tmp": "^0.2.1",
"tmp-promise": "^3.0.2"
},
"devDependencies": {
"@types/tmp": "^0.2.1",
"typescript": "^3.8.3"
}
}
+20
View File
@@ -0,0 +1,20 @@
import {UploadOptions} from './internal/upload-options'
import {UploadResponse} from './internal/upload-response'
import {DownloadOptions} from './internal/download-options'
import {DownloadResponse} from './internal/download-response'
import {ArtifactClient, DefaultArtifactClient} from './internal/artifact-client'
export {
ArtifactClient,
UploadResponse,
UploadOptions,
DownloadResponse,
DownloadOptions
}
/**
* Constructs an ArtifactClient
*/
export function create(): ArtifactClient {
return DefaultArtifactClient.create()
}
@@ -0,0 +1,51 @@
/**
* Mocks default limits for easier testing
*/
export function getUploadFileConcurrency(): number {
return 1
}
export function getUploadChunkConcurrency(): number {
return 1
}
export function getUploadChunkSize(): number {
return 4 * 1024 * 1024 // 4 MB Chunks
}
export function getRetryLimit(): number {
return 2
}
export function getRetryMultiplier(): number {
return 1.5
}
export function getInitialRetryIntervalInMilliseconds(): number {
return 10
}
export function getDownloadFileConcurrency(): number {
return 1
}
/**
* Mocks the 'ACTIONS_RUNTIME_TOKEN', 'ACTIONS_RUNTIME_URL' and 'GITHUB_RUN_ID' env variables
* that are only available from a node context on the runner. This allows for tests to run
* locally without the env variables actually being set
*/
export function getRuntimeToken(): string {
return 'totally-valid-token'
}
export function getRuntimeUrl(): string {
return 'https://www.example.com/'
}
export function getWorkFlowRunId(): string {
return '15'
}
export function getRetentionDays(): string | undefined {
return '45'
}
@@ -0,0 +1,282 @@
import * as core from '@actions/core'
import {
UploadSpecification,
getUploadSpecification
} from './upload-specification'
import {UploadHttpClient} from './upload-http-client'
import {UploadResponse} from './upload-response'
import {UploadOptions} from './upload-options'
import {DownloadOptions} from './download-options'
import {DownloadResponse} from './download-response'
import {
createDirectoriesForArtifact,
createEmptyFilesForArtifact
} from './utils'
import {checkArtifactName} from './path-and-artifact-name-validation'
import {DownloadHttpClient} from './download-http-client'
import {getDownloadSpecification} from './download-specification'
import {getWorkSpaceDirectory} from './config-variables'
import {normalize, resolve} from 'path'
export interface ArtifactClient {
/**
* Uploads an artifact
*
* @param name the name of the artifact, required
* @param files a list of absolute or relative paths that denote what files should be uploaded
* @param rootDirectory an absolute or relative file path that denotes the root parent directory of the files being uploaded
* @param options extra options for customizing the upload behavior
* @returns single UploadInfo object
*/
uploadArtifact(
name: string,
files: string[],
rootDirectory: string,
options?: UploadOptions
): Promise<UploadResponse>
/**
* Downloads a single artifact associated with a run
*
* @param name the name of the artifact being downloaded
* @param path optional path that denotes where the artifact will be downloaded to
* @param options extra options that allow for the customization of the download behavior
*/
downloadArtifact(
name: string,
path?: string,
options?: DownloadOptions
): Promise<DownloadResponse>
/**
* Downloads all artifacts associated with a run. Because there are multiple artifacts being downloaded, a folder will be created for each one in the specified or default directory
* @param path optional path that denotes where the artifacts will be downloaded to
*/
downloadAllArtifacts(path?: string): Promise<DownloadResponse[]>
}
export class DefaultArtifactClient implements ArtifactClient {
/**
* Constructs a DefaultArtifactClient
*/
static create(): DefaultArtifactClient {
return new DefaultArtifactClient()
}
/**
* Uploads an artifact
*/
async uploadArtifact(
name: string,
files: string[],
rootDirectory: string,
options?: UploadOptions | undefined
): Promise<UploadResponse> {
core.info(
`Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging`
)
checkArtifactName(name)
// Get specification for the files being uploaded
const uploadSpecification: UploadSpecification[] = getUploadSpecification(
name,
rootDirectory,
files
)
const uploadResponse: UploadResponse = {
artifactName: name,
artifactItems: [],
size: 0,
failedItems: []
}
const uploadHttpClient = new UploadHttpClient()
if (uploadSpecification.length === 0) {
core.warning(`No files found that can be uploaded`)
} else {
// Create an entry for the artifact in the file container
const response = await uploadHttpClient.createArtifactInFileContainer(
name,
options
)
if (!response.fileContainerResourceUrl) {
core.debug(response.toString())
throw new Error(
'No URL provided by the Artifact Service to upload an artifact to'
)
}
core.debug(`Upload Resource URL: ${response.fileContainerResourceUrl}`)
core.info(
`Container for artifact "${name}" successfully created. Starting upload of file(s)`
)
// Upload each of the files that were found concurrently
const uploadResult = await uploadHttpClient.uploadArtifactToFileContainer(
response.fileContainerResourceUrl,
uploadSpecification,
options
)
// Update the size of the artifact to indicate we are done uploading
// The uncompressed size is used for display when downloading a zip of the artifact from the UI
core.info(
`File upload process has finished. Finalizing the artifact upload`
)
await uploadHttpClient.patchArtifactSize(uploadResult.totalSize, name)
if (uploadResult.failedItems.length > 0) {
core.info(
`Upload finished. There were ${uploadResult.failedItems.length} items that failed to upload`
)
} else {
core.info(
`Artifact has been finalized. All files have been successfully uploaded!`
)
}
core.info(
`
The raw size of all the files that were specified for upload is ${uploadResult.totalSize} bytes
The size of all the files that were uploaded is ${uploadResult.uploadSize} bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads \r\n`
)
uploadResponse.artifactItems = uploadSpecification.map(
item => item.absoluteFilePath
)
uploadResponse.size = uploadResult.uploadSize
uploadResponse.failedItems = uploadResult.failedItems
}
return uploadResponse
}
async downloadArtifact(
name: string,
path?: string | undefined,
options?: DownloadOptions | undefined
): Promise<DownloadResponse> {
const downloadHttpClient = new DownloadHttpClient()
const artifacts = await downloadHttpClient.listArtifacts()
if (artifacts.count === 0) {
throw new Error(
`Unable to find any artifacts for the associated workflow`
)
}
const artifactToDownload = artifacts.value.find(artifact => {
return artifact.name === name
})
if (!artifactToDownload) {
throw new Error(`Unable to find an artifact with the name: ${name}`)
}
const items = await downloadHttpClient.getContainerItems(
artifactToDownload.name,
artifactToDownload.fileContainerResourceUrl
)
if (!path) {
path = getWorkSpaceDirectory()
}
path = normalize(path)
path = resolve(path)
// During upload, empty directories are rejected by the remote server so there should be no artifacts that consist of only empty directories
const downloadSpecification = getDownloadSpecification(
name,
items.value,
path,
options?.createArtifactFolder || false
)
if (downloadSpecification.filesToDownload.length === 0) {
core.info(
`No downloadable files were found for the artifact: ${artifactToDownload.name}`
)
} else {
// Create all necessary directories recursively before starting any download
await createDirectoriesForArtifact(
downloadSpecification.directoryStructure
)
core.info('Directory structure has been setup for the artifact')
await createEmptyFilesForArtifact(
downloadSpecification.emptyFilesToCreate
)
await downloadHttpClient.downloadSingleArtifact(
downloadSpecification.filesToDownload
)
}
return {
artifactName: name,
downloadPath: downloadSpecification.rootDownloadLocation
}
}
async downloadAllArtifacts(
path?: string | undefined
): Promise<DownloadResponse[]> {
const downloadHttpClient = new DownloadHttpClient()
const response: DownloadResponse[] = []
const artifacts = await downloadHttpClient.listArtifacts()
if (artifacts.count === 0) {
core.info('Unable to find any artifacts for the associated workflow')
return response
}
if (!path) {
path = getWorkSpaceDirectory()
}
path = normalize(path)
path = resolve(path)
let downloadedArtifacts = 0
while (downloadedArtifacts < artifacts.count) {
const currentArtifactToDownload = artifacts.value[downloadedArtifacts]
downloadedArtifacts += 1
core.info(
`starting download of artifact ${currentArtifactToDownload.name} : ${downloadedArtifacts}/${artifacts.count}`
)
// Get container entries for the specific artifact
const items = await downloadHttpClient.getContainerItems(
currentArtifactToDownload.name,
currentArtifactToDownload.fileContainerResourceUrl
)
const downloadSpecification = getDownloadSpecification(
currentArtifactToDownload.name,
items.value,
path,
true
)
if (downloadSpecification.filesToDownload.length === 0) {
core.info(
`No downloadable files were found for any artifact ${currentArtifactToDownload.name}`
)
} else {
await createDirectoriesForArtifact(
downloadSpecification.directoryStructure
)
await createEmptyFilesForArtifact(
downloadSpecification.emptyFilesToCreate
)
await downloadHttpClient.downloadSingleArtifact(
downloadSpecification.filesToDownload
)
}
response.push({
artifactName: currentArtifactToDownload.name,
downloadPath: downloadSpecification.rootDownloadLocation
})
}
return response
}
}
@@ -0,0 +1,67 @@
// The number of concurrent uploads that happens at the same time
export function getUploadFileConcurrency(): number {
return 2
}
// When uploading large files that can't be uploaded with a single http call, this controls
// the chunk size that is used during upload
export function getUploadChunkSize(): number {
return 8 * 1024 * 1024 // 8 MB Chunks
}
// The maximum number of retries that can be attempted before an upload or download fails
export function getRetryLimit(): number {
return 5
}
// With exponential backoff, the larger the retry count, the larger the wait time before another attempt
// The retry multiplier controls by how much the backOff time increases depending on the number of retries
export function getRetryMultiplier(): number {
return 1.5
}
// The initial wait time if an upload or download fails and a retry is being attempted for the first time
export function getInitialRetryIntervalInMilliseconds(): number {
return 3000
}
// The number of concurrent downloads that happens at the same time
export function getDownloadFileConcurrency(): number {
return 2
}
export function getRuntimeToken(): string {
const token = process.env['ACTIONS_RUNTIME_TOKEN']
if (!token) {
throw new Error('Unable to get ACTIONS_RUNTIME_TOKEN env variable')
}
return token
}
export function getRuntimeUrl(): string {
const runtimeUrl = process.env['ACTIONS_RUNTIME_URL']
if (!runtimeUrl) {
throw new Error('Unable to get ACTIONS_RUNTIME_URL env variable')
}
return runtimeUrl
}
export function getWorkFlowRunId(): string {
const workFlowRunId = process.env['GITHUB_RUN_ID']
if (!workFlowRunId) {
throw new Error('Unable to get GITHUB_RUN_ID env variable')
}
return workFlowRunId
}
export function getWorkSpaceDirectory(): string {
const workspaceDirectory = process.env['GITHUB_WORKSPACE']
if (!workspaceDirectory) {
throw new Error('Unable to get GITHUB_WORKSPACE env variable')
}
return workspaceDirectory
}
export function getRetentionDays(): string | undefined {
return process.env['GITHUB_RETENTION_DAYS']
}
@@ -0,0 +1,76 @@
export interface ArtifactResponse {
containerId: string
size: number
signedContent: string
fileContainerResourceUrl: string
type: string
name: string
url: string
}
export interface CreateArtifactParameters {
Type: string
Name: string
RetentionDays?: number
}
export interface PatchArtifactSize {
Size: number
}
export interface PatchArtifactSizeSuccessResponse {
containerId: number
size: number
signedContent: string
type: string
name: string
url: string
uploadUrl: string
}
export interface UploadResults {
/**
* The size in bytes of data that was transferred during the upload process to the actions backend service. This takes into account possible
* gzip compression to reduce the amount of data that needs to be transferred
*/
uploadSize: number
/**
* The raw size of the files that were specified for upload
*/
totalSize: number
/**
* An array of files that failed to upload
*/
failedItems: string[]
}
export interface ListArtifactsResponse {
count: number
value: ArtifactResponse[]
}
export interface QueryArtifactResponse {
count: number
value: ContainerEntry[]
}
export interface ContainerEntry {
containerId: number
scopeIdentifier: string
path: string
itemType: string
status: string
fileLength?: number
fileEncoding?: number
fileType?: number
dateCreated: string
dateLastModified: string
createdBy: string
lastModifiedBy: string
itemLocation: string
contentLocation: string
fileId?: number
contentId: string
}
@@ -0,0 +1,355 @@
import * as fs from 'fs'
import * as core from '@actions/core'
import * as zlib from 'zlib'
import {
getArtifactUrl,
getDownloadHeaders,
isSuccessStatusCode,
isRetryableStatusCode,
isThrottledStatusCode,
getExponentialRetryTimeInMilliseconds,
tryGetRetryAfterValueTimeInMilliseconds,
displayHttpDiagnostics,
getFileSize,
rmFile,
sleep
} from './utils'
import {URL} from 'url'
import {StatusReporter} from './status-reporter'
import {performance} from 'perf_hooks'
import {ListArtifactsResponse, QueryArtifactResponse} from './contracts'
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {HttpManager} from './http-manager'
import {DownloadItem} from './download-specification'
import {getDownloadFileConcurrency, getRetryLimit} from './config-variables'
import {IncomingHttpHeaders} from 'http'
import {retryHttpClientRequest} from './requestUtils'
export class DownloadHttpClient {
// http manager is used for concurrent connections when downloading multiple files at once
private downloadHttpManager: HttpManager
private statusReporter: StatusReporter
constructor() {
this.downloadHttpManager = new HttpManager(
getDownloadFileConcurrency(),
'@actions/artifact-download'
)
// downloads are usually significantly faster than uploads so display status information every second
this.statusReporter = new StatusReporter(1000)
}
/**
* Gets a list of all artifacts that are in a specific container
*/
async listArtifacts(): Promise<ListArtifactsResponse> {
const artifactUrl = getArtifactUrl()
// use the first client from the httpManager, `keep-alive` is not used so the connection will close immediately
const client = this.downloadHttpManager.getClient(0)
const headers = getDownloadHeaders('application/json')
const response = await retryHttpClientRequest('List Artifacts', async () =>
client.get(artifactUrl, headers)
)
const body: string = await response.readBody()
return JSON.parse(body)
}
/**
* Fetches a set of container items that describe the contents of an artifact
* @param artifactName the name of the artifact
* @param containerUrl the artifact container URL for the run
*/
async getContainerItems(
artifactName: string,
containerUrl: string
): Promise<QueryArtifactResponse> {
// the itemPath search parameter controls which containers will be returned
const resourceUrl = new URL(containerUrl)
resourceUrl.searchParams.append('itemPath', artifactName)
// use the first client from the httpManager, `keep-alive` is not used so the connection will close immediately
const client = this.downloadHttpManager.getClient(0)
const headers = getDownloadHeaders('application/json')
const response = await retryHttpClientRequest(
'Get Container Items',
async () => client.get(resourceUrl.toString(), headers)
)
const body: string = await response.readBody()
return JSON.parse(body)
}
/**
* Concurrently downloads all the files that are part of an artifact
* @param downloadItems information about what items to download and where to save them
*/
async downloadSingleArtifact(downloadItems: DownloadItem[]): Promise<void> {
const DOWNLOAD_CONCURRENCY = getDownloadFileConcurrency()
// limit the number of files downloaded at a single time
core.debug(`Download file concurrency is set to ${DOWNLOAD_CONCURRENCY}`)
const parallelDownloads = [...new Array(DOWNLOAD_CONCURRENCY).keys()]
let currentFile = 0
let downloadedFiles = 0
core.info(
`Total number of files that will be downloaded: ${downloadItems.length}`
)
this.statusReporter.setTotalNumberOfFilesToProcess(downloadItems.length)
this.statusReporter.start()
await Promise.all(
parallelDownloads.map(async index => {
while (currentFile < downloadItems.length) {
const currentFileToDownload = downloadItems[currentFile]
currentFile += 1
const startTime = performance.now()
await this.downloadIndividualFile(
index,
currentFileToDownload.sourceLocation,
currentFileToDownload.targetPath
)
if (core.isDebug()) {
core.debug(
`File: ${++downloadedFiles}/${downloadItems.length}. ${
currentFileToDownload.targetPath
} took ${(performance.now() - startTime).toFixed(
3
)} milliseconds to finish downloading`
)
}
this.statusReporter.incrementProcessedCount()
}
})
)
.catch(error => {
throw new Error(`Unable to download the artifact: ${error}`)
})
.finally(() => {
this.statusReporter.stop()
// safety dispose all connections
this.downloadHttpManager.disposeAndReplaceAllClients()
})
}
/**
* Downloads an individual file
* @param httpClientIndex the index of the http client that is used to make all of the calls
* @param artifactLocation origin location where a file will be downloaded from
* @param downloadPath destination location for the file being downloaded
*/
private async downloadIndividualFile(
httpClientIndex: number,
artifactLocation: string,
downloadPath: string
): Promise<void> {
let retryCount = 0
const retryLimit = getRetryLimit()
let destinationStream = fs.createWriteStream(downloadPath)
const headers = getDownloadHeaders('application/json', true, true)
// a single GET request is used to download a file
const makeDownloadRequest = async (): Promise<IHttpClientResponse> => {
const client = this.downloadHttpManager.getClient(httpClientIndex)
return await client.get(artifactLocation, headers)
}
// check the response headers to determine if the file was compressed using gzip
const isGzip = (incomingHeaders: IncomingHttpHeaders): boolean => {
return (
'content-encoding' in incomingHeaders &&
incomingHeaders['content-encoding'] === 'gzip'
)
}
// Increments the current retry count and then checks if the retry limit has been reached
// If there have been too many retries, fail so the download stops. If there is a retryAfterValue value provided,
// it will be used
const backOff = async (retryAfterValue?: number): Promise<void> => {
retryCount++
if (retryCount > retryLimit) {
return Promise.reject(
new Error(
`Retry limit has been reached. Unable to download ${artifactLocation}`
)
)
} else {
this.downloadHttpManager.disposeAndReplaceClient(httpClientIndex)
if (retryAfterValue) {
// Back off by waiting the specified time denoted by the retry-after header
core.info(
`Backoff due to too many requests, retry #${retryCount}. Waiting for ${retryAfterValue} milliseconds before continuing the download`
)
await sleep(retryAfterValue)
} else {
// Back off using an exponential value that depends on the retry count
const backoffTime = getExponentialRetryTimeInMilliseconds(retryCount)
core.info(
`Exponential backoff for retry #${retryCount}. Waiting for ${backoffTime} milliseconds before continuing the download`
)
await sleep(backoffTime)
}
core.info(
`Finished backoff for retry #${retryCount}, continuing with download`
)
}
}
const isAllBytesReceived = (
expected?: string,
received?: number
): boolean => {
// be lenient, if any input is missing, assume success, i.e. not truncated
if (
!expected ||
!received ||
process.env['ACTIONS_ARTIFACT_SKIP_DOWNLOAD_VALIDATION']
) {
core.info('Skipping download validation.')
return true
}
return parseInt(expected) === received
}
const resetDestinationStream = async (
fileDownloadPath: string
): Promise<void> => {
destinationStream.close()
await rmFile(fileDownloadPath)
destinationStream = fs.createWriteStream(fileDownloadPath)
}
// keep trying to download a file until a retry limit has been reached
while (retryCount <= retryLimit) {
let response: IHttpClientResponse
try {
response = await makeDownloadRequest()
} catch (error) {
// if an error is caught, it is usually indicative of a timeout so retry the download
core.info('An error occurred while attempting to download a file')
// eslint-disable-next-line no-console
console.log(error)
// increment the retryCount and use exponential backoff to wait before making the next request
await backOff()
continue
}
let forceRetry = false
if (isSuccessStatusCode(response.message.statusCode)) {
// The body contains the contents of the file however calling response.readBody() causes all the content to be converted to a string
// which can cause some gzip encoded data to be lost
// Instead of using response.readBody(), response.message is a readableStream that can be directly used to get the raw body contents
try {
const isGzipped = isGzip(response.message.headers)
await this.pipeResponseToFile(response, destinationStream, isGzipped)
if (
isGzipped ||
isAllBytesReceived(
response.message.headers['content-length'],
await getFileSize(downloadPath)
)
) {
return
} else {
forceRetry = true
}
} catch (error) {
// retry on error, most likely streams were corrupted
forceRetry = true
}
}
if (forceRetry || isRetryableStatusCode(response.message.statusCode)) {
core.info(
`A ${response.message.statusCode} response code has been received while attempting to download an artifact`
)
resetDestinationStream(downloadPath)
// if a throttled status code is received, try to get the retryAfter header value, else differ to standard exponential backoff
isThrottledStatusCode(response.message.statusCode)
? await backOff(
tryGetRetryAfterValueTimeInMilliseconds(response.message.headers)
)
: await backOff()
} else {
// Some unexpected response code, fail immediately and stop the download
displayHttpDiagnostics(response)
return Promise.reject(
new Error(
`Unexpected http ${response.message.statusCode} during download for ${artifactLocation}`
)
)
}
}
}
/**
* Pipes the response from downloading an individual file to the appropriate destination stream while decoding gzip content if necessary
* @param response the http response received when downloading a file
* @param destinationStream the stream where the file should be written to
* @param isGzip a boolean denoting if the content is compressed using gzip and if we need to decode it
*/
async pipeResponseToFile(
response: IHttpClientResponse,
destinationStream: fs.WriteStream,
isGzip: boolean
): Promise<void> {
await new Promise((resolve, reject) => {
if (isGzip) {
const gunzip = zlib.createGunzip()
response.message
.on('error', error => {
core.error(
`An error occurred while attempting to read the response stream`
)
gunzip.close()
destinationStream.close()
reject(error)
})
.pipe(gunzip)
.on('error', error => {
core.error(
`An error occurred while attempting to decompress the response stream`
)
destinationStream.close()
reject(error)
})
.pipe(destinationStream)
.on('close', () => {
resolve()
})
.on('error', error => {
core.error(
`An error occurred while writing a downloaded file to ${destinationStream.path}`
)
reject(error)
})
} else {
response.message
.on('error', error => {
core.error(
`An error occurred while attempting to read the response stream`
)
destinationStream.close()
reject(error)
})
.pipe(destinationStream)
.on('close', () => {
resolve()
})
.on('error', error => {
core.error(
`An error occurred while writing a downloaded file to ${destinationStream.path}`
)
reject(error)
})
}
})
return
}
}
@@ -0,0 +1,7 @@
export interface DownloadOptions {
/**
* Specifies if a folder is created for the artifact that is downloaded (contents downloaded into this folder),
* defaults to false if not specified
* */
createArtifactFolder?: boolean
}
@@ -0,0 +1,11 @@
export interface DownloadResponse {
/**
* The name of the artifact that was downloaded
*/
artifactName: string
/**
* The full Path to where the artifact was downloaded
*/
downloadPath: string
}
@@ -0,0 +1,87 @@
import * as path from 'path'
import {ContainerEntry} from './contracts'
export interface DownloadSpecification {
// root download location for the artifact
rootDownloadLocation: string
// directories that need to be created for all the items in the artifact
directoryStructure: string[]
// empty files that are part of the artifact that don't require any downloading
emptyFilesToCreate: string[]
// individual files that need to be downloaded as part of the artifact
filesToDownload: DownloadItem[]
}
export interface DownloadItem {
// Url that denotes where to download the item from
sourceLocation: string
// Information about where the file should be downloaded to
targetPath: string
}
/**
* Creates a specification for a set of files that will be downloaded
* @param artifactName the name of the artifact
* @param artifactEntries a set of container entries that describe that files that make up an artifact
* @param downloadPath the path where the artifact will be downloaded to
* @param includeRootDirectory specifies if there should be an extra directory (denoted by the artifact name) where the artifact files should be downloaded to
*/
export function getDownloadSpecification(
artifactName: string,
artifactEntries: ContainerEntry[],
downloadPath: string,
includeRootDirectory: boolean
): DownloadSpecification {
// use a set for the directory paths so that there are no duplicates
const directories = new Set<string>()
const specifications: DownloadSpecification = {
rootDownloadLocation: includeRootDirectory
? path.join(downloadPath, artifactName)
: downloadPath,
directoryStructure: [],
emptyFilesToCreate: [],
filesToDownload: []
}
for (const entry of artifactEntries) {
// Ignore artifacts in the container that don't begin with the same name
if (
entry.path.startsWith(`${artifactName}/`) ||
entry.path.startsWith(`${artifactName}\\`)
) {
// normalize all separators to the local OS
const normalizedPathEntry = path.normalize(entry.path)
// entry.path always starts with the artifact name, if includeRootDirectory is false, remove the name from the beginning of the path
const filePath = path.join(
downloadPath,
includeRootDirectory
? normalizedPathEntry
: normalizedPathEntry.replace(artifactName, '')
)
// Case insensitive folder structure maintained in the backend, not every folder is created so the 'folder'
// itemType cannot be relied upon. The file must be used to determine the directory structure
if (entry.itemType === 'file') {
// Get the directories that we need to create from the filePath for each individual file
directories.add(path.dirname(filePath))
if (entry.fileLength === 0) {
// An empty file was uploaded, create the empty files locally so that no extra http calls are made
specifications.emptyFilesToCreate.push(filePath)
} else {
specifications.filesToDownload.push({
sourceLocation: entry.contentLocation,
targetPath: filePath
})
}
}
}
}
specifications.directoryStructure = Array.from(directories)
return specifications
}
@@ -0,0 +1,35 @@
import {HttpClient} from '@actions/http-client/index'
import {createHttpClient} from './utils'
/**
* Used for managing http clients during either upload or download
*/
export class HttpManager {
private clients: HttpClient[]
private userAgent: string
constructor(clientCount: number, userAgent: string) {
if (clientCount < 1) {
throw new Error('There must be at least one client')
}
this.userAgent = userAgent
this.clients = new Array(clientCount).fill(createHttpClient(userAgent))
}
getClient(index: number): HttpClient {
return this.clients[index]
}
// client disposal is necessary if a keep-alive connection is used to properly close the connection
// for more information see: https://github.com/actions/http-client/blob/04e5ad73cd3fd1f5610a32116b0759eddf6570d2/index.ts#L292
disposeAndReplaceClient(index: number): void {
this.clients[index].dispose()
this.clients[index] = createHttpClient(this.userAgent)
}
disposeAndReplaceAllClients(): void {
for (const [index] of this.clients.entries()) {
this.disposeAndReplaceClient(index)
}
}
}
@@ -0,0 +1,82 @@
import {info} from '@actions/core'
/**
* Invalid characters that cannot be in the artifact name or an uploaded file. Will be rejected
* from the server if attempted to be sent over. These characters are not allowed due to limitations with certain
* file systems such as NTFS. To maintain platform-agnostic behavior, all characters that are not supported by an
* individual filesystem/platform will not be supported on all fileSystems/platforms
*
* FilePaths can include characters such as \ and / which are not permitted in the artifact name alone
*/
const invalidArtifactFilePathCharacters = new Map<string, string>([
['"', ' Double quote "'],
[':', ' Colon :'],
['<', ' Less than <'],
['>', ' Greater than >'],
['|', ' Vertical bar |'],
['*', ' Asterisk *'],
['?', ' Question mark ?'],
['\r', ' Carriage return \\r'],
['\n', ' Line feed \\n']
])
const invalidArtifactNameCharacters = new Map<string, string>([
...invalidArtifactFilePathCharacters,
['\\', ' Backslash \\'],
['/', ' Forward slash /']
])
/**
* Scans the name of the artifact to make sure there are no illegal characters
*/
export function checkArtifactName(name: string): void {
if (!name) {
throw new Error(`Artifact name: ${name}, is incorrectly provided`)
}
for (const [
invalidCharacterKey,
errorMessageForCharacter
] of invalidArtifactNameCharacters) {
if (name.includes(invalidCharacterKey)) {
throw new Error(
`Artifact name is not valid: ${name}. Contains the following character: ${errorMessageForCharacter}
Invalid characters include: ${Array.from(
invalidArtifactNameCharacters.values()
).toString()}
These characters are not allowed in the artifact name due to limitations with certain file systems such as NTFS. To maintain file system agnostic behavior, these characters are intentionally not allowed to prevent potential problems with downloads on different file systems.`
)
}
}
info(`Artifact name is valid!`)
}
/**
* Scans the name of the filePath used to make sure there are no illegal characters
*/
export function checkArtifactFilePath(path: string): void {
if (!path) {
throw new Error(`Artifact path: ${path}, is incorrectly provided`)
}
for (const [
invalidCharacterKey,
errorMessageForCharacter
] of invalidArtifactFilePathCharacters) {
if (path.includes(invalidCharacterKey)) {
throw new Error(
`Artifact path is not valid: ${path}. Contains the following character: ${errorMessageForCharacter}
Invalid characters include: ${Array.from(
invalidArtifactFilePathCharacters.values()
).toString()}
The following characters are not allowed in files that are uploaded due to limitations with certain file systems such as NTFS. To maintain file system agnostic behavior, these characters are intentionally not allowed to prevent potential problems with downloads on different file systems.
`
)
}
}
}
@@ -0,0 +1,79 @@
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {
isRetryableStatusCode,
isSuccessStatusCode,
sleep,
getExponentialRetryTimeInMilliseconds,
displayHttpDiagnostics
} from './utils'
import * as core from '@actions/core'
import {getRetryLimit} from './config-variables'
export async function retry(
name: string,
operation: () => Promise<IHttpClientResponse>,
customErrorMessages: Map<number, string>,
maxAttempts: number
): Promise<IHttpClientResponse> {
let response: IHttpClientResponse | undefined = undefined
let statusCode: number | undefined = undefined
let isRetryable = false
let errorMessage = ''
let customErrorInformation: string | undefined = undefined
let attempt = 1
while (attempt <= maxAttempts) {
try {
response = await operation()
statusCode = response.message.statusCode
if (isSuccessStatusCode(statusCode)) {
return response
}
// Extra error information that we want to display if a particular response code is hit
if (statusCode) {
customErrorInformation = customErrorMessages.get(statusCode)
}
isRetryable = isRetryableStatusCode(statusCode)
errorMessage = `Artifact service responded with ${statusCode}`
} catch (error) {
isRetryable = true
errorMessage = error.message
}
if (!isRetryable) {
core.info(`${name} - Error is not retryable`)
if (response) {
displayHttpDiagnostics(response)
}
break
}
core.info(
`${name} - Attempt ${attempt} of ${maxAttempts} failed with error: ${errorMessage}`
)
await sleep(getExponentialRetryTimeInMilliseconds(attempt))
attempt++
}
if (response) {
displayHttpDiagnostics(response)
}
if (customErrorInformation) {
throw Error(`${name} failed: ${customErrorInformation}`)
}
throw Error(`${name} failed: ${errorMessage}`)
}
export async function retryHttpClientRequest(
name: string,
method: () => Promise<IHttpClientResponse>,
customErrorMessages: Map<number, string> = new Map(),
maxAttempts = getRetryLimit()
): Promise<IHttpClientResponse> {
return await retry(name, method, customErrorMessages, maxAttempts)
}
@@ -0,0 +1,78 @@
import {info} from '@actions/core'
/**
* Status Reporter that displays information about the progress/status of an artifact that is being uploaded or downloaded
*
* Variable display time that can be adjusted using the displayFrequencyInMilliseconds variable
* The total status of the upload/download gets displayed according to this value
* If there is a large file that is being uploaded, extra information about the individual status can also be displayed using the updateLargeFileStatus function
*/
export class StatusReporter {
private totalNumberOfFilesToProcess = 0
private processedCount = 0
private displayFrequencyInMilliseconds: number
private largeFiles = new Map<string, string>()
private totalFileStatus: NodeJS.Timeout | undefined
constructor(displayFrequencyInMilliseconds: number) {
this.totalFileStatus = undefined
this.displayFrequencyInMilliseconds = displayFrequencyInMilliseconds
}
setTotalNumberOfFilesToProcess(fileTotal: number): void {
this.totalNumberOfFilesToProcess = fileTotal
this.processedCount = 0
}
start(): void {
// displays information about the total upload/download status
this.totalFileStatus = setInterval(() => {
// display 1 decimal place without any rounding
const percentage = this.formatPercentage(
this.processedCount,
this.totalNumberOfFilesToProcess
)
info(
`Total file count: ${
this.totalNumberOfFilesToProcess
} ---- Processed file #${this.processedCount} (${percentage.slice(
0,
percentage.indexOf('.') + 2
)}%)`
)
}, this.displayFrequencyInMilliseconds)
}
// if there is a large file that is being uploaded in chunks, this is used to display extra information about the status of the upload
updateLargeFileStatus(
fileName: string,
chunkStartIndex: number,
chunkEndIndex: number,
totalUploadFileSize: number
): void {
// display 1 decimal place without any rounding
const percentage = this.formatPercentage(chunkEndIndex, totalUploadFileSize)
info(
`Uploaded ${fileName} (${percentage.slice(
0,
percentage.indexOf('.') + 2
)}%) bytes ${chunkStartIndex}:${chunkEndIndex}`
)
}
stop(): void {
if (this.totalFileStatus) {
clearInterval(this.totalFileStatus)
}
}
incrementProcessedCount(): void {
this.processedCount++
}
private formatPercentage(numerator: number, denominator: number): string {
// toFixed() rounds, so use extra precision to display accurate information even though 4 decimal places are not displayed
return ((numerator / denominator) * 100).toFixed(4).toString()
}
}
@@ -0,0 +1,74 @@
import * as fs from 'fs'
import * as zlib from 'zlib'
import {promisify} from 'util'
const stat = promisify(fs.stat)
/**
* GZipping certain files that are already compressed will likely not yield further size reductions. Creating large temporary gzip
* files then will just waste a lot of time before ultimately being discarded (especially for very large files).
* If any of these types of files are encountered then on-disk gzip creation will be skipped and the original file will be uploaded as-is
*/
const gzipExemptFileExtensions = [
'.gzip',
'.zip',
'.tar.lz',
'.tar.gz',
'.tar.bz2',
'.7z'
]
/**
* Creates a Gzip compressed file of an original file at the provided temporary filepath location
* @param {string} originalFilePath filepath of whatever will be compressed. The original file will be unmodified
* @param {string} tempFilePath the location of where the Gzip file will be created
* @returns the size of gzip file that gets created
*/
export async function createGZipFileOnDisk(
originalFilePath: string,
tempFilePath: string
): Promise<number> {
for (const gzipExemptExtension of gzipExemptFileExtensions) {
if (originalFilePath.endsWith(gzipExemptExtension)) {
// return a really large number so that the original file gets uploaded
return Number.MAX_SAFE_INTEGER
}
}
return new Promise((resolve, reject) => {
const inputStream = fs.createReadStream(originalFilePath)
const gzip = zlib.createGzip()
const outputStream = fs.createWriteStream(tempFilePath)
inputStream.pipe(gzip).pipe(outputStream)
outputStream.on('finish', async () => {
// wait for stream to finish before calculating the size which is needed as part of the Content-Length header when starting an upload
const size = (await stat(tempFilePath)).size
resolve(size)
})
outputStream.on('error', error => {
// eslint-disable-next-line no-console
console.log(error)
reject
})
})
}
/**
* Creates a GZip file in memory using a buffer. Should be used for smaller files to reduce disk I/O
* @param originalFilePath the path to the original file that is being GZipped
* @returns a buffer with the GZip file
*/
export async function createGZipFileInBuffer(
originalFilePath: string
): Promise<Buffer> {
return new Promise(async resolve => {
const inputStream = fs.createReadStream(originalFilePath)
const gzip = zlib.createGzip()
inputStream.pipe(gzip)
// read stream into buffer, using experimental async iterators see https://github.com/nodejs/readable-stream/issues/403#issuecomment-479069043
const chunks = []
for await (const chunk of gzip) {
chunks.push(chunk)
}
resolve(Buffer.concat(chunks))
})
}
@@ -0,0 +1,563 @@
import * as fs from 'fs'
import * as core from '@actions/core'
import * as tmp from 'tmp-promise'
import * as stream from 'stream'
import {
ArtifactResponse,
CreateArtifactParameters,
PatchArtifactSize,
UploadResults
} from './contracts'
import {
getArtifactUrl,
getContentRange,
getUploadHeaders,
isRetryableStatusCode,
isSuccessStatusCode,
isThrottledStatusCode,
displayHttpDiagnostics,
getExponentialRetryTimeInMilliseconds,
tryGetRetryAfterValueTimeInMilliseconds,
getProperRetention,
sleep
} from './utils'
import {
getUploadChunkSize,
getUploadFileConcurrency,
getRetryLimit,
getRetentionDays
} from './config-variables'
import {promisify} from 'util'
import {URL} from 'url'
import {performance} from 'perf_hooks'
import {StatusReporter} from './status-reporter'
import {HttpCodes} from '@actions/http-client'
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {HttpManager} from './http-manager'
import {UploadSpecification} from './upload-specification'
import {UploadOptions} from './upload-options'
import {createGZipFileOnDisk, createGZipFileInBuffer} from './upload-gzip'
import {retryHttpClientRequest} from './requestUtils'
const stat = promisify(fs.stat)
export class UploadHttpClient {
private uploadHttpManager: HttpManager
private statusReporter: StatusReporter
constructor() {
this.uploadHttpManager = new HttpManager(
getUploadFileConcurrency(),
'@actions/artifact-upload'
)
this.statusReporter = new StatusReporter(10000)
}
/**
* Creates a file container for the new artifact in the remote blob storage/file service
* @param {string} artifactName Name of the artifact being created
* @returns The response from the Artifact Service if the file container was successfully created
*/
async createArtifactInFileContainer(
artifactName: string,
options?: UploadOptions | undefined
): Promise<ArtifactResponse> {
const parameters: CreateArtifactParameters = {
Type: 'actions_storage',
Name: artifactName
}
// calculate retention period
if (options && options.retentionDays) {
const maxRetentionStr = getRetentionDays()
parameters.RetentionDays = getProperRetention(
options.retentionDays,
maxRetentionStr
)
}
const data: string = JSON.stringify(parameters, null, 2)
const artifactUrl = getArtifactUrl()
// use the first client from the httpManager, `keep-alive` is not used so the connection will close immediately
const client = this.uploadHttpManager.getClient(0)
const headers = getUploadHeaders('application/json', false)
// Extra information to display when a particular HTTP code is returned
// If a 403 is returned when trying to create a file container, the customer has exceeded
// their storage quota so no new artifact containers can be created
const customErrorMessages: Map<number, string> = new Map([
[
HttpCodes.Forbidden,
'Artifact storage quota has been hit. Unable to upload any new artifacts'
],
[
HttpCodes.BadRequest,
`The artifact name ${artifactName} is not valid. Request URL ${artifactUrl}`
]
])
const response = await retryHttpClientRequest(
'Create Artifact Container',
async () => client.post(artifactUrl, data, headers),
customErrorMessages
)
const body: string = await response.readBody()
return JSON.parse(body)
}
/**
* Concurrently upload all of the files in chunks
* @param {string} uploadUrl Base Url for the artifact that was created
* @param {SearchResult[]} filesToUpload A list of information about the files being uploaded
* @returns The size of all the files uploaded in bytes
*/
async uploadArtifactToFileContainer(
uploadUrl: string,
filesToUpload: UploadSpecification[],
options?: UploadOptions
): Promise<UploadResults> {
const FILE_CONCURRENCY = getUploadFileConcurrency()
const MAX_CHUNK_SIZE = getUploadChunkSize()
core.debug(
`File Concurrency: ${FILE_CONCURRENCY}, and Chunk Size: ${MAX_CHUNK_SIZE}`
)
const parameters: UploadFileParameters[] = []
// by default, file uploads will continue if there is an error unless specified differently in the options
let continueOnError = true
if (options) {
if (options.continueOnError === false) {
continueOnError = false
}
}
// prepare the necessary parameters to upload all the files
for (const file of filesToUpload) {
const resourceUrl = new URL(uploadUrl)
resourceUrl.searchParams.append('itemPath', file.uploadFilePath)
parameters.push({
file: file.absoluteFilePath,
resourceUrl: resourceUrl.toString(),
maxChunkSize: MAX_CHUNK_SIZE,
continueOnError
})
}
const parallelUploads = [...new Array(FILE_CONCURRENCY).keys()]
const failedItemsToReport: string[] = []
let currentFile = 0
let completedFiles = 0
let uploadFileSize = 0
let totalFileSize = 0
let abortPendingFileUploads = false
this.statusReporter.setTotalNumberOfFilesToProcess(filesToUpload.length)
this.statusReporter.start()
// only allow a certain amount of files to be uploaded at once, this is done to reduce potential errors
await Promise.all(
parallelUploads.map(async index => {
while (currentFile < filesToUpload.length) {
const currentFileParameters = parameters[currentFile]
currentFile += 1
if (abortPendingFileUploads) {
failedItemsToReport.push(currentFileParameters.file)
continue
}
const startTime = performance.now()
const uploadFileResult = await this.uploadFileAsync(
index,
currentFileParameters
)
if (core.isDebug()) {
core.debug(
`File: ${++completedFiles}/${filesToUpload.length}. ${
currentFileParameters.file
} took ${(performance.now() - startTime).toFixed(
3
)} milliseconds to finish upload`
)
}
uploadFileSize += uploadFileResult.successfulUploadSize
totalFileSize += uploadFileResult.totalSize
if (uploadFileResult.isSuccess === false) {
failedItemsToReport.push(currentFileParameters.file)
if (!continueOnError) {
// fail fast
core.error(`aborting artifact upload`)
abortPendingFileUploads = true
}
}
this.statusReporter.incrementProcessedCount()
}
})
)
this.statusReporter.stop()
// done uploading, safety dispose all connections
this.uploadHttpManager.disposeAndReplaceAllClients()
core.info(`Total size of all the files uploaded is ${uploadFileSize} bytes`)
return {
uploadSize: uploadFileSize,
totalSize: totalFileSize,
failedItems: failedItemsToReport
}
}
/**
* Asynchronously uploads a file. The file is compressed and uploaded using GZip if it is determined to save space.
* If the upload file is bigger than the max chunk size it will be uploaded via multiple calls
* @param {number} httpClientIndex The index of the httpClient that is being used to make all of the calls
* @param {UploadFileParameters} parameters Information about the file that needs to be uploaded
* @returns The size of the file that was uploaded in bytes along with any failed uploads
*/
private async uploadFileAsync(
httpClientIndex: number,
parameters: UploadFileParameters
): Promise<UploadFileResult> {
const fileStat: fs.Stats = await stat(parameters.file)
const totalFileSize = fileStat.size
const isFIFO = fileStat.isFIFO()
let offset = 0
let isUploadSuccessful = true
let failedChunkSizes = 0
let uploadFileSize = 0
let isGzip = true
// the file that is being uploaded is less than 64k in size to increase throughput and to minimize disk I/O
// for creating a new GZip file, an in-memory buffer is used for compression
// with named pipes the file size is reported as zero in that case don't read the file in memory
if (!isFIFO && totalFileSize < 65536) {
core.debug(
`${parameters.file} is less than 64k in size. Creating a gzip file in-memory to potentially reduce the upload size`
)
const buffer = await createGZipFileInBuffer(parameters.file)
// An open stream is needed in the event of a failure and we need to retry. If a NodeJS.ReadableStream is directly passed in,
// it will not properly get reset to the start of the stream if a chunk upload needs to be retried
let openUploadStream: () => NodeJS.ReadableStream
if (totalFileSize < buffer.byteLength) {
// compression did not help with reducing the size, use a readable stream from the original file for upload
core.debug(
`The gzip file created for ${parameters.file} did not help with reducing the size of the file. The original file will be uploaded as-is`
)
openUploadStream = () => fs.createReadStream(parameters.file)
isGzip = false
uploadFileSize = totalFileSize
} else {
// create a readable stream using a PassThrough stream that is both readable and writable
core.debug(
`A gzip file created for ${parameters.file} helped with reducing the size of the original file. The file will be uploaded using gzip.`
)
openUploadStream = () => {
const passThrough = new stream.PassThrough()
passThrough.end(buffer)
return passThrough
}
uploadFileSize = buffer.byteLength
}
const result = await this.uploadChunk(
httpClientIndex,
parameters.resourceUrl,
openUploadStream,
0,
uploadFileSize - 1,
uploadFileSize,
isGzip,
totalFileSize
)
if (!result) {
// chunk failed to upload
isUploadSuccessful = false
failedChunkSizes += uploadFileSize
core.warning(`Aborting upload for ${parameters.file} due to failure`)
}
return {
isSuccess: isUploadSuccessful,
successfulUploadSize: uploadFileSize - failedChunkSizes,
totalSize: totalFileSize
}
} else {
// the file that is being uploaded is greater than 64k in size, a temporary file gets created on disk using the
// npm tmp-promise package and this file gets used to create a GZipped file
const tempFile = await tmp.file()
core.debug(
`${parameters.file} is greater than 64k in size. Creating a gzip file on-disk ${tempFile.path} to potentially reduce the upload size`
)
// create a GZip file of the original file being uploaded, the original file should not be modified in any way
uploadFileSize = await createGZipFileOnDisk(
parameters.file,
tempFile.path
)
let uploadFilePath = tempFile.path
// compression did not help with size reduction, use the original file for upload and delete the temp GZip file
// for named pipes totalFileSize is zero, this assumes compression did help
if (!isFIFO && totalFileSize < uploadFileSize) {
core.debug(
`The gzip file created for ${parameters.file} did not help with reducing the size of the file. The original file will be uploaded as-is`
)
uploadFileSize = totalFileSize
uploadFilePath = parameters.file
isGzip = false
} else {
core.debug(
`The gzip file created for ${parameters.file} is smaller than the original file. The file will be uploaded using gzip.`
)
}
let abortFileUpload = false
// upload only a single chunk at a time
while (offset < uploadFileSize) {
const chunkSize = Math.min(
uploadFileSize - offset,
parameters.maxChunkSize
)
const startChunkIndex = offset
const endChunkIndex = offset + chunkSize - 1
offset += parameters.maxChunkSize
if (abortFileUpload) {
// if we don't want to continue in the event of an error, any pending upload chunks will be marked as failed
failedChunkSizes += chunkSize
continue
}
const result = await this.uploadChunk(
httpClientIndex,
parameters.resourceUrl,
() =>
fs.createReadStream(uploadFilePath, {
start: startChunkIndex,
end: endChunkIndex,
autoClose: false
}),
startChunkIndex,
endChunkIndex,
uploadFileSize,
isGzip,
totalFileSize
)
if (!result) {
// Chunk failed to upload, report as failed and do not continue uploading any more chunks for the file. It is possible that part of a chunk was
// successfully uploaded so the server may report a different size for what was uploaded
isUploadSuccessful = false
failedChunkSizes += chunkSize
core.warning(`Aborting upload for ${parameters.file} due to failure`)
abortFileUpload = true
} else {
// if an individual file is greater than 8MB (1024*1024*8) in size, display extra information about the upload status
if (uploadFileSize > 8388608) {
this.statusReporter.updateLargeFileStatus(
parameters.file,
startChunkIndex,
endChunkIndex,
uploadFileSize
)
}
}
}
// Delete the temporary file that was created as part of the upload. If the temp file does not get manually deleted by
// calling cleanup, it gets removed when the node process exits. For more info see: https://www.npmjs.com/package/tmp-promise#about
core.debug(`deleting temporary gzip file ${tempFile.path}`)
await tempFile.cleanup()
return {
isSuccess: isUploadSuccessful,
successfulUploadSize: uploadFileSize - failedChunkSizes,
totalSize: totalFileSize
}
}
}
/**
* Uploads a chunk of an individual file to the specified resourceUrl. If the upload fails and the status code
* indicates a retryable status, we try to upload the chunk as well
* @param {number} httpClientIndex The index of the httpClient being used to make all the necessary calls
* @param {string} resourceUrl Url of the resource that the chunk will be uploaded to
* @param {NodeJS.ReadableStream} openStream Stream of the file that will be uploaded
* @param {number} start Starting byte index of file that the chunk belongs to
* @param {number} end Ending byte index of file that the chunk belongs to
* @param {number} uploadFileSize Total size of the file in bytes that is being uploaded
* @param {boolean} isGzip Denotes if we are uploading a Gzip compressed stream
* @param {number} totalFileSize Original total size of the file that is being uploaded
* @returns if the chunk was successfully uploaded
*/
private async uploadChunk(
httpClientIndex: number,
resourceUrl: string,
openStream: () => NodeJS.ReadableStream,
start: number,
end: number,
uploadFileSize: number,
isGzip: boolean,
totalFileSize: number
): Promise<boolean> {
// prepare all the necessary headers before making any http call
const headers = getUploadHeaders(
'application/octet-stream',
true,
isGzip,
totalFileSize,
end - start + 1,
getContentRange(start, end, uploadFileSize)
)
const uploadChunkRequest = async (): Promise<IHttpClientResponse> => {
const client = this.uploadHttpManager.getClient(httpClientIndex)
return await client.sendStream('PUT', resourceUrl, openStream(), headers)
}
let retryCount = 0
const retryLimit = getRetryLimit()
// Increments the current retry count and then checks if the retry limit has been reached
// If there have been too many retries, fail so the download stops
const incrementAndCheckRetryLimit = (
response?: IHttpClientResponse
): boolean => {
retryCount++
if (retryCount > retryLimit) {
if (response) {
displayHttpDiagnostics(response)
}
core.info(
`Retry limit has been reached for chunk at offset ${start} to ${resourceUrl}`
)
return true
}
return false
}
const backOff = async (retryAfterValue?: number): Promise<void> => {
this.uploadHttpManager.disposeAndReplaceClient(httpClientIndex)
if (retryAfterValue) {
core.info(
`Backoff due to too many requests, retry #${retryCount}. Waiting for ${retryAfterValue} milliseconds before continuing the upload`
)
await sleep(retryAfterValue)
} else {
const backoffTime = getExponentialRetryTimeInMilliseconds(retryCount)
core.info(
`Exponential backoff for retry #${retryCount}. Waiting for ${backoffTime} milliseconds before continuing the upload at offset ${start}`
)
await sleep(backoffTime)
}
core.info(
`Finished backoff for retry #${retryCount}, continuing with upload`
)
return
}
// allow for failed chunks to be retried multiple times
while (retryCount <= retryLimit) {
let response: IHttpClientResponse
try {
response = await uploadChunkRequest()
} catch (error) {
// if an error is caught, it is usually indicative of a timeout so retry the upload
core.info(
`An error has been caught http-client index ${httpClientIndex}, retrying the upload`
)
// eslint-disable-next-line no-console
console.log(error)
if (incrementAndCheckRetryLimit()) {
return false
}
await backOff()
continue
}
// Always read the body of the response. There is potential for a resource leak if the body is not read which will
// result in the connection remaining open along with unintended consequences when trying to dispose of the client
await response.readBody()
if (isSuccessStatusCode(response.message.statusCode)) {
return true
} else if (isRetryableStatusCode(response.message.statusCode)) {
core.info(
`A ${response.message.statusCode} status code has been received, will attempt to retry the upload`
)
if (incrementAndCheckRetryLimit(response)) {
return false
}
isThrottledStatusCode(response.message.statusCode)
? await backOff(
tryGetRetryAfterValueTimeInMilliseconds(response.message.headers)
)
: await backOff()
} else {
core.error(
`Unexpected response. Unable to upload chunk to ${resourceUrl}`
)
displayHttpDiagnostics(response)
return false
}
}
return false
}
/**
* Updates the size of the artifact from -1 which was initially set when the container was first created for the artifact.
* Updating the size indicates that we are done uploading all the contents of the artifact
*/
async patchArtifactSize(size: number, artifactName: string): Promise<void> {
const resourceUrl = new URL(getArtifactUrl())
resourceUrl.searchParams.append('artifactName', artifactName)
const parameters: PatchArtifactSize = {Size: size}
const data: string = JSON.stringify(parameters, null, 2)
core.debug(`URL is ${resourceUrl.toString()}`)
// use the first client from the httpManager, `keep-alive` is not used so the connection will close immediately
const client = this.uploadHttpManager.getClient(0)
const headers = getUploadHeaders('application/json', false)
// Extra information to display when a particular HTTP code is returned
const customErrorMessages: Map<number, string> = new Map([
[
HttpCodes.NotFound,
`An Artifact with the name ${artifactName} was not found`
]
])
// TODO retry for all possible response codes, the artifact upload is pretty much complete so it at all costs we should try to finish this
const response = await retryHttpClientRequest(
'Finalize artifact upload',
async () => client.patch(resourceUrl.toString(), data, headers),
customErrorMessages
)
await response.readBody()
core.debug(
`Artifact ${artifactName} has been successfully uploaded, total size in bytes: ${size}`
)
}
}
interface UploadFileParameters {
file: string
resourceUrl: string
maxChunkSize: number
continueOnError: boolean
}
interface UploadFileResult {
isSuccess: boolean
successfulUploadSize: number
totalSize: number
}
@@ -0,0 +1,35 @@
export interface UploadOptions {
/**
* Indicates if the artifact upload should continue if file or chunk fails to upload from any error.
* If there is a error during upload, a partial artifact will always be associated and available for
* download at the end. The size reported will be the amount of storage that the user or org will be
* charged for the partial artifact. Defaults to true if not specified
*
* If set to false, and an error is encountered, all other uploads will stop and any files or chunks
* that were queued will not be attempted to be uploaded. The partial artifact available will only
* include files and chunks up until the failure
*
* If set to true and an error is encountered, the failed file will be skipped and ignored and all
* other queued files will be attempted to be uploaded. The partial artifact at the end will have all
* files with the exception of the problematic files(s)/chunks(s) that failed to upload
*
*/
continueOnError?: boolean
/**
* Duration after which artifact will expire in days.
*
* By default artifact expires after 90 days:
* https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#downloading-and-deleting-artifacts-after-a-workflow-run-is-complete
*
* Use this option to override the default expiry.
*
* Min value: 1
* Max value: 90 unless changed by repository setting
*
* If this is set to a greater value than the retention settings allowed, the retention on artifacts
* will be reduced to match the max value allowed on server, and the upload process will continue. An
* input of 0 assumes default retention setting.
*/
retentionDays?: number
}
@@ -0,0 +1,22 @@
export interface UploadResponse {
/**
* The name of the artifact that was uploaded
*/
artifactName: string
/**
* A list of all items that are meant to be uploaded as part of the artifact
*/
artifactItems: string[]
/**
* Total size of the artifact in bytes that was uploaded
*/
size: number
/**
* A list of items that were not uploaded as part of the artifact (includes queued items that were not uploaded if
* continueOnError is set to false). This is a subset of artifactItems.
*/
failedItems: string[]
}
@@ -0,0 +1,94 @@
import * as fs from 'fs'
import {debug} from '@actions/core'
import {join, normalize, resolve} from 'path'
import {checkArtifactFilePath} from './path-and-artifact-name-validation'
export interface UploadSpecification {
absoluteFilePath: string
uploadFilePath: string
}
/**
* Creates a specification that describes how each file that is part of the artifact will be uploaded
* @param artifactName the name of the artifact being uploaded. Used during upload to denote where the artifact is stored on the server
* @param rootDirectory an absolute file path that denotes the path that should be removed from the beginning of each artifact file
* @param artifactFiles a list of absolute file paths that denote what should be uploaded as part of the artifact
*/
export function getUploadSpecification(
artifactName: string,
rootDirectory: string,
artifactFiles: string[]
): UploadSpecification[] {
// artifact name was checked earlier on, no need to check again
const specifications: UploadSpecification[] = []
if (!fs.existsSync(rootDirectory)) {
throw new Error(`Provided rootDirectory ${rootDirectory} does not exist`)
}
if (!fs.lstatSync(rootDirectory).isDirectory()) {
throw new Error(
`Provided rootDirectory ${rootDirectory} is not a valid directory`
)
}
// Normalize and resolve, this allows for either absolute or relative paths to be used
rootDirectory = normalize(rootDirectory)
rootDirectory = resolve(rootDirectory)
/*
Example to demonstrate behavior
Input:
artifactName: my-artifact
rootDirectory: '/home/user/files/plz-upload'
artifactFiles: [
'/home/user/files/plz-upload/file1.txt',
'/home/user/files/plz-upload/file2.txt',
'/home/user/files/plz-upload/dir/file3.txt'
]
Output:
specifications: [
['/home/user/files/plz-upload/file1.txt', 'my-artifact/file1.txt'],
['/home/user/files/plz-upload/file1.txt', 'my-artifact/file2.txt'],
['/home/user/files/plz-upload/file1.txt', 'my-artifact/dir/file3.txt']
]
*/
for (let file of artifactFiles) {
if (!fs.existsSync(file)) {
throw new Error(`File ${file} does not exist`)
}
if (!fs.lstatSync(file).isDirectory()) {
// Normalize and resolve, this allows for either absolute or relative paths to be used
file = normalize(file)
file = resolve(file)
if (!file.startsWith(rootDirectory)) {
throw new Error(
`The rootDirectory: ${rootDirectory} is not a parent directory of the file: ${file}`
)
}
// Check for forbidden characters in file paths that will be rejected during upload
const uploadPath = file.replace(rootDirectory, '')
checkArtifactFilePath(uploadPath)
/*
uploadFilePath denotes where the file will be uploaded in the file container on the server. During a run, if multiple artifacts are uploaded, they will all
be saved in the same container. The artifact name is used as the root directory in the container to separate and distinguish uploaded artifacts
path.join handles all the following cases and would return 'artifact-name/file-to-upload.txt
join('artifact-name/', 'file-to-upload.txt')
join('artifact-name/', '/file-to-upload.txt')
join('artifact-name', 'file-to-upload.txt')
join('artifact-name', '/file-to-upload.txt')
*/
specifications.push({
absoluteFilePath: file,
uploadFilePath: join(artifactName, uploadPath)
})
} else {
// Directories are rejected by the server during upload
debug(`Removing ${file} from rawSearchResults because it is a directory`)
}
}
return specifications
}
+293
View File
@@ -0,0 +1,293 @@
import {debug, info, warning} from '@actions/core'
import {promises as fs} from 'fs'
import {HttpCodes, HttpClient} from '@actions/http-client'
import {BearerCredentialHandler} from '@actions/http-client/auth'
import {IHeaders, IHttpClientResponse} from '@actions/http-client/interfaces'
import {IncomingHttpHeaders} from 'http'
import {
getRuntimeToken,
getRuntimeUrl,
getWorkFlowRunId,
getRetryMultiplier,
getInitialRetryIntervalInMilliseconds
} from './config-variables'
/**
* Returns a retry time in milliseconds that exponentially gets larger
* depending on the amount of retries that have been attempted
*/
export function getExponentialRetryTimeInMilliseconds(
retryCount: number
): number {
if (retryCount < 0) {
throw new Error('RetryCount should not be negative')
} else if (retryCount === 0) {
return getInitialRetryIntervalInMilliseconds()
}
const minTime =
getInitialRetryIntervalInMilliseconds() * getRetryMultiplier() * retryCount
const maxTime = minTime * getRetryMultiplier()
// returns a random number between the minTime (inclusive) and the maxTime (exclusive)
return Math.trunc(Math.random() * (maxTime - minTime) + minTime)
}
/**
* Parses a env variable that is a number
*/
export function parseEnvNumber(key: string): number | undefined {
const value = Number(process.env[key])
if (Number.isNaN(value) || value < 0) {
return undefined
}
return value
}
/**
* Various utility functions to help with the necessary API calls
*/
export function getApiVersion(): string {
return '6.0-preview'
}
export function isSuccessStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return false
}
return statusCode >= 200 && statusCode < 300
}
export function isForbiddenStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return false
}
return statusCode === HttpCodes.Forbidden
}
export function isRetryableStatusCode(statusCode: number | undefined): boolean {
if (!statusCode) {
return false
}
const retryableStatusCodes = [
HttpCodes.BadGateway,
HttpCodes.GatewayTimeout,
HttpCodes.InternalServerError,
HttpCodes.ServiceUnavailable,
HttpCodes.TooManyRequests,
413 // Payload Too Large
]
return retryableStatusCodes.includes(statusCode)
}
export function isThrottledStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return false
}
return statusCode === HttpCodes.TooManyRequests
}
/**
* Attempts to get the retry-after value from a set of http headers. The retry time
* is originally denoted in seconds, so if present, it is converted to milliseconds
* @param headers all the headers received when making an http call
*/
export function tryGetRetryAfterValueTimeInMilliseconds(
headers: IncomingHttpHeaders
): number | undefined {
if (headers['retry-after']) {
const retryTime = Number(headers['retry-after'])
if (!isNaN(retryTime)) {
info(`Retry-After header is present with a value of ${retryTime}`)
return retryTime * 1000
}
info(
`Returned retry-after header value: ${retryTime} is non-numeric and cannot be used`
)
return undefined
}
info(
`No retry-after header was found. Dumping all headers for diagnostic purposes`
)
// eslint-disable-next-line no-console
console.log(headers)
return undefined
}
export function getContentRange(
start: number,
end: number,
total: number
): string {
// Format: `bytes start-end/fileSize
// start and end are inclusive
// For a 200 byte chunk starting at byte 0:
// Content-Range: bytes 0-199/200
return `bytes ${start}-${end}/${total}`
}
/**
* Sets all the necessary headers when downloading an artifact
* @param {string} contentType the type of content being uploaded
* @param {boolean} isKeepAlive is the same connection being used to make multiple calls
* @param {boolean} acceptGzip can we accept a gzip encoded response
* @param {string} acceptType the type of content that we can accept
* @returns appropriate headers to make a specific http call during artifact download
*/
export function getDownloadHeaders(
contentType: string,
isKeepAlive?: boolean,
acceptGzip?: boolean
): IHeaders {
const requestOptions: IHeaders = {}
if (contentType) {
requestOptions['Content-Type'] = contentType
}
if (isKeepAlive) {
requestOptions['Connection'] = 'Keep-Alive'
// keep alive for at least 10 seconds before closing the connection
requestOptions['Keep-Alive'] = '10'
}
if (acceptGzip) {
// if we are expecting a response with gzip encoding, it should be using an octet-stream in the accept header
requestOptions['Accept-Encoding'] = 'gzip'
requestOptions[
'Accept'
] = `application/octet-stream;api-version=${getApiVersion()}`
} else {
// default to application/json if we are not working with gzip content
requestOptions['Accept'] = `application/json;api-version=${getApiVersion()}`
}
return requestOptions
}
/**
* Sets all the necessary headers when uploading an artifact
* @param {string} contentType the type of content being uploaded
* @param {boolean} isKeepAlive is the same connection being used to make multiple calls
* @param {boolean} isGzip is the connection being used to upload GZip compressed content
* @param {number} uncompressedLength the original size of the content if something is being uploaded that has been compressed
* @param {number} contentLength the length of the content that is being uploaded
* @param {string} contentRange the range of the content that is being uploaded
* @returns appropriate headers to make a specific http call during artifact upload
*/
export function getUploadHeaders(
contentType: string,
isKeepAlive?: boolean,
isGzip?: boolean,
uncompressedLength?: number,
contentLength?: number,
contentRange?: string
): IHeaders {
const requestOptions: IHeaders = {}
requestOptions['Accept'] = `application/json;api-version=${getApiVersion()}`
if (contentType) {
requestOptions['Content-Type'] = contentType
}
if (isKeepAlive) {
requestOptions['Connection'] = 'Keep-Alive'
// keep alive for at least 10 seconds before closing the connection
requestOptions['Keep-Alive'] = '10'
}
if (isGzip) {
requestOptions['Content-Encoding'] = 'gzip'
requestOptions['x-tfs-filelength'] = uncompressedLength
}
if (contentLength) {
requestOptions['Content-Length'] = contentLength
}
if (contentRange) {
requestOptions['Content-Range'] = contentRange
}
return requestOptions
}
export function createHttpClient(userAgent: string): HttpClient {
return new HttpClient(userAgent, [
new BearerCredentialHandler(getRuntimeToken())
])
}
export function getArtifactUrl(): string {
const artifactUrl = `${getRuntimeUrl()}_apis/pipelines/workflows/${getWorkFlowRunId()}/artifacts?api-version=${getApiVersion()}`
debug(`Artifact Url: ${artifactUrl}`)
return artifactUrl
}
/**
* Uh oh! Something might have gone wrong during either upload or download. The IHtttpClientResponse object contains information
* about the http call that was made by the actions http client. This information might be useful to display for diagnostic purposes, but
* this entire object is really big and most of the information is not really useful. This function takes the response object and displays only
* the information that we want.
*
* Certain information such as the TLSSocket and the Readable state are not really useful for diagnostic purposes so they can be avoided.
* Other information such as the headers, the response code and message might be useful, so this is displayed.
*/
export function displayHttpDiagnostics(response: IHttpClientResponse): void {
info(
`##### Begin Diagnostic HTTP information #####
Status Code: ${response.message.statusCode}
Status Message: ${response.message.statusMessage}
Header Information: ${JSON.stringify(response.message.headers, undefined, 2)}
###### End Diagnostic HTTP information ######`
)
}
export async function createDirectoriesForArtifact(
directories: string[]
): Promise<void> {
for (const directory of directories) {
await fs.mkdir(directory, {
recursive: true
})
}
}
export async function createEmptyFilesForArtifact(
emptyFilesToCreate: string[]
): Promise<void> {
for (const filePath of emptyFilesToCreate) {
await (await fs.open(filePath, 'w')).close()
}
}
export async function getFileSize(filePath: string): Promise<number> {
const stats = await fs.stat(filePath)
debug(
`${filePath} size:(${stats.size}) blksize:(${stats.blksize}) blocks:(${stats.blocks})`
)
return stats.size
}
export async function rmFile(filePath: string): Promise<void> {
await fs.unlink(filePath)
}
export function getProperRetention(
retentionInput: number,
retentionSetting: string | undefined
): number {
if (retentionInput < 0) {
throw new Error('Invalid retention, minimum value is 1.')
}
let retention = retentionInput
if (retentionSetting) {
const maxRetention = parseInt(retentionSetting)
if (!isNaN(maxRetention) && maxRetention < retention) {
warning(
`Retention days is greater than the max value allowed by the repository setting, reduce retention to ${maxRetention} days`
)
retention = maxRetention
}
}
return retention
}
export async function sleep(milliseconds: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
+11
View File
@@ -0,0 +1,11 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"baseUrl": "./",
"outDir": "./lib",
"rootDir": "./src"
},
"include": [
"./src"
]
}
+9
View File
@@ -0,0 +1,9 @@
The MIT License (MIT)
Copyright 2019 GitHub
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+44
View File
@@ -0,0 +1,44 @@
# `@actions/cache`
> Functions necessary for caching dependencies and build outputs to improve workflow execution time.
See ["Caching dependencies to speed up workflows"](https://help.github.com/github/automating-your-workflow-with-github-actions/caching-dependencies-to-speed-up-workflows) for how caching works.
Note that GitHub will remove any cache entries that have not been accessed in over 7 days. There is no limit on the number of caches you can store, but the total size of all caches in a repository is limited to 10 GB. If you exceed this limit, GitHub will save your cache but will begin evicting caches until the total size is less than 10 GB.
## Usage
This package is used by the v2+ versions of our first party cache action. You can find an example implementation in the cache repo [here](https://github.com/actions/cache).
#### Save Cache
Saves a cache containing the files in `paths` using the `key` provided. The files would be compressed using zstandard compression algorithm if zstd is installed, otherwise gzip is used. Function returns the cache id if the cache was saved succesfully and throws an error if cache upload fails.
```js
const cache = require('@actions/cache');
const paths = [
'node_modules',
'packages/*/node_modules/'
]
const key = 'npm-foobar-d5ea0750'
const cacheId = await cache.saveCache(paths, key)
```
#### Restore Cache
Restores a cache based on `key` and `restoreKeys` to the `paths` provided. Function returns the cache key for cache hit and returns undefined if cache not found.
```js
const cache = require('@actions/cache');
const paths = [
'node_modules',
'packages/*/node_modules/'
]
const key = 'npm-foobar-d5ea0750'
const restoreKeys = [
'npm-foobar-',
'npm-'
]
const cacheKey = await cache.restoreCache(paths, key, restoreKeys)
```
+58
View File
@@ -0,0 +1,58 @@
# @actions/cache Releases
### 0.1.0
- Initial release
### 0.2.0
- Fixes issues with the zstd compression algorithm on Windows and Ubuntu 16.04 [#469](https://github.com/actions/toolkit/pull/469)
### 0.2.1
- Fix to await async function getCompressionMethod
### 1.0.0
- Downloads Azure-hosted caches using the Azure SDK for speed and reliability
- Displays download progress
- Includes changes that break compatibility with earlier versions, including:
- `retry`, `retryTypedResponse`, and `retryHttpClientResponse` moved from `cacheHttpClient` to `requestUtils`
### 1.0.1
- Fix bug in downloading large files (> 2 GBs) with the Azure SDK
### 1.0.2
- Use posix archive format to add support for some tools
### 1.0.3
- Use http-client v1.0.9
- Fixes error handling so retries are not attempted on non-retryable errors (409 Conflict, for example)
- Adds 5 second delay between retry attempts
### 1.0.4
- Use @actions/core v1.2.6
- Fixes uploadChunk to throw an error if any unsuccessful response code is received
### 1.0.5
- Fix to ensure Windows cache paths get resolved correctly
### 1.0.6
- Make caching more verbose [#650](https://github.com/actions/toolkit/pull/650)
- Use GNU tar on macOS if available [#701](https://github.com/actions/toolkit/pull/701)
### 1.0.7
- Fixes permissions issue extracting archives with GNU tar on macOS ([issue](https://github.com/actions/cache/issues/527))
### 1.0.8
- Increase the allowed artifact cache size from 5GB to 10GB ([issue](https://github.com/actions/cache/discussions/497))
### 1.0.9
- Use @azure/ms-rest-js v2.6.0
- Use @azure/storage-blob v12.8.0
### 1.0.10
- Update `lockfileVersion` to `v2` in `package-lock.json [#1022](https://github.com/actions/toolkit/pull/1022)
### 1.0.11
- Fix file downloads > 2GB([issue](https://github.com/actions/cache/issues/773))
### 2.0.0
- Added support to check if Actions cache service feature is available or not [#1028](https://github.com/actions/toolkit/pull/1028)
+5
View File
@@ -0,0 +1,5 @@
name: 'Set env variables'
description: 'Sets certain env variables so that e2e restore and save cache can be tested in a shell'
runs:
using: 'node12'
main: 'index.js'
+1
View File
@@ -0,0 +1 @@
hello world
+14
View File
@@ -0,0 +1,14 @@
// Certain env variables are not set by default in a shell context and are only available in a node context from a running action
// In order to be able to restore and save cache e2e in a shell when running CI tests, we need these env variables set
const fs = require('fs');
const os = require('os');
const filePath = process.env[`GITHUB_ENV`]
fs.appendFileSync(filePath, `ACTIONS_RUNTIME_TOKEN=${process.env.ACTIONS_RUNTIME_TOKEN}${os.EOL}`, {
encoding: 'utf8'
})
fs.appendFileSync(filePath, `ACTIONS_CACHE_URL=${process.env.ACTIONS_CACHE_URL}${os.EOL}`, {
encoding: 'utf8'
})
fs.appendFileSync(filePath, `GITHUB_RUN_ID=${process.env.GITHUB_RUN_ID}${os.EOL}`, {
encoding: 'utf8'
})
+14
View File
@@ -0,0 +1,14 @@
import * as cache from '../src/cache'
test('isFeatureAvailable returns true if server url is set', () => {
try {
process.env['ACTIONS_CACHE_URL'] = 'http://cache.com'
expect(cache.isFeatureAvailable()).toBe(true)
} finally {
delete process.env['ACTIONS_CACHE_URL']
}
})
test('isFeatureAvailable returns false if server url is not set', () => {
expect(cache.isFeatureAvailable()).toBe(false)
})
+141
View File
@@ -0,0 +1,141 @@
import {downloadCache, getCacheVersion} from '../src/internal/cacheHttpClient'
import {CompressionMethod} from '../src/internal/constants'
import * as downloadUtils from '../src/internal/downloadUtils'
import {DownloadOptions, getDownloadOptions} from '../src/options'
jest.mock('../src/internal/downloadUtils')
test('getCacheVersion with one path returns version', async () => {
const paths = ['node_modules']
const result = getCacheVersion(paths)
expect(result).toEqual(
'b3e0c6cb5ecf32614eeb2997d905b9c297046d7cbf69062698f25b14b4cb0985'
)
})
test('getCacheVersion with multiple paths returns version', async () => {
const paths = ['node_modules', 'dist']
const result = getCacheVersion(paths)
expect(result).toEqual(
'165c3053bc646bf0d4fac17b1f5731caca6fe38e0e464715c0c3c6b6318bf436'
)
})
test('getCacheVersion with zstd compression returns version', async () => {
const paths = ['node_modules']
const result = getCacheVersion(paths, CompressionMethod.Zstd)
expect(result).toEqual(
'273877e14fd65d270b87a198edbfa2db5a43de567c9a548d2a2505b408befe24'
)
})
test('getCacheVersion with gzip compression does not change vesion', async () => {
const paths = ['node_modules']
const result = getCacheVersion(paths, CompressionMethod.Gzip)
expect(result).toEqual(
'b3e0c6cb5ecf32614eeb2997d905b9c297046d7cbf69062698f25b14b4cb0985'
)
})
test('downloadCache uses http-client for non-Azure URLs', async () => {
const downloadCacheHttpClientMock = jest.spyOn(
downloadUtils,
'downloadCacheHttpClient'
)
const downloadCacheStorageSDKMock = jest.spyOn(
downloadUtils,
'downloadCacheStorageSDK'
)
const archiveLocation = 'http://www.actionscache.test/download'
const archivePath = '/foo/bar'
await downloadCache(archiveLocation, archivePath)
expect(downloadCacheHttpClientMock).toHaveBeenCalledTimes(1)
expect(downloadCacheHttpClientMock).toHaveBeenCalledWith(
archiveLocation,
archivePath
)
expect(downloadCacheStorageSDKMock).toHaveBeenCalledTimes(0)
})
test('downloadCache uses storage SDK for Azure storage URLs', async () => {
const downloadCacheHttpClientMock = jest.spyOn(
downloadUtils,
'downloadCacheHttpClient'
)
const downloadCacheStorageSDKMock = jest.spyOn(
downloadUtils,
'downloadCacheStorageSDK'
)
const archiveLocation = 'http://foo.blob.core.windows.net/bar/baz'
const archivePath = '/foo/bar'
await downloadCache(archiveLocation, archivePath)
expect(downloadCacheStorageSDKMock).toHaveBeenCalledTimes(1)
expect(downloadCacheStorageSDKMock).toHaveBeenCalledWith(
archiveLocation,
archivePath,
getDownloadOptions()
)
expect(downloadCacheHttpClientMock).toHaveBeenCalledTimes(0)
})
test('downloadCache passes options to download methods', async () => {
const downloadCacheHttpClientMock = jest.spyOn(
downloadUtils,
'downloadCacheHttpClient'
)
const downloadCacheStorageSDKMock = jest.spyOn(
downloadUtils,
'downloadCacheStorageSDK'
)
const archiveLocation = 'http://foo.blob.core.windows.net/bar/baz'
const archivePath = '/foo/bar'
const options: DownloadOptions = {downloadConcurrency: 4}
await downloadCache(archiveLocation, archivePath, options)
expect(downloadCacheStorageSDKMock).toHaveBeenCalledTimes(1)
expect(downloadCacheStorageSDKMock).toHaveBeenCalled()
expect(downloadCacheStorageSDKMock).toHaveBeenCalledWith(
archiveLocation,
archivePath,
getDownloadOptions(options)
)
expect(downloadCacheHttpClientMock).toHaveBeenCalledTimes(0)
})
test('downloadCache uses http-client when overridden', async () => {
const downloadCacheHttpClientMock = jest.spyOn(
downloadUtils,
'downloadCacheHttpClient'
)
const downloadCacheStorageSDKMock = jest.spyOn(
downloadUtils,
'downloadCacheStorageSDK'
)
const archiveLocation = 'http://foo.blob.core.windows.net/bar/baz'
const archivePath = '/foo/bar'
const options: DownloadOptions = {useAzureSdk: false}
await downloadCache(archiveLocation, archivePath, options)
expect(downloadCacheHttpClientMock).toHaveBeenCalledTimes(1)
expect(downloadCacheHttpClientMock).toHaveBeenCalledWith(
archiveLocation,
archivePath
)
expect(downloadCacheStorageSDKMock).toHaveBeenCalledTimes(0)
})
+34
View File
@@ -0,0 +1,34 @@
import {promises as fs} from 'fs'
import * as path from 'path'
import * as cacheUtils from '../src/internal/cacheUtils'
test('getArchiveFileSizeInBytes returns file size', () => {
const filePath = path.join(__dirname, '__fixtures__', 'helloWorld.txt')
const size = cacheUtils.getArchiveFileSizeInBytes(filePath)
expect(size).toBe(11)
})
test('unlinkFile unlinks file', async () => {
const testDirectory = await fs.mkdtemp('unlinkFileTest')
const testFile = path.join(testDirectory, 'test.txt')
await fs.writeFile(testFile, 'hello world')
await expect(fs.stat(testFile)).resolves.not.toThrow()
await cacheUtils.unlinkFile(testFile)
// This should throw as testFile should not exist
await expect(fs.stat(testFile)).rejects.toThrow()
await fs.rmdir(testDirectory)
})
test('assertDefined throws if undefined', () => {
expect(() => cacheUtils.assertDefined('test', undefined)).toThrowError()
})
test('assertDefined returns value', () => {
expect(cacheUtils.assertDefined('test', 5)).toBe(5)
})
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# Validate args
prefix="$1"
if [ -z "$prefix" ]; then
echo "Must supply prefix argument"
exit 1
fi
path="$2"
if [ -z "$path" ]; then
echo "Must supply path argument"
exit 1
fi
mkdir -p $path
echo "$prefix $GITHUB_RUN_ID" > $path/test-file.txt
+160
View File
@@ -0,0 +1,160 @@
import * as core from '@actions/core'
import {DownloadProgress} from '../src/internal/downloadUtils'
test('download progress tracked correctly', () => {
const progress = new DownloadProgress(1000)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(0)
expect(progress.segmentIndex).toBe(0)
expect(progress.segmentOffset).toBe(0)
expect(progress.segmentSize).toBe(0)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(0)
expect(progress.isDone()).toBe(false)
progress.nextSegment(500)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(0)
expect(progress.segmentIndex).toBe(1)
expect(progress.segmentOffset).toBe(0)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(0)
expect(progress.isDone()).toBe(false)
progress.setReceivedBytes(250)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(250)
expect(progress.segmentIndex).toBe(1)
expect(progress.segmentOffset).toBe(0)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(250)
expect(progress.isDone()).toBe(false)
progress.setReceivedBytes(500)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(500)
expect(progress.segmentIndex).toBe(1)
expect(progress.segmentOffset).toBe(0)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(500)
expect(progress.isDone()).toBe(false)
progress.nextSegment(500)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(0)
expect(progress.segmentIndex).toBe(2)
expect(progress.segmentOffset).toBe(500)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(500)
expect(progress.isDone()).toBe(false)
progress.setReceivedBytes(250)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(250)
expect(progress.segmentIndex).toBe(2)
expect(progress.segmentOffset).toBe(500)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(750)
expect(progress.isDone()).toBe(false)
progress.setReceivedBytes(500)
expect(progress.contentLength).toBe(1000)
expect(progress.receivedBytes).toBe(500)
expect(progress.segmentIndex).toBe(2)
expect(progress.segmentOffset).toBe(500)
expect(progress.segmentSize).toBe(500)
expect(progress.displayedComplete).toBe(false)
expect(progress.timeoutHandle).toBeUndefined()
expect(progress.getTransferredBytes()).toBe(1000)
expect(progress.isDone()).toBe(true)
})
test('display timer works correctly', done => {
const progress = new DownloadProgress(1000)
const infoMock = jest.spyOn(core, 'info')
infoMock.mockImplementation(() => {})
const check = (): void => {
expect(infoMock).toHaveBeenLastCalledWith(
expect.stringContaining('Received 500 of 1000')
)
}
// Validate no further updates are displayed after stopping the timer.
const test2 = (): void => {
check()
expect(progress.timeoutHandle).toBeUndefined()
done()
}
// Validate the progress is displayed, stop the timer, and call test2.
const test1 = (): void => {
check()
progress.stopDisplayTimer()
progress.setReceivedBytes(1000)
setTimeout(() => test2(), 500)
}
// Start the timer, update the received bytes, and call test1.
const start = (): void => {
progress.startDisplayTimer(10)
expect(progress.timeoutHandle).toBeDefined()
progress.setReceivedBytes(500)
setTimeout(() => test1(), 500)
}
start()
})
test('display does not print completed line twice', () => {
const progress = new DownloadProgress(1000)
const infoMock = jest.spyOn(core, 'info')
infoMock.mockImplementation(() => {})
progress.display()
expect(progress.displayedComplete).toBe(false)
expect(infoMock).toHaveBeenCalledTimes(1)
progress.nextSegment(1000)
progress.setReceivedBytes(500)
progress.display()
expect(progress.displayedComplete).toBe(false)
expect(infoMock).toHaveBeenCalledTimes(2)
progress.setReceivedBytes(1000)
progress.display()
expect(progress.displayedComplete).toBe(true)
expect(infoMock).toHaveBeenCalledTimes(3)
progress.display()
expect(progress.displayedComplete).toBe(true)
expect(infoMock).toHaveBeenCalledTimes(3)
})
+54
View File
@@ -0,0 +1,54 @@
import {
DownloadOptions,
UploadOptions,
getDownloadOptions,
getUploadOptions
} from '../src/options'
const useAzureSdk = true
const downloadConcurrency = 8
const timeoutInMs = 30000
const uploadConcurrency = 4
const uploadChunkSize = 32 * 1024 * 1024
test('getDownloadOptions sets defaults', async () => {
const actualOptions = getDownloadOptions()
expect(actualOptions).toEqual({
useAzureSdk,
downloadConcurrency,
timeoutInMs
})
})
test('getDownloadOptions overrides all settings', async () => {
const expectedOptions: DownloadOptions = {
useAzureSdk: false,
downloadConcurrency: 14,
timeoutInMs: 20000
}
const actualOptions = getDownloadOptions(expectedOptions)
expect(actualOptions).toEqual(expectedOptions)
})
test('getUploadOptions sets defaults', async () => {
const actualOptions = getUploadOptions()
expect(actualOptions).toEqual({
uploadConcurrency,
uploadChunkSize
})
})
test('getUploadOptions overrides all settings', async () => {
const expectedOptions: UploadOptions = {
uploadConcurrency: 2,
uploadChunkSize: 16 * 1024 * 1024
}
const actualOptions = getUploadOptions(expectedOptions)
expect(actualOptions).toEqual(expectedOptions)
})
+147
View File
@@ -0,0 +1,147 @@
import {retry} from '../src/internal/requestUtils'
import {HttpClientError} from '@actions/http-client'
interface ITestResponse {
statusCode: number
result: string | null
error: Error | null
}
function TestResponse(
action: number | Error,
result: string | null = null
): ITestResponse {
if (action instanceof Error) {
return {
statusCode: -1,
result,
error: action
}
} else {
return {
statusCode: action,
result,
error: null
}
}
}
async function handleResponse(
response: ITestResponse | undefined
): Promise<ITestResponse> {
if (!response) {
fail('Retry method called too many times')
}
if (response.error) {
throw response.error
} else {
return Promise.resolve(response)
}
}
async function testRetryExpectingResult(
responses: ITestResponse[],
expectedResult: string | null
): Promise<void> {
responses = responses.reverse() // Reverse responses since we pop from end
const actualResult = await retry(
'test',
async () => handleResponse(responses.pop()),
(response: ITestResponse) => response.statusCode,
2, // maxAttempts
0 // delay
)
expect(actualResult.result).toEqual(expectedResult)
}
async function testRetryConvertingErrorToResult(
responses: ITestResponse[],
expectedStatus: number,
expectedResult: string | null
): Promise<void> {
responses = responses.reverse() // Reverse responses since we pop from end
const actualResult = await retry(
'test',
async () => handleResponse(responses.pop()),
(response: ITestResponse) => response.statusCode,
2, // maxAttempts
0, // delay
(e: Error) => {
if (e instanceof HttpClientError) {
return {
statusCode: e.statusCode,
result: null,
error: null
}
}
}
)
expect(actualResult.statusCode).toEqual(expectedStatus)
expect(actualResult.result).toEqual(expectedResult)
}
async function testRetryExpectingError(
responses: ITestResponse[]
): Promise<void> {
responses = responses.reverse() // Reverse responses since we pop from end
expect(
retry(
'test',
async () => handleResponse(responses.pop()),
(response: ITestResponse) => response.statusCode,
2, // maxAttempts,
0 // delay
)
).rejects.toBeInstanceOf(Error)
}
test('retry works on successful response', async () => {
await testRetryExpectingResult([TestResponse(200, 'Ok')], 'Ok')
})
test('retry works after retryable status code', async () => {
await testRetryExpectingResult(
[TestResponse(503), TestResponse(200, 'Ok')],
'Ok'
)
})
test('retry fails after exhausting retries', async () => {
await testRetryExpectingError([
TestResponse(503),
TestResponse(503),
TestResponse(200, 'Ok')
])
})
test('retry fails after non-retryable status code', async () => {
await testRetryExpectingError([TestResponse(500), TestResponse(200, 'Ok')])
})
test('retry works after error', async () => {
await testRetryExpectingResult(
[TestResponse(new Error('Test error')), TestResponse(200, 'Ok')],
'Ok'
)
})
test('retry returns after client error', async () => {
await testRetryExpectingResult(
[TestResponse(400), TestResponse(200, 'Ok')],
null
)
})
test('retry converts errors to response object', async () => {
await testRetryConvertingErrorToResult(
[TestResponse(new HttpClientError('Test error', 409))],
409,
null
)
})
+271
View File
@@ -0,0 +1,271 @@
import * as core from '@actions/core'
import * as path from 'path'
import {restoreCache} from '../src/cache'
import * as cacheHttpClient from '../src/internal/cacheHttpClient'
import * as cacheUtils from '../src/internal/cacheUtils'
import {CacheFilename, CompressionMethod} from '../src/internal/constants'
import {ArtifactCacheEntry} from '../src/internal/contracts'
import * as tar from '../src/internal/tar'
jest.mock('../src/internal/cacheHttpClient')
jest.mock('../src/internal/cacheUtils')
jest.mock('../src/internal/tar')
beforeAll(() => {
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
jest.spyOn(cacheUtils, 'getCacheFileName').mockImplementation(cm => {
const actualUtils = jest.requireActual('../src/internal/cacheUtils')
return actualUtils.getCacheFileName(cm)
})
})
test('restore with no path should fail', async () => {
const paths: string[] = []
const key = 'node-test'
await expect(restoreCache(paths, key)).rejects.toThrowError(
`Path Validation Error: At least one directory or file path is required`
)
})
test('restore with too many keys should fail', async () => {
const paths = ['node_modules']
const key = 'node-test'
const restoreKeys = [...Array(20).keys()].map(x => x.toString())
await expect(restoreCache(paths, key, restoreKeys)).rejects.toThrowError(
`Key Validation Error: Keys are limited to a maximum of 10.`
)
})
test('restore with large key should fail', async () => {
const paths = ['node_modules']
const key = 'foo'.repeat(512) // Over the 512 character limit
await expect(restoreCache(paths, key)).rejects.toThrowError(
`Key Validation Error: ${key} cannot be larger than 512 characters.`
)
})
test('restore with invalid key should fail', async () => {
const paths = ['node_modules']
const key = 'comma,comma'
await expect(restoreCache(paths, key)).rejects.toThrowError(
`Key Validation Error: ${key} cannot contain commas.`
)
})
test('restore with no cache found', async () => {
const paths = ['node_modules']
const key = 'node-test'
jest.spyOn(cacheHttpClient, 'getCacheEntry').mockImplementation(async () => {
return Promise.resolve(null)
})
const cacheKey = await restoreCache(paths, key)
expect(cacheKey).toBe(undefined)
})
test('restore with server error should fail', async () => {
const paths = ['node_modules']
const key = 'node-test'
jest.spyOn(cacheHttpClient, 'getCacheEntry').mockImplementation(() => {
throw new Error('HTTP Error Occurred')
})
await expect(restoreCache(paths, key)).rejects.toThrowError(
'HTTP Error Occurred'
)
})
test('restore with restore keys and no cache found', async () => {
const paths = ['node_modules']
const key = 'node-test'
const restoreKey = 'node-'
jest.spyOn(cacheHttpClient, 'getCacheEntry').mockImplementation(async () => {
return Promise.resolve(null)
})
const cacheKey = await restoreCache(paths, key, [restoreKey])
expect(cacheKey).toBe(undefined)
})
test('restore with gzip compressed cache found', async () => {
const paths = ['node_modules']
const key = 'node-test'
const cacheEntry: ArtifactCacheEntry = {
cacheKey: key,
scope: 'refs/heads/main',
archiveLocation: 'www.actionscache.test/download'
}
const getCacheMock = jest.spyOn(cacheHttpClient, 'getCacheEntry')
getCacheMock.mockImplementation(async () => {
return Promise.resolve(cacheEntry)
})
const tempPath = '/foo/bar'
const createTempDirectoryMock = jest.spyOn(cacheUtils, 'createTempDirectory')
createTempDirectoryMock.mockImplementation(async () => {
return Promise.resolve(tempPath)
})
const archivePath = path.join(tempPath, CacheFilename.Gzip)
const downloadCacheMock = jest.spyOn(cacheHttpClient, 'downloadCache')
const fileSize = 142
const getArchiveFileSizeInBytesMock = jest
.spyOn(cacheUtils, 'getArchiveFileSizeInBytes')
.mockReturnValue(fileSize)
const extractTarMock = jest.spyOn(tar, 'extractTar')
const unlinkFileMock = jest.spyOn(cacheUtils, 'unlinkFile')
const compression = CompressionMethod.Gzip
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValue(Promise.resolve(compression))
const cacheKey = await restoreCache(paths, key)
expect(cacheKey).toBe(key)
expect(getCacheMock).toHaveBeenCalledWith([key], paths, {
compressionMethod: compression
})
expect(createTempDirectoryMock).toHaveBeenCalledTimes(1)
expect(downloadCacheMock).toHaveBeenCalledWith(
cacheEntry.archiveLocation,
archivePath,
undefined
)
expect(getArchiveFileSizeInBytesMock).toHaveBeenCalledWith(archivePath)
expect(extractTarMock).toHaveBeenCalledTimes(1)
expect(extractTarMock).toHaveBeenCalledWith(archivePath, compression)
expect(unlinkFileMock).toHaveBeenCalledTimes(1)
expect(unlinkFileMock).toHaveBeenCalledWith(archivePath)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
test('restore with zstd compressed cache found', async () => {
const paths = ['node_modules']
const key = 'node-test'
const infoMock = jest.spyOn(core, 'info')
const cacheEntry: ArtifactCacheEntry = {
cacheKey: key,
scope: 'refs/heads/main',
archiveLocation: 'www.actionscache.test/download'
}
const getCacheMock = jest.spyOn(cacheHttpClient, 'getCacheEntry')
getCacheMock.mockImplementation(async () => {
return Promise.resolve(cacheEntry)
})
const tempPath = '/foo/bar'
const createTempDirectoryMock = jest.spyOn(cacheUtils, 'createTempDirectory')
createTempDirectoryMock.mockImplementation(async () => {
return Promise.resolve(tempPath)
})
const archivePath = path.join(tempPath, CacheFilename.Zstd)
const downloadCacheMock = jest.spyOn(cacheHttpClient, 'downloadCache')
const fileSize = 62915000
const getArchiveFileSizeInBytesMock = jest
.spyOn(cacheUtils, 'getArchiveFileSizeInBytes')
.mockReturnValue(fileSize)
const extractTarMock = jest.spyOn(tar, 'extractTar')
const compression = CompressionMethod.Zstd
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValue(Promise.resolve(compression))
const cacheKey = await restoreCache(paths, key)
expect(cacheKey).toBe(key)
expect(getCacheMock).toHaveBeenCalledWith([key], paths, {
compressionMethod: compression
})
expect(createTempDirectoryMock).toHaveBeenCalledTimes(1)
expect(downloadCacheMock).toHaveBeenCalledWith(
cacheEntry.archiveLocation,
archivePath,
undefined
)
expect(getArchiveFileSizeInBytesMock).toHaveBeenCalledWith(archivePath)
expect(infoMock).toHaveBeenCalledWith(`Cache Size: ~60 MB (62915000 B)`)
expect(extractTarMock).toHaveBeenCalledTimes(1)
expect(extractTarMock).toHaveBeenCalledWith(archivePath, compression)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
test('restore with cache found for restore key', async () => {
const paths = ['node_modules']
const key = 'node-test'
const restoreKey = 'node-'
const infoMock = jest.spyOn(core, 'info')
const cacheEntry: ArtifactCacheEntry = {
cacheKey: restoreKey,
scope: 'refs/heads/main',
archiveLocation: 'www.actionscache.test/download'
}
const getCacheMock = jest.spyOn(cacheHttpClient, 'getCacheEntry')
getCacheMock.mockImplementation(async () => {
return Promise.resolve(cacheEntry)
})
const tempPath = '/foo/bar'
const createTempDirectoryMock = jest.spyOn(cacheUtils, 'createTempDirectory')
createTempDirectoryMock.mockImplementation(async () => {
return Promise.resolve(tempPath)
})
const archivePath = path.join(tempPath, CacheFilename.Zstd)
const downloadCacheMock = jest.spyOn(cacheHttpClient, 'downloadCache')
const fileSize = 142
const getArchiveFileSizeInBytesMock = jest
.spyOn(cacheUtils, 'getArchiveFileSizeInBytes')
.mockReturnValue(fileSize)
const extractTarMock = jest.spyOn(tar, 'extractTar')
const compression = CompressionMethod.Zstd
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValue(Promise.resolve(compression))
const cacheKey = await restoreCache(paths, key, [restoreKey])
expect(cacheKey).toBe(restoreKey)
expect(getCacheMock).toHaveBeenCalledWith([key, restoreKey], paths, {
compressionMethod: compression
})
expect(createTempDirectoryMock).toHaveBeenCalledTimes(1)
expect(downloadCacheMock).toHaveBeenCalledWith(
cacheEntry.archiveLocation,
archivePath,
undefined
)
expect(getArchiveFileSizeInBytesMock).toHaveBeenCalledWith(archivePath)
expect(infoMock).toHaveBeenCalledWith(`Cache Size: ~0 MB (142 B)`)
expect(extractTarMock).toHaveBeenCalledTimes(1)
expect(extractTarMock).toHaveBeenCalledWith(archivePath, compression)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
+188
View File
@@ -0,0 +1,188 @@
import * as core from '@actions/core'
import * as path from 'path'
import {saveCache} from '../src/cache'
import * as cacheHttpClient from '../src/internal/cacheHttpClient'
import * as cacheUtils from '../src/internal/cacheUtils'
import {CacheFilename, CompressionMethod} from '../src/internal/constants'
import * as tar from '../src/internal/tar'
jest.mock('../src/internal/cacheHttpClient')
jest.mock('../src/internal/cacheUtils')
jest.mock('../src/internal/tar')
beforeAll(() => {
jest.spyOn(console, 'log').mockImplementation(() => {})
jest.spyOn(core, 'debug').mockImplementation(() => {})
jest.spyOn(core, 'info').mockImplementation(() => {})
jest.spyOn(core, 'warning').mockImplementation(() => {})
jest.spyOn(core, 'error').mockImplementation(() => {})
jest.spyOn(cacheUtils, 'getCacheFileName').mockImplementation(cm => {
const actualUtils = jest.requireActual('../src/internal/cacheUtils')
return actualUtils.getCacheFileName(cm)
})
jest.spyOn(cacheUtils, 'resolvePaths').mockImplementation(async filePaths => {
return filePaths.map(x => path.resolve(x))
})
jest.spyOn(cacheUtils, 'createTempDirectory').mockImplementation(async () => {
return Promise.resolve('/foo/bar')
})
})
test('save with missing input should fail', async () => {
const paths: string[] = []
const primaryKey = 'Linux-node-bb828da54c148048dd17899ba9fda624811cfb43'
await expect(saveCache(paths, primaryKey)).rejects.toThrowError(
`Path Validation Error: At least one directory or file path is required`
)
})
test('save with large cache outputs should fail', async () => {
const filePath = 'node_modules'
const primaryKey = 'Linux-node-bb828da54c148048dd17899ba9fda624811cfb43'
const cachePaths = [path.resolve(filePath)]
const createTarMock = jest.spyOn(tar, 'createTar')
const cacheSize = 11 * 1024 * 1024 * 1024 //~11GB, over the 10GB limit
jest
.spyOn(cacheUtils, 'getArchiveFileSizeInBytes')
.mockReturnValueOnce(cacheSize)
const compression = CompressionMethod.Gzip
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValueOnce(Promise.resolve(compression))
await expect(saveCache([filePath], primaryKey)).rejects.toThrowError(
'Cache size of ~11264 MB (11811160064 B) is over the 10GB limit, not saving cache.'
)
const archiveFolder = '/foo/bar'
expect(createTarMock).toHaveBeenCalledTimes(1)
expect(createTarMock).toHaveBeenCalledWith(
archiveFolder,
cachePaths,
compression
)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
test('save with reserve cache failure should fail', async () => {
const paths = ['node_modules']
const primaryKey = 'Linux-node-bb828da54c148048dd17899ba9fda624811cfb43'
const reserveCacheMock = jest
.spyOn(cacheHttpClient, 'reserveCache')
.mockImplementation(async () => {
return -1
})
const createTarMock = jest.spyOn(tar, 'createTar')
const saveCacheMock = jest.spyOn(cacheHttpClient, 'saveCache')
const compression = CompressionMethod.Zstd
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValueOnce(Promise.resolve(compression))
await expect(saveCache(paths, primaryKey)).rejects.toThrowError(
`Unable to reserve cache with key ${primaryKey}, another job may be creating this cache.`
)
expect(reserveCacheMock).toHaveBeenCalledTimes(1)
expect(reserveCacheMock).toHaveBeenCalledWith(primaryKey, paths, {
compressionMethod: compression
})
expect(createTarMock).toHaveBeenCalledTimes(0)
expect(saveCacheMock).toHaveBeenCalledTimes(0)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
test('save with server error should fail', async () => {
const filePath = 'node_modules'
const primaryKey = 'Linux-node-bb828da54c148048dd17899ba9fda624811cfb43'
const cachePaths = [path.resolve(filePath)]
const cacheId = 4
const reserveCacheMock = jest
.spyOn(cacheHttpClient, 'reserveCache')
.mockImplementation(async () => {
return cacheId
})
const createTarMock = jest.spyOn(tar, 'createTar')
const saveCacheMock = jest
.spyOn(cacheHttpClient, 'saveCache')
.mockImplementationOnce(() => {
throw new Error('HTTP Error Occurred')
})
const compression = CompressionMethod.Zstd
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValueOnce(Promise.resolve(compression))
await expect(saveCache([filePath], primaryKey)).rejects.toThrowError(
'HTTP Error Occurred'
)
expect(reserveCacheMock).toHaveBeenCalledTimes(1)
expect(reserveCacheMock).toHaveBeenCalledWith(primaryKey, [filePath], {
compressionMethod: compression
})
const archiveFolder = '/foo/bar'
const archiveFile = path.join(archiveFolder, CacheFilename.Zstd)
expect(createTarMock).toHaveBeenCalledTimes(1)
expect(createTarMock).toHaveBeenCalledWith(
archiveFolder,
cachePaths,
compression
)
expect(saveCacheMock).toHaveBeenCalledTimes(1)
expect(saveCacheMock).toHaveBeenCalledWith(cacheId, archiveFile, undefined)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
test('save with valid inputs uploads a cache', async () => {
const filePath = 'node_modules'
const primaryKey = 'Linux-node-bb828da54c148048dd17899ba9fda624811cfb43'
const cachePaths = [path.resolve(filePath)]
const cacheId = 4
const reserveCacheMock = jest
.spyOn(cacheHttpClient, 'reserveCache')
.mockImplementation(async () => {
return cacheId
})
const createTarMock = jest.spyOn(tar, 'createTar')
const saveCacheMock = jest.spyOn(cacheHttpClient, 'saveCache')
const compression = CompressionMethod.Zstd
const getCompressionMock = jest
.spyOn(cacheUtils, 'getCompressionMethod')
.mockReturnValue(Promise.resolve(compression))
await saveCache([filePath], primaryKey)
expect(reserveCacheMock).toHaveBeenCalledTimes(1)
expect(reserveCacheMock).toHaveBeenCalledWith(primaryKey, [filePath], {
compressionMethod: compression
})
const archiveFolder = '/foo/bar'
const archiveFile = path.join(archiveFolder, CacheFilename.Zstd)
expect(createTarMock).toHaveBeenCalledTimes(1)
expect(createTarMock).toHaveBeenCalledWith(
archiveFolder,
cachePaths,
compression
)
expect(saveCacheMock).toHaveBeenCalledTimes(1)
expect(saveCacheMock).toHaveBeenCalledWith(cacheId, archiveFile, undefined)
expect(getCompressionMock).toHaveBeenCalledTimes(1)
})
+267
View File
@@ -0,0 +1,267 @@
import * as exec from '@actions/exec'
import * as io from '@actions/io'
import * as path from 'path'
import {CacheFilename, CompressionMethod} from '../src/internal/constants'
import * as tar from '../src/internal/tar'
import * as utils from '../src/internal/cacheUtils'
// eslint-disable-next-line @typescript-eslint/no-require-imports
import fs = require('fs')
jest.mock('@actions/exec')
jest.mock('@actions/io')
const IS_WINDOWS = process.platform === 'win32'
const IS_MAC = process.platform === 'darwin'
const defaultTarPath = process.platform === 'darwin' ? 'gtar' : 'tar'
function getTempDir(): string {
return path.join(__dirname, '_temp', 'tar')
}
beforeAll(async () => {
jest.spyOn(io, 'which').mockImplementation(async tool => {
return tool
})
process.env['GITHUB_WORKSPACE'] = process.cwd()
await jest.requireActual('@actions/io').rmRF(getTempDir())
})
afterAll(async () => {
delete process.env['GITHUB_WORKSPACE']
await jest.requireActual('@actions/io').rmRF(getTempDir())
})
test('zstd extract tar', async () => {
const mkdirMock = jest.spyOn(io, 'mkdirP')
const execMock = jest.spyOn(exec, 'exec')
const archivePath = IS_WINDOWS
? `${process.env['windir']}\\fakepath\\cache.tar`
: 'cache.tar'
const workspace = process.env['GITHUB_WORKSPACE']
await tar.extractTar(archivePath, CompressionMethod.Zstd)
expect(mkdirMock).toHaveBeenCalledWith(workspace)
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${defaultTarPath}"`,
[
'--use-compress-program',
'zstd -d --long=30',
'-xf',
IS_WINDOWS ? archivePath.replace(/\\/g, '/') : archivePath,
'-P',
'-C',
IS_WINDOWS ? workspace?.replace(/\\/g, '/') : workspace
]
.concat(IS_WINDOWS ? ['--force-local'] : [])
.concat(IS_MAC ? ['--delay-directory-restore'] : []),
{cwd: undefined}
)
})
test('gzip extract tar', async () => {
const mkdirMock = jest.spyOn(io, 'mkdirP')
const execMock = jest.spyOn(exec, 'exec')
const archivePath = IS_WINDOWS
? `${process.env['windir']}\\fakepath\\cache.tar`
: 'cache.tar'
const workspace = process.env['GITHUB_WORKSPACE']
await tar.extractTar(archivePath, CompressionMethod.Gzip)
expect(mkdirMock).toHaveBeenCalledWith(workspace)
const tarPath = IS_WINDOWS
? `${process.env['windir']}\\System32\\tar.exe`
: defaultTarPath
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${tarPath}"`,
[
'-z',
'-xf',
IS_WINDOWS ? archivePath.replace(/\\/g, '/') : archivePath,
'-P',
'-C',
IS_WINDOWS ? workspace?.replace(/\\/g, '/') : workspace
].concat(IS_MAC ? ['--delay-directory-restore'] : []),
{cwd: undefined}
)
})
test('gzip extract GNU tar on windows', async () => {
if (IS_WINDOWS) {
jest.spyOn(fs, 'existsSync').mockReturnValueOnce(false)
const isGnuMock = jest
.spyOn(utils, 'isGnuTarInstalled')
.mockReturnValue(Promise.resolve(true))
const execMock = jest.spyOn(exec, 'exec')
const archivePath = `${process.env['windir']}\\fakepath\\cache.tar`
const workspace = process.env['GITHUB_WORKSPACE']
await tar.extractTar(archivePath, CompressionMethod.Gzip)
expect(isGnuMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"tar"`,
[
'-z',
'-xf',
archivePath.replace(/\\/g, '/'),
'-P',
'-C',
workspace?.replace(/\\/g, '/'),
'--force-local'
],
{cwd: undefined}
)
}
})
test('zstd create tar', async () => {
const execMock = jest.spyOn(exec, 'exec')
const archiveFolder = getTempDir()
const workspace = process.env['GITHUB_WORKSPACE']
const sourceDirectories = ['~/.npm/cache', `${workspace}/dist`]
await fs.promises.mkdir(archiveFolder, {recursive: true})
await tar.createTar(archiveFolder, sourceDirectories, CompressionMethod.Zstd)
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${defaultTarPath}"`,
[
'--posix',
'--use-compress-program',
'zstd -T0 --long=30',
'-cf',
IS_WINDOWS ? CacheFilename.Zstd.replace(/\\/g, '/') : CacheFilename.Zstd,
'-P',
'-C',
IS_WINDOWS ? workspace?.replace(/\\/g, '/') : workspace,
'--files-from',
'manifest.txt'
]
.concat(IS_WINDOWS ? ['--force-local'] : [])
.concat(IS_MAC ? ['--delay-directory-restore'] : []),
{
cwd: archiveFolder
}
)
})
test('gzip create tar', async () => {
const execMock = jest.spyOn(exec, 'exec')
const archiveFolder = getTempDir()
const workspace = process.env['GITHUB_WORKSPACE']
const sourceDirectories = ['~/.npm/cache', `${workspace}/dist`]
await fs.promises.mkdir(archiveFolder, {recursive: true})
await tar.createTar(archiveFolder, sourceDirectories, CompressionMethod.Gzip)
const tarPath = IS_WINDOWS
? `${process.env['windir']}\\System32\\tar.exe`
: defaultTarPath
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${tarPath}"`,
[
'--posix',
'-z',
'-cf',
IS_WINDOWS ? CacheFilename.Gzip.replace(/\\/g, '/') : CacheFilename.Gzip,
'-P',
'-C',
IS_WINDOWS ? workspace?.replace(/\\/g, '/') : workspace,
'--files-from',
'manifest.txt'
].concat(IS_MAC ? ['--delay-directory-restore'] : []),
{
cwd: archiveFolder
}
)
})
test('zstd list tar', async () => {
const execMock = jest.spyOn(exec, 'exec')
const archivePath = IS_WINDOWS
? `${process.env['windir']}\\fakepath\\cache.tar`
: 'cache.tar'
await tar.listTar(archivePath, CompressionMethod.Zstd)
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${defaultTarPath}"`,
[
'--use-compress-program',
'zstd -d --long=30',
'-tf',
IS_WINDOWS ? archivePath.replace(/\\/g, '/') : archivePath,
'-P'
]
.concat(IS_WINDOWS ? ['--force-local'] : [])
.concat(IS_MAC ? ['--delay-directory-restore'] : []),
{cwd: undefined}
)
})
test('zstdWithoutLong list tar', async () => {
const execMock = jest.spyOn(exec, 'exec')
const archivePath = IS_WINDOWS
? `${process.env['windir']}\\fakepath\\cache.tar`
: 'cache.tar'
await tar.listTar(archivePath, CompressionMethod.ZstdWithoutLong)
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${defaultTarPath}"`,
[
'--use-compress-program',
'zstd -d',
'-tf',
IS_WINDOWS ? archivePath.replace(/\\/g, '/') : archivePath,
'-P'
]
.concat(IS_WINDOWS ? ['--force-local'] : [])
.concat(IS_MAC ? ['--delay-directory-restore'] : []),
{cwd: undefined}
)
})
test('gzip list tar', async () => {
const execMock = jest.spyOn(exec, 'exec')
const archivePath = IS_WINDOWS
? `${process.env['windir']}\\fakepath\\cache.tar`
: 'cache.tar'
await tar.listTar(archivePath, CompressionMethod.Gzip)
const tarPath = IS_WINDOWS
? `${process.env['windir']}\\System32\\tar.exe`
: defaultTarPath
expect(execMock).toHaveBeenCalledTimes(1)
expect(execMock).toHaveBeenCalledWith(
`"${tarPath}"`,
[
'-z',
'-tf',
IS_WINDOWS ? archivePath.replace(/\\/g, '/') : archivePath,
'-P'
].concat(IS_MAC ? ['--delay-directory-restore'] : []),
{cwd: undefined}
)
})
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
# Validate args
prefix="$1"
if [ -z "$prefix" ]; then
echo "Must supply prefix argument"
exit 1
fi
path="$2"
if [ -z "$path" ]; then
echo "Must specify path argument"
exit 1
fi
# Sanity check GITHUB_RUN_ID defined
if [ -z "$GITHUB_RUN_ID" ]; then
echo "GITHUB_RUN_ID not defined"
exit 1
fi
# Verify file exists
file="$path/test-file.txt"
echo "Checking for $file"
if [ ! -e $file ]; then
echo "File does not exist"
exit 1
fi
# Verify file content
content="$(cat $file)"
echo "File content:\n$content"
if [ -z "$(echo $content | grep --fixed-strings "$prefix $GITHUB_RUN_ID")" ]; then
echo "Unexpected file content"
exit 1
fi
+1102
View File
File diff suppressed because it is too large Load Diff
+55
View File
@@ -0,0 +1,55 @@
{
"name": "@actions/cache",
"version": "2.0.0",
"preview": true,
"description": "Actions cache lib",
"keywords": [
"github",
"actions",
"cache"
],
"homepage": "https://github.com/actions/toolkit/tree/main/packages/cache",
"license": "MIT",
"main": "lib/cache.js",
"types": "lib/cache.d.ts",
"directories": {
"lib": "lib",
"test": "__tests__"
},
"files": [
"lib",
"!.DS_Store"
],
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/actions/toolkit.git",
"directory": "packages/cache"
},
"scripts": {
"audit-moderate": "npm install && npm audit --json --audit-level=moderate > audit.json",
"test": "echo \"Error: run tests from root\" && exit 1",
"tsc": "tsc"
},
"bugs": {
"url": "https://github.com/actions/toolkit/issues"
},
"dependencies": {
"@actions/core": "^1.2.6",
"@actions/exec": "^1.0.1",
"@actions/glob": "^0.1.0",
"@actions/http-client": "^1.0.9",
"@actions/io": "^1.0.1",
"@azure/ms-rest-js": "^2.6.0",
"@azure/storage-blob": "^12.8.0",
"semver": "^6.1.0",
"uuid": "^3.3.3"
},
"devDependencies": {
"typescript": "^3.8.3",
"@types/semver": "^6.0.0",
"@types/uuid": "^3.4.5"
}
}
+208
View File
@@ -0,0 +1,208 @@
import * as core from '@actions/core'
import * as path from 'path'
import * as utils from './internal/cacheUtils'
import * as cacheHttpClient from './internal/cacheHttpClient'
import {createTar, extractTar, listTar} from './internal/tar'
import {DownloadOptions, UploadOptions} from './options'
export class ValidationError extends Error {
constructor(message: string) {
super(message)
this.name = 'ValidationError'
Object.setPrototypeOf(this, ValidationError.prototype)
}
}
export class ReserveCacheError extends Error {
constructor(message: string) {
super(message)
this.name = 'ReserveCacheError'
Object.setPrototypeOf(this, ReserveCacheError.prototype)
}
}
function checkPaths(paths: string[]): void {
if (!paths || paths.length === 0) {
throw new ValidationError(
`Path Validation Error: At least one directory or file path is required`
)
}
}
function checkKey(key: string): void {
if (key.length > 512) {
throw new ValidationError(
`Key Validation Error: ${key} cannot be larger than 512 characters.`
)
}
const regex = /^[^,]*$/
if (!regex.test(key)) {
throw new ValidationError(
`Key Validation Error: ${key} cannot contain commas.`
)
}
}
/**
* isFeatureAvailable to check the presence of Actions cache service
*
* @returns boolean return true if Actions cache service feature is available, otherwise false
*/
export function isFeatureAvailable(): boolean {
return !!process.env['ACTIONS_CACHE_URL']
}
/**
* Restores cache from keys
*
* @param paths a list of file paths to restore from the cache
* @param primaryKey an explicit key for restoring the cache
* @param restoreKeys an optional ordered list of keys to use for restoring the cache if no cache hit occurred for key
* @param downloadOptions cache download options
* @returns string returns the key for the cache hit, otherwise returns undefined
*/
export async function restoreCache(
paths: string[],
primaryKey: string,
restoreKeys?: string[],
options?: DownloadOptions
): Promise<string | undefined> {
checkPaths(paths)
restoreKeys = restoreKeys || []
const keys = [primaryKey, ...restoreKeys]
core.debug('Resolved Keys:')
core.debug(JSON.stringify(keys))
if (keys.length > 10) {
throw new ValidationError(
`Key Validation Error: Keys are limited to a maximum of 10.`
)
}
for (const key of keys) {
checkKey(key)
}
const compressionMethod = await utils.getCompressionMethod()
// path are needed to compute version
const cacheEntry = await cacheHttpClient.getCacheEntry(keys, paths, {
compressionMethod
})
if (!cacheEntry?.archiveLocation) {
// Cache not found
return undefined
}
const archivePath = path.join(
await utils.createTempDirectory(),
utils.getCacheFileName(compressionMethod)
)
core.debug(`Archive Path: ${archivePath}`)
try {
// Download the cache from the cache entry
await cacheHttpClient.downloadCache(
cacheEntry.archiveLocation,
archivePath,
options
)
if (core.isDebug()) {
await listTar(archivePath, compressionMethod)
}
const archiveFileSize = utils.getArchiveFileSizeInBytes(archivePath)
core.info(
`Cache Size: ~${Math.round(
archiveFileSize / (1024 * 1024)
)} MB (${archiveFileSize} B)`
)
await extractTar(archivePath, compressionMethod)
core.info('Cache restored successfully')
} finally {
// Try to delete the archive to save space
try {
await utils.unlinkFile(archivePath)
} catch (error) {
core.debug(`Failed to delete archive: ${error}`)
}
}
return cacheEntry.cacheKey
}
/**
* Saves a list of files with the specified key
*
* @param paths a list of file paths to be cached
* @param key an explicit key for restoring the cache
* @param options cache upload options
* @returns number returns cacheId if the cache was saved successfully and throws an error if save fails
*/
export async function saveCache(
paths: string[],
key: string,
options?: UploadOptions
): Promise<number> {
checkPaths(paths)
checkKey(key)
const compressionMethod = await utils.getCompressionMethod()
core.debug('Reserving Cache')
const cacheId = await cacheHttpClient.reserveCache(key, paths, {
compressionMethod
})
if (cacheId === -1) {
throw new ReserveCacheError(
`Unable to reserve cache with key ${key}, another job may be creating this cache.`
)
}
core.debug(`Cache ID: ${cacheId}`)
const cachePaths = await utils.resolvePaths(paths)
core.debug('Cache Paths:')
core.debug(`${JSON.stringify(cachePaths)}`)
const archiveFolder = await utils.createTempDirectory()
const archivePath = path.join(
archiveFolder,
utils.getCacheFileName(compressionMethod)
)
core.debug(`Archive Path: ${archivePath}`)
try {
await createTar(archiveFolder, cachePaths, compressionMethod)
if (core.isDebug()) {
await listTar(archivePath, compressionMethod)
}
const fileSizeLimit = 10 * 1024 * 1024 * 1024 // 10GB per repo limit
const archiveFileSize = utils.getArchiveFileSizeInBytes(archivePath)
core.debug(`File Size: ${archiveFileSize}`)
if (archiveFileSize > fileSizeLimit) {
throw new Error(
`Cache size of ~${Math.round(
archiveFileSize / (1024 * 1024)
)} MB (${archiveFileSize} B) is over the 10GB limit, not saving cache.`
)
}
core.debug(`Saving Cache (ID: ${cacheId})`)
await cacheHttpClient.saveCache(cacheId, archivePath, options)
} finally {
// Try to delete the archive to save space
try {
await utils.unlinkFile(archivePath)
} catch (error) {
core.debug(`Failed to delete archive: ${error}`)
}
}
return cacheId
}
+311
View File
@@ -0,0 +1,311 @@
import * as core from '@actions/core'
import {HttpClient} from '@actions/http-client'
import {BearerCredentialHandler} from '@actions/http-client/auth'
import {IRequestOptions, ITypedResponse} from '@actions/http-client/interfaces'
import * as crypto from 'crypto'
import * as fs from 'fs'
import {URL} from 'url'
import * as utils from './cacheUtils'
import {CompressionMethod} from './constants'
import {
ArtifactCacheEntry,
InternalCacheOptions,
CommitCacheRequest,
ReserveCacheRequest,
ReserveCacheResponse
} from './contracts'
import {downloadCacheHttpClient, downloadCacheStorageSDK} from './downloadUtils'
import {
DownloadOptions,
UploadOptions,
getDownloadOptions,
getUploadOptions
} from '../options'
import {
isSuccessStatusCode,
retryHttpClientResponse,
retryTypedResponse
} from './requestUtils'
const versionSalt = '1.0'
function getCacheApiUrl(resource: string): string {
const baseUrl: string = process.env['ACTIONS_CACHE_URL'] || ''
if (!baseUrl) {
throw new Error('Cache Service Url not found, unable to restore cache.')
}
const url = `${baseUrl}_apis/artifactcache/${resource}`
core.debug(`Resource Url: ${url}`)
return url
}
function createAcceptHeader(type: string, apiVersion: string): string {
return `${type};api-version=${apiVersion}`
}
function getRequestOptions(): IRequestOptions {
const requestOptions: IRequestOptions = {
headers: {
Accept: createAcceptHeader('application/json', '6.0-preview.1')
}
}
return requestOptions
}
function createHttpClient(): HttpClient {
const token = process.env['ACTIONS_RUNTIME_TOKEN'] || ''
const bearerCredentialHandler = new BearerCredentialHandler(token)
return new HttpClient(
'actions/cache',
[bearerCredentialHandler],
getRequestOptions()
)
}
export function getCacheVersion(
paths: string[],
compressionMethod?: CompressionMethod
): string {
const components = paths.concat(
!compressionMethod || compressionMethod === CompressionMethod.Gzip
? []
: [compressionMethod]
)
// Add salt to cache version to support breaking changes in cache entry
components.push(versionSalt)
return crypto
.createHash('sha256')
.update(components.join('|'))
.digest('hex')
}
export async function getCacheEntry(
keys: string[],
paths: string[],
options?: InternalCacheOptions
): Promise<ArtifactCacheEntry | null> {
const httpClient = createHttpClient()
const version = getCacheVersion(paths, options?.compressionMethod)
const resource = `cache?keys=${encodeURIComponent(
keys.join(',')
)}&version=${version}`
const response = await retryTypedResponse('getCacheEntry', async () =>
httpClient.getJson<ArtifactCacheEntry>(getCacheApiUrl(resource))
)
if (response.statusCode === 204) {
return null
}
if (!isSuccessStatusCode(response.statusCode)) {
throw new Error(`Cache service responded with ${response.statusCode}`)
}
const cacheResult = response.result
const cacheDownloadUrl = cacheResult?.archiveLocation
if (!cacheDownloadUrl) {
throw new Error('Cache not found.')
}
core.setSecret(cacheDownloadUrl)
core.debug(`Cache Result:`)
core.debug(JSON.stringify(cacheResult))
return cacheResult
}
export async function downloadCache(
archiveLocation: string,
archivePath: string,
options?: DownloadOptions
): Promise<void> {
const archiveUrl = new URL(archiveLocation)
const downloadOptions = getDownloadOptions(options)
if (
downloadOptions.useAzureSdk &&
archiveUrl.hostname.endsWith('.blob.core.windows.net')
) {
// Use Azure storage SDK to download caches hosted on Azure to improve speed and reliability.
await downloadCacheStorageSDK(archiveLocation, archivePath, downloadOptions)
} else {
// Otherwise, download using the Actions http-client.
await downloadCacheHttpClient(archiveLocation, archivePath)
}
}
// Reserve Cache
export async function reserveCache(
key: string,
paths: string[],
options?: InternalCacheOptions
): Promise<number> {
const httpClient = createHttpClient()
const version = getCacheVersion(paths, options?.compressionMethod)
const reserveCacheRequest: ReserveCacheRequest = {
key,
version
}
const response = await retryTypedResponse('reserveCache', async () =>
httpClient.postJson<ReserveCacheResponse>(
getCacheApiUrl('caches'),
reserveCacheRequest
)
)
return response?.result?.cacheId ?? -1
}
function getContentRange(start: number, end: number): string {
// Format: `bytes start-end/filesize
// start and end are inclusive
// filesize can be *
// For a 200 byte chunk starting at byte 0:
// Content-Range: bytes 0-199/*
return `bytes ${start}-${end}/*`
}
async function uploadChunk(
httpClient: HttpClient,
resourceUrl: string,
openStream: () => NodeJS.ReadableStream,
start: number,
end: number
): Promise<void> {
core.debug(
`Uploading chunk of size ${end -
start +
1} bytes at offset ${start} with content range: ${getContentRange(
start,
end
)}`
)
const additionalHeaders = {
'Content-Type': 'application/octet-stream',
'Content-Range': getContentRange(start, end)
}
const uploadChunkResponse = await retryHttpClientResponse(
`uploadChunk (start: ${start}, end: ${end})`,
async () =>
httpClient.sendStream(
'PATCH',
resourceUrl,
openStream(),
additionalHeaders
)
)
if (!isSuccessStatusCode(uploadChunkResponse.message.statusCode)) {
throw new Error(
`Cache service responded with ${uploadChunkResponse.message.statusCode} during upload chunk.`
)
}
}
async function uploadFile(
httpClient: HttpClient,
cacheId: number,
archivePath: string,
options?: UploadOptions
): Promise<void> {
// Upload Chunks
const fileSize = utils.getArchiveFileSizeInBytes(archivePath)
const resourceUrl = getCacheApiUrl(`caches/${cacheId.toString()}`)
const fd = fs.openSync(archivePath, 'r')
const uploadOptions = getUploadOptions(options)
const concurrency = utils.assertDefined(
'uploadConcurrency',
uploadOptions.uploadConcurrency
)
const maxChunkSize = utils.assertDefined(
'uploadChunkSize',
uploadOptions.uploadChunkSize
)
const parallelUploads = [...new Array(concurrency).keys()]
core.debug('Awaiting all uploads')
let offset = 0
try {
await Promise.all(
parallelUploads.map(async () => {
while (offset < fileSize) {
const chunkSize = Math.min(fileSize - offset, maxChunkSize)
const start = offset
const end = offset + chunkSize - 1
offset += maxChunkSize
await uploadChunk(
httpClient,
resourceUrl,
() =>
fs
.createReadStream(archivePath, {
fd,
start,
end,
autoClose: false
})
.on('error', error => {
throw new Error(
`Cache upload failed because file read failed with ${error.message}`
)
}),
start,
end
)
}
})
)
} finally {
fs.closeSync(fd)
}
return
}
async function commitCache(
httpClient: HttpClient,
cacheId: number,
filesize: number
): Promise<ITypedResponse<null>> {
const commitCacheRequest: CommitCacheRequest = {size: filesize}
return await retryTypedResponse('commitCache', async () =>
httpClient.postJson<null>(
getCacheApiUrl(`caches/${cacheId.toString()}`),
commitCacheRequest
)
)
}
export async function saveCache(
cacheId: number,
archivePath: string,
options?: UploadOptions
): Promise<void> {
const httpClient = createHttpClient()
core.debug('Upload cache')
await uploadFile(httpClient, cacheId, archivePath, options)
// Commit Cache
core.debug('Commiting cache')
const cacheSize = utils.getArchiveFileSizeInBytes(archivePath)
core.info(
`Cache Size: ~${Math.round(cacheSize / (1024 * 1024))} MB (${cacheSize} B)`
)
const commitCacheResponse = await commitCache(httpClient, cacheId, cacheSize)
if (!isSuccessStatusCode(commitCacheResponse.statusCode)) {
throw new Error(
`Cache service responded with ${commitCacheResponse.statusCode} during commit cache.`
)
}
core.info('Cache saved successfully')
}
+125
View File
@@ -0,0 +1,125 @@
import * as core from '@actions/core'
import * as exec from '@actions/exec'
import * as glob from '@actions/glob'
import * as io from '@actions/io'
import * as fs from 'fs'
import * as path from 'path'
import * as semver from 'semver'
import * as util from 'util'
import {v4 as uuidV4} from 'uuid'
import {CacheFilename, CompressionMethod} from './constants'
// From https://github.com/actions/toolkit/blob/main/packages/tool-cache/src/tool-cache.ts#L23
export async function createTempDirectory(): Promise<string> {
const IS_WINDOWS = process.platform === 'win32'
let tempDirectory: string = process.env['RUNNER_TEMP'] || ''
if (!tempDirectory) {
let baseLocation: string
if (IS_WINDOWS) {
// On Windows use the USERPROFILE env variable
baseLocation = process.env['USERPROFILE'] || 'C:\\'
} else {
if (process.platform === 'darwin') {
baseLocation = '/Users'
} else {
baseLocation = '/home'
}
}
tempDirectory = path.join(baseLocation, 'actions', 'temp')
}
const dest = path.join(tempDirectory, uuidV4())
await io.mkdirP(dest)
return dest
}
export function getArchiveFileSizeInBytes(filePath: string): number {
return fs.statSync(filePath).size
}
export async function resolvePaths(patterns: string[]): Promise<string[]> {
const paths: string[] = []
const workspace = process.env['GITHUB_WORKSPACE'] ?? process.cwd()
const globber = await glob.create(patterns.join('\n'), {
implicitDescendants: false
})
for await (const file of globber.globGenerator()) {
const relativeFile = path
.relative(workspace, file)
.replace(new RegExp(`\\${path.sep}`, 'g'), '/')
core.debug(`Matched: ${relativeFile}`)
// Paths are made relative so the tar entries are all relative to the root of the workspace.
paths.push(`${relativeFile}`)
}
return paths
}
export async function unlinkFile(filePath: fs.PathLike): Promise<void> {
return util.promisify(fs.unlink)(filePath)
}
async function getVersion(app: string): Promise<string> {
core.debug(`Checking ${app} --version`)
let versionOutput = ''
try {
await exec.exec(`${app} --version`, [], {
ignoreReturnCode: true,
silent: true,
listeners: {
stdout: (data: Buffer): string => (versionOutput += data.toString()),
stderr: (data: Buffer): string => (versionOutput += data.toString())
}
})
} catch (err) {
core.debug(err.message)
}
versionOutput = versionOutput.trim()
core.debug(versionOutput)
return versionOutput
}
// Use zstandard if possible to maximize cache performance
export async function getCompressionMethod(): Promise<CompressionMethod> {
if (process.platform === 'win32' && !(await isGnuTarInstalled())) {
// Disable zstd due to bug https://github.com/actions/cache/issues/301
return CompressionMethod.Gzip
}
const versionOutput = await getVersion('zstd')
const version = semver.clean(versionOutput)
if (!versionOutput.toLowerCase().includes('zstd command line interface')) {
// zstd is not installed
return CompressionMethod.Gzip
} else if (!version || semver.lt(version, 'v1.3.2')) {
// zstd is installed but using a version earlier than v1.3.2
// v1.3.2 is required to use the `--long` options in zstd
return CompressionMethod.ZstdWithoutLong
} else {
return CompressionMethod.Zstd
}
}
export function getCacheFileName(compressionMethod: CompressionMethod): string {
return compressionMethod === CompressionMethod.Gzip
? CacheFilename.Gzip
: CacheFilename.Zstd
}
export async function isGnuTarInstalled(): Promise<boolean> {
const versionOutput = await getVersion('tar')
return versionOutput.toLowerCase().includes('gnu tar')
}
export function assertDefined<T>(name: string, value?: T): T {
if (value === undefined) {
throw Error(`Expected ${name} but value was undefiend`)
}
return value
}
+23
View File
@@ -0,0 +1,23 @@
export enum CacheFilename {
Gzip = 'cache.tgz',
Zstd = 'cache.tzst'
}
export enum CompressionMethod {
Gzip = 'gzip',
// Long range mode was added to zstd in v1.3.2.
// This enum is for earlier version of zstd that does not have --long support
ZstdWithoutLong = 'zstd-without-long',
Zstd = 'zstd'
}
// The default number of retry attempts.
export const DefaultRetryAttempts = 2
// The default delay in milliseconds between retry attempts.
export const DefaultRetryDelay = 5000
// Socket timeout in milliseconds during download. If no traffic is received
// over the socket during this period, the socket is destroyed and the download
// is aborted.
export const SocketTimeout = 5000
+25
View File
@@ -0,0 +1,25 @@
import {CompressionMethod} from './constants'
export interface ArtifactCacheEntry {
cacheKey?: string
scope?: string
creationTime?: string
archiveLocation?: string
}
export interface CommitCacheRequest {
size: number
}
export interface ReserveCacheRequest {
key: string
version?: string
}
export interface ReserveCacheResponse {
cacheId: number
}
export interface InternalCacheOptions {
compressionMethod?: CompressionMethod
}
+279
View File
@@ -0,0 +1,279 @@
import * as core from '@actions/core'
import {HttpClient} from '@actions/http-client'
import {IHttpClientResponse} from '@actions/http-client/interfaces'
import {BlockBlobClient} from '@azure/storage-blob'
import {TransferProgressEvent} from '@azure/ms-rest-js'
import * as buffer from 'buffer'
import * as fs from 'fs'
import * as stream from 'stream'
import * as util from 'util'
import * as utils from './cacheUtils'
import {SocketTimeout} from './constants'
import {DownloadOptions} from '../options'
import {retryHttpClientResponse} from './requestUtils'
/**
* Pipes the body of a HTTP response to a stream
*
* @param response the HTTP response
* @param output the writable stream
*/
async function pipeResponseToStream(
response: IHttpClientResponse,
output: NodeJS.WritableStream
): Promise<void> {
const pipeline = util.promisify(stream.pipeline)
await pipeline(response.message, output)
}
/**
* Class for tracking the download state and displaying stats.
*/
export class DownloadProgress {
contentLength: number
segmentIndex: number
segmentSize: number
segmentOffset: number
receivedBytes: number
startTime: number
displayedComplete: boolean
timeoutHandle?: ReturnType<typeof setTimeout>
constructor(contentLength: number) {
this.contentLength = contentLength
this.segmentIndex = 0
this.segmentSize = 0
this.segmentOffset = 0
this.receivedBytes = 0
this.displayedComplete = false
this.startTime = Date.now()
}
/**
* Progress to the next segment. Only call this method when the previous segment
* is complete.
*
* @param segmentSize the length of the next segment
*/
nextSegment(segmentSize: number): void {
this.segmentOffset = this.segmentOffset + this.segmentSize
this.segmentIndex = this.segmentIndex + 1
this.segmentSize = segmentSize
this.receivedBytes = 0
core.debug(
`Downloading segment at offset ${this.segmentOffset} with length ${this.segmentSize}...`
)
}
/**
* Sets the number of bytes received for the current segment.
*
* @param receivedBytes the number of bytes received
*/
setReceivedBytes(receivedBytes: number): void {
this.receivedBytes = receivedBytes
}
/**
* Returns the total number of bytes transferred.
*/
getTransferredBytes(): number {
return this.segmentOffset + this.receivedBytes
}
/**
* Returns true if the download is complete.
*/
isDone(): boolean {
return this.getTransferredBytes() === this.contentLength
}
/**
* Prints the current download stats. Once the download completes, this will print one
* last line and then stop.
*/
display(): void {
if (this.displayedComplete) {
return
}
const transferredBytes = this.segmentOffset + this.receivedBytes
const percentage = (100 * (transferredBytes / this.contentLength)).toFixed(
1
)
const elapsedTime = Date.now() - this.startTime
const downloadSpeed = (
transferredBytes /
(1024 * 1024) /
(elapsedTime / 1000)
).toFixed(1)
core.info(
`Received ${transferredBytes} of ${this.contentLength} (${percentage}%), ${downloadSpeed} MBs/sec`
)
if (this.isDone()) {
this.displayedComplete = true
}
}
/**
* Returns a function used to handle TransferProgressEvents.
*/
onProgress(): (progress: TransferProgressEvent) => void {
return (progress: TransferProgressEvent) => {
this.setReceivedBytes(progress.loadedBytes)
}
}
/**
* Starts the timer that displays the stats.
*
* @param delayInMs the delay between each write
*/
startDisplayTimer(delayInMs = 1000): void {
const displayCallback = (): void => {
this.display()
if (!this.isDone()) {
this.timeoutHandle = setTimeout(displayCallback, delayInMs)
}
}
this.timeoutHandle = setTimeout(displayCallback, delayInMs)
}
/**
* Stops the timer that displays the stats. As this typically indicates the download
* is complete, this will display one last line, unless the last line has already
* been written.
*/
stopDisplayTimer(): void {
if (this.timeoutHandle) {
clearTimeout(this.timeoutHandle)
this.timeoutHandle = undefined
}
this.display()
}
}
/**
* Download the cache using the Actions toolkit http-client
*
* @param archiveLocation the URL for the cache
* @param archivePath the local path where the cache is saved
*/
export async function downloadCacheHttpClient(
archiveLocation: string,
archivePath: string
): Promise<void> {
const writeStream = fs.createWriteStream(archivePath)
const httpClient = new HttpClient('actions/cache')
const downloadResponse = await retryHttpClientResponse(
'downloadCache',
async () => httpClient.get(archiveLocation)
)
// Abort download if no traffic received over the socket.
downloadResponse.message.socket.setTimeout(SocketTimeout, () => {
downloadResponse.message.destroy()
core.debug(`Aborting download, socket timed out after ${SocketTimeout} ms`)
})
await pipeResponseToStream(downloadResponse, writeStream)
// Validate download size.
const contentLengthHeader = downloadResponse.message.headers['content-length']
if (contentLengthHeader) {
const expectedLength = parseInt(contentLengthHeader)
const actualLength = utils.getArchiveFileSizeInBytes(archivePath)
if (actualLength !== expectedLength) {
throw new Error(
`Incomplete download. Expected file size: ${expectedLength}, actual file size: ${actualLength}`
)
}
} else {
core.debug('Unable to validate download, no Content-Length header')
}
}
/**
* Download the cache using the Azure Storage SDK. Only call this method if the
* URL points to an Azure Storage endpoint.
*
* @param archiveLocation the URL for the cache
* @param archivePath the local path where the cache is saved
* @param options the download options with the defaults set
*/
export async function downloadCacheStorageSDK(
archiveLocation: string,
archivePath: string,
options: DownloadOptions
): Promise<void> {
const client = new BlockBlobClient(archiveLocation, undefined, {
retryOptions: {
// Override the timeout used when downloading each 4 MB chunk
// The default is 2 min / MB, which is way too slow
tryTimeoutInMs: options.timeoutInMs
}
})
const properties = await client.getProperties()
const contentLength = properties.contentLength ?? -1
if (contentLength < 0) {
// We should never hit this condition, but just in case fall back to downloading the
// file as one large stream
core.debug(
'Unable to determine content length, downloading file with http-client...'
)
await downloadCacheHttpClient(archiveLocation, archivePath)
} else {
// Use downloadToBuffer for faster downloads, since internally it splits the
// file into 4 MB chunks which can then be parallelized and retried independently
//
// If the file exceeds the buffer maximum length (~1 GB on 32-bit systems and ~2 GB
// on 64-bit systems), split the download into multiple segments
// ~2 GB = 2147483647, beyond this, we start getting out of range error. So, capping it accordingly.
const maxSegmentSize = Math.min(2147483647, buffer.constants.MAX_LENGTH)
const downloadProgress = new DownloadProgress(contentLength)
const fd = fs.openSync(archivePath, 'w')
try {
downloadProgress.startDisplayTimer()
while (!downloadProgress.isDone()) {
const segmentStart =
downloadProgress.segmentOffset + downloadProgress.segmentSize
const segmentSize = Math.min(
maxSegmentSize,
contentLength - segmentStart
)
downloadProgress.nextSegment(segmentSize)
const result = await client.downloadToBuffer(
segmentStart,
segmentSize,
{
concurrency: options.downloadConcurrency,
onProgress: downloadProgress.onProgress()
}
)
fs.writeFileSync(fd, result)
}
} finally {
downloadProgress.stopDisplayTimer()
fs.closeSync(fd)
}
}
}
+136
View File
@@ -0,0 +1,136 @@
import * as core from '@actions/core'
import {HttpCodes, HttpClientError} from '@actions/http-client'
import {
IHttpClientResponse,
ITypedResponse
} from '@actions/http-client/interfaces'
import {DefaultRetryDelay, DefaultRetryAttempts} from './constants'
export function isSuccessStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return false
}
return statusCode >= 200 && statusCode < 300
}
export function isServerErrorStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return true
}
return statusCode >= 500
}
export function isRetryableStatusCode(statusCode?: number): boolean {
if (!statusCode) {
return false
}
const retryableStatusCodes = [
HttpCodes.BadGateway,
HttpCodes.ServiceUnavailable,
HttpCodes.GatewayTimeout
]
return retryableStatusCodes.includes(statusCode)
}
async function sleep(milliseconds: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
export async function retry<T>(
name: string,
method: () => Promise<T>,
getStatusCode: (arg0: T) => number | undefined,
maxAttempts = DefaultRetryAttempts,
delay = DefaultRetryDelay,
onError: ((arg0: Error) => T | undefined) | undefined = undefined
): Promise<T> {
let errorMessage = ''
let attempt = 1
while (attempt <= maxAttempts) {
let response: T | undefined = undefined
let statusCode: number | undefined = undefined
let isRetryable = false
try {
response = await method()
} catch (error) {
if (onError) {
response = onError(error)
}
isRetryable = true
errorMessage = error.message
}
if (response) {
statusCode = getStatusCode(response)
if (!isServerErrorStatusCode(statusCode)) {
return response
}
}
if (statusCode) {
isRetryable = isRetryableStatusCode(statusCode)
errorMessage = `Cache service responded with ${statusCode}`
}
core.debug(
`${name} - Attempt ${attempt} of ${maxAttempts} failed with error: ${errorMessage}`
)
if (!isRetryable) {
core.debug(`${name} - Error is not retryable`)
break
}
await sleep(delay)
attempt++
}
throw Error(`${name} failed: ${errorMessage}`)
}
export async function retryTypedResponse<T>(
name: string,
method: () => Promise<ITypedResponse<T>>,
maxAttempts = DefaultRetryAttempts,
delay = DefaultRetryDelay
): Promise<ITypedResponse<T>> {
return await retry(
name,
method,
(response: ITypedResponse<T>) => response.statusCode,
maxAttempts,
delay,
// If the error object contains the statusCode property, extract it and return
// an ITypedResponse<T> so it can be processed by the retry logic.
(error: Error) => {
if (error instanceof HttpClientError) {
return {
statusCode: error.statusCode,
result: null,
headers: {}
}
} else {
return undefined
}
}
)
}
export async function retryHttpClientResponse(
name: string,
method: () => Promise<IHttpClientResponse>,
maxAttempts = DefaultRetryAttempts,
delay = DefaultRetryDelay
): Promise<IHttpClientResponse> {
return await retry(
name,
method,
(response: IHttpClientResponse) => response.message.statusCode,
maxAttempts,
delay
)
}
+155
View File
@@ -0,0 +1,155 @@
import {exec} from '@actions/exec'
import * as io from '@actions/io'
import {existsSync, writeFileSync} from 'fs'
import * as path from 'path'
import * as utils from './cacheUtils'
import {CompressionMethod} from './constants'
async function getTarPath(
args: string[],
compressionMethod: CompressionMethod
): Promise<string> {
switch (process.platform) {
case 'win32': {
const systemTar = `${process.env['windir']}\\System32\\tar.exe`
if (compressionMethod !== CompressionMethod.Gzip) {
// We only use zstandard compression on windows when gnu tar is installed due to
// a bug with compressing large files with bsdtar + zstd
args.push('--force-local')
} else if (existsSync(systemTar)) {
return systemTar
} else if (await utils.isGnuTarInstalled()) {
args.push('--force-local')
}
break
}
case 'darwin': {
const gnuTar = await io.which('gtar', false)
if (gnuTar) {
// fix permission denied errors when extracting BSD tar archive with GNU tar - https://github.com/actions/cache/issues/527
args.push('--delay-directory-restore')
return gnuTar
}
break
}
default:
break
}
return await io.which('tar', true)
}
async function execTar(
args: string[],
compressionMethod: CompressionMethod,
cwd?: string
): Promise<void> {
try {
await exec(`"${await getTarPath(args, compressionMethod)}"`, args, {cwd})
} catch (error) {
throw new Error(`Tar failed with error: ${error?.message}`)
}
}
function getWorkingDirectory(): string {
return process.env['GITHUB_WORKSPACE'] ?? process.cwd()
}
export async function extractTar(
archivePath: string,
compressionMethod: CompressionMethod
): Promise<void> {
// Create directory to extract tar into
const workingDirectory = getWorkingDirectory()
await io.mkdirP(workingDirectory)
// --d: Decompress.
// --long=#: Enables long distance matching with # bits. Maximum is 30 (1GB) on 32-bit OS and 31 (2GB) on 64-bit.
// Using 30 here because we also support 32-bit self-hosted runners.
function getCompressionProgram(): string[] {
switch (compressionMethod) {
case CompressionMethod.Zstd:
return ['--use-compress-program', 'zstd -d --long=30']
case CompressionMethod.ZstdWithoutLong:
return ['--use-compress-program', 'zstd -d']
default:
return ['-z']
}
}
const args = [
...getCompressionProgram(),
'-xf',
archivePath.replace(new RegExp(`\\${path.sep}`, 'g'), '/'),
'-P',
'-C',
workingDirectory.replace(new RegExp(`\\${path.sep}`, 'g'), '/')
]
await execTar(args, compressionMethod)
}
export async function createTar(
archiveFolder: string,
sourceDirectories: string[],
compressionMethod: CompressionMethod
): Promise<void> {
// Write source directories to manifest.txt to avoid command length limits
const manifestFilename = 'manifest.txt'
const cacheFileName = utils.getCacheFileName(compressionMethod)
writeFileSync(
path.join(archiveFolder, manifestFilename),
sourceDirectories.join('\n')
)
const workingDirectory = getWorkingDirectory()
// -T#: Compress using # working thread. If # is 0, attempt to detect and use the number of physical CPU cores.
// --long=#: Enables long distance matching with # bits. Maximum is 30 (1GB) on 32-bit OS and 31 (2GB) on 64-bit.
// Using 30 here because we also support 32-bit self-hosted runners.
// Long range mode is added to zstd in v1.3.2 release, so we will not use --long in older version of zstd.
function getCompressionProgram(): string[] {
switch (compressionMethod) {
case CompressionMethod.Zstd:
return ['--use-compress-program', 'zstd -T0 --long=30']
case CompressionMethod.ZstdWithoutLong:
return ['--use-compress-program', 'zstd -T0']
default:
return ['-z']
}
}
const args = [
'--posix',
...getCompressionProgram(),
'-cf',
cacheFileName.replace(new RegExp(`\\${path.sep}`, 'g'), '/'),
'-P',
'-C',
workingDirectory.replace(new RegExp(`\\${path.sep}`, 'g'), '/'),
'--files-from',
manifestFilename
]
await execTar(args, compressionMethod, archiveFolder)
}
export async function listTar(
archivePath: string,
compressionMethod: CompressionMethod
): Promise<void> {
// --d: Decompress.
// --long=#: Enables long distance matching with # bits.
// Maximum is 30 (1GB) on 32-bit OS and 31 (2GB) on 64-bit.
// Using 30 here because we also support 32-bit self-hosted runners.
function getCompressionProgram(): string[] {
switch (compressionMethod) {
case CompressionMethod.Zstd:
return ['--use-compress-program', 'zstd -d --long=30']
case CompressionMethod.ZstdWithoutLong:
return ['--use-compress-program', 'zstd -d']
default:
return ['-z']
}
}
const args = [
...getCompressionProgram(),
'-tf',
archivePath.replace(new RegExp(`\\${path.sep}`, 'g'), '/'),
'-P'
]
await execTar(args, compressionMethod)
}
+109
View File
@@ -0,0 +1,109 @@
import * as core from '@actions/core'
/**
* Options to control cache upload
*/
export interface UploadOptions {
/**
* Number of parallel cache upload
*
* @default 4
*/
uploadConcurrency?: number
/**
* Maximum chunk size in bytes for cache upload
*
* @default 32MB
*/
uploadChunkSize?: number
}
/**
* Options to control cache download
*/
export interface DownloadOptions {
/**
* Indicates whether to use the Azure Blob SDK to download caches
* that are stored on Azure Blob Storage to improve reliability and
* performance
*
* @default true
*/
useAzureSdk?: boolean
/**
* Number of parallel downloads (this option only applies when using
* the Azure SDK)
*
* @default 8
*/
downloadConcurrency?: number
/**
* Maximum time for each download request, in milliseconds (this
* option only applies when using the Azure SDK)
*
* @default 30000
*/
timeoutInMs?: number
}
/**
* Returns a copy of the upload options with defaults filled in.
*
* @param copy the original upload options
*/
export function getUploadOptions(copy?: UploadOptions): UploadOptions {
const result: UploadOptions = {
uploadConcurrency: 4,
uploadChunkSize: 32 * 1024 * 1024
}
if (copy) {
if (typeof copy.uploadConcurrency === 'number') {
result.uploadConcurrency = copy.uploadConcurrency
}
if (typeof copy.uploadChunkSize === 'number') {
result.uploadChunkSize = copy.uploadChunkSize
}
}
core.debug(`Upload concurrency: ${result.uploadConcurrency}`)
core.debug(`Upload chunk size: ${result.uploadChunkSize}`)
return result
}
/**
* Returns a copy of the download options with defaults filled in.
*
* @param copy the original download options
*/
export function getDownloadOptions(copy?: DownloadOptions): DownloadOptions {
const result: DownloadOptions = {
useAzureSdk: true,
downloadConcurrency: 8,
timeoutInMs: 30000
}
if (copy) {
if (typeof copy.useAzureSdk === 'boolean') {
result.useAzureSdk = copy.useAzureSdk
}
if (typeof copy.downloadConcurrency === 'number') {
result.downloadConcurrency = copy.downloadConcurrency
}
if (typeof copy.timeoutInMs === 'number') {
result.timeoutInMs = copy.timeoutInMs
}
}
core.debug(`Use Azure SDK: ${result.useAzureSdk}`)
core.debug(`Download concurrency: ${result.downloadConcurrency}`)
core.debug(`Request timeout (ms): ${result.timeoutInMs}`)
return result
}
+12
View File
@@ -0,0 +1,12 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"baseUrl": "./",
"outDir": "./lib",
"rootDir": "./src",
"lib": ["es6", "dom"]
},
"include": [
"./src"
]
}
+9
View File
@@ -0,0 +1,9 @@
The MIT License (MIT)
Copyright 2019 GitHub
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+178 -6
View File
@@ -16,11 +16,14 @@ import * as core from '@actions/core';
#### Inputs/Outputs
Action inputs can be read with `getInput`. Outputs can be set with `setOutput` which makes them available to be mapped into inputs of other actions to ensure they are decoupled.
Action inputs can be read with `getInput` which returns a `string` or `getBooleanInput` which parses a boolean based on the [yaml 1.2 specification](https://yaml.org/spec/1.2/spec.html#id2804923). If `required` set to be false, the input should have a default value in `action.yml`.
Outputs can be set with `setOutput` which makes them available to be mapped into inputs of other actions to ensure they are decoupled.
```js
const myInput = core.getInput('inputName', { required: true });
const myBooleanInput = core.getBooleanInput('booleanInputName', { required: true });
const myMultilineInput = core.getMultilineInput('multilineInputName', { required: true });
core.setOutput('outputKey', 'outputVal');
```
@@ -62,11 +65,10 @@ catch (err) {
// setFailed logs the message and sets a failing exit code
core.setFailed(`Action failed with error ${err}`);
}
```
Note that `setNeutral` is not yet implemented in actions V2 but equivalent functionality is being planned.
```
#### Logging
Finally, this library provides some utilities for logging. Note that debug logging is hidden from the logs by default. This behavior can be toggled by enabling the [Step Debug Logs](../../docs/action-debugging.md#step-debug-logs).
@@ -82,7 +84,16 @@ try {
core.warning('myInput was not set');
}
if (core.isDebug()) {
// curl -v https://github.com
} else {
// curl https://github.com
}
// Do stuff
core.info('Output to the actions build log')
core.notice('This is a message that will also emit an annotation')
}
catch (err) {
core.error(`Error ${err}, action may still succeed though`);
@@ -106,11 +117,123 @@ const result = await core.group('Do something async', async () => {
})
```
#### Annotations
This library has 3 methods that will produce [annotations](https://docs.github.com/en/rest/reference/checks#create-a-check-run).
```js
core.error('This is a bad error. This will also fail the build.')
core.warning('Something went wrong, but it\'s not bad enough to fail the build.')
core.notice('Something happened that you might want to know about.')
```
These will surface to the UI in the Actions page and on Pull Requests. They look something like this:
![Annotations Image](../../docs/assets/annotations.png)
These annotations can also be attached to particular lines and columns of your source files to show exactly where a problem is occuring.
These options are:
```typescript
export interface AnnotationProperties {
/**
* A title for the annotation.
*/
title?: string
/**
* The name of the file for which the annotation should be created.
*/
file?: string
/**
* The start line for the annotation.
*/
startLine?: number
/**
* The end line for the annotation. Defaults to `startLine` when `startLine` is provided.
*/
endLine?: number
/**
* The start column for the annotation. Cannot be sent when `startLine` and `endLine` are different values.
*/
startColumn?: number
/**
* The start column for the annotation. Cannot be sent when `startLine` and `endLine` are different values.
* Defaults to `startColumn` when `startColumn` is provided.
*/
endColumn?: number
}
```
#### Styling output
Colored output is supported in the Action logs via standard [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code). 3/4 bit, 8 bit and 24 bit colors are all supported.
Foreground colors:
```js
// 3/4 bit
core.info('\u001b[35mThis foreground will be magenta')
// 8 bit
core.info('\u001b[38;5;6mThis foreground will be cyan')
// 24 bit
core.info('\u001b[38;2;255;0;0mThis foreground will be bright red')
```
Background colors:
```js
// 3/4 bit
core.info('\u001b[43mThis background will be yellow');
// 8 bit
core.info('\u001b[48;5;6mThis background will be cyan')
// 24 bit
core.info('\u001b[48;2;255;0;0mThis background will be bright red')
```
Special styles:
```js
core.info('\u001b[1mBold text')
core.info('\u001b[3mItalic text')
core.info('\u001b[4mUnderlined text')
```
ANSI escape codes can be combined with one another:
```js
core.info('\u001b[31;46mRed foreground with a cyan background and \u001b[1mbold text at the end');
```
> Note: Escape codes reset at the start of each line
```js
core.info('\u001b[35mThis foreground will be magenta')
core.info('This foreground will reset to the default')
```
Manually typing escape codes can be a little difficult, but you can use third party modules such as [ansi-styles](https://github.com/chalk/ansi-styles).
```js
const style = require('ansi-styles');
core.info(style.color.ansi16m.hex('#abcdef') + 'Hello world!')
```
#### Action state
You can use this library to save state and get state for sharing information between a given wrapper action:
You can use this library to save state and get state for sharing information between a given wrapper action:
**action.yml**:
**action.yml**
```yaml
name: 'Wrapper action sample'
inputs:
@@ -131,10 +254,59 @@ core.saveState("pidToKill", 12345);
```
In action's `cleanup.js`:
```js
const core = require('@actions/core');
var pid = core.getState("pidToKill");
process.kill(pid);
```
#### OIDC Token
You can use these methods to interact with the GitHub OIDC provider and get a JWT ID token which would help to get access token from third party cloud providers.
**Method Name**: getIDToken()
**Inputs**
audience : optional
**Outputs**
A [JWT](https://jwt.io/) ID Token
In action's `main.ts`:
```js
const core = require('@actions/core');
async function getIDTokenAction(): Promise<void> {
const audience = core.getInput('audience', {required: false})
const id_token1 = await core.getIDToken() // ID Token with default audience
const id_token2 = await core.getIDToken(audience) // ID token with custom audience
// this id_token can be used to get access token from third party cloud providers
}
getIDTokenAction()
```
In action's `actions.yml`:
```yaml
name: 'GetIDToken'
description: 'Get ID token from Github OIDC provider'
inputs:
audience:
description: 'Audience for which the ID token is intended for'
required: false
outputs:
id_token1:
description: 'ID token obtained from OIDC provider'
id_token2:
description: 'ID token obtained from OIDC provider'
runs:
using: 'node12'
main: 'dist/index.js'
```
+40
View File
@@ -1,5 +1,45 @@
# @actions/core Releases
### 1.6.0
- [Added OIDC Client function `getIDToken`](https://github.com/actions/toolkit/pull/919)
- [Added `file` parameter to `AnnotationProperties`](https://github.com/actions/toolkit/pull/896)
### 1.5.0
- [Added support for notice annotations and more annotation fields](https://github.com/actions/toolkit/pull/855)
### 1.4.0
- [Added the `getMultilineInput` function](https://github.com/actions/toolkit/pull/829)
### 1.3.0
- [Added the trimWhitespace option to getInput](https://github.com/actions/toolkit/pull/802)
- [Added the getBooleanInput function](https://github.com/actions/toolkit/pull/725)
### 1.2.7
- [Prepend newline for set-output](https://github.com/actions/toolkit/pull/772)
### 1.2.6
- [Update `exportVariable` and `addPath` to use environment files](https://github.com/actions/toolkit/pull/571)
### 1.2.5
- [Correctly bundle License File with package](https://github.com/actions/toolkit/pull/548)
### 1.2.4
- [Be more lenient in accepting non-string command inputs](https://github.com/actions/toolkit/pull/405)
- [Add Echo commands](https://github.com/actions/toolkit/pull/411)
### 1.2.3
- [IsDebug logging](README.md#logging)
### 1.2.2
- [Fix escaping for runner commands](https://github.com/actions/toolkit/pull/302)
### 1.2.1
- [Remove trailing comma from commands](https://github.com/actions/toolkit/pull/263)
- [Add \"types\" to package.json](https://github.com/actions/toolkit/pull/221)
### 1.2.0
- saveState and getState functions for wrapper tasks (on finally entry points that run post job)

Some files were not shown because too many files have changed in this diff Show More