Optimize webhooks JSON with compact format and string interning
- Convert params to compact array format (type-based dispatch) - Intern duplicate property names into string table - Use negative indices for object references to distinguish from string indices - Rename objects.json → webhooks.objects.json, add webhooks.strings.json - Move event filters to JSON for maintainability - Add CI validation job to verify optimization correctness Reduces combined minified size by ~67% (453 KB → 148 KB), ~27% gzipped (23 KB → 17 KB).
This commit is contained in:
eric sciple
+99
-26
@@ -6,8 +6,9 @@ This document describes the JSON data files used by the language service package
|
||||
|
||||
The language service uses several JSON files containing schema definitions, webhook payloads, and other metadata. To reduce bundle size, these files are:
|
||||
|
||||
1. **Optimized at generation time** — unused events are dropped, unused fields are stripped
|
||||
2. **Minified at build time** — whitespace is removed to produce `.min.json` files
|
||||
1. **Optimized at generation time** — unused events are dropped, unused fields are stripped, shared objects are deduplicated, property names are interned
|
||||
2. **Compacted using a space-efficient format** — params use type-based dispatch arrays instead of objects
|
||||
3. **Minified at build time** — whitespace is removed to produce `.min.json` files
|
||||
|
||||
The source `.json` files are human-readable and checked into the repository. The `.min.json` files are generated during build and gitignored.
|
||||
|
||||
@@ -18,7 +19,8 @@ The source `.json` files are human-readable and checked into the repository. The
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `src/context-providers/events/webhooks.json` | Webhook event payload schemas for autocompletion |
|
||||
| `src/context-providers/events/objects.json` | Deduplicated shared object definitions referenced by webhooks |
|
||||
| `src/context-providers/events/webhooks.objects.json` | Deduplicated shared object definitions referenced by webhooks |
|
||||
| `src/context-providers/events/webhooks.strings.json` | Interned property names shared by webhooks and objects |
|
||||
| `src/context-providers/events/schedule.json` | Schedule event context data |
|
||||
| `src/context-providers/events/workflow_call.json` | Reusable workflow call context data |
|
||||
| `src/context-providers/descriptions.json` | Context variable descriptions for hover |
|
||||
@@ -33,7 +35,7 @@ The source `.json` files are human-readable and checked into the repository. The
|
||||
|
||||
### Webhooks and Objects
|
||||
|
||||
The `webhooks.json` and `objects.json` files are generated from the [GitHub REST API description](https://github.com/github/rest-api-description):
|
||||
The `webhooks.json`, `webhooks.objects.json`, and `webhooks.strings.json` files are generated from the [GitHub REST API description](https://github.com/github/rest-api-description):
|
||||
|
||||
```bash
|
||||
cd languageservice
|
||||
@@ -44,9 +46,10 @@ This script:
|
||||
1. Fetches webhook schemas from the GitHub API description
|
||||
2. **Validates** all events are categorized (fails if new events are found)
|
||||
3. **Drops** events that aren't valid workflow triggers (see [Dropped Events](#dropped-events))
|
||||
4. **Strips** unused fields like `description` and `summary` (see [Stripped Fields](#stripped-fields))
|
||||
5. **Deduplicates** shared object definitions into `objects.json`
|
||||
6. Writes the optimized, pretty-printed JSON files
|
||||
4. **Compacts** params into a space-efficient array format, keeping only `name`, `description`, and `childParamsGroups` (see [Compact Format](#compact-format))
|
||||
5. **Deduplicates** shared object definitions into `webhooks.objects.json`
|
||||
6. **Interns** duplicate property names into `webhooks.strings.json` (see [String Interning](#string-interning))
|
||||
7. Writes the optimized, pretty-printed JSON files
|
||||
|
||||
### Handling New Webhook Events
|
||||
|
||||
@@ -67,9 +70,9 @@ Action required:
|
||||
|
||||
1. Check [Events that trigger workflows](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows)
|
||||
|
||||
2. Edit `languageservice/script/webhooks/index.ts`:
|
||||
- Add to `KEPT_EVENTS` if it's a valid workflow trigger
|
||||
- Add to `DROPPED_EVENTS` if it's GitHub App or API-only
|
||||
2. Edit `languageservice/src/context-providers/events/event-filters.json`:
|
||||
- Add to `kept` array if it's a valid workflow trigger
|
||||
- Add to `dropped` array if it's GitHub App or API-only
|
||||
|
||||
3. Run `npm run update-webhooks` and commit the changes
|
||||
|
||||
@@ -101,13 +104,15 @@ The code imports the minified versions:
|
||||
|
||||
```ts
|
||||
import webhooks from "./events/webhooks.min.json"
|
||||
import objects from "./events/webhooks.objects.min.json"
|
||||
import strings from "./events/webhooks.strings.min.json"
|
||||
```
|
||||
|
||||
## CI Verification
|
||||
|
||||
CI verifies that generated source files are up-to-date:
|
||||
|
||||
1. Runs `npm run update-webhooks` to regenerate webhooks.json and objects.json
|
||||
1. Runs `npm run update-webhooks` to regenerate webhooks.json, webhooks.objects.json, and webhooks.strings.json
|
||||
2. Checks for uncommitted changes with `git diff --exit-code`
|
||||
|
||||
The `.min.json` files are generated at build time and are not committed to the repository.
|
||||
@@ -118,33 +123,101 @@ If the build fails, run `cd languageservice && npm run update-webhooks` locally
|
||||
|
||||
Webhook events that aren't valid workflow `on:` triggers are dropped (e.g., `installation`, `ping`, `member`, etc.). These are GitHub App or API-only events.
|
||||
|
||||
See `DROPPED_EVENTS` in `script/webhooks/index.ts` for the full list.
|
||||
See `dropped` array in `src/context-providers/events/event-filters.json` for the full list.
|
||||
|
||||
## Stripped Fields
|
||||
## Compact Format
|
||||
|
||||
Unused fields are stripped to reduce bundle size. For example:
|
||||
Params are converted from verbose objects into compact arrays, keeping only the fields needed for autocompletion and hover docs (`name`, `description`, `childParamsGroups`). Unused fields like `type`, `in`, `isRequired`, `enum`, and `default` are discarded.
|
||||
|
||||
| Format | Meaning |
|
||||
|--------|---------|
|
||||
| `"name"` | Name only (no description, no children) |
|
||||
| `[name, desc]` | Name + description (arr[1] is a string) |
|
||||
| `[name, children]` | Name + children (arr[1] is an array) |
|
||||
| `[name, desc, children]` | Name + description + children |
|
||||
|
||||
The reader uses `typeof arr[1]` to determine the format: if it's a string, it's a description; if it's an array, it's children.
|
||||
|
||||
**Example:**
|
||||
|
||||
```json
|
||||
// Before (from webhooks.all.json)
|
||||
// Before (object format)
|
||||
{
|
||||
"type": "object",
|
||||
"name": "issue",
|
||||
"in": "body",
|
||||
"description": "The issue itself.",
|
||||
"isRequired": true,
|
||||
"childParamsGroups": [...]
|
||||
"childParamsGroups": [
|
||||
{ "name": "id" },
|
||||
{ "name": "title", "description": "Issue title" }
|
||||
]
|
||||
}
|
||||
|
||||
// After (webhooks.json)
|
||||
// After (compact format)
|
||||
["issue", "The issue itself.", [
|
||||
"id",
|
||||
["title", "Issue title"]
|
||||
]]
|
||||
```
|
||||
|
||||
## String Interning
|
||||
|
||||
Property names that appear 2+ times are "interned" into a shared string table (`webhooks.strings.json`). In the compact arrays, these names are replaced with non-negative numeric indices:
|
||||
|
||||
```json
|
||||
// webhooks.strings.json
|
||||
["url", "id", "name", ...] // Index 0 = "url", 1 = "id", 2 = "name"
|
||||
|
||||
// webhooks.json - uses indices instead of strings
|
||||
{
|
||||
"name": "issue",
|
||||
"description": "The issue itself.",
|
||||
"childParamsGroups": [...]
|
||||
"push": {
|
||||
"default": {
|
||||
"p": [
|
||||
[0, "The URL..."], // 0 = "url" from string table
|
||||
[1, "Unique ID"], // 1 = "id"
|
||||
2 // 2 = "name" (name-only, no description)
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Only `name`, `description`, and `childParamsGroups` are kept — these are used for autocompletion and hover docs.
|
||||
**How to distinguish indices from other values:**
|
||||
|
||||
To compare all fields vs stripped, run `npm run update-webhooks -- --all` and diff the `.all.json` files against the regular ones.
|
||||
- **Negative numbers** → Object indices: `-1` = object 0, `-2` = object 1, etc. (formula: `-(index + 1)`)
|
||||
- **Non-negative numbers** → String indices (references into `webhooks.strings.json`)
|
||||
- **Literal strings** → Singletons (names appearing only once, not interned)
|
||||
|
||||
See `EVENT_ACTION_FIELDS` and `BODY_PARAM_FIELDS` in `script/webhooks/index.ts` to modify what gets stripped.
|
||||
Singletons are kept as literal strings for readability and to avoid the overhead of adding rarely-used names to the string table.
|
||||
|
||||
## Deduplication
|
||||
|
||||
Shared object definitions are extracted into `webhooks.objects.json` and referenced by negative index:
|
||||
|
||||
```json
|
||||
// webhooks.objects.json
|
||||
[
|
||||
["url", "The URL"], // Index 0 (referenced as -1)
|
||||
["id", "Unique identifier"], // Index 1 (referenced as -2)
|
||||
[...]
|
||||
]
|
||||
|
||||
// webhooks.json - negative numbers reference objects
|
||||
{
|
||||
"push": {
|
||||
"default": {
|
||||
"p": [-1, -2, ["ref", "The git ref"]] // -1 = object 0, -2 = object 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This reduces duplication when the same object structure appears in multiple events (e.g., `repository`, `sender`, `organization`).
|
||||
|
||||
## Size Reduction
|
||||
|
||||
The optimizations achieve approximately 99% file size reduction:
|
||||
|
||||
| Stage | Minified | Gzip |
|
||||
|-------|----------|------|
|
||||
| Original (webhooks.full.json) | 15.8 MB | 968 KB |
|
||||
| After optimization (combined) | 152 KB | 15.6 KB |
|
||||
| **Reduction** | **99%** | **98%** |
|
||||
|
||||
Reference in New Issue
Block a user