Skip to content
← 回到文章列表

用 Eval 系统地测试 Agent Skill ​

用 Eval 系统地测试 Agent Skill

When you’re iterating on a skill for an agent like Codex, it’s hard to tell whether you’re actually improving it or just changing its behavior. One version feels faster, another seems more reliable, and then a regression slips in: the skill doesn’t trigger, it skips a required step, or it leaves extra files behind.

给 Codex 这类 Agent 反复改一个 skill 的时候,你很难说清自己到底是在改进它,还是只是让它的行为变了个样。这一版感觉快了点,那一版好像更稳了,然后回归(regression)就悄悄混进来:skill 没触发,跳过了必须做的某一步,或者留下了一堆多余的文件。

At its core, a skill is an organized collection of prompts and instructions for an LLM. The most reliable way to improve a skill over time is to evaluate it the same way you would any other prompt for LLM applications.

说到底,skill 就是写给 LLM 的一套整理好的提示词和指令。想让它越改越好,最靠谱的办法就是像评估 LLM 应用里的其他提示词那样去评估它。

Evals (short for evaluations) check whether a model’s output, and the steps it took to produce it, match what you intended. Instead of asking “does this feel better?” (or relying on vibes), evals let you ask concrete questions like:

Eval(evaluation 的简称,评测)检查的是模型的输出,以及它得出这个输出走过的步骤,是不是符合你的本意。有了 eval,你不用再问「这样感觉是不是好一点?」(或者全凭感觉),而是可以问一些具体的问题,比如:

  • Did the agent invoke the skill?Agent 调用这个 skill 了吗?
  • Did it run the expected commands?它跑了该跑的命令吗?
  • Did it produce outputs that follow the conventions you care about?它的产出符合你在意的那些约定吗?

Concretely, an eval is: a prompt → a captured run (trace + artifacts) → a small set of checks → a score you can compare over time.

具体来说,一个 eval 就是:一条提示词 → 一次完整记录下来的运行,包括轨迹(trace)和产物→ 一小组检查 → 一个可以跨时间比较的分数。

In practice, evals for agent skills look a lot like lightweight end-to-end tests: you run the agent, record what happened, and score the result against a small set of rules.

实际做起来,Agent skill 的 eval 很像轻量的端到端测试:把 Agent 跑一遍,记下发生了什么,再按一小组规则给结果打分。

This post walks through a clear pattern for doing that with Codex, starting from defining success, then adding deterministic checks and rubric-based grading so improvements (and regressions) are clear.

这篇文章用 Codex 走一遍这套做法:先定义什么叫成功,再加上确定性检查和基于评分细则(rubric)的打分,让改进(和回归)都一目了然。

1. Define success before you write the skill1. 写 skill 之前,先定义成功 ​

Before writing the skill itself, write down what “success” means in terms you can actually measure. A useful way to think about this is to split your checks into a few categories:

动手写 skill 之前,先用真正能衡量的说法把「成功」写下来。一个好用的思路是把检查分成几类:

  • Outcome goals: Did the task complete? Does the app run?结果目标: 任务完成了吗?应用能跑起来吗?
  • Process goals: Did Codex invoke the skill and follow the tools and steps you intended?过程目标: Codex 调用这个 skill 了吗?用的工具、走的步骤是不是你想要的?
  • Style goals: Does the output follow the conventions you asked for?风格目标: 产出符合你要求的约定吗?
  • Efficiency goals: Did it get there without thrashing (for example, unnecessary commands or excessive token use)?效率目标: 它是不是没有来回折腾就做到了(比如跑了多余的命令,或者用掉了过多的 token)?

Keep this list small and focused on must-pass checks. The goal isn’t to encode every preference up front, but to capture the behaviors you care about most.

这份清单要短,只放必须通过的检查。目的不是一开始就把所有偏好都写进去,而是抓住你最在意的那些行为。

In this post, for example, the guide evaluates a skill that sets up a demo app. Some checks are concrete. Did it run npm install? Did it create package.json? The guide pairs those with a structured style rubric to evaluate conventions and layout.

拿这篇文章来说,要评测的是一个搭建演示应用的 skill。有些检查很具体:它跑了 npm install 吗?它创建了 package.json 吗?除了这些,还配了一份结构化的风格评分细则,用来评约定和目录布局。

This mix is intentional. You want fast, targeted signals that surface specific regressions early, rather than a single pass/fail verdict at the end.

这样搭配是故意的。你要的是快速、有针对性的信号,能早早暴露出具体是哪里回归了,而不是到最后才给一个笼统的通过/不通过。

2. Create the skill2. 创建 skill ​

A Codex skill is a directory with a SKILL.md file that includes YAML front matter (name, description), followed by the Markdown instructions that define the skill’s behavior and optional resources and scripts. The name and description matter more than they might seem. They’re the primary signals Codex uses to decide whether to invoke the skill at all, and when to inject the rest of SKILL.md into the agent’s context. If these are vague or overloaded, the skill won’t trigger reliably.

一个 Codex skill 就是一个目录,里面有一个 SKILL.md 文件:开头是 YAML front matter(name、description),后面是定义 skill 行为的 Markdown 指令,另外还可以带一些资源和脚本。name 和 description 比看上去要重要得多。Codex 主要就靠它们来决定要不要调用这个 skill,以及什么时候把 SKILL.md 的其余部分注入 Agent 的上下文。这两项写得含糊,或者塞了太多东西,skill 就没法稳定触发。

The fastest way to get started is to use Codex’s built-in skill creator (which itself is also a skill). It walks you through:

上手最快的办法是用 Codex 内置的 skill creator(它本身也是一个 skill)。它会一步步带你走:

shell
$skill-creator

The creator asks you what the skill does, when it should trigger, and whether it’s instruction-only or script-backed (instruction-only is the default recommendation). To learn more about creating a skill, check out the documentation.

它会问你这个 skill 做什么、什么时候该触发、是只有指令还是带脚本(默认推荐只有指令)。想多了解怎么创建 skill,可以看看文档。

A sample skill一个示例 skill ​

This post uses an intentionally minimal example: a skill that sets up a small React demo app in a predictable, repeatable way.

这篇文章特意用了一个极简的例子:一个用可预测、可重复的方式搭一个小型 React 演示应用的 skill。

This skill will:

这个 skill 会:

  • Scaffold a project using Vite’s React + TypeScript template用 Vite 的 React + TypeScript 模板搭出项目骨架
  • Configure Tailwind CSS using the official Vite plugin approach按官方的 Vite 插件方式配置 Tailwind CSS
  • Enforce a minimal, consistent file structure强制使用一套最小、一致的文件结构
  • Define a clear “definition of done” so success is straightforward to evaluate定一个清楚的「完成标准」,这样成功与否很好判断

Below is a compact draft you can paste either into:

下面是一份精简的草稿,你可以把它粘贴到以下任一位置:

  • .codex/skills/setup-demo-app/SKILL.md (repo-scoped), or.codex/skills/setup-demo-app/SKILL.md(仓库级),或者
  • ~/.codex/skills/setup-demo-app/SKILL.md (user-scoped).~/.codex/skills/setup-demo-app/SKILL.md(用户级)。
markdown
---
name: setup-demo-app
description: Scaffold a Vite + React + Tailwind demo app with a small, consistent project structure.
---

## When to use this

Use when you need a fresh demo app for quick UI experiments or reproductions.

## What to build

Create a Vite React TypeScript app and configure Tailwind. Keep it minimal.

Project structure after setup:

- src/
  - main.tsx (entry)
  - App.tsx (root UI)
  - components/
    - Header.tsx
    - Card.tsx
  - index.css (Tailwind import)
- index.html
- package.json

Style requirements:

- TypeScript components
- Functional components only
- Tailwind classes for styling (no CSS modules)
- No extra UI libraries

## Steps

1. Scaffold with Vite using the React TS template:
   npm create vite@latest demo-app -- --template react-ts

2. Install dependencies:
   cd demo-app
   npm install

3. Install and configure Tailwind using the Vite plugin.
   - npm install tailwindcss @tailwindcss/vite
   - Add the tailwind plugin to vite.config.ts
   - In src/index.css, replace contents with:
     @import "tailwindcss";

4. Implement the minimal UI:
   - Header: app title and short subtitle
   - Card: reusable card container
   - App: render Header + 2 Cards with placeholder text

## Definition of done

- npm run dev starts successfully
- package.json exists
- src/components/Header.tsx and src/components/Card.tsx exist

This sample skill takes an opinionated stance on purpose. Without clear constraints, there’s nothing concrete to evaluate.

这个示例 skill 故意把话说得很死。没有明确的约束,就没有什么具体的东西可评。

3. Manually trigger the skill to expose hidden assumptions3. 手动触发 skill,把隐藏的假设逼出来 ​

Because skill invocation depends so much on the name and description in SKILL.md, the first thing to check is whether the setup-demo-app skill triggers when you expect it to.

skill 会不会被调用,很大程度上取决于 SKILL.md 里的 name 和 description,所以第一件要查的事,就是 setup-demo-app 这个 skill 有没有在你预期的时候触发。

Early on, explicitly activate the skill, either via the /skills slash command or by referencing it with the $ prefix, in a real repository or a scratch directory, and watch where it breaks. This is where you surface the misses: cases where the skill doesn’t trigger at all, triggers too eagerly, or runs but deviates from the intended steps.

早期阶段,在一个真实仓库或者临时目录里显式地启用这个 skill,可以用 /skills 斜杠命令,也可以用 $ 前缀引用它,然后看它在哪儿出问题。漏掉的情况就是在这一步暴露出来的:skill 压根没触发、触发得太积极,或者跑了但偏离了预定的步骤。

At this stage, you’re not optimizing for speed or polish. You’re looking for hidden assumptions the skill is making, such as:

这个阶段不追求速度,也不追求打磨。你要找的是 skill 暗中做了哪些假设,比如:

  • Triggering assumptions: Prompts like “set up a quick React demo” that should invoke setup-demo-app but don’t, or more generic prompts (“add Tailwind styling”) that unintentionally trigger it.触发上的假设:像「搭一个简单的 React 演示」这样的提示词本该调用 setup-demo-app 却没调用,或者更泛的提示词(「加点 Tailwind 样式」)却意外触发了它。
  • Environment assumptions: The skill assumes it’s running in an empty directory, or that npm is available and preferred over other package managers.环境上的假设:skill 默认自己跑在一个空目录里,或者默认 npm 可用、而且比别的包管理器优先。
  • Execution assumptions: The agent skips npm install because it assumes dependencies are already installed, or configures Tailwind before the Vite project exists.执行上的假设:Agent 以为依赖已经装好了,就跳过了 npm install;或者 Vite 项目还没建出来,就先去配 Tailwind。

Once you’re ready to make these runs repeatable, switch to codex exec. It’s designed for automation and CI: it streams progress to stderr and writes only the final result to stdout, which makes runs easier to script, capture, and inspect.

等你准备让这些运行可以重复,就换成 codex exec。它是为自动化和 CI 设计的:进度实时输出到 stderr,stdout 里只写最终结果,这样写脚本、记录、检查起来都更方便。

By default, codex exec runs in a restricted sandbox. If your task needs to write files, run it with --full-auto. As a general rule, especially when automating, use the least permissions needed to get the job done.

codex exec 默认跑在一个受限的沙箱里。任务要写文件的话,就加上 --full-auto。通用的原则是,尤其在自动化的时候,只给完成任务所需的最小权限。

A basic manual run might look like:

一次基本的手动运行大概是这样:

shell
codex exec --full-auto \
  'Use the $setup-demo-app skill to create the project in this directory.'

This first hands-on pass is less about validating correctness and more about discovering edge cases. Every manual fix you make here, such as adding a missing npm install, correcting the Tailwind setup, or tightening the trigger description, is a candidate for a future eval, so you can lock in the intended behavior before evaluating at scale.

第一轮亲手跑,重点不在验证对不对,而在发现边界情况。你在这里做的每一处手动修正,比如补上漏掉的 npm install、改对 Tailwind 的配置、把触发用的 description 收紧,都可以变成以后的一条 eval,这样在大规模评测之前就能把想要的行为固定下来。

4. Use a small, targeted prompt set to catch regressions early4. 用一小组有针对性的提示词,尽早抓住回归 ​

You don’t need a large benchmark to get value from evals. For a single skill, a small set of 10–20 prompts is enough to surface regressions and confirm improvements early.

eval 要有用,并不需要一个很大的基准测试。对单个 skill 来说,10 到 20 条提示词就够早早暴露回归、确认改进了。

Start with a small CSV and grow it over time as you encounter real failures during development or usage. Each row should represent a situation where you care whether the setup-demo-app skill does or does not activate, and what success looks like when it does.

先从一个小 CSV 开始,开发或使用中碰到真实的失败,就往里加。每一行都应该对应一个场景:你在意 setup-demo-app 这个 skill 在这里该不该启用,以及启用之后什么样算成功。

For example, an initial evals/setup-demo-app.prompts.csv might look like this:

比如,最初的 evals/setup-demo-app.prompts.csv 可能长这样:

plaintext
id,should_trigger,prompt
test-01,true,"Create a demo app named `devday-demo` using the $setup-demo-app skill"
test-02,true,"Set up a minimal React demo app with Tailwind for quick UI experiments"
test-03,true,"Create a small demo app to showcase the Responses API"
test-04,false,"Add Tailwind styling to my existing React app"

Each of these cases is testing something slightly different:

这几条测的东西各有一点不同:

  • Explicit invocation (test-01) This prompt names the skill directly. It ensures that Codex can invoke setup-demo-app when asked, and that changes to the skill’s name, description, or instructions don’t break direct usage.显式调用(test-01) 这条提示词直接点了 skill 的名字。它保证被要求时 Codex 能调用 setup-demo-app,而且改了 skill 的 name、description 或指令之后,直接点名调用还是好使的。
  • Implicit invocation (test-02) This prompt describes exactly the scenario the skill targets, setting up a minimal React + Tailwind demo, without mentioning the skill by name. It tests whether the name and description in SKILL.md are strong enough for Codex to select the skill on its own.隐式调用(test-02) 这条提示词描述的正好就是 skill 针对的场景,也就是搭一个最小的 React + Tailwind 演示,但没提 skill 的名字。它测的是 SKILL.md 里的 name 和 description 够不够有力,能不能让 Codex 自己选中这个 skill。
  • Contextual invocation (test-03) This prompt adds domain context (the Responses API) but still requires the same underlying setup. It checks that the skill triggers in realistic, slightly noisy prompts, and that the resulting app still matches the expected structure and conventions.带上下文的调用(test-03) 这条提示词加了领域上下文(Responses API),但底下需要的还是同一套搭建。它检查的是:在真实、略带干扰的提示词下 skill 也能触发,而且搭出来的应用仍然符合预期的结构和约定。
  • Negative control (test-04) This prompt should not invoke setup-demo-app. It’s a common adjacent request (“add Tailwind to an existing app”) that can unintentionally match the skill’s description (“React + Tailwind demo”). Including at least one should_trigger=false case helps catch false positives, where Codex selects the skill too eagerly and scaffolds a new project when the user wanted an incremental change to an existing one.阴性对照(Negative control,test-04) 这条提示词不该调用 setup-demo-app。它是一个很常见的相邻请求(「给现有应用加上 Tailwind」),可能无意中跟 skill 的 description(「React + Tailwind 演示」)对上。至少放一条 should_trigger=false 的用例,能帮你抓到误触发(false positive):用户只想在现有项目上做点增量修改,Codex 却过于积极地选中了这个 skill,新搭了一个项目。

This mix is intentional. Some evals should confirm that the skill behaves correctly when invoked explicitly; others should check that it activates in real-world prompts where the user never mentions the skill at all.

这样搭配是故意的。有些 eval 要确认 skill 被显式调用时行为正确;另一些则要检查,在用户根本没提这个 skill 的真实提示词里,它也会启用。

As you discover misses, prompts that fail to trigger the skill, or cases where the output drifts from your expectations, add them as new rows. Over time, this small CSV becomes a living record of the scenarios the setup-demo-app skill must continue to get right.

每发现一处遗漏,比如没能触发 skill 的提示词,或者产出偏离了预期的情况,就加成新的一行。时间长了,这个小 CSV 就成了一份活的记录,记着 setup-demo-app 这个 skill 必须一直做对的那些场景。

Over time, this small dataset becomes a living record of what the skill must continue to get right.

时间长了,这个小数据集就成了一份活的记录,记着这个 skill 必须一直做对的事。

5. Get started with lightweight deterministic graders5. 从轻量的确定性评分器开始 ​

This is the core of the evaluation step: use codex exec --json so your eval harness can score what actually happened, not just whether the final output looks right.

这是评测这一步的核心:用 codex exec --json,让你的评测框架(eval harness)能给实际发生了什么打分,而不只是看最终输出像不像样。

When you enable --json, stdout becomes a JSONL stream of structured events. That makes it straightforward to write deterministic checks tied directly to the behavior you care about, for example:

打开 --json 之后,stdout 会变成一条由结构化事件组成的 JSONL 流。这样写确定性检查就很容易,可以直接对准你关心的行为,比如:

  • Did it run npm install?它跑了 npm install 吗?
  • Did it create package.json?它创建了 package.json 吗?
  • Did it invoke the expected commands, in the expected order?它是不是按预期的顺序调用了预期的命令?

These checks are intentionally lightweight. They give you fast, explainable signals before you add any model-based grading.

这些检查故意做得很轻。在你加上任何基于模型的打分之前,它们就能给你快速、说得清原因的信号。

A minimal Node.js runner一个最小的 Node.js 运行脚本 ​

A “good enough” approach looks like this:

一个「够用」的做法是这样的:

  1. For each prompt, run codex exec --json --full-auto "<prompt>"对每条提示词,运行 codex exec --json --full-auto "<prompt>"
  2. Save the JSONL trace to disk把 JSONL 轨迹存到磁盘上
  3. Parse the trace and run deterministic checks over the events解析轨迹,对其中的事件跑确定性检查
javascript
// evals/run-setup-demo-app-evals.mjs
import { spawnSync } from "node:child_process";
import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
import path from "node:path";

function runCodex(prompt, outJsonlPath) {
  const res = spawnSync(
    "codex",
    [
      "exec",
      "--json", // REQUIRED: emit structured events
      "--full-auto", // Allow file system changes
      prompt,
    ],
    { encoding: "utf8" }
  );

  mkdirSync(path.dirname(outJsonlPath), { recursive: true });

  // stdout is JSONL when --json is enabled
  writeFileSync(outJsonlPath, res.stdout, "utf8");

  return { exitCode: res.status ?? 1, stderr: res.stderr };
}

function parseJsonl(jsonlText) {
  return jsonlText
    .split("\n")
    .filter(Boolean)
    .map((line) => JSON.parse(line));
}

// deterministic check: did the agent run `npm install`?
function checkRanNpmInstall(events) {
  return events.some(
    (e) =>
      (e.type === "item.started" || e.type === "item.completed") &&
      e.item?.type === "command_execution" &&
      typeof e.item?.command === "string" &&
      e.item.command.includes("npm install")
  );
}

// deterministic check: did `package.json` get created?
function checkPackageJsonExists(projectDir) {
  return existsSync(path.join(projectDir, "package.json"));
}

// Example single-case run
const projectDir = process.cwd();
const tracePath = path.join(projectDir, "evals", "artifacts", "test-01.jsonl");

const prompt =
  "Create a demo app named demo-app using the $setup-demo-app skill";

runCodex(prompt, tracePath);

const events = parseJsonl(readFileSync(tracePath, "utf8"));

console.log({
  ranNpmInstall: checkRanNpmInstall(events),
  hasPackageJson: checkPackageJsonExists(path.join(projectDir, "demo-app")),
});

The value here is that everything is deterministic and debuggable.

这里的好处是,一切都确定、可调试。

If a check fails, you can open the JSONL file and see exactly what happened. Every command execution appears as an item.* event, in order. That makes regressions straightforward to explain and fix, which is exactly what you want at this stage.

某项检查没过,你可以打开那个 JSONL 文件,看清楚到底发生了什么。每一次命令执行都按顺序记成一条 item.* 事件。所以回归很好解释、也很好修,而这正是这个阶段你想要的。

6. Conduct qualitative checks with Codex and rubric-based grading6. 用 Codex 和评分细则做定性检查 ​

Deterministic checks answer “did it do the basics?” but they don’t answer “did it do it the way you wanted?”

确定性检查回答的是*「基本的事做了没有?」,但回答不了「是不是按你想要的方式做的?」*

For skills like setup-demo-app, many requirements are qualitative: component structure, styling conventions, or whether Tailwind follows the intended configuration. These are hard to capture with basic file existence checks or command counts alone.

像 setup-demo-app 这样的 skill,很多要求是定性的:组件结构、样式约定,或者 Tailwind 的配置是不是按预期来的。光靠检查文件在不在、数命令跑了几次,很难把这些抓住。

A pragmatic solution is to add a second, model-assisted step to your eval pipeline:

一个务实的办法,是在评测流水线里加第二步,由模型来辅助:

  1. Run the setup skill (this writes code to disk)运行搭建用的 skill(它会把代码写到磁盘上)
  2. Run a read-only style check against the resulting repository对生成的仓库跑一遍只读的风格检查
  3. Require a structured response that your harness can score consistently要求返回结构化的结果,让评测框架能稳定地打分

Codex supports this directly via --output-schema, which constrains the final response to a JSON Schema you define.

Codex 通过 --output-schema 直接支持这一点,它会把最终回复约束成你定义的 JSON Schema。

A small rubric schema一份小小的评分细则 schema ​

Start by defining a small schema that captures the checks you care about. For example, create evals/style-rubric.schema.json:

先定义一个小 schema,把你关心的检查装进去。比如,创建 evals/style-rubric.schema.json:

json
{
  "type": "object",
  "properties": {
    "overall_pass": { "type": "boolean" },
    "score": { "type": "integer", "minimum": 0, "maximum": 100 },
    "checks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "pass": { "type": "boolean" },
          "notes": { "type": "string" }
        },
        "required": ["id", "pass", "notes"],
        "additionalProperties": false
      }
    }
  },
  "required": ["overall_pass", "score", "checks"],
  "additionalProperties": false
}

This schema gives you stable fields (overall_pass, score, per-check results) that you can combine, diff, and track over time.

这个 schema 给了你一组稳定的字段(overall_pass、score、每项检查的结果),可以合并、对比差异,也可以长期跟踪。

The style-check prompt风格检查的提示词 ​

Next, run a second codex exec that only inspects the repository and emits a rubric-compliant JSON response:

接下来,再跑一个 codex exec,这次它只检查仓库,输出一份符合评分细则的 JSON:

shell
codex exec \
  "Evaluate the demo-app repository against these requirements:
   - Vite + React + TypeScript project exists
   - Tailwind is configured via @tailwindcss/vite and CSS imports tailwindcss
   - src/components contains Header.tsx and Card.tsx
   - Components are functional and styled with Tailwind utility classes (no CSS modules)
   Return a rubric result as JSON with check ids: vite, tailwind, structure, style." \
  --output-schema ./evals/style-rubric.schema.json \
  -o ./evals/artifacts/test-01.style.json

This is where --output-schema is handy. Instead of free-form text that’s hard to parse or compare, you get a predictable JSON object that your eval harness can score across many runs.

--output-schema 好用就好用在这里。你拿到的不再是难解析、难比较的自由文本,而是一个格式可预期的 JSON 对象,评测框架可以在很多次运行之间统一打分。

If you later move this eval suite into CI, the Codex GitHub Action explicitly supports passing --output-schema through codex-args, so you can enforce the same structured output in automated workflows.

如果以后把这套 eval 搬进 CI,Codex 的 GitHub Action 明确支持通过 codex-args 传入 --output-schema,所以在自动化工作流里也能强制同样的结构化输出。

7. Extending your evals as the skill matures7. skill 成熟之后,再扩展 eval ​

Once you have the core loop in place, you can extend your evals in the directions that matter most for your skill. Start small, then layer in deeper checks only where they add real confidence.

核心的循环跑通之后,就可以往对你的 skill 最要紧的方向扩展 eval。从小处开始,只在真正能增加把握的地方,才一层层加上更深的检查。

Some examples include:

举几个例子:

  • Command count and thrashing: Count command_execution items in the JSONL trace to catch regressions where the agent starts looping or re-running commands. Token usage is also available in turn.completed events.命令数量和来回折腾: 数一数 JSONL 轨迹里 command_execution 条目的个数,抓住 Agent 开始兜圈子、重复跑命令这类回归。turn.completed 事件里还能拿到 token 用量。
  • Token budget: Track usage.input_tokens and usage.output_tokens to spot accidental prompt bloat and compare efficiency across versions.token 预算: 跟踪 usage.input_tokens 和 usage.output_tokens,发现提示词不知不觉膨胀起来,也能比较不同版本的效率。
  • Build checks: Run npm run build after the skill completes. This acts as a stronger end-to-end signal and catches broken imports or incorrectly configured tooling.构建检查: skill 跑完之后执行 npm run build。这是一个更强的端到端信号,能抓到坏掉的 import 或者配错的工具链。
  • Runtime smoke checks: Start npm run dev and hit the dev server with curl, or run a lightweight Playwright check if you already have one. Use this selectively. It adds confidence but costs time.运行时冒烟检查: 启动 npm run dev,用 curl 请求一下开发服务器;如果手头已经有轻量的 Playwright 检查,也可以跑一下。这个要挑着用:它能增加把握,但费时间。
  • Repository cleanliness: Ensure the run generates no unwanted files and that git status --porcelain is empty (or matches an explicit allow list).仓库干净程度: 确认这次运行没有生成多余的文件,git status --porcelain 的输出为空(或者跟一份明确的白名单对得上)。
  • Sandbox and permission regressions: Verify the skill still works without escalating permissions beyond what you intended. Least-privilege defaults matter most once you automate.沙箱和权限方面的回归: 确认 skill 在不把权限提到超出你预期的情况下仍然能用。一旦自动化起来,默认最小权限就是最要紧的。

The pattern is consistent: begin with fast checks that explain behavior, then add slower, heavier checks only when they reduce risk.

套路始终是一样的:先上能解释行为的快检查,只有在能降低风险的时候,才加更慢、更重的检查。

8. Key takeaways8. 要点 ​

This small setup-demo-app example shows the shift from “it feels better” to “proof”: run the agent, record what happened, and grade it with a small set of checks. Once that loop exists, every tweak becomes easier to confirm, and every regression becomes clear. Here are the key takeaways:

setup-demo-app 这个小例子展示的,是从「感觉更好了」到「拿得出证据」的转变:跑一遍 Agent,记下发生了什么,再用一小组检查给它打分。有了这个循环,每一处微调都更容易确认,每一次回归也都看得清清楚楚。要点如下:

  • Measure what matters. Good evals make regressions clear and failures explainable.衡量要紧的东西。 好的 eval 让回归一目了然,让失败说得清原因。
  • Start from a checkable definition of done. Use $skill-creator to bootstrap, then tighten the instructions until success is unambiguous.从一个能检查的完成标准出发。 用 $skill-creator 起个头,再把指令一点点收紧,直到成功与否没有歧义。
  • Ground evals in behavior. Capture JSONL with codex exec --json and write deterministic checks against command_execution events.让 eval 立足于行为。 用 codex exec --json 记下 JSONL,针对 command_execution 事件写确定性检查。
  • Use Codex where rules fall short. Add a structured, rubric-based pass with --output-schema to grade style and conventions reliably.规则管不到的地方,交给 Codex。 用 --output-schema 加一轮结构化、基于评分细则的检查,稳定地给风格和约定打分。
  • Let real failures drive coverage. Every manual fix is a signal. Turn it into a test so the skill keeps getting it right.让真实的失败决定覆盖面。 每一次手动修正都是一个信号。把它变成一条测试,让 skill 一直做对。

最后更新: