> ## Documentation Index
> Fetch the complete documentation index at: https://cndoc-langchain.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 使用 openevals 包运行评估

> 使用开源的 openevals 和 agentevals 包与 LangSmith 一起运行评估。

LangSmith 与开源的 `openevals` 包集成，提供了一套评估工具和提示，你可以将其作为评估的起点。

<Note>
  本操作指南将演示如何设置和运行一种类型的评估器（LLM 作为评判者）。有关完整的评估工具和提示列表及使用示例，请参阅 [openevals](https://github.com/langchain-ai/openevals) 和 [agentevals](https://github.com/langchain-ai/agentevals) 仓库。
</Note>

## 设置

你需要安装 `openevals` 包才能使用 LLM 作为评判者的评估器。

<CodeGroup>
  ```bash Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  pip install -U openevals
  ```

  ```bash TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  yarn add openevals @langchain/core
  ```
</CodeGroup>

你还需要将 OpenAI API 密钥设置为环境变量，不过你也可以选择其他提供商：

```bash theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
export OPENAI_API_KEY="your_openai_api_key"
```

我们还将使用 LangSmith 的 [pytest](/langsmith/pytest) 集成（用于 Python）和 [Vitest/Jest](/langsmith/vitest-jest)（用于 TypeScript）来运行我们的评估。`openevals` 也能与 [`evaluate`](https://docs.smith.langchain.com/reference/python/evaluation/langsmith.evaluation._runner.evaluate) 方法无缝集成。有关设置说明，请参阅[相应的指南](/langsmith/pytest)。

## 运行评估器

一般流程很简单：从 `openevals` 导入评估器或工厂函数，然后在你的测试文件中使用输入、输出和参考输出来运行它。LangSmith 会自动将评估器的结果记录为反馈。

请注意，并非所有评估器都需要每个参数（例如，精确匹配评估器只需要输出和参考输出）。此外，如果你的 LLM 作为评判者的提示需要额外的变量，将它们作为关键字参数传递会将它们格式化到提示中。

按如下方式设置你的测试文件：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import pytest
  from langsmith import testing as t
  from openevals.llm import create_llm_as_judge
  from openevals.prompts import CORRECTNESS_PROMPT

  correctness_evaluator = create_llm_as_judge(
      prompt=CORRECTNESS_PROMPT,
      feedback_key="correctness",
      model="openai:o3-mini",
  )

  # 你的应用程序的模拟替代品
  def my_llm_app(inputs: dict) -> str:
      return "Doodads have increased in price by 10% in the past year."

  @pytest.mark.langsmith
  def test_correctness():
      inputs = "How much has the price of doodads changed in the past year?"
      reference_outputs = "The price of doodads has decreased by 50% in the past year."
      outputs = my_llm_app(inputs)

      t.log_inputs({"question": inputs})
      t.log_outputs({"answer": outputs})
      t.log_reference_outputs({"answer": reference_outputs})

      correctness_evaluator(
          inputs=inputs,
          outputs=outputs,
          reference_outputs=reference_outputs
      )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import * as ls from "langsmith/vitest";
  // import * as ls from "langsmith/jest";
  import { createLLMAsJudge, CORRECTNESS_PROMPT } from "openevals";

  const correctnessEvaluator = createLLMAsJudge({
      prompt: CORRECTNESS_PROMPT,
      feedbackKey: "correctness",
      model: "openai:o3-mini",
  });

  // 你的应用程序的模拟替代品
  const myLLMApp = async (_inputs: Record<string, unknown>) => {
      return "Doodads have increased in price by 10% in the past year.";
  };

  ls.describe("Correctness", () => {
      ls.test("incorrect answer", {
          inputs: {
              question: "How much has the price of doodads changed in the past year?"
          },
          referenceOutputs: {
              answer: "The price of doodads has decreased by 50% in the past year."
          }
      }, async ({ inputs, referenceOutputs }) => {
          const outputs = await myLLMApp(inputs);
          ls.logOutputs({ answer: outputs });
          await correctnessEvaluator({
              inputs,
              outputs,
              referenceOutputs,
          });
      });
  });
  ```
</CodeGroup>

`feedback_key`/`feedbackKey` 参数将用作你实验中反馈的名称。

在终端中运行评估将产生类似以下的结果：

<img src="https://mintcdn.com/other-405835d4/4uxGFmQj9AQhEaXK/langsmith/images/prebuilt-eval-result.png?fit=max&auto=format&n=4uxGFmQj9AQhEaXK&q=85&s=e83e683fc2990b23fc768b865b38034a" alt="预构建评估器终端结果" width="2114" height="614" data-path="langsmith/images/prebuilt-eval-result.png" />

如果你已经在 LangSmith 中创建了数据集，也可以将评估器直接传递给 `evaluate` 方法。如果使用 Python，这需要 `langsmith>=0.3.11`：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langsmith import Client
  from openevals.llm import create_llm_as_judge
  from openevals.prompts import CONCISENESS_PROMPT

  client = Client()
  conciseness_evaluator = create_llm_as_judge(
      prompt=CONCISENESS_PROMPT,
      feedback_key="conciseness",
      model="openai:o3-mini",
  )

  experiment_results = client.evaluate(
      # 这是一个虚拟的目标函数，请替换为你实际的基于 LLM 的系统
      lambda inputs: "What color is the sky?",
      data="Sample dataset",
      evaluators=[
          conciseness_evaluator
      ]
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { evaluate } from "langsmith/evaluation";
  import { createLLMAsJudge, CONCISENESS_PROMPT } from "openevals";

  const concisenessEvaluator = createLLMAsJudge({
      prompt: CONCISENESS_PROMPT,
      feedbackKey: "conciseness",
      model: "openai:o3-mini",
  });

  await evaluate((inputs) => "What color is the sky?", {
      data: datasetName,
      evaluators: [concisenessEvaluator],
  });
  ```
</CodeGroup>

有关可用评估工具和提示的完整列表，请参阅 [openevals](https://github.com/langchain-ai/openevals) 和 [agentevals](https://github.com/langchain-ai/agentevals) 仓库。

***

<div className="source-links">
  <Callout icon="terminal-2">
    [将这些文档](/use-these-docs)通过 MCP 连接到 Claude、VSCode 等，以获取实时答案。
  </Callout>

  <Callout icon="edit">
    [在 GitHub 上编辑此页面](https://github.com/langchain-ai/docs/edit/main/src/langsmith/openevals.mdx) 或 [提交问题](https://github.com/langchain-ai/docs/issues/new/choose)。
  </Callout>
</div>