> ## Documentation Index
> Fetch the complete documentation index at: https://cndoc-langchain.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 如何定义汇总评估器

某些指标只能在整个实验级别上定义，而不是针对实验的单次运行。例如，您可能希望计算评估目标在整个数据集所有示例上的总体通过率或F1分数。这些被称为汇总评估器。

## 基本示例

在此，我们将计算F1分数，它是精确率和召回率的组合。

这类指标只能在实验的所有示例上计算，因此我们的评估器接收一个输出列表和一个参考输出列表。

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  def f1_score_summary_evaluator(outputs: list[dict], reference_outputs: list[dict]) -> dict:
      true_positives = 0
      false_positives = 0
      false_negatives = 0

      for output_dict, reference_output_dict in zip(outputs, reference_outputs):
          output = output_dict["class"]
          reference_output = reference_output_dict["class"]

          if output == "Toxic" and reference_output == "Toxic":
              true_positives += 1
          elif output == "Toxic" and reference_output == "Not toxic":
              false_positives += 1
          elif output == "Not toxic" and reference_output == "Toxic":
              false_negatives += 1

      if true_positives == 0:
          return {"key": "f1_score", "score": 0.0}

      precision = true_positives / (true_positives + false_positives)
      recall = true_positives / (true_positives + false_negatives)
      f1_score = 2 * (precision * recall) / (precision + recall)

      return {"key": "f1_score", "score": f1_score}
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  function f1ScoreSummaryEvaluator({ outputs, referenceOutputs }: {
      outputs: Record<string, any>[],
      referenceOutputs: Record<string, any>[]
  }) {
      let truePositives = 0;
      let falsePositives = 0;
      let falseNegatives = 0;

      for (let i = 0; i < outputs.length; i++) {
          const output = outputs[i]["class"];
          const referenceOutput = referenceOutputs[i]["class"];

          if (output === "Toxic" && referenceOutput === "Toxic") {
              truePositives += 1;
          } else if (output === "Toxic" && referenceOutput === "Not toxic") {
              falsePositives += 1;
          } else if (output === "Not toxic" && referenceOutput === "Toxic") {
              falseNegatives += 1;
          }
      }

      if (truePositives === 0) {
          return { key: "f1_score", score: 0.0 };
      }

      const precision = truePositives / (truePositives + falsePositives);
      const recall = truePositives / (truePositives + falseNegatives);
      const f1Score = 2 * (precision * recall) / (precision + recall);

      return { key: "f1_score", score: f1Score };
  }
  ```
</CodeGroup>

然后，您可以按如下方式将此评估器传递给 `evaluate` 方法：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langsmith import Client

  ls_client = Client()
  dataset = ls_client.clone_public_dataset(
      "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d"
  )

  def bad_classifier(inputs: dict) -> dict:
      return {"class": "Not toxic"}

  def correct(outputs: dict, reference_outputs: dict) -> bool:
      """行级正确性评估器。"""
      return outputs["class"] == reference_outputs["label"]

  results = ls_client.evaluate(
      bad_classified,
      data=dataset,
      evaluators=[correct],
      summary_evaluators=[pass_50],
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { Client } from "langsmith";
  import { evaluate } from "langsmith/evaluation";
  import type { EvaluationResult } from "langsmith/evaluation";

  const client = new Client();
  const datasetName = "Toxic queries";
  const dataset = await client.clonePublicDataset(
      "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d",
      { datasetName: datasetName }
  );

  function correct({ outputs, referenceOutputs }: {
      outputs: Record<string, any>,
      referenceOutputs?: Record<string, any>
  }): EvaluationResult {
      const score = outputs["class"] === referenceOutputs?.["label"];
      return { key: "correct", score };
  }

  function badClassifier(inputs: Record<string, any>): { class: string } {
      return { class: "Not toxic" };
  }

  await evaluate(badClassifier, {
      data: datasetName,
      evaluators: [correct],
      summaryEvaluators: [summaryEval],
      experimentPrefix: "Toxic Queries",
  });
  ```
</CodeGroup>

在 LangSmith UI 中，您将看到汇总评估器的分数及其对应的键显示出来。

<img src="https://mintcdn.com/other-405835d4/ozzEsp14Hd9jH9J-/langsmith/images/summary-eval.png?fit=max&auto=format&n=ozzEsp14Hd9jH9J-&q=85&s=21014858e578e17e4334d524af8ab2cc" alt="summary_eval.png" width="1535" height="122" data-path="langsmith/images/summary-eval.png" />

## 汇总评估器参数

汇总评估器函数必须具有特定的参数名称。它们可以接受以下参数的任意子集：

* `inputs: list[dict]`：对应于数据集中单个示例的输入列表。
* `outputs: list[dict]`：每个实验在给定输入上产生的字典输出列表。
* `reference_outputs/referenceOutputs: list[dict]`：与示例关联的参考输出列表（如果可用）。
* `runs: list[Run]`：两个实验在给定示例上生成的完整 [Run](/langsmith/run-data-format) 对象列表。如果您需要访问中间步骤或每次运行的元数据，请使用此参数。
* `examples: list[Example]`：所有数据集 [Example](/langsmith/example-data-format) 对象，包括示例输入、输出（如果可用）和元数据（如果可用）。

## 汇总评估器输出

汇总评估器应返回以下类型之一：

Python 和 JS/TS

* `dict`：形式为 `{"score": ..., "name": ...}` 的字典允许您传递数值或布尔分数以及指标名称。

目前仅限 Python

* `int | float | bool`：这被解释为一个连续指标，可以进行平均、排序等操作。函数名称用作指标的名称。

***

<div className="source-links">
  <Callout icon="terminal-2">
    [将这些文档连接](/use-these-docs)到 Claude、VSCode 等，通过 MCP 获取实时答案。
  </Callout>

  <Callout icon="edit">
    [在 GitHub 上编辑此页面](https://github.com/langchain-ai/docs/edit/main/src/langsmith/summary.mdx) 或 [提交问题](https://github.com/langchain-ai/docs/issues/new/choose)。
  </Callout>
</div>