Skip to main content
许多聊天或问答应用程序在嵌入和向量存储之前需要对输入文档进行分块。 来自 Pinecone 的这些说明提供了一些有用的提示： 
When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.
如前所述，分块通常旨在将具有共同上下文的文本保持在一起。考虑到这一点，我们可能希望专门遵循文档本身的结构。例如，Markdown 文件按标题组织。在特定标题组内创建块是一种直观的想法。为了解决这一挑战，我们可以使用 MarkdownHeaderTextSplitter。它将按照一组指定的标题分割 Markdown 文件。 例如，如果我们想分割以下 Markdown： 
md = '# Foo\n\n ## Bar\n\nHi this is Jim  \nHi this is Joe\n\n ## Baz\n\n Hi this is Molly'
我们可以指定要分割的标题： 
[("#", "Header 1"),("##", "Header 2")]
内容将按公共标题分组或分割： 
{'content': 'Hi this is Jim  \nHi this is Joe', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Bar'}}
{'content': 'Hi this is Molly', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Baz'}}
下面让我们看一些示例。

​
基本用法：

pip install -qU langchain-text-splitters

from langchain_text_splitters import MarkdownHeaderTextSplitter

markdown_document = "# Foo\n\n    ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits

[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim  \nHi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]

type(md_header_splits[0])

langchain_core.documents.base.Document
默认情况下，MarkdownHeaderTextSplitter 会从输出块内容中去除被分割的标题。可以通过设置 strip_headers = False 来禁用此行为。 
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits

[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='# Foo  \n## Bar  \nHi this is Jim  \nHi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='### Boo  \nHi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='## Baz  \nHi this is Molly')]
默认的 MarkdownHeaderTextSplitter 会去除空格和换行符。要保留 Markdown 文档的原始格式，请参阅 ExperimentalMarkdownSyntaxTextSplitter。

​
如何将 Markdown 行作为单独文档返回

默认情况下，MarkdownHeaderTextSplitter 根据 headers_to_split_on 中指定的标题聚合行。我们可以通过指定 return_each_line 来禁用此行为： 
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on,
    return_each_line=True,
)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits

[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]
注意，这里标题信息保留在每个文档的 metadata 中。

​
如何限制块大小：

在每个 Markdown 组内，我们可以应用任何我们想要的文本分割器，例如 RecursiveCharacterTextSplitter，它可以进一步控制块大小。 
markdown_document = "# Intro \n\n    ## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
]

# MD splits
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on, strip_headers=False
)
md_header_splits = markdown_splitter.split_text(markdown_document)

# Char-level splits
from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 250
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(md_header_splits)
splits

[Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='# Intro  \n## History  \nMarkdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9]'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='## Rise and divergence  \nAs Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for  \nadditional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='#### Standardization  \nFrom 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Implementations'}, page_content='## Implementations  \nImplementations of Markdown are available for over a dozen programming languages.')]

​
疑难解答：chunk_overlap 似乎不生效

  • 在基于标题的分割（如 MarkdownHeaderTextSplitter）之后，使用 split_documents(docs)（而非 split_text），这样重叠才会在每个章节内应用，并且每个章节的元数据（标题）会保留在块中。
  • 重叠仅在单个章节超过 chunk_size 并被分割为多个块时出现。
  • 重叠不会跨越章节/文档边界（例如 # H1 → ## H2）。
  • 如果标题变成了一个微小的第一块，请考虑将 strip_headers 设置为 True，以避免标题行成为独立的块。
  • 如果您的文本缺少换行符/空格，请在 separators 中保留一个回退的 ""，以便分割器仍然可以分割并应用重叠。
在 GitHub 上编辑此页面 或提交问题。
连接这些文档 到 Claude、VSCode 等，通过 MCP 获取实时答案。