> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-docs-zh.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 分割 Markdown - 文本分割器集成

> 使用 LangChain Python 集成 Split markdown 文本分割器。

许多对话或问答应用在嵌入和向量化存储之前需要对输入文档进行分块处理。

[这些笔记](https://www.pinecone.io/learn/chunking-strategies/)来自Pinecone提供了一些实用建议：

```wrap theme={null}
When a full paragraph or document is embedded, the embedding process considers both the overall context and the relationships between the sentences and phrases within the text. This can result in a more comprehensive vector representation that captures the broader meaning and themes of the text.
```

如前所述，分块的目标是将具有共同上下文的内容保留在一起。基于此，我们可能希望特别尊重文档自身的结构。例如，Markdown文件通过标题组织内容。在特定标题组内创建分块是一个直观的想法。为解决此挑战，我们可以使用[MarkdownHeaderTextSplitter](https://reference.langchain.com/python/langchain-text-splitters/markdown/MarkdownHeaderTextSplitter)。这将根据指定的标题集对Markdown文件进行分割。

例如，如果我们想分割此Markdown：

```markdown theme={null}
md = '# Foo\n\n ## Bar\n\nHi this is Jim  \nHi this is Joe\n\n ## Baz\n\n Hi this is Molly'
```

我们可以指定要分割的标题：

```python theme={null}
[("#", "Header 1"),("##", "Header 2")]
```

内容将根据共同标题进行分组或分割：

```
{'content': 'Hi this is Jim  \nHi this is Joe', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Bar'}}
{'content': 'Hi this is Molly', 'metadata': {'Header 1': 'Foo', 'Header 2': 'Baz'}}
```

让我们来看一些示例。

### 基础用法：

```python theme={null}
pip install -qU langchain-text-splitters
```

```python theme={null}
from langchain_text_splitters import MarkdownHeaderTextSplitter
```

```python theme={null}
markdown_document = "# Foo\n\n    ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits
```

```text theme={null}
[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim  \nHi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]
```

```python theme={null}
type(md_header_splits[0])
```

```text theme={null}
langchain_core.documents.base.Document
```

默认情况下，`MarkdownHeaderTextSplitter`会从输出分块的文档内容中移除被分割的标题。此行为可通过设置`strip_headers = False`禁用。

```python theme={null}
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits
```

```text theme={null}
[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='# Foo  \n## Bar  \nHi this is Jim  \nHi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='### Boo  \nHi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='## Baz  \nHi this is Molly')]
```

> [!NOTE]
>
> **默认的`MarkdownHeaderTextSplitter`会移除空白和换行符。如需保留原始Markdown格式，请参考[`ExperimentalMarkdownSyntaxTextSplitter`](https://reference.langchain.com/python/langchain-text-splitters/markdown/ExperimentalMarkdownSyntaxTextSplitter)。**

### 如何将Markdown行作为独立文档返回

默认情况下，`MarkdownHeaderTextSplitter`会根据`headers_to_split_on`中指定的标题进行行聚合。可通过指定`return_each_line`禁用此行为：

```python theme={null}
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on,
    return_each_line=True,
)
md_header_splits = markdown_splitter.split_text(markdown_document)
md_header_splits
```

```text theme={null}
[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Joe'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]
```

注意此处`metadata`会为每个文档保留标题信息。

### 如何约束分块大小：

在每个Markdown组内，我们可以应用任何文本分割器，例如`RecursiveCharacterTextSplitter`，以进一步控制分块大小。

```python theme={null}
markdown_document = "# Intro \n\n    ## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
]

# MD splits
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on, strip_headers=False
)
md_header_splits = markdown_splitter.split_text(markdown_document)

# Char-level splits
from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 250
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(md_header_splits)
splits
```

```text theme={null}
[Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='# Intro  \n## History  \nMarkdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9]'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='## Rise and divergence  \nAs Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for  \nadditional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='#### Standardization  \nFrom 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort.'),
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Implementations'}, page_content='## Implementations  \nImplementations of Markdown are available for over a dozen programming languages.')]
```

## 故障排除：`chunk_overlap`似乎未生效

* 在基于标题的分割后（例如`MarkdownHeaderTextSplitter`），请使用\*\*`split_documents(docs)`**（而非`split_text`）以确保重叠应用于**每个章节\*\*，并保留章节级元数据（标题）。
* 重叠仅在单个章节超过`chunk_size`并被分割为多个分块时生效。
* 重叠**不会跨越**章节/文档边界（例如`# H1` → `## H2`）。
* 若标题成为极小的独立分块，请将`strip_headers`设置为`True`以避免标题行成为独立分块。
* 若文本缺乏换行/空白，请确保在`separators`中保留备用`""`，使分割器仍能执行分块和重叠处理。

***

> 官方原文：[src/oss/python/integrations/splitters/markdown\_header\_metadata\_splitter.mdx](https://github.com/langchain-ai/docs/blob/main/src/oss/python/integrations/splitters/markdown_header_metadata_splitter.mdx) · 非官方翻译，译文由 AI 生成，以英文原文为准。


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.