> ## Documentation Index > Fetch the complete documentation index at: https://langchain-docs-zh.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # 按 Token 分割 - 文本分割器集成指南 > 语言模型有 Token 限制。您不应超过 Token 限制。当您 将文本分割 成块时，因此计算 Token 数量是一个好主意。有很多 Tokenizer。当您在文本中计算 Token 时，您应该使用与语言模型中相同的 Tokenizer。 语言模型有 Token 限制。您不应超过 Token 限制。当您 [将文本分割](https://langchain-docs-zh.mintlify.site/langchain/oss/integrations/splitters.md) 成块时，因此计算 Token 数量是一个好主意。有很多 Tokenizer。当您在文本中计算 Token 时，您应该使用与语言模型中相同的 Tokenizer。 :::python ## tiktoken > [!NOTE] > > **[tiktoken](https://github.com/openai/tiktoken) 是由 `OpenAI` 创建的一个快速 `BPE` Tokenizer。** 我们可以使用 `tiktoken` 来估计使用的 Token。它可能对 OpenAI 模型更准确。 1. 文本是如何分割的：通过传入的字符。 2. 块大小是如何测量的：通过 `tiktoken` Tokenizer。 @\[CharacterTextSplitter], @\[RecursiveCharacterTextSplitter], 和 @\[TokenTextSplitter] 可以直接与 `tiktoken` 一起使用。 ```python theme={null} pip install --upgrade --quiet langchain-text-splitters tiktoken ``` ```python theme={null} from langchain_text_splitters import CharacterTextSplitter # This is a long document we can split up. with open("state_of_the_union.txt") as f: state_of_the_union = f.read() ``` 要使用 @\[CharacterTextSplitter] 分割，然后使用 `tiktoken` 合并块，请使用它的 `.from_tiktoken_encoder()` 方法。请注意，此方法的分割可能大于由 `tiktoken` Tokenizer 测量的块大小。 `.from_tiktoken_encoder()` 方法接受 `encoding_name` 作为参数（例如 `cl100k_base`），或接受 `model_name`（例如 `gpt-4`）。所有额外的参数，如 `chunk_size`、`chunk_overlap` 和 `separators`，用于实例化 `CharacterTextSplitter`： ```python theme={null} text_splitter = CharacterTextSplitter.from_tiktoken_encoder( encoding_name="cl100k_base", chunk_size=100, chunk_overlap=0 ) texts = text_splitter.split_text(state_of_the_union) ``` ```python theme={null} print(texts[0]) ``` ```text theme={null} Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. ``` 要实现块大小的硬约束，我们可以使用 `RecursiveCharacterTextSplitter.from_tiktoken_encoder`，其中每个分割如果大小较大，将被递归分割： ```python theme={null} from langchain_text_splitters import RecursiveCharacterTextSplitter text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder( model_name="gpt-4", chunk_size=100, chunk_overlap=0, ) ``` 我们还可以加载一个 `TokenTextSplitter` 分割器，它可以直接与 `tiktoken` 一起使用，并确保每个分割都小于块大小。 ```python theme={null} from langchain_text_splitters import TokenTextSplitter text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0) texts = text_splitter.split_text(state_of_the_union) print(texts[0]) ``` ```text theme={null} Madam Speaker, Madam Vice President, our ``` 某些书写语言（例如中文和日语）有字符，这些字符编码为两个或多个 Token。直接使用 `TokenTextSplitter` 可能会在两个块之间分割字符的 Token，导致 Unicode 字符格式错误。使用 `RecursiveCharacterTextSplitter.from_tiktoken_encoder` 或 `CharacterTextSplitter.from_tiktoken_encoder` 确保块包含有效的 Unicode 字符串。 ::: :::js ## js-tiktoken > [!NOTE] > > **[js-tiktoken](https://github.com/dqbd/tiktoken) 是由 `OpenAI` 创建的 `BPE` Tokenizer 的 JavaScript 版本。** 我们可以使用 `tiktoken` 和 @\[TokenTextSplitter] 来估计使用的 Token。它可能对 OpenAI 模型更准确。 1. 文本是如何分割的：通过传入的字符。 2. 块大小是如何测量的：通过 `tiktoken` Tokenizer。 ```bash npm theme={null} npm install @langchain/textsplitters ``` ```bash pnpm theme={null} pnpm install @langchain/textsplitters ``` ```bash yarn theme={null} yarn add @langchain/textsplitters ``` ```bash bun theme={null} bun add @langchain/textsplitters ``` ```ts theme={null} import { TokenTextSplitter } from "@langchain/textsplitters"; import { readFileSync } from "fs"; // Example: read a long document const stateOfTheUnion = readFileSync("state_of_the_union.txt", "utf8"); ``` 要使用 @\[TokenTextSplitter] 分割，然后使用 `tiktoken` 合并块，在初始化 @\[TokenTextSplitter] 时传入一个 `encodingName`（例如 cl100k\_base）。请注意，此方法的分割可能大于由 `tiktoken` Tokenizer 测量的块大小。 ```ts theme={null} import { TokenTextSplitter } from "@langchain/textsplitters"; // Example: use cl100k_base encoding const splitter = new TokenTextSplitter({ encodingName: "cl100k_base", chunkSize: 10, chunkOverlap: 0 }); const texts = splitter.splitText(stateOfTheUnion); console.log(texts[0]); ``` ```text theme={null} Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. ``` ::: :::python ## spaCy > [!NOTE] > > [spaCy](https://spacy.io/) 是一个用于高级自然语言处理的开源软件库，用 Python 和 Cython 编写。 LangChain 实现了基于 [spaCy Tokenizer](https://spacy.io/api/tokenizer) 的分割器。 1. 文本是如何分割的：通过 `spaCy` Tokenizer。 2. 块大小是如何测量的：通过字符数量。 ```python theme={null} pip install --upgrade --quiet spacy ``` ```python theme={null} # This is a long document we can split up. with open("state_of_the_union.txt") as f: state_of_the_union = f.read() ``` ```python theme={null} from langchain_text_splitters import SpacyTextSplitter text_splitter = SpacyTextSplitter(chunk_size=1000) texts = text_splitter.split_text(state_of_the_union) print(texts[0]) ``` ```text theme={null} Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. ``` ## SentenceTransformers [`SentenceTransformersTokenTextSplitter`](https://reference.langchain.com/python/langchain-text-splitters/sentence_transformers/SentenceTransformersTokenTextSplitter) 是一个专门用于与句子转换器模型一起使用的文本分割器。默认行为是将文本分割成适合您希望使用的句子转换器模型的 Token 窗口的块。 要分割文本并根据句子转换器 Tokenizer 限制 Token 计数，请实例化一个 `SentenceTransformersTokenTextSplitter`。您可以可选地指定： * `chunk_overlap`：Token 重叠的整数计数； * `model_name`：句子转换器模型名称，默认为 `"sentence-transformers/all-mpnet-base-v2"`； * `tokens_per_chunk`：每个块所需的 Token 计数。 ```python theme={null} from langchain_text_splitters import SentenceTransformersTokenTextSplitter splitter = SentenceTransformersTokenTextSplitter(chunk_overlap=0) text = "Lorem " count_start_and_stop_tokens = 2 text_token_count = splitter.count_tokens(text=text) - count_start_and_stop_tokens print(text_token_count) ``` ```text theme={null} 2 ``` ```python theme={null} token_multiplier = splitter.maximum_tokens_per_chunk // text_token_count + 1 # `text_to_split` does not fit in a single chunk text_to_split = text * token_multiplier print(f"tokens in text to split: {splitter.count_tokens(text=text_to_split)}") ``` ```text theme={null} tokens in text to split: 514 ``` ```python theme={null} text_chunks = splitter.split_text(text=text_to_split) print(text_chunks[1]) ``` ```text theme={null} lorem ``` ## NLTK > [!NOTE] > > **[自然语言工具包](https://en.wikipedia.org/wiki/Natural_Language_Toolkit)，或更常见的 [NLTK](https://www.nltk.org/)，是一套用于英语符号和统计自然语言处理（NLP）的库和程序。** 我们不仅可以基于 "\n\n" 分割，还可以使用 `NLTK` 基于 [NLTK Tokenizer](https://www.nltk.org/api/nltk.tokenize.html) 分割。 1. 文本是如何分割的：通过 `NLTK` Tokenizer。 2. 块大小是如何测量的：通过字符数量。 ```python theme={null} # pip install nltk ``` ```python theme={null} # This is a long document we can split up. with open("state_of_the_union.txt") as f: state_of_the_union = f.read() ``` ```python theme={null} from langchain_text_splitters import NLTKTextSplitter text_splitter = NLTKTextSplitter(chunk_size=1000) ``` ```python theme={null} texts = text_splitter.split_text(state_of_the_union) print(texts[0]) ``` ```text theme={null} Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. ``` ## KoNLPY > [!NOTE] > > [KoNLPy: Python 中的韩语 NLP](https://konlpy.org/en/latest/) 是一个用于韩语自然语言处理（NLP）的 Python 包。 Token 分割涉及将文本分割成更小的、更易于管理的单元，称为 Token。这些 Token 通常是单词、短语、符号或其他对进一步处理和分析至关重要的有意义元素。在英语等语言中，Token 分割通常涉及通过空格和标点符号分隔单词。Token 分割的有效性很大程度上取决于 Tokenizer 对语言结构的理解，确保生成有意义的 Token。由于为英语设计的 Tokenizer 无法理解其他语言（如韩语）的独特语义结构，因此它们不能有效地用于韩语语言处理。 ### 使用 KoNLPY 的 kkma 分析器进行韩语 Token 分割 对于韩语文本，KoNLPY 包含一个称为 `Kkma`（韩语知识词素分析器）的形态分析器。`Kkma` 提供了韩语文本的详细形态分析。它将句子分解成单词，并将单词分解成相应的词素，为每个 Token 识别词性。它可以分割文本块成单个句子，这对于处理长文本特别有用。 ### 使用注意事项 虽然 `Kkma` 以其详细分析而闻名，但请注意，这种精确度可能会影响处理速度。因此，`Kkma` 最适合分析深度优先于快速文本处理的应用。 ```python theme={null} # pip install konlpy ``` ```python theme={null} # This is a long Korean document that we want to split up into its component sentences. with open("./your_korean_doc.txt") as f: korean_document = f.read() ``` ```python theme={null} from langchain_text_splitters import KonlpyTextSplitter text_splitter = KonlpyTextSplitter() ``` ```python theme={null} texts = text_splitter.split_text(korean_document) # The sentences are split with "\n\n" characters. print(texts[0]) ``` ```text theme={null} 춘향전 옛날에 남원에 이 도령이라는 벼슬아치 아들이 있었다. 그의 외모는 빛나는 달처럼 잘생겼고, 그의 학식과 기예는 남보다 뛰어났다. 한편, 이 마을에는 춘향이라는 절세 가인이 살고 있었다. 춘 향의 아름다움은 꽃과 같아 마을 사람들 로부터 많은 사랑을 받았다. 어느 봄날, 도령은 친구들과 놀러 나갔다가 춘 향을 만 나 첫 눈에 반하고 말았다. 두 사람은 서로 사랑하게 되었고, 이내 비밀스러운 사랑의 맹세를 나누었다. 하지만 좋은 날들은 오래가지 않았다. 도령의 아버지가 다른 곳으로 전근을 가게 되어 도령도 떠나 야만 했다. 이별의 아픔 속에서도, 두 사람은 재회를 기약하며 서로를 믿고 기다리기로 했다. 그러나 새로 부임한 관아의 사또가 춘 향의 아름다움에 욕심을 내 어 그녀에게 강요를 시작했다. 춘 향 은 도령에 대한 자신의 사랑을 지키기 위해, 사또의 요구를 단호히 거절했다. 이에 분노한 사또는 춘 향을 감옥에 가두고 혹독한 형벌을 내렸다. 이야기는 이 도령이 고위 관직에 오른 후, 춘 향을 구해 내는 것으로 끝난다. 두 사람은 오랜 시련 끝에 다시 만나게 되고, 그들의 사랑은 온 세상에 전해 지며 후세에까지 이어진다. - 춘향전 (The Tale of Chunhyang) ``` ## Hugging Face 分词器 [Hugging Face](https://huggingface.co/docs/tokenizers/index) 有很多 Tokenizer。 我们使用 Hugging Face tokenizer，即 [GPT2TokenizerFast](https://huggingface.co/Ransaka/gpt2-tokenizer-fast) 来计算文本的 Token 长度。 1. 文本是如何分割的：通过传入的字符。 2. 块大小是如何测量的：通过 `Hugging Face` Tokenizer 计算的 Token 数量。 ```python theme={null} from transformers import GPT2TokenizerFast tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") ``` ```python theme={null} # This is a long document we can split up. with open("state_of_the_union.txt") as f: state_of_the_union = f.read() from langchain_text_splitters import CharacterTextSplitter ``` ```python theme={null} text_splitter = CharacterTextSplitter.from_huggingface_tokenizer( tokenizer, chunk_size=100, chunk_overlap=0 ) texts = text_splitter.split_text(state_of_the_union) ``` ```python theme={null} print(texts[0]) ``` ```text theme={null} Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. ``` ::: *** > 官方原文：[src/oss/integrations/splitters/split\_by\_token.mdx](https://github.com/langchain-ai/docs/blob/main/src/oss/integrations/splitters/split_by_token.mdx) · 非官方翻译，译文由 AI 生成，以英文原文为准。 This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.