> ## Documentation Index > Fetch the complete documentation index at: https://langchain-docs-zh.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # AI21SemanticTextSplitter 集成 > 使用 LangChain Python 集成 AI21SemanticTextSplitter 文档转换器。 本示例演示如何在LangChain中使用AI21SemanticTextSplitter。 ## 安装 ```python theme={null} pip install langchain-ai21 ``` ## 环境配置 我们需要获取AI21 API密钥并设置 `chunk_size` 环境变量： ```python theme={null} import os from getpass import getpass if "AI21_API_KEY" not in os.environ: os.environ["AI21_API_KEY"] = getpass() ``` ## 示例用法 ### 按语义含义分割文本 本示例演示如何使用AI21SemanticTextSplitter根据语义含义将文本分割为块。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter = AI21SemanticTextSplitter() chunks = semantic_text_splitter.split_text(TEXT) print(f"The text has been split into {len(chunks)} chunks.") for chunk in chunks: print(chunk) print("====") ``` ### 带合并的语义分割 本示例演示如何使用 `chunk_size` AI21SemanticTextSplitter将文本按语义分割后合并块。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter_chunks = AI21SemanticTextSplitter(chunk_size=1000) chunks = semantic_text_splitter_chunks.split_text(TEXT) print(f"The text has been split into {len(chunks)} chunks.") for chunk in chunks: print(chunk) print("====") ``` ### 将文本分割为文档 本示例演示如何使用AI21SemanticTextSplitter根据语义含义将文本分割为文档。每个文档的元数据将包含类型标识。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter = AI21SemanticTextSplitter() documents = semantic_text_splitter.split_text_to_documents(TEXT) print(f"The text has been split into {len(documents)} Documents.") for doc in documents: print(f"type: {doc.metadata['source_type']}") print(f"text: {doc.page_content}") print("====") ``` ### 带元数据的文档创建 本示例演示如何使用AI21SemanticTextSplitter从文本创建文档，并为每个文档添加自定义元数据。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter = AI21SemanticTextSplitter() texts = [TEXT] documents = semantic_text_splitter.create_documents( texts=texts, metadatas=[{"pikachu": "pika pika"}] ) print(f"The text has been split into {len(documents)} Documents.") for doc in documents: print(f"metadata: {doc.metadata}") print(f"text: {doc.page_content}") print("====") ``` ### 带起始索引的文档分割 本示例演示如何使用AI21SemanticTextSplitter根据语义含义将文本分割为文档。每个文档的元数据将包含起始索引。 **注意**：起始索引仅表示块之间的顺序而非实际起始位置。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter = AI21SemanticTextSplitter(add_start_index=True) documents = semantic_text_splitter.create_documents(texts=[TEXT]) print(f"The text has been split into {len(documents)} Documents.") for doc in documents: print(f"start_index: {doc.metadata['start_index']}") print(f"text: {doc.page_content}") print("====") ``` ### 文档分割 本示例演示如何使用AI21SemanticTextSplitter根据语义含义将文档列表分割为块。 ```python theme={null} from langchain_ai21 import AI21SemanticTextSplitter from langchain_core.documents import Document TEXT = ( "We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, " "legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\n" "Imagine a company that employs hundreds of thousands of employees. In today's information " "overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise " "here, given that some of these documents are long and convoluted on purpose (did you know that " "reading through all your privacy policies would take almost a quarter of a year?). Aside from " "inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of " "Employees Read Their Employment Contracts Entirely Before Signing!).\nThis is where AI-driven summarization " "tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, " "users can (ideally) quickly extract relevant information from a text. With large language models, " "the development of those tools is easier than ever, and you can offer your users a summary that is " "specifically tailored to their preferences.\nLarge language models naturally follow patterns in input " "(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed " 'them with several examples in the input ("few-shot prompt"), so they can follow through. ' "The process of creating the correct prompt for your problem is called prompt engineering, " "and you can read more about it here." ) semantic_text_splitter = AI21SemanticTextSplitter() document = Document(page_content=TEXT, metadata={"hello": "goodbye"}) documents = semantic_text_splitter.split_documents([document]) print(f"The document list has been split into {len(documents)} Documents.") for doc in documents: print(f"text: {doc.page_content}") print(f"metadata: {doc.metadata}") print("====") ``` *** > 官方原文：[src/oss/python/integrations/document\_transformers/ai21\_semantic\_text\_splitter.mdx](https://github.com/langchain-ai/docs/blob/main/src/oss/python/integrations/document_transformers/ai21_semantic_text_splitter.mdx) · 非官方翻译，译文由 AI 生成，以英文原文为准。 This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.