Package com.github.hakenadu.javalangchains.chains.qa.split

Class TiktokenTextSplitter

java.lang.Object

com.github.hakenadu.javalangchains.chains.qa.split.MaxLengthBasedTextSplitter

com.github.hakenadu.javalangchains.chains.qa.split.TiktokenTextSplitter

All Implemented Interfaces:

TextSplitter

public final class TiktokenTextSplitter
extends MaxLengthBasedTextSplitter

This TextSplitter splits documents based on their tiktoken token count. For that purpose jtokkit is utilized.

Constructor Summary

Constructors

Constructor	Description
TiktokenTextSplitter(com.knuddels.jtokkit.api.Encoding encoding, int maxTokens)	creates an instance of TiktokenTextSplitter with sentence based text streaming
TiktokenTextSplitter(com.knuddels.jtokkit.api.Encoding encoding, int maxTokens, TextStreamer textStreamer)	creates an instance of TiktokenTextSplitter

Method Summary

Modifier and Type	Method	Description
protected int	getLength(String textPart)	provide the length value for a text part

Methods inherited from class com.github.hakenadu.javalangchains.chains.qa.split.MaxLengthBasedTextSplitter

split

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TiktokenTextSplitter

public TiktokenTextSplitter(com.knuddels.jtokkit.api.Encoding encoding, int maxTokens, TextStreamer textStreamer)

creates an instance of TiktokenTextSplitter

Parameters:

encoding - encoding

maxTokens - max amount of tokens for each chunk

textStreamer - the TextStreamer used for streaming the base text

TiktokenTextSplitter

public TiktokenTextSplitter(com.knuddels.jtokkit.api.Encoding encoding, int maxTokens)

creates an instance of TiktokenTextSplitter with sentence based text streaming

Parameters:

encoding - encoding

maxTokens - max amount of tokens for each chunk

Method Details

getLength

protected int getLength(String textPart)

Description copied from class: MaxLengthBasedTextSplitter

provide the length value for a text part

Specified by:

getLength in class MaxLengthBasedTextSplitter

Parameters:

textPart - the text part which needs to be measured

Returns:

the length for the passed textPart