Package com.github.hakenadu.javalangchains.chains.qa.split

Class MaxLengthBasedTextSplitter

java.lang.Object

com.github.hakenadu.javalangchains.chains.qa.split.MaxLengthBasedTextSplitter

All Implemented Interfaces:

TextSplitter

Direct Known Subclasses:

TiktokenTextSplitter

public abstract class MaxLengthBasedTextSplitter
extends Object
implements TextSplitter

abstract base class for TextSplitter implementations that use a max length for deciding when to split

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	MaxLengthBasedTextSplitter(int maxLength)	creates a MaxLengthBasedTextSplitter using sentence wise text streaming
protected	MaxLengthBasedTextSplitter(int maxLength, TextStreamer textStreamer)

Method Summary

Modifier and Type	Method	Description
protected abstract int	getLength(String textPart)	provide the length value for a text part
List<String>	split(String text)	Splits a text into one or more subtexts

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MaxLengthBasedTextSplitter

protected MaxLengthBasedTextSplitter(int maxLength, TextStreamer textStreamer)

Parameters:

maxLength - maxLength

textStreamer - textStreamer

MaxLengthBasedTextSplitter

protected MaxLengthBasedTextSplitter(int maxLength)

creates a MaxLengthBasedTextSplitter using sentence wise text streaming

Parameters:

maxLength - maxLength

Method Details

getLength

protected abstract int getLength(String textPart)

provide the length value for a text part

Parameters:

textPart - the text part which needs to be measured

Returns:

the length for the passed textPart

split

public final List<String> split(String text)

Description copied from interface: TextSplitter

Splits a text into one or more subtexts

Specified by:

split in interface TextSplitter

Parameters:

text - text to split

Returns:

List with text partitions