Function: cjkSentenceTokenizer()

cjkSentenceTokenizer(sentence): string[]

Tokenizes sentences. Suitable for Chinese, Japanese, and Korean. Use instead of defaultSentenceTokenizer.

Parameters

• sentence: string

Returns

string[]

string[]

Source

packages/llamaindex/src/TextSplitter.ts:41