Package xyz.felh.openai.jtokkit

Class TokenEncoder

java.lang.Object

xyz.felh.openai.jtokkit.TokenEncoder

public final class TokenEncoder
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DUMMY_RANK
static final int	MAX_RANK
static final String	VERY_LARGE_TOKENIZER_BYTE_THRESHOLD_KEY

Constructor Summary

Constructors

Constructor	Description
TokenEncoder(Map<byte[],Integer> encoder)

Method Summary

Modifier and Type	Method	Description
byte[]	decodeToken(int token, xyz.felh.openai.jtokkit.SpecialEncoder specialEncoder)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

VERY_LARGE_TOKENIZER_BYTE_THRESHOLD_KEY

public static final String VERY_LARGE_TOKENIZER_BYTE_THRESHOLD_KEY

See Also:

• Constant Field Values

DUMMY_RANK

public static final int DUMMY_RANK

See Also:

• Constant Field Values

MAX_RANK

public static final int MAX_RANK

See Also:

• Constant Field Values

Constructor Details

TokenEncoder

public TokenEncoder(Map<byte[],Integer> encoder)

Method Details

decodeToken

public byte[] decodeToken(int token,
 xyz.felh.openai.jtokkit.SpecialEncoder specialEncoder)