Uncategorized

Is there any port of python library “transformers.tokenization_auto” to java?


I’m trying to instantiate “intfloat/multilingual-e5-large” in jvm.
To do this, I’ll use this library:

<dependency>
    <groupId>com.microsoft.onnxruntime</groupId>
    <artifactId>onnxruntime</artifactId>
    <version>1.15.1</version>

This model requires tokenizer and contains configuration for this.

onnxruntime does not contain tools for such things. I only know about transformers.tokenization_auto here. Maybe there’s some sort of java version of this library, or some alternatives?



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *