1

I'm trying to make a chatbot model using Tensorflow Javascript and I want to tokenize the text such that I can separate the context, the past messages and the current prompt to output a reply. So I was thinking that for the model to work properly it has to know the difference between each data I will input.

But the problem is that I can't figure out a way to add special tokens to my text using tfjs(this is possible in python using beet tokenizer). So I would really appreciate your help on this issue. Looking forward to your reply guys😊

I checked out the Universal Sentence Encoder, and it works perfectly in tokenizing text, but I'm still struggling with the adding special tokens.

Improve this question
1
  • Please provide enough code so others can better understand or reproduce the problem. Commented Apr 18, 2023 at 2:41
Related questions
1
React Native with TypeScript tfjs-models/universal-sentence-encoder error on model load
0
How to replace BERT tokenizer special tokens
0
Textual representation of LaBSE preprocessor output?
Load 4 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.