목록로 토큰나이징 하기 (1)

무회blog

python: pytorch,Bert, 로 토큰나이징 하기

In [1]: import torch from transformers import AutoModel,AutoTokenizer, BertTokenizer print(torch.__version__) torch.set_grad_enabled(False) 1.6.0 Out[1]: In [2]: # 모델 저장하기 , Store the model we want to use MODEL_NAME = "bert-base-cased" # We need to create the model and tokenizer model = AutoModel.from_pretrained(MODEL_NAME) tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) print('model : ',m..
Python/TA 2020. 8. 13. 18:15
Prev 1 Next