I'm trying to run OpenAssistant's pythia-12b model but I'm getting the following error:
NameError: name 'init_empty_weights' is not defined I have Accelerate installed, and I'm running Transformers version 4.25.1
from transformers import AutoModelForCausalLM, GPTNeoXForCausalLM, AutoTokenizer Model = "OpenAssistant/oasst-sft-1-pythia-12b" tokenizer = AutoTokenizer.from_pretrained(Model, cache_dir='models_hf') model = GPTNeoXForCausalLM.from_pretrained(Model, device_map="auto", load_in_8bit=True, cache_dir='models_hf',low_cpu_mem_usage=True) message = "Hello, How are you?" inp = "<|prompter|>"+message+"<|endoftext|><|assistant|>" data = tokenizer([inp], return_tensors="pt") data = {k: v.to(model.device) for k, v in data.items() if k in ("input_ids", "attention_mask")} outputs = model.generate(**data) print(tokenizer.decode(outputs[0])) 1 Answer
In my case, installing accelerate package and re-starting the runtime was enough. The same issue was also discussed here.
1
