![]() ![]() ![]() loss.backward() + accelerator.backward(loss) optimizer.step()Īs you can see in this example, by adding 5-lines to any standard PyTorch training script you can now run on any kind of single or distributed node setting (single CPU, single GPU, multi-GPUs and TPUs) as well as with or without mixed precision (fp8, fp16, bf16). + model, optimizer, data = accelerator.prepare(model, optimizer, data) ain() Optimizer = (model.parameters())ĭata = (dataset, shuffle=True) + from accelerate import Accelerator + accelerator = Accelerator() - device = 'cpu' + device = vice model = torch.nn.Transformer().to(device) □ Accelerate abstracts exactly and only the boilerplate code related to multi-GPUs/TPU/fp16 and leaves the rest of your code unchanged. □ Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boilerplate code needed to use multi-GPUs/TPU/fp16. ![]() Run your *raw* PyTorch training script on any kind of device ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |