Exception just before LoRA is saved after training

[ArachneRomBor]

Again, after updating the WebUI, an exception occurs after the training is finished and before the LoRA is saved. LoRA seems to be stored fine.
Is this error critical?

f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\autograd\_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")

Exception in thread Thread-3 (threaded_run):
Traceback (most recent call last):
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "f:\work\programs\oobabooga_windows\text-generation-webui\modules\training.py", line 461, in threaded_run
    trainer.train()
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\transformers\trainer.py", line 1645, in train
    return inner_training_loop(
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\transformers\trainer.py", line 1987, in _inner_training_loop
    self.accelerator.clip_grad_norm_(
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\accelerator.py", line 1893, in clip_grad_norm_
    self.unscale_gradients()
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\accelerator.py", line 1856, in unscale_gradients
    self.scaler.unscale_(opt)
  File "f:\work\programs\oobabooga_windows\installer_files\env\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 275, in unscale_
    raise RuntimeError("unscale_() has already been called on this optimizer since the last update().")
RuntimeError: unscale_() has already been called on this optimizer since the last update().

2023-06-16 09:33:27 INFO:Training complete, saving...
2023-06-16 09:33:27 INFO:Training complete!```

[harpomaxx]

Same issue here. I've updated from current main branch and the problem persists. Not sure if LoRa model is created correctly.

[FartyPants]

Were bitsandbytes updated in the requirements recently?

[harpomaxx]

Not sure about that. I'm new to webui. I'm using bnb==0.39

[harpomaxx]

I've been testing different LoRA setups and if I reduce the size of the Microbatch and Batch Size I can finish LoRA training without raising the error.

[ArachneRomBor]

That's a good suggestion, but on the other hand, I found setting higher parameter values a way to improve the training level.
So it would be good to have this bug fixed, as its side effect is that a training_log.json file is not created, nor is this JSON written to the console.

[ArachneRomBor]

I reinstalled transformers using the following script and the error seems to be gone.

pip install git+https://github.com/huggingface/transformers@de9255de27abfcae4a1f816b904915f0b1e23cd9