Repository navigation
Fix low_cpu_mem_usage Flag Conflict with DeepSpeed Zero 3 in from_pretrained for Models with keep_in_fp32_modules" - #27762
Merged
asartran merged 1 commit intoDec 15, 2023
Conversation
for `low_cpu_mem_usage` with DeepSpeed Zero3
pacman100
approved these changes
Dec 15, 2023
pacman100
left a comment
pacman100
left a comment
Contributor
There was a problem hiding this comment.
Thank you @kotarotanahashi for fixing this bug!
iantbutler01
pushed a commit
to BismuthCloud/transformers
that referenced
this pull request
Dec 16, 2023
…pretrained` for Models with `keep_in_fp32_modules`" (huggingface#27762) Fix `from_pretrained` Logic for `low_cpu_mem_usage` with DeepSpeed Zero3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request addresses a compatibility issue in the
from_pretrainedmethod. Specifically, when using models withkeep_in_fp32_modulesnot set to None (e.g., BLIP2, T5) in conjunction with DeepSpeed's Zero 3, an unexpected error occurs due to the improper handling of thelow_cpu_mem_usage.Problem Description
Currently, in the
from_pretrainedmethod, thelow_cpu_mem_usageflag is set toTrueifuse_keep_in_fp32_modulesisTrueand accelerate is available. This logic does not account for the incompatibility with DeepSpeed Zero 3. When Zero 3 is enabled, settinglow_cpu_mem_usageto True can lead to unexpected errors.Proposed Change
I propose to modify the condition to check whether DeepSpeed Zero3 is enabled. The
low_cpu_mem_usageshould be set toTrueonly if Accelerate is available and DeepSpeed Zero3 is not enabled. The revised code snippet is:This change prevents the
low_cpu_mem_usageflag from being incorrectly set in scenarios where DeepSpeed Zero 3 is in use, thereby avoiding the aforementioned issue.