[jongwook]

No description provided.

[formater]

I encountered an issue with this patch.

File "/usr/local/src/transcriber_rt/lib/python3.11/site-packages/whisper/model.py", line 111, in forward
    wv, qk = self.qkv_attention(q, k, v, mask)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/transcriber_rt/lib/python3.11/site-packages/whisper/model.py", line 124, in qkv_attention
    a = scaled_dot_product_attention(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In this part of the code of model.py:

def qkv_attention(
        self, q: Tensor, k: Tensor, v: Tensor, mask: Optional[Tensor] = None
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        n_batch, n_ctx, n_state = q.shape
        scale = (n_state // self.n_head) ** -0.25
        q = q.view(*q.shape[:2], self.n_head, -1).permute(0, 2, 1, 3)
        k = k.view(*k.shape[:2], self.n_head, -1).permute(0, 2, 1, 3)
        v = v.view(*v.shape[:2], self.n_head, -1).permute(0, 2, 1, 3)

        if SDPA_AVAILABLE and MultiHeadAttention.use_sdpa:
            a = scaled_dot_product_attention(
                q, k, v, is_causal=mask is not None and n_ctx > 1
            )

Somehow we got Tensor and not a boolean in is_causal. If mask is a tensor, the resulting is_causal can itself be a tensor, depending on how it is handled. So maybe a potential solution would be to cast it as bool like:
is_causal = bool(mask is not None and n_ctx > 1) ?

[JonahMMay]

I have the same issue as @formater where is_causal is receiving a Tensor instead of a bool.

[affinanti333]

Final