Sj( &UdZddlZddlZgZeeed<dejdefdZ dejde dejfd Z d e d e dzde fd Z dd ejdejdejdejdzddf dZy)zCDefines utilities for interacting with scaled_dot_product_attentionN__all__tensorsreturnc&td|DS)z0Returns True if any of the tensors requires gradc34K|]}|jyw)N) requires_grad).0ts X/var/www/html/ia-enhanced/venv/lib/python3.12/site-packages/torch/nn/attention/_utils.py z'_input_requires_grad..s01q0s)any)rs r _input_requires_gradr s 00 00 inpt_tensorog_sizec@|jd|k7r |dd|fS|S)z'Handles the unpad of the last dimension.N)size)rrs r _postprocess_flash_outputrs.w&3=)) r head_dim_sizescalec:||Sdtj|z S)z For FlashAttention we pad the head dimension to be a multiple of 8 so we need to scale the output by the original head size and not the padded. g?)mathsqrt)rrs r _calculate_scalers#   =) ))rquerykeyvalue attn_maskc |se|j|jk7s|j|jk7r3td|jd|jd|jd|j|jk7s|j|jk7r3td|jd|jd|jd|jdks&|jdks|jdkr?td |jd |jd |jdy) NzLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: z instead.zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 2 dimensional, but got query.dim: z , key.dim: z and value.dim: )dtype ValueErrordevicedim)rrrr dropout_p is_causalr allow_lowp_kvs r _validate_sdpa_inputr)"s%  ;;#)) #u{{ekk'A((- }M#))M$$)KK= ;   ||szz!U\\U\\%A%%*\\N. M!!&i 9   yy{Q#'')a-599;?cyy{m;swwyk1A%))+i Y  ,;r)NgFNF)__doc__rtorchrliststr__annotations__Tensorboolrintrfloatrr)rr r4sI c15<<1D1 5<<#%,,*C* **&*  <<   << ||d"    r