LjZ6UdZddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z ddl Z ddlmZeegefdzed<e j$j'rdd lmZndZiZeee fed <iZeeefed <Gd d ZGddZGddeZeegefdzed<e j<j'rddlmZ ndZ GddeZ!eegefdzed<e jDj'rddlm#Z$ndZ$GddeZ%e GddZ&GddeZ'GddeZ(Gd d!eZ)iZ*eee+efed"<d#a,d$ee jZzd%e+ed&dfd'Z.d$ee jZzd&e+efd(Z/d&ee0ee+effd)Z1d+d*Z2y),a Device abstraction layer for TorchDynamo and Inductor backends. This module provides a unified interface for different hardware backends (CUDA, XPU, CPU, MPS, MTIA) through a common device interface. Key components include: - DeviceInterface: Base class defining the common API for all device types - Device-specific implementations: CudaInterface, XpuInterface, CpuInterface, MpsInterface, MtiaInterface - Device registration system for managing available backends - Worker APIs for multi-processing scenarios - Stream and event management across different devices - Device property caching for worker processes The abstraction layer enables device-agnostic code in TorchDynamo while allowing specialized implementations for each hardware backend's unique features. N namedtuple)CallableIterable) dataclass)AnyLiteral has_torch_tpuget_cuda_stream)_cuda_getCurrentRawStream caching_worker_device_propertiescaching_worker_current_devicesc eZdZdZGddZGddZGddZGdd Zed e fd Z ede jjd d fd Zede d e fdZede d e fdZed e fdZed efdZede j d efdZed e j fdZede j d d fdZede de de d d fdZede d e fdZed&de jjd d fdZed&de jjd efdZed&de jjd efdZed'ded efd Ze d'd!e j>ded efd"Z ed&de jjd e fd#Z!ed&de jjd efd$Z"ed&de jjd d fd%Z#y )(DeviceInterfacez This is a simple device runtime interface for Inductor. It enables custom backends to be integrated with Inductor in a device-agnostic semantic. cDeZdZdejj defdZy)DeviceInterface.devicedevicereturnctNNotImplementedErrorclsrs ]/var/www/html/ia-enhanced/venv/lib/python3.12/site-packages/torch/_dynamo/device_interface.py__new__zDeviceInterface.device.__new__/s% %N)__name__ __module__ __qualname__torchtypesDevicerrrrrr.s &!3!3 & &rrc eZdZdededefdZy)DeviceInterface.Eventargskwargsrctd)NzYEvent should be inherited from torch.Event, otherwise, it couldn't be captured by dynamo.rrr(r)s rrzDeviceInterface.Event.__new__3s%k rNrr r!rrr%rrEventr'2  s s rr-c eZdZdededefdZy)DeviceInterface.Streamr(r)rctd)Nz[Stream should be inherited from torch.Stream, otherwise, it couldn't be captured by dynamo.rr+s rrzDeviceInterface.Stream.__new__9s%m rNr,r%rrStreamr08r.rr2ceZdZdZededdfdZedefdZeddejjde fdZ y) DeviceInterface.Workera Worker API to query device properties that will work in multi processing workers that cannot use the GPU APIs (due to processing fork() and initialization time issues). Properties are recorded in the main process before we fork the workers. rrNctrrrs r set_devicez!DeviceInterface.Worker.set_deviceF% %rctrrr%rrcurrent_devicez%DeviceInterface.Worker.current_deviceJr8rctrrr6s rget_device_propertiesz,DeviceInterface.Worker.get_device_propertiesNr8rr) rr r!__doc__ staticmethodintr7r:r"r#r$rr<r%rrWorkerr4>sp   &s &t &  &  & &  &  &%++*<*< & &  &rr@rctrrr%rrr:zDeviceInterface.current_deviceR!!rNctrrr6s rr7zDeviceInterface.set_deviceVrBrctrrr6s rmaybe_exchange_devicez%DeviceInterface.maybe_exchange_deviceZrBrctrrr6s rexchange_devicezDeviceInterface.exchange_device^rBrctrrr%rr device_countzDeviceInterface.device_countbrBrctrrr%rr is_availablezDeviceInterface.is_availablefrBrstreamctrrrLs rrLzDeviceInterface.streamjrBrctrrr%rrcurrent_streamzDeviceInterface.current_streamnrBrctrrrNs r set_streamzDeviceInterface.set_streamrrBr stream_id device_index device_typectrr)rSrTrUs r_set_stream_by_idz!DeviceInterface._set_stream_by_idvrBr device_idxctrrrXs rget_raw_streamzDeviceInterface.get_raw_streamzrBrctrrr6s r synchronizezDeviceInterface.synchronize~rBrc8|jj|Sr)r@r<rs rr<z%DeviceInterface.get_device_propertiesszz//77rctrrr6s rget_compute_capabilityz&DeviceInterface.get_compute_capabilityrBrincluding_emulationctrrras ris_bf16_supportedz!DeviceInterface.is_bf16_supportedrBrdtypecN|tjk7xs|j|Sr)r"bfloat16rdrreras ris_dtype_supportedz"DeviceInterface.is_dtype_supporteds$&T#*?*?@S*TTrctrrr6s rmemory_allocatedz DeviceInterface.memory_allocatedrBrcy)z Returns True if the device has Triton support, False otherwise, even if the appropriate Triton backend is not available. Fr%r6s ris_triton_capablez!DeviceInterface.is_triton_capables rc:|js tdy)aH Raises a `RuntimeError` with the appropriate human-readable instructions to resolve the issue if Triton is not available for the given device, or the default device if `device` is `None`. The caller should ensure the presence of the 'triton' package before calling this method. z/This device is not capable of supporting TritonN)rm RuntimeErrorrs rraise_if_triton_unavailablez+DeviceInterface.raise_if_triton_unavailables!$$&PQ Q'rrF)$rr r!r=rr-r2r@r>r?r:r"r#r$r7rErGrIboolrKrrLrPrRrWr[r] classmethodr<r`rdrerirkrmrpr%rrrr(s &&  &&("C"""5;;--"$"""c"c"""""""#"""$"""u||""""ELL"""5<<"D"""S""#"RV"""3"3"""EKK.."$""85;;+=+=888"u{{'9'9"S"""t"""=BUKKU6:U UU "!3!3"s""%++"4"4 R1C1C Rt R Rrrc VeZdZdZdeededzddfdZd dZde d e d e de d fd Z y) DeviceGuarda_ This class provides a context manager for device switching. This is a stripped down version of torch.{device_name}.device. The context manager changes the current device to the given device index on entering the context and restores the original device on exiting. The device is switched using the provided device interface. device_interfaceindexNrc.||_||_d|_y)N)rvidxprev_idx)selfrvrws r__init__zDeviceGuard.__init__s!1 rcr|j+|jj|j|_yyr)rzrvrGr{)r|s r __enter__zDeviceGuard.__enter__s- 88  11AA$((KDM rtypevalue tracebackFcp|j*|jj|j|_y)NF)rzrvrEr{)r|rrrs r__exit__zDeviceGuard.__exit__s+ 88 ,,BB4==QDHrrN) rr r!r=rrr?r}rrr rr%rrrurusX $_ 5>ADj LSrructeZdZejj Zejj ZejjZGddZe ejjZ e ejjZ e ejjZ e ejjZ e ejjZe ejjZe ejj Ze ejj"Ze ejj$Ze eZe ejj*Ze ejj.Ze ejj2Ze ejj4Ze defdZe d dej:j<dee zfdZ!e d dej:j<defdZ"e d dej:j<ddfd Z#y) CudaInterfaceceZdZededdfdZedefdZeddejjde fdZ y)CudaInterface.WorkerrrNc|td<yNcudarr6s rr7zCudaInterface.Worker.set_device 5; *6 2rc`dtvr tdStjjSr)rr"rr:r%rrr:z#CudaInterface.Worker.current_device)775f==::,,. .rc|\t|tr&tj|}|jdk(sJt|tjr |j }|t jj}dtvrZttjjDcgc]!}tjj|#}}|td<td|Scc}wr) isinstancestrr"rrrwrr@r:rrangerrIr<ri device_props rr<z*CudaInterface.Worker.get_device_properties!fc*"\\&1F!;;&000fell3#\\F~&--<<>==#5::#:#:#<=JJ44Q7 r?r7r:r"r#r$rr<r%rrr@rm  tjjSr)r"rrKr%rrrKzCudaInterface.is_availableszz&&((rNrctjj*tjj |\}}|dz|zStjj |j jdddS)N :r)r"versionhiprget_device_capabilityr< gcnArchNamesplit)rmajormins rr`z$CudaInterface.get_compute_capabilitysh ==   $99&AJE32:# #::33F;GGMMcSTUVWX Xrctjjduxs,tjj |j dk\S)N)r"rrrr<rr6s rrmzCudaInterface.is_triton_capable s> MM  T ) Czz//7==B rcxddlm}tj|s:tj j |}||tjddl }tjj$d|jjvr tdyd|jjvr tdy)Nr)GPUTooOldForTritonamdz'triton not built with the 'amd' backendnvidiaz*triton not built with the 'nvidia' backend)torch._inductor.excrrrmr"rr<inspect currentframetriton.backendsrrbackendsro)rr device_propstritons rrpz)CudaInterface.raise_if_triton_unavailables:..v6 ::;;FCL$\73G3G3IJ J ==   (FOO444"#LMM5 V__55 5KL L6rr)$rr r!r"rrr-r2r@r>r:r7rIrLrPrRrWr]r<r r[_exchange_devicerG_maybe_exchange_devicerErkrdrrrKr#r$r?rr`rmrpr%rrrrs ZZ  F JJ  E ZZ  FDD>"%**";";#>?O()J)JK#EJJ$?$?@$UZZ%A%AB)$))Yu{{'9'9YS3YYY %++"4"4    MEKK,>,> M$ M Mrrget_mtia_stream)_mtia_getCurrentRawStreamcneZdZejj Zejj ZejjZGddZe ejjZ e ejjZ e ejjZ e ejjZ e ejjZe ejjZe ejj Ze ejj"Ze ejj$Ze eZe ejj*Ze ejj.Ze ejj2Ze ejj4Ze defdZe d dej:j<defdZ e d dej:j<defdZ!e d dej:j<ddfd Z"y) MtiaInterfaceceZdZededdfdZedefdZeddejjde fdZ y)MtiaInterface.WorkerrrNc|td<yNmtiarr6s rr7zMtiaInterface.Worker.set_device3rrc`dtvr tdStjjSr)rr"rr:r%rrr:z#MtiaInterface.Worker.current_device7rrc|\t|tr&tj|}|jdk(sJt|tjr |j }|t jj}dtvrZttjjDcgc]!}tjj|#}}|td<td|Scc}wr)rrr"rrrwrr@r:rrrrIr<rs rr<z*MtiaInterface.Worker.get_device_properties=rrrrr%rrr@r2rrr@rcBtjj}|Sr)r"rrK)rets rrKzMtiaInterface.is_availableasjj%%' rNrcDtjj|}|Sr)r"rrrccs rr`z$MtiaInterface.get_compute_capabilityfs ZZ - -f 5 rcyNTr%r6s rrmzMtiaInterface.is_triton_capablekrcRddl}d|jjvr tdy)Nrrz(triton not built with the 'mtia' backendrrrorrs rrpz)MtiaInterface.raise_if_triton_unavailableos( 11 1IJ J 2rr)#rr r!r"rrr-r2r@r>r:r7rIrLrPrRrWr]r<rr[rrGrrErkrdrrrKr#r$rr`rmrpr%rrrr,s ZZ  F JJ  E ZZ  FDD>"%**";";#>?O()J)JK#EJJ$?$?@$UZZ%A%AB$u{{'9'9S%++"4"4KEKK,>,>K$KKrrget_xpu_stream)_xpu_getCurrentRawStreamcReZdZejj Zejj ZejjZGddZe ejjZ e ejjZ e ejjZ e ejjZ e ejjZe ejjZe ejj Ze ejj"Ze ejj$Ze eZe ejj*Ze ejj.Ze ejj2Ze defdZe d dej8j:defdZe d dedefd Z e d dej8j:defd Z!e d dej8j:ddfd Z"y) XpuInterfaceceZdZededdfdZedefdZeddejjde fdZ y)XpuInterface.WorkerrrNc|td<yNxpurr6s rr7zXpuInterface.Worker.set_devices 4: *5 1rc`dtvr tdStjjSr)rr"rr:r%rrr:z"XpuInterface.Worker.current_devices)665e<<99++- -rc|\t|tr&tj|}|jdk(sJt|tjr |j }|t jj}dtvrZttjjDcgc]!}tjj|#}}|td<td|Scc}wr)rrr"rrrwrr@r:rrrrIr<rs rr<z)XpuInterface.Worker.get_device_propertiess!fc*"\\&1F!;;%///fell3#\\F~%,,;;=<<#599#9#9#;<II33A6 ;F073E:6B B rrrr%rrr@rsm  ;s ;t ;  ;  . .  .  C%++*<*< C C  Crr@rc>tjjSr)r"rrKr%rrrKzXpuInterface.is_availablesyy%%''rNrcDtjj|}|Sr)r"rrrs rr`z#XpuInterface.get_compute_capabilitys YY , ,V 4 rrac>tjjSr)r"rrdrcs rrdzXpuInterface.is_bf16_supportedsyy**,,rcyrr%r6s rrmzXpuInterface.is_triton_capablerrcRddl}d|jjvr tdy)Nrintelz)triton not built with the 'intel' backendrrs rrpz(XpuInterface.raise_if_triton_unavailables( &//22 2JK K 3rrrq)#rr r!r"rrr-r2r@r>r:r7rIrLrPrRrWr]r<rr[rrGrrErkrrrKr#r$rr`rdrmrpr%rrrr~s YY  F IIOOE YY  FCC>"%))":":;Neii223J 6 67L %))** +F!%))":":;Neii223J$UYY%@%@Auyy445K()H)HI!.1N"599#=#=>O()I)IJ#EII$>$>?($((u{{'9'9S-t---%++"4"4LEKK,>,>L$LLrrceZdZUeed<y)CpuDevicePropertiesmulti_processor_countN)rr r!r?__annotations__r%rrrrsrrceZdZGddejZGddZedefdZeddedefdZ edd ejjde fd Z ed edefd ZedefdZedd ejjdd fdZedd ejjdefdZedd ejjdd fdZy ) CpuInterfacec@eZdZd deddfdZdedefdZd deddfdZy) CpuInterface.Event enable_timingrNcd|_y)Ngtime)r|rs rr}zCpuInterface.Event.__init__s DIrotherc:|j|jz dzS)Nir)r|rs r elapsed_timezCpuInterface.Event.elapsed_timesJJ*d2 2rrLc6tj|_yr)r perf_counter)r|rLs rrecordzCpuInterface.Event.records))+DIr)Tr) rr r!rrr}rfloatrrr%rrr-rs; $ $  3c 3e 3 , , ,rr-cReZdZe ddej j defdZy)CpuInterface.WorkerNrrc@ddl}|j}t|SNr)multiprocessing cpu_countr)rrrs rr<z)CpuInterface.Worker.get_device_propertiess  #'113I&y1 1rr) rr r!r>r"r#r$rr<r%rrr@rs1 )- 2KK&& 2  2  2rr@rcyrr%r%rrrKzCpuInterface.is_availablerrracyrr%rcs rrdzCpuInterface.is_bf16_supportedrrNrcyNr%r6s rr`z#CpuInterface.get_compute_capabilityrrXcyrr%rZs rr[zCpuInterface.get_raw_streamrcyrr%r%rrr:zCpuInterface.current_devicerrcyrr%r6s rr]zCpuInterface.synchronizes rcyrr%r6s rrmzCpuInterface.is_triton_capablerrcRddl}d|jjvr tdy)Nrcpuz'triton not built with the 'cpu' backendrrs rrpz(CpuInterface.raise_if_triton_unavailables( 00 0HI I 1rrqr)rr r!r"r-r@r>rrrKrdr#r$rr`rr?r[r:r]rmrpr%rrrrsJ, ,22$tu{{'9'9S33C EKK.. $  %++"4"4JEKK,>,>J$JJrrc&eZdZeddedefdZe ddejdedefdZ edefdZ ede fdZ edd ejjdefd Zedd ejjddfd ZGd d Zy) MpsInterfacerarcyrr%rcs rrdzMpsInterface.is_bf16_supported rrrecJ|tjtjfvryy)NFT)r"float64 complex128rhs rrizMpsInterface.is_dtype_supporteds! U]]E$4$45 5rcRtjjjSr)r"rmpsrKr%rrrKzMpsInterface.is_availables~~!!..00rcyrr%r%rrr:zMpsInterface.current_devicerrNrcyrr%r6s rr`z#MpsInterface.get_compute_capability rrc@tjjyr)r"r r]r6s rr]zMpsInterface.synchronize$s rcfeZdZeddej j defdZede fdZ y)MpsInterface.WorkerNrrcttddgtjjj S)N MPSPropertiesr)rr"rr get_core_countr6s rr<z)MpsInterface.Worker.get_device_properties*s2I:o0G/HI""113 rcyrr%r%rrr:z"MpsInterface.Worker.current_device0rr rr r!r>r"r#r$rr<r?r:r%rrr@r)G  %++*<*<         rr@rqr)rr r!r>rrrdrsr"rerirKr?r:r#r$rr`r]r@r%rrrr st=BKK6: 1$11Cu{{'9'9S EKK.. $    rrceZdZed dedefdZe d dejdedefdZ edefdZ ede fdZ edd ejjdefd ZGd d Zy) TpuInterfacerarcyrr%rcs rrdzTpuInterface.is_bf16_supported6rrrec|tjtjtjtjtj fvSr)r"r complex32 complex64r halfrhs rrizTpuInterface.is_dtype_supported:s; MM OO OO    JJ    rctSrr r%rrrKzTpuInterface.is_availableFs rcyrr%r%rrr:zTpuInterface.current_deviceJrrNrcyrr%r6s rr`z#TpuInterface.get_compute_capabilityNrrcfeZdZeddej j defdZede fdZ y)TpuInterface.WorkerNrrc(tddgdS)N TPUPropertiesrrrr6s rr<z)TpuInterface.Worker.get_device_propertiesTsI:o0G/HI rcyrr%r%rrr:z"TpuInterface.Worker.current_deviceZrrrrr%rrr@r#Srrr@rqr)rr r!r>rrrdrsr"rerirKr?r:r#r$rr`r@r%rrrr5st=B  KK  6:      $Cu{{'9'9S  rrdevice_interfacesFrrvrcbt|tjr |j}|t|<yr)rr"rrr')rrvs rregister_interface_for_devicer)cs&&%,,' 0frct|tjr |j}ts t |t vr t |Std|)NzNo interface for device )rr"rr_device_initializedinit_device_regr'rr6s rget_interface_for_devicer-ksI&%,,'  "" ((  8A BBrcJts ttjSr)r+r,r'itemsr%rr get_registered_device_interfacesr0us   " " $$rcBtdtttjj D]}td|ttdt ttjj D]}td|t tdtttjj D]}td|ttdttdttd td a y) Nrzcuda:rzxpu:rzmtia:rr tpuT)r)rrr"rrIrrrrrrrr+)rs rr,r,{s!&-8 5::**, -B%aSk=AB"%6 599))+ ,@%QCj,?@"&-8 5::**, -B%aSk=AB"%6!%6!%6rr)3r=rr collectionsrcollections.abcrr dataclassesrtypingrr r"torch.utils._pallasr r?rr _is_compiledtorch._Cr r rdictrrrrurrrrrrrrrrrrrr'rr+rr)r-tupler0r,r%rrr<s" ".! -3%*%,,::EO46 $sCx.513S#X3CRCRL6YMOYMx3%*%,,::EOHKOHKV#$t++99CNJL?JLZ  8J?8Jv'?'T'?'T794T/22381 %,, 1262G1 1CS5<<%7CDS9S3T*U% r