Lj:ddlmZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z mZddlZddlmZddlmZddlmZd d lmZmZmZd d lmZerd d lmZmZd d lmZm Z m!Z!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d dl+m,Z,ejZe.Z/ej`jce.dZ2erddl3m4Z4 dEdZ5dFdZ6dFdZ7 dFdZ8 dFdZ9e GddZ:dGdZ;dHdZdZ?dLdZ@dZA dMd ZB dNd!ZC dOd"ZDd#ZEd$ZFd%ZG dPd&ZH dQd'ZI dRd(ZJ dSd)ZK dTd*ZL dUd+ZM dVd,ZN dWd-ZO dXd.ZP dYd/ZQ dZd0ZR d[d1ZS d\d2ZTe Gd3d4ZU dXd5ZV d]d6ZW d^d7ZX dTd8ZY d_d9ZZ dWd:Z[ d`d;Z\dFd<Z]dad=Z^d>Z_d?Z` dFd@ZadbdAZbdcdBZcdCZd dddDZey)e) annotationsN) defaultdict) dataclass)Any TYPE_CHECKING)trace_structured)StorageWeakRef) OrderedSet)config config_commsir)WeakDep)IRNode Operation)estimate_peak_memory_allocfreeFreeableInputBufferget_freeable_input_buf SNodeMemory)contains_collective contains_waitfind_recursive_deps_of_nodefind_recursive_users_of_node is_collectiveis_fallback_opis_wait)Voverlap)BaseSchedulerNodecRddlm}i}i}|D]&}|j||<||||||<(ddlm}ddlm}|j}|}t|D cgc]} g} } |j| t|j|tjtj| djj} t!|j#D] \} }| | ||<|D]}||vr||||<|||_ycc} w)Nr)_get_mm_like_fn)_get_default_group)dim)torch._inductor.schedulerr!get_estimated_runtimetorch.distributed distributed"torch.distributed.distributed_c10dr"get_world_sizerangeall_gather_objectlistvaluestorchmediantensortolist enumeratekeysoverride_estimated_runtime) snodesr!runtime_estimationsruntime_estimations_for_mmssnodedistr" world_sizepg_$gathered_runtime_estimations_for_mms"median_runtime_estimations_for_mmsidxs T/var/www/html/ia-enhanced/venv/lib/python3.12/site-packages/torch/_inductor/comms.py6align_runtime_estimations_across_all_distributed_ranksrA4s[:"$L%*%@%@%BE" 5 ! -1DU1K ' .L %E$$&J  B*%??(? , ( / / 12  */ 9:* fVVX' ; @ @ BCU U-OPS-T#E*UF / /)DU)K  &+>u+E(F?s& D$c t|dddS)z7 Greedily schedules waits as late as possible. FT raise_comms sink_waitsreorder_for_overlap_schedule_for_commr5s r@rErEZs Ed c t|dddS)z8 Greedily schedules comms as early as possible. TFrCrGrIs r@rDrDcs DU rJc t|dddS)a This achieves the following overall scheduling procedure: Step 1: Given that we've currently scheduled comm N, we now schedule all compute nodes that are required for comm N + 1 but do not depend on comm N, to run at the same time with comm N. Step 2: If all those compute nodes are sufficient to overlap comm N, we're done. Otherwise, we now need to look elsewhere to find compute that overlaps with comm N. We prioritize compute nodes that are needed sooner. Step 3: We schedule the compute nodes dependent on comm N and required for comm N + 1. Step 4: We schedule comm N + 1. Repeat this for subsequent comm nodes. TrCrGrIs r@reorder_compute_for_overlaprMls DTt rJc"t|\}}|S)a Reorders communication ops relative to computation ops to improve communication-compute overlapping and hide comm latency. Stops moving a particular op if it reaches a point that would have increased the peak memory footprint. Currently, follows these heuristics (subject to change or tune): - never reorders collectives relative to one another, for SPMD safety - has an option for per-collective prefetch limit, but does not enable it by default - limits the total number of reorder steps to some factor of the graph size to prevent worst-case quadratic performance Prerequisite: sink_comms_and_waits - ensure comm and wait nodes are scheduled as late as possible, respecting data dependencies. That allows reorder_communication_preserving_peak_memory to take a best case peak-memory snapshot, and then monotonically improve latency by moving collectives backward in time. Peak memory impact is computed in an iterative fashion. First, memory use at each timestep is computed, and global peak memory is computed as a max over timesteps. Then, when swapping any two adjacent nodes, only the curr-memory for the earlier of the nodes after the swap is affected. This enables checking step by step whether a swap is peak-memory-safe, and bailing out if not. Example: 0 n0 C0 1 n1 C0 + Allocs(n1) - Frees(n1) 2 n2 C0 + Allocs(n1) - Frees(n1) + Allocs(n2) - Frees(n2) 0 n0 C0 1 n2 C0 + Allocs(n2) - Frees(n2) <-- After moving n2 to Time 1, only time1 memory changes 2 n1 C0 + Allocs(n2) - Frees(n2) + Allocs(n1) - Frees(n1) )6_reorder_communication_preserving_peak_memory_internal)r5reordered_snodes node_statss r@,reorder_communication_preserving_peak_memoryrRs@ ?vF!j rJceZdZUdZdZded<dZded<dZded<d Zded <d Z d ed <d Z d ed<d Z d ed<d Z d ed<dZ ded<edZy) ReorderInfozE Debug info describing how an individual snode was reordered Nonestrlimiting_factorrintmovesgrouped grouped_infofloat comm_time comp_timeinitial_exposed final_exposed overlap_infoc4|j|jz SNrarbselfs r@ improvementzReorderInfo.improvement##d&8&888rJN)__name__ __module__ __qualname____doc__rW__annotations__rYrZr\r_r`rarbrcpropertyrirJr@rTrTst"OS!E3NGSL#IuIu!OU!M5L# 99rJrTc|yt|tjjjj ryt |ddx}rd|vryy)NFTpython_kernel_nameextern_kernels)rr.opsaten#_scaled_dot_product_flash_attentiondefaultgetattr)noderss r@ is_gemm_liker{sX |  ::BB&d,@$GGG 0 0 rJcddlm}t||rtd|jDSt |j S)NrGroupedSchedulerNodec32K|]}t|ywre)contains_gemm_like).0xs r@ z%contains_gemm_like..s?Q%a(?s)r$r~ isinstanceanyr5r{rz)r8r~s r@rrs4>%-.?%,,???EJJ''rJcddlm}t||r&|jr|jD] }|| y||y)Nrr})r$r~r temp_groupingr5)r8fnr~_snodes r@_temp_group_visit_leavesrs;>%-.53F3Fll F vJ  5 rJcd}|jD]<}|r|dz }||jz }|s |t|jz }>|S)Nr[r<)r5get_namer,get_buffer_names)r8 with_bufsretns r@ _group_namers[ C \\7  3JC qzz|  d511345 6C 7 JrJc>t|txr |jSre)rris_fake)ds r@ _is_fake_deprs a ! /aii/rJcfdj|Dcgc]}|jc}Scc}w)N~)joinr)gnsgns r@ _group_namesrs% 88S1rR[[]1 221s.ct||}t|||\}}}}tt||}d|d<t |} |||||| fS)z*Initialize memory tracking data structures)rrN)rrdictzip_build_candidate_buffer_map) r5 graph_inputs graph_outputsname_to_freeable_input_buf peak_memorysnodes_curr_memorysnodes_allocfreebuf_to_snode_last_use _curr_memorycandidate_buffer_maps r@_initialize_memory_trackingrsz!7 !M& .  MK#%57L F$678LL77LM "  rJci}i}t|D]5\}}|dkDr||dz nd||<|t|dz kr||dznd||<7|d}|||fS)z/Create double-linked list structure from snodesrr N)r2len)r5_prev_nextir8_heads r@_initialize_double_linked_listr s{ E Ef%F5()Ava!e}4e ()CK!O(;va!e}e F 1IE % rJctt}|D]1}|jjD]}||j |3t |S)a} Build inverted index: node -> set of buffers where node appears in successors. This optimization reduces buffer iteration from O(total_buffers) to O(buffers_per_node). Since buffer successors are immutable during reordering, this map doesn't need updates. Returns: dict mapping each node to the set of buffers that have this node in their successors )rr mpi_buffer succ_nodesaddr)rnode_to_candidate_bufsbuf succ_nodes r@rrs`CNC%722 7I "9 - 1 1# 6 77 & ''rJc j|Dcic]#}|td|jD%c}Scc}w)z Pre-compute output name sets for all nodes. This optimization avoids creating OrderedSet objects repeatedly during exposed time calculations. Returns: dict mapping each node to a set of its output names c3<K|]}|jywrerros r@rz/_precompute_node_output_sets..@sD1!**,D)r get_outputs)r5r8s r@_precompute_node_output_setsr3s<SY INzD0A0A0CDDD  s(0cXt|rtjStjSre)rr *reorder_sink_runtime_estimations_comm_mult.reorder_sink_runtime_estimations_non_comm_multr8s r@_op_runtime_estimate_multrDs$5!FFF  F FFrJc@t|jddx}rd|vryy)z Filtering out ops that contain Collective and Wait inside and considered as Collectives. See contains_collective function. If the op contains Wait inside - consider as Synchronous compute. rsNz-torch.ops._dtensor.shard_dim_alltoall.defaultFT)ryrz)r8rss r@is_async_collectiverMs- %UZZ1EtLLL :>P P rJc"t|tSre)rrrs r@contains_async_collectiverZs u&9 ::rJcNg}|} ||j|||k(r |S||}!)aA Traverse doubly-linked list from head to tail and return nodes as a list. Args: head: Starting node of the segment tail: Ending node of the segment (inclusive) next_dict: Dictionary mapping each node to its next node Returns: List of nodes from head to tail (inclusive) )append)headtail next_dictrrs r@_group_nodes_from_linked_listr^s@ C A  = JJqM 9  J aL rJc2||}||}t||zS)zr Check if a wait node corresponds to a given collective node. Uses pre-computed sets for O(1) lookup. )bool)collective_snode wait_snodenode_output_sets node_dep_setscollective_outs unmet_depss r@!_is_corresponding_collective_waitrys)''78Oz*J  _, --rJc|}d||}d}g}||} | || } | |zr|d| jdz }nt| r#t| sn||j| || } Rt | rd} |D]} t || ||sd} n| rn<} d fd }t | |}|d| jd || z dz }|| } | ||fS) z Calculate exposed communication time by iterating directly over linked list. Avoids O(N) list construction for each call. The collective_snode is the starting point, iteration continues via next_dict. r[z->W[]FTc|z yrerqrr`runtimess r@accumulate_timez9_coll_exposed_communication_time..accumulate_time &) )IrJ+[rrreturnrU)rrrrrrr)rrrrrr_rrccollectives_foundr8rhas_wait_for_collectives_found_collcomp_time_beforercomp_time_afterr`s ` @r@ _coll_exposed_communication_timersD)*II&'78OL13 & 'E  "5)  ' d5>>#3"4A6 6L  u %,U3!((/!%(  -2 ** 4$e-=}6:2   .$ * !8#!ENN,-QAQ/Q.RRSTT % G  J i --rJcd}dd}g}||} | t| r|j| t| r]t| |||r| }|d| j dz }nnt | sd|| } g|D]} t| | ||sdn|| } } dfd } t | | } |d| j d| | z dz }|| } | ||fS) z Calculate exposed communication time for a wait operation by iterating directly over linked list backwards. Avoids O(N) list construction. Iterates from wait_snode backwards using prev_dict to find corresponding collective. rr[z->C[rc|z yrerqrs r@rz9_wait_exposed_communication_time..accumulate_timerrJrrr)rrrrrrr)rr prev_dictrrrr_rc waits_foundr8wrrrr`s ` @r@ _wait_exposed_communication_timers>IIL+-K j !E       u % u %0z#3]%UO $u~~'7&8 :: ,U3 !%($A8q"2M%(   "%($ * !8#!ENN,-QAQ/Q.RRSTT % G  J i --rJcl||}|r|||<|||<||}|r|||<|||<|||<|||<||k(r|S|S)a Swap positions of candidate and group in doubly-linked list. Transforms: candidate_prev -> candidate -> group_head...group_tail -> group_tail_next Into: candidate_prev -> group_head...group_tail -> candidate -> group_tail_next Args: candidate: Node to swap with group group_head: First node of group group_tail: Last node of group prev_dict: Dictionary mapping nodes to their previous nodes next_dict: Dictionary mapping nodes to their next nodes head: Current head of the linked list Returns: New head of the linked list (may change if candidate was the head) rq) candidate group_head group_tailrrrcandidate_prevgroup_tail_nexts r@ _perform_double_linked_list_swaprss8y)N$. .!*Ij +O%. /"*Ii&Ii%Ij y KrJcni}d} |s)t||z ||d|z |jz} | |fS| } |D]R} || d| z} | || <t| | } |j| } | 3| D]}| |jjz } T||d| z|jz}|||<t| |} | |fS)a Calculate potential peak memory after swapping candidate with group (reorder version). Computes new memory levels for all affected nodes and returns the potential peak memory along with cached post-allocation memory values for each node. Args: candidate: Node being moved gns: Group nodes group_tail: Last node of group group_peak_memory: Current peak memory within the group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_by_candidate: Buffers whose deallocation moves to candidate curr_memory: Current memory state dict Returns: Tuple of (potential_peak_memory, post_alloc_update_dict) rr )max size_allocgetr size_free)rrrgroup_peak_memorycandidate_delta_memcandidate_allocfree/group_n_to_bufs_after_swap_dealloc_by_candidate curr_memory_post_alloc_updatepotential_peakmem_after_reorder_deltargn_post_alloc_membufsrcandidate_mem_post_allocs r@(_calculate_potential_peak_memory_reorderr1s+>8:N :  3 3  #A &! "!,, -  111%8#7 D'OA.1HH!22^->?>BB2F   D'3>>+C+CC' D D J" ! "  ( ( ) %=y!)ABN - --rJc |sK|D]} || } | d|z | d|z f|| <||d|jz} | |jz } | | f||<y|jD]}|D]}|||< d}|D]R}||}td||D}||z }| |xj|zc_|| |jz }||f||<T||} | |xj|z c_| | |jz }| |f||<y)aN Update memory tracking structures after swap (reorder version). Updates curr_memory, buf_to_snode_last_use, and snodes_allocfree dictionaries to reflect the new memory state after swapping candidate with group. Args: candidate: Node that was moved gns: Group nodes group_tail: Last node of group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_by_candidate: Buffers whose deallocation moves to candidate post_alloc_update: Cached post-allocation memory values curr_memory: Current memory state dict (mutated) buf_to_snode_last_use: Buffer to last-use node mapping (mutated) snodes_allocfree: Node allocation/free info dict (mutated) rr Nc3HK|]}|jjywrerrrrs r@rz=_update_memory_tracking_after_swap_reorder..s"2  NN $ $2  ")rrr-sum)rrrrrrpost_alloc_updaterrrrcm_candidate_post_alloc_mem_candidate_post_free_memrr"size_free_to_move_to_candidate_sumr_gn_post_alloc_memsize_free_to_move_to_candidategn_post_free_memcandidate_post_free_mems r@*_update_memory_tracking_after_swap_reorderrts< ; BRB1++1++KO   #A &)<)G)G G " &(;(E(E E ! & $" I @FFH3 3C)2 !# & 33/0&  @"3A"6.12 FqI2 / & +.LL*%%)GG% 25Ea5H5R5R R,.>? A @!2) <Y))-OO)!$4Y$?$I$II "K rJctt}|j|t}t|}|D] }||}||vs ||j |"|S)a Find buffers whose last use will change after swapping candidate with group. When we swap [candidate [group]] to [[group] candidate], some buffers that were last used by a group node will now be last used by candidate instead. This affects memory deallocation timing. Args: candidate: The node being moved gns: Group nodes being swapped with candidate buf_to_snode_last_use: Mapping of buffers to their current last-use nodes candidate_buffer_map: Pre-computed map of node -> buffers using that node Returns: Dict mapping group nodes to buffers that will change their last-use node )rr,rr r) rrrrrcandidate_bufsgns_setrsnode_last_uses r@#_find_buffers_with_changed_last_userso0 D4 *--iFNoGX.s3 W $ ;N K R RSV WX ;:rJctjs t|rdd|jfStjs t |ryy)aa Check if a candidate node can be grouped with collective during reordering. This pass processes collectives left to right, so we avoid grouping with already-processed collectives based on configuration. Args: candidate: Node to check for groupability Returns: Tuple of (is_groupable, reason_if_not_groupable) Fzcandidate contains_collective )FrTN)r (reorder_iterative_group_with_collectivesrr)reorder_iterative_use_runtime_estimationsrrs r@_is_node_groupable_for_reorderr"sO$  @ @ $Y /01C1C1E0FG   A A i (. rJc|}|Dcic]}|||j} }t| Dcgc]}| | c}} t|Dcgc]}||jc}} d| d| dgd} |jD cgc]\}} t || j dz | j dz | jdz | jdz | jdz | j| j| j| j| jg }}} tjjdrddlm}||| z n8d z t#| d zz d j%t't"|z t)|d |}t+||k(sJt-|||\}}}}d |z d|z t.j1t3ddfd|Scc}wcc}wcc}wcc} }w)a Format reordering statistics, log them, and return final node list. Computes improvement metrics, creates a formatted table (using tabulate if available), validates the reordered node count, recalculates peak memory, and logs all information. Args: stats: Per-node reordering statistics head: Head of the reordered linked list next_dict: Linked list next pointers original_snodes_num: Original number of nodes (for validation) peak_memory: Initial peak memory before reordering name_to_freeable_input_buf: Buffer memory tracking info graph_outputs: Graph output names Returns: Final reordered list of scheduler nodes zAreorder_communication_preserving_peak_memory improved overlap by z ns after z reorders. ) zCollective node comm_time(us) comp_time(us)initial exposed(us)final exposed(us)improvement(us)limiting factorrYrZr\rc@@tabulaterr+headers>Please `pip install tabulate` to nicely render overlap stats.  Nz peak_memory_before:z peak_memory_after:artifactc dddS)NrRstringnameencodingrqrqrJr@z2_format_and_log_reordering_stats..asB  rJcSrerq)reorder_log_strsr@r7z2_format_and_log_reordering_stats..es?rJ metadata_fn payload_fn)rir rYitems node_summaryr_r`rarbrWrZr\rc importlibutil find_specr+rVrmaprrr overlap_loginfor)statsrroriginal_snodes_numrrrrQr8ritotal_improvement total_movesr. node_inforowsr+ new_snodesnew_peak_memoryr<r9s @r@ _format_and_log_reordering_statsrMsF8JEOPE5*U+777PKP[IE[/IJJG5z%(..GHK LL]K^_l , G6!+ 0 0 2  E9     # %   # %  % % +  # #c )  ! !C '  % % OO     " "  " "  D  ~~ +%8     M  3w<$..499Sd^44.tT9EJ z?1 11 1=.  OQ1/ }==O..?@@O_% +  KQIG& sG, G1G6BG;c  d}|D]}t|sd}n|s|ifSt|}ttjj j }ttjj}t|||\}}}} } } |Dcic]}|t|t|z} }|Dcic]#}|td|jD%} }|Dcic]}|td|jD!}}i}d}t|\}}}tj }d}|}tj"}d}|||||}|rnt%|s|} |||k\rn|dz }t'x}||<t)||| | |\}}}||_||_||z x|_|_||_||}|}|} i}!d}"||d}#t||}$t| |}%| tj4r4|j0tj6 |j*zkr d |_ntj:st|r d |_n|$|%z }&|&sd}'n| |}(t=|(|&z}'|'rt?|\})}*|)r|}|$jA|||%jA| |tj4r9tC|rtE|||| | |\}}}+||f|!|<t%|s|"| |z }"tG|#||d}#|xjHdz c_$||};d |jKd |jD,cgc]},|,jKc},d |*}-|-|_ntj4rWt%|s| |}.|.dkDrAt|!dkDr2tGd|j*|j,z }/tGd|j*|j,z |.z }0|0|/z }1|!jMD].\}2}3tGd|2|3z }4tGd|2|3z |.z}5|1|5|4z z }10|1dkDrd|.d|1|_n|!jOD]\}6\}2}3|2|3|.z f|!|6<ni|"dkDrdt)||| | |\}}}+tGd||z }/tGd||z |"z}0|0|/z }1|1dkDr&d|jKd|"d|1d|d| |_ntQ|| |}7||}8|8jR|8jTz }9tW||7| | }:tY||7| |#|9|8|:|\};}<|;|z |tjZzkDrd|;d||_n|xj\dz c_.|dz }t_||| |||}t)||| | |\}}}||_||_||_||z |_ta||7| |9|8|:|<|| | |r/ddl1m2}=|=||7tg|7tQ|d|| ||||d|: }|rn ||}| |}|||tjhstQ|d|}>|>|fStk|||||| |}>|>|fScc}wcc}wcc}wcc},w)z Internal testing helper that also returns debug info. Returns: - reordered snodes list - dict {snode: ReorderInfo} FTc3<K|]}|jywrerrs r@rzI_reorder_communication_preserving_peak_memory_internal..C!Crc3LK|]}t|r|jywrerr5rrs r@rzI_reorder_communication_preserving_peak_memory_internal.. LOAFF $$rNr r unexposed by runtime estimationszcollective ordering$data dependency detected candidate:z(outs:z) non_group_reason:candidate has compute z,, group contains waits, total_exposed_delta z candidate z is collective, group_runtime:z, exposed_delta:z c_comm_time:z c_comp_time:peak memory new: vs base:!_debug_iterative_memory_recomputerR)6rrr rgraphrr3get_output_namesrestimate_op_runtimer frozensetrunmet_dependenciesrr (reorder_iterative_debug_limit_to_reorder(reorder_iterative_debug_memory_recomputerrTrr_r`rarbrcr )reorder_iterative_extra_comm_comp_overlaprW,reorder_iterative_unsafe_collectives_reorderrr"updaterrrrZrr-r=rrrrr$reorder_iterative_peak_memory_budgetrYrr comms_debugr\rreorder_sink_verbose_loggingrM)?r5has_collectivesr8rFrrrrrrrrrrrrErHrrr debug_num_collectives_to_reordernum_processed_collectivescurr debug_iterative_memory_recomputeiterative_recompute_error _next_currrDr_r`rcrrr group_waits group_runtimergroup_unmet_deps_namesgroup_output_namesdata_deps_namesdata_depcandidate_out_namesis_groupable_resultgrouping_reasonr<rmsg c_runtimeexposed_before exposed_after exposed_delta gw_comm_time gw_comp_timegw_exposed_beforegw_exposed_aftergwrrrrrrr\rKs? r@rOrOksO u %"O  rzf+$.qww/C/C/H/H/J$KL%/0H0H0J%KM $FL-H" 0  "5),Ee,LLL0H0RXAHMyCu/@/@/BCCCAA >  y !44   >M>35EK8@E5% ==%&'%*D==%!&  uT{64[ $ (.D  + 7 %)I I !Q&!(]*uT{-M %#3]. * 9l#"4= 4IIt1($K     (.q1",M$,?!@'(8(>?#EE""III..!!(J$!MM' 2'<$57IIO" '7y&A# 3o EF7U84#_'!*J+11- 2JK&--.>y.IJ#MM(36V ) % % ( 0 - 73Iy!6? 5IK 28C)Xi-@@M(+)< +B1+E)%LLA%L %i 0I((1(:(:(<'=V[d[p[p[rDsVWQZZ\DsCtu//>.?A ,/D(EE0; ( 3I 1}[)9A)=),Q0O)P(+t~~>J) )6(F :E:L:L:NR6L,03A|l7R0S-/2 !<,#>#J0,*-=@Q-QQMR)1,"8 DNN[_!^!0" "-!2!2!4"!% , ,%1$09$<3" B "%q(2R%uh8H-3/ 9a*-Q I0E)F(+Ay9/D}/T(U (5(F (1,",Y-?-?-A,BC22?A22? i[Xefoep!r!0 ",IJ,C0@ /J #..1D1N1NN 4s$9;O <9%''C   /N., Q QQR'~&6i }M$ JJ!OJ 1 K4:z5%E2ReX'72 .Iy,'DN&DN ,D !*Y!6D  6##?"%  0K,M %1%uE.! $BC -)-j)IC#DK  uT{6N  4 425$F 5  1   "J u o 0 A>pEtsY?(Y-$Y Y ci}iiiict|D]y\}}|jD]}|||< |jD]}||< ||j<|j} tj | <d| <|| <{d} |D]}|rZt |rO| |j<|jD]'} | j} t| | | <)| dz } _|sbt|snd|j<Gfdd|Dcic]}|td|jD!c}gtt|Dcic]}|t|c}jD]J\}} t| dk(rt!j"|| D]}|j%|Lgfdfdfd}r@t!j&j(}|rt |r ||n|r@j+D]} t| dk(rJd Scc}wcc}w) a Schedule `snodes` for various comm optimization objectives. Args: snodes: the nodes to be scheduled. raise_comms: whether to greedily schedule collectives as early as possible sink_wait: whether to greedily schedule waits as late as possible reorder_compute_for_overlap: whether to reorder compute nodes to optimize for compute/communication overlapping. Returns: The new schedule order. Some notes on the synergy between different options: - `raise_comms` provides more overlapping oppurtunies for `reorder_compute_for_overlap`. - When both `raise_comms` and `sink_waits` is `True`, `raise_comms` is prioritized. rr c&eZdZdfd ZdZy)$_schedule_for_comm..Runnablec||_tt|j}|j }|||f|_yre)r8nextiterget_operation_namesrscore)rhr8r5 fused_namename_to_fused_nodescores_0scores_1scores_2s r@__init__z-_schedule_for_comm..Runnable.__init__sVDJU6689:D+D1::.Runnable.__lt__s:: + +rJN)rrU)rkrlrmrr)rrrrsr@Runnablers   ,rJrc34K|]}|jywre)r5)rdeps r@rz%_schedule_for_comm..sGs#((Gscj||jD]N}|D]D}|j|t|dk(s)t j |FPy)zU Schedules `snode` and put all unblocked nodes onto the ready queue. rN)rrremoverheapqheappush)r8buf_namer buffer_usersready scheduledrs r@schedulez$_schedule_for_comm..schedule#sv ..0 ;H%h/ ;5!((2z%()Q.NN5(5/: ; ;rJcDcgc].}t|jst|js|0}}t|dk(ryt |dScc}w)zh Return the next node in the ready queue that's neither a collective or a wait. rNc|jSrerrs r@r7zG_schedule_for_comm..get_overlapping_candidate..:s QWWrJkey)rr8rrmin)r candidatesrs r@get_overlapping_candidatez5_schedule_for_comm..get_overlapping_candidate.s] &qww/ agg8N   z?a :#455 s3Act|sJ||}|dkDrMx}Dj||j||jz}|dkDr x}Dtjy)z Schedules collective node `snode`, along with one or more compute nodes to overlap with it. The strategy is described in the comment of `reorder_compute_for_overlap`. rN)rrr8rheapify)r8collective_costrrrr snode_to_costs r@schedule_collective_for_overlapz;_schedule_for_comm..schedule_collective_for_overlap<s #5)))'. a 799F LL # Y__ % }Y__= =O a 799F  erJz;Detected unscheduled nodes. Nodes with unmet dependencies: )r2rrrsysmaxsizer ancestorsrrr rarr_r=rrrrheappopr8r-)r5rDrErFbuf_name_to_snoder?r8rop_name node_namecomm_idxancestoranc_fused_namedepsrrrrrrrrrrrrrrs @@@@@@@@@@@@r@rHrHsL#%r2 Hh' " U..0 0H*/ h ' 0002 0G*/ w ' 0/45>>+,NN$ !kk! "H+ .u5)1HU^^% &!OO S!3H!=!F!F!H+.x/G+R( S MH M%0)*HU^^% &+ , , <  zGe.F.FGGG<J E=H=TLDJK5U/66KM!'')) t t9> NN5(5/ 2 )C   ! !% ( )) I ; ; 6*  e$** #6u#= +E 2 UO !!# 4yA~ I* V ~  U<Ls 9$I15I6c ztjjs|S|Dcgc]}t|s|}}t dt |D]b}t t||j}||dz jD]"}||jt||d$d|Scc}w)z Decide global ordering of comms, by just enforcing the ordering that's in the input graph (might not be the same ordering as the eager mode program). TODO: Come up with a better approach r T mutating_bufr) r.r' is_availablerr*rrrr add_fake_depr)nodes name_to_bufrr comm_nodesrrrs r@decide_global_ordering_of_commsr_s    ) ) + "=&9!&s B8B8ceZdZUdZded<dZded<dZded<dZded<d Zded <d Z d ed <d Z d ed<d Z d ed<d Z d ed<d Z ded<edZy) SinkWaitInforrXrZr[rVr\rY moves_inforUrWr]r^r_r`rarbrcc4|j|jz Srerfrgs r@rizSinkWaitInfo.improvementrjrJN)rkrlrmrZror\rYrrWr_r`rarbrcrprirqrJr@rrwswGSL#E3NJ!OS!IuIu!OU!M5L# 99rJrc$t|rdd|jfSt|rdd|jfStjs@t |rdd|jfSt |rdd|jfSy)aP Check if a candidate node can be grouped during sink_waits pass. Sink Waits traverses waits right to left, so we don't group with processed waits on the right or with async collectives. Args: candidate: Node to check for groupability Returns: Tuple of (is_groupable, reason_if_not_groupable) Fzcandidate contains wait $candidate contains_async_collective zcandidate contains collective zcandidate contains gemm_like r)rrrr &sink_iterative_use_runtime_estimationsrrr!s r@!_is_node_groupable_for_sink_waitsrs"Y01C1C1E0FGGG + 293E3E3G2H I   > > y )01C1C1E0FG  i (/ 0B0B0D/EF  rJc d|d} || d|| jz } |sC| |jz} | | |jz f||<|D]} || } | d|z| d|zf|| <y|g|D]F}||}||xj|j|dz c_||||jz f||<Hy)a Update memory tracking structures after swap (sink_waits version). Updates curr_memory and snodes_allocfree dictionaries to reflect the new memory state after swapping candidate with group. Args: candidate: Node that was moved gns: Group nodes candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_instead_of_candidate: Buffers whose deallocation moves from candidate to group post_alloc_update: Cached post-allocation memory values size_free_delta_update: Cached size-free delta values curr_memory: Current memory state dict (mutated) snodes_allocfree: Node allocation/free info dict (mutated) rr N)rrr)rrrr7group_n_to_bufs_after_swap_dealloc_instead_of_candidatersize_free_delta_updaterrr pre_group_memcandidate_post_allocrrr post_allocs r@-_update_memory_tracking_after_swap_sink_waitsrs 8QJ +A.1A*1M1X1XXM B,/B/M/MM #6#@#@ @" I BRB1++1++KO   #  &q) %%)?)C)CAq)II%  )!,66 6 A rJc ||d||jz } i} i} d} |s!t||z| |jz} | | | fS| |jz} | | |<| } tdtjj |j D}| | |<||z}|D]S}||d|z}|| |<t| |} d}||vr*||}|D]}||jjz }|| |<||z}U| | | fS)a Calculate potential peak memory after swapping candidate with group (sink_waits version). Computes new memory levels for all affected nodes and returns the potential peak memory along with cached post-allocation and size-free delta values. Args: candidate: Node being moved gns: Group nodes group_head: First node of group group_peak_memory: Current peak memory within the group candidate_delta_mem: Net memory change from candidate (alloc - free) candidate_allocfree: Candidate's allocation/free info group_n_to_bufs_after_swap_dealloc_instead_of_candidate: Buffers whose deallocation moves from candidate to group curr_memory: Current memory state dict snodes_allocfree: Allocation/free info for all nodes Returns: Tuple of (potential_peak_memory, post_alloc_update_dict, size_free_delta_update_dict) rc3HK|]}|jjywrer r s r@rz>_calculate_potential_peak_memory_sink_waits..s"&    &r ) rrr  itertoolschain from_iterabler-rr)rrrrrrrrrrr_size_free_delta_updaterrcandidate_size_free_to_move delta_memr gn_post_allocgn_size_free_to_addrrs r@+_calculate_potential_peak_memory_sink_waitsrs{> +A.1A*1M1X1XXM79<>N B  3 3 /:: : 13JJJ(+>+I+II$8y!)N"%&??00 C J J L &# +F)EI&#&AAI )#B*Y6 !.2^]; H HJ2ND @#s~~'?'??# @*= #B '((  ) -/F FFrJcl||}|r|||<|||<||}|r|||<|||<|||<|||<||k(r|S|S)a Swap positions of candidate and group in doubly-linked list (sink_waits version). Transforms (moves candidate to the left): group_head_prev -> group_head...group_tail -> candidate -> candidate_next Into: group_head_prev -> candidate -> group_head...group_tail -> candidate_next Args: candidate: Node to swap with group group_head: First node of group group_tail: Last node of group prev_dict: Dictionary mapping nodes to their previous nodes next_dict: Dictionary mapping nodes to their next nodes head: Current head of the linked list Returns: New head of the linked list (may change if group_head was the head) rq)rrrrrrgroup_head_prevcandidate_nexts r@+_perform_double_linked_list_swap_sink_waitsr/ss8 +O%. /"*Iiy)N$. .!*Ij&Ij%IiT KrJc*gd}|jD cgc]\}} t|| jdz | jdz | jdz | j dz | j dz | j| j| j| j| j| jg } }} dtjjdrddlm} | | |z n8dz t#|d zz d j%t't"| z t(j+t-|d |} t/| |k(sJt1| ||\} }}}d |z d | z t3d dfd| Scc} }w)a Format sink_waits statistics, log them, and return final node list. Computes improvement metrics, creates a formatted table (using tabulate if available), validates the reordered node count, recalculates peak memory, and logs all information. Args: stats: Per-node sink_waits statistics head: Head of the reordered linked list next_dict: Linked list next pointers original_snodes_num: Original number of nodes (for validation) peak_memory: Initial peak memory before reordering name_to_freeable_input_buf: Buffer memory tracking info graph_outputs: Graph output names Returns: Final reordered list of scheduler nodes ) z Wait noder$r%r&r'r(r)rZr\rYrrcr*r[r+rr,r-r/r0Nz* sink_waits_iterative peak_memory_before:z) sink_waits_iterative peak_memory_after:r1c dddS)Nsink_waits_iterative_infor3r4rqrqrJr@r7z2_format_and_log_sink_waits_stats..s/  rJcSrerq)log_strsr@r7z2_format_and_log_sink_waits_stats..s7rJr:)r=r>r_r`rarbrirWrZr\rYrrcr?r@rAr+rVrrBrCrDrrrr)rErrrFrrrr.r8rDrJr+rKrLr<rs @r@ _format_and_log_sink_waits_statsr`s8 G:!;;=  E4   NNS NNS  3 &    $   s "  LL    JJ OO     D "G~~ +%8    TT3w<$&&499Sd^,,W.tT9EJ z?1 11 1=.  OQ1 <[MJJG ;O;LMMG #  [ sBFctt}|j|t}|D]J}||}||k7r|jj }d} |D] } | |vs| } | 7|| j |L|S)a Find buffers whose last use will change after swapping in sink_waits pass. When we swap [group] candidate to candidate [group], some buffers that were last used by candidate will now be last used by a group node instead. This is the opposite direction from the reorder version. Args: candidate: The node being moved (currently last use) gns: Group nodes being swapped with candidate buf_to_snode_last_use: Mapping of buffers to their current last-use nodes candidate_buffer_map: Pre-computed map of node -> buffers using that node Returns: Dict mapping group nodes to buffers that will change their last-use node N)rr,rr rrr) rrrrrrrrr last_succ_gnrs r@._find_buffers_with_changed_last_use_sink_waitsrs0 D< *--iFN .s3 Y & ^^..   "BZ!  "    @ MTT  % , CBrJc  t|}|dk(r|ifSttjjj }ttjj }t|||\}}}}}} t|\} } } i} |Dcic]}|t|t|z}}|Dcic]#}|td|jD%}}|Dcic]}|td|jD!}}|d}t}tj}tj }d}|| || |}|rn|t||k\rnt#|r||vs|}6|j%|t'x}| |<t)|| | |||\}}}||z x|_|_||_||_||_| |}|}|}i}d} ||d}!t||}"t5|}#|.tj6r4|j,tj8 |j.zkr d|_n|"sd}$n||}%t=|%|"z}$tj>s%tA|rd|jC|_n|#xr t5|}&|$s|&rtE|\}'}(|'r|}|"jG|||#xs t5|}#tj6r8t5|r-tI|| |||\}}})||f||<tA|s| ||z } tK|!||d}!|xjLd z c_&| |}R|$s,tjNs9|&r7d |jC|_nd |jCd |(|_ntj6rAtQ|jRrIt)|| | |||\}}})tKd||z }*tKd||z | z}+|+|*kDrd |*d|+|_n%tA|s||},|,dkDrt|dkDrtKd|j.|j0z }-tKd|j.|j0z |,z  |jUD]*\}.}/|-tKd|.|/z tKd|.|/z |,zz z }-,|-dkDrd|,d|-|_ng|jWD]\}0\}.}/|.|/|,z f||0<tY||| }1||}2|2jZ|2j\z }3t_||1|| }4ta||1||!|3|2|4|| \}5}6}7|5|z |tjbzkDrd|5d||_n|xjdd z c_2|xjfd|jCz c_3ti|||| | | } t)|| | |||\}}}||_||_||z |_||_tk||1|3|2|4|6|7|| |r/d dl6m7}8|8||1tq|1tY| d| |||||d|4 }|rn | |}|.|}|| |tjrstY| d| }9|9| fStu| | | ||||}9|9| fScc}wcc}wcc}w)Nrc3<K|]}|jywrerrs r@rz1_sink_waits_iterative_internal.. rPrc3LK|]}t|r|jywrerRrSs r@rz1_sink_waits_iterative_internal..rTrUFrrVrr z$collective ordering with candidate:rWz non_group_reason:z"candidate is wait, exposed_before:z vs exposed_after:rXz2, group contains collectives, total_exposed_delta rYrZrr[sink_waits_iterative);rr rr]rr3r^rrr_rr`rrar rc(sink_waits_iterative_debug_limit_to_sinkrrrrrarbr_r`rcrr&sink_iterative_extra_comm_comp_overlaprWr*sink_waits_iterative_swap_with_collectivesrrrrfrrrZ/sink_waits_iterative_unsafe_collectives_reorderrrzr-r=rrrrr!sink_iterative_peak_memory_budgetrYrrrrhr\rrir):r5rFrrrrrrrrrrrrEr8rrrrmprocessed_waitsrndebug_num_sink_waits_to_reorderro _prev_currrDr_r`rcrrr group_collsrrrrtgroup_contains_collectivervcandidate_dep_namesboth_contain_comms _is_groupablegroupable_reasonr<r|r}r{r~ gc_comm_time gc_comp_timegcrrrrrrrr\rKs: r@_sink_waits_iterative_internalrsf+arz$.qww/C/C/H/H/J$KL%/0H0H0J%KM $FL-H"9@E5%35E0  "5),Ee,LLL0H0RXAHMyCu/@/@/BCCCAA >  y !44   >M>&,BZD lO==% ==$!&  uT{64[ $  + 7O$(GG d#O(CD D!)^+uT{-M %*:M. * 9l5> 4IIt1""($K     (.q1((8(>?$7$=!#BB""FFFWX(J$& '4I&># 36H HI  JJ,Y7>y?Q?Q?S>TU(";"?R@ -2S3/ /!!*J'--.>y.IJ1S5H5S. %KK/ :2R%uh8H-3/ 9a3>*/O! (% /+Iy!&)I ,A%BN$'9y+@=+P$QM%~5//=.>>PQ^P_a,1;!) 3I 1}[)9A)=), NNT^^;) Q ?) KLL:E:L:L:N6L,)SL<4O-PSV !<,#>#JT.M )1,"8 D88E!H!0" "-!2!2!4"!% , ,%1$09$<3" B ",IJ,C0@ /J #..1D1N1NN ?s$9;O D<%''K $  HN.0G, N NNO'~&6i }M$ JJ!OJ OO9#5#5#7"89 9O?:z5%E2ReUH.> 2 .Iy,'DN&DN!*Y!6D  ,D  9##G"'  0J,M %1%uE.! $*K -)-j)IQ#RY  uT{6\  4 425$F 5  1   "J u o 0 A>sY0;(Y5)$Y:ct|dS)a Similarly to reorder_communication_preserving_peak_memory this pass will try to iteratively push Wait nodes later, recomputing estimated peak memory before each swap, and preventing peak memory regressions. Pass will be applied to every Wait node. If there are immediate dependencies with next node, pass will try to group them together and on the next step to swap the group with next candidate. If _inductor.config_comms.sink_iterative_use_runtime_estimations is set True, pass will stop reordering of Wait once corresponding Collective is unexposed, based on runtime estimations. inductor.config_comms.sink_iterative_peak_memory_budget allows to tune how much pass can regress initial peak memory. E.g.: sink_iterative_peak_memory_budget == 0.0 - No regression of initial peak memory is allowed sink_iterative_peak_memory_budget == 0.2 - Pass can improve comm-compute overlap, sacrificing 20% of initial peak memory value. inductor.config_comms.sink_iterative_extra_comm_comp_overlap config allows to more aggressively sink waits, stopping only when overlap_compute >= (1 + extra_comm_comp_overlap) * comm_time r)rrIs r@rr@s. *& 1! 44rJctjdk(r|j}|SttjsJtj|}|S)z; Returns estimated op runtime in milliseconds (ms) rx)r r_r%callable)r8runtimes r@r_r_ZsR!!Y.--/ N22333,,U3 NrJc b|j}t|dk(rd}t|jtj tj frd|jDcgc]}|jc}}d|jDcgc]}|jc}}d|jd|jjd|d|d }|jDcgc]}|jj}}d j|D cgc];} t| tjrd | jd | j d nd=c} } |jj#} |jj&j(|| d| d|j+d dSg} |D]} | j-t/| |j&j(ddj| Scc}wcc}wcc}wcc} w#t$$rd} YwxYw)Nr r[zouts:zins: z (z) (),z (size=z , stride=z.0fz ns): z, ) get_nodesrrrzrExternKernelOut_CollectiveKernelrrrar5rsget_output_specrLayoutsizestridemaybe_get_nameAttributeError __class__rkr%rr>)r8r5detailrouts_strrins_strchildlayoutslayoutout_tensor_infor summaries child_snodes r@r>r>fs __ F 6{a ejj2#5#5r7K7K"L Me6G6G6IJ JKLHe.F.FGaffGHIG)*"UZZ-J-J,K4PXzYZ[bZccdeF=B__=NOE5::--/OO(( &  fbii0&++i a@    113I**&&//08II;VXY^YtYtYvwzX{{A AI4 k234oo&& 'r$))I*>)? @@/ KGO  I s+)H H/!H!AH)H H.-H.c&d}d}d}t|D]\}}|^t|r|t|z }|j}n$t |jrn|t|z }||t |ft|r.|t|z }|j}||t |t |jr||t |d}||dt |t jd|dz dz y)Nrc:tj|dd|y)Nz>6r )rCdebug)steprzs r@step_logz#visualize_overlap..step_logsT"IRu-.rJz| zEst. runtime (ms): i)r2rr_rzrr>rCr )ordertotal_est_runtime cur_comm_noder"r!r8s r@visualize_overlapr&s #M/!'; e  "5)!%8%??! % $!%8%??! Tl512 4"5)!%8%??! % ,u"5!68$,u"5!68 $ L$7#89:-;. /$6=>?rJc|}tjD]I}t|tr|t vr t |}t |s Jd|d||}K|S)Nz3Invalid reorder_compute_and_comm_for_overlap pass: z is not callable)r 'reorder_for_compute_comm_overlap_passesrrVglobalsr)r5r#ps r@$reorder_compute_and_comm_for_overlapr+sm E  ; ; a !wy. ! A{ A!DT U {%  LrJc t|jtttttD]\}}|jdk(s|j t jjjjusQ|jdjdk(sJd|d|jdd|jd}|jd}|dkDr|j||j|fd}tt}tD]\}}|jdk(s|j t jjjjusQ|}|jdjdk(sJd d |d|s|j|d }d D]}|jdk(st|j t j j"sB|j j$j&sc||rl||j)sJd |d|j+D]!\} t| D] \} } | }|jdusJ|j\} | dz} | t-| dz kr| | dzn t-dz }| |}t/fd|DrJdd|d|d|D]}|jdk(s|jvs"|j t jjjjk7s^t1fd|jD}||_ $|j3D]} | D]} | }|j5|!D]p}|jdk(s|j t jjjjusN|jd|vs`|j5|ry)a This FX graph pass replaces uses of FSDP2 unsharded params with their corresponding graph intermediates that were fsdp.copy_ into the unsharded params in the original graph. NOTE: Can only apply this pass to any of the FSDP2 unsharded params that have this pattern (or repetition of): `resize_(full) -> copy_ -> resize_(0)`. Because of this, for partial-graph case where `resize_(full) -> copy_` is in one graph and `resize_(0)` is in another graph, we can't remove these resize and copy ops and thus we will have worse performance there. In other words, "do we try to remove all the resize_(full) -> copy_ -> resize_(0) nodes for this unsharded param" is actually a per-unsharded-param decision, since for each unsharded param, we look at its resize sequence pattern (in `check_resize_pattern()`) to determine if its set of resize and copy nodes can be removed. call_functionr placeholderz1Resize can only operate on graph inputs, but got z# which is resizing non-graph-input r0r clj|g}j|g}t|t|k7r2tjd|dt|dt|dyt ||D]7\}}||k\s tjd|d|d|d |d|d yy ) NzH Unequal number of resize-to-full and resize-to-0 nodes for graph input z: z vs. zK. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass. Fz For graph input z: resize-to-full node z at index z happens after resize-to-0 node zd. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass for that unsharded param. T)rrlogwarningr) graph_inputresized_to_full_idxesresized_to_0_idxesresize_to_full_idxresize_to_0_idx&graph_input_to_resized_to_0_node_idxes)graph_input_to_resized_to_full_node_idxes node_lists r@check_resize_patternzLremove_fsdp2_unsharded_param_graph_input_usage..check_resize_patterns !J M M ! DGG UWX $ %-?)@ @ KKHHS}UE#&8"9!:; 47 !#54  / "_4  3I>P4Q3RR\]o\pq )/ :;:oEVW rJz\ Assumed all FSDP2 `unsharded_param`s to be graph input, but it's not true! Offending node: z . Graph: c|jtjjjj uxs:|jtjj jj uSre)targetr.rufsdpcopy_rxinductorresize_storage_bytes_)rzs r@is_allowed_mutationzKremove_fsdp2_unsharded_param_graph_input_usage..is_allowed_mutation sO KK599>>//77 7 O{{eii00FFNNN rJc nt|jtjjr^t |jj jDcgc])\}}|j|jjr|+c}}ng}t|Dcgc]5}t|j|jdj7c}}t|Dcgc](}t|jdj*c}}t||zdkDScc}}wcc}wcc}w)Nvalr)rr<r._ops OpOverloadr2_schema arguments alias_infois_writer r argsmetauntyped_storager)rzunsharded_paramsrrmutated_arg_idxesmutated_node_arg_storagesunsharded_paramstorages_of_unsharded_paramss r@-is_node_mutating_unsharded_param_or_its_aliaszeremove_fsdp2_unsharded_param_graph_input_usage..is_node_mutating_unsharded_param_or_its_alias s$++uzz'<'<=&dkk&9&9&C&CD Aq<<+ 0E0E   %/+ tyy|007GGIJ % ! (2(8 #33E:JJLM ( $ ,/KKLqPP)    s.D':D-"-D2zdUser mutation on FSDP2 unsharded param is not allowed when Traceable FSDP2 is used. Violating node: c32K|]}|gywrerq)rrzrRrPs r@rzAremove_fsdp2_unsharded_param_graph_input_usage..Y s#>d_DUVsz(Assumed no ops mutating unsharded param z in subgraph z, but it's not true! Graph: c3.K|] }|urn|ywrerq)rarg replacementrPs r@rzAremove_fsdp2_unsharded_param_graph_input_usage..f s%%(+o'= 3F%sN)r,rrr2opr<r.rur?r@rxrJrr=r>rrDrErF is_mutabler3r=rrtupler- erase_node)r]r?rzr2new_sizer:'unsharded_param_to_fsdp_copy_node_idxesfsdp_copy_noderAfsdp_copy_node_idxesrfsdp_copy_node_idxr<subgraph_start_idxsubgraph_end_idxsubgraph_nodesnew_argsr7r8rRr9rVrPs @@@@@@r@.remove_fsdp2_unsharded_param_graph_input_usagerds{U[[!I1*y) P T GG & uyy11GGOOO99Q ##..'-D=BBD eeidjk  : 1 6 6 8") %./C%D ) !A!&'9:N!&&q)_< <<+00NA{!3a!7 s/0144%QU+^a'  ''9:JKN* ))8(9~FVW w ' )GG.'4994 uyy'9'9'O'O'W'WW$%#'99% H!)DI )) ) ")J!H N N P-"6 - &'9:N   ^ , -- # GG & uyy11GGOOO ! GG   T " #rJc ddl jjsJ jjj r jjj sJ ddl m }m }m }m}m} fd}|}|| jjj j |t"j$| jj&j(j |d|d|d|d|d |d |d |d |d d fd }|||j+|y#tttf$rYywxYw)Nrr ) CallFunction KeywordArgMatchPatternMatcherPassregister_graph_patterncHt|j}|D]}|jtjus |j djj jjjusd|j ddk(sw|j|y)Nrr ) r,rr<operatorgetitemrJrur=all_gather_copy_inrxrZ)gr9rr.s r@remove_unused_getitemz8reinplace_fsdp_all_gather..remove_unused_getitem suM  AH,,,FF1I$$ (I(I(Q(QQFF1IN Q  rJall_gather_inputsall_gather_outputinp_split_sizesall_gather_input_numelrankitem_idx group_size group_namec&|jddk(S)Nrvr)kwargs)matchs r@r7z+reinplace_fsdp_all_gather.. s%,,z":a"?rJ) pass_dict extra_checkc lfd}|j||d|d|d|d|d|d|dgy) Nc|dd}|d}|d}jjjj|}|d}|d}jjj j||||}|S)Nrrr )out)rur=rnrx_c10d_functionalall_gather_into_tensor_out) rJ copy_in_argsrwrxrnrm getitem_1all_gather_into_tensorr.s r@replzEreinplace_fsdp_all_gather..reinplace_all_gather..repl s 9LbJbJ!J!B!B!J!J" )+G*1-I **EEMMZN # * )rJrqrrrsrtrurwrx)replace_by_example)r{rJrzrr.s r@reinplace_all_gatherz7reinplace_fsdp_all_gather..reinplace_all_gather s[. *$   *+*+()/0v|$|$  rJ)r{rh)5torch.distributed.fsdp._fully_shard._fsdp_collectivesr'rrurrr ImportErrorrAssertionErrorpattern_matcherrfrgrhrirjrxrlrmr=rnapply) r]rfrgrhrirjrp graph_passrr.s @r@reinplace_fsdp_all_gatherr| sd D  --/// II & & = = **EE  FE   $%J II & & = = E E   IINN55==23230178v& :&  | $ | $ $?), -, B% UA  8sA"EE10E1ct|tjjjtjjj frJt |jddS)N)rr. _inductor schedulerFusedSchedulerNoder~rXrrs r@ get_op_idxr s] OO % % 8 8 OO % % : :   u~~# $$rJc ~  !ddlm g}tt}d}d}i}i}i! !fd} |D]} t | j t jjjjrtfd| jDrd}| } t} t| | |tt jjjjt jjjjt jjj jgt#| | | fd t%| d } t'| }d }t)t'| D]W}| |}t+|j t jjj jr|dz }|dkDsU|}n| d|} d}t)t'| dz D]3}t-| |dzj t.j0s.|dz}n|J| | d|}| | |d}|||<'t+| j t jjj2jskd}| }t}t#|||t%|d }d}t)t'|dz D]3}t-||dzj t.j0s.|dz}n|J| |d|}| ||d}|||<t'!d kDsJ|rt'|d kDsJ|rt'|d kDsJ|D]N} | j5!vr!| j5} | |vr-|j7| |j9| Pd}|j;D]k\}}|bt=t?|jA}|jCD]-}|jEtG|j5|d/|}md}|j;D]k\}}|bt=t?|jA}|jCD]-}|jEtG|j5|d/|}m|S)Nr )rFcjj|}|D]}||j<||j<|Sre)r~creater)snodes_to_group group_noder8rsnode_name_to_final_snodes r@_create_group_nodez:enforce_comm_ordering_for_fsdp.._create_group_node sV33::?K $ EE:D %enn&6 7 E;E!*"5"5"78rJ)rWc3K|]I}t|jtjjj j Kywre)rrzr.rur=rnrx)rrrs r@rz1enforce_comm_ordering_for_fsdp.. sD  "1%**EIINN,M,M,U,U  sAATct|jxs0t|jxr|jjv Sre)rNopKernelSchedulerNodeExternKernelSchedulerNoderz op_overload)r allowed_opsrs r@r7z0enforce_comm_ordering_for_fsdp... sFq)"B"BC"1i&I&IJ>FF..+= 'rJ) criteria_cbct|Srerrs r@r7z0enforce_comm_ordering_for_fsdp..9 JqMrJrrct|Srerrs r@r7z0enforce_comm_ordering_for_fsdp..j rrJr)$r[rr rrrzr.rurrrxrrr wait_tensorr=split_with_sizes_copyrsortedrr*rrr _WaitKernel chunk_catrrrr=rrrrrr)"r5rr new_orderr ag_exists rs_exists$ag_grouped_node_to_wait_grouped_node$rs_grouped_node_to_wait_grouped_noderr8ag_snodeag_related_snode_setag_related_snodesend_idx_of_current_ag_blockcopy_out_countr cur_snode wait_node_idx ag_group_nodeag_wait_group_noders_snoders_related_snode_setrs_related_snodes rs_group_noders_wait_group_node prev_ag_waitwait_group_noderr prev_rs_waitrrrs" ` @@@r@enforce_comm_ordering_for_fsdpr s )+I3!III+-(+-( "nU  JJ59955PPXX  __   IHLVL  ($"  %II..IIQQII..::BBIINN88@@K )$" !'$*A!  +..?*@ 'N3012 -a0 !NNEIINN$H$H$P$P#a'N!A%23/ !22N3N O !M301A56 /A6;;R^^L$%EM !, ,,./@-/PQM"44Emn4U!V BT 0 ?EJJ (@(@(H(H IIHMWL ($"  !'$*A!  !M301A56 /A6;;R^^L$%EM !, ,,./@-/PQM"44Emn4U!V BT 0 ?]nU` ( )A -- -781<<<781<<< >> 8 8-enn.>?E I   e L*N*T*T*V'&   #]%C%C%E FGL!--/ **AJJL|TR ' 'L*N*T*T*V'&   #]%C%C%E FGL!--/ **AJJL|TR ' ' rJ)r5list[BaseSchedulerNode])r5rrr)rzzIRNode | Operation | Nonerr)r8rrr)r8r)F)rrV)rrrrV)r5rrz~tuple[dict[BaseSchedulerNode, BaseSchedulerNode | None], dict[BaseSchedulerNode, BaseSchedulerNode | None], BaseSchedulerNode])rrr#dict[BaseSchedulerNode, OrderedSet])r5rrz(dict[BaseSchedulerNode, OrderedSet[str]])rBaseSchedulerNode | Nonerrr1dict[BaseSchedulerNode, BaseSchedulerNode | None]rr) rrrrr'dict[BaseSchedulerNode, frozenset[str]]rrrr) rrrrrdict[BaseSchedulerNode, float]rrrrrtuple[float, float, str])rrrrrrrrrrrrrr)rrrrrrrrrrrrrr)rrrrrrrrXrrXrrrrrrrz(tuple[int, dict[BaseSchedulerNode, int]])rrrrrrrrXrrrrrdict[BaseSchedulerNode, int]rrrrrrrrU) rrrrrrrrrz8dict[BaseSchedulerNode, list[FreeableInputBuffer | Any]])rrrztuple[bool, str | None])rEz$dict[BaseSchedulerNode, ReorderInfo]rrrrrFrXrrXrrrOrderedSet[str]rr)r5rrzDtuple[list[BaseSchedulerNode], dict[BaseSchedulerNode, ReorderInfo]]) r5rrDrrErrFrrr)rrrr)rrrrrrXrrrrrrrrrrrrrrU)rrrrrrrrXrrXrrrrrrrrrzFtuple[int, dict[BaseSchedulerNode, int], dict[BaseSchedulerNode, int]])rEz%dict[BaseSchedulerNode, SinkWaitInfo]rrrrrFrXrrXrrrrrr)r5rrzEtuple[list[BaseSchedulerNode], dict[BaseSchedulerNode, SinkWaitInfo]])r8rrr^)r]torch.fx.Graph)r]rrrU)r51list[torch._inductor.scheduler.BaseSchedulerNode]rz4dict[str, torch._inductor.scheduler.SchedulerBuffer]rzdict[str, BaseSchedulerNode]rr)f __future__rrr?rloggingrlr collectionsr dataclassesrtypingrrr.torch._loggingr torch.multiprocessing.reductionsr torch.utils._ordered_setr r[r r r dependenciesrrrmemoryrrrrutilsrrrrrrr virtualizedr getLoggerrkr0_logginggetArtifactLoggerrCr$rrArErDrMrRrTr{rrrrrrrrrrrrrrrrrrrrr"rMrOrHrrrrrrrrrrr_r>r&r+rdrrrrqrJr@rs#  #!% +;/&&!% g!nn..xC ;#F ##FL #&# ###L 99 9("(032 #"((((0 #-"G ; " "A 6 .' .! .> .; .  .9.'9.@9.-9.> 9. ; 9.  9.x9.!9. 9.A9.- 9. > 9. ; 9.9.x. .!.".A . A .  ..b@. @. @."@. @.  @. % @.6:@.@..@.FJ J J"J J % J 6: J4JJ JJ JZ$; $; $; $;> $; > $;N B 2 4 2 92 2 2  2 jCG CG CG"CG CG  CG % CG>BCGCGCGLCGL. .!.".A . A .  ..bW 0W WAW W  W !% W#WWt4C 4C 4C 4C> 4C > 4CnL #LJL^ 54 A>#L  #   ~#BjZ%n =nEn5n7 nrJ