Mj&. UddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZm Z ddl!m"Z"dd l#m$Z$m%Z%m&Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.m/Z/dd l0m1Z1dd l2m3Z3dd l4m5Z6ddl7m8Z8gdZ9dZ:dZ;dZe?Z@e1ezejzeBzeCze?zZDeDeEeDzeFeDzeGe?dfzZHeGe?eHfZIeEeIZJeGe?eIeJzfZKe>ZLe>eeMd<ejdZOe GddZPe GddePZQ dIdejde?de?deSdeSd e@f d!ZTGd"d#ZUdJd$ZVddd%dejd&eFejjd'fd(eSd)e>ejdzd*ePdzd eQf d+ZYd,eGe?eHfd-eKd.eQd dfd/ZZd0ejejjzd1e?d efd2Z[d3eGe?efd.eQd eGe?effd4Z\ejdejd.eQd eGe?eHffd5Z^ejdejd3eGe?eHfd.eQd e3fd6Z_d7ejjd dfd8Z`d3eKd eGe?eHffd9Zad7ejjd3eGe?eHfd.eQd eKfd:Zbejdejd;eFejjd'fd.eQd eKfd<Zcdejd7ejjd-eKd.eQd eKf d=Zdejdejd;eFejjd'fd3eKd.eQd df d>Zeddd%dejd)e>ejdzd*ePdzd eGe?eHffd?Zfddd%dejd;ejjeejjzd)e>ejdzd*ePdzd eKf d@Zgddd%dejd;ejjeejjzd)e>ejdzd*ePdzd eFeGe?eHfeKff dAZhdejd3eGejeGe?eHffeGe?eHfzd eGe?eHffdBZiddCdejd,eGe?eHfd*ePdzd e3fdDZjddCdejd;ejjeejjzd-eKd*ePdzd df dEZkddCdejd;ejjeejjzd,eGe?eHfd-eKd*ePdzd e3f dFZleddCdejd*ePdzd dfdGZmeddCdejd;eFejjd'fd*ePdzd dfdHZny)KN)Callable GeneratorIterable)asdict dataclassfield)chain)Anycast no_type_check) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater%_patched_state_dictc#Ktj}tj d|rtjyy#|rtjwwxYwwN)gc isenableddisableenable) is_enableds f/var/www/html/ia-enhanced/venv/lib/python3.12/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextr=QsDJJJL  IIK : IIK s)A$AA$A!!A$ceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <y )r)ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__r?bool__annotations__r@rArBrCrDrErGstrr<r)r)\s_"H"OT!K!&$&$(T(FD!&$&). $.-s-rPr)cXeZdZUeeZeeejze ejzfe d<eeZ eeejze ejzfe d<ee Z e ee d<dZee d<dZee d<ej$Zee d<eeZeej0e d <y ) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rHrIrJrdictrTrNtorchTensorr#rMrUsetrVrWrLrX contextlib nullcontextrYrlistrZnnModulerOrPr<rRrRs d#t ell $ d#4 ell $$)#=C=L$L$'33L(3$)$$?L$ryy/?rPrRmodelnamerGskip_ddp_prefixskip_compiler_prefixreturnc|jtd}d|vr|hS|jd}g}|}t|D]\}} t |t r5| dk7rt d| d|j}|r:|j| Lt |tr|t|dz krW||dztk(rHdj|} t|t} | r| d} | jD chc]} | |  c} cSt|t}| tk7s|j| t|| }t |t j"j$j&r7| dk7rt d| d|j(}|rN|j| at+||r:t||j-| x} rt+|| r t|| }|j| | t.j0jj2k(r|t|dz k7st5d t|| } dj|jtdhScc} w) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulezExpected 'module', got '' _orig_modzExpected '_orig_mod', got 'z-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorrlappendFSDPlen _FLAT_PARAMjoingetattr_fqnsrr\_dynamo eval_frameOptimizedModulerohasattrgetrbmodules_EXTRA_STATE_KEY_SUFFIX RuntimeError)rdrerGrfrg obj_names fqn_obj_namescurr_obji curr_obj_nameprefix flat_paramfqn removed_fqns r< _get_fqnsrsS. <<*B /D $v  3IMH%i0&<= h $($'? a%PQQH"$$]3 $ '3y>A%%)AE*:k*I-0$X{;  &xq\F4>4D4DES6(3%(EEx)<=H 33$$]3"8]; %--":":"J"J K +$'B=/QR%STT))H'$$]3x!23"F'(4E"F"H"L"L!#;x5#*8[#A   / 1 1 I III**&'VWW"8];M& C DD7Fs0 I0c eZdZy) _EXTRA_STATEN)rHrIrJrOrPr<rrsrPrc#Ktdtjdtdtffd |dEd{y7w)Nrlcurr_fqnrhc3Kj||r|dnd}|jD]T\}}|vr t|r'|t|j vr|dd}n||}||Ed{Vt |j d|jdD] \}}||jvr||}||f"t|jdtjjtjjk7r7|tjjj}|t!fyy7׭w)NrkrjF)recurseget_extra_state)addnamed_childrenrr{valuesr named_buffersnamed_parameters_non_persistent_buffers_set __class__rbrcrrrlrr) rlrre submodulenew_fqnobjrGrvisited_moduless r<rz+_iterate_valid_model_state..recursesrF#%-hZq>2%446 3OD)O+ 12>GF,=>@GGII#3B-%Jtf-y'2 2 2 3   /1H1HQV1H1W  ID#v999! 4&)G3,    F$$&79R9R Syy(( )" 2::#4#4#L#L"MNG<>) )  ) 3sA;E>E?CErj)r^rbrcrNr)rdrGrrs `@@r<_iterate_valid_model_staters?&)eO * *S *Y *Dub!!!s;AAA) submodulesoptionsoptims. optim_onlyrrc |rtjdtd|r |s td|xs t }i}i}t |D]\}}t |trt||} |j|} | 2ttt||j| ||||<n| j||<| D]} t |tr||| <t|j!D])\} } | D]} tt"j$| || <!+t} |ret|}|j'D]G\}}||vr t||} t)| dk7r t+d| jd| DI|j,r|j.s t1dt3j4|}|r|j.rat7|j8|j8 }t;|j8|j8xs |j, }t<j>}nz"_verify_options..Ks%@CQi%@sz?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc3Ktj5tjddtt j ||||5dddddddy#1swYxYw#1swYyxYww)NignorezFSDP.state_dict_type)messagecategoryrlstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningrwrrs r<$fsdp_state_dict_type_without_warningz=_verify_options..fsdp_state_dict_type_without_warningisy((* ''&<}))!$3&7,C       s4A;6A/ A#A/ A;#A, (A//A84A;rr)rTrUrVrYrZrWrXrO),rwarnrrr)rrsrrrr r^rNupdatecopyraitemsr\r] named_modulesrxrurDr? ValueErrorrwrZrr@rrFULL_STATE_DICTrrSHARDED_STATE_DICTr_contextmanager functoolspartialr`rRrrbrc)rdrrrrrTrUreparamfqnsrparam_fqns_rVrlrZrrrrrYs r<_verify_optionsrs#  I   & I  +)+GKMOQ1%8/ e e\ * %##E* ? S,U3 4 ; ;D A+>O  " "  # $!(( 0+/$;  "--   / +3-!$ryy/<8#^&kAo  rPmodel_state_dictoptim_state_dictinfoc |jD]}t|}|td|jrk|si|js]|j sQ|j r |js9|jr-|js!tdtjd|jr4|s2|j r |js|jstd||D]}t|vs t|dtdy)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rkzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rZrrurWrVrAr@r?rCrDrdistget_rankrXry)rrrrl fsdp_statekeys r<_verify_state_dictrs ##NCFK   !LM MN  ''))!!d&:&: KK)) 'mmo'q *    %%$*>*>..::J9KM    # %z+/** rPrapict||}|tvr+tjt|j||}|S)N)self)r{r4rrr)rrcalls r<_state_dict_fnrs9 3 D ""  !<3G KrP state_dictc|jrF|jrtjj sdnd}t ||j|S|jr t |S|S)NrO)r)r@ ranks_only)r?r@r\ distributedis_initializedrr)rrrs r<_maybe_full_or_cpu_state_dictrsn $$E,=,=,L,L,N   " D$4$4     )*55rPc f|jsiS|j5t|d}dddtj D]}t ||}t |dk7rtd|dt |d|tt|}||k7sTdtfd}|||std|d ||j|||<|jrYi}|D]P}|jD]?}|j|s|jr ||||<*|t |d} |||| <AR|}|j rI|j#D]6\}} | j$rt ||}|D]}|j|8t'||S#1swYwxYw) NrrnExpected 1 FQN for key '', got z: rhc t|t|k\ry|jd}|jd}d}t|D]:\}}|||k(r'|dz }|t|k(s"|t|dz k(cS|dvr:yy)NFrkrrn)rlroT)rxrqrr)rr fqn_split key_splitfqn_idxkey_idxkey_names r<verifyz%_get_model_state_dict..verifyss8s3x' IIcN IIcN )29)=%%GX9W#551 "c)n4#*c)nq.@#@@!%<< $%rPzAn unexpected key, z, exists. FQN is )rWrYrrakeysrrxrunextiterrLrpoprV startswithrBrAr requires_gradr) rdrrrrrrnew_state_dictrrrs r<_get_model_state_dictrs      ;8^E<8: ;JOO%&2$ t9> *3%ws4ykD6J 4: #: D "#s#"%8=Nse#TUU(nnS1JsO=2@ /1 >C11 >~~f-//*4S/N3'!#f+-0G.8oN7+ > >$    002 $JC""UC(D $s# $  $ )T ::s;;s F&&F0c |jr|s|js tiiSi}t||jD]\}}t |||j}t |||jdd}t ||D]f\}} |jrtjdk(r9|| k7r4|j|d} | |jrtd|d| || <||| <hd} |js |jrnt} |jD]G}tj |s|j#dkDs-| j%|j&Itj&d| vr&| j)tj&dd} t+| dk(r.| j%tj,j/nt+| dkDr t1d |jr3t3||| j|j|j4 n(|jrt7||| j |j9||j;5t=tt?|d ||j| cdddS#1swYyxYw)NF)rfrgrz Missing key: rkmetaTrnzMultiple devices found)devicerCr@rload_state_dict)rrCassign) rWrDr rrGrziprrrrCrr?r^rr\ is_tensordimrrremoverxdistributed_c10d_get_pg_default_devicerrr@rrrYr r) rdrrlocal_state_dictrvaluerfqns_with_prefixrfqn_with_prefix load_valuerdevicess r<_load_model_state_dictrsq   Z8Q8Q R((08N8NO6 UT%;%;<$    " "!!&  %(.>$? 6 C--A1E('^^C6 %{{*]3%q+ABB2 S!   JJtJ))+ ;  # K + OOO%rPc < dtttfdtdtttff fd d i}t t |t jD]C\}}t d|}|j t tttf||Et t|tD]\}|jt}t tt|D]+}|jD]\}}||td|d|<-^|S)a3 This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_groups": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_groups.layer1.weight.lr": 0.1, "param_groups.layer2.weight.lr": 0.1, "param_groups.layer1.weight.betas": (0.9, 0.95), "param_groups.layer2.weight.betas": (0.9, 0.95), } The "state" section supports arbitrary levels of nesting for optimizers like Shampoo. nested_dictrrhci}|jD]Q\}}t|}|r|d|n|}t|tr|j ||E||||<S|S)a Recursively flatten a nested dictionary with dot-separated keys. Args: nested_dict: The dictionary to flatten prefix: The prefix to prepend to all keys Returns: Flattened dictionary with dot-separated keys rk)rrNrsr[r) rr flattenedrrstr_keyfull_key_flatten_state_nested_dict_raise_if_type_not_supporteds r<rz=_flatten_optim_state_dict.._flatten_state_nested_dicts+- %++- ,JC#hG06&7),GH%&  !;E8!LM-U3&+ (# ,rPct|tjttt fst dt|dy)Nz[Flattening optimizer state_dict only supports tensor, int, float, dict states now. Type is rk)rsr\r]intfloatr[NotImplementedErrortype)vs r<rz?_flatten_optim_state_dict.._raise_if_type_not_supporteds@!ellC=>%7)1& ?rPrk)r[rNr r%r r&_STATErrr'_PGrr ra) rretrr3 state_prefixrrkr!rrs @@r<_flatten_optim_state_dictr's R#s(^-0 c9n <!#C=*V*<=CCE U 3%(  &tDcNE'BL Q  -z#?, w'S 4( ,C#))+ ,1*+se1SE1#&' , ,, JrPc dtdtttfdtttffd}i}g}t|t|i}|j D]}|j tgi|tD]}|j|D]} | |jvr)d} |D]!} | tk(r td| d| } | |vrd} nnd} | s?|dt} t| tstd t| | j | |jsi|| <|j|D]M}td| d|}||vr |||}|t!t"|| |<5||t!t"|| |<O t!tt|dtd }|D]V} | tk(r |td|d| }| |dvr ||d| <.|d| |k7s:t%d |d| d |d |d| d |S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). Supports arbitrary levels of nesting in the state section through recursive reconstruction. See the docstring of _flatten_optim_state_dict() for more detail. flattened_keyflattened_dictrhc|d}i}|jD]q\}}|j|s|t|d}|jd}|}|ddD])} | |vri|| <t || t st || }+|||d<s|S)z Reconstructs a potentially nested value from flattened keys. For non-nested values, returns the value directly. For nested values, reconstructs the nested structure with string keys. rkNr)rrrxrqrsr[ru) r)r*rrrr remaining_keypartscurrentparts r<_reconstruct_nested_dictz=_unflatten_optim_state_dict.._reconstruct_nested_dicts"?!$&( )..0 'JC>>&) F .M"'',E!Gcr  (w&$&GDM!'$-6((!$- ("'GE"I ; '@rPFrkTrExpected list, got rzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rNr[r%r"r#r1rvr rTrUrsrarur rr3r r&r)rrrr0r3pg_state return_osdrrr in_paramsr& flatten_keyr2 state_nameflattened_state_keyreconstructed_valuefirst_param_fqnrs r<_unflatten_optim_state_dictr:s00,0i,@0 c9n 0dE"$H&,eS(%CJ))A " & )1 E--e40 $444 %I(<$),Qse1QC&8 &*4(,I !%I !"g.!&$/(+>tF|n)MNN c"** c "'++e"4J-3HAcU!J<*H'**<.F//+ 0]E#J7 C GQ/G]E#J7 C?0 1 ftCy(2,w*?@C AG|#a'8!=>E $"' Q"aE)"==Lrt5t@tC|}tE||S#1swYxYwcc}wcc}w) Nrroz _orig_mod.rjc3.K|] }|tywr6)r )rgs r<rz(_get_optim_state_dict..s-UQaj-Usrnrr)#rXr"r#rrrZrYrwrrarrrpr r from_iterabler1r[rrangerxrrrurrr r&rr'extendrEr(r'r)rdr;rrrosdr&r>r2param_pid_mappingfqn_pid_mappingrrrrpidgroups r<_get_optim_state_dictrGus    ,2BR+@4H% 1nUL13   ""$ ?++E5#> ?#f+**,- R!#?B6{q?QCK , ;< RX $?@zJ!!))L"5JJ#'  $%---U%BTBT-UUVF $Ss6{1C%D E  O#446 + U ,t9>(23%ws4ykJ4:& 11'.'*$'*$ + CK,,./ 8%c*$'v;??3#7F C  8 S RBG.!Q3/#"6!Qg R  ],V45<} t'ttt| td k(s.|j | @t'|dk7r t+d t'|tt'|j k7r t+d t'|tdz |t) <wtt|tD]M}|j-t)|d}|dk(r$|j/D]\}}|tk(r||||<O|S) a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. c3<K|]}t|tywr6)rsr)rr&s r<rz*_split_optim_state_dict..s U!:a  UsFTrr1z'Missing optimizer state for parameter 'z' in checkpoint. The parameter requires gradients but has no saved optimizer state. To load anyway, use StateDictOptions(strict=False).rnrzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)r"r#allr r&r1rvr rTrUr'rarNrsrur rrCrrxidrrr)rdrrrr3r2r3 pg_mappingrrrr4loaded_param_groupr2r$pg_idxrrs r<_split_optim_state_dictrOsI*E"$H&,eS(%CJ!#J UtM;KF;S'T UU))7J " & ) VE--e4 V$444 %I.2)+;C+@/"*$tCy2DW2M"NN(,I! "!%I !"g.!&$/(+>tF|n)MNN c"&&d=2B62JKK%)-9I&9Q%RSV%Wc *EcUKRR +/%'7'<+V&d49.@.IJJ=@C=QTU=U 2&8#9: V7 V VD {7# $ )C&*+<>Ns>S&T 3"tDI'9''BCDIJJ12 33x1} 1#C()S1C1C-DD =25Z_1E1IJr,- .o7Jr-/?/DE * ;4 R< %++- *JCg~$)HV S !  * * rPc 0|jsy|D]}t||r@t|vrt||||}n+t |t t ttf||}ni}|jrl|jD]&\}}t||}t||d} || k(r't|dk7rtd|dt||j} | j} |tD]N} t t tt f| } | t"Dcgc]}|j%| | }}|| t"<Pt t&|t}t)|j+D]+}| |vs|j|||j%| | <-)|j-5t/j0|||}dddn|j2rd|_t5||f|}d|_dfd}t7t8j:||} tdt=|\}}t=|\}}|j>rtA|| ntC|| |D]*}||vs||vrtd |d ||||<||||<,tE||}|tD]/}t"|vs gt t ttf|t"<1tG|d | ycc}w#1swY%xYw)NF)rgrnzExpected 1 FQN for 'rTc|jdkDr*|j|S|jk7r td|S)NrzDevice mismatch)rrr)trs r<_devicez'_load_optim_state_dict.._device^sD557Q;~!" 188+():;;rPzExpected device to be setrzExpected key 'z' in osd_mappingr)r)$rXrr"rOr:r r[rNr%rZrrrxrurr#r r rpr&rarrYrwoptim_state_dict_to_loadr?rGr"r\r]rrDrrrr)rdr;rrrr original_fqn_rfqns_with_compilerrfqn_with_compilerr>valrr2 osd_stater&rrS flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keypgrs @r<_load_optim_state_dictra!st   [N% ##:5*d$ $?4S)^ 4jA4$  "    $)#9#9#; X a  5%.(.|nGCI;Ohhj$6$:$:$<!)#.*AtCH~q1CGJ7|@C C):;F$*CL * !0@0HI inn./XAaxGP}}UVGW !))C1B"CDX+ X2""$ #'#@#@5"2$    ! !#(D 4UUHdK #'D F ellG5EFA~$%@AA':;K'L $K3FGW3X 0 0((%k3DVT&{4EfU ) J $55 3,,YK7GH4?y3I%i03>y3I%i0 J 5!#4  's+ A"$>@Dc9n-r27; A 1u/0>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrcN) r=rsr\rrerfrrrGr)rdr;rrrrrs r<r,r,sP 2*ekk&;&;<Mz"    !  1=0 DI+-=tD!11!222s A,BB c |siSttt|jtj rt jdtdtttj tttff|}i}|jD]\}}|jD]}\}}||k7r t||}t!|dk7r t#dtt|d} |j%|jD cic] \} } | | z|  c} } |Sttttf|Scc} } w)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rrrnz/FQNs for a submodule should only have 1 elementrk)rsrrrrbrcrrrr r[rNr%rrrrxrur) rdrcast_state_dictrrsub_state_dictremrrsubfqnrs r<_unflatten_model_state_dictrm6sF  $tJOO-./;  "   tBIItCN/C$CDjQ/1)8)>)>)@  %I~ ..0 a > -t9>(I!d,-Q/%%AOAUAUAWX Vf_e+X  Di(*55 YsE)rct||}t5t|dd|}t|i|t |||cdddS#1swYyxYw)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] rOFrrN)rmr=rrr)rdrrrs r<r-r-[s`:.I . EubUGL+R6%e-=tD EEEs )A  Act5t|tjjr|fn t |}t ||d|}ti||t||||dddy#1swYyxYw)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TroN) r=rsr\rrerfrrra)rdr;rrrs r<r.r.sz>  J*ekk&;&;<Mz"  ujT7S2/6uj2BDI J J Js AA11A:c.t||}t5t|tjj r|fn t |}t||| |}t|||t||||t|||cdddS#1swYyxYw)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType roN) rmr=rsr\rrerfrrrar)rdr;rrrrs r<r/r/s\.I .  E*ekk&;&;<Mz"   :.>*>  +-=tDuj2BDI%e-=tD E E Es A*B  Bc$tjt||fd}||_tjt||dt t tfffd }||_tj|tj|y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rdrcSr6rO_state_dict_callsr<state_dict_callz0_patch_model_state_dict..state_dict_call !!rPrc|y)N)rrOr_load_state_dict_calls r<load_state_dict_callz5_patch_model_state_dict..load_state_dict_call z:rPN) rrr*rr-r[rNr rr4r)rdrrvr{rzrus @@r<_patch_model_state_dictr}s6!(( "'E%-- ;c3h;1EO,01rPctjt|||fd}tjt|||dtt t fffd }tj|tj|t|tjjr|fn t|}|D]}||_||_y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rdr;rcSr6rOrtsr<rvz4_patch_optimizer_state_dict..state_dict_callLrwrPrc|y)N)rrOrys r<r{z9_patch_optimizer_state_dict..load_state_dict_callVr|rPN)rrr+r.r[rNr r4rrsr\rrerfrr)rdr;rrvr{rrzrus @@r<_patch_optimizer_state_dictr&s>!((  "&--  ;c3h;O,01 j%++"7"7 8  :  5* 45rP)rFTT)rF)or_rr7rcollections.abcrrr dataclassesrrr itertoolsr typingr r r r\torch.distributedrrtorch.nnrb'torch.distributed._shard.sharded_tensorr #torch.distributed._state_dict_utilsrrrrrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrrwrrrrr$torch.distributed.fsdp._common_utilsrrtorch.distributed.tensorrtorch.nn.modules.moduler torch.nn.parallelr!rttorch.utils._pytreer"__all__ryr#r r"r^rNr#r]rrr$rarfr[r%r&r'r(r4rMrr=r)rRrcrLrrrrrerrrrno_gradrrrr'r:rGrOrar*r+r,rmr-r.r/r}rrOrPr<rsq 9900++ A   -5<- "    S-'%,,6>c;FVAWW S)^$ '#}/@@@A&)US]*  ,.,. ,.^  @% @  @&. !% FE 99FE FEFE FE  FE  FER  %"Z)-'+ } 99} %++'', -}} BII% }  $ }}@+3 >*+(+ + +\ EKK$9$99S#X&4 #s(^$?; 99?;*?; #y.?;?;DA 99A S)^$A  A  A A H'&U[[22'&t'&T_*<_c9nAU_DE ;; ES)^$E E EPDA 99DAekk++S01DA DA DADANa 99a ;; a)a  a  aHdN 99dNekk++S01dN#dN  dN  dNdNT)-'+ " 99" BII%"  $ "  #y. " R)-'+ * 99*  %%1F1F(GG* BII% *  $ *  * b)-'+ X2 99X2 %%1F1F(GGX2BII% X2  $ X2  4Y !3 34 X2v"6 99"6RYYS)^ 445S)^8LL"6 #y."6R(, $E 99$E3 >*$E $ $E  $EX(, (J 99(J %%1F1F(GG(J)(J  $ (J  (Jb(, =E 99=E %%1F1F(GG=E3 >* =E ) =E  $ =E=ED(,12 9912 $12 1212l (, ;5 99;5ekk++S01;5 $ ;5  ;5;5rP