Mjh7UddlZddlZddlmZmZmZddlmZddlm Z m Z ddl Z ddl m Z ddl mZddlmZddlmZgZeeed<ej0eZGd d ej6Zd eed efd Zy)N)Callable CollectionMapping)deepcopy)Anyoverload)optim) ShardedTensor)FullyShardedDataParallel__all__ceZdZdZ ddeeejezfde jde eee fdzde jdzdee dfd eee fd dfd Zdd Zd eee ffd ZedddZedegefd efdZddegefdzd edzfdZed eeje ffdZdeee fd dfdZdeee fd dfdZddZdeee fd eee ffdZdeee fd eee ffdZy)_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmoduleargs.kwargsreturnctjjd||_|j t ||_||j jn|}||g|i||_||_ |)t|j j|_ ntjdd|j jD cic]\}} | | } }} g} |D]3} | dD])} | | vrt!d| d| j#| | +5| |_ |jj|_ycc} }w)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.) stacklevelparamszExpect param name z% found in param group but is missing.)torch_C_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrrrparams_for_optimizerkeyparam param_to_keyr#groups f/var/www/html/ia-enhanced/venv/lib/python3.12/site-packages/torch/distributed/optim/named_optimizer.py__init__z_NamedOptimizer.__init__?sk $$%NO;G   " $%5 6.:.BD ! ! ( ( *  *       &*4+@+@+E+E+G&HD # MMN  :>9N9N9T9T9VW:3E3JWLW!# % C"8_CEL0(07\]'--l5.AB C C'9D # OO88Xs E c||j|jD]}t|ts tdd|vr td|d}t|tj r|g}t |}|D]=}t|tj rtdt j|z||d<yy)Nparam group must be a dictrz#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is ) r isinstancerAssertionErrorrTensorr! TypeErrortypename)r) param_grouprr,s r/rz#_NamedOptimizer._param_groups_checkjs    (#00 / !+t4()EFF;.()NOO$X.fell3$XFf#E%eU\\:'8:?..:OP )/ H% / )c|jj}|d}|djDcic]\}}|j||}}}g}|D]k}|dDcgc]}|j|} }dt | i} |jD]\} } | dk7s t | | | <|j | m|j||dScc}}wcc}w)z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rstater)r;r)r state_dictr&r#sortedrr(_post_state_dict) r)r<rst_key state_val ret_state ret_groupsr.r, param_keys ret_groupkvs r/r<z_NamedOptimizer.state_dict}s__//1 !.1 &0%8%>%>%@ !   # #F +Y 6   ! )EFKHoVU$11%8VJV!6*#56I  /1=#+A;IaL /   i (  )$$y*%UVV Ws C!CclosurecyNr)rGs r/stepz_NamedOptimizer.steps25r9cyrIrJrKs r/rLz_NamedOptimizer.steps;>r9c:|jj|S)z Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rG)r rLrKs r/rLz_NamedOptimizer.steps##G#44r9c.|jjSrI)r r;)r)s r/r;z_NamedOptimizer.states$$$r9r<c |jj}|j|}|d}|d}t|dk(r t dt |j D]\}}||vr t||t||k7r,t dt||d|dt||||jD]l\}}|||vrt d|d|d|||} t|trt| tstt|j} t| j} | | k7rt d | d | d |d |t|j| jD]8\} } | jjj| j:t|t j"rAt| t j"st|jj| \t%| |||<o|d }|d }i}|D]}t'|d}||t)|< i}|D]:}g}|dD] }|j+|j |"||t)|<<|jD]\}}||vr ||}t|t|k7r't dt|d|d t|d|D]/}||vrt d|d|d|dk7st%||||<1|jj-|y)a Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. r;rzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rrz"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r r<_pre_load_state_dictlenr' enumerater#r&r3r r4 local_shardsziptensordetachcopy_rr5rr!_gen_param_group_keyr(load_state_dict)r)r<new_state_dictr; new_stateidx param_key state_keyr@ src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr.rC new_group_map new_group group_key src_grouprEs r/r]z_NamedOptimizer.load_state_dicts6335..z: 7#"7+ y>Q \ ((?(?@" HNC%5#$IcN(;; .s9S>/B.C?S\R]]ijmnst}n~jjAB)2#(<(<(> H$ 9E)$44$( ?9+_]!&i 0 ; i7%m]C,,!$Y%;%;%=!>J%()C)C)E%FN!^3(@@PP[\f[gglmvlwwxzCyDE-0!..0-2L2L2N-F(y ++-33I4D4DEF 5<<8%mU\\B,,$$&,,];080GIcN9-3 H" HJ&n5).9 % DEeHo.J>CM.z: ; D ) HIJ&x0 F !!$"9"9)"DE F>GM.z: ;  H %2$7$7$9 : Iy -%i0I9~Y/ 8Y8H T]S^^ijmnwjxiyyz{ :I%$,QC/? {Jjk=#+IaL#9IaL  : :$ ''7r9r8ct|ts td|d}t|tjr|g|d<nt ||d<|j jDcic]\}}|| }}}|dD]/}||vr td|jj||1|jj||jj|_ ycc}}w)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r2rz%some parameters are not in the moduleN)r3rr4rr5r!rr&r'r#r(r add_param_groupr)r)r8rr+r,r-s r/rpz_NamedOptimizer.add_param_groups +t, !=> >X& fell +%+HK !$(LK !595J5J5P5P5RSzsEs S S * @EL( !HII  # # * *<+> ? @ '' 4 OO88Ts- C+c|jjD]H}|jstj|}tj j ||_J|jdy)z Run a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrO) rr requires_gradr zeros_likeautogradVariablegradrL)r)r,ts r/ init_statez_NamedOptimizer.init_state-sa **113 8E""$$U+"^^44Q7  8 $ r9ct|jtr-tj|j|j|dS|S)NT)is_named_optimizer)r3rFSDPoptim_state_dict_to_loadr r)r<s r/rTz$_NamedOptimizer._pre_load_state_dict:s> dkk4 (00 T__jT r9ct|jtr+tj|j|j||SrI)r3rr{optim_state_dictr r}s r/r>z _NamedOptimizer._post_state_dictCs2 dkk4 (  ! !$++t Kr9)NN)rNrI)rGNrN)__name__ __module__ __qualname____doc__rstrrr5r r OptimizerrrnnModuletuplerr0rr<rrLrfloatpropertyr;r]rprxrTr>rJr9r/rrs)^>B#' )9!#u||m'C"CD)9)9!c!23d: )9  D )9 S#X )9sCx.)9 )9V/&WDcNW455 >HRY/>E>>5HRY/$65%$,5%wu||S01%%h8$sCx.h8Th8T9738+<992  tCH~$sCx.4S>d38nr9rrCrc6djt|S)zFConcatenate all param keys as a unique identifier for one param group.rR)joinr=)rCs r/r\r\Ks 88F:& ''r9)loggingr$collections.abcrrrcopyrtypingrrrtorch.nnrr 'torch.distributed._shard.sharded_tensorr torch.distributed.fsdpr r{r r!r__annotations__ getLoggerrloggerrrr\rJr9r/rsr99 ACc   8 $ueooup (T#Y(3(r9