Learning Rate Schedulers

LNS optimizers can be used in conjunction with learning rate schedulers from the xlnstorch.optim.lr_scheduler module. These schedulers allow for dynamic adjustment of the learning rate during training, which can help improve convergence and performance.

LNS learning rate schedulers work in the same was as PyTorch schedulers but are designed to handle LNSTensor learning rates. They can be applied to any LNS optimizer and support both float and LNSTensor learning rates. For example:

import xlnstorch as xltorch

func = lambda epoch: 1 / (epoch + 1)
scheduler = xltorch.optim.lr_scheduler.LNSLambdaLR(optimizer, func)

for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

xlnstorch.optim.lr_scheduler.get_lr_bases(optimizer)

Returns a list of base values for the learning rates of each parameter group in the optimizer.

Parameters:

optimizer (LNSOptimizer) – The optimizer from which to extract the base learning rates.

Returns:

A list of base learning rates for each parameter group.

Return type:

List[torch.Tensor]

Raises:

TypeError – If the provided optimizer is not an instance of LNSOptimizer.

LNSLambdaLR	An LNS learning rate scheduler that sets the learning rate of each parameter group to the initial learning rate multipled by a given function of the epoch.
LNSMultiplicativeLR	An LNS learning rate scheduler that sets the learning rate of each parameter group to the previous learning rate multipled by a given multiplicative factor.
LNSStepLR	An LNS learning rate scheduler that decays the learning rate of each parameter group by a factor of gamma every step_size epochs.
LNSMultiStepLR	An LNS learning rate scheduler that decays the learning rate of each parameter group by a factor of gamma at specified epochs.
LNSConstantLR	An LNS learning rate scheduler that sets the learning rate of each parameter group to a constant value until a pre-determined number of epochs is reached.
LNSLinearLR	An LNS learning rate scheduler that sets the learning rate of each parameter group to a linearly decaying value.
LNSExponentialLR	An LNS learning rate scheduler that decays the learning rate of each parameter group by gamma each epoch.
LNSPolynomialLR	An LNS learning rate scheduler that decays the learning rate of each parameter group by a polynomial factor in the given total_iters.
LNSReduceLROnPlateau	An LNS learning rate scheduler that reduces the learning rate of each parameter group when a metric has stopped improving.
LNSChainedScheduler	A scheduler that chains multiple schedulers together.
LNSSequentialLR	A scheduler that applies a sequence of schedulers in order.