Optimizers

Optimizers#

The xlnstorch.optim module provides LNS-compatible optimization algorithms for training neural networks with LNSTensor parameters. These optimizers are designed to be analogous to the PyTorch optimizers. Currently, the optimizers are missing support for foreach and fused operations, but they are fully functional for standard LNS training tasks.

Overview#

All optimizers in xlnstorch follow the same interface as PyTorch optimizers but are specifically designed to handle LNSTensor parameters. They support:

  • LNS and floating point learning rates and hyperparameters

  • Automatic gradient handling for LNS arithmetic

  • Parameter groups from LNS layers using model.lns_parameters()

  • Standard optimizer features like momentum, weight decay, and adaptive learning rates

Basic Usage#

Here’s a basic example of using an LNS optimizer:

import torch
import xlnstorch as xltorch

# Create model and data
model = xltorch.nn.LNSLinear(3, 3, bias=True)
input = xltorch.randn(3, requires_grad=True)
target = xltorch.lnstensor([1.0, 1.0, 1.0])

# Initialize optimizer with model parameters
optimizer = xltorch.optim.LNSSGD(model.lns_parameters(), lr=0.1)
loss_fn = torch.nn.MSELoss(reduction='mean')

# Training loop
for i in range(20):
optimizer.zero_grad()

output = model(input)
loss = loss_fn(output, target)

loss.backward()
optimizer.step()

Parameter Groups#

LNS optimizers work with parameter groups obtained from LNS layers using the lns_parameters() method. This method returns an iterable of parameter dictionaries that contain both weights and biases in LNS format.

# Get parameter groups from a model
model = xltorch.nn.LNSLinear(10, 5, bias=True)
param_groups = model.lns_parameters()

# Initialize optimizer with parameter groups
optimizer = xltorch.optim.LNSAdam(param_groups, lr=0.001)

Learning Rate and Hyperparameters#

All LNS optimizers accept both regular Python floats and LNSTensor objects for learning rates and other hyperparameters. Using LNSTensor hyperparameters allows for control over the base of the LNSTensor since other hyperparameters will be converted to LNSTensors with the default base.

# Using float learning rate
optimizer1 = xltorch.optim.LNSAdam(params, lr=0.001)

# Using LNSTensor learning rate
lr_tensor = xltorch.lnstensor(0.001)
optimizer2 = xltorch.optim.LNSAdam(params, lr=lr_tensor)

Available Optimizers#

xlnstorch provides LNS-compatible versions of popular PyTorch optimizers:

Quick Reference#

  • LNSSGD - Stochastic Gradient Descent with momentum support

  • LNSAdam - Adam optimizer with bias correction

  • LNSAdamW - Adam optimizer with decoupled weight decay

  • LNSAdamax - Adamax optimizer (infinity norm variant of Adam)

  • LNSNAdam - Nesterov-accelerated Adam optimizer

  • LNSRAdam - Rectified Adam optimizer

  • LNSAdagrad - Adaptive gradient algorithm

  • LNSAdadelta - Adadelta optimizer

  • LNSRMSprop - RMSprop optimizer

  • LNSRprop - Resilient backpropagation algorithm

  • LNSASGD - Averaged Stochastic Gradient Descent

  • LNSSignMul - Sign-based multiplication optimizer (experimental)

  • LNSMul - Simple multiplication optimizer (experimental)

  • LNSMadam - Multiplicative Adam optimizer (experimental)

  • LNSHybridMul - Hybrid multiplication optimizer (experimental)

Stochastic Gradient Descent (SGD)#

class xlnstorch.optim.LNSSGD(params, lr=0.001, momentum=0.0, dampening=0.0, weight_decay=0.0, nesterov=False, *, maximize=False)#

Bases: LNSOptimizer

Implements stochastic gradient descent (SGD) with support for momentum, dampening, weight decay, and nesterov momentum.

This optimizer is analogous to PyTorch’s torch.optim.SGD, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the SGD algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.001). Must be a non-negative LNSTensor or float.

  • momentum (LNSTensor, float, optional) – Momentum factor (default: 0.0). Must be a non-negative LNSTensor or float.

  • dampening (LNSTensor, float, optional) – Dampening for momentum (default: 0.0). Must be a non-negative LNSTensor or float.

  • weight_decay (LNSTensor, float, optional) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • nesterov (bool, optional) – Enables Nesterov momentum if set to True (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

Examples

>>> optimizer = xlnstorch.optim.LNSSGD(model.lns_parameters(), lr=0.1, momentum=0.9)
>>> optimizer.zero_grad() # Clear gradients before the step
>>> loss_fn(model(input), target).backward() # Compute gradients
>>> optimizer.step() # Update parameters based on gradients
step(closure=None)#

Performs a single optimization step.

Adam Optimizers#

class xlnstorch.optim.LNSAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, amsgrad=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the Adam optimization algorithm for LNSTensor parameters, including optional weight–decay regularisation, the AMSGrad variant, and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.Adam, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the Adam algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.001). Must be a non-negative LNSTensor or float.

  • betas (Tuple[float, float] or Tuple[LNSTensor, LNSTensor], optional) – Coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)). Must be two non-negative LNSTensor or float values in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • weight_decay (LNSTensor or float) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • amsgrad (bool, optional) – Uses the AMSGrad variant that maintains the maximum of past squared gradients (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSAdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the AdamW optimization algorithm for LNSTensor parameters, including optional weight–decay regularisation, the AMSGrad variant, and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.AdamW, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the AdamW algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.001). Must be a non-negative LNSTensor or float.

  • betas (Tuple[float, float] or Tuple[LNSTensor, LNSTensor], optional) – Coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)). Must be two non-negative LNSTensor or float values in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • weight_decay (LNSTensor or float) – Weight decay (L2 penalty) (default: 0.01). Must be a non-negative LNSTensor or float.

  • amsgrad (bool, optional) – Uses the AMSGrad variant that maintains the maximum of past squared gradients (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSAdamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, *, maximize=False)#

Bases: LNSOptimizer

Implements the Adamax optimization algorithm for LNSTensor parameters, including optional weight–decay regularisation, and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.Adamax, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the Adamax algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.002). Must be a non-negative LNSTensor or float.

  • betas (Tuple[float, float] or Tuple[LNSTensor, LNSTensor], optional) – Coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)). Must be two non-negative LNSTensor or float values in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • weight_decay (LNSTensor or float) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSNAdam(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, momentum_decay=0.004, decoupled_weight_decay=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the Adam optimization algorithm for LNSTensor parameters, including decoupled weight decay, momentum decay, and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.NAdam, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the NAdam algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.002). Must be a non-negative LNSTensor or float.

  • betas (Tuple[float, float] or Tuple[LNSTensor, LNSTensor], optional) – Coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)). Must be two non-negative LNSTensor or float values in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • weight_decay (LNSTensor or float) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • momentum_decay (LNSTensor or float, optional) – Decay factor for the momentum term (default: 0.004). Must be a non-negative LNSTensor or float.

  • decoupled_weight_decay (bool, optional) – If True, applies decoupled weight decay (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSRAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, decoupled_weight_decay=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the rectified Adam optimization algorithm for LNSTensor parameters, including decoupled weight decay, and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.RAdam, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the RAdam algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.002). Must be a non-negative LNSTensor or float.

  • betas (Tuple[float, float] or Tuple[LNSTensor, LNSTensor], optional) – Coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)). Must be two non-negative LNSTensor or float values in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • weight_decay (LNSTensor or float) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • decoupled_weight_decay (bool, optional) – If True, applies decoupled weight decay (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

Adaptive Learning Rate Optimizers#

class xlnstorch.optim.LNSAdagrad(params, lr=0.01, lr_decay=0.0, weight_decay=0.0, initial_accumulator_value=0, eps=1e-10, *, maximize=False)#

Bases: LNSOptimizer

Implements the Adagrad algorithm with support for learning rate decay, weight decay, and an initial accumulator value.

This optimizer is analogous to PyTorch’s torch.optim.Adagrad, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the Adagrad algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • lr_decay (LNSTensor, float, optional) – Learning rate decay factor (default: 0.0). Must be a non-negative LNSTensor or float.

  • weight_decay (LNSTensor, float, optional) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • initial_accumulator_value (LNSTensor, float, optional) – Initial value for the accumulator (default: 0). Must be a non-negative LNSTensor or float.

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-10).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSAdadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0.0, *, maximize=False)#

Bases: LNSOptimizer

Implements the LNSAdadelta algorithm for LNSTensor parameters, supporting weight decay and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.Adamax, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the Adamax algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.002). Must be a non-negative LNSTensor or float.

  • rho (LNSTensor, float, optional) – Coefficient used for computing running averages of gradient (default: 0.9). Must be a non-negative LNSTensor or float in the range (0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-6). Must be a non-negative LNSTensor or float.

  • weight_decay (LNSTensor or float, optional) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSRMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0.0, momentum=0.0, centered=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the RMSprop algorithm with support for weight decay, momentum, and centered variants.

This optimizer is analogous to PyTorch’s torch.optim.RMSprop, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the RMSprop algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • alpha (LNSTensor, float, optional) – Smoothing constant (default: 0.99). Must be a non-negative LNSTensor or float in the range [0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator to improve numerical stability (default: 1e-08). Must be a non-negative LNSTensor or float.

  • weight_decay (LNSTensor, float, optional) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • momentum (LNSTensor, float, optional) – Momentum factor (default: 0.0). Must be a non-negative LNSTensor or float.

  • centered (bool, optional) – If True, computes the centered RMSprop variant (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

Other Optimizers#

class xlnstorch.optim.LNSRprop(params, lr=0.01, etas=(0.5, 1.2), step_sizes=(1e-06, 50.0), *, maximize=False)#

Bases: LNSOptimizer

Implements the Rprop (resilient backpropagation) algorithm.

This optimizer is analogous to PyTorch’s torch.optim.Rprop, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the Rprop algorithm.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • etas (tuple of LNSTensor, tuple of float, optional) – Tuple of two factors (η₋, η₊) for decreasing and increasing the step size (default: (0.5, 1.2)). Must satisfy 0 < η₋ < 1 and η₊ > 1.

  • step_sizes (tuple of LNSTensor, tuple of float, optional) – Tuple of two step sizes (Γ_min, Γ_max) for clamping the step size (default: (1e-6, 50.0)). Must satisfy 0 < Γ_min < Γ_max.

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSASGD(params, lr=0.01, lambd=0.0001, alpha=0.75, t0=1000000.0, weight_decay=0.0, *, maximize=False)#

Bases: LNSOptimizer

Implements the ASGD algorithm for LNSTensor parameters, including optional weight decay and a “maximize” mode.

This optimizer is analogous to PyTorch’s torch.optim.ASGD, but is designed to work with LNSTensor objects. See the PyTorch documentation for more details on the ASGD algorithm.

Note that this optimizer doesn’t seem to implement the ASGD algorithm correctly, but it is made to match the PyTorch implementation as closely as possible.

Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.001). Must be a non-negative LNSTensor or float.

  • lambd (LNSTensor, float, optional) – Coefficient for the learning rate decay (default: 0.0001). Must be a non-negative LNSTensor or float.

  • alpha (LNSTensor, float, optional) – Exponent for the learning rate decay (default: 0.75). Must be a non-negative LNSTensor or float in the range (0.0, 1.0].

  • t0 (LNSTensor, float, optional) – The point at which the learning rate decay starts (default: 1000000.0). Must be a non-negative LNSTensor or float.

  • weight_decay (LNSTensor or float, optional) – Weight decay (L2 penalty) (default: 0.0). Must be a non-negative LNSTensor or float.

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

LNS Experimental Optimizers#

Note

Note that for all multiplicative optimizers below, weights should be initialized to non-zero values as multiplicative updates cannot change a zero weight.

class xlnstorch.optim.LNSSignMul(params, lr=0.01, p_scale=10.0, use_pow=False, *, maximize=False)#

Bases: LNSOptimizer

Implements a simple sign multiplication algorithm for LNSTensor parameters.

\[\begin{split}\begin{aligned} &\rule{120mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\; \theta_{0} \text{ (params)},\; f(\theta) \text{ (objective)},\; \textit{use_pow},\; \\ &\hspace{17mm} \textit{maximize},\; \sigma \text{ (weight scale)} \\ &\textbf{initialize} : \\ &\hspace{5mm}\alpha \; = \; \begin{cases} 2^{\gamma}, & \text{if } \textit{use_pow} \\ 1 + \gamma, & \text{otherwise} \end{cases} \quad\text{(primary multiplier)}, \\ &\hspace{5mm} \alpha^{-1} \; = \; 1 / \alpha \quad\text{(inverse multiplier)}, \\ &\hspace{5mm} \sigma^{*} \leftarrow \sigma \cdot \operatorname{RMS}\left(\theta_{0}\right) \text{ (max weight)} \\[-1.ex] &\rule{120mm}{0.4pt} \\ &\textbf{for } t = 1 \textbf{ to } \ldots \textbf{ do} \\ &\hspace{5mm}\textbf{if } \textit{maximize}: \\ &\hspace{10mm} g_t \leftarrow -\nabla_{\theta} f_t \left(\theta_{t-1}\right) \\ &\hspace{5mm}\textbf{else}: \\ &\hspace{10mm} g_t \leftarrow \nabla_{\theta} f_t \left(\theta_{t-1}\right) \\ &\hspace{5mm} u_t \leftarrow \begin{cases} 1, &g_t = 0 \\ \alpha^{-1}, &\operatorname{sign} \bigl(\theta_{t-1}\bigr) = \operatorname{sign} \left(g_t\right) \\ \alpha, &\text{otherwise} \end{cases} \\ &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} \cdot u_t \\ &\hspace{5mm} \theta_t \leftarrow \operatorname{clamp_{\sigma^{*}}} \left(\theta_t\right) \\[-1.ex] &\rule{120mm}{0.4pt} \\[-1.ex] &\textbf{return } \theta_t \\[-1.ex] &\rule{120mm}{0.4pt} \\ \end{aligned}\end{split}\]
Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • use_pow (bool, optional) – If True, uses a power-based multiplier (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSMul(params, lr=0.01, use_pow=False, *, maximize=False)#

Bases: LNSOptimizer

Implements a simple multiplication algorithm for LNSTensor parameters.

\[\begin{split}\begin{aligned} &\rule{120mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\; \theta_{0} \text{ (params)},\; f(\theta) \text{ (objective)},\; \textit{use_pow},\; \\ &\hspace{17mm} \textit{maximize} \\[-1.ex] &\rule{120mm}{0.4pt} \\ &\textbf{for } t = 1 \textbf{ to } \ldots \textbf{ do} \\ &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t \left(\theta_{t-1}\right) \\ &\hspace{5mm} \textbf{if } \textit{use_pow}: \\ &\hspace{10mm} u_t \leftarrow 2 ^ {-\gamma g_t \operatorname{sign} \bigl(\theta_{t-1}\bigr)} \\ &\hspace{10mm} \textbf{if } \textit{maximize}: \\ &\hspace{15mm} u_t \leftarrow 1 / u_t \\ &\hspace{5mm} \textbf{else}: \\ &\hspace{10mm} u_t \leftarrow 1 + \gamma \lvert g_t \rvert \\ &\hspace{10mm} \textbf{if } \left(\operatorname{sign} \bigl(\theta_{t-1}\bigr) \neq \operatorname{sign} \left(g_t\right)\right) \oplus \textit{maximize}: \\ &\hspace{15mm} u_t \leftarrow 1 / u_t \\ &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} \cdot u_t \\[-1.ex] &\rule{120mm}{0.4pt} \\[-1.ex] &\textbf{return } \theta_t \\[-1.ex] &\rule{120mm}{0.4pt} \\ \end{aligned}\end{split}\]
Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • use_pow (bool, optional) – If True, uses a power-based multiplier (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSMadam(params, lr=0.01, beta=0.999, eps=1e-08, p_scale=3.0, g_bound=10.0, use_pow=False, *, maximize=False)#

Bases: LNSOptimizer

Implements the Madam optimizer for LNSTensor parameters. See

for more details on the algorithm.

\[\begin{split}\begin{aligned} &\rule{130mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\; \beta \text{ (beta)},\; \epsilon \text{ (epsilon)},\; \mu \text{ (max perturbation)},\; \\ &\hspace{17mm} \sigma \text{ (weight scale)},\; \theta_{0} \text{ (params)},\; f(\theta) \text{ (objective)},\; \\ &\hspace{17mm} \textit{use_pow},\; \textit{maximize} \\ &\textbf{initialize} : \sigma^{*} \leftarrow \sigma \cdot \operatorname{RMS}\left(\theta_{0}\right) \text{ (max weight)},\; \\ &\hspace{26mm} v_0 \leftarrow 0 \text{ (second moment)} \\[-1.ex] &\rule{130mm}{0.4pt} \\ &\textbf{for } t = 1 \textbf{ to } \ldots \textbf{ do} \\ &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t \left(\theta_{t-1}\right) \\ &\hspace{5mm} \rho_t \leftarrow 1 - \beta^{t} \\ &\hspace{5mm} v_t \leftarrow (1 - \beta) g_t^2 + \beta v_{t-1} \\ &\hspace{5mm} g^{*}_t \leftarrow g_t / \sqrt{v_t / \rho_t + \epsilon} \\ &\hspace{5mm} \chi_t \leftarrow -\gamma \operatorname{sign} \bigl(\theta_{t-1}\bigr) \operatorname{clamp_{\mu}} \bigl(g^{*}_t\bigr) \\ &\hspace{5mm} \textbf{if } \textit{maximize}: \\ &\hspace{10mm} \chi_t \leftarrow -\chi_t \\ &\hspace{5mm} \textbf{if } \textit{use_pow}: \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} \cdot \exp \left(\chi_t\right) \\ &\hspace{5mm} \textbf{else}: \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} \cdot \left(1 + \chi_t\right) \\ &\hspace{5mm} \theta_t \leftarrow \operatorname{clamp_{\sigma^{*}}} \left(\theta_t\right) \\[-1.ex] &\rule{130mm}{0.4pt} \\[-1.ex] &\textbf{return } \theta_t \\[-1.ex] &\rule{130mm}{0.4pt} \\ \end{aligned}\end{split}\]
Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

  • beta (LNSTensor, float, optional) – Coefficient used for computing the running average of the gradient (default: 0.999). Must be a non-negative LNSTensor or float in the range (0.0, 1.0).

  • eps (LNSTensor, float, optional) – Term added to the denominator for numerical stability (default: 1e-8).

  • p_scale (LNSTensor, float, optional) – Scaling factor for the parameter update (default: 3.0). Must be a non-negative LNSTensor or float.

  • g_bound (LNSTensor, float, optional) – Bound for the gradient norm (default: 10.0). Must be a non-negative LNSTensor or float.

  • use_pow (bool, optional) – If True, uses a power-based multiplier (default: False).

  • maximize (bool, optional) – If True, optimizes the parameters for maximization instead of minimization (default: False).

step(closure=None)#

Performs a single optimization step.

class xlnstorch.optim.LNSHybridMul(params, lr=0.01)#

Bases: LNSOptimizer

Implements a hybrid multiplication algorithm for LNSTensor parameters. This optimizer uses a heuristic to decide between using a standard multiplicative update, a sign-based update or a gradient descent-like update.

\[\begin{split}\begin{aligned} &\rule{120mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\; \theta_{0} \text{ (params)},\; f(\theta) \text{ (objective)} \\[-1.ex] &\rule{120mm}{0.4pt} \\ &\textbf{for } t = 1 \textbf{ to } \ldots \textbf{ do} \\ &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t \left(\theta_{t-1}\right) \\ &\hspace{5mm} \textbf{if } \operatorname{sign} \bigl(\theta_{t-1}\bigr) = \operatorname{sign} \left(g_t\right) \lor \lvert g_t \rvert < \gamma \lor \lvert \theta_{t-1} \rvert < \gamma: \\ &\hspace{10mm} u_t \leftarrow 1 / \left(1 + \gamma \lvert g_t \rvert \right) \\ &\hspace{5mm} \textbf{else}: \\ &\hspace{10mm} u_t \leftarrow \max \left( 2 ^ {\gamma}, 1 + \gamma \cdot \lvert g_t / \theta_{t-1} \rvert \right) \\ &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} \cdot u_t \\[-1.ex] &\rule{120mm}{0.4pt} \\[-1.ex] &\textbf{return } \theta_t \\[-1.ex] &\rule{120mm}{0.4pt} \\ \end{aligned}\end{split}\]
Parameters:

  • params (iterable) – An iterable of parameters to optimize or dicts defining parameter groups. This should be obtained from a model’s lns_parameters() method.

  • lr (LNSTensor, float, optional) – Learning rate (default: 0.01). Must be a non-negative LNSTensor or float.

step(closure=None)#

Performs a single optimization step.

Contents