博主根据自身多年的深度学习算法研发经验，整理分享以下十条必知。
含参考资料链接，部分附上相关代码实现。
独乐乐不如众乐乐，希望对各位看客有所帮助。
待回头有时间再展开细节说一说深度学习里的那些道道。
有什么技术需求需要有偿解决的也可以邮件或者QQ联系博主。
邮箱QQ同ID：gaozhihan@vip.qq.com
当然除了这十条，肯定还有其他“必知”，
欢迎评论分享更多，这里只是暂时拟定的十条，别较真哈。
主要学习其中的思路，切记，以下思路在个别场景并不适用 。
1.数据回流
[1907.05550] Faster Neural Network Training with Data Echoing
def data_echoing(factor): return lambda image, label: tf.data.Dataset.from_tensors((image, label)).repeat(factor)
作用:
数据集加载后，在数据增广前后重复当前批次进模型的次数，减少数据的加载耗时。
等价于让模型看n次当前的数据，或者看n个增广后的数据样本。
2.AMP 自动精度混合
在bert4keras中使用混合精度和XLA加速训练 - 科学空间|Scientific Spaces
tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})作用:
降低显存占用，加速训练，将部分网络计算转为等价的低精度计算，以此降低计算量。
3.优化器节省显存
3.1 [1804.04235]Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
mesh/optimize.py at master · tensorflow/mesh · GitHub
3.2 [1901.11150] Memory-Efficient Adaptive Optimization
google-research/sm3 at master · google-research/google-research (github.com)
作用:
节省显存，加速训练，
主要是对二阶动量进行特例化解构，减少显存存储。
4.权重标准化(归一化)
[2102.06171] High-Performance Large-Scale Image Recognition Without Normalization
deepmind-research/nfnets at master · deepmind/deepmind-research · GitHub
class WSConv2D(tf.keras.layers.Conv2D): def __init__(self, *args, **kwargs): super(WSConv2D, self).__init__( kernel_initializer=tf.keras.initializers.VarianceScaling( scale=1.0, mode='fan_in', distribution='untruncated_normal', ), use_bias=False, kernel_regularizer=tf.keras.regularizers.l2(1e-4), *args, **kwargs ) self.gain = self.add_weight( name='gain', shape=(self.filters,), initializer="ones", trainable=True, dtype=self.dtype ) def standardize_weight(self, eps): mean, var = tf.nn.moments(self.kernel, axes=[0, 1, 2], keepdims=True) fan_in = np.prod(self.kernel.shape[:-1]) # Manually fused normalization, eq. to (w - mean) * gain / sqrt(N * var) scale = tf.math.rsqrt( tf.math.maximum( var * fan_in, tf.convert_to_tensor(eps, dtype=self.dtype) ) ) * self.gain shift = mean * scale return self.kernel * scale - shift def call(self, inputs): eps = 1e-4 weight = self.standardize_weight(eps) return tf.nn.conv2d( inputs, weight, strides=self.strides, padding=self.padding.upper(), dilations=self.dilation_rate ) if self.bias is None else tf.nn.bias_add( tf.nn.conv2d( inputs, weight, strides=self.strides, padding=self.padding.upper(), dilations=self.dilation_rate ), self.bias)
作用:
通过对kernel进行标准化或归一化，相当于对kernel做一个先验约束，以此加速模型训练收敛。
5.自适应梯度裁剪
deepmind-research/agc_optax.py at master · deepmind/deepmind-research · GitHub
def unitwise_norm(x): if len(tf.squeeze(x).shape) <= 1: # Scalars and vectors axis = None keepdims = False elif len(x.shape) in [2, 3]: # Linear layers of shape IO axis = 0 keepdims = True elif len(x.shape) == 4: # Conv kernels of shape HWIO axis = [0, 1, 2, ] keepdims = True else: raise ValueError(f'Got a parameter with shape not in [1, 2, 3, 4]! {x}') square_sum = tf.reduce_sum(tf.square(x), axis, keepdims=keepdims) return tf.sqrt(square_sum)def gradient_clipping(grad, var): clipping = 0.01 max_norm = tf.maximum(unitwise_norm(var), 1e-3) * clipping grad_norm = unitwise_norm(grad) trigger = (grad_norm > max_norm) clipped_grad = (max_norm / tf.maximum(grad_norm, 1e-6)) return grad * tf.where(trigger, clipped_grad, tf.ones_like(clipped_grad))作用:
防止梯度爆炸，稳定训练。通过梯度和参数的关系，对梯度进行裁剪，约束学习率。
6.recompute_grad
[1604.06174] Training Deep Nets with Sublinear Memory Cost
google-research/recompute_grad.py at master · google-research/google-research (github.com)
bojone/keras_recompute: saving memory by recomputing for keras (github.com)
作用:
通过梯度重计算，节省显存。
7.归一化
[2003.05569] Extended Batch Normalization (arxiv.org)
from keras.layers.normalization.batch_normalization import BatchNormalizationBaseclass ExtendedBatchNormalization(BatchNormalizationBase): def __init__(self, axis=-1, momentum=0.99, epsilon=1e-3, center=True, scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones', beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None, renorm=False, renorm_clipping=None, renorm_momentum=0.99, trainable=True, name=None, **kwargs): # Currently we only support aggregating over the global batch size. super(ExtendedBatchNormalization, self).__init__( axis=axis, momentum=momentum, epsilon=epsilon, center=center, scale=scale, beta_initializer=beta_initializer, gamma_initializer=gamma_initializer, moving_mean_initializer=moving_mean_initializer, moving_variance_initializer=moving_variance_initializer, beta_regularizer=beta_regularizer, gamma_regularizer=gamma_regularizer, beta_constraint=beta_constraint, gamma_constraint=gamma_constraint, renorm=renorm, renorm_clipping=renorm_clipping, renorm_momentum=renorm_momentum, fused=False, trainable=trainable, virtual_batch_size=None, name=name, **kwargs) def _calculate_mean_and_var(self, x, axes, keep_dims): with tf.keras.backend.name_scope('moments'): y = tf.cast(x, tf.float32) if x.dtype == tf.float16 else x replica_ctx = tf.distribute.get_replica_context() if replica_ctx: local_sum = tf.math.reduce_sum(y, axis=axes, keepdims=True) local_squared_sum = tf.math.reduce_sum(tf.math.square(y), axis=axes, keepdims=True) batch_size = tf.cast(tf.shape(y)[0], tf.float32) y_sum = replica_ctx.all_reduce(tf.distribute.ReduceOp.SUM, local_sum) y_squared_sum = replica_ctx.all_reduce(tf.distribute.ReduceOp.SUM, local_squared_sum) global_batch_size = replica_ctx.all_reduce(tf.distribute.ReduceOp.SUM, batch_size) axes_vals = [(tf.shape(y))[i] for i in range(1, len(axes))] multiplier = tf.cast(tf.reduce_prod(axes_vals), tf.float32) multiplier = multiplier * global_batch_size mean = y_sum / multiplier y_squared_mean = y_squared_sum / multiplier # var = E(x^2) - E(x)^2 variance = y_squared_mean - tf.math.square(mean) else: # Compute true mean while keeping the dims for proper broadcasting. mean = tf.math.reduce_mean(y, axes, keepdims=True, name='mean') variance = tf.math.reduce_mean( tf.math.squared_difference(y, tf.stop_gradient(mean)), axes, keepdims=True, name='variance') if not keep_dims: mean = tf.squeeze(mean, axes) variance = tf.squeeze(variance, axes) variance = tf.math.reduce_mean(variance) if x.dtype == tf.float16: return (tf.cast(mean, tf.float16), tf.cast(variance, tf.float16)) else: return mean, variance
作用:
一个简易改进版的Batch Normalization，思路简单有效。
8.学习率策略
[1506.01186] Cyclical Learning Rates for Training Neural Networks (arxiv.org)
作用:
一个推荐的学习率策略方案，特定情况下可以取得更好的泛化。
9.重参数化
[1908.03930] ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
https://zhuanlan.zhihu.com/p/361090497
作用：
通过同时训练多份参数，合并权重的思路来提升模型泛化性。
10.长尾学习
[2110.04596] Deep Long-Tailed Learning: A Survey (arxiv.org)
Jorwnpay/A-Long-Tailed-Survey: 本项目是 Deep Long-Tailed Learning: A Survey 文章的中译版 (github.com)
作用:
解决长尾问题，可以加速收敛，提升模型泛化，稳定训练。