B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Apr. 2017, pp. 1273–1282.
communication-efficient的含义并不是相较于传输整体数据或整个网络结构，只传输参数更新会降低通信开销。而是和同步的SGD（仅用所有本地数据训练一次就进行参数合并，是当时的基于数据中心训练方法的SOTA）相比，在更少的通信次数下就能达到目标准确率（减少10到100倍的通信次数）。
our goal is to use additional computation in order to decrease the number of rounds of communication needed to train a model
对于不传输本地数据这一点，作者强调的是隐私保护，而不是节省通信开销。
每次更新只针对当前模型，因此不建议利用连续两次更新的相关性
Since these updates are specific to improving the current model, there is no reason to store them once they have been applied.
每一轮的用户参与并不是越多越好，需要考虑一个性能和通信的折衷
We only select a fraction of clients for efficiency, as our experiments show diminishing returns for adding more clients beyond a certain point.
FedAvg的有很强的鲁棒性，作者推测是因为带来了类似于dropout的正则化作用
averaging provides any advantage (vs. actually diverging) when we naively average the parameters of models trained on entirely different pairs of digits. Thus, we view this as strong evidence for the robustness of this approach
We conjecture that in addition to lowering communication costs, model averaging produces a regularization benefit similar to that achieved by dropout [36]
batch size只要和硬件相匹配，减少它就不会显著增加计算时间
As long as B is large enough to take full advantage of available parallelism on the client hardware, there is essentially no cost in computation time for lowering it, and so in practice this should be the first parameter tuned.
文章的训练对象是mobile devices，因此和通信结合是自然而然的
the identification of the problem of training on decentralized data from mobile devices as an important research direction
异构数据
数据初始分布不同有何影响（每个用户的损失函数都不同）
\(F_k\) could be an arbitrarily bad approximation to \(f\)
\[f(w)=\sum_{k=1}^{K} \frac{n_{k}}{n} F_{k}(w) \quad \text { where } \quad F_{k}(w)=\frac{1}{n_{k}} \sum_{i \in \mathcal{P}_{k}} f_{i}(w)\]
训练中数据的增删有何影响
数据的上线时段不同有何影响
在不平衡的数据分布下，小数据集的过拟合程度很大，也没有影响吗？
网络参数传输
部分网络传输
one-shot averaging（多半是正则的）训练完后直接合并
本地训练的过拟合程度和发散究竟有何关系？
This result suggests that for some models, especially in the later stages of convergence, it may be useful to decay the amount of local computation per round (moving to smaller E or larger B) in the same way decaying learning rates can be useful.
新意100×有效1000×研究问题100
当两个模型采用同一套参数初始值时，过拟合训练后直接参数平均就能提高模型性能！所以和分布式SGD的每本地训练一次就上传相比，大大减少了通信的次数。
这个发现是在IID的情况下做的，仿真下发现在non IID下也有显著提升。但是没有IID下提升那么明显，可能是个可以挖的坑。
Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly well-behaved and in particular less prone to bad local minima than previously thought [11, 17, 9].
we find that naive parameter averaging works surprisingly well
the average of these two models, \(\frac{1}{2}w+ \frac{1}{2}w^\prime\), achieves significantly lower loss on the full MNIST training set than the best model achieved by training on either of the small datasets independently.
