A. Collaborative Dictionary Learning Using Cloud K-SVD

Consider a collection of N geographically-distributed sites that are interconnected to each other according to a fixed topology. Mathematically, we represent this collection and their interconnections through an undirected graph \mathcal{G}=(\mathcal{N},\ \mathcal{E}), where \mathcal{N}=\{1, 2,\ \cdots,\ N\} denotes the sites and \mathcal{E} denotes edges in the graph with (i,\ i)\in \mathcal{E} and (i, j)\in \mathcal{E} whenever there is a connection between site i and site j. The only assumption we make about the topology of \mathcal{G} is that it is a connected graph. Next, we assume that each site i has a massive collection of local data, expressed as a matrix Y_{i}\in \mathbb{R}^{n\times S_{\tau}} with S_{i}\gg n representing the number of data samples at the i^{th} site. We can express this distributed data as a single matrix Y=[Y_{1}\ Y_{2}... \ Y_{N}]\in \mathbb{R}^{n\times S}, where S=\sum_{i=1}^{N}S_{i} denotes the total number of data samples distributed across the N sites. In this setting, the fundamental objective is for each site to collaboratively learn a dictionary that underlies the global (distributed) data Y.

Assuming that global data Y is available at a centralized location, the dictionary learning problem can be expressed as \begin{equation*} (D,\displaystyle \ X)=\arg\min_{D, X}\Vert Y-DX \Vert_{F}^{2} \mathrm{s.t.} \forall s, \Vert x_{s}\Vert_{0}\leq T_{0}, \tag{1} \end{equation*}View Source\begin{equation*} (D,\displaystyle \ X)=\arg\min_{D, X}\Vert Y-DX \Vert_{F}^{2} \mathrm{s.t.} \forall s, \Vert x_{s}\Vert_{0}\leq T_{0}, \tag{1} \end{equation*} where D\in \mathbb{R}^{n\times K} with K > n is an overcomplete dictionary having unit \ell_{2}-norm columns, X\in \mathbb{R}^{K\times S} corresponds to representation coefficients of the data having no more than T_{0}\ll n nonzero coefficients per sample, and x_{s}, s=1, \ldots, S denotes the s^{th} column in X. Unlike classical dictionary learning, however, we do not have the global data Y available at a centralized location. Therefore, our goal is to have individual sites collaboratively learn dictionaries \{\hat{D}_{i}\}_{i\in N} from global data Y such that these collaborative dictionaries are close to a dictionary D that could have been learned from Y in a centralized fashion.

Cloud K-SVD, given as Algorithm 1, was developed in [5] to accomplish the goal of collaborative dictionary learning from distributed data. This algorithm is a distributed variant of the famous K-SVD algorithm, which consists of two main steps, sparse coding and dictionary update. The sparse coding step in cloud K-SVD is performed only for locally available data at each site, while the dictionary update step is performed in a distributed manner using distributed power method. This makes the dictionary update step in cloud K-SVD as the most important and challenging step, Specifically, we recall from the description of K-SVD in [2] that dictionary update involves singular value decomposition (SVD) of the error matrix E_{k}^{(t)}=[\overline{E}_{1, k}^{(t)} \ldots\ E_{N_{)}k}^{(t)}] in iteration t, which is available to K-SVD at one location. Cloud K-SVD, on the other hand, can only compute E^{(t)}_{i, k} at any site i due to local sparse coding (cf. Step 5 of Algorithm 1), where \hat{E}_{i, k}^{(t)} denotes a perturbed version of the centralized E_{i, k}^{(t)}. Next, define an ordered set \overline{\omega}_{i, k}^{(t)}=\{s:1\leq s\leq S,\overline{x}_{i, k, T}^{(t)}(s)\neq 0\}, where \overline{x}_{i_{\backslash }k, T}^{(t)}(s) denotes the s^{th} element of \overline{x}_{i, k,\mathcal{I}}^{(t)}, and an S\times\vert \overline{\omega}_{i, k}^{(t)} binary matrix \overline{\Omega}_{i, k}^{(t)} that has ones at (\overline{\omega}_{i, k}^{(t)}(s),\ s) locations and zeros everywhere else. Then each site i in cloud K-SVD only has access to \hat{E}_{i, k, R}^{(t)}=\hat{E}_{i, k}^{(t)}\overline{\Omega}_{i, k}^{(t)} and \overline{x}_{i, k, R}^{(t)}=\overline{x}_{i, k, T}^{(t)}\overline{\Omega}_{i, k}^{(t)}, whereas the centralized K-SVD has access to E_{i, k, R}^{(t)}=B_{i, k}^{(t)}\Omega_{i, k}^{(t)} and x_{i.k.R}^{(t)}=x_{i.kT}^{(t)}\Omega_{i.i}^{(t}. Finally, each site i in cloud K-SVD can only rely on \overline{\Omega}_{i, k}^{(t)}, while K-SVD has access to the matrices \Omega_{k}^{(t)}=diag(\Omega_{1, k}^{(t)},\ \ldots,\ \Omega_{N, k}^{(t)}) and E_{k, R}^{(t)}=E_{k}^{(t)}\Omega_{k}^{(t)}. Steps 4–21 in Algorithm 1 are designed to address these limitations of distributed dictionary learning; we refer the reader to [8] for further details on these steps.

B. Main Challenge

Even after K-SVD and cloud K-SVD are identically initialized, D^{(0)}=\hat{D}_{i}^{(0)}, we will have D^{(1)}\neq\hat{D}_{i}^{(1)} at the end of iteration 1 due to finite power method and consensus iterations in cloud K-SVD. This error in \hat{D}_{i}^{(1)} will then cause errors in sparse coding (Step 3 of Algorithm 1). Next, it can be seen from Steps 5–6 of Algorithm 1 that errors in sparse coding and \hat{D}_{i}^{(1)} will result in deviation of \hat{E}_{i, k, R}^{(2)} from the centralized E_{i, k, R}^{(2)}. This means that the updated k^{th} atom in iteration 2 will have an error due to perturbation of E_{i, k, R}^{(2)} and due to errors caused by finite numbers of consensus and power method iterations. All these errors will keep on accumulating in the same way for any iteration t > 2. In summary, the main sources of error in cloud K-SVD are as follows:

Algorithm 1: Cloud K-SVD for dictionary learning

1. Error in sparse coding due to perturbed dictionaries at the start of any iteration t > 1.

2. Error in \overline{E}_{k, R}^{(t)} due to errors in dictionaries from the previous iteration and errors in sparse codes in the current iteration. This error in E_{k, R}^{(t)} will result in an error during the dictionary update step even if there is no error in computing the principal eigenvector of \hat{Fj}_{k.R}^{(t)}\hat{E}_{k.R}^{(t)^{\mathrm{T}}}.

3. Error in computing the principal eigenvector of \hat{E}_{k, R}^{(t)}\hat{E}_{k, R}^{(t)^{1}} due to finite numbers of power method and consensus iterations.

Our goal in this paper is to analyze how these errors are accumulating in each iteration and how to control these errors such that the errors in dictionaries \hat{D}_{i}^{(t)} stay below some threshold after T_{d} dictionary learning iterations.

A. Preliminaries

We will start by providing algorithmic specification of the sparse coding steps in both algorithms. While the sparse coding step as stated in Step 3 of Algorithm 1 has combinatorial complexity, various low-complexity computational approaches can be used to solve this step in practice. Our analysis in the following will be focused on the case when sparse coding in both cloud K-SVD and centralized K-SVD is carried out using the lasso [9]. Specifically, we assume sparse coding is carried out by solving \begin{equation*} x_{i, s}=\displaystyle \arg\min_{x\in \mathrm{R}^{K}}\frac{1}{2}\Vert y_{i, s}-Dx\Vert_{2}^{2}+\tau\Vert x\Vert_{1} \tag{2} \end{equation*}View Source\begin{equation*} x_{i, s}=\displaystyle \arg\min_{x\in \mathrm{R}^{K}}\frac{1}{2}\Vert y_{i, s}-Dx\Vert_{2}^{2}+\tau\Vert x\Vert_{1} \tag{2} \end{equation*} with the regularization parameter \tau > 0 selected in a way that \Vert x_{i, s}\Vert_{0}\leq T_{0}\ll n. This can be accomplished, for example, by making use of the least angle regression algorithm [10]. Note that the lasso also has a dual, constrained form, given by \begin{equation*} x_{i, s}=\displaystyle \arg\min_{x\in \mathrm{R}^{K}}\frac{1}{2}\Vert y_{i, s}-Dx\Vert_{2}^{2} s.t. \Vert x\Vert_{1}\leq\eta, \tag{3} \end{equation*}View Source\begin{equation*} x_{i, s}=\displaystyle \arg\min_{x\in \mathrm{R}^{K}}\frac{1}{2}\Vert y_{i, s}-Dx\Vert_{2}^{2} s.t. \Vert x\Vert_{1}\leq\eta, \tag{3} \end{equation*} and the solutions of (2) and (3) are identical for an appropriate \eta_{\tau}=\eta(\tau) [11].

We also assume identical centralized and distributed initializations, i.e., \hat{D}_{i}^{(0)}\simeq D^{(0)}, i\simeq 1, \ldots, N, where D^{(t)}, t\geq 0, in the following denotes the centralized K-SVD dictionary estimate in the t^{th} iteration. Despite identical initialization, the cloud K-SVD dictionaries get perturbed in each iteration due to imperfect power method and consensus averaging. In order to ensure these perturbations do not cause the cloud K-SVD dictionaries to diverge from the centralized solution after T_{\mathrm{d}} iterations, we need the dictionary estimates returned by centralized K-SVD in each iteration to satisfy the following properties.

1. [P1] Let x_{i, s}^{(t)} denote the solution of the lasso (i.e., (2)) for D=D^{(t-1)} and \tau=\tau^{(t)}, t=1, \ldots, T_{\mathrm{d}}. Then there exists some C_{1} > 0 such that the following holds: \begin{equation*} \displaystyle \min_{t, i, s, j\not\in \mathrm{supp}(x_{is}^{(t)})} \tau^{(t)}-\vert \langle d_{j}^{(t)}, y_{i, s}-D^{(t-1)}x_{i, s}^{(t)}\rangle\vert > C_{1}. \end{equation*}View Source\begin{equation*} \displaystyle \min_{t, i, s, j\not\in \mathrm{supp}(x_{is}^{(t)})} \tau^{(t)}-\vert \langle d_{j}^{(t)}, y_{i, s}-D^{(t-1)}x_{i, s}^{(t)}\rangle\vert > C_{1}. \end{equation*}

   For collection \{\tau^{(t)}\}_{t=1}^{T_{d}}, we also define the smallest regularization parameter \tau_{\min} = \displaystyle \min_{f}\tau^{(t)}, and the largest dual parameter among the (dual) collection \{\eta_{\tau}^{(t)}=-\eta(\tau^{(t)})\}_{t=1}^{T_{d}} as \displaystyle \eta_{\tau,\max}=\max_{t}\eta_{\tau}^{(t)}.

2. [P2] Define \Sigma_{T_{0}}=\{\mathcal{I}\subset\{1,\ \ldots,\ K\}:\vert \mathcal{I}\vert =\tau_{0}\}. Then there exists some C_{2}' > \displaystyle \frac{C_{1}^{4}\tau_{\min}^{2}}{1936}, such that the following holds, \displaystyle \min_{t=1,\ldots, T_{d},\mathcal{I}\in\Sigma_{T_{0}}}.\sigma_{T_{0}}(D_{\vert \mathcal{I}j}^{(t-1)})\geq\sqrt{C_{2}'}, where \sigma_{T_{0}}(\cdot) denotes the T_{0}^{th} (ordered) singular value of a matrix. In our analysis, we will be using the parameter C_{2}=(\displaystyle \sqrt{C_{2}'}-\frac{c_{1^{\mathcal{T}_{\min}}}^{2^{\vee}}}{44})^{2}.

3. [P3] Let \lambda_{1, k}^{(t)} > \lambda_{2, k}^{(t)}\geq\ldots\lambda_{n, k}^{(t)}\geq denote the eigenvalues of the centralized “reduced” matrix E_{kR}^{(t)}E_{kR}^{(t)^{\mathrm{T}}}, k\in \{ 1, \ldots, K\}, in the t^{th} iteration, t \in \{ 1, \ldots, T_{d}\}. Then there exists some C_{3}' < 1 such that the following holds, \max_{t, k^{\frac{\lambda_{2k}^{(t)}}{\lambda_{1k}^{(t)}}}}-\leq C_{\mathrm{q}, 0}'. Now define C_{3}= \displaystyle \max\{1, \displaystyle \frac{1}{\min_{tk}\lambda_{1k}^{(t)}(1-C_{3}')}\}, which we will use in our forthcoming analysis.

Properties P1 and P2 correspond to sufficient conditions for x_{i, k}^{(t)} to be a unique solution of (2) [12] and guarantee that the centralized K-SVD generates a unique collection of sparse codes in each dictionary learning iteration. Property P3, on the other hand, ensures that algorithms such as the power method can be used to compute the dominant eigenvector of E_{k, R}^{(t)}E_{k, R}^{(t)^{\mathrm{T}}} in each dictionary learning iteration (Steps 8–18 in Algorithm 1) [13]. In addition to these properties, our final analytical result for cloud K-SVD will also be a function of a certain parameter of the centralized error matrices \{E_{k}^{(t)}\}_{k=1}^{K} generated by the centralized K-SVD in each iteration. We define this parameter as follows. Let E_{i, k}^{(t)}, i= 1, \ldots, N, denote part of the centralized error matrix E_{k}^{(t)} associated with the data of the i^{th} site in the t^{th} iteration, i.e., E_{k}^{(t)}= \lfloor E_{1, k}^{(t)} E_{2, k}^{(t)} E_{N, k}^{(t)}\rfloor, k=1, \ldots, K, t\simeq 1, \ldots, T_{d}. Then C_{4}=\displaystyle \max\{1, \displaystyle \max_{t^{\rightarrow}i, k}\Vert E_{i, k}^{(t)}\Vert_{2}\}.

B. Main Result

We are now ready to state the main result of this paper. This result is given in terms of the \Vert\cdot\Vert_{2} norm mixing time, T_{mix}, of the Markov chain associated with the doubly-stochastic weight matrix W used for consensus averaging, defined as \begin{equation*} T_{mix}=\displaystyle \max_{=}\inf_{i1,\ldots, Nt\in \mathrm{N}}\{t: \displaystyle \Vert \mathrm{e}_{i}^{\mathrm{T}}W^{t}-\frac{1}{N}1^{\mathrm{T}}\Vert_{2}\leq\frac{1}{2}\}. \tag{4} \end{equation*}View Source\begin{equation*} T_{mix}=\displaystyle \max_{=}\inf_{i1,\ldots, Nt\in \mathrm{N}}\{t: \displaystyle \Vert \mathrm{e}_{i}^{\mathrm{T}}W^{t}-\frac{1}{N}1^{\mathrm{T}}\Vert_{2}\leq\frac{1}{2}\}. \tag{4} \end{equation*}

Here, \mathrm{e}_{i}\in \mathbb{R}^{N} denotes the i^{th} column of the identity matrix I_{N}. In the following, main convergence results for cloud K-SVD along with brief discussions are presented. For detailed proofs and discussions we refer the reader to [8].

\hat{\theta}_{k}^{(t)} \in [0,\ \pi/2] as \hat{\theta}_{k}^{(t)} arccos (\displaystyle \frac{\vert \langle u_{1k}^{(t)}, q^{init}\rangle\vert }{\Vert u_{1k}^{(t)}\Vert_{2}\Vert q^{init}\Vert_{2}}), \mu=\displaystyle \max\{1, \displaystyle \max_{k, t}\mathrm{t}an(\hat{\theta}_{k}^{(t)})\}, and \zeta K\displaystyle \sqrt{2S_{\max}}(\frac{6\sqrt{KT_{0}}}{\tau_{\min}C_{2}}+\eta_{\tau,\max}), where S_{\max}=\displaystyle \max_{i}S_{i}, u_{1, k}^{(t)} is the dominant eigenvector of \hat{E}_{k, R}^{(t)}\hat{E}_{k, R}^{(t)^{T}}, \hat{\lambda}_{1k}^{(t)} and \hat{\lambda}_{2, k}^{(t)} are first and second largest eigenvalues of \hat{E}_{k, R}^{(t)}\hat{E}_{k, R}^{(t)^{\mathcal{T}}}, respectively, and q_{c, t, k}^{(t_{p})} denotes the iterates of a centralized power method initialized with q^{init} for estimation of the dominant eigenvector of \hat{E}_{k, R}^{(t)}\hat{E}_{k, R}^{(t)^{7}}. Then, assuming \displaystyle \min_{t, k}\vert \langle u_{1, k}^{(t)}, q^{init}\rangle\vert > 0, and fixing any \in(0, \displaystyle \min\{(10\alpha^{2}\beta^{2})^{-1/3T_{p}},\ (\frac{1-\nu}{4})^{1/3}\}) and \delta_{d}\in(0, \displaystyle \min\{\frac{1}{\sqrt{2}},\ \frac{C_{1}^{2}\tau_{\min}}{44\sqrt{2K}}\}), we have \begin{equation*} \max_{i=1,\ldots, N_{k=1,\ldots, K}}\Vert^{\wedge}d_{i, k}^{(T_{d})^{\wedge}}d_{i, k}^{(T_{d})^{\mathcal{T}}}-d_{k}^{(T_{d})}d_{k}^{(T_{d})^{\mathcal{T}}}\Vert_{2}\leq\delta_{d} \tag{5} \end{equation*}View Source\begin{equation*} \max_{i=1,\ldots, N_{k=1,\ldots, K}}\Vert^{\wedge}d_{i, k}^{(T_{d})^{\wedge}}d_{i, k}^{(T_{d})^{\mathcal{T}}}-d_{k}^{(T_{d})}d_{k}^{(T_{d})^{\mathcal{T}}}\Vert_{2}\leq\delta_{d} \tag{5} \end{equation*} as long as the number of power method iterations T_{p}\geq \displaystyle \frac{2(T_{d}K-2)\log(8C_{3}C_{4}^{2^{\mathrm{z}}}N+5)+(T_{d}-1)\log(1+\zeta)+\log(8C_{3}C_{4}\mu N\sqrt{n}\delta_{d}^{-1})}{\log\lceil(l/+4\epsilon^{3})^{-1}\rceil} and the number of consensus iterations T_{c}= \Omega(T_{p}T_{mix}\log(2\alpha\beta\epsilon^{-1})+T_{mix}\log(\alpha^{-1}\gamma\sqrt{N})).

We now comment on the major implications of Theorem 1. First. the theorem establishes that the distributed dictionaries \{\hat{D}_{i}^{(T_{d})}\} can indeed remain arbitrarily close to the centralized dictionary D^{(T_{d})} after T_{\mathrm{d}} dictionary learning iterations (cf. (5)). Second, the theorem shows that this can happen as long as the number of distributed power method iterations T_{p} scale in a certain manner. In particular, Theorem 1 calls for this scaling to be at least linear in T_{d}K (modulo the \log N multiplication factor), which is the total number of SVDs that K-SVD needs to perform in T_{\mathrm{d}} dictionary learning iterations. On the other hand, T_{p} need only scale logarithmically with S_{\text{maec}}, which is significant in the context of big data problems. Other main problem parameters that affect the scaling of T_{p} include T_{0}, n, and \delta_{d}^{-1}, all of which enter the scaling relation in a logarithmic fashion. Finally, Theorem 1 dictates that the number of consensus iterations T_{c} should also scale at least linearly with T_{p}T_{mix} (modulo some log factors) for the main result to hold. In summary, Theorem 1 guarantees that the distributed dictionaries learned by cloud K-SVD can remain close to the centralized dictionary without requiring excessive numbers of power method and consensus averaging iterations.

We now provide a brief heuristic understanding of the roadmap needed to prove Theorem 1. In the first dictionary learning iteration (t=1), we have \{\hat{D}_{i}^{(t-1)}\equiv D^{(t-1)}\} due to identical initializations. While this means both K-SVD and cloud K-SVD result in identical sparse codes for t=1, the distributed dictionaries begin to deviate from the centralized dictionary after this step. The perturbations in \{\hat{d}_{i, k}^{(1)}\} happen due to the finite numbers of power method and consensus averaging iterations for k=1, whereas they happen for k > 1 due to this reason as well as due to the earlier perturbations in \{\hat{d}_{i, j}^{(1)},\hat{x}_{i_{)}j, T}^{(1)}\}, j < k. In subsequent dictionary learning iterations (t > 1), therefore, cloud K-SVD starts with already perturbed distributed dictionaries \{\hat{D}_{i}^{(t-1)}\}. This in turn also results in deviations of the sparse codes computed by K-SVD and cloud K-SVD, which then adds another source of perturbations in \{\hat{d}_{i, k}^{(t)}\} during the dictionary update steps. To summarize, imperfect power method and consensus averaging in cloud K-SVD introduce errors in the top eigenvector estimates of (centralized) E_{1R}^{(1)}E_{1R}^{(1)^{\mathrm{T}}} at individual sites, which then accumulate for (k,\ t)\neq(1, 1) to also cause errors in estimate \hat{E}_{k, R}^{(t)}\hat{E}_{k, R}^{(t)^{\mathrm{T}}} of the matrix E_{k, R}^{(t)}E_{k, R}^{(t)^{\mathrm{T}}} available to cloud K-SVD. Collectively, these two sources of errors cause deviations of the distributed dictionaries from the centralized dictionary and the proof of Theorem 1 mainly relies on our ability to control these two sources of errors.

C. Roadmap to Theorem 1

The first main result needed for the proof of Theorem 1 looks at the errors in the estimates of the dominant eigenvector u_{1} of an arbitrary symmetric matrix M=\sum_{i=1}^{N}M_{i} obtained at individual sites using imperfect power method and consensus averaging when the M_{2}'s are distributed across the N sites (cf. Algorithm 1). The following result effectively helps us control the errors in cloud K-SVD dictionaries due to Steps 8–18 in Algorithm 1.

Proof of this theorem can be find in our earlier work [5]. Theorem 2 states that \hat{q_{i}}\rightarrow^{T_{p}}\pm u_{1} at an exponential rate at each site as long as enough consensus iterations are performed in each iteration of the distributed power method. In the case of a finite number of distributed power method iterations, (6) in Theorem 2 tells us that the maximum error in estimates of the dominant eigenvector is bounded by the sum of two terms, with the first term due to finite number of power method iterations and the second term due to finite number of consensus iterations.

The second main result needed to prove Theorem 1 looks at the errors between individual blocks of the reduced distributed error matrix \hat{E}_{k, R}^{(t)}=\vert f\hat{f}_{1, k, R}^{(t)}, f\hat{fj}_{2, k, R}^{(t)}, \cdots,\hat{]}X_{N, k, R}^{(t)}\vert and the reduced centralized error matrix E_{k, R}^{(t)} \lceil E_{1, k, R}^{(t)}, E_{2, k, R}^{(t)}, \cdots, E_{N, k, R}^{(t)}\rceil for k \in \{ 1, 2, \cdots, K\} and t\in\{1, 2,\ \cdots,\ T_{d}\}. This result helps us control the error in Step 6 of Algorithm 1 and, together with Theorem 2, characterizes the major sources of errors in cloud K-SVD in relation to centralized K-SVD. The following theorem provides a bound on error in E_{i, k, R}^{(t)}

Theorem 3 tells us that the error in matrix E_{i, k, R}^{(t)} can be made arbitrarily small through a suitable choice of T_{p} and \epsilon as long as all of the assumptions of Theorem 1 are satisfied. One of the steps in proving Theorem 1 involves proving that the assumption on \varepsilon is satisfied as long as we are performing power method iterations and consensus iterations as required by Theorem 1 (see to [8] for complete proof). In the following, we provide a brief sketch of the proof of Theorem 3 and refer the reader to [8] for complete details.

We can prove Theorem 3 by induction over dictionary learning iteration t. But first we need to have a way to bound \Vert B_{i, k+1, R}^{(t)}\Vert_{2} using bounds on \{\Vert B_{i, j, R}^{(t)}\Vert_{2}\}_{j=1}^{k} and also we need to have a method to bound \Vert B_{i, 1, R}^{(\check{t}+1)}\Vert_{2} using bounds on \{\Vert B_{i, j, R}^{(t)}\Vert_{2}\}_{j=1}^{K}. Notice from Algorithm 1 that sparse coding is always performed before update of the first dictionary atom. But we do not perform sparse coding before updating any other dictionary atom. Due to this distinction, we address the problem of error accumulation in matrix E_{i, k, R}^{(t)} (t) (t) for first dictionary atom (\Vert B_{i, 1, R}^{(t+1)}\Vert_{2}) differently than for any other dictionary atom (\{\Vert B_{i, j, R}^{(i)}\Vert_{2}\}_{j=2}^{K}). Proof of Theorem 3 can then be divided into three steps.