A. Gossip Algorithms for In-Network Processing

This paper presents an overview of gossip algorithms and issues related to their use for in-network processing in wireless sensor networks. Gossip algorithms have been widely studied in the computer science community for information dissemination and search [14]–​[16]. More recently, they have been developed and studied for information processing in sensor networks. They have the attractive property that no specialized routing is required. Each node begins with a subset of the data in the network. At each iteration, information is exchanged between a subset of nodes, and then this information is processed by the receiving nodes to compute a local update.

Gossip algorithms for in-network processing have primarily been studied as solutions to consensus problems, which capture the situation where a network of agents must achieve a consistent opinion through local information exchanges with their neighbors. Early work includes that of Tsitsiklis et al. [17], [18]. Consensus problems have arisen in numerous applications including: load balancing [19]; alignment, flocking, and multiagent collaboration [20], [21]; vehicle formation [22], tracking and data fusion [23], and distributed inference [24].

The canonical example of a gossip algorithm for information aggregation is a randomized protocol for distributed averaging. The problem setup is such that each node in an n-node network initially has a scalar measurement value, and the goal is to have every node compute the average of all n initial values—often referred to as the average consensus. In pairwise randomized gossiping [25], each node maintains an estimate of the network average, which it initializes with its own measurement value. Let x(t) denote the vector of estimates of the global averages after the tth gossip round, where x(0) is the vector of initial measurements; that is, xi(t) is the estimate² at node i after t iterations. In one iteration, a randomly selected pair of neighboring nodes in the network exchange their current estimates, and then update their estimates by setting xi(t+1)=xj(t+1)=(xi(t)+xj(t))/2. A straightforward analysis of such an algorithm shows that the estimates at each node are guaranteed to converge to the average xave=(1/n)∑ni=1xi(0) as long as the network is connected (information can flow between all pairs of nodes), and as long as each pair of neighboring nodes gossips frequently enough; this is made more precise in Section II. Note that the primitive described above can be used to compute any function of the form ∑ni=1fi(xi(0)). Computations of this form appear in many canonical sensor network applications such as distributed estimation (e.g., computation of means and variances), source localization, and compression; we will return to these examples in Section IV.

Gossip algorithms can be classified as being randomized or deterministic. The scheme described above is randomized and asynchronous, since at each iteration a random pair of nodes is active. In deterministic, synchronous gossip algorithms, at each iteration, node i updates xi(t+1) with a convex combination of its own values and the values received from all of its neighbors, e.g., as discussed in [26]. Asynchronous gossip is much better suited to wireless sensor network applications, where synchronization itself is a challenging task. Asynchronous gossip can be implemented using the framework described in [18] and [27]. Each node runs an independent Poisson clock, and when node i's clock “ticks,” it randomly selects and gossips with one neighbor. In this formulation, denoting the probability that node i chooses a neighbor j by Pi,j, conditions for convergence can be expressed directly as properties of these probabilities. Gossip and consensus algorithms have also been the subject of study within the systems and control community, with a focus on characterizing conditions for convergence and stability of synchronous gossiping, as well as optimization of the algorithm parameters Pi,j; see the excellent surveys by Olfati-Saber and Murray [28], and Ren et al. [29], and references therein.

B. Paper Outline

Our overview of gossip algorithms begins on the theoretical side and progresses towards sensor network applications. Each gossip iteration requires wireless trans-mission and thus consumes valuable bandwidth and energy resources. Section II discusses techniques for bounding rates of convergence for gossip, and thus the number of transmissions required. Because standard pairwise gossip converges slowly on wireless network topologies, a large body of work has focused on developing faster gossip algorithms for wireless networks, and this work is also described. When transmitting over a wireless channel, one must also consider issues such as noise and coding. Section III discusses the effects of finite transmission rates and quantization on convergence of gossip algorithms. Finally, Section IV illustrates how gossip algorithms can be applied to accomplish distributed signal processing tasks such as distributed estimation and compression.

A. Analysis of Gossip Algorithms

In pairwise gossip, only two nodes exchange information at each iteration. More generally, a subset of nodes may average their information. Many of the gossip algorithms that we will be interested in can be described by an equation of the formx(t+1)=W(t)x(t)(1)View Sourcex(t+1)=W(t)x(t)\eqno{\hbox{(1)}} where W(t) are randomly selected averaging matrices, selected independently across time, and x(t)∈Rn is the vector of gossip states after t iterations. When restricted to pairwise averaging algorithms, in each gossip round only the values of two nodes i,j are averaged (as in [25]) and the corresponding W(t) matrices have 1/2 in the coordinates (i,i),(i,j),(j,i),(j,j) and a diagonal identity for every other node. When pairwise gossip is performed on a graph G=(V,E), only the matrices that average nodes that are neighbors on G (i.e., (i,j)∈E) are selected with nonzero probability. More generally, we will be interested in matrices that average sets of node values and leave the remaining nodes unchanged. A matrix W(t) acting on a vector x(t) is a set averaging matrix for a set S of nodes, ifxi(t+1)=1|S|∑i∈Sxi(t),i∈S(2)View Sourcex_{i}(t+1)={1\over\vert S\vert}\sum_{i\in S}x_{i}(t),\qquad i\in S\eqno{\hbox{(2)}} and xi(t+1)=xi(t), i∉S. Such matrices therefore have entry 1/|S| at the coordinates corresponding to the set S and a diagonal identity for all other entries.

It is therefore easy to see that all such matrices will have the following properties: {1⃗ TW(t)=1⃗ TW(t)1⃗ =1⃗ (3)View Source\cases{\vec{1}^{T}W(t)=\vec{1}^{T}\cr W(t)\vec{1}= \vec{1}}\eqno{\hbox{(3)}} which, respectively, ensure that the average is preserved at every iteration, and that 1⃗ , the vector of ones, is a fixed point. Further, any set averaging matrix W is symmetric and doubly stochastic. A matrix is doubly stochastic if its rows sum to unity, and its columns also sum to unity, as implied in (3). The well-known Birkhoff–von Neumann theorem states that a matrix is doubly stochastic if and only if it is a convex combination of permutation matrices. In the context of gossip, the only permutation matrices which contribute to the convex combination are those which permute nodes in S to other nodes in S, and keep all other nodes not in S fixed. The matrix W must also be a projection matrix; i.e., W2=W since averaging the same set twice no longer changes the vector x(t). It then follows that W must also be positive semidefinite.

We are now ready to understand the evolution of the estimate vector x(t) through the product of these randomly selected set averaging matrices x(t+1)=W(t)x(t)=∏k=0tW(k)x(0).(4)View Sourcex(t+1)=W(t)x(t)=\prod_{k=0}^{t}W(k)x(0).\eqno{\hbox{(4)}} Since W(t) are selected independently across time, EW(t)=EW(0) and we can drop the time index and simply refer to the expected averaging matrix \BBE W, which is the average of symmetric, doubly stochastic, positive-semidefinite matrices and therefore also has these properties. The desired behavior is that x(t+1)\rightarrow x_{\rm ave}\vec{1} that is equivalent to asking that\prod_{k=0}^{t}W(k) \rightarrow{1\over n}\!\vec{\,1}\vec{1}^{\,T}.\eqno{\hbox{(5)}}∏k=0tW(k)→1n1→1⃗ T.(5)View Source\prod_{k=0}^{t}W(k) \rightarrow{1\over n}\!\vec{\,1}\vec{1}^{\,T}.\eqno{\hbox{(5)}}

B. Expected Behavior

We start by looking at the expected evolution of the random vector x(t) by taking expectations on both sides of (4) \BBE x(t+1)=\BBE\left(\prod_{k=0}^{t}W(k)\right)x(0)=(\BBE W)^{t+1}x(0)\eqno{\hbox{(6)}}Ex(t+1)=E(∏k=0tW(k))x(0)=(EW)t+1x(0)(6)View Source\BBE x(t+1)=\BBE\left(\prod_{k=0}^{t}W(k)\right)x(0)=(\BBE W)^{t+1}x(0)\eqno{\hbox{(6)}} where the second equality is true because the matrices are selected independently. Since \BBE W is a convex combination of the matrices W(t), which all satisfy the conditions (3), it is clear that \BBE W is also a doubly stochastic matrix. We can see that the expected evolution of the estimation vector follows a Markov chain that has the \bar{x}_{\rm ave}\vec{1} vector as its stationary distribution. In other words, \vec{1} is an eigenvector of \BBE W with eigenvalue 1. Therefore, if the Markov chain corresponding to \BBE W is irreducible and aperiodic, our estimate vector will converge in expectation to the desired average. Let \lambda_{2}(\BBE[W]) be the second largest eigenvalue of \BBE W. If condition (3) holds and if \lambda_{2}(\BBE[W])\ <\ 1, then x(t) converges to x_{\rm ave}\vec{1} in expectation and in mean square. Further precise conditions for convergence in expectation and in mean square can be found in [30].

C. Convergence Rate

The problem with the expectation analysis is that it gives no estimate on the rate of convergence, a key parameter for applications. Since the algorithms are randomized, we need to specify what we mean by convergence. One notion that yields clean theoretical results involves defining convergence as the first time where the normalized error is small with high probability, and controlling both error and probability with one parameter \epsilon.

The key technical theorem used in the analysis of gossip algorithms is the following connection between the averaging time and the second largest eigenvalue of \BBE W.

The topology of the network influences the convergence time of the gossip algorithm, and using this theorem, this is precisely quantified; the matrix \BBE[W] is completely specified by the network topology and the selection probabilities of which nodes gossip. The rate at which the spectral gap 1-\lambda_{2}(\BBE[W]) approaches zero, as n increases, controls the \epsilon-averaging time T_{\rm ave}. The spectral gap is related to the mixing time (see, e.g., [32]) of a random walk on the network topology. Roughly, the gossip averaging time is the mixing time of the simple random walk on the graph times a factor of n. One therefore would like to understand how the spectral gap scales for different models of networks and gossip algorithms.

This was first analyzed for the complete graph⁴ and uniform pairwise gossiping [15], [25], [30]. For this case it was shown that \lambda_{2}(\BBE[W])=1-1/n, and therefore, T_{\rm ave}=\Theta(n\log\epsilon^{-1}). Since only nearest neighbors interact, each gossip round costs two transmitted messages, and therefore, \Theta(n\log\epsilon^{-1}) gossip messages need to be exchanged to converge to the global average within \epsilon accuracy. This yields \Theta(n\log n) messages to have a vanishing error with probability 1/n, an excellent performance for a randomized algorithm with no coordination that averages n nodes on the complete graph. For other well-connected graphs (including expanders and small world graphs), uniform pairwise gossip converges very quickly, asymptotically requiring the same number of messages (\Theta(n\log\epsilon^{-1})) as the complete graph. Note that any algorithm that averages n numbers with a constant error and constant probability of success should require \Omega(n) messages.

If the network topology is fixed, one can ask what is the selection of pairwise gossiping probabilities that maximizes the convergence rate (i.e., maximizes the spectral gap). This problem is equivalent to designing a Markov chain which approaches stationarity optimally fast and, interestingly, it can be formulated as a semidefinite program which can be solved efficiently [25], [26], [33]. Unfortunately, for random geometric graphs (RGGs) ⁵ and grids, which are the relevant topologies for large wireless ad hoc and sensor networks, even the optimized version of pairwise gossip is extremely wasteful in terms of communication requirements. For example, for a grid topology, the number of required messages scales like \Theta(n^{2}\log\epsilon^{-1}) [25], [35]. Observe that this is of the same order as the energy required for every node to flood its estimate to all other nodes. On the contrary, the obvious solution of averaging numbers on a spanning tree and flooding back the average to all the nodes requires only O(n) messages. Constructing and maintaining a spanning tree in dynamic and ad hoc networks introduces significant overhead and complexity, but a quadratic number of messages is a high price to pay for fault tolerance.

D. Faster Gossip Algorithms

Pairwise gossip converges very slowly on grids and random geometric graphs because of its diffusive nature. Information from nodes is essentially performing random walks, and, as is well known, a random walk on the 2-D lattice has to perform d^{2} steps to cover distance d. One approach to gossiping faster is to modify the algorithm so that there is some directionality in the underlying diffusion of information. Assuming that nodes have knowledge of their geographic location, we can use a modified algorithm called geographic gossip [35]. The idea of geographic gossip is to combine gossip with greedy geographic routing towards a randomly selected location. If each node has knowledge of its own location and under some mild assumptions on the network topology, greedy geographic routing can be used to build an overlay network where any pair of nodes can communicate. The overlay network is a complete graph on which pairwise uniform gossip converges with \Theta(n\log\epsilon^{-1}) iterations. At each iteration, we perform greedy routing, which costs \Theta(\sqrt{n/\log n}) messages on a random geometric graph (also the order of the diameter of the network). In total, geographic gossip thus requires \Theta(n^{1.5}\log\epsilon^{-1}/\sqrt{\log n}) messages. The technical part of the analysis involves understanding how this can be done with only local information: assuming that each node only knows their own location, routing towards a randomly selected location is not identical to routing towards a randomly selected node. If the nodes are evenly spaced, however, these two processes are almost the same and the \Theta(n^{1.5}) message scaling still holds [35].

Li and Dai [36] recently proposed location-aided distributed averaging (LADA), a scheme that uses partial locations and Markov chain lifting to create fast gossiping algorithms. Lifting of gossip algorithms is based on the seminal work of Diaconis et al. [37] and Chen et al. [38] on lifting Markov chain samplers to accelerate convergence rates. The basic idea is to lift the original chain to one with additional states; in the context of gossiping, this corresponds to replicating each node and associating all replicas of a node with the original. LADA creates one replica of a node for each neighbor and associates the policy of a node given it receives a message from the neighbor with that particular lifted state. In this manner, LADA suppresses the diffusive nature of reversible Markov chains that causes pairwise randomized gossip to be slow. The cluster-based LADA algorithm performs slightly better than geographic gossip, requiring \Theta(n^{1.5}\log\epsilon^{-1}/(\log n)^{1.5}) messages for random geometric graphs. While the theoretical machinery is different, LADA algorithms also use directionality to accelerate gossip, but can operate even with partial location information and have smaller total delay compared to geographic gossip, at the cost of a somewhat more complicated algorithm. A related scheme was proposed concurrently by Jung et al. [39], [40] and is based on the idea of lifting to an expander graph. ⁶ Mosk-Aoyama and Shah [41] use an algorithm based on the work of Flajolet and Martin [42] to compute averages and bound the averaging time in terms of a “spreading time” associated with the communication graph, with a similar scaling for the number of messages on grids and RGGs.

Just as algorithms based on lifting incorporate additional memory at each node (by way of additional states in the lifted Markov chain), another collection of algorithms seek to accelerate gossip computations by having nodes remember a few previous state values and incorporate these values into the updates at each iteration. These memory-based schemes can be viewed as predicting the trajectory as seen by each node, and using this prediction to accelerate convergence. The schemes are closely related to shift-register methods studied in numerical analysis to accelerate linear system solvers. The challenge of this approach is to design local predictors that provide speedups without creating instabilities. Empirical evidence that such schemes can accelerate convergence rates is shown in [43], and numerical methods for designing linear prediction filters are presented in [44] and [45]. Recent work of Oreshkin et al. [46] shows that improvements in convergence rate on par with geographic gossip are achieved by a deterministic, synchronous gossip algorithm using only one extra tap of memory at each node. Extending these theoretical results to asynchronous gossip algorithms remains an open area of research.

The geographic gossip algorithm uses location information to route packets on long paths in the network. One natural extension of the algorithm is to allow all the nodes on the routed path to be averaged jointly. This can be easily performed by aggregating the sum and the hop length while routing. As long as the information of the average can be routed back on the same path, all the intermediate nodes can replace their estimates with updated value. This modified algorithm is called geographic gossip with path averaging. It was recently shown [47] that this algorithm converges much faster, requiring only \Theta(\sqrt{n}) gossip interactions and \Theta(n\log\epsilon^{-1}) messages, which is clearly minimal.

A related distributed algorithm was introduced by Savas et al. [48], using multiple random walks that merge in the network. The proposed algorithm does not require any location information and uses the minimal number of messages \Theta(n\log n) to average on grid topologies with high probability. The coalescence of information reduces the number of nodes that update information, resulting in optimal communication requirements but also less fault tolerance. In most gossip algorithms, all nodes keep updating their information which, as we discuss in the next section, adds robustness with respect to changes to the network and noise in communications.

Finally, we note the recent development of schemes that exploit the broadcast nature of wireless communications in order to accelerate gossip rates of convergence [49]–​[51], either by having all neighbors that overhear a transmission execute a local update, or by having nodes eavesdrop on their neighbors' communication and then using this information to strategically select which neighbor to gossip with next. The next section discusses issues arising when gossiping specifically over wireless networks.

A. Are Rate Constraints Significant?

In most sensor network architectures today, the overhead of packet headers and reliable communication is so great that using a few bytes to encode the gossip state variables exchanged leads to negligible additional cost while practically giving a precision that can be seen as infinite. Moreover, we can ignore bit errors in transmissions, which very rarely go undetected thanks to cyclic redundancy check bits. It is natural to ask: why should one bother studying rate constraints at all?

One should bother because existing sensor network modems are optimized to transmit long messages, infrequently, to nearby neighbors, in order to promote spatial bandwidth reuse, and were not designed with decentralized iterative computation in mind. Transmission rates are calculated amortizing the overhead of establishing the link over the duration of very long transmission sessions.

Optimally encoding for computation in general (and for gossiping in particular) is an open problem; very few have treated the subject of communication for computation in an information theoretic sense (see, e.g., [53] and [54]) and consensus gossiping is nearly absent in the landscape of network information theory. This is not an accident. Broken up in parts, consensus gossip contains the elements of complex classical problems in information theory, such as multiterminal source coding, the two-way channel, the feedback channel, the multiple access of correlated sources, and the relay channel [55]; this is a frightening collection of open questions. However, as the number of possible applications of consensus gossip primitives expands, designing source and channel encoders to solve precisely this class of problems more efficiently, even though perhaps not optimally, is a worthy task. Desired features are efficiency in exchanging frequently, and possibly in an optimal order, few correlated bits, and exchanging with nodes that are (at least occasionally) very far, to promote rapid diffusion. Such forms of communications are very important in sensor networks and network control.

Even if fundamental limits are hard to derive, there are several heuristics that have been applied to the problem to yield some achievable bound. Numerous papers have studied the effects of intermittent or lossy links in the context of gossip algorithms [independent identically distributed (i.i.d.) and correlated models, symmetric and asymmetric] [56]–​[65]. In these models, lossy links correspond to masking some edges from the topology at each iteration, and, as we have seen above, the topology directly affects the convergence rate. Interestingly, a common thread running through all of the work in this area is that, so long as the network remains connected on average, convergence of gossip algorithms is not affected by lossy or intermittent links, and convergence speeds degrade gracefully.

Another aspect that has been widely studied is that of source coding for average consensus and is the one that we consider next in Section III-B. It is fair to say that, particularly in wireless networks, the problem of channel coding is essentially open, as we will discuss in Section III-C.

B. Quantized Consensus

Quantization maps the state variable exchanged x_{j}(t) onto codes that correspond to discrete points Q_{t,j}(x_{j}(t))= q_{j}(t)\in{\cal Q}_{t,j}\subset\BBR. The set {\cal Q}_{t,j} is referred to as the code used at time t by node j; the points q_{j}(t) are used to generate an approximation \mathhat{x}_{j}(t) of the state x_{j}(t)\in\BBR that each node needs to transmit; the quantizer rate, in bits, is R_{t,j}=\log_{2}\vert{\cal Q}_{t,j}\vert, where \vert{\cal A}\vert is the cardinality of the set {\cal A}. Clearly, under the constraints specified previously on the network update matrix W(t), the consensus states \{c\vec{1}:c\in\BBR\} are fixed points. The evolution of the nodes' quantized states is that of an automaton; under asynchronous random exchanges, the network state forms a Markov chain with \prod_{j=1}^{n}\vert{\cal Q}_{t,j}\vert possible states and consensus states \{c\vec{1}:c\in\BBR\} that are absorbing states. The cumulative number of bits that quantized consensus diffuses throughout the network asymptotically isR_{\rm tot}^{\infty}=\sum_{t=1}^{\infty}R_{t,{\rm tot}}=\sum_{t=1}^{\infty}\sum_{j=1}^{n}R_{t,j}.\eqno{\hbox{(9)}}View SourceR_{\rm tot}^{\infty}=\sum_{t=1}^{\infty}R_{t,{\rm tot}}=\sum_{t=1}^{\infty}\sum_{j=1}^{n}R_{t,j}.\eqno{\hbox{(9)}}

The first simple question is: for a fixed uniform quantizer with step-size \Delta, i.e.,\mathhat{x}_{j}(t)={\rm uni}_{\Delta}\left(x_{j}(t)\right)={\rm arg}\!\min_{q\in{\cal Q}}\left\vert x_{j}(t)-q\right \vertView Source\mathhat{x}_{j}(t)={\rm uni}_{\Delta}\left(x_{j}(t)\right)={\rm arg}\!\min_{q\in{\cal Q}}\left\vert x_{j}(t)-q\right \vert where {\cal Q}\!=\!\{0,\pm\Delta,\pm 2 \Delta,\ldots,\pm(2^{R-1}\!-\!1)\Delta\}, do the states x(t) always converge (in a probabilistic sense) to the fixed points c\vec{1}? The second is: what is the distortion d(\lim_{k\rightarrow\infty}x(t),(1/n)\sum_{i=1}^{n}x_{i}(0) \vec{1}) due to limited R_{k,i} or a total budget R_{\rm tot}? Fig. 1 illustrates the basic answers through numerical simulation. Interestingly, with a synchronous gossip update, quantization introduces new fixed points other than consensus [Fig. 1(a)], and asynchronous gossiping in general reaches consensus, but without guarantees on the location of outcome [Fig. 1(b)].

Fig. 1.

Quantized consensus over a random geometric graph with n=50 nodes transmission radius r=0.3 and initial states \in[0,1] with uniform quantization with 128 quantization levels. (a) Synchronous updates and (b) pairwise exchange.

Show All

Kashyap et al. [66] first considered a fixed code quantized consensus algorithm, which preserves the network average at every iteration. In their paper, the authors draw an analogy between quantization and load balancing among processors, which naturally comes with an integer constraint since the total number of tasks is finite and divisible only by integers (see, e.g., [19], [67], and [68]). Distributed policies to attain a balance among loads were previously proposed in [69] and [70]. Assuming that the average can be written as (1/n)\sum_{j=1}^{n}x_{j}(0)=S/n and denoting L\buildrel{\Delta}\over{=}S\ {\rm mod}\ n, under these updates in [66], it is proven that any algorithm meeting the aforementioned conditions makes every node converge to either L or L+1, thereby approximating the average. The random gossip algorithm analyzed in [71] leads to a similar result, where the final consensus state differs at most by one bin from the true average; the same authors discuss bounds on the rate of convergence in [72]. In these protocols, the agents will be uncertain on what interval contains the actual average: the nodes whose final value is L will conclude that the average is in [L-1,L+1] and those who end with L+1 will think that the average is in [L,L+2]. Benezit et al. [73] proposed a slight modification of the policy, considering a fixed rate class of quantization strategies that are based on voting, requiring only 2 bits of memory per agent and attaining a consensus on the interval that contains the actual average.

To overcome the fact that not all nodes end up having the same quantized value, a simple variant on the quantized consensus problem that guarantees almost sure convergence to a unique consensus point was proposed concurrently in [74] and [75]. The basic idea is to dither the state variables by adding a uniform random variable u\sim{\cal U}(-(\Delta/2),\Delta/2) prior to quantizing the states, i.e., \mathhat{x}_{i}(t)={\rm uni}_{\Delta}(x_{i}(t)+u). This modest change enables gossip to converge to a consensus almost surely, as shown in [76]. This guarantees that the nodes will make exactly the same decision. However, the algorithm can deviate more from the actual average than the quantized consensus policies considered in [66]. The advantage of using a fixed code is the low complexity, but with relatively modest additional cost, the performance can considerably improve.

Carli et al. [77] noticed that the issue of quantizing for consensus averaging has analogies with the problem of stabilizing a system using quantized feedback [78], which amounts to partitioning the state space into sets whose points can be mapped to an identical feedback control signal. Hence, the authors resorted to control theoretic tools to infer effective strategies for quantization. In particular, instead of using a static mapping, they model the quantizer Q_{t}(x_{i}(t)) at each node i as a dynamical system with internal state \xi_{i}(t), which is coupled with the consensus update through a quantized error variable q_{i}(t) (see Fig. 2). They study two particular strategies. They refer to the first as the zoom-in–zoom-out uniform coder/decoder where they adaptively quantize the state as follows. The node states are defined as \xi_{i}(t)=\left(\mathhat{x}_{-1,i}(t),f_{i}(t) \right).\eqno{\hbox{(10)}}View Source\xi_{i}(t)=\left(\mathhat{x}_{-1,i}(t),f_{i}(t) \right).\eqno{\hbox{(10)}} The quantized feedback and its update are \eqalignno{\mathhat{x}_{i}(t)=&\, \mathhat{x}_{-1,i}(t+1)=\mathhat{x}_{-1,i}(t)+f_{i}(t)q_{i}(t)&\hbox{(11)}\cr q_{i}(t)=&\,{\rm uni}_{\Delta} \left({x_{i}(t)-\mathhat{x}_{-1,i}(t)\over f_{i}(t)}\right)&\hbox{(12)}}View Source\eqalignno{\mathhat{x}_{i}(t)=&\, \mathhat{x}_{-1,i}(t+1)=\mathhat{x}_{-1,i}(t)+f_{i}(t)q_{i}(t)&\hbox{(11)}\cr q_{i}(t)=&\,{\rm uni}_{\Delta} \left({x_{i}(t)-\mathhat{x}_{-1,i}(t)\over f_{i}(t)}\right)&\hbox{(12)}} which is basically a differential encoding, and f_{i}(t) is the stepsize, updated according tof_{i}(t+1)=\cases{k_{\rm in}f_{i}(t),&if $\left\vert q_{i}(t)\right\vert\ <\ 1$\cr k_{\rm out}f_{i}(t),&if $\left\vert q_{i}(t)\right\vert=1$}\eqno{\hbox{(13)}}View Sourcef_{i}(t+1)=\cases{k_{\rm in}f_{i}(t),&if $\left\vert q_{i}(t)\right\vert\ <\ 1$\cr k_{\rm out}f_{i}(t),&if $\left\vert q_{i}(t)\right\vert=1$}\eqno{\hbox{(13)}} which allows the encoder to adaptively zoom in and out, depending on the range of q_{i}(t). The second strategy has the same node states but uses a logarithmic quantizer\eqalignno{\mathhat{x}_{i}(t)=&\,\xi_{i}(t+1)=\xi_{i}(t)+q_{i}(t)&\hbox{(14)}\cr q_{i}(t)=&\,{\log}_{\delta}\left(x_{i}(t)-\xi_{i}(t)\right)&\hbox{(15)}}View Source\eqalignno{\mathhat{x}_{i}(t)=&\,\xi_{i}(t+1)=\xi_{i}(t)+q_{i}(t)&\hbox{(14)}\cr q_{i}(t)=&\,{\log}_{\delta}\left(x_{i}(t)-\xi_{i}(t)\right)&\hbox{(15)}} where the logarithmic quantization amounts to\eqalignno{q_{i}(t)=&\,{\rm sign}\left(x_{i}(t)-\xi_{i}(t)\right)\left({1+\delta\over 1-\delta}\right)^{\ell_{i}(t)}\cr\ell_{i}(t):{1\over 1-\delta}\leq&\,\left\vert x_{i}(t)-\xi_{i}(t)\right\vert\left({1+\delta\over 1-\delta}\right)^{-\ell_{i}(t)}\cr\leq&\,{1\over 1+\delta}.&\hbox{(16)}}View Source\eqalignno{q_{i}(t)=&\,{\rm sign}\left(x_{i}(t)-\xi_{i}(t)\right)\left({1+\delta\over 1-\delta}\right)^{\ell_{i}(t)}\cr\ell_{i}(t):{1\over 1-\delta}\leq&\,\left\vert x_{i}(t)-\xi_{i}(t)\right\vert\left({1+\delta\over 1-\delta}\right)^{-\ell_{i}(t)}\cr\leq&\,{1\over 1+\delta}.&\hbox{(16)}} In [77], numerical results are provided for the convergence of the zoom-in–zoom-out quantizer, while the properties of the logarithmic quantizer are studied analytically. Remarkably, the authors prove that if the state average is preserved and if 0\ <\ \delta\ <\ ((1+\lambda_{\min}(W))/(3-\lambda_{\min}(z))), then the network reaches asymptotically exactly the same state as the unquantized average consensus. In other words, for all i, \lim_{k\rightarrow\infty}x_{i}(t)=(1/n)\sum_{i=1}^{n}x_{i}(0). One needs to observe that the logarithmic quantizer replaces state values in an uncountable set \BBR with discrete countable outputs \ell\in\BBN, in the most efficient way [78], but there are still infinite many such sets; in other words, the logarithmic quantizer has unlimited range, and therefore, R_{t,j}=\infty. Hence, in practice, one will have to accept a penalty in accuracy when its range is limited.

Fig. 2.

Quantized consensus node i encoder and node j decoder, with memory.

Show All

The vast signal processing literature on sampling and quantization can obviously be applied to the consensus problem as well to find heuristics. It is not hard to recognize that the quantizers analyzed in [77] are equivalent to predictive quantizers. Noting that the states are both temporally and spatially correlated, it is clear that encoding using the side information that is available at both transmitter and receiver can yield improved performance and lower cost; this is the tenet of the work in [79] and [80], which analyzed a more general class of quantizers. They can be captured in a similar framework as that of [77] by adding an auxiliary state variable \zeta_{ij}(t), which affects the state of the decoder only (see the decoder in Fig. 3). The idea is similar, since \mathhat{x}_{-1,i}(t+1) in (11) is replaced in [80] by the optimum linear minimum mean squared error prediction, performed using k previous states \mathhat{x}_{-1,i}(t)=\sum_{l=1}^{k}a_{i,k}(l)\mathhat{x}_{i}(t-l). Similarly, the receiver state is introduced to utilize the idea of coding with side information [81], where the side information about x_{i}(t) that is available at, say, receiver j consists of the receiver's present state x_{j}(t), as well as possibly its own past states and those of neighbors in communication with node j. The decoder augmented state (\zeta_{ij}(t), \xi_{i}(t)) in [80] is useful to reap the benefits of the refinement in the prediction of x_{i}(t) that the decoder can obtain using its own side information. This prediction \mathhat{x}_{-1,ij}(t) can more closely approximate the true state x_{i}(t) compared to the transmitter \mathhat{x}_{-1,i}(t) and this, in turn, means that (12) can be replaced by a nested quantizer, such as, for example, the nested lattice quantizers in [82]. In practice, to keep the complexity at bay, one can use a static nested lattice quantizer at the transmitter without any memory, while using the current local state as the j-node decoder state, i.e., \zeta_{ij}(t)=x_{j}(t). The main analytical result in [80] is the conclusion that, even with the lowest complexity (i.e., prediction memory k=1 only or \zeta_{ij}(t)=x_{j}(t) and no memory) one needs finite R_{\rm tot}^{\infty}\ <\ \infty to guarantee that the network will reach consensus with a bounded error d(\lim_{k\rightarrow\infty}x(t),(1/n)\sum_{i=1}^{n}x_{i}(0)\vec{1})\leq D_{\rm tot}^{\infty} that decreases as a function of R_{\rm tot}^{\infty}. This is useful to establish since one may argue that, as long as the network is finite, flooding each value from each node, rather than gossiping, would require a total cost in term of transmission bits that is finite, which can also be reduced via optimal joint source and network coding methods. It is meaningful to ask if gossiping can also lead to a similar rate-distortion tradeoff and the result in [80] suggests that this is, indeed, the case.

Fig. 3.

Node j decoder, with memory and side information.

Show All

Recent work has begun to investigate information-theoretic performance bounds for gossip. These bounds characterize the rate-distortion tradeoff either i) as a function of the underlying network topology assuming that each link has a finite capacity [83], or ii) as a function of the rate of the information source providing new measurements to each sensor [84].

C. Wireless Channel Coding for Average Consensus

Quantization provides a source code, but equally important is the channel code that is paired with it. First, using the separation of source and channel coding in wireless networks is not optimal in general. Second, and more intuitively, in a wireless network, there is a variety of rates that can be achieved with a variety of nodes under different traffic conditions. The two key elements that determine what communications can take place are scheduling and channel coding. Theoretically, there is no fixed-range communication; any range can be reached albeit with lower capacity. Also, there is no such thing as a collision; rather, there is a tradeoff between the rate that multiple users can simultaneously access the channel.

The computational codes proposed in [85] aim to strike a near-optimal tradeoff for each gossip iteration, by utilizing the additive noise multiple-access channel as a tool for directly computing the average of the neighborhood. The idea advocated by the authors echoes their previous work [54]: nodes send lattice codes that, when added through the channel, result in a lattice point that encodes a specific algebraic sum of the inputs. Owing to the algebraic structure of the channel codes and the linearity of the channel, each recipient decodes directly the linear combination of the neighbors' states, which provides a new estimate of the network average when added to the local state. The only drawbacks of this approach is that 1) it requires channel state information at the transmitter, and 2) only one recipient can be targeted at the time. The scenario considered is closer to that in [84], since a stream of data needs to be averaged, and a finite round is dedicated to each input. The key result proven is that the number of rounds of gossip grows as O(\log n^{2}/r^{2}) where r is the radius of the neighborhood.

A. Robust Gossip for Distributed Linear Parameter Estimation

The present section focuses on robust gossiping for distributed linear parameter estimation of a vector of parameters with low-dimensional observations at each sensor. We describe the common assumptions on sensing, the network topology, and the gossiping protocols. Although we focus on estimation, the formulation is quite general and applies to many inference problems, including distributed detection and distributed localization.

3. Stochastic Approximation Algorithm

Let {\cal N}_{i}(t) denote the neighbors of node i at iteration t; that is, j\in{\cal N}_{i}(t) if i can receive a transmission from j at time t. In this manner, we allow the connectivity of the network to vary with time. Based on the current state x_{i}(t), the quantized exchanged data {\{Q(x_{j}(t)+\nu_{ij}(t))\}}_{j\in{\cal N}_{i}(t)}, and the observation z_{i}(t), the updated estimate at node i is\displaylines{x_{i}(t+1)\!=\!x_{i}(t)-\alpha(t)\!\!\left[b\!\sum_{j\in{\cal N}_{i}(t)}\!\! \left(x_{i}(t)\!-\!Q\!\left(x_{j}(t)\!+\!\nu_{ij}(t)\right)\right)\right.\hfill\cr\hfill\left.{\vphantom{b\!\sum_{j \in{\cal N}_{i}(t)}\!\!\left(x_{i}(t)\!-\!Q\!\left(x_{j}(t)\!+\!\nu_{ij}(t)\right)\right)}}-H_{i}^{T} \left(z_{i}(t)-H_{i}x_{i}(t)\right)\right].\quad\hbox{(19)}}View Source\displaylines{x_{i}(t+1)\!=\!x_{i}(t)-\alpha(t)\!\!\left[b\!\sum_{j\in{\cal N}_{i}(t)}\!\! \left(x_{i}(t)\!-\!Q\!\left(x_{j}(t)\!+\!\nu_{ij}(t)\right)\right)\right.\hfill\cr\hfill\left.{\vphantom{b\!\sum_{j \in{\cal N}_{i}(t)}\!\!\left(x_{i}(t)\!-\!Q\!\left(x_{j}(t)\!+\!\nu_{ij}(t)\right)\right)}}-H_{i}^{T} \left(z_{i}(t)-H_{i}x_{i}(t)\right)\right].\quad\hbox{(19)}} In (19), b>0 is a constant and \{\alpha(t)\}_{t\geq 0} is a sequence of weights satisfying the persistence condition⁸:\alpha(t)\geq 0,\quad\sum_{t}\alpha(t)=\infty,\quad\sum_{t}\alpha^{2}(t)\ <\ \infty.\eqno{\hbox{(20)}}View Source\alpha(t)\geq 0,\quad\sum_{t}\alpha(t)=\infty,\quad\sum_{t}\alpha^{2}(t)\ <\ \infty.\eqno{\hbox{(20)}} Algorithm (19) is distributed because for sensor n it involves only the data from the sensors in its neighborhood {\cal N}_{i}(t).

The following result from [92] characterizes the desired statistical properties of the distributed parameter estimation algorithm just described. The flavor of these results is common to other stochastic approximation algorithms [105]. First, we have a law of large-numbers-like result that guarantees that the estimates at each node will converge to the true parameter estimates\BBP\left(\lim_{t \rightarrow\infty}x_{i}(t)=\theta,\forall i\right)=1.\eqno{\hbox{(21)}}View Source\BBP\left(\lim_{t \rightarrow\infty}x_{i}(t)=\theta,\forall i\right)=1.\eqno{\hbox{(21)}} If, in addition to the conditions mentioned above, the weight sequence is taken to be\alpha(t)={a\over t+1}\eqno{\hbox{(22)}}View Source\alpha(t)={a\over t+1}\eqno{\hbox{(22)}} for some constant a>0, we also obtain a central-limit-theorem-like result, describing the distribution of estimation error over time. Specifically, for a sufficiently large, we have that the error\sqrt{t}\left(x(t)-\vec{1}\otimes\theta\right)View Source\sqrt{t}\left(x(t)-\vec{1}\otimes\theta\right) converges in distribution to a zero-mean multivariate normal with covariance matrix that depends on the observation matrices, the quantization parameters, the variance of the measurement noise w_{i}(t), and the constants a and b. The two most common techniques for analyzing stochastic approximation algorithms are stochastic Lyapunov functions and the ordinary differential equations method [105]. For the distributed estimation algorithm (19), the results just mentioned can be derived using the Lyapunov approach [92], [106].

Performance analysis of the algorithm for an example network is illustrated in Fig. 4. An example network of n=45 sensors are deployed randomly on a 25 × 25 grid, where sensors communicate in a fixed radius and are further constrained to have a maximum of six neighbors per node. The true parameter {\mmb\theta}^{\ast}\in \BBR^{45}. Each node is associated with a single component of {\mmb\theta}^{\ast}. For the experiment, each component of {\mmb\theta}^{\ast} is generated by an instantiation of a zero-mean Gaussian random variable of variance 25. (Note that the parameter {\mmb\theta}^{\ast} here has a physical significance and may represent the state of the field to be estimated. In this example, the field is assumed to be white, stationary and hence each sample of the field has the same Gaussian distribution and independent of the others. More generally, the components of {\mmb\theta}^{\ast} may correspond to random field samples, as dictated by the sensor deployment, representing a discretization of the partial differential equation governing the field.) Each sensor observes the corresponding field component in additive Gaussian noise. For example, sensor 1 observes z_{1}(t)=\theta_{1}^{\ast}+w_{1}(t), where w_{1}(t)\sim{\cal N}(0,1). Clearly, such a model satisfies the distributed observability conditionG= \sum_{i}H_{i}^{T}H_{i}=I=G^{-1}.\eqno{\hbox{(23)}}View SourceG= \sum_{i}H_{i}^{T}H_{i}=I=G^{-1}.\eqno{\hbox{(23)}}(Note here that H_{i}={\bf e}_{i}^{T}, where {\bf e}_{i} is the standard unit vector with 1 at the ith component and zeros elsewhere.) Fig. 4(a) shows the network topology, and Fig. 4(b) shows the normalized error of each sensor plotted against the iteration index t for an instantiation of the algorithm. The normalized error for the ith sensor at time t is given by the quantity \Vert{\bf x}_{i}(t)-{\mmb \theta}^{\ast}\Vert/45, i.e., the estimation error normalized by the dimension of {\mmb\theta}^{\ast}. We note that the errors converge to zero as established by the theoretical findings. The decrease is rapid at the beginning and slows down as t increases. This is a standard property of stochastic-approximation-based algorithms and is attributed to the decreasing weight sequence \alpha(t) required for convergence.

Fig. 4.

Illustration of distributed linear parameter estimation. (a) Example network deployment of 45 nodes. (b) Convergence of normalized estimation error at each sensor.

Show All

It is interesting to note that, although the individual sensors suffer from low-rank observations of the true parameter, by collaborating, each of them can reconstruct the true parameter value. The asymptotic normality shows that the estimation error at each sensor decays as 1/\sqrt{t}, the decay rate being similar to that of a centralized estimator having access to all the sensor observations at all times. The efficiency of the distributed estimator is measured in terms of its asymptotic variance, the lower limit being the Fisher information rate of the corresponding centralized estimator. As expected, because of the distributed nature of the protocol (information needs to disseminate across the entire network) and quantized (noisy) intersensor communication, the achieved asymptotic variance is larger than the centralized Fisher information rate. In the absence of quantization (perfect communication), it can be shown that the parameter a in (22) can be designed appropriately so that the asymptotic variance of the decentralized estimator matches the centralized Fisher information rate, showing that the distributed estimator described above is efficient for the centralized estimation problem. An example of interest, with Gaussian observation noise, is studied in [92], where it is shown that asymptotic variance attainable by the distributed algorithm is the same as that of the optimum (in the sense of Cramér–Rao) centralized estimator having access to all information simultaneously. This is an interesting result, as it holds irrespective of the network topology given, it remains connected on average. Such a phenomenon is attributed to a time-scale separation between the consensus potential and the innovation rate (rate of new information entering the network), when intersensor communication is unquantized (perfect) with possible link failures.

As noted before, the observation model need not be linear for distributed parameter estimation. In [92], a large class of nonlinear observation models were considered and a notion of distributed nonlinear observability called separably estimable observable models introduced. Under the separably estimable condition, there exist local transforms under which the updates can be made linear. However, such a state transformation induces different time scales on the consensus potential and the innovation update, giving the algorithm a mixed time-scale behavior (see [92] and [107] for details). This mixed time-scale behavior and the effect of biased perturbations leads to the inapplicability of standard stochastic approximation techniques.

B. Source Localization

A canonical problem, encompassing many of the challenges that commonly arise in wireless sensor network applications, is that of estimating the location of an energy-emitting source [1]. Patwari et al. [108] present an excellent overview of the many approaches that have been developed for this problem. The aim in this section is to illustrate how gossip algorithms can be used for source localization using received signal strength (RSS) measurements.

Let \theta\in\BBR^{2} denote the coordinates of the unknown source, and, for i=1,\ldots,n, let y_{i}\in\BBR^{2} denote the location of the ith sensor. The RSS measurement at node i is modeled as f_{i}={\alpha\over\Vert y_{i}-\theta\Vert^{\beta}}+w_{i}\eqno{\hbox{(24)}}View Sourcef_{i}={\alpha\over\Vert y_{i}-\theta\Vert^{\beta}}+w_{i}\eqno{\hbox{(24)}} where \alpha>0 is the signal strength emitted at the source, \beta is the path-loss coefficient, and w_{j} is additive white Gaussian noise. Typical values of \beta are between 2 and 4. This model was validated experimentally in [109]. Centralized maximum-likelihood estimators for single- and multiple-source scenarios based on this model are presented in [110] and [111]. Because the maximum-likelihood problem is, in general, nonlinear and nonconvex, it is challenging to solve in a decentralized fashion. Distributed approaches based on finding a cyclic route through the network are presented in [9] and [10].

An alternative approach, using gossip algorithms [112], forms a location estimate \widehat{\theta} by taking a linear combination of the sensor locations weighted by a function of their RSS measurement \widehat{\theta}={\sum_{i=1}^{n}y_{i}K(f_{i})\over \sum_{i=1}^{n}K(f_{i})}\eqno{\hbox{(25)}}View Source\widehat{\theta}={\sum_{i=1}^{n}y_{i}K(f_{i})\over \sum_{i=1}^{n}K(f_{i})}\eqno{\hbox{(25)}} where K:\BBR^{+}\rightarrow\BBR^{+} is a monotone increasing function satisfying K(0)=0 and \lim_{f\rightarrow\infty}K(f)\ <\ \infty. Intuitively, nodes that are close to the source measure high RSS values, so their locations should be given more weight than nodes that are further away, which will measure lower RSS values. Taking K(f)=1_{\{f\ \geq\ \gamma\}}, where \gamma>0 is a positive threshold, and where 1_{\{\cdot\}} is the indicator function, (25) reduces to \widehat{\theta}_{1}={\sum_{i=1}^{n}y_{i}1_{\left\{\Vert y_{i}-\theta\Vert\leq\gamma^{-1/\beta}\right\}}\over\sum_{i=1}^{n}1_{\left\{\Vert y_{i}-\theta\Vert\leq \gamma^{-1/\beta}\right\}}}\eqno{\hbox{(26)}}View Source\widehat{\theta}_{1}={\sum_{i=1}^{n}y_{i}1_{\left\{\Vert y_{i}-\theta\Vert\leq\gamma^{-1/\beta}\right\}}\over\sum_{i=1}^{n}1_{\left\{\Vert y_{i}-\theta\Vert\leq \gamma^{-1/\beta}\right\}}}\eqno{\hbox{(26)}} which is simply the centroid of the locations of sensors that are no further than \gamma^{-1/\beta} from the source. In [112], it was shown that this estimator benefits from some attractive properties. First, if the sensor locations y_{i} are modeled as uniform and random over the region being sensed, then \widehat{\theta}_{1} is a consistent estimator, as the number of sensors grows. It is interesting to note that one does not necessarily need to know the parameters \alpha or \beta precisely to implement this estimator. In particular, because (25) is self-normalizing, the estimator automatically adapts to the source signal strength \alpha. In addition, Rabbat et al. [112] show that this estimator is robust to choice of \gamma. In particular, even if \beta is not known precisely, the performance of (25) degrades gracefully. On the other hand, the maximum-likelihood approach is very sensitive to model mismatch and estimating \alpha and \beta can be challenging.

Note that (25) is a ratio of linear functions of the measurements at each node. To compute (25), we run two parallel instances of gossip over the network, one each for the numerator and the denominator. If each node initializes x_{i}^{N}(0)=y_{i}K(f_{i}), and x_{i}^{D}(0)=K(f_{i}), then executing gossip iterations will cause the values at each node to converge to \lim_{t\rightarrow\infty}x_{i}^{N}(t) \!=\!(1/n)\sum_{j=1}^{n}y_{j}K(f_{j}) and \lim_{t\rightarrow\infty}x_{i}^{D}\!=(1/n)\sum_{j=1}^{n}K(f_{j}). Of course, in a practical implementation, one would stop gossiping after a fixed number of iterations t_{stop}, which depends on the desired accuracy and network topology. Then, each node can locally compute the estimate x_{i}^{N}(t_{stop})/x_{i}^{D}(t_{stop}) of the source's location. Note that throughout this section it was assumed that each node knows its own location. This can also be accomplished using a gossip-style algorithm, as described in [113].

C. Distributed Compression and Field Estimation

Extracting information in an energy-efficient and communication-efficient manner is a fundamental challenge in wireless sensor network systems. In many cases, users are interested in gathering data to see an “image” of activity or sensed values over the entire region. Let f_{i}\in \BBR denote the measurement at node i, and let f\in\BBR^{n} denote the network signal obtained by stacking these values into a vector. Having each sensor transmit f_{i} directly to an information sink is inefficient in many situations. In particular, when the values at different nodes are correlated or the signal is compressible, then one can transmit less data without loosing the salient information in the signal. Distributed source coding approaches attempt to reduce the total number of bits transmitted by leveraging the celebrated results of Slepian and Wolf [114] to code with side information [115], [116]. These approaches make assumptions about statistical characteristics of the underlying data distribution that may be difficult to verify in practice.

An alternative approach is based on linear transform coding, gossip algorithms, and compressive sensing. It has been observed that many natural signals are compressible under some linear transformation. That is, although f may have energy in all locations (i.e., f_{i}>0 for all i), there is a linear basis transformation matrix T\in\BBR^{n\times n}, such that when f is represented in terms of the basis T by computing \theta=Tf, the transformed signal \theta is compressible (i.e., \theta_{j}\approx 0 for many j). For example, it is well known that smooth 1-D signals are well approximated using the Fourier basis, and piecewise smooth images with smooth boundaries (a reasonable model for images) are well approximated using wavelet bases [117].

To formally capture the notion of compressibility using ideas from the theory of nonlinear approximation [118], we reorder the coefficients \theta_{j} in order of decreasing magnitude \left\vert\theta_{(1)}\right\vert\geq\left\vert \theta_{(2)}\right\vert\geq\left\vert\theta_{(3)}\right\vert\geq\cdots\geq\left\vert\theta_{(n)}\right\vert\eqno{\hbox{(27)}}View Source\left\vert\theta_{(1)}\right\vert\geq\left\vert \theta_{(2)}\right\vert\geq\left\vert\theta_{(3)}\right\vert\geq\cdots\geq\left\vert\theta_{(n)}\right\vert\eqno{\hbox{(27)}} and then define the best m-term approximation of f in T as f^{(m)}=\sum_{j=1}^{m}\theta_{(j)}T_{:,(j)}, where T_{:,j} denotes the jth column of T. This is analogous to projecting f onto the m-dimensional subspace of T that captures the most energy in f. We then say that f is \alpha-compressible in T, for \alpha\geq 1, when the mean squared approximation error behaves like{1 \over n}\left\Vert f-f^{(m)}\right\Vert^{2}\leq Cm^{-2\alpha}\eqno{\hbox{(28)}}View Source{1 \over n}\left\Vert f-f^{(m)}\right\Vert^{2}\leq Cm^{-2\alpha}\eqno{\hbox{(28)}} for some constant C>0. Since the error exhibits a power-law decay in m for compressible signals, it is possible to achieve a small mean squared approximation error while only computing and/or communicating the few most significant coefficients \theta_{(1)},\ldots,\theta_{(m)}. Fig. 5 shows an example where 500 nodes forming a random geometric graph sample a smooth function. As a compressing basis T, we use the eigenvectors of the normalized graph Laplacian (a function of the network topology), which are analogous to the Fourier basis vectors for signals supported on G [119].

Fig. 5.

Example illustrating compression of a smooth signal. (a) Original smooth signal which is sampled at 500 random node locations, and nodes are connected as in a random geometric graph. (b) The m-term approximation error decay in both the original basis and using the eigenvectors of the graph Laplacian as a transform, which is analogous to taking a Fourier transform of signals supported on the network. (c) The reconstruction error after gossiping on random linear combinations of the sensor measurements and reconstructing using compressed sensing techniques. Note that using more random linear projections (larger k) gives lower error, but the number of projections used is much smaller than the network size.

Show All

Observe that each coefficient \theta_{j} is a linear function of the data at each node, and so one could conceivably compute these coefficients using gossip algorithms. Assuming that each node i knows the values {\{T_{j,i}\}}_{j=1}^{n} in each basis vector, to compute \theta_{j}, we can initialize x_{i}(0)=nT_{j,i}f_{i}, and, by gossiping, each node will compute \lim_{t\rightarrow\infty}x_{i}(t)= \sum_{k=1}^{n}T_{j,k}f_{k}=\theta_{j}. The main challenge with this approach is that the indices of the most significant coefficients are very signal specific, and are generally not known in advance.

We can avoid this issue by making use of the recent theory of compressive sensing [120]–​[122], which says that one can recover sparse signals from a small collection of random linear combinations of the measurements. In the present setting, to implement the gathering of k compressive sensing measurements using gossip algorithms, each node initializes k parallel instances of gossip with x_{i,j}(0)=nA_{i,j}f_{i}, j=1,\ldots,k, where A_{i,j}, e.g., are i.i.d. zero-mean normal random variables with variance 1/n. Let \bar{x}_{j} denote the limiting value of the jth gossip instance at each node. Stacking these into the vector \bar{x}, any node can recover an estimate of the signal f by solving the optimization\min_{\theta}{\Vert\bar{x}-AT^{T}\theta \Vert}^{2}+\tau\sum_{i=1}^{n}\vert\theta_{i}\vert\eqno{\hbox{(29)}}View Source\min_{\theta}{\Vert\bar{x}-AT^{T}\theta \Vert}^{2}+\tau\sum_{i=1}^{n}\vert\theta_{i}\vert\eqno{\hbox{(29)}} where \tau>0 is a regularization parameter. In practice, the values A_{i,j} can be pseudorandomly generated at each node using a predefined seeding mechanism. Then, any user can retrieve the gossip values {\{x_{i,j}(t)\}}_{j=1}^{k} from any node i and solve the reconstruction. Moreover, note that the compressing transformation T only needs to be known at reconstruction time, and to initialize the gossip instances each node only needs its measurement and pseudorandomly generated values A_{i,j}. In general, there is a tradeoff between 1) k, the number of compressed sensing measurements collected, 2) \epsilon, the accuracy to which the gossip algorithm is run, 3) the number of transmissions required for this computation, and 4) the average reconstruction accuracy available at each node. For an \alpha-compressible signal f, compressed sensing theory provides bounds on the mean squared reconstruction error as a function of k and \alpha, assuming that the values \bar{x} are calculated precisely. Larger k corresponds to lower error, and the error decays rapidly with k (similar to the m-term approximation), so one can obtain a very accurate estimate of f with k\ll n measurements. Inaccurate computation of the compressed sensing values \bar{x}, due to gossiping for a finite number of iterations, can be thought of as adding noise to the values \bar{x}, and increases the overall reconstruction error. Fig. 5(c) illustrates, via numerical simulation, the tradeoff between varying k and the number of gossip iterations. For more on the theoretical performance guarantees achievable in this formulation, see [123] and [124]. A gossip-based approach to solving the reconstruction problem in a distributed fashion is described in [125]. For an alternative approach to using gossip for distributed field estimation, see [126].