When Do Low-Rate Concatenated Codes Approach the Gilbert–Varshamov Bound?

4 Most Linear Codes \(\mathcal{C}_{\mathrm{out}}\) Work Well

Definition 41 Multiplicity and parity matrices, Definition 4.1

For a sequence \(\mathcal{V}= ((\alpha _1,b_1),\dots ,(\alpha _r,b_r))\), let \(V = V(\mathcal{V}) \in \mathbb {N}^{n \times n_0}\) be its “unordered” version, where \(V[\alpha ,b]\) is the number of times \((\alpha ,b)\) appears in \(\mathcal{V}\). Let \(B = B(\mathcal{V}) = V \bmod 2\) (entrywise). The weight \(\| B\| \) is the number of \(1\)-entries of \(B\).

Fix a linear code \(\mathcal{C}_{\mathrm{in}}\subseteq \mathbb {F}_2^{n_0}\) of dimension \(k_0\) that is \(\tau \)-nice for \(\tau = 1/\sqrt{n_0}\). Let \(\varepsilon \triangleq k_0/n_0\) and \(r \in \mathbb {N}\). Let \(G_0 \in \mathbb {F}_2^{n_0\times k_0}\) have left kernel \(\mathcal{C}_{\mathrm{in}}^\perp \), and set

\[ W = \big\{ \mathcal{V}\in ([n]\times \Omega )^r : B(\mathcal{V})\cdot G_0 = 0\big\} . \]

Then

\[ |W| \leq (8N)^r \cdot (r/N)^{r/2}\cdot 2^{\frac{2N}{\sqrt{n_0}} + \log _2 N} \cdot \max \Big\{ 1, \Big(\tfrac {re}{\varepsilon ^2 N}\Big)^{r/2}\Big\} . \]
Proof

Let \(W' = \{ V(\mathcal{V}) : \mathcal{V}\in W\} \). Since \(V(\mathcal{V})\) retains all of \(\mathcal{V}\) up to ordering, \(|W| \leq r!\, |W'|\). We bound \(|W'|\).

Fix \(V \in W'\) and write \(V = B + E\) where \(B \in \{ 0,1\} ^{n\times n_0}\) is the parity matrix and \(E\) has all entries even. The total entry-sum of \(V\) is \(r\); write \(p = r/N\), so the entry-sum is \(Np\). For \(0 \leq m \leq Np\) let \(W'_m\) be the set of \(V = B+E \in W'\) with \(\| B\| = m\). We count the choices of \(E\) and \(B\) separately.

Choosing \(E\). The matrix \(E\) has entry-sum \(Np - m\). Each nonzero entry of \(E\) is even, hence \(\geq 2\), so \(E\) has at most \(\frac{Np-m}{2}\) nonzero entries. The number of ways to choose their positions is at most \(\sum _{t=0}^{(Np-m)/2}\binom {N}{t} \leq 2^{N\cdot h_2\left(\frac{Np-m}{2N}\right)}\) by Lemma 32. Given the positions, the number of even value-assignments is at most \(\binom {\frac{Np-m}{2}+Np-m-1}{Np-m}\leq 2^{3Np/2}\) by Lemma 37.

Choosing \(B\). Let \(I = \{ i \in [n] : B_i \neq 0\} \) be the set of nonzero rows; the number of choices of \(I\) is at most \(2^n\). Write \(|I| = \gamma n\). Given \(m\) and \(I\), we claim there are at most \(2^{N\gamma (h_2(m/\gamma N) - \varepsilon + 1/\sqrt{n_0})}\) choices for \(B\). Indeed, every row of \(B\) must lie in \(\mathcal{C}_{\mathrm{in}}^\perp \). Let \(B'\) be a uniformly random matrix among those of weight \(m\) whose nonzero rows are exactly \(I\); the number of such matrices is at most \(\binom {n_0|I|}{m} = \binom {\gamma n n_0}{m}\leq 2^{\gamma Nh_2(m/\gamma N)}\). Conditioning on the weight \(w\) of each row, \(\tau \)-niceness with \(\tau = 1/\sqrt{n_0}\) gives, for each \(i \in I\),

\[ \Pr \big[B'_i \in \mathcal{C}_{\mathrm{in}}^\perp \mid |B'_i| = w\big] = \frac{|\{ x \in \mathcal{C}_{\mathrm{in}}^\perp : |x| = w\} |}{\binom {n_0}{w}} \leq 2^{-n_0\varepsilon + \sqrt{n_0}} . \]

Since the rows are independent under this conditioning, the probability that all \(\gamma n\) nonzero rows of \(B'\) lie in \(\mathcal{C}_{\mathrm{in}}^\perp \) is at most \(2^{\gamma n(-n_0\varepsilon + \sqrt{n_0})} = 2^{\gamma N(-\varepsilon + 1/\sqrt{n_0})}\) (using \(n\sqrt{n_0} = N/\sqrt{n_0}\)). Multiplying the count of weight-\(m\) matrices by this probability proves the claim.

Combining. Writing \(m = \alpha N\) and combining the bounds,

\[ \frac{\log _2|W'_m|}{N} \leq \max _{\gamma \in [0,1]} \Big\{ \tfrac {3p}{2} + \tfrac {n}{N} + h_2\! \big(\tfrac {p-\alpha }{2}\big) + \gamma \big(h_2(\tfrac {\alpha }{\gamma }) - \varepsilon + \tfrac {1}{\sqrt{n_0}}\big)\Big\} . \]

Using \(h_2(x) \leq -x\log _2 x + 2x\) (Lemma 34) and optimising over \(\gamma \) (the choice \(\gamma = \alpha /(\varepsilon \ln 2)\) maximises the bracket) gives, after simplification,

\[ \frac{\log _2|W'_m|}{N} \leq 3p + \tfrac {1}{\sqrt{n_0}} + \tfrac {n}{N} + \tfrac {p-\alpha }{2}\log _2\! \big(\tfrac {2}{p-\alpha }\big) + \alpha \log _2\! \big(\tfrac {1}{\varepsilon }\big). \]

The last expression is maximised at \(\alpha = \max \{ p - \varepsilon ^2/e, 0\} \): its derivative in \(\alpha \) is \(\frac{1}{2\ln 2}\big(1 + \ln \frac{p-\alpha }{ \varepsilon ^2}\big)\), which is negative throughout \([0,p]\) when \(p {\lt} \varepsilon ^2/e\) (so the max is at \(\alpha = 0\)), and vanishes at \(\alpha = p - \varepsilon ^2/e\) when \(p \geq \varepsilon ^2/e\). Plugging in each case yields

\[ \frac{\log _2|W'_m|}{N} \leq 3p + \tfrac {1}{\sqrt{n_0}} + \tfrac {n}{N} + \tfrac {p}{2}\log _2\! \Big(\max \big\{ \tfrac {1}{p}, \tfrac {e}{\varepsilon ^2}\big\} \Big). \]

Therefore, summing over \(0 \leq m \leq Np\) and using \(|W| \leq r!\, |W'| \leq (Np)!\sum _{m}|W'_m|\),

\[ |W| \leq N^{Np+1}\cdot 2^{3Np + N/\sqrt{n_0} + n}\cdot p^{Np/2}\cdot \big(\max \{ 1, p/\varepsilon ^2\} \big)^{Np/2} \leq (8N)^r (r/N)^{r/2} 2^{2N/\sqrt{n_0}+\log _2 N} \big(\max \{ 1, \tfrac {r}{\varepsilon ^2 N}\} \big)^{r/2}, \]

where we substituted \(p = r/N\) and used \(n = N/n_0 \leq N/\sqrt{n_0}\).

There exist constants \(c, \bar{c}, \tilde{c} {\gt} 0\) such that the following holds. Fix an integer \(k {\gt} 0\), any sufficiently small \(\varepsilon {\gt} 0\) (in terms of \(\bar c\)), and a power of two \(q = 2^{k_0}\). Let \(n_0 = k_0/\varepsilon \), \(n = k/\varepsilon \), \(N = n_0 n\), and suppose \(q \geq 2^{\tilde c/\varepsilon ^3}\). Let \(\mathcal{C}_{\mathrm{in}}\subseteq \mathbb {F}_2^{n_0}\) be a linear code of dimension \(k_0\) that is \(\tau \)-nice for \(\tau = 1/\sqrt{n_0}\), and let \(\mathcal{C}_{\mathrm{out}}\subseteq \mathbb {F}_q^n\) be an independent random linear code of dimension \(k\). Then with probability at least \(1 - 2^{-\varepsilon ^2 N/\bar c}\) over \(\mathcal{C}_{\mathrm{out}}\), the relative distance of \(\mathcal{C}= \mathcal{C}_{\mathrm{out}}\circ \mathcal{C}_{\mathrm{in}}\subseteq \mathbb {F}_2^N\) is at least \(\frac12 - c\varepsilon \).

Proof

Choose \(r = \varepsilon ^2 N\) (the smallest even integer larger than \(\varepsilon ^2 N\); assume WLOG equality). By Lemma 40, for any fixed \(\mathcal{C}_{\mathrm{in}}\) (hence fixed \(\Omega \)),

\begin{equation} \mathbb {E}_{m \sim \mathbb {F}_q^k\setminus \{ 0\} }[X_m^r(\mathcal{C}_{\mathrm{out}},\Omega )] = \frac{1}{q^k-1}\sum _{\mathcal{V}\in ([n]\times \Omega )^r} \big(q^k\mathbf{1}[g_\mathcal{V}\in \mathcal{C}_{\mathrm{out}}^\perp ]-1\big). \tag {8} \end{equation}
1

Take the expectation over the random \(\mathcal{C}_{\mathrm{out}}\). Since \(\mathcal{C}_{\mathrm{out}}^\perp \) is a random linear code of dimension \(n-k\), for any fixed \(\mathcal{V}\), \(\Pr _{\mathcal{C}_{\mathrm{out}}}[g_\mathcal{V}\in \mathcal{C}_{\mathrm{out}}^\perp ] = 1\) if \(g_\mathcal{V}= 0\) and \(\frac{q^{n-k}-1}{q^n-1}\) otherwise. Hence

\begin{equation} \mathbb {E}_{\mathcal{C}_{\mathrm{out}},m}[X_m^r] = \frac{1}{q^k-1}\sum _\mathcal{V}\big(q^k\Pr _\mathcal{C}_{\mathrm{out}}[g_\mathcal{V}\in \mathcal{C}_{\mathrm{out}}^\perp ]-1\big) = \sum _\mathcal{V}\Big(\mathbf{1}[g_\mathcal{V}=0] - \tfrac {1}{q^n-1}\mathbf{1}[g_\mathcal{V}\neq 0]\Big) \leq \sum _\mathcal{V}\mathbf{1}[g_\mathcal{V}= 0]. \tag {9} \end{equation}
2

Now \(g_\mathcal{V}= 0\) iff every row of \(B(\mathcal{V})\) is a left-kernel vector of \(G_0\), i.e. iff every row of \(B(\mathcal{V})\) lies in \(\mathcal{C}_{\mathrm{in}}^\perp \), i.e. iff \(\mathcal{V}\in W\) as in Claim 42. Thus \(\sum _\mathcal{V}\mathbf{1}[g_\mathcal{V}=0] = |W|\), and Claim 42 with \(r = \varepsilon ^2 N\) (so \(r/N = \varepsilon ^2\) and \(\max \{ 1,(re/\varepsilon ^2 N)^{r/2}\} = e^{r/2}\)) gives

\begin{equation} \mathbb {E}_{\mathcal{C}_{\mathrm{out}},m}[X_m^r] \leq |W| \leq (8N)^r(e\varepsilon ^2)^{r/2} 2^{2N\varepsilon ^2/\sqrt{\tilde c}}\cdot N = (32\sqrt e\, N\varepsilon )^r\cdot N, \tag {10} \end{equation}
3

using \(q \geq 2^{\tilde c/\varepsilon ^3} \Rightarrow n_0 \geq \tilde c/\varepsilon ^4\) (so \(2N/\sqrt{n_0}\leq 2N\varepsilon ^2/\sqrt{\tilde c}\)) and WLOG \(\tilde c \geq 1\) so that \(8\sqrt e\cdot 2^{2/\sqrt{\tilde c}}\leq 32\sqrt e\).

By Markov’s inequality (Lemma 23), using that \(r\) is even so \(X_m^r = |X_m|^r\), and \(q^k = 2^{N\varepsilon ^2} = 2^r\),

\[ \Pr _\mathcal{C}_{\mathrm{out}}\big[\exists m\neq 0,\ X_m {\gt} c\varepsilon N\big] \leq \sum _{m\neq 0}\frac{\mathbb {E}_\mathcal{C}_{\mathrm{out}}[X_m^r]}{(c\varepsilon N)^r} = \frac{(q^k-1)\mathbb {E}_{\mathcal{C}_{\mathrm{out}},m}[X_m^r]}{(c\varepsilon N)^r} \leq \frac{2^r\, \mathbb {E}_{\mathcal{C}_{\mathrm{out}},m}[X_m^r]}{(c\varepsilon N)^r}. \]

Plugging in (10) gives \(\big(\frac{64\sqrt e\, N\varepsilon }{c\varepsilon N}\big)^r\cdot N = (\frac{64\sqrt e}{c})^r\cdot N\). Choosing \(c = 128\sqrt e\) yields \(\Pr _\mathcal{C}_{\mathrm{out}}[\exists m\neq 0,\ X_m {\gt} c\varepsilon N]\leq N\cdot 2^{-r} = N\cdot 2^{-N\varepsilon ^2}\). Since \(n_0 \geq \tilde c/\varepsilon ^4\) implies \(n_0 \leq 2^{n_0\varepsilon ^2}\) (for small \(\varepsilon \)) and hence \(N = n n_0 \leq 2^{N\varepsilon ^2}\), for sufficiently large \(N\) we have \(N \leq 2^{N\varepsilon ^2/2}\), so the probability is at most \(2^{-N\varepsilon ^2/2} \leq 2^{-\varepsilon ^2 N/\bar c}\) for \(\bar c \geq 2\). When no message has \(X_m {\gt} c\varepsilon N\), every nonzero codeword of \(\mathcal{C}\) has weight at least \(\frac12 - c\varepsilon \) times \(N\) (by Definition 39), so the relative distance is at least \(\frac12 - c\varepsilon \).