Suppose we use $n$ polarizers, each with efficiency $\alpha\in (0,1)$, and we orient them at angles $\theta_1,\dots,\theta_n$, as shown below. Our task is to pick the angles $\theta_i$ so that the ratio $\frac{|OB|}{|OA|}$ is maximized.
When polarizer $i$ is applied, the vector gets multiplied by $\alpha\cos(\theta_i)$. Therefore, our task is to solve the following optimization problem for the overall efficiency $\beta_n$:
\begin{align}
\beta \,\,=\,\,
\underset{\theta_1,\dots,\theta_n}{\text{maximize}}\quad &\alpha^n \prod_{i=1}^n \cos(\theta_i) \\
\text{subject to}\quad &\theta_1+\cdots+\theta_n=\tfrac{\pi}{2} \\
& \theta_i \geq 0\quad\text{for }i=1,\dots,n
\end{align}
It turns out that the optimal configuration is for all angles to be equal. To see why this is the case, note that we can equivalently maximize the log of this product. In other words, maximize
\[
\sum_{i=1}^n \log \cos (\theta_i)
\]Consider the function $f(x)=\log\cos(x)$. Since $f'{}'(x) = -\sec^2(x) \leq 0$ on $x\in[0,\tfrac{\pi}{2}]$, it follows that $f$ is a concave function. By Jensen’s inequality,
\[
\frac{f(x_1)+\cdots+f(x_n)}{n} \leq f\biggl(\frac{x_1+\cdots+x_n}{n}\biggr)
\]Applying this to our function, we conclude that:
\[
\sum_{i=1}^n \log \cos(\theta_i) \leq n \log\cos\biggl(\frac{\theta_1+\cdots+\theta_n}{n}\biggr) = n \log\cos\biggl(\frac{\pi}{2n}\biggr),
\]where the equality is due to the fact that we know $\theta_1+\cdots+\theta_n=\tfrac{\pi}{2}$. We can also achieve this equality by setting $\theta_1=\ldots=\theta_n=\tfrac{\pi}{2n}$, so the optimal thing to do is to make all angles equal. Substituting this into our optimization problem, we obtain a formula for the overall efficiency $\beta$:
$\displaystyle
\beta_n \,=\, \alpha^n \cos^n\bigl(\tfrac{\pi}{2n}\bigr)
$
The question is: what value of $n$ maximizes this? For the case where 1% of the light is reflected ($\alpha=0.99$), we can plot $\beta_n$ as a function of $n$ and we obtain (on a log scale):
Closer examination reveals that the optimal number of polarizers is $n=11$, and this leads to an optimal efficiency of:
$\displaystyle
\beta^\star \,=\, (0.99)^{11} \cos^{11}\bigl(\tfrac{\pi}{22}\bigr) \approx 0.800042
$
or about 80%. We can ask a more general question of how the overall efficiency varies for different values of $\alpha$ and $n$. Here is a plot comparing a wide range of possible values:
Limiting cases
The optimal $n$ occurs when $\tfrac{\mathrm{d}}{\mathrm{d}n}\beta_n = 0$, or equivalently, $\tfrac{\mathrm{d}}{\mathrm{d}n}\log(\beta_n) = 0$. This leads to the equation
\[
\log (\alpha )+\frac{\pi}{2n} \tan \left(\frac{\pi }{2 n}\right)+\log \left(\cos \left(\frac{\pi }{2 n}\right)\right) = 0
\]We can solve this for $\alpha$ and obtain
\[
\alpha = \exp\left[-\frac{\pi}{2n} \tan\left(\frac{\pi }{2 n}\right)-\log \cos\left(\frac{\pi }{2 n}\right)\right]
\]This is an exact expression relating the $n$ that is optimal for a given $\alpha$. Of course, this assumes we can use a fractional number of polarizers, but it should be adequate to study the limit when $\alpha\to 1$ (and therefore $n\to\infty$). Taking an asymptotic expansion of the right-hand side, we obtain $\alpha \approx 1-\frac{\pi^2}{8n^2}$, or equivalently
$\displaystyle
n \approx \frac{\pi }{\sqrt{8}\sqrt{1-\alpha }}
$
This allows us to estimate the number of layers required as a function of each polarizer’s efficiency. When $\alpha=0.99$, we obtain $n\approx 11.1$, which is close to the result we found previously. Similarly, if $\alpha=0.9999$, we obtain $n=111$, which matches the peak of the red curve in the plot above.
We might also be interested in the heights of these peaks; so what is the maximum overall efficiency $\beta$ that we can obtain if our individual polarizers each have efficiency $\alpha$? Performing a similar expansion to the one for $\alpha$, we obtain:
\[
\beta = \alpha^n \cos^n\left(\frac{\pi}{2n}\right) \approx 1-\frac{\pi ^2}{4 n}+\frac{\pi ^4}{32 n^2}
\]Eliminating $n$ from the $\alpha$ and $\beta$ equations, we obtain the approximation
\[
\beta \approx 1-\frac{\pi}{\sqrt{2}}\sqrt{1-\alpha }+\frac{\pi^2}{4}(1-\alpha )
\]And as anticipated, as $\alpha\to1$, we obtain $\beta\to 1$.