Cohl (2011) gives a generalisation of the standard generating function for the Gegenbauer polynomials $ C_n^\mu(x)$ :

$ (1 -2tx + t^2)^{-\nu} = A_{\mu,\nu} \frac{(1-t^2)^{-\nu+\mu+1/2}}{t^{\mu+1/2}} \sum_{n=0}^\infty (n+\mu) \: Q^{\nu-\mu-1/2}_{n+\mu-1/2}\left(\frac{1+t^2}{2t}\right) \: C^{\mu}_n(x)$ ,

where $ Q^\mu_\nu(x)$ are associated Legendre functions of the second kind, and $ A_{\mu,\nu} = \frac{\Gamma(\mu) e^{i\pi(\mu-\nu+1/2)}}{\sqrt{\pi}\Gamma(\nu)}$ .

The traditional form is recovered when $ \nu=\mu$ , thanks to the simple closed-form of $ Q^{1/2}_\mu(x)$ .

This is certainly not an ‘ordinary’ generating function, as the polynomials are not simply summed against monomials $ t^n$ . How does one ‘use’ this generating function in practice? For example, how would one extract the $ n$ -th polynomial, or prove the Gegenbauer polynomial recursion relation, etc.?

This generalised form is useful to me because the free parameter $ \nu$ enables a certain integral over a set of functions (where the $ n$ -th function depends in some way on the $ n$ -th Gegenbauer polynomial) to be performed, but the unusual form of this generating function is preventing me from getting a result ‘out the other end’ as it were, as I cannot simply match terms according to the power of $ t$ .

I note that taking $ t \to 0$ and using DLMF 14.8.15 produces the ‘correct’ powers of $ t$ (i.e. $ t^n$ appears), but I believe it would be invalid to interchange the derivatives and limits in order to extract a particular polynomial from the series.