Modulational Instability in the Whitham Equation for Water Waves

We show that periodic traveling waves with sufficiently small amplitudes of the Whitham equation, which incorporates the dispersion relation of surface water waves and the nonlinearity of the shallow water equations, are spectrally unstable to long-wavelengths perturbations if the wave number is greater than a critical value, bearing out the Benjamin-Feir instability of Stokes waves; they are spectrally stable to square integrable perturbations otherwise. The proof involves a spectral perturbation of the associated linearized operator with respect to the Floquet exponent and the small-amplitude parameter. We extend the result to related, nonlinear dispersive equations.