The lifetimes of sub THz acoustic phonon modes determine the intrinsic quality factor of nanomechanical resonators, and control the ultimate limits to sensing mass change, liquid density, charge and temperature with such devices. Recent experiments have provided direct measurements of longitudinal acoustic phonon lifetimes in the higher GHz to THz regime for silicon. However, the results do not definitively resolve the relative contributions of intrinsic mechanisms (such as Akhiezer) versus extrinsic mechanisms (such as boundary scattering), particularly at the higher frequencies. This work focuses on understanding how these mechanisms influence phonon transport through acoustic measurements in nanostructures with well-characterized surface morphologies. We employ a femtosecond laser pump-probe setup to excite and measure the lifetimes of longitudinal acoustic phonons in ultrathin silicon membranes with thicknesses down to 36 nm. We show that the phonon lifetime for membranes thicker than 200 nm is limited intrinsically by Akhiezer mechanism. In thinner membranes, boundary scattering is the most dominant dissipation mechanism. We use a surface specularity parameter based on Kirchhoff's approximation to correctly predict the observed trend. Our results provide insights to understanding thermal and acoustic transport in nanostructures.