Disordered packings of colloidal spheres show angle-independent structural color when the particles are on the scale of the wavelength of visible light. Previous work has shown that the positions of the peaks in the reflectance spectra can be predicted accurately from a single-scattering model that accounts for the effective refractive index of the material. This agreement shows that the main color peak arises from short-range correlations between particles. However, the single-scattering model does not quantitatively reproduce the observed color: the main peak in the reflectance spectrum is much broader and the reflectance at low wavelengths is much larger than predicted by the model. We use a combination of experiment and theory to understand these features. We find that one significant contribution to the breadth of the main peak is light that is scattered, totally internally reflected from the boundary of the sample, and then scattered again. The high reflectance at low wavelengths also results from multiple scattering but can be traced to the increase in the scattering cross section of individual particles with decreasing wavelength. Both of these effects tend to reduce the saturation of the structural color, which limits the use of these materials in applications. We show that while the single-scattering model cannot reproduce the observed saturations, it can be used as a design tool to reduce the amount of multiple scattering and increase the color saturation of materials, even in the absence of absorbing components.