Cerebral lateralization is a fundamental property of the human brain and a marker of successful development. Here we provide evidence that multiple mechanisms control asymmetry for distinct brain systems. Using intrinsic activity to measure asymmetry in 300 adults, we mapped the most strongly lateralized brain regions. Both men and women showed strong asymmetries with a significant, but small, group difference. Factor analysis on the asymmetric regions revealed 4 separate factors that each accounted for significant variation across subjects. The factors were associated with brain systems involved in vision, internal thought (the default network), attention, and language. An independent sample of right- and left-handed individuals showed that hand dominance affects brain asymmetry but differentially across the 4 factors supporting their independence. These findings show the feasibility of measuring brain asymmetry using intrinsic activity fluctuations and suggest that multiple genetic or environmental mechanisms control cerebral lateralization.