Physical inactivity (PI) and obesity are risk factors for many health conditions, including knee pain (KP). The purpose of the present study was to examine the 6-year effects of PI and obesity on gait speed (GS) among older adults with frequent KP. This prospective cohort study used data from the Osteoarthritis Initiative (OAI). At baseline, we studied 1788 adults aged 45 to 79 years old. We grouped the participants into four categories according to baseline scores on the Physical Activity Scale for the Elderly (PASE) and body mass index (BMI). GS was measured using the 20-m timed walk test. Frequent KP was assessed with a self-report questionnaire, and obesity was assessed by BMI (30 kg/m² or greater). General linear mixed models were conducted using data collected at baseline and 12, 24, 36, 48, and 72 months. After adjusting for all covariates, lower levels of physical activity and obesity were associated with a decrease in GS (β = -0.095, SE = 0.011, p < 0.0001). Our results suggest that both PI and obesity are associated with decreased GS over time in older adults with frequent KP.