Chemotherapy improves breast cancer survival but is underused more often in black than in white women. We examined associations between patient-physician relationships and chemotherapy initiation and timeliness of initiation among black and white patients. Women with primary invasive, non-metastatic breast cancer were recruited via hospitals (in Washington, DC and Detroit) and community outreach between July 2006 and April 2011. Data were collected via telephone interviews and medical records. Logistic regression models evaluated associations between chemotherapy initiation and independent variables. Since there were race interactions, analyses were race-stratified. Factors associated with time from surgery to chemotherapy initiation and delay of ≥90 days were evaluated with linear and logistic regressions, respectively. Among eligible women, 82.8 % were interviewed and 359 (90.9 %) of those had complete data. The odds of initiating chemotherapy were 3.26 times (95 % CI: 1.51, 7.06) higher among black women reporting greater communication with physicians (vs. lesser), after considering covariates. In contrast, the odds of starting chemotherapy were lower for white women reporting greater communication (vs. lesser) (adjusted OR 0.22, 95 % CI: 0.07, 0.73). The opposing direction of associations was also seen among the sub-set of black and white women with definitive clinical indications for chemotherapy. Among those initiating treatment, black women had longer mean time to the start of chemotherapy than whites (71.8 vs. 55.0 days, p = 0.005), but race was not significant after considering trust in oncologists, where initiation time decreased as trust increased, controlling for covariates. Black women were also more likely to delay ≥90 days than whites (27 vs. 8.3 %; p = 0.024), but this was not significant after considering religiosity. The patient-physician dyad and sociocultural factors may represent leverage points to improve chemotherapy patterns in black women.