Previous evidence has suggested that vitamins might be beneficial for cognition. This systematic review aimed to investigate the efficacy of B vitamins, antioxidant vitamins, and vitamin D on the cognitive function of non-demented middle-aged or older people. Randomized or quasi-randomized controlled trials of individuals aged 40 years or older were included. PubMed/MEDLINE, EMBASE, CINAHL, PsycINFO, Cochrane Library databases, and other grey literature sources were searched up to November 2019. Their methodological quality was evaluated using the Cochrane Risk of Bias tool. Twenty-three studies on B vitamins (n = 22–1053; comprising folate, B6, and B12), nine on antioxidant vitamins (n = 185–20,469), and six on vitamin D (n = 55–4122) were included. Taking B vitamins for over 3 months was beneficial for global cognition (standardized mean difference (SMD) −0.18, 95% CI −0.30 to −0.06) and episodic memory (SMD −0.09, 95% CI −0.15 to −0.04). However, antioxidant vitamins (SMD −0.02, 95% CI −0.08 to 0.03) and vitamin D (SMD −0.06, 95% CI −0.36 to 0.23) were not. Antioxidant vitamins were beneficial for global cognition in sensitivity analyses using final measurement data as mean difference estimates (SMD, −0.04, 95% CI −0.08 to −0.01). Taking B vitamins and possibly antioxidant vitamins may be beneficial for the cognitive function of non-demented people.