We consider a quantum particle in a potential  subject to a (spatially homogeneous) time-dependent electric field, which plays the role of the control. Under generic assumptions on the potential, this system is approximately controllable, as proved by Boscain, Caponigro, Chambrion and Sigalotti. In the present article, we show that this approximate controllability result is false in small time. As a consequence, the result by Boscain et al. is, in some sense, optimal with respect to the control time.