Ferroelectric and ferromagnetic materials possess spontaneous electric and magnetic order, respectively, which can be switched by the corresponding applied electric and magnetic fields. Multiferroics combine these properties in a single material, providing an avenue for controlling electric polarization with a magnetic field and magnetism with an electric field. These materials have been intensively studied in recent years, both for their fundamental scientific interest as well as their potential applications in a broad range of magnetoelectric devices [1, 2, 3, 4]. However, the microscopic origins of magnetism and ferroelectricity are quite different, and the mechanisms producing strong coupling between them are not always well understood. Hence, gaining a deeper understanding of magnetoelectric coupling in these materials is the key to their rational design. Here, we use ultrafast optical spectroscopy to show that quantum charge fluctuations can govern the interplay between electric polarization and magnetic ordering in the charge-ordered multiferroic LuFe2O4.