We study a nonlinear generalization of the Landau-Zener resonance-crossing problem relevant to coherent photo- and magneto-association of ultracold atoms. Due to the structure of the corresponding classical phase space, the adiabatic theorem breaks down even at very small sweep rates, and the adiabatic approximation diverges because of the crossing of a separatrix. First, by introducing a complex term into the Hamiltonian of the system, we eliminate this divergence and construct a valid zero-order approximation. Further, taking into account that the molecular conversion efficiency and the change of the classical adiabatic invariant at the separatrix crossing are related quantities, we calculate the change of the action for the situation when the system starts from the all-atomic state that corresponds to the case of zero initial action. The absolute error of the presented formula for the change in the action is of the order of or less than 10^-4.