We report on a measurement of the asymmetry in the scattering of transversely polarized electrons off unpolarized protons, A$_\perp$, at two Q$^2$ values of \qsquaredaveragedlow (GeV/c)$^2$ and \qsquaredaveragedhighII (GeV/c)$^2$ and a scattering angle of $30^\circ < \theta_e < 40^\circ$. The measured transverse asymmetries are A$_{\perp}$(Q$^2$ = \qsquaredaveragedlow (GeV/c)$^2$) = (\experimentalasymmetry alulowcorr $\pm$ \statisticalerrorlow$_{\rm stat}$ $\pm$ \combinedsyspolerrorlowalucor$_{\rm sys}$) $\times$ 10$^{-6}$ and A$_{\perp}$(Q$^2$ = \qsquaredaveragedhighII (GeV/c)$^2$) = (\experimentalasymme tryaluhighcorr $\pm$ \statisticalerrorhigh$_{\rm stat}$ $\pm$ \combinedsyspolerrorhighalucor$_{\rm sys}$) $\times$ 10$^{-6}$. The first errors denotes the statistical error and the second the systematic uncertainties. A$_\perp$ arises from the imaginary part of the two-photon exchange amplitude and is zero in the one-photon exchange approximation. From comparison with theoretical estimates of A$_\perp$ we conclude that $\pi$N-intermediate states give a substantial contribution to the imaginary part of the two-photon amplitude. The contribution from the ground state proton to the imaginary part of the two-photon exchange can be neglected. There is no obvious reason why this should be different for the real part of the two-photon amplitude, which enters into the radiative corrections for the Rosenbluth separation measurements of the electric form factor of the proton.