Detections of gravitational waves emitted by binary black holes allow for tests of general relativity in the strong-field regime. In particular, deviations from general relativity can be observed by comparing incoming signals to waveform templates that include parametrized deviations from general relativity. However, it is essential that the general relativity sector of these templates accounts for all predictable physics. Otherwise, missing physics might be mimicked by the "beyond general relativity" sector of the templates, leading the analysis to report apparent deviations from general relativity. Current parametrized tests implement templates that omit physical phenomena such as orbital eccentricity and higher-order modes. In this paper, we show how the omission of higher modes can lead to false deviations from general relativity when these effects are strong enough. We study the extent of these deviations as a function of the mass ratio and the orbital orientation. We find that significant false deviations can arise when current tests are performed on signals emitted by asymmetric binaries whose orbital angular momentum is orthogonal to the line of sight. We estimate that the Advanced LIGO-Virgo network operating at its design sensitivity can observe false violations with a significance above 5σ as often as once per year. Similar results are expected for other tests of general relativity that compare the data to waveforms where some physical effects are omitted.