This paper proposes a new instrumental variables approach for consistent and asymptotically efficient estimation of panel data models with weakly exogenous or endogenous regressors and residuals generated by a multi-factor error structure. In this case, the standard dynamic panel estimators fail to provide consistent estimates of the parameters. The novelty of our approach is that we introduce new parameters to represent the unobserved covariances between the instruments and the factor component of the residual; these parameters are estimable when N is large. Some important estimation and identification issues are studied in detail. The finite sample performance of the proposed estimators is investigated using simulated data. The results show that the method produces reliable estimates of the parameters over several parameterisations.