In this article, we consider semiparametric estimation in a partially linear single-index panel data model with fixed effects. Without taking the difference explicitly, we propose using a semiparametric minimum average variance estimation (SMAVE) based on a dummy variable method to remove the fixed effects and obtain consistent estimators for both the parameters and the unknown link function. As both the cross-section size and the time series length tend to infinity, we not only establish an asymptotically normal distribution for the estimators of the parameters in the single index and the linear component of the model, but also obtain an asymptotically normal distribution for the nonparametric local linear estimator of the unknown link function. The asymptotically normal distributions of the proposed estimators are similar to those obtained in the random effects case. In addition, we study several partially linear single-index dynamic panel data models. The methods and results are augmented by simulation studies and illustrated by application to two real data examples. This article has online supplementary materials.