During the past twenty years, there has been a rapid growth in life expectancy and an increased attention on funding for old age. Attempts to forecast improving life expectancy have been boosted by the development of stochastic mortality modeling, for example the Cairns-Blake-Dowd (CBD) 2006 model. The most common optimization method for these models is maximum likelihood estimation (MLE) which relies on the assumption that the number of deaths follows a Poisson distribution. However, several recent studies have found that the true underlying distribution of death data is overdispersed in nature (see Cairns etal. 2009 and Dowd etal. 2010). Semiparametric models have been applied to many areas in economics but there are very few applications of such models in mortality modeling. In this paper we propose a local linear panel fitting methodology to the CBD model which would free the Poisson assumption on number of deaths. The parameters in the CBD model will be considered as smooth functions of time instead of being treated as a bivariate random walk with drift process in the current literature. Using the mortality data of several developed countries, we find that the proposed estimation methods provide comparable fitting results with the MLE method but without the need of additional assumptions on number of deaths. Further, the 5-year-ahead forecasting results show that our method significantly improves the accuracy of the forecast.