Non-covalent functionalization of graphene materials with responsive polymers is a promising approach for synthesizing new, hybrid composites with improved dispersibility and functional properties. However, the interplay between various components of the hybrid systems, their structural configurations, and stimuli-responsive behavior are not yet well understood at the atomic level. Here, we investigate the temperature-responsive behavior of physisorbed poly(N-isopropylacrylamide) (PNIPAM) on to graphene (G) and graphene oxide (GO) sheets in aqueous solution using large scale molecular dynamics simulations. It was observed that PNIPAM can be spontaneously anchored to the surfaces of both G and GO at 290 K with a macromolecular coil shape. However, the configuration of PNIPAM on G is markedly different in comparison with that on GO, leading to its distinct thermoresponsive behavior. Specifically, the adsorption on G gives rise to an increase in the temperature of the coil-to-globule transition when compared to the native polymer, the origin of which can be interpreted in terms of the interactions and the solvation behavior. The results obtained here are of significance to the design and manipulation of graphene-based stimuli-responsive hybrid systems with optimal functional properties.