We report semiconductor-based resistive switching nonvolatile memory devices with graphene oxide (GO) as an active layer which is sandwiched between aluminum (Al) metal and semiconductors such as Si and Ge. Semiconductors (p-Si or p-Ge) are used as bottom electrodes on which a layer of GO is deposited and Al electrodes are then formed on the top of it by thermal evaporation. From current-voltage characteristics, it is found that the devices show diode like rectifying switching behavior, which can suppress the cross talk between adjacent cells. In these structures, during initial voltage biasing, the current conduction is found to be due to thermionic emission and in later stages, it is driven by space charge. The maximum on/off ratio in Al/GO/p-Si and Al/GO/p-Ge structures is 110 (at -1.2 V) and 76 (at -1.7 V), respectively. However, breakdown occurs in the memory cells fabricated on p-Ge after switching to low resistance state due to lack of stable oxide at the interface between Ge and GO unlike in the cells on Si where stable native SiO2 prevents such breakdown. The mechanism of resistive switching in semiconductor based memory cells has been explained using X-ray photoelectron spectroscopy and capacitance-voltage characteristics.