Recent studies suggest that we rapidly and effortlessly associate neutral information with the self, leading to subsequent prioritization of this information in perception. However, the exact underlying processes behind these effects are not fully known. Here, we focus specifically on top-down and bottom-up processes involved in self-prioritization, and report results from three experiments involving face detection, using a sequential match-non-match task. Across the three experiments we asked participants to associate an unfamiliar face with the self (Experiment 1), to associate one's face with a stranger's name (Experiment 2), and to establish both associations simultaneously (Experiment 3). We found that while participants showed evidence of bottom-up prioritization of their real faces, they did not show such an effect for self-associated strangers' faces. However, the participants showed a robust self-related top-down effect; when presented with a self-related cue, they were later faster at classifying both subsequent correct and incorrect targets. Together, our results suggest that self-prioritization is underpinned by distinct top-down and bottom-up processes. We discuss our findings in the context of the proposal that the self acts as an "integrative glue", and suggest an interpretation of our results within the framework of predictive coding.