Recently, the first subtype-selective allosteric modulators of the M5 muscarinic acetylcholine receptor (mAChR) have been described, but their molecular mechanisms of action remain unknown. Using radioligand-binding and functional assays of inositol phosphate (IP) accumulation and Ca2+ mobilization in a recombinant cell line stably expressing the human M5 mAChR, we investigated the effects of the positive allosteric modulator (PAM), ML380, and negative allosteric modulator, ML375. In functional assays, ML380 caused robust enhancements in the potency of the full agonists, acetylcholine (ACh), carbachol, and oxotremorine-M, while significantly increasing the maximal response to the partial agonist, pilocarpine. ML380 also demonstrated direct allosteric agonist activity. In contrast, ML375 displayed negative cooperativity with each of the agonists in a manner that varied with the pathway investigated and progressively reduced the maximal pilocarpine response. Radioligand-binding affinity cooperativity estimates were consistent with values derived from functional assays in some instances but not others, suggesting additional allosteric effects on orthosteric ligand efficacy. For ML375 this was confirmed in IP assays performed after reduction of receptor reserve by the alkylating agent, phenoxybenzamine, as it reduced the maximal ACh response. In contrast, ML380 enhanced only ACh potency after receptor alkylation, with no effect on maximal response, consistent with studies of the M1 mAChR with the prototypical PAM, BQZ12. Interaction studies between ML380 and ML375 also indicated that they most likely used an overlapping allosteric site. Our findings indicate that novel small-molecule modulators of the M5 mAChR display mixed mechanisms of action compared with previously characterized modulators of other mAChRs.