Artificial Intelligence (AI) techniques like optimisation and machine learning often present themselves as end-to-end solutions. Typically, an operator runs an AI model to obtain e.g., a forecast, schedule or decision, a numerically optimal solution that the user is expected to accept. However, often multiple solutions are close to optimal, and an operator might prefer one of these over the computed solution for reasons not encoded in the model. In other cases, the preferred solution might not be possible to achieve due to inconsistencies between the model and the real-world. Ideally, an AI system would take into account human preferences and automatically adjust provided solutions to match (or provide explanations when that is impossible /suboptimal). This PhD project aims to apply such AI-based human-in-the-loop techniques within a microgrid setting.