In current physical rehabilitation protocols, patients typically perform exercises with intermittent feedback or guidance following the initial demonstrations from the physiotherapist. Although many patient-centered systems have been developed for home rehabilitation, few systems have been developed to aid the physiotherapist as well as patients in the rehabilitation clinic. This article proposes the Automated Rehabilitation System (ARS), a system designed specifically for rehabilitation clinics using an iterative design process, developed with physiotherapists and patients in a knee and hip replacement clinic. ARS consists of body-worn inertial measurement units that continuously measure the patient’s pose. The measured pose is graphically represented as an animation and overlaid with the instructed motion on a visual display shown to the patient during exercise performance. ARS allows physiotherapists to quantitatively measure patient movement, assess recovery progress, and manage and schedule exercise regimens for patients. The system requirements and design requirements were derived through a focus group with 13 physiotherapists. For patients, ARS provides visual feedback and a novel exercise guidance feature to aid them while exercising. The patient interface was evaluated in a user study with 26 outpatients. The results show that performing the exercises with the visual guidance tool improves the quality of exercise performance.