Augmented video calls on mobile devices

We present a system and a method to segment the head-shoulder image of participants in a video call using mobile devices such as a smartphone or a tablet. Participants can choose to send only the segmented head-shoulder foreground image and overlay it on top of a static background image or a background video on the receiver side of the video call, as well as to replace the background of the caller himself/herself during the video call. Our proposed method extracts the head-shoulder area of each video frame based on detected face region, superpixel clustering, and efficient label propagation.