Topic Title: Computer Vision for Creativity


Technical Area: Computer Vision



Creative images play a key role for better user experience of internet applications. They are also one of the most important medias the advertisers can leverage to deliver business information to their customers. For e-commerce website like Taobao, the largest e-commerce platform in China, ads are displayed with human designed creative images to attract customers. However, advertisers design these images without considering user preferences of the designs. Furthermore, there are billions of items displayed and traded in Taobao. Due to the heavy human effort designing creatives for each item, only a very small part of items have corresponding advertising creatives. To resolve these problems, automatically compositing personalized creatives for each user is the emerging trend in creative technology development. Computer vision algorithms, especially high-quality image segmentation and natural composition with harmony lighting and scene effects, form the basis of an intelligent creative design system. These techniques might also help emerging applications such as live streaming, short videos and AR/VR.



To obtain a visually feasible result, the quality of segmentation is critical. However, current state-of-the-art segmentation methods do not fulfill the requirements. For example, the widely adopted large scale dataset COCO is coarsely labeled and lacks fine details, in which even the ground truth is not able to apply to creative design. And current models built for this task mainly focus on the main parts and hardly reveal the structural details. We strive to investigate a new segmentation method which is able to pull the fine structural details automatically.

Naturally compositing is also important for the final visual appearance of creatives and is a challenging task. It needs to take the lighting, scenes and color

effects into consideration. We propose to investigate a way to automatically adjust these factors and make them harmony to get an attractive and photo-realistic composition.


Related Research Topics

Semantic segmentation, Matting, Image editing