SketchyScenes: Understanding Scene Sketches

We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Finally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its components to alternative choices.

Publications

teaser image of Language-based Colorization of Scene Sketches

Language-based Colorization of Scene Sketches

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.
Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization, interactive graphics, interactive perception, augmented communication
teaser image of SketchyScene: Richly-Annotated Scene Sketches

SketchyScene: Richly-Annotated Scene Sketches

European Conference on Computer Vision (ECCV), 2018.
Keywords: sketch dataset, scene sketch, sketch segmentation, interactive graphics

Videos

Language-based Colorization of Scene Sketches


LUCSS: Language-based User-customized Colourization of Scene Sketches


Talks

Cited By

  • Sketch-Based Creativity Support Tools Using Deep Learning. Artificial Intelligence for Human Computer Interaction: A Modern Approach.Forrest Huang, Eldon Schoop, David Ha, Jeffrey Nichols, and John Canny. source | cite | search
  • Emergent Graphical Conventions in a Visual Communication Game. https://arxiv.org/pdf/2111.14210.pdf.Shuwen Qiu, Sirui Xie, Lifeng Fan, Tao Gao, Song-Chun Zhu, and Yixin Zhu. source | cite | search
  • Write-an-Animation: High-level Text-based Animation Editing With Character-Scene Interaction. Computer Graphics Forum.Jia-Qi Zhang, Xiang Xu, Zhi-Meng Shen, Ze-Huan Huang, Yang Zhao, Yan-Pei Cao, Pengfei Wan, and Miao Wang. source | cite | search
  • One Sketch for All: One-Shot Personalized Sketch Segmentation. https://arxiv.org/pdf/2112.10838.pdf.Anran Qi, Yulia Gryaditskaya, Tao Xiang, and Yi-Zhe Song. source | cite | search
  • FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context. https://arxiv.org/pdf/2203.02113.pdf.Pinaki Nath Chowdhury, Aneeshan Sain, Yulia Gryaditskaya, Ayan Kumar Bhunia, Tao Xiang, and Yi-Zhe Song. source | cite | search
  • Delaunay Painting: Perceptual Image Colouring From Raster Contours With Gaps. Computer Graphics Forum.Amal Dev Parakkat, Pooran Memari, and Marie-Paule Cani. source | cite | search
  • SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs. IEEE Transactions on Image Processing.Fang Liu, Xiaoming Deng, Changqing Zou, Yu-Kun Lai, Keqi Chen, Ran Zuo, Cuixia Ma, Yong-Jin Liu, and Hongan Wang. source | cite | search
  • A Review of Image and Video Colorization: From Analogies to Deep Learning. Visual Informatics.Shu-Yu Chen, Jia-Qi Zhang, You-You Zhao, Paul L. Rosin, Yu-Kun Lai, and Lin Gao. source | cite | search
  • Exemplar-Based Sketch Colorization With Cross-Domain Dense Semantic Correspondence. Mathematics.Jinrong Cui, Haowei Zhong, Hailong Liu, and Yulu Fu. source | cite | search
  • Neural Image Recolorization for Creative Domains. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Boyi Li, Serge Belongie, Ser-nam Lim, and Abe Davis. source | cite | search
  • SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition. ACM Transactions on Interactive Intelligent Systems.Fang Liu, Xiaoming Deng, Jiancheng Song, Yu-Kun Lai, Yong-Jin Liu, Hao Wang, Cuixia Ma, Shengfeng Qin, and Hongan Wang. source | cite | search
  • Modulating Bottom-Up and Top-Down Visual Processing Via Language-Conditional Filters. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Ilker Kesen, Ozan Arkan Can, Erkut Erdem, Aykut Erdem, and Deniz Yuret. source | cite | search
  • Unsupervised Scene Sketch to Photo Synthesis. arXiv.2209.02834.Jiayun Wang, Sangryul Jeon, Stella X. Yu, Xi Zhang, Himanshu Arora, and Yu Lou. source | cite | search
  • Cartoon Image Processing: A Survey. International Journal of Computer Vision.Yang Zhao, Diya Ren, Yuan Chen, Wei Jia, Ronggang Wang, and Xiaoping Liu. source | cite | search
  • DrawMon: A Distributed System for Detection of Atypical Sketch Content in Concurrent Pictionary Games. Proceedings of the 30th ACM International Conference on Multimedia.Nikhil Bansal, Kartik Gupta, Kiruthika Kannan, Sivani Pentapati, and Ravi Kiran Sarvadevabhatla. source | cite | search
  • Generating Compositional Color Representations From Text. https://arxiv.org/abs/2109.10477.Paridhi Maheshwari, Nihal Jain, Praneetha Vaddamanu, Dhananjay Raut, Shraiysh Vaishay, and Vishwa Vinay. source | cite | search
  • Painting Style-Aware Manga Colorization Based on Generative Adversarial Networks. 2021 IEEE International Conference on Image Processing (ICIP).Yugo Shimizu, Ryosuke Furuta, Delong Ouyang, Yukinobu Taniguchi, Ryota Hinami, and Shonosuke Ishiwatari. source | cite | search
  • Adversarial Segmentation Loss for Sketch Colorization. 2021 IEEE International Conference on Image Processing (ICIP).Samet Hicsonmez, Nermin Samet, Emre Akbas, and Pinar Duygulu. source | cite | search
  • Focusing on Persons. 4.Xin Jin, Zhonglan Li, Ke Liu, Dongqing Zou, Xiaodong Li, Xingfan Zhu, Ziyin Zhou, Qilong Sun, and Qingyu Liu. source | cite | search
  • Sketchy Scene Captioning: Learning Multi-level Semantic Information From Sparse Visual Scene Cues. Lecture Notes in Computer Science.Lian Zhou, Yangdong Chen, and Yuejie Zhang. source | cite | search
  • Multi-style Chinese Art Painting Generation of Flowers. IET Image Processing.Feifei Fu, Jiancheng Lv, Chenwei Tang, and Mao Li. source | cite | search
  • TediGAN: Text-Guided Diverse Face Image Generation and Manipulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. source | cite | search
  • DanbooRegion: An Illustration Region Dataset. Computer Vision –ECCV 2020.Lvmin Zhang, Yi Ji, and Chunping Liu. source | cite | search
  • SketchyDepth: From Scene Sketches to RGB-D Images. ICCV 2021.Gianluca Berardi, Samuele Salti, and Luigi Di Stefano. source | cite | search
  • Generative Adversarial Networks\textendashEnabled Human\textendashArtificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends. Frontiers in Artificial Intelligence.Rowan T. Hughes, Liming Zhu, and Tomasz Bednarz. source | cite | search
  • XCI-Sketch: Extraction of Color Information From Images for Generation of Colored Outlines and Sketche. https://arxiv.org/abs/2108.11554.Harsh Rathod, Manisimha Varma, Parna Chowdhury, Sameer Saxena, V. Manushree, Ankita Ghosh, and Sahil Khose. source | cite | search
  • Generating Compositional Color Representations From Text. https://arxiv.org/abs/2109.10477.Paridhi Maheshwari, Nihal Jain, Praneetha Vaddamanu, Dhananjay Raut, Shraiysh Vaishay, and Vishwa Vinay. source | cite | search
  • Text As Neural Operator:Image Manipulation by Text Instruction. MM '21: Proceedings of the 29th ACM International Conference on Multimedia.Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, and Irfan Essa. source | cite | search
  • DLA-Net for FG-SBIR. 5.Jiaqing Xu, Haifeng Sun, Qi Qi, Jingyu Wang, Ce Ge, Lejian Zhang, and Jianxin Liao. source | cite | search
  • Grayscale Image Colorization Using a Convolutional Neural Network. Journal of the Korean Society for Industrial and Applied Mathematics.Minje Jwa and Myungjoo Kang. source | cite | search
  • Exploring Local Detail Perception for Scene Sketch Semantic Segmentation. IEEE Transactions on Image Processing.Ce Ge, Haifeng Sun, Yi-Zhe Song, Zhanyu Ma, and Jianxin Liao. source | cite | search
  • Partially Does It: Towards Scene-Level FG-SBIR With Partial Input. arXiv.2203.14804.Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Viswanatha Reddy Gajjala, Aneeshan Sain, Tao Xiang, and Yi-Zhe Song. source | cite | search
  • ColorizeDiffusion: Adjustable Sketch Colorization With Reference Image and Text. arXiv.2401.01456.Dingkun Yan, Liang Yuan, Yuma Nishioka, Issei Fujishiro, and Suguru Saito. source | cite | search
  • OpenSketch: A Richly-annotated Dataset of Product Design Sketches. ACM Transactions on Graphics.Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Frédo Dur, and Adrien Bousseau. source | cite | search
  • Language-based Photo Color Adjustment for Graphic Designs. ACM Transactions on Graphics.Zhenwei Wang, Nanxuan Zhao, Gerhard Hancke, and Rynson W.H. Lau. source | cite | search
  • Semi-supervised Reference-based Sketch Extraction Using a Contrastive Learning Framework. ACM Transactions on Graphics.Chang Wook Seo, Amirsaman Ashtari, and Junyong Noh. website, source | cite | search
  • Stroke-based Semantic Segmentation for Scene-level Free-hand Sketches. The Visual Computer.Zhengming Zhang, Xiaoming Deng, Jinyao Li, Yukun Lai, Cuixia Ma, Yongjin Liu, and Hongan Wang. source | cite | search
  • Deep Learning for Studying Drawing Behavior: A Review. Frontiers in Psychology.Benjamin Beltzung, Marie Pel{\'{e}}, Julien P. Renoult, and C{\'{e}}dric Sueur. source | cite | search
  • Human\textendashmachine Hybrid Intelligence for the Generation of Car Frontal Forms. Advanced Engineering Informatics.Yu Wu, Lisha Ma, Xiaofang Yuan, and Qingnan Li. source | cite | search
  • SVCNet: Scribble-based Video Colorization Network With Temporal Aggregation. arXiv.2303.11591.Yuzhi Zhao, Lai-Man Po, Kangcheng Liu, Xuehui Wang, Wing-Yin Yu, Pengfei Xian, Yujia Zhang, and Mengyang Liu. source | cite | search
  • Adversarial Interactive Cartoon Sketch Colourization With Texture Constraint and Auxiliary Auto-Encoder. Computer Graphics Forum.Xiaoyu Liu, Shaoqiang Zhu, Yao Zeng, and Junsong Zhang. source | cite | search
  • Fine-Grained Video Retrieval With Scene Sketches. IEEE Transactions on Image Processing.Ran Zuo, Xiaoming Deng, Keqi Chen, Zhengming Zhang, Yu-Kun Lai, Fang Liu, Cuixia Ma, Hao Wang, Yong-Jin Liu, and Hongan Wang. source | cite | search
  • A State-of-Art Review on~Intelligent Systems for~Drawing Assisting. Lecture Notes in Computer Science.Juexiao Qin, Xiaohua Sun, and Weijian Xu. source | cite | search
  • Improving Sketch Colorization Using Adversarial Segmentation Consistency. arXiv.2301.08590.Samet Hicsonmez, Nermin Samet, Emre Akbas, and Pinar Duygulu. source | cite | search
  • CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing. arXiv.2402.17624.Chufeng Xiao and Hongbo Fu. source | cite | search
  • Stay In Touch