SmartPaint: a co-creative drawing system based on generative adversarial networks

Lingyun SUN; Pei CHEN; Wei XIANG; Peng CHEN; Wei-yue GAO; Ke-jun ZHANG

doi:10.1631/FITEE.1900386

Your Location：

Home >

Browse articles >

SmartPaint: a co-creative drawing system based on generative adversarial networks

Special Feature on Intelligent Design | Updated：2022-05-19

- SmartPaint: a co-creative drawing system based on generative adversarial networks
- SmartPaint:一种基于生成式对抗神经网络的机协同绘画系统
- Frontiers of Information Technology & Electronic Engineering Vol. 20, Issue 12, Pages: 1644-1656(2019)
- Affiliations：
  
  Key Laboratory of Design Intelligence and Digital Creativity of Zhejiang Province, Hangzhou 310027, China
  State Key Lab of CAD & CG, Zhejiang University, Hangzhou 310058, China
  Alibaba-Zhejiang University Joint Institute of Frontier Technologies, Hangzhou 310027, China
- Author bio：
  
  [ "Lingyun SUN, E-mail: sunly@zju.edu.cn" ]
  [ "Pei CHEN, chenpei@zju.edu.cn" ]
  Wei XIANG, wxiang@zju.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900386
  CLC： TP391
- Received：30 July 2019，
  
  Revised：2019-12-24，
  
  Published：2019-12
- Accepted：
Scan QR Code
Lingyun SUN, Pei CHEN, Wei XIANG, et al. SmartPaint: a co-creative drawing system based on generative adversarial networks[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(12): 1644-1656.
DOI：

Lingyun SUN, Pei CHEN, Wei XIANG, et al. SmartPaint: a co-creative drawing system based on generative adversarial networks[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(12): 1644-1656. DOI： 10.1631/FITEE.1900386.

摘要

Abstract

Artificial intelligence (AI) has played a significant role in imitating and producing large-scale designs such as e-commerce banners. However

it is less successful at creative and collaborative design outputs. Most humans express their ideas as rough sketches

and lack the professional skills to complete pleasing paintings. Existing AI approaches have failed to convert varied user sketches into artistically beautiful paintings while preserving their semantic concepts. To bridge this gap

we have developed SmartPaint

a co-creative drawing system based on generative adversarial networks (GANs)

enabling a machine and a human being to collaborate in cartoon landscape painting. SmartPaint trains a GAN using triples of cartoon images

their corresponding semantic label maps

and edge detection maps. The machine can then simultaneously understand the cartoon style and semantics

along with the spatial relationships among the objects in the landscape images. The trained system receives a sketch as a semantic label map input

and automatically synthesizes its edge map for stable handling of varied sketches. It then outputs a creative and fine painting with the appropriate style corresponding to the human's sketch. Experiments confirmed that the proposed SmartPaint system successfully generates high-quality cartoon paintings.

关键词

Keywords

references

G Atarsaikhan , , , BK Iwana , , , A Narusawa , , , 等 . . Neural font style transfer . . Proc 14 th IAPR Int Conf on Document Analysis and Recognition , , 2017 . . p.51 - - 56 . . DOI: 10.1109/ICDAR.2017.328 http://doi.org/10.1109/ICDAR.2017.328 . .

S Belongie , , , J Malik , , , J Puzicha . . Shape context: a new descriptor for shape matching and object recognition . . Proc 13 th Int Conf on Neural Information Processing Systems , , 2001 . . p.798 - - 804 . . . .

L Benedetti , , , H Winnemöller , , , M Corsini , , , 等 . . Painting with Bob: assisted creativity for novices . . Proc 27 th Annual ACM Symp on User Interface Software and Technology , , 2014 . . p.419 - - 428 . . DOI: 10.1145/2642918.2647415 http://doi.org/10.1145/2642918.2647415 . .

SR Bowman , , , L Vilnis , , , O Vinyals , , , 等 . . Generating sentences from a continuous space . . Proc 20 th SIGNLL Conf on Computational Natural Language Learning , , 2016 . . p.10 - - 21 . . DOI: 10.18653/v1/K16-1002 http://doi.org/10.18653/v1/K16-1002 . .

J Canny . . A computational approach to edge detection . . In: Fischler MA, Firschein O (Eds.), Readings in Computer Vision: Issues, Problem, Principles, and Paradigms. Elsevier, Amsterdam , , 1987 . . p.184 - - 203 . . DOI: 10.1016/B978-0-08-051581-6.50024-6 http://doi.org/10.1016/B978-0-08-051581-6.50024-6 . .

AJ Champandard . . Semantic style transfer and turning . . two-bit doodles into fine artworks , , 2016 . . https://arxiv.org/abs/1603.01768 https://arxiv.org/abs/1603.01768 , , . .

C Chen , , , JC Lin , , , MH Liao , , , 等 . . Learning to detect salient curves of cartoon images based on composition rules . . Proc 11 th Int Conf on Computer Science & Education , , 2016 . . p.808 - - 813 . . DOI: 10.1109/ICCSE.2016.7581686 http://doi.org/10.1109/ICCSE.2016.7581686 . .

T Chen , , , MM Cheng , , , P Tan , , , 等 . . Sketch2Photo: Internet image montage . . ACM Trans Graph , , 2009 . . 28 ( ( 5 ): ): Article 124 DOI: 10.1145/1618452.1618470 http://doi.org/10.1145/1618452.1618470 . .

Y Chen , , , YK Lai , , , YJ Liu . . CartoonGAN: generative adversarial networks for photo cartoonization . . IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.9465 - - 9474 . . DOI: 10.1109/CVPR.2018.00986 http://doi.org/10.1109/CVPR.2018.00986 . .

NSH Chu , , , CL Tai . . Real-time painting with an expressive virtual Chinese brush . . IEEE Comput Graph Appl , , 2004 . . 24 ( ( 5 ): ): 76 - - 85 . . DOI: 10.1109/MCG.2004.37 http://doi.org/10.1109/MCG.2004.37 . .

YZ Ci , , , XZ Ma , , , ZH Wang , , , 等 . . User-guided deep anime line art colorization with conditional adversarial networks . . Proc 26 th ACM Int Conf on Multimedia , , 2018 . . p.1536 - - 1544 . . DOI: 10.1145/3240508.3240661 http://doi.org/10.1145/3240508.3240661 . .

D Cummmings , , , F Vides , , , T Hammond . . I donot believe my eyes . . Geometric sketch recognition for a computer art tutorial. Proc Int Symp on Sketch-Based Interfaces and Modeling , , 2012 . . p.97 - - 106 . . DOI: 10.2312/SBM/SBM12/097-106 http://doi.org/10.2312/SBM/SBM12/097-106 . .

NM Davis . . Human-computer co-creativity: blending human and computational creativity . . Proc 9 th Artificial Intelligence and Interactive Digital Entertainment Conf , , 2013 . . 9 - - 12 . . . .

NM Davis , , , CP Hsiao , , , KY Singh , , , 等 . . Drawing apprentice: an enactive co-creative agent for artistic collaboration . . Proc ACM SIGCHI Conf on Creativity and Cognition , , 2015 . . p.185 - - 186 . . DOI: 10.1145/2757226.2764555 http://doi.org/10.1145/2757226.2764555 . .

NM Davis , , , CP Hsiao , , , KY Singh , , , 等 . . Co-creative drawing agent with object recognition . . Proc 12th Artificial Intelligence and Interactive Digital Entertainment Conf , , 2016a . . p.9 - - 15 . . . .

NM Davis , , , CP Hsiao , , , K Yashraj Singh , , , 等 . . Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent . . Proc 21 st Int Conf on Intelligent User Interfaces , , 2016b . . p.196 - - 207 . . DOI: 10.1145/2856767.2856795 http://doi.org/10.1145/2856767.2856795 . .

T Dekel , , , C Gan , , , D Krishnan , , , 等 . . Sparse, smart contours to represent and edit images . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.3511 - - 3520 . . DOI: 10.1109/CVPR.2018.00370 http://doi.org/10.1109/CVPR.2018.00370 . .

D Dixon , , , M Prasad , , , T Hammond . . iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces . . Proc SIGCHI Conf on Human Factors in Computing Systems , , 2010 . . p.897 - - 906 . . DOI: 10.1145/1753326.1753459 http://doi.org/10.1145/1753326.1753459 . .

M Eitz , , , R Richter , , , K Hildebrand , , , 等 . . Photosketcher: interactive sketch-based image synthesis . . IEEE Comput Graph Appl , , 2011 . . 31 ( ( 6 ): ): 56 - - 66 . . DOI: 10.1109/MCG.2011.67 http://doi.org/10.1109/MCG.2011.67 . .

LA Gatys , , , AS Ecker , , , M Bethge . . Image style transfer using convolutional neural networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.2414 - - 2423 . . DOI: 10.1109/CVPR.2016.265 http://doi.org/10.1109/CVPR.2016.265 . .

I Goodfellow , , , J Pouget-Abadie , , , M Mirza , , , 等 . . Generative adversarial nets . . Proc 27th Int Conf on Neural Information Processing Systems , , 2014 . . p.2672 - - 2680 . . . .

Y Güçlütürk , , , U Güçlü , , , R van Lier , , , 等 . . Convolutional sketch inversion. European Conf on Computer Vision , , 2016 . . p.810 - - 824 . . DOI: 10.1007/978-3-319-46604-0_56 http://doi.org/10.1007/978-3-319-46604-0_56 . .

D Ha , , , D Eck . . A neural representation of sketch drawings , , 2017 . . https://arxiv.org/abs/1704.03477 https://arxiv.org/abs/1704.03477 , , . .

HZ Huang , , , H Wang , , , WH Luo , , , 等 . . Real-time neural style transfer for videos . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.783 - - 791 . . DOI: 10.1109/CVPR.2017.745 http://doi.org/10.1109/CVPR.2017.745 . .

P Isola , , , JY Zhu , , , TH Zhou , , , 等 . . Image-to-image translation with conditional adversarial networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.1125 - - 1134 . . DOI: 10.1109/CVPR.2017.632 http://doi.org/10.1109/CVPR.2017.632 . .

P Karimi , , , N Davis , , , K Grace , , , 等 . . Deep learning for identifying potential conceptual . . shifts for co-creative drawing , , 2018 . . https://arxiv.org/abs/1801.00723 https://arxiv.org/abs/1801.00723 , , . .

YJ Lee , , , CL Zitnick , , , MF Cohen . . ShadowDraw: realtime user guidance for freehand drawing . . ACM Trans Graph , , 2011 . . 30 ( ( 4 ): ): Article 27 DOI: 10.1145/2010324.1964922 http://doi.org/10.1145/2010324.1964922 . .

MJ Li , , , HZ Huang , , , L Ma , , , 等 . . Unsupervised image-toimage translation with stacked cycle-consistent adversarial networks . . Proc 15 th European Conf on Computer Vision , , 2018 . . p.186 - - 201 . . DOI: 10.1007/978-3-030-01240-3_12 http://doi.org/10.1007/978-3-030-01240-3_12 . .

YF Liu , , , ZC Qin , , , T Wan , , , 等 . . Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks . . Neurocomputing , , 2018 . . 311 78 - - 87 . . DOI: 10.1016/j.neucom.2018.05.045 http://doi.org/10.1016/j.neucom.2018.05.045 . .

FJ Luan , , , S Paris , , , E Shechtman , , , 等 . . Deep photo style transfer . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.4990 - - 4998 . . DOI: 10.1109/CVPR.2017.740 http://doi.org/10.1109/CVPR.2017.740 . .

M Mirza , , , S Osindero . . Conditional generative . . adversarial nets , , 2014 . . https://arxiv.org/abs/1411.1784 https://arxiv.org/abs/1411.1784 , , . .

X Ning , , , H Laga , , , S Saito , , , 等 . . Contour-driven Sumi-e rendering of real photos . . Comput Graph , , 2011 . . 35 ( ( 1 ): ): 122 - - 134 . . DOI: 10.1016/j.cag.2010.11.017 http://doi.org/10.1016/j.cag.2010.11.017 . .

C Oh , , , J Song , , , J Choi , , , 等 . . I lead, you help but only with enough details: understanding user experience of co-creation with artificial intelligence . . Proc CHI Conf on Human Factors in Computing Systems, Article , , 2018 . . Article 649 DOI: 10.1145/3173574.3174223 http://doi.org/10.1145/3173574.3174223 . .

T Portenier , , , QY Hu , , , A SzabȮ , , , 等 . . Faceshop: deep sketch-based face image editing . . ACM Trans Graph , , 2018 . . 37 ( ( 4 ): ): Article 99 DOI: 10.1145/3197517.3201393 http://doi.org/10.1145/3197517.3201393 . .

A Roberts , , , J Engel , , , D Eck . . Hierarchical variational autoencoders for music . . Workshop on Machine Learning for Creativity and Design, NIPS , , 2017 . . .

P Sangkloy , , , JW Lu , , , C Fang , , , 等 . . Scribbler: controlling deep image synthesis with sketch and color . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.5400 - - 5409 . . DOI: 10.1109/cvpr.2017.723 http://doi.org/10.1109/cvpr.2017.723 . .

A Selim , , , M Elgharib , , , L Doyle . . Painting style transfer for head portraits using convolutional neural networks . . ACM Trans Graph , , 2016 . . 35 ( ( 4 ): ): Article 129 DOI: 10.1145/2897824.2925968 http://doi.org/10.1145/2897824.2925968 . .

E Simo-Serra , , , S Iizuka , , , K Sasaki , , , 等 . . Learning to simplify: fully convolutional networks for rough sketch cleanup . . ACM Trans Graph , , 2016 . . 35 ( ( 4 ): ): Article 121 DOI: 10.1145/2897824.2925972 http://doi.org/10.1145/2897824.2925972 . .

TC Wang , , , MY Liu , , , JY Zhu , , , 等 . . High-resolution image synthesis and semantic manipulation with conditional GANs . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . 8798 - - 8807 . . DOI: 10.1109/CVPR.2018.00917 http://doi.org/10.1109/CVPR.2018.00917 . .

WQ Xian , , , P Sangkloy , , , V Agrawal , , , 等 . . TextureGAN: controlling deep image synthesis with texture patches . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.8456 - - 8465 . . DOI: 10.1109/cvpr.2018.00882 http://doi.org/10.1109/cvpr.2018.00882 . .

YK Zhang , , , KK Hu , , , PR Ren , , , 等 . . Layout style modeling for automating banner design . . Proc Thematic Workshops of ACM Multimedia , , 2017 . . p.451 - - 459 . . DOI: 10.1145/3126686.3126718 http://doi.org/10.1145/3126686.3126718 . .

NX Zhao , , , Y Cao , , , RWH Lau . . What characterizes personalities of graphic designs . . ACM Trans Graph , , 2018 . . 37 ( ( 4 ): ): Article 116 DOI: 10.1145/3197517.3201355 http://doi.org/10.1145/3197517.3201355 . .

JY Zhu , , , P Krähenbȹhl , , , E Shechtman , , , 等 . . Generative visual manipulation on the natural image manifold . . Proc 14th European Conf on Computer Vision , , 2016 . . p.597 - - 613 . . DOI: 10.1007/978-3-319-46454-1_36 http://doi.org/10.1007/978-3-319-46454-1_36 . .

JY Zhu , , , T Park , , , P Isola , , , 等 . . Unpaired imageto-image translation using cycle-consistent adversarial networks . . Proc IEEE Int Conf on Computer Vision , , 2017 . . p.2223 - - 2232 . . DOI: 10.1109/ICCV.2017.244 http://doi.org/10.1109/ICCV.2017.244 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

Jingqi Li

Yuan Cao

Zilong Li

Yiming Lei

Dongsheng Li

Linbo Qiao

Hao Zheng

Related Institution

Shanghai Center for Brain Science and Brain-Inspired Technology

MOE Frontiers Center for Brain Science, Fudan University

Institute of Science and Technology for Brain-inspired Intelligence, Fudan University

Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University

National University of Defense Technology

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰