Powered by GANs, the prototype AI tool translates sketches into an imaginary creature   

Drawing or sketching has always been a fun part of human life. Through sketching, we can give a shape of our imaginations and even emotions. Now the modern technology revolution is making it more exciting and enjoyable. To this effort, search engine giant Google brought an artificial intelligence tool that turns a sketch into imagery creatures. The tool, named Chimera Painter, developed by Big G, uses machine learning to generate images based on users’ rough sketches.

Powered by generative adversarial networks (GANs), the AI tool creates new content by pitting two neural networks against each other. A generator that creates new images and a discriminator that recognizes which of the designs have been artificially created. According to the Google AI blog post, Chimera Painter is a demo application that trained on thousands of computer-generated creature images, involving features and textures to a creature outline with a segmentation map, showing body part labels, such as “wings” or “claws,” when the user clicks the “transform” button.

When creating the machine learning model behind Chimera Painter, the developers faced some challenges. To develop artwork for a fictional fantasy card game, researchers created a digital card game prototype around the concept of coalescing creatures into new hybrids that can then battle each other. The game allows a player to play with cards of real-world animals and make them more powerful by combining them. Through this, developers presented a creative environment to demonstrate an image-generating model, as the number of possible chimeras demanded a method for quickly designing large volumes of artistic assets that could be combined naturally. This also retained identifiable visual characteristics of the original creatures.

To make high-quality creature card images guided by artist input, the team behind Chimera Painter experimented with GANs to create creature images that would be appropriate for the fantasy card game prototype. They also created a dataset of full-color images with single-species creature outlines adapted from 3D creature models to train the GANs. Each image paired with a segmentation map that divided the creatures’ body parts into anatomical parts like claws, snouts, legs, etc.

Once the model was trained, it was tasked with generating multi-species chimeras, based on outlines provided by artists. Developers then incorporated the best performing model into Chimera Painter that created single-species creatures, as well as the more complex multi-species chimeras.

Using GANs to create creatures, developers also found issues of loss of anatomical and spatial coherence when rendering subtle or low-contrast parts of images, despite being of high perceptual importance to humans. “It is our hope that these GAN models and the Chimera Painter demonstration tool might inspire others to think differently about their art pipeline,” the AI tool’s developers wrote in the blog post.

Moreover, Google is not alone in this landscape. Nvidia also built an AI prototype software that turns doodles into realistic landscapes. The software, which leverages GANs, provides users a smart paintbrush, so they can make a very basic outline of a scene before filling in their rough sketch with natural textures like grass, clouds, forests, or rocks.