Using AI Generated Images In Games [Discussion / Devlog]

Showcases
#1

Hello everyone! Hope you had a Merry Christmas and a happy holiday.

Earlier this year my friends and I came across AI image generators. While extremely impressive technically, I wondered if it had a practical application for game development. We settled on remaking Super Mario Bros Level 1-1 as a test of the limits for this AI.

We generated images and quickly learned the AI struggles with faces, limbs, and anything vaguely human. Here are some of the funnier images the AI generated for us:

chick_wing_mario pringles_mario cursed_one_up bowser

More concrete objects like clothing, bricks, etc seemed to be fine, I’d argue that this is where the
The most interesting (and where I think this AI shines) was with ground / rock textures and fire.

blue_cobblestone brown_brick perfect_fireball

That’s the quick version of everything. If you’d like to see the full process (and more hilarious / nightmare art) then here’s a video on it:

My conclusions are:

  • This tech is extremely impressive BUT…
  • The art generated come in many different styles and it’s hard to get consistency.
  • Additional work is needed to get the art into a form that can be used in game.
  • Tools like this maybe good for background art, textures, and prototyping.
  • Ultimately, I’m of the opinion that tools like this aren’t going to replace artists, and I don’t think they will for some time (if ever).

What are your thoughts on AI image generators?
Have you tried messing around with them at all?
I’d love to hear your thoughts!

5 Likes
Creating A Five Nights At Freddy's Style Game In Defold
#2

I tinkered a little bit with most of the popular AIs. My master thesis was about artificial neural networks and it was soo cool :smiley:
Nowadays, I see a lot of progress in AI I can’t even comprehend, its capabilities and everyday growing and it’s definitely something revolutionary.

For the image generators - there is a lot of work that can be done to polish the images, but free versions are limited - the best results you can get when polishing one output of a very good prompt - in paid Midjourney or your own hosted version of Stable Diffusion (or online Dreamstudio at least). People are able to make amazing things there. Basic prompts and usage of the free versions of AI is like you experienced - unrealistic and you can easily see it was generated.

My latest discovery - ChatGPT is something amazing too. I remember chatting with Cleverbot, but it was immensely frustrating it didn’t remembered the context of the conversation, like I would talk with someone with no short-term memory. And ChatGPT surprised me with this and even a long conversation has so much sense. And it knows a lot - like I learned a lot about AI and e.g. quantum physics by asking ChatGPT and asking for explanations. It’s few levels higher than using Google and I believe it is the future.

3 Likes
#3

No kidding! This still is beyond me, I wouldn’t even know where to start programming something like that.

I do think part of the charm (and many good laughs) came from the absurd things the AI generated. I might try to do something like this again in the future using one of the paid AIs. For now I have other projects lined up.

Yeah, I’ve seen some of what that AI can do. I recorded my video back in September but only was motivated to finish it by people making videos of generating (almost) entire games using ChatGPT.

2 Likes
#4

At the end of this year I also decided to get a little insight into what I could get with new technology and this is what I got.

Comic book style

Here I used a dataset of certain cartoons to teach the model, and the result is some pretty good stuff.

Tom Gauld_Tom Gauld, comic book style, fine art, sunny, Meadows of red poppies, snow-capped mountains in the background_image-2_1672313067
Tom Gauld_Tom Gauld, comic book style, fine art, sunny, Meadows of red poppies, snow-capped mountains in the background_image-2_1672313067512×512 146 KB
Tom Gauld_Tom Gauld, An old goblin in the coat, vampires around, Deep night_image-0_1672072582
Tom Gauld_Tom Gauld, An old goblin in the coat, vampires around, Deep night_image-0_1672072582512×512 522 KB
Tom Gauld_Tom Gauld, comic book style, fine lining, trending on Artstation, A soviet train on the railways, The sunny summer forest_image-7_1672003952
Tom Gauld_Tom Gauld, comic book style, fine lining, trending on Artstation, A soviet train on the railways, The sunny summer forest_image-7_1672003952512×512 125 KB
Tom Gauld_Tom Gauld, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, snowy, windy, close-up portrait, An yeti with a frightened face_image-6_1672002887
Tom Gauld_Tom Gauld, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, snowy, windy, close-up portrait, An yeti with a frightened face_image-6_1672002887512×512 493 KB
Tom Gauld_Tom Gauld, A snowman with a frightened face, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, portrait, snowy, windy_image-3_1672002232
Tom Gauld_Tom Gauld, A snowman with a frightened face, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, portrait, snowy, windy_image-3_1672002232512×512 513 KB
Tom Gauld_Tom Gauld, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, snowy, windy, close-up portrait, An yeti with a frightened face_image-1_1672002887
Tom Gauld_Tom Gauld, A winter night forest in the background, comic book style, fine lining, trending on Artstation, blue lighting, snowy, windy, close-up portrait, An yeti with a frightened face_image-1_1672002887512×512 549 KB

Tetromino shapes.

It was a longer process here.

I started by training the model on pictures filled with tetromino shapes and then generated something with aztec patterns:

IMAGE 2022-12-31 17:14:35
IMAGE 2022-12-31 17:14:35512×512 45.8 KB
IMAGE 2022-12-31 17:14:33
IMAGE 2022-12-31 17:14:33512×512 65.1 KB
IMAGE 2022-12-31 17:14:32
IMAGE 2022-12-31 17:14:32512×512 56.9 KB
IMAGE 2022-12-31 17:14:31
IMAGE 2022-12-31 17:14:31512×512 68.9 KB

Then I manually cut out the results that interested me and made collages from them:

IMAGE 2022-12-31 17:14:38
IMAGE 2022-12-31 17:14:381280×1070 234 KB

Manually sketched out the inaccuracies:

Screenshot 2022-12-31 at 17.14.20
Screenshot 2022-12-31 at 17.14.20898×1572 178 KB
IMAGE 2022-12-31 17:14:42
IMAGE 2022-12-31 17:14:421070×900 86.5 KB

And sent them back to generator:

1
1512×512 220 KB
5
5512×512 236 KB
705801001_a_set_of_tetromino_shapes_made_from_stone__aztec_patterns__bright_colors__very_detailed__hq__8k__anc
705801001_a_set_of_tetromino_shapes_made_from_stone__aztec_patterns__bright_colors__very_detailed__hq__8k__anc512×512 266 KB
2892996577_a_set_of_tetromino_shapes_made_from_wood__plants_and_grass_on_the_corners__aztec_patterns__bright_co
2892996577_a_set_of_tetromino_shapes_made_from_wood__plants_and_grass_on_the_corners__aztec_patterns__bright_co512×512 248 KB

In the end, I couldn’t achieve the desired result with more casual patterns and I abandoned the idea for now due to lack of time. But… it’s interesting, anyway, I’ll get back to it next year.

Merry Christmas and Happy Holidays! :christmas_tree:

8 Likes
#5

Yeah, something I’ve noticed is that the current AI tools seem really good at organic drawings but not great at geometric shapes, pixel art, or anything else with perfectly hard edges. But for those drawings you can still get reasonably close and then use them as reference images for your own drawings.

#6

I freaking love this, it’s hilarious!

1 Like