Machine Learning + Stories: Raising Machines to Dream

Have you ever wondered what happens when you teach a machine to build stories from images and then bring those stories to life on paper?

Imagine a dark room with an antique table. On top, a lamp illuminates an open book. The dim light casts long shadows, giving the scene a mysterious and almost magical atmosphere. As you approach the table, letters and images begin to float out of the book, transforming into lights and movements. A holographic display of stories unfolds before your eyes, dancing in the air with a life of their own.

In this enchanting room, where reality meets fantasy, we find ourselves lost in a world of tales that blur the lines between the real and the imagined. It’s as if Gabriel García Márquez himself had conjured this scene, where the boundaries of time and space dissolve, and the ordinary becomes extraordinary. The book is not just a book; it is a portal to countless worlds, each page a doorway to another adventure. The words lift off the pages and transform into shimmering figures, telling stories that swirl around us like ghosts in a dream.

Why are we talking about this now? Probably because Diana (my daughter) is in Switzerland. During our WhatsApp video calls, we play a game. We draw things from our day, share the stories behind them, and, of course, create new “stories.” This playful exchange has been our way of staying connected, our shared language of imagination and creativity.

Will Wright, the creator of The Sims, believes that we learn through play and storytelling. Stories are powerful tools for learning language, fostering imagination, and building empathy. These are essential skills we pass on to our children through bedtime stories, teaching them to dream, to understand, and to create.

Now, imagine raising a machine in the same way we raise a child. What if we could teach machines to dream? This thought sparked the idea for our prototype. What if a machine could see our drawings—using a camera or any other visual system—and start creating stories by itself? Picture a hat and a frog transforming into the Mad Hatter visiting a witch’s tea store to sample new flavors of tea. Whimsical, isn’t it?

For this to happen, machines need to be nurtured with imagination, fantasy, and dreams. They need to interpret a drawing of a hat from different perspectives, each angle adding a layer of rich narrative. It’s about teaching machines not just to see, but to imagine.

Here comes our prototype, a step towards that magical story-installation table from our imagination. It consists of a webcam capturing hand-drawn sketches and recognizing them. The computer runs a recognition system where we can teach what each drawing is, a machine learning system for children’s drawings. Our hands-on investigation is a part of the journey to create that magical scene, where reality blends with fantasy.

For this prototype, let’s gather our materials. Grab some paper and start drawing. I used plain white paper folded into a booklet and some cutout drawings. The first ingredient in our creative recipe is a webcam to capture the images. Connect it to a computer and process the images using Teachable Machine. Once the machine has learned from the images, transfer them to MadMapper Home to project them. And voilà! You can trigger different scenes just by showing different drawings.

Let’s follow Julio Cortázar’s playful spirit from “Instructions on How to Wind a Watch” and approach our task with a sense of wonder and curiosity. Imagine teaching a machine to weave stories from mere sketches. We draw, the machine learns, and together, we create.

Now, let’s end with a provocative twist. Imagine a future where machines learn to tell better stories, developing language, empathy, and imagination. And as we return to our dark room with the antique table, the figure approaching the book is revealed to be a robot. With a delicate motion, it tips its hat, inviting us into a world where the lines between human and machine storytellers blur.

Who else is excited to see what stories our machines can tell? Let’s dream together and expand the horizons of our (cyborg?) imaginations.

*

This video is in Portuguese, but here’s a translation of my delightful conversation with Paulina Cho:

Paulina: “I’m recording what you do on a sunny day.”

Lina: “Lina tells the machine a story to sleep…”

*

What did I learn from this?

• The Power of Playfulness in Learning: Through playful experimentation, I learned that creativity thrives when we approach problems with a sense of wonder and curiosity, much like how children play and learn.
• The Importance of Storytelling: Teaching a machine to recognize and create stories from drawings underscored the profound impact of storytelling. It’s a universal language that not only fosters empathy and imagination but also bridges the gap between technology and human experience.
• Hands-on Learning with Technology: Engaging directly with tools like Teachable Machine and MadMapper highlighted the significance of hands-on learning. By physically interacting with the technology, I gained a deeper understanding of how machine learning systems work and how they can be applied creatively.
• Collaborative Creativity: Working with Diana and integrating her drawings into the project reinforced the importance of collaboration. It showed me that combining different perspectives and skills can lead to richer and more innovative outcomes.
• Blending Reality with Fantasy: The process taught me how technology can be used to blur the lines between reality and fantasy, creating immersive and magical experiences. It’s a powerful reminder that the boundaries of our imagination are the only limits to what we can create.
• Iterative Process: The development of the prototype reminded me of the importance of iteration. Each trial and error brought new insights and improvements, demonstrating that the journey is just as important as the final outcome.
• I need a new notebook: You can hear the processing fan whirring away, and it sounds like it’s about to launch into orbit! 😂 Maybe it’s time to give the machine a rest. Zzz…

*

Here’s a glimpse of the prototype in action and below you can find a gallery with more photos and videos (Go to Flickr to play the videos, just click on top of the image).

Images and videos were made with the incredible help of [[Paulina Cho]]. 🫀🎥

[!warning] No machines were harmed in this episode.