Category Archives: AI

Turning stories into AI videos

Leave a reply

This is an update on my attempt to turn my story Thin Red Lines into a video. The experience was both more rewarding and more challenging than I thought. I am linking to the six-minute video of Thin Red Lines but it is not great. It tells the story but looks like AI slop, something which I did not want to happen.  

However, I did learn a few things along the way, some of which pertain to me having a better understanding of the story itself, which I think you will find interesting.

The Tools

There are a lot of AI tools out there for creating video. It seems like Runway, Higgsfield and Open Art are some of the more common ones. There can be quite a bit of a cost to this, so I decided to stick with the most tools I already have paid subscriptions to: ChatGPT, Google AI (using Google Flow). I used ElevenLabs on the free tier but will likely subscribe to as I continue the video creation journey.  I manually put everything together and edited it in the free version of the amazing Davinci Resolve.

The Process

I spent a lot of time watching You Tube videos on AI Video creation, which helped me get a good footing.  My process ended up being the following, and note that this will likely change and evolve as I learn how to do a better job:

  1. Feed the story into ChatGPT to have it break it into a scene story board
  2. Use ChatGPT to create “Character Sheets” for characters, locations and important objects (Will include an example image of one of these below)
  3. Use Eleven labs to pick out voices for each of the characters. Google Flow does put voices and sound with the videos but does not keep the audio consistent.
  4. I used the character sheets and short sections of dialog to prompt Google Flow for each of the scenes.
  5. I cut the scenes together using Davinci Resolve as my editing platform. Replacing most of the voices generated in the Google Flow clips with the voices from Eleven Labs in order to keep the character voices consistent.
  6. Some still shots, like that of the house were generated in Chat GPT.

The Bad

Continuity issues: If you look closely at the video you will see things that “change” from scene to scene, like how the table in one shot has flowers on it and later on it doesn’t.

Character consistency This one drove me crazy. It was really hard for me to figure out how to keep the characters consistent. Even with me insisting the AI stay true to the character sheets, Cal and Andy, in particular, change quite a bit throughout the video.

Lip syncing the Eleven labs audio to video: This is more a matter of my ignorance, I now know there is a way to take the original audio Google Flow produces and run it through Eleven Labs usin their “Voice Changer” option to get audio that matches the shots.

Audio sounds hollow and out of place. Creating the voices in Eleven Labs works well, but they often sound hollow and out of context with the surroundings. Apparently having separate tracks for scene ambiance can help with this but I didn’t have as much success with that as I would have liked.

Uncanny Valley:  There are a lot of places with what I would call AI awkwardness:  AI creates a lot of weird pauses and having characters looking around for no reason, or over acting. Sometimes a general feel of stiffness that just doesn’t seem right. It is deceptively hard to get natural looking motion and character actions.  

Google Flow limits scene length to only 8 seconds so I had to stitch together longer scenes.

It took way longer than I thought it would have. I spent about 3 full weekends on the six minutes of Thin Red Lines. I will certainly need to get my process more efficient if I want to keep doing this.

The Good:

I love seeing the story come to life and being able to tweak the characters to what I envisioned them to be when I wrote them.  

AI did surprisingly well at times. While the voices and appearance didn’t quite match the characters, after its first attempt at it, I kept the end scene in its entirety  because I thought it was funny.

What I learned:

I need to know a lot more about video production. Timing, shot selection, how to choose a Point of view color grading and the importance of audio are all very important. If I am to be the director behind what AI is doing, then I need to learn about these things.

Having to break down the story into scenes also really helped me to understand where I had holes in the story. In Thin Red Lines (Spoiler alert), Cynthia, the protagonist, goes on about how much the home means to her but in the end goes after money to make things right. Even though she technically “wins,” this is not a satisfying ending.

Converting my short story to video has been an interesting and fun experience and one I plan to continue with for my other stories. Let me know what you think in the comments below.

-James

The character sheet for Cynthia (Protagonist)

Character Sheet for Andy (Antagonist):

Example of environment characters sheet for the kitchen

Will AI ever write like we do?

Leave a reply

Regular readers of this Blog have probably picked up on my fascination with AI and its impact on writing.

The term AI-Slop is thrown around a lot today. And certainly, there is a lot of it. But this is not AI’s doing. If you believe AI is a tool, like I do, then you can quickly see how it is the improper use of that tool by PEOPLE that creates the slop. The slop comes from people. AI is how they make it.  

Imagine if everyone suddenly got a shop full of new power tools and a free supply of lumber and nails. Some wouldn’t have any interest in using the tools at all. But there would still be a lot of people attempting to build furniture and other creations with their new tool set. Unfortunately, only a few of those people would create something truly worthwhile.  I suspect those people would likely be the ones who already knew how to build something beautiful. I think this is true with AI as well.

It is never the tool; it’s the craftsperson behind it.

When it comes to AI and writing, I also like to think about the emotional involvement we tend to have in our own stories. Do we care as much about something written in part, or whole, by AI?  I suspect not.

My mother has written two books containing various stories about her life and the people she has known through the years. If all of those stories were fed into AI, I suspect AI could have come up with a book as well. I am sure it would read factually and be grammatically perfect.

But I am also certain it could not tell those stories in anywhere close to the same way my mother can tell a story. She is a natural storyteller. In conversation she often leads with a shocking opening:

“I suppose you heard so-and-so was killed by a runaway giraffe at the Zoo.”

OK, that giraffe bit was made up, but this is her exact “shocker” structure. I always found the “I suppose you heard” bit interesting. I think she chooses that opening phrase as it has such a casual, conversational, feel that it heightens how much you get sandbagged by the shocking part.

She also likes to build to a twist or surprise when she can. Here is an example of something she actually told me (Note the name has been changed):

              “I suppose you heard your Terri is in Las Vegas getting married.”

              “What? When did this happen?”

              “We’ll there’s probably a reason she had to get married so quickly”

That there, my friends, is a pro in action.

I am analytical, so my default is to deliver information fast and factual. My tale would have been:

              “Terri got pregnant so she and her boyfriend went to Las Vegas to get married.”

Still shocking news but not that sweet double-punch set-up of the natural story teller.

I don’t see AI doing that kind of set-up. Not just yet anyway. Will it be able to in the future? Maybe. Probably, in fact. But it still won’t have the connection. AI doesn’t care who Terri is. It doesn’t care about her current situation nor know how any of that that should impact the audience.

So, while there will undoubtedly be a lot of AI generated content that we will need to to swim through in our collective future, I have faith that the hand of the human author still has a place to write and express in a way AI will never be able to emulate completely.

– James

Where are the audiences?

Leave a reply

AI helped me to confirm something I have long suspected: The audience for short story fiction is smaller than other markets. (And, yes, whether you can believe AI is certainly always in question).

When I asked it what markets and media have the most appetite for fiction it came up with this chart below (note that I also found it interesting how it put in “Money” and “Fame potential” as metrics along side “audience size”):

While I am not sure if this is exactly correct (graphic novels & webtoons ranked way higher than I would have thought), it does confirm my suspicions that people are watching a lot more than reading. The one exception being novels still ranking really high.

This doesn’t mean there is no market for the written word, just that it will be an uphill battle if your goal is to get your short story published in a written format.

Since I am interested in storytelling in general I am working on converting my fiction story Thin Red Lines to a short You Tube video. And, yeah, in order to do that in the most efficient and cost effective way, I am going to use AI.

Before you start searching for your pitchforks, know that I draw the line at AI editing or creating any of the story I have written – the words will still be mine – I am having it do the video and audio. And, sure, maybe this is me justifying it to myself.

I am going to include this disclaimer at the beginning of the video:

You should also know that I am not after a quicky, wham-bam AI slop production. Like many of us, this story is my offspring, one of my literary children, so I want the final product to represent the tale I envision.

I can tell you, after a few days of working with Chat GPT for images, Google Flow for visuals and Eleven labs for audio, getting AI to provide anything close to what I envision is proving an exercise in frustration.

I often liken using AI iteratively like this as the same as working with a petulant child – sometimes they do exactly what you want and other times they stubbornly refuse to do what you are after no matter how much you try to persuade, explain, and re-frame.

Let me know if you have ever tried creating AI videos for your stories – or if you ever would?

-James

Antigravity

Leave a reply

I have been working with Google’s Antigravity code development tool recently. And while that seems like a strange thing to open with for a writing blog, here is why I think it might matter to us:

Antigravity is a development environment for coding that has AI built in so that instead of writing all of the lines of code yourself, you can “vibe code” which is basically talking with the program in plain language and telling it what you want.

While “programming” with Antigravity I found that the more detailed and deliberate I was with my description, the better the results. This made me realize that our talents as fiction writers will be very useful when working with AI now and in the future.

AI can do a lot of very amazing (and scary) things but it needs us to be able to describe in detail what it is that we want it to do.  The better we are at that, the more useful we will be when working with AI, which I suspect will be a very desirable skill in the near future.

As I try to follow the rapidly changing landscape of AI, I find this a refreshing thought that gives me a glimmer of hope.

At least for now…

Let me know what you think.

-James