This video is a little bit sloppy, but it's about a development feature - it will be in a release soon.
I just did some videos about how to set up the AI Interpolator to generate extra media data for the Media module for image, document and audio types.
This got me thinking about a way to do video as well, and I spent some hours last night coding this. It's currently available in the development version of the module, but hopefully it will also soon be available in the next release.
What this does is rasterize all the scene cuts/keyframes from the video onto multiple images. Then it also extracts the audio and sends that to OpenAI Whisper to get a transcription.
Then your prompt is sent together with these two components as context, for it to be able to "see" and "hear" the video.
Note that this only works with FFmpeg installed on your server.
This video is from before the AI Interpolator module became AI Automators, part of the AI module . Module and menu names have changed since, and some steps look different today.
YouTube's automatic captions, lightly corrected. Click a line to jump to that moment in the video.
0:01
hey everyone Marcus here um so you see YouTube in front of me yesterday I was talking about uh the media module how to improve and add metadata to like stuff like you know images and documents and
0:11
stuff and those things are fairly easy to do what I said yesterday is also as soon as there is a service that does video we're going to add that but that got me thinking that I mean there there
0:24
there is actually a way to solve this today so uh I started developing that yesterday night just because I got inspired and now there is a rule that can basically describe a video so that's
0:37
what I want to show you now um I have just you know ripped a video from YouTube so this is the video that was talking about uh that I'm going to use as example just so you can you can
0:51
basically rerun this yourself to make sure uh that this works but also see the description that I create now actually has something to do with what you see and what you hear in this video right uh
1:04
I will explain how this works but but because the um generation takes a long time I will just start setting up everything so uh the AI Interpolator is already installed AI Interpolator OpenAI is
1:17
already installed with a key I just uh enabled the media module because that's what we're going to be working with here uh then we go to Media types and Video in this case then
1:31
uh and we have a video file so I can just you know show you what you get basically if we do a video here so I just going to upload this video here uh save it and then we have a video
1:45
file right uh and I will I'm not allowed to climb this Tower I did ask but not only just climbing this yeah you see that is that video right without AIO anything um what we're going to do now
1:58
is that we want want some metadata from this right and in this case I'm just going to do uh the simplest thing which is basically describe it so I have only added this so far to text formatted long
2:13
in theory this could be added like the image description to any type of field you know is there emails in this field uh does this video or sorry is there emails in this video represented or are
2:26
they talking about emails or uh so it becomes a May in that case or if it's a Boolean like it's this video about Tom Scott which it is in this case and get checked
2:39
and so on so I could add that uh just comment or create a ticket if you think that's actually something that would be useful it will take some time to do this but um I could do that if if people find
2:51
it useful the reason I'm a little bit hesitant is because I think OpenAI will you know come with a feature for this too I hope that's better than what I have done here but that's if you want
3:05
something in the meantime basically so I'm just going to do a video description here field so it's formatted text takes formatted long that's the only place this rule works at the
3:15
moment uh and as I said this is only in the dev Branch you would have to get the dev Branch version of it uh it's not stable in this case and then I just enable the AI Interpolator and there is
3:27
this rule OpenAI video text uh and we just say which base field we do and then basically what we want to get out of it so describe the the video from start to finish in one value in up to five
3:53
paragraphs with up to four sentences each you make may use HTML tags like headers uh strong italic underline oh it's not italic it's em um also describe the video graphy so you
4:22
can see because otherwise it would be easy to just get it from the transcription what's actually happening right um and how it matters to your description and once again this is prompting um you
4:38
have to write the right prompt and you have to write what you want out the prompt also so you could ask basically uh do we do we see Tom Scott in this video and it would probably give
4:48
you a very short answer like yes Tom Scott is in this video If it knows about Tom Scott which I assume it because uh what's important to know here there's no Advanced uh prompting in this case you
5:00
can take many tokens I could fix this later if you can think of a reason to do this but right now there's actually not even placeholders so you just write what you want
5:10
basically uh normally you should definitely do bat share uh sorry cron because this is going to take forever but I have I have long timeout so I can run it on the web server locally no
5:23
problem I just do that it here you can also say that you only want the video to matter or only the audio but in this case we're going to do both I just save this uh and then I go back to the
5:38
content uh ah sorry some media and we're going to have that field under here yeah we should probably move it up by the way so it looks a little bit better oh sorry I was supposed to
5:54
press manage form display there video description will put there on the video file save reload so you can see that the video description is empty right we click save and this is going to take
6:05
forever so while we're doing this I'm going to show you what is happening now so it's of course it's transcribing the video but then it's also uh taking and this is not my idea I'm not I mean I
6:20
have thought about it but this guy on like OpenAI Forum gave me a much better idea so my idea was basically take each of the uh scene Cuts so this is you oh by the way this is using FFmpeg in the
6:34
background you need FFmpeg to be able to do this if you don't have FFmpeg the rule is not going to show up uh but anyway with FFmpeg you can do something that basically says that if if there's a
6:46
scam cut which is usually where key frames happen so if you know video you know what a key frame is but if there's a scene cut uh take a screenshot if something
6:56
completely different happens take a screenshot and my idea was to send all these images but then I realized you know that was a this Tom Scott video is like a 7 minute
7:07
video uh it would generate like 60 images so then this guy said like okay we can take a video and his case he just taking every second or something I think and generate uh um like a rer from it
7:23
so basically what I did was to combine these two things where I said that oh it's uh finished by the way just so you know how long it takes but before we see the results I'm just going to explain
7:35
everything uh that we restor all these key frames in multiple images and send all these images and I also add the time stamp so it can basically connect that whatever
7:49
is said at 6 seconds that comes back from the transcription as have being said at 6 seconds is connected to this image right uh uh and that's it so we're sending all these restor images we're
8:03
sending the transcript and basically tell us tell OpenAI this is a video please answer the prompt uh and then you get something back like this oh here we have the JSON
8:13
error again so this would basically be something like this I guess uh so the video is the history and current St mod blah so uh yeah you could just you know you can just P pause the
8:28
video on YouTube and compare this to the video that uh Tom Scott made because you know which video it is uh and and then you can basically see that it's using kind of the video you can see here by
8:44
the way because this you could get from transcription but the footage feature various shots that include aial views closeup of towers historical references so you can see that it also actually
8:55
takes the images uh so while this is not a uh definitely not a perfect solution of doing video descriptions compared to the models that will come in I'm sure not that long time
9:11
there is a way to do it today so yeah that's it uh if you want to try it you have to get the dev version as I Said So yeah thank you for watching
