In this workflow we will use the AI Interpolator OpenAI module in Drupal to take a video and a prompt for what you want to extract from the video, and extract that into one concatenated file or multiple snippet files.
That way you can take a long video and say something like "Extract all the places where he talks about Microsoft" or "Get me all the scenes where there is a person with a flag in their hand" and you get video snippets back.
This is using FFmpeg together with OpenAI Vision and OpenAI Whisper to be able to read the videos. Then it asks for timestamps in the video and uses FFmpeg to extract this and save it back.
In the end we have created a search function for video, both on the audio and the visual part. Neat, isn't it?
The OpenAI cost for a 5 minute video is around 10 cents.
This video is from before the AI Interpolator module became AI Automators, part of the AI module . Module and menu names have changed since, and some steps look different today.
0:01
hey there Marcus here I'm going to show you a really cool workflow uh today so we're going to take the Ed Sheeran perfect video we're going to take the Monica Lewinsky I did not have sex no
0:10
sorry Bill Clinton I did not have sexual intercourse with that woman video and we're going to cut out for uh cut out segments from it using uh prompting so I'm just going to start before I I uh um
0:28
tell you the background of the and how it works so I will just write I want all the shots where a pizza is featured and click save and this is going to start processing now so what
0:46
you're seeing in front of you is Drupal Drupal is a CMS that's open source uh it's running something called the AI Interpolator which which is a module I made that can chain a lot of AI
0:58
processes and work together with the complex field types that exist in Drupal and this is using Drupal rather as a CMS automated tool than a cmf automated tools so the content management
1:10
framework rather than the CMS there's probably some editors that could use this uh just so you know there is JSON:API GraphQL modules uh connected to Drupal so if you want to set this up
1:22
it's very easy after I shown you this proof of concept I'm going to show you how you can set this up in 15 minutes basically using these modules uh the only thing you need is a OpenAI account
1:34
so and and the cost is actually minimal so now I got a cut here basically so I want I we will see now basically if it produced I want all the shots where a pizza is
1:46
[Music] [Applause] feure right so you see it worked right uh actually what I could have written here also so this is important for video editors you could have write that I I
2:04
want them as video Snippets and you would get multiple videos back instead of one video right so maybe you want to import them into Premiere Pro or something like that so
2:17
uh Next Step then we do Bill Clinton and uh we're going to ask then um can you out the part where he is refuting uh that he had six so I'm not using his words but by context it will
2:43
be able to understand that that is sentence that we're looking for uh I'm going to click save this will probably be a lot faster because it's a a short video in bad resolution and we got
2:56
something back let's see here I'm just going to show you the original video video first so you see that it's uh not just this small part but a long video president so you can see it's alore
3:12
talking before and so on and this is what it cut out I did not have sexual relations with that woman Miss Lewinsky so that works as well the reason I wanted to show these two videos what was
3:26
because the first one was a proof of concept that is using that actual uh uh images in the video right and the second part is that it's using the audio in the video uh to to figure out what it should
3:41
cut uh what is important before I show you how to set this up this is actually not that expensive so what I'm doing in this case is that I'm uh looking for all the scene cuts and key frames so
3:55
basically where something changes in the scene I take a screenshot of this uh and then we put this into a grid where we show nine screenshots per image and then we're asking OpenAI uh how uh
4:09
basically based on this context and getting the transcription from whisper uh with all the time stamps and we also put the time stamps into the images uh you know answer whatever instructions I
4:23
have there and then we're just cutting it out uh another thing that's important to know before we show this is that uh this is Drupal it's running on a normal LAMP server you can host it anywhere but
4:35
this specific functionality requires FFmpeg P because there's a lot of uh both video reading and video editing uh being done um so that's it uh proof of concept now I'm going to show you how you can
4:49
set this up by yourself from basically from scratch hey there so now I'm going to show you basically how to set up this uh almost from scratch so what I'm I will not show
5:10
in this case is how to set up a web server in this case a local uh local uh development environment so I'm using ddev in this case uh for local en uh development
5:25
environment what you need basically is a LAMP server right and one more thing so what I've added in if you use ddev you can add extra packages you have to have FFM this is uh crucial otherwise nothing
5:38
of this workflow that we showed today works this is a requirement basically um so I have added that the other thing I've added is minimum stability Dev because we're going to be
5:51
working with modules that are still in beta in this case so we have to allow uh minimum stability Dev hopefully they will be in production later I have used the dev to install a plain standard
6:06
installation of Drupal uh with nothing extra so it's the standard profile this is usually what you get when you install Drupal what I will add is admin toolbar which just
6:19
makes life simpler because the toolbar for the admin becomes a lot more usable then uh the engine of this workflow is the AI Interpolator OpenAI so this is a plugin of the uh family of
6:34
plugins that's AI Interpolator which is basically AI chaining mod modules but also uh modules to create context for AI and also extracting uh uh extracting logic and stuff like this as well so
6:50
there's a lot of different services that you can chain here using uh using only GUI so I everything I show you now will require no code at all you you don't need to be a developer
7:02
to set this up then we need the token module to make the prompting feel smarter so you can dynamically load uh uh text into the prompt basically so I'm going to go ahead and install this so we
7:17
just copy the composer command and run it I have to prefix it with DDEV because I'm running DDEV and then you see it it downloads admin toolbar in this case uh for me so if you're from some other uh
7:32
programming language like Java you might know Maven or uh node you know mpm so composer is just the same thing basically um so we're just getting these three then the next step we do is we log
7:49
in on the website uh and we're click going to click manage and extend and this is where you install these modules that we get uh so what I need is the admin toolbar extra tools and admin toolbar AI
8:05
interpolator OpenAI and uh the token module so we just install these it's going to say that it's going to add the OpenAI Core because that's a dependency that it got as well so it will install
8:18
this and this will complain now that there's no OpenAI key uh so we have to have have to add one so we can add one here we can go to configuration OpenAI settings then I just open where you get
8:31
the API keys from the OpenAI uh web page so you just log in and get something uh and go to API keys there basically uh so we take video testing how we call this copy this
8:45
key paste it in here and we are all set if I reload the error is gone now right so the next thing we're going to do is uh since I'm not doing any coding in this case I'm just going to use
8:58
something called content types in Drupal so this are usually for the CMS part of Drupal but if you want to use this functionality or anything else in the workflows that I've done so shout out
9:10
workflows-of-ai.com so workflows Dash wait let me show you instead flows of ai.com it's this web page right so you can find all the different workflows you can use this chaining tools for um if
9:27
you want to use anything of it uh you can set it up without any code and uh it's also important to know if you just want this as a headless service API or something there is a JSON:API uh
9:43
module you can just click extend it part of core and then you have a API interface where you can basically send the video in the prompt and you get the videos back so we're going to do video
9:55
extractor we're going to call this content type I just clicked save as I said this is just CMS stuff that I just want to get rid of some stuff and then we also get rid of the body field which
10:06
is a typical WYSIWYG field we don't care about this create a new field that we call the origin video origin video which is file upload and a file so these are the field types that basically triggers
10:22
I mean you use them in Drupal for uh creating complex entity types content types with a a interpolator is basically saying that from one field type to another you can do something so uh this
10:36
is what the AI Interpolator does so we take origin video and we say this should be MP4 and it should be required because you need to upload something right and then the next thing we do is take a
10:47
plain text text plane long which is just a text area which is the prompt create this uh this is also required both of these are limited to uh adding one right so with the video in
11:03
theory you could actually add multiple uh and it would extract from all those videos but that would take forever so I um it's better to just keep one video the last thing we need to do
11:17
is add another uh file field that is the destinate oh have it there destination video right so this is where the generated video is going to end up since we can ask for snip instead of one
11:30
concatenated video we're going to set unlimited here so we can say that hey create multiple videos here if you want it's also going to be MP4 and then we have the thing that the AI Interpolator
11:42
does and this is just we're saying enable AI Interpolator and here we can choose different rules depending on what the origin uh origin field is and what the field we're trying to generate this
11:54
right so but in this case there is only OpenAI video extract extraction so we're using base mode and we're getting from the origin video and then we put a cutting prompt
12:06
here but since we want this to be flexible so you can write a different prompt for different videos you upload uh we're going to use from the token module that we installed so these
12:16
are the tokens that exist and these we can put here so I can take the node get the prompt and I can just click here and this is going to pre fill out so instead
12:27
of hardcoding some prompt here it's going to fetch from this field now uh advanced settings is uh if you're going to set this up for real you should run this on the Chrome job right but now
12:40
I I have a long timeout so I'm fine with running it directly when you have a chain of of uh prompts you can also do the batch mode so you chain them using JavaScript so
12:54
you don't get timeouts uh we save this and and believe it or not we are set up so let's try this in action now then so I'm just going to take uh another thing here baby
13:08
shark so this is the famous baby shark video right uh and I write cut out the video that cut out the scenes where there are visual confirmation that you should Subs
13:30
subscribe so that's it basically there's in the end of the video there is like I think 10 seconds or something where it says like subscribe to us so right now it's just going on the visual part
13:42
right uh and hopefully this will work and we have set it up and it yeah how long have I been recording 8 minutes so you can set this up extremely extremely fast and setting up JSON:API takes
13:57
another 10 minutes maybe and then you have this as a service uh running uh and then we have the destination video we try it baby shark baby shark babyar so you see it uh cut out the 10
14:21
seconds of the video that was the Subscribe we have it working that's it you can you know build a whole product out of this I'm going to expand this later so it actually also can
14:36
um uh reformat the resolution so maybe you want a mobile video 69 then there's going to be another prompt where you can basically say uh so if you take the Ed shm video focus on the pizza all the
14:50
time so basically the camera moves around and focus on the pizza so this is uh a future add-on I'm going to add soon uh and that's it thank you for watching
