In this workflow we will work with the AI Interpolator OpenAI module and its video to HTML rule, where we will take a video and generate full working HTML.
The video in this example is "Drupal at Web Summit 2023" where Baddý Sonja Breidert, CEO of 1xInternet, talks about Drupal.
We will then give the following prompt to it:
Cut out between 4-6 images. Write 4 parts starting with a h2 header and between 2-4 paragraphs per part. You can use the following other tags: strong, a, quote, pre, code, prequote and em.
Write it as a copywriter that wants to sell it to ad agencies. The person on stage is Baddý Sonja Breidert, CEO of 1xInternet. Write a story about how great Drupal is and how well she explained it.
Make sure that at least one image shows Baddý. If you want to show PowerPoint slides, cut them out to only show the slide.
Also provide one selling quote from Baddý.
Give attribution to the video located at https://www.youtube.com/watch?v=ccfwTVEwTxU&ab_channel=1xINTERNET
If possible, organically link to https://www.drupal.org.
And out comes a full editorialized HTML page, including cropped images and links.
This is built on Drupal - all open source and free, so feel free to do it after you see it here.
This video is from before the AI Interpolator module became AI Automators, part of the AI module . Module and menu names have changed since, and some steps look different today.
0:01
hi there Marcus here in today's video I'm going to show you how to set up a workflow where you can take a video upload the video give a prompt how you want to create a web page and then get a
0:12
web page from it so you can think of it as aiz uh you know someone telling an editor that hey look at this video on YouTube and please write an article about it so this we're going to automize
0:25
now the video at question is uh Drupal at the Web Summit 20 23 so it's from the CEO of 1xINTERNET, Baddý Sonja Breidert uh where she exceptionally well talks about Drupal and why you should use
0:42
Drupal um as usual this is op Source it's free it's a module based on Drupal that's why I chose the video so that means after uh I am finish showing the results here I will show you how to set
0:56
this up all the code is free you can have it set up directly afterwards so I'm just going to go ahead now and click save and this HTML field will then be filled out and it's going
1:07
to show a web page I will be uh uh clear here that I have actually saved the result from running this once so it takes around 54 seconds uh to generate uh a web page from this then 18 minute
1:25
long video just so you don't expect it to go this fast I just didn't want to have a award silence here uh and this is what we have generated I'm just going to click edit so we can go through the
1:35
prompt more exact here so first I'm going to um talk about how uh it actually can generate images so there's cut out between four to six images make sure that at least one image shows body
1:51
if you want to show PowerPoint slides cut them out to only show the slide so not only is it editorial not only does it take screenshots of of the actual video it can also crop those screenshots
2:04
so if we look at the results here it has generated four Images one of those images is of body it's not perfect I just kept it here so you will see it's not perfect they couldn't understand
2:17
that this is a PowerPoint slide probably because of the white frame but in this case and in this case uh I will just show you in the video wherever these passwords are so if we follow here you
2:28
can see this is never show full screen so it has actually gone and cut out this so it only shows it uh shows the slides from the PowerPoint and same in this case right it also added um image
2:41
captions so this it does automatically if you don't prompt this prompt it otherwise then right four parts starting with a H2 header uh between two to four paragraphs Ed you know you can use
2:53
strong in some other HTML so you can see here there's four sections 1 two 3 four with H2S uh it's using you know strong to emphasize certain words open sustainable and
3:06
universally accessible uh and this also comes then from the prompt on how to write this so write it as a coperate they want to sell it to add agencies write a story about
3:18
how great Drupal is and how well she explained it right uh I also add context because in this case I think she says her full name but if she wouldn't I'm just writing
3:29
here the person on stages body Sonja Breidert CEO of 1xINTERNET it's also important so uh OpenAI or the a large language modu model in this case can connect B bad to the person on stage
3:43
because I say later make sure at least one image show body uh and without context it wouldn't actually connect that though it would probably take the assumption that it is the personal stage
3:54
uh also provide one selling quote from body so there's a selling quote and it even added it in uh quote or pre quote here so Drupal is either the Pioneer in CMS using these passwords or have
4:07
already integrated in within a weeks in our project so if we just go to more show transcript and I'm going to search for Pioneer is probably a good word it's here but it's probably almost at the end
4:24
there so I just played this you're seeing every single day and I'm going to tell you something through is either the Pioneer in CMS using these passwords and integrating those technology
4:36
technologies that allow us to use this either it's a Pioneer or we have already integrated it within weeks into our project and why is that so actually it didn't quote her word by word but I
4:50
guess that's usually how a editor would do as well you know create a readable quote of um the gist of what she's trying to say I guess you this is also a prompting question you could ask it to
5:04
you know quote her word by word uh lastly but not least I said that it should of course give attribution this doesn't change just because it's AI if you if you actually you know write
5:15
article about someone else's work you have to give attribution so please attribute this uh video link and also if possible organically linked to drupal.org and here it actually
5:27
surprised me so by default when you click on an image it links to a larger version of this image but I think it did the connection to that okay this is the YouTube video so when you click here it
5:40
should go to the YouTube video so this is what it did instead and if you don't want this you have to you know further create a better prompt and then in the end it has naturally added there to the
5:50
be part of this groundbreaking yourne to the call of action is clear Embrace drup ball and this is then linking to drupal.org um so that's it that's the results and um
6:03
now I'm going to go ahead and show you how to set this up by yourself using Drupal so before I start with uh showcasing on how to set up this I just want to uh show my website workflows of
6:20
AI so uh it's on workflows uh- off- a.com you can see up here there you can see all the different workflows that I have done together with the module the AI interpolator so the module AI
6:35
interpolator is open source it's free all of the workflows you see here you can basically do by yourself after you have seen it it's a chaining tool for automation of AI running in the content
6:47
management framework Drupal so the advantage you can do here instead of you know you can subscribe to the YouTube channel if you're you know you want to see new videos coming up but here you
6:58
can actually uh uh filter on all the different uh plugins that exist for the AI Interpolator so you can see uh specific videos that you might be interested right the AI Interpolator is uh since a week
7:15
fully documented that means that every rule that you can run uh so a rule is basically one part of the chain so you can think of it like uh you know you have something that scrapes a web page
7:29
and then you maybe have another uh to uh another rule that uh categorizes that web page so that's a simple chaining rule so the rule we're going to use today is OpenAI video to HTML so it's
7:42
very straightforward there's just one rule um but if you want to know more about it you know just Google the interpolator documentation I uh and there you can get
7:54
a list of all the rules I think we're up to 60 65 rules now that you can do on different field types and so on um for today then we're going to use the AI interpolator OpenAI module so that's
8:08
one of the plugins to the AI interpolator that holds this Rule and we're going to need the token mod module in this case because we we want to basically have Dynamic prompts so if you
8:20
you know in the Showcase I did this was about a very specific video uh uh about body talking about Drupal and of course this prompt then become specific to this
8:32
video uh so if you don't want a fully automatic workflow where you have some generic way of you know looking at videos where you can have a generic prompt you need the
8:45
token module as well so we're just um well I'm going to show you as well what I have done is I'm running ddev in this uh showcase but you know you can run Docksal, Lando whatever you want for local
8:58
development what's what important though this will not work without FFmpeg and I know this is a showstopper for people that does manage hosting because they will never install FFmpeg but if you want
9:09
this to work there is no other way around it because we're analyzing video we're having FFmpeg actually looking at the video basically and also doing all the uh different extractions from the video
9:21
and so on so if you're using DDEV it's just in the config there is this uh web image extra packages you can just ffmpeg restart DDEV and that's it the other thing I just added here is by the time you see
9:36
this video it might not be true anymore but right now if you look at the AI interpolator it's in beta mode which means that we have to set uh General minimum stability to dep
9:50
basically so uh the thing we just need to do now is we just need to basically uh copy the composer command uh and do DD and the composer command and then it's going to fetch this code so this is
10:09
a package manager for PHP for you who don't work with PHP uh on the regular and then we're just going to go ahead and do the same with the token module and oh right so what I just said
10:25
I have to prefix with d all forgot myself so we prefix this and you also see that it actually downloads this oh by the way I have installed admin toolbar beforehand
10:36
because otherwise it just becomes unmanageable to use the website the next step is just to log in uh and enable this module so I just click extend uh and what I need
10:54
to uh extend then is the AI uh install sorry uh AI inter OpenAI and the token modor are the two that we need and this is going to install the OpenAI module then and the OpenAI
11:09
module will um complain that we need to add the OpenAI key so I'm just going to go ahead and do that here this is to be honest not how you should be doing it putting it in uh completely like this
11:23
because that would be exported like config you should rather put it in settings.php but for all purposes when you trying this locally this is good enough so I'm just doing a HTML key here
11:34
temporarily temporary key that I will remove after I've done this video so you can't copy and use it and then you can just click here or you can go to configuration OpenAI settings paste it
11:47
in save and if you reload the error message is gone and we have set up OpenAI uh so the next step that we want to do is to uh do uh structure content type I'm just
12:02
going to create a new uh content type here I call video to HTML because I'm using content types and that makes sense in this case we're doing web pages uh there's a lot of
12:14
extra fields that comes with the CMS part of Drupal right so I will just disable this and remove some stuff that I usually do so no menu save and we want to add
12:26
Fields uh we can't use the normal body field I haven't done a rule yet for the text format long with summary so there is only for text formatt long but the first thing we want to add is the things
12:38
that we base this on and that's then file upload file and that's this is then the video source so this is the video we upload right we just take one and we say MP4s
12:51
are allowed but in in the you can do anything that FFmpeg can handle which is basically every video format in the world so you can do whatever you want there but we just set M4 in this case
13:03
create another field that is then a text plane text plane long which is we call the prompt to generate HTML so that's the prompt we're going to use to uh generate uh the HTML right uh
13:21
we just well sorry we should require this of course and we should have required the video source as well because without if you don't f out these two Fields nothing is going to happen
13:31
right and then in the end we want to generate the HTML uh field right and this is what is the actual website so this is a text formatted long uh I'm just going to call it uh full page HTML
13:45
in this case continue and this is where the magic happens with AI interpolator then uh so this is also just going to be limited to one we're going to allow full
13:54
HTML because we want to allow a lot of tags so if you don't know Drupal these are basically filters for for what is allowed so you can read more about text formats Drupal if you don't know it
14:06
enable a an poor so this is where the magic happens so here are all the rules depending on what we have installed and there is one then called OpenAI video to HTML we're going to enable this we're
14:20
going to run this in base mode there is no other choice in this case and we have to give a base field so this is basically where is the video located and that is in the field video source
14:32
and then we're going to add the generating prompt but since we want it dynamic in this case we don't want one prompt that we run on every instance of the video we upload we're going to use
14:41
the token module so we're just going to paste the field that we created that was prompt to generate HTML and this you can just click as long as the text area is
14:51
focused and then it's added here so this will basically replace that prompt with whatever prompt we write right then in advanced settings I would suggest that you run this on the Q Chron
15:05
job because generating these videos can take so long time and if if you either have a net uh you know web server Timeout on nginx or Apache or a PHP timeout this might actually time out
15:18
before it's generated and you just get a error so unless you can set a longer time out I would set that you do this on qron make sure that you have a Chron job that runs uh well often every minute
15:30
would be the best and then it happens on this crun job where it's allowed to take long time and this this is a q worker by the way so it's not going to stop the
15:38
Cron job just because it's running a long time and you can also use the Dr Q command to run this thing but in our case I so I can visualize it well I'm not actually going to run it now uh you
15:50
have already seen it by the time I have run it right so uh in this case I will just it direct there uh I will just do some uh fixing of the forms that we show off
16:07
because there's a lot of stuff that's connected to Drupal CMS that I don't just want to show you and then of course on the display you don't actually want to see the video you don't want to see
16:18
the prompt to generate HTML you just want to see the HTML and we don't want a label on it so it's shouldn't say full page HTML over it and and we are done it's not more complex than this so what
16:33
you can do now is go ahead and do a video to HTML write a title write a source write how to uh generate HTML click save and wait and that is uh talking about costs
16:49
by the way you can uh assume one to two cents per minute of video so this uses a pretty uh budget saving way of doing this because uh if you look into OpenAI you might
17:05
actually notice that video doesn't exist as a multim multimodal context yet I'm I'm guessing it's coming soon but it doesn't exist so how this works in the background for you who are
17:17
interested uh and this by the way I can also mention this is open source the code is there so if someone wants to build a product around this feel free uh be nice and give attri ution to
17:29
me but feel free to do it um I don't have any commercial interest in this I love open source so uh anyway how it works is that it uh looks for scam Cuts or key frames so uh yeah read up on
17:45
video if you want to know what that is but basically it takes the most important Snippets uh out of the vide as images put this in a grid of nine uh images per image put time stamps on this
17:59
then it uses whisper API to uh get a transcript of what's being said and when it's being said and then you can connect this via image to the audio and that way create a prompt where you feed it the
18:13
images and the transcript and your prompt and overlying prompt that's part of the module if you look into the source code uh that gives back HTML and that's how it works in the end basically so uh
18:28
it is fairly complex if you haven't worked with AI before but you don't have to set it up if you work with Drupal just get the module and it does it for you so thank you for watching and see
18:41
you soon if you like these videos and please subscribe if you like the videos that's what the YouTubers always says thank you
