0:00
This video is sponsored by FreelyGive. If you need help with Drupal and AI, get in touch with us. So, this is another video of the examples I gave in Atlanta. So, here you're going to see how you actually set this up. So, the
0:14
idea is this. Gordon Ramsay comes and contacts us. He says, "I have 500 videos on YouTube. I need a recipe site tomorrow. You need to have filled out all the content." Well, luckily for us, there is the umami website. So there is
0:29
a website with all the structure that you need for a recipe website, but filling out this content is going to take forever. So I I actually tried with this video. The video is just four 4
0:40
minutes and 20 seconds, but filling out this manually and trying to figure out going back and forth in the video. Well, it takes around 40 minutes to fill out one content and then you have 500. So you're going to need to hire a lot of
0:54
people. uh or you can let the AI do it. So, what we have set up here is that instead of filling out all this, just need to fill out this and the YouTube link. Click
1:07
save and you're going to end up with something like this. So, this is actually going to be the AI looking at the video trying to figure out uh how to create this dish from uh what it sees
1:20
and what it hears, well, what it transcribes to hear. and uh write out directions, ingredients, all this uh metadata that we need and even take a screenshot from the video of the
1:33
finished dish. Uh so if you're interested in figuring out how this work, just follow along. Right. So we have uh umami here. It's just uh vanilla umami. I've gotten
1:51
I've downloaded four different modules. admin toolbar. It just makes life easier. Token is required for the automators. Then we have the AI module and the OpenAI provider. So you can actually use any vision module and you
2:06
can use any transcription model. Deepgram for instance is actually a better transcription model for this. But um since many people use OpenAI, I'm just going to uh assume those for this. So I turn on admin toolbars. I turn on
2:23
automators and oh yeah I should turn on OpenAI as well. I forot that but let's do that a second step here take some time and uh the next step after that is that we have to set up OpenAI as
2:56
well. There we go. So we have it set up but we also want the OpenAI provider. So we're also going to set up that. While I'm doing that, I will also uh log to the machine because this is also a requirement for the thing we're going to do. Since we're going to look
3:30
at video, we need to be able to actually have something that can decode video. And what you need for that is uh ffmpeg. So, you have to have ffmpeg running on on the server where this runs. But be aware, it's the coding. It's not encoding. So it's not going to
3:49
kill off you know all your CPU usage or anything like that. This is a pretty light operation that is running. So uh let's see how it went here. Yeah. So we go to AI provider settings OpenAI uh authentication. We have to create a
4:11
key. Um so you're adding it here. I'm not going to showcase this so you don't see my key here. I'm instead gonna do that off screen.
4:23
Uh so let me just add one [Music] here that I'm calling OpenAI key. So I know this I mean the advantage you have you can forward in the video it's not maybe that fun to listen to someone like coding but I want to showcase actually how long time it takes to set up
5:05
something. I don't want to add a lot of um steps where you know I cut or make it look faster than it actually is. So uh we now have the OpenAI installed. We have ffmpeg installed. Uh so that is a good
5:30
start. So uh there is another library and installation we have to do in ddev. So it's this drupal YouTubedl. And you might ask yourself what why is this not on drupal.org? This is actually a module. So you can just use composer require this as normal. This is also my
5:50
um my repo even boring. So the reason for that is that YouTube DL that is the underlying uh application running here is kind of um debated. Let's say it like that. It's actually um have forbidden where I live in Germany. Germany have
6:10
very hard laws on privacy. You should never use Pirate Bay here. So because of that I you know GitHub I think actually got into legal threats because of their hosting YouTube DL. So because of that
6:25
I'm uh keeping it off drupal.org. So if you want to use it just get it from here. Uh you can either use the YouTube DL uh binary or you can use this uh yt-dlp.
6:39
Doesn't really matter. It works with both. So, I'm just going to do the copy the instructions here to install the binary. And I'm just going to uh run that uh uh let's see
7:03
here. Oh, failure I think. Okay, let's do this. Maybe there is some Uh well let's do this instead. Let's store it under user user bin and do sudo. Uh all right we have to also write the name.
7:36
So, and then we need to set the chod on it. So, we're just installing it and oops, sudo and then we're going to just see. So, it's running. Yep. So, we have it there. So, I have in gotten this module as well. So, I'm just going to go ahead
8:01
and install that as well. Let's close here. So no, it's not called YouTube. It's called video downloader because YouTube even if it's called YouTube downloader, it can actually take
8:15
as you see here like thousand different vid video providers. So in this case you can create a workflow where you basically uh have one uh video field and one file
8:28
field and it will just download this as a video. So now we can go to recipe. Then uh what is important for us to be able to try this out is that first of all we need to turn off a lot of required fields because we will actually want to
8:50
uh install those ourselves. Right? So we have preparation time. Uh we're going to unrequire that There we go. What do we have? Number of servings. So, this might be boring to look at, but I I want everything to be
9:19
uh realistic how long time this actually takes so people get an idea how long time it would take to set up something similar. Of course, I have kind of thought through this beforehand. So
9:32
there's some time to think through what you want to solve as well. But hopefully all these uh things that I'm doing gives you a good idea what you can do. So I'm going to turn off this as a
9:48
required field summary directions. Let's do like this. That should be it. So title of course that is required and that we're going to have to keep just uh so so if we update here now hopefully it
10:16
looks a lot better. It's just the recipe name that we want to use. So, we're going to create our first um we're going to create our first uh chain here. So, we start uh with a link and we call this YouTube link.
10:38
We're going to use YouTube in this case, right? Um, and we we set this required because this of course you have to to give. We don't need a link text and we don't need to enable the AI
10:52
automator because this is only a input field. It's not a field that actually does anything. This is the field that doesn't something. So, we're going to call this YouTube
11:04
video and it's a file. Add it. Now, I have forgotten what it creates by default if it's MP4 or um maybe it could. Let's see. Uh uh uh if it's Matroska how do I even forgot what that is. MKV could be to create MKV I don't
11:36
know. Uh and in this case we enable AI automator and there is video downloader link to video uh file. We use our YouTube link as base mode. Uh we're going to use patches because this is all going to take a long time. We set a very low weight because this is the first
11:55
thing that has to run. So, we're just going to save this. Uh, oops. What did I do wrong here? Uh, is that not how you write it? You just write like this.
12:12
Maybe this works better. Yep, that'll work better. And then we want to make sure we pull this up so we can see it in the beginning of the form field. Uh you will see here it
12:25
created an automator status and that's just if you're using this API you can actually see the status of it working. So we're going to put well we can put the video up there but the
12:38
important thing is the link. So, and uh then we will test this out. And the recipe we're going to use is this one. So, it is from uh it is from Gordon Ramsay. It's not Gordon Ramsay himself cooking it. Um, and it is a chef basically
13:10
explaining how to do it. She says what she's adding into it. She talks about it, but there isn't actually like a sign where it says exactly how the recipe is done. So, this is going to be basically the uh AI looking at the video and
13:29
trying to figure out all of these different things that we need to uh do. But first of course we have to try. So the downloader works. So we have to give a recipe name. So these are the two
13:41
inputs that we require. The rest we don't require. And this will take some time because it's downloading a video now. Um but I I I will not as I said forward here. I want this to be real time. So you get the idea how long time this
13:59
takes. So, if we go to edit now, you can see uh that we have a video here. I'm just going to Today we're going to learn how to make our lobster. We have the video. So, first step done. So, uh for the second step, I have
14:24
to basically go back to the manage fields. We can close this again. Uh, and I'm going to create a new field. But I created a new tab because I want to see everything I need to Well, well,
14:37
actually, let's do this. Let's go to our content and we're going to check here what we need to fill in because what we're going to do now is that we're going to take a
14:50
uh should we take format? No, just take plain text uh text plain long. And we call this video description. And this is where basically the magic happens. So what this is going to do, it's going to look at the video based on our prompt and try
15:08
to extract all these things that we're actually wanting here. So there is video to text and that's what we're going to use. So uh we're going to use the YouTube video which is the file field as our base
15:22
mode. Uh before I start writing this prompt, I will just set this to batch, but it should be higher than 50 that we used before. Uh speech to text provider, we use OpenAI whisper one and we use the
15:35
default advanced JSON uh model here. Uh we don't really have to give any extra prompts there. So you are expert and turn them into recipes.
16:24
Um so preparation time try to uh figure out approximate how long time this takes.
16:41
Also, this is a people meet online. Also note that this is a professional shift that normal people probably need double the time preparation time in minutes.
17:07
Uh cooking time in minutes. This always answer something. Uh So hard. Oh, this actually we can infer from the recipe later. So uh category and tags we don't have set media image comes
18:18
later a summary of three four sentences of what this recipe is about. all the ingredients with name, measure length and amount uh directions in uh uh ordered ordered list format and that is what we need I guess.
19:02
Oh, let's try this. We might need to improve it. Let's see. So, I'm just going to reload here now. Oh, no. I'm not just going to reload. I'm going to make sure that this field is higher up. So, we see the
19:19
results right away. Now, the video. Now, we can reload here. So we see we have video description and that is what we're going to try to get the results for now. So we just going to click save here
19:39
now then. And this is also going to take some time. So uh just so you get an idea what it's doing here. So it can't actually look at a video and because the only one that can do that right now is Gemini. So
19:52
there is a process going on here that I I think Drupal was almost the first with uh doing and I even got you know I've gotten two people contacting me to create startups around this idea. Uh but it is actually not very
20:09
uh complex in the end. So what it's doing it's transcribing the audio if time stamps exist it has time stamps as well and then it's using ffmpeg uh to take out all the key frames.
20:22
So I used to work as a video engineer. This is why this is easy for me probably. But uh it's taking out the key frames. And a key frame you can think of is like a scene. Something happens in the video that is so different that it
20:35
can't use the other frames to encode this frame. So it creates a completely new frame. You can think of it as like a new image. Creates images and overlays those with the actual time stamp so you know when it's happening in the video. And then you provide this to the
20:52
algorithm. So you prov it also puts that in a raster. So you don't feed it, you know, I don't know, this is a 4 minute video. You don't want to feed it like 400 images, but instead it's it's like a raster of 16 images per image, I think.
21:08
Uh, and that is what it's looking for. So you can see we have a lot of very unstructured information but it is enough for it to actually start filling out the other fields. So that is what
21:24
we're going to do now and this is simpler. So we just go back here click manage fields I guess uh and we're just going to go through them one by one now then. So cooking time is an integer. So we just enable a
21:43
And now the actual chaining doesn't really matter anymore. Uh so we're just going to use the video description that it yeah video description which is the context here. Uh and try to find the part of the uh
22:04
text where it says cooking time. So that's supposed to be somewhere here, right? uh and extract the value. Uh wait, we don't have to use the word context. We just use the word text. I still do batch though. It looks nicer.
22:24
And we just try this first. So this actually works with filling out preparation time. And then I'm just going to go ahead and fill out all of them. So let's save this and see. So it works. We click edit.
22:40
Scroll down. Oh, that didn't work. Why didn't that work? Uh, try to find the part of the text where it says cooking extracted uh integer value. I don't know if we need to specify it like this. Data
23:01
issue. Sometimes you might need to be exact. So, it might have been trying to save 30 minutes, which won't be savable when I enter the uh No, now it works. Cooking time. 30 minutes. It was cooking time. Yeah.
23:17
Cool. Uh then, let's see here. We're going to just add all of them now. So, difficulty. This is a list text. So, it's a little bit uh little bit more things you have to add here. Actually,
23:37
something that's good here is to copy the uh placeholder. So, there's Scott Euser is working on um prompt module that we're going to implement here. So, you can have example prompt so you don't have to copy from there. So
23:56
uh based on the text find the difficulty uh and fill it out possible difficult this uh and yeah that's it batch So we have difficulties directions I don't remember was
24:29
direction one field was it? Yeah, what field? I can see it there. One field. We want the full HTML. Enable H AI automator. We use LLM text. Simple. So there's LLM text. LLM text is very good when you want to look at the text and maybe split this up into multiple text
24:48
like give me 10 suggestions on how you can do this. But the LLM text simple is just it's not going to try to structure anything. It just takes whatever it outputs. And this works a lot better
24:59
with simple models as well. Um, can you extract the part that is instructions here and output that using HTML? Be sure to use for the list. So uh O L I and also
25:27
emphasize I have no clue to spell that but luckily enough uh AI is very good at understand anyway. to emphasize uh words with strong em uh yeah that's good I guess context is the
25:54
context and we just do batch and we say that we should always use full HTML otherwise it will try to figure out from the chosen ones which one to use but it's better to do this. Oh, maybe I go
26:08
back. See, so I actually using HTML output is always good to write here. Uh uh with HTML the limiters is even better to write because if you're using this extracting it can actually extract just that part very
26:29
easily. Layout we don't have to I will come to media image later. That's the last thing we do. Uh number of servings is that actually there did I ask for that? Yeah, it's
26:43
there. Uh so once again video description. Can you extract the number oops number of servings from this recipe from this text of our recipe and save the entire view and context
27:11
context do batch and of course this I'm doing very simple prompts now that might fail from time to time. When you're doing this professionally, you should really test the prompts out over and over until you
27:25
know you see that it working really well. So, in this case, I'm just going to copy from cooking time because this is just the same more or less.
27:39
Um, preparation time, What do we have left here? Uh, recipe category. Oh, I come to recipe category and tags later. So, summary. This is also HTML. So, uh actually let's do this.
28:10
Uh a word where did you have directions was the thing field is called instruction but the title is directions anyway [Music] uh part that is a summary of the recipe here and I'll put that
28:37
uh Oh, let's do this. Strong nm uh that summary tags video description. We don't Oh, the rest are stuff we have already, but we're missing something. It should be So, there's directions. There
28:58
should be some ingredients. I forgot that problem. Yeah, forgot that. So, uh, can you look at the text and, uh, uh, extract the ingredients. Make sure to extract one ingredient per value. So
29:34
this is where we're going to use oh we're going to use LLM text not LLM text simple. So this is a typical example where we're taking one one text and this is a multiext as you can see uh uh
29:47
field. So we make sure to extract one ingredient per value. Let's do this. So just if you're going to do as I just reiterate this, you're going to do this in production, don't keep these prompts. Try to get better prompts, but they
30:06
should be kind of good enough now. So we're going to try this out now. Uh before we continue with the categories, um so we just save. And now it's going to be many batch jobs, I'm guessing. Let's see.
30:30
That doesn't sound like enough. Didn't change more than three. Let's see. I might have missed something or I might have said something to not that mistake. See if there is something not
30:44
added there. We have preparation time cooking time, number of servings, difficulty. We have a summary. Oh, did I forget to set forgot to set it before HTML probably the word here. Uh, and it has filled out also the
31:04
ingredients there. Uh, oh, but there is one obvious thing amount. Yeah, this is not great. So, I'm just going to I'm going to showcase how you actually fix when something is not correctly set. pump. So
31:23
you can just remove everything because then it's going to rerun. So if we go to ingredients now I also going to see set because it took very literally that we just want ingredients including it's uh measurement
31:43
uh and uh measurement and amount. So we do this, we save and then we can as soon as this is saved, we can resave.
31:56
So everything else seem to work but not that one. So that's a way how you can reiterate. You see that most of it is working but this was not good. So we have to reiterate to make sure that this is
32:09
correct. Uh whoops. That still was not good. Let's see what was wrong now. You look at the thing including make sure that you one ingredient one set. So small words can make a huge difference here. Uh measure
32:32
measurement amount and ingredients. Let's see if that works better. And yeah, even if this works now, it might not work on the next ingredient. So, as I said, you will have to try it with a couple uh another video. I mean, you will have to try it
32:52
with a couple of ones until you find the correct prompts that basically creates the things you want. So, now it's working well finally. Uh so, what we want to do now
33:07
is tags, I guess. Uh well, there was yeah, there was recipe category and tags. That's two different things. And that's interesting because they do kind of different things in the automator. So
33:18
I'm going to enable the AI automator again. Uh we're going to use LM taxonomy. Uh and we're going to use uh in this. Let's try to no I think actually we can just use the simple one. Uh is that a multiffield? Yep, it
33:40
is. So we can actually once again copy the placeholder. Uh choose up to two categories from the category that fits the text. Just use the value option comma and use the context context. Right? Uh we set this
34:00
to batch. Uh in this this doesn't mean anything in this case. We don't have to do anything more. That's just saving this. And then for tags, it's more interesting because I'm guessing this tags also has uh yeah create referenced entities if
34:19
they don't already exist. So in this case, the automator is also able to create, but it can also look at uh uh taxonomies that exist. And this is quite interesting. So we can just paste this
34:34
again. Choose uh or create between one to three category tags, we write from the tags context that fits the text or create them if they do not exist. And it has the value options,
34:54
comma, it has the context here. But something that's interesting, you can actually remove this if you want. uh if you have a lot of tags, it's going to cost a lot of money to do this. So, what you can actually do instead is uh do
35:10
this find similar tags. Uh and this will basically use the model to try to find similar tags that exist. Uh and and uh well, think of this example. you know, if you're saving this as
35:32
uh uh I don't know, rice dish, it adds a tag called rice dish, but there is already a tag called dish of rice. You don't want those two tags to exist at the same time. So, this is what's making
35:45
sure that that doesn't happen basically. And then we can make sure maybe we want everything lowerase. I'm not sure if that is the standard. I haven't looked actually how it is. Uh maybe I should do
35:56
that. Let me just know that I can see under structure taxonomy tags how they actually write this. No first uh oops what am I doing? I want to go back here. It should be first character uppercase. And let's see if this works.
36:22
Save. Oh, I would I should go back and look. I probably forgot to set batch on some of these. That's why it didn't actually count down here. Uh, and we check now if we have so
36:37
dinner party, seafood, seasonal, and it's a main course. So, it has set this. So, last thing we want to do is the image.
36:49
So uh we go to media image, we enable AI automator, we use the LLM video to media image. So this can extract images from uh a video using a prompt and it has the
37:03
audio track as well. So you could actually ask it like cut out where he says this or something like that. So cut out where the whole dish is seen and looking sulfury. We're going to use as prompt uh cut out oh sorry one image that's
37:23
important you can of course set this to unlimited or something and it will create multiple medias for you if you write something else there but we just want one image and you have to choose image uh as the media type under
37:36
advanced settings we also did batch job here then again and then we're just going to go ahead and save and this is actually even more complex than the other uh video to
37:48
text uh thing. Uh so what this does is it's doing the same thing. It's getting all a raster of all the key frames uh and it's looking at the and looking for candidates based on the prompt you wrote. So right now it's looking for
38:07
okay it looks like the dish can probably be here the whole dish. And then it takes those candidates and creates another raster from uh where this is where it takes the average key frame distance in time and goes back that much
38:24
time and goes forward that much time. So that's usually around two to 10 seconds depending on how much compression you want in a video and getting all the frames there and looking at those frames
38:35
and then taking a final decision. And the reason for that is because when you're looking at a video that's zooming in or zooming out, you don't see that it's a lot of artifacts and blur and and
38:47
things like this. But there might actually just be two, three frames in such a scene where uh the dish is fully in focus and the rest is just zooming in or being out of focus and so on. So it's important to get the actual right frame
39:02
where it looks kind of uh uh in focus at at least. So that's why we're doing this. So let's see here if it found a good image. Yeah, that looks okay, I guess.
39:17
Um, and yeah, that's it actually. So what can have what we do now, of course, is we set this to published. This you could also use the automator for. There is actually a workflows automator that
39:30
could look at all the results and see does this does this sound like a recipe. So it's not, you know, saying that you should put gasoline in the food or something like that. And when it's allowing it, it can auto publish. But
39:42
right now, we want to do it ourselves. So, we save it. And then, let's see here. We have our results. So, we have the Gordon Ramsay's Hell's Kitchen Butternut Squash Lobster is a recipe. That's a main course. Seafood seasonal
39:57
dinner party with a nice little uh introduction here. It has set number of servings, cooking time, preparation time, difficulty. It has set the ingredients.
40:10
Um, not all of these are said in the video. So, it's trying to estimate actually. But that is probably similar to how uh someone that looked at the video and would have to write it down would do as well, unless you actually go
40:24
ahead then and do the recipe. That's that's of course what you really should be doing. But if you if you have this scenario where you have 500 videos and you maybe have a couple of you know days
40:36
to finish this then that's not going to work. You can just do all these stitches and it has given uh directions as well.
40:48
Um and that is it. Thank you for watching.
