In the last video, called "Elevate Drupal Media module document field with AI elements" , I showed you how to improve the metadata for a PDF or Word document.
This is a follow-up video where we do the same for audio and images.
The idea is basically that the end user does not have to do much or any editorial work to get the media entities well described and indexable by Search API.
This makes it possible to show improved previews and makes it easy for both the editors and the end users to find the correct results.
This video is from before the AI Interpolator module became AI Automators, part of the AI module . Module and menu names have changed since, and some steps look different today.
0:01
hey there um so I just did a video about how you can improve the media entity when it comes to the document media type I just want to show you how to do the same for audio and to
0:17
do the same for a image media so you get that idea as well um this is building up on the other video so if you haven't seen the other video please look at that first um to get some idea what I was
0:30
doing there um so where I will start is that for the Audio I need this module AI interor Deepgram for the image we're going to just going to use um uh OpenAI St 3 in this case in
0:47
theory we also have the DreamStudio plugin so if you prefer Stable Diffusion you can use that but I will just use it and this is already installed in the old video but for the audio we need the Deep
0:58
CR so we will just go in here and run ddev to uh get this module um so we're downloading the Deepgram module and then we will just go here to extend I I can just preview so this is just
1:19
some audio about difference between TCP and UDP whenever a computer wants to communicate with another computer the communication so that's just a previous someone talking basically about what's
1:31
the difference between TCP and UDP for the image I will just Showcase with this Lamborghini car um right so uh let's see we have ai interpolator Deepgram install oh sorry
1:47
download it we install it so we go to configuration web services Deepgram config to set up this you have to log into to Deepgram I'm logged in here so I'm just going to create a uh
2:00
API key test API key uh well can set the owner set so it expires later tomorrow so I don't forget to remove it and someone of you steals something here and I just create a key here just
2:19
going to copy this key and save it here so that's to basically authenticate to Deepgram so we have Deepgram set up now oh that's not have so many things here so what we're going to do is take the media
2:33
type audio in this case now then and improve it I can just start by uploading the audio so you see there's no meta Fields uh connected on this at all uh we just going to take this TCP versus UDP
2:49
upload it and there's yeah as you can see there's no metadata basically so we need to add Fields here so we're going to do uh I'm just going to open open the document field so I can copy some of the
3:02
prompts actually we're going to reuse existing Fields right so uh we can actually use the media document uh sorry no no no media document I mean the original text I mean it's it's a
3:18
transcription in this case so maybe we should call transcription but I will just reuse this uh and in this case I'm going to use Deepgram transcribe right and we're going to use
3:32
uh I guess in this case it's well let just take General there's no monologue there and we use the audio file because that's the file field we have right and uh we just set this to batch mode this
3:48
is the first thing we need to do so this is just going to replicate basically what we did with the other things so we're going to add search words here enable in AI interpolator we can just
3:59
copy this prompt that we have here oh sorry that's the help text here from uh oh no right right what we want to do is we want to start with the long description that's how we did it in the
4:14
old one so we create a long description and then we generate everything from that right so uh I'm just going to take the long description here so we do OpenAI text long from the
4:28
original text and we're just creating a summary I'm doing this very fast now you can look at the other video to understand exactly what I'm doing because this is basically uh doing the
4:41
same thing over again so we use GPT-4 turbo preview here then we're going to uh oh the search word actually got copied there so we have to set it [Music] up so we're just going
4:56
to basically uh create that short string here so we take that from the long description we have a prom 10 to 15 15 search words um this has to be 10 1002 because it has to happen after we have
5:12
transcribed it and made a summary uh turbo preview we save that and we're going to also do the short description so we're so we're basically going to have the say metadata Fields
5:30
here right uh enable that we take OpenAI text long from the long description we use as context and we're going to create a two sentence summary same thing here this has to have
5:45
1002 batch GPT-4 turbo preview and we save that and the last thing is we're going to do the uh header image prompt that creates the header image right or preview image that's what I I
6:07
wrote that a little bit wrong so OpenAI string long take from the long description generate a prompt for a image 2002 batch Modo GPT-4 Turbo preview save it and lastly but not least oh I hope so
6:29
supposed to reuse we're going to do um the preview image enable AI Interpolator d e from the header image prompt we just do the context and this one we need to do 103 because it has to happen after the
6:50
header image is done to a little bit better quality save it and then oh sorry sorry we go back to the media just going to re um resave this and you will see that
7:07
all these fields will basically be automatically filled out now then by the A interpolator and then we're going to do the same for image well we're going to do it a little bit different because for
7:18
the image of course we don't need to create a preview image the preview image is just the actual image um we also don't necessarily need to uh have a base text field we can create
7:34
the long uh description directly from the image right because the image itself doesn't uh doesn't have any text in it so this might take some time while we do that we can actually start right away
7:53
with the image just to set it up the same way so we have a scene how it gets set up uh so in this case we're going to start with long description that's still running enable
8:06
AI interpolator but in this case we're uh going to do let's see here um text from docu oh right right so this is a little bit special case because uh the AI interpolator forces you to use in
8:21
context unless then you use the advanced settings uh Advanced uh mode which requires that you have the token module installed so I'm going to install that now and I'm going to show you how I can
8:34
do this so I just reload because the long description is there on the image we enable theator and then we can do uh oh this doesn't do advanced mode yes it does OpenAI text long advanced
8:52
mode uh because what we can do now instead in this case we don't really care about uh context of some other text right so we can just write in the prompt based on the image I just still need to
9:10
see how I wrote The Long description so we we can still copy this but we're not going to copy the context So based on the uh image create a summary of to say one paragraph describing the image
9:26
right uh and then it's important that we use one of the uh Vision models so we're going to do GPT-4 Vision preview and then we can say that we use the image field
9:37
as the actual image we upload right set batch mode and save this is then going to take the uploaded image and create a description of it and then we can just do the same thing that we did before we
9:50
do create a short description here just copy that one so P AI text long from long description and this just has to have one higher so oh 101 in this case GPT 4 Turbo I
10:16
want save it and then we also get the search words in the case with the search words I think it actually makes more sense to uh for AI to look at the image because the long description might just
10:31
describe it very abstractly while the search word you might want to have very specific Keys like what colors and stuff right um so because of that we actually are going to base that on the image in
10:45
this case so I just copy this but I'm not uh let's see here oh right right no no we have to uh base it on the long description so this is something that will will come in the next version of
11:01
the OpenAI module that you can use short text to to uh get from the um get from the image you can't do that actually at the moment so it only works currently with um no it does work
11:19
sorry I'm I'm I'm not I'm just not thinking right now sorry here so what we want to do is sorry Advance mode So based on the image I don't even know my own model my own code sorry
11:36
based on the image suggests 10 to 15 search words that are unique descriptive and important when searching for this document single out colors and forms no not single out colors and forms
11:58
are important let's right like that and this can be uh this can be 100 basically because it's based on the image right which does exist so we're going to do a GPT-4 vision preview in this case because
12:15
we once again we need a vision model do it on image uh then preview image we don't need header image we don't need original text we don't need right in this case so
12:27
this should actually be enough so I'm just going to close this now so we're going to start with this one that we did before the audio field and you see that the whole transcription is
12:39
there uh there's a long description and there's a short description about TCP UDP right uh and of course TCP and UDP it's important search words in this case but also a lot of other important search
12:54
words uh yeah visualizing TCP UDP is probably not the easiest let's see what it did here yeah okay uh yeah we I should probably add to the prompt that it shouldn't try to do text because it's
13:09
really bad at doing text still but you see this like a network it's kind of symbolizing youp so let's just do the last thing here now then we're going to upload our
13:21
image of our uh Lamborghini here right uh Lambo and then all this should just hopefully be filled out so the search word the short description the long description so let's see if that works
13:37
as well and then um soon there's going to come plugins to do this on video as well well first of all Deepgram soon in the next version it's going to be able to take video as input and if you have
13:52
something like FFmpeg running on your server it's going to be able to convert this and uh get the trans transcription of of the video but uh after that there are models
14:07
that can read video already so hopefully this will be integrated soon so you can see here if we click edit that it has done a long description a short description and also some search words
14:22
red sport car right that's what it is Sleek body work and so on so yeah that's it so video is not there yet but if you want to have a automated process where you get
14:38
descriptions uh where you get search words or even preview images in those cases where it's not the image from this uh get go then uh this is how you do it automized so thank you for
14:52
watching
