0:00
hey there Marcus here um today we're going to do a little bit more complex workflow than usual so what we're going to do is we're going to try to recreate Metacritic and how how it works but
0:11
we're just going to use Drupal AI to do it very quickly uh first of all I'm I'm just going to create the data structure and how to fill the data in this case so I'm
0:23
not going to show you how to style it how to create the views and so on uh the loog logic for aggre ating values together you can use views agregate for that and so on I'm not going to show
0:34
that part there's a lot of videos showing how Drupal Works in general so you can look at those uh for this use case I'm just going to show you how you can use Drupal to
0:46
basically uh fill all the data that you need so for people that don't know Metacritic it's a awesome website for basically figuring out if a game is good or great or worthless and so on
1:00
uh same for movies TV show music and I don't know they might have other stuff as well that's not here in the menu so how it works is compared to a normal reviewing website it Aggregates all the
1:13
reviews and normalizes so if if one review site might have one to five as the rating and another might not even have actual numeric rating they kind of normalized that to 0 to 100 and then uh
1:29
as a aage of those so you can figure out if something is really great depend uh depending on a lot of reviews and not just one single reviewer right um and this of course means if something gets
1:43
like close to 100 I think Salk Arena time has 100 for instance which makes sense it's one of the greatest games ever right that means it's awesome uh while if something has a low rating or a
1:57
average rating it might have been that one reviewer gave it a very good rating but the others didn't right um and for this workflow we're going to use um well I will say first of
2:11
all so I said Drupal in the beginning we're going to build this in Drupal using the AI Interpolator that's what all my videos are about if you don't know drol aw some CMS cmf uh it gets you
2:22
started very quickly to set up a website especially data heavy website uh data structure heavy website uh it's open source it's free you anything I show you you can just start
2:35
doing after I showed it and you can set up a copy of exactly what I do so the modules we're going to do uh use sorry and that I already installed is then the AI Interpolator OpenAI uh so we're going to
2:48
use that basically to um get a summarization of uh oh I forgot to say what I'm going to do I'm not going to do this on games and so on I have some uh Berlin restaurants here
3:01
that I'm going to do it on so uh say that I want to create Metacritic for but for restaurants in this case right so we're going to use it to summarize uh a short couple of sentences about the
3:13
restaurant uh then uh yeah what it's also going to do what's more important then it's going to normalize the rating so do semantic analysis uh we're going to use AI
3:24
interpolator Ser so that creates search results from a search word um in this case uh what it's going to do it's just going to create a search uh result uh for the general search term so
3:39
I can create a summary uh like I was talking about without actually having to copy and paste or write something myself then we're going to use ScrapingBot ScrapingBot is for scraping websites uh
3:52
what's unique with ScrapingBot compared to using curl or guzzle or something uh is that it's can run headless so it it or sorry it can run a browser that can run against headless websites which
4:06
means that a website that requires JavaScript so if it's written on react or angular or something it can actually render this so there is a simple crawler if you don't want to pay ScrapingBot
4:17
but that um it has advantages because it can run locally and it's a lot faster of course but it also has a lot of uh or has this big disadvantage that it's not an actual browser and you can't run it
4:30
from different countries and so on unless you set up Proxes yourself uh the last one is AI Interpolator screenshot so we're going to take a screenshot also of the websites we
4:42
review and this could be good for uh proving that they actually wrote what they did because maybe the article goes down maybe they change the article right so you have a screenshot of how it
4:54
looked exactly when you did the the normalization of the rating uh just to Showcase it I'm going to use the address module so you don't have to type in address for every restaurant
5:07
it's going to find that by itself uh there is another uh video where you can see how to actually map this on Google Maps and so on but in this case we're just going to create the address
5:19
structured token module is needed to have have token for advanced prompting uh in in Drupal I'm not actually sure if we're goingon to need that but it's good that you're aware of it as well so you
5:34
can do a lot more advanced prompting where you can use many fields together in a prompt instead of just using one field as the context so let's get started I'm just
5:47
going to log in then uh so as I said uh so this is running locally it's just a base Drupal setup with these modules uh I have the admin tool bar here to make the toolbar nice as well
6:01
I've also set up all the API keys so that you do on the configuration web services all the ScrapingBot Ser ScreenshotOne except for OpenAI because it's using the OpenAI module so
6:11
it's OpenAI settings to set that up um and that's it basically so this is the only thing I have done got in the modules using composer and um then set up the API keys
6:26
so let's get started then so we're going to add two different typee of context content types in this case we're going to use uh we're going to have a content type that's the restaurant and that's
6:36
the overall restaurant and then this will have different reviews connected to it right so we start with the restaurants uh rounds and this will have a title field that's called
6:51
name wait name um and here you there's a lot of settings that you would normally care about if you were setting up the website end to endend how to do that in Drupal there's multiple
7:06
there's thousands of videos on YouTube you can search so I remove the body field because I'm not going to create that General article what I will add first is basically uh search field so
7:21
this is going to be a formated text um formatted uh oh we have to give it a label we call it search word so this is basically what we need to search for to find it the address and also to find
7:37
some about text right uh and we just set the full HTML oh sorry I forgot one thing there as well I'm going to set this required because that's one of two field that you
7:48
will have to fill out and the rest you should not really have to fill out so the next thing we want to do is uh add a address field uh sorry I forget this restront
8:02
address so this is going to get filled out automatically now so we're going to use uh we're going to turn off that we need first name middle name last name because these are not people instead
8:15
organization name the restaurant name is required right and then we're going to use enable AI Interpolator and we're going to use the rule that's called text address finder and this uses base mode
8:26
and it uses the search word and I'm just going copy this so basically what this is uh saying is from the search word uh oh sorry there I copy too much just this
8:40
uh from the search word that is the search word that we add find up place in um using Google places so that's why we add the API key and get the address and store it basically uh we're going to use
8:56
batch mode in this case we use GPT 4 so batch mode is basically so you can see uh the different AI jobs running one and after another so if you just have one job that's very fast this is not
9:10
required because you can use direct so it's just when you click save on the saving the entity it's just going to happen but you might have time out and so on on server so you want to split it
9:21
up into a queue where you see it happening right gp4 is just because it's better at understanding stuff which is important in this case um so we do search for right um and then we're going to create
9:36
the next field which is going to be uh links and this is going to be search links so we're going just going to take the first three uh uh links that we find and we're going to scrape those and
9:52
we're going to use that as context to create a summary right so we enable AI Interpolator in this case as well we do Ser to link this has a lot of settings uh but the most important is what do you
10:06
use as the base field so that's search word and what is important uh otherwise is of course if you're on a specific Market you should maybe not search from us or in English so you can set all that
10:19
up but that's not so important in this case once again we do uh batch set 100 so in this case the weight is going to be important because we need to do this in three steps basically
10:32
first we need to find the links right uh oh there's one thing I forgot there search result okay no that's set automatically right um so we're going to find the links second step is that we're
10:43
going to scrape it third step is we're going to use the scrape data to write a summary so because of this weight is important so the next step is that we're going to scrape it right so to do this
10:55
we use formatted text text format long and the these are uh basically HTML bodies we can call it and this is important that we have uh at least three I mean I just said
11:10
unlimited now so because there's going to be three links uh we have to have as many um bodies to scrape from right set this to full HTML enable AI Interpolator and we're using ScrapingBot crawler in
11:26
this case base Mode still and we're going to use the search link as the uh as the base field so that's what we're using to scrape we're going to use Chrome in this case that's important
11:36
right um that we're using a browser can set weight for Network maybe someone set it up in react really poorly or so on and it takes time before it actually renders this of course takes longer to
11:50
run but that's okay in this case what's important is that we use uh article segmentation as calller mode so you can do raw dump to get the whole web website but we don't care about the menus and
12:00
stuff you can use HTML selector like jQuery to just select a specific part but in this case we're using the Firefox readability Library so that's the um thing in Firefox if you ever went to
12:13
article and you just wanted to see the text of the article this can figure it out so in this case this has to happen after we have the linked right uh and we're going to use batch mode again so
12:25
we're using 101 a little bit higher than 100 so it happens later and then we're going to have the last thing which is the description um and we're going to use text format long for this as well and
12:44
this can be one uh and we're going to enable so we set this to full HTML we're going to enable H AI Interpolator we're going to use the OpenAI text long in this
12:55
case uh we're going to use bace mode and no in this case I think we use token actually because we don't want to run through them one and one we want all of them right so this is the advanced mode
13:10
then so we're going to go to nodes and find our uh HTML bodies field and we're just going to copy the whole thing oh I have to press here first before I copy it and this is going to be the context
13:24
then right so this is going to be a um this is going to be all the bodies together in one Big Blob and then we just write here based on the following data generate a one
13:41
paragraph three sentence uh description of the rest restaurant and then we're just going to copy the title in here because that's the restaurant title right uh uh let's write context actually based on
14:04
the following context and then it's going to create this right uh for this we're going to use Turbo 16k I think to save some money because that's usually kind of good at writing small
14:19
descriptions it hopefully has the context links that's that you're is needed for reading these three articles it depends on how long they are of course worst case you have to use um
14:30
like gp4 turbo preview for instance that's I think is 32k or we can use stupid it4 just so I don't get stuck on this and it's too I get a warning message that the context L stays too
14:42
long right so we're going to set batch mode and this has to be 102 because this is the last thing that happens right then and we're going to save this and we can start by trying a
14:55
restaurant now then so I'm going to add a restaurant uh and what we're going to do is use Mustafa's Gemüse Kebap so it's one of the most popular derer places in the world
15:10
uh it's for I I think personally a little bit overrated but if you're a tourist it might make sense to visit it if you go to Berlin so you see there's a lot of fields here that's just empty I'm
15:23
just not going to fill those out those are going to be filled out now then by AI so get that description in the end and we get the address basically so if I click save now you see it's going to
15:35
start working here nicely what I can do in a new tab while we're doing this I'm just going to set the manage display of the restaurant so we don't have to see the
15:46
HTML we don't care about the search links or the search word not this links as well so we're just going to show the restaurant address and the description on the output of it right
15:59
so this is going to take some time now um especially finding out the address can take some time for sure um so now it's finished with the first the second then the third one is uh should
16:18
be scraping of course uh which will take some time because it's actually going to the website make sure that they have finished rendering scrapes it and stores it and then uh last thing now it's going
16:29
to write a description then of one paragraph if everything works out as it should be uh okay so this output was not perfect so um what you can do here basically is that
16:45
you remove the description here and resave then it's going to run this job again so this um is one of the reasons why it's still in beta I'm trying to uh basically get better stability so I'm I
17:01
I'm not going to stop this uh worklow here so it is a little bit realistic to the current status of it so this could happen to you and then you know what you can do as
17:14
well uh so it got a description it got the restaurant address right uh and this I know miring them 32 it's the correct address so what we're going to do now is to create the second thing then which is
17:31
the reviews uh and yeah what we need to do in this case is so because we use content types title is required so I I'm just going to keep it there we going to have to manually
17:51
fill out something basically so we remove the body once again so in this case what we basically we need to give is a link so this is the link that we're going to get the review from it just
18:05
needs to be one it's a required field because it needs to be filled out and then this is going to scrape of course so we take um formatted text text formatted long and HTML body in this
18:22
case we call it because it's only one enable the an ScrapingBot and it's using the link same thing as before we use batch mode get the art actual article and nothing
18:40
else uh and now then we're going to set a number on this we're going to set a rating so we just set rating so it's going to read through uh this re view now and set a rating on it right there's
18:54
of course a lot of other stuff we could get here that I'm not going to show show you using the anle layer so you could get the title uh of the actual uh URL we going to um you could
19:09
get the host name so you know which place it is that reviewing it so you could group those together so you could see a view like what's what's the typical review of this web page and so
19:21
on uh but I'm not going to show in this case so we're do minimum maximum 0 to 100 we're going to use OpenAI integer uh base mode is okay in this case we're going to use the a h HTML
19:37
body and what we can do here is we can just copy this because this is basically the example placeholder is pretty okay here based on the content text add a sentiment rating between Min and Max so
19:50
you can see here Min and Max is the numeric value basically uh of the minimum that we set here so this is just going to be replaced by 0 and 100 where zero means really negative sentiment and
20:01
Max means really great sentiment answer with a full number and context is then this HDM body this is of course not a perfect prompt because if you want it to all respond
20:16
similarly um you need to give a longer prompt with a lot more context like maybe give some background who this person that actually sets the center senent is what give examples on what
20:31
good and bad sentiment is in this context and so on but I'm not going to do that I'm just going to do the simplest here this case we definitely need GPT-4 because it's a lot better at semantic uh
20:45
anal analysis we need to set 1001 because this has to happen after we scraped it in this case we can give an OpenAI role as well so you have seen this as well you are a an editor of a
21:02
restant meta view website that tries to normalize reviews into a rating of 0 to 100 right so this is this is also important to give some background who the person is right in this case it's
21:21
important when you write a summary and so on is probably not that important unless you want it in a specific language I can also say OpenAI moderation I made it possible to turn
21:32
this off there's never a reason to turn this off what this does is it checks against the moderation endpoint that OpenAI has uh beforehand to check that what you're going to send is actually
21:43
not against their policies uh because what could happen when we automate everything so there's stuff one after another so take the description part right it searches for M
21:54
Mustafa Gus about Berlin but somehow by mistake it might um find I don't know something about that what's happening in Israel right now which might write something that
22:11
goes against the policy of what you're allowed to prompt into OpenAI and if you do this often enough you actually lose your your uh access to it so just keep this on because this will basically
22:23
if that happens their moderation end point will say that this is not the you will get a warning message in Drupal that this got stuck on the moderation queue and you don't get the result
22:34
basically because of this I will show as well that I will create a image in this case um because so it's file upload images we want to take a screenshot as well so we have a screenshot saved of it
22:52
right um so there's once again a lot of CMS settings not Import in our case we don't want the alt field we enable the AI Interpolator take the rule ScreenshotOne image uh and we can say width and height
23:10
here of it uh but in this case we're just going to care about width so we can take one 1280 is enough we don't have to make it bigger but we're going to do full
23:21
page uh so this is going to take a full page screenshot what's cool with ScreenshotOne and the reason I'm using using them instead of a lot of other services and
23:30
why I create a module for it is that they remove the cookie banners which is super important because otherwise you just get a screenshot of a cookie Banner basically it's using the link based
23:41
field in this case we don't have to set a wait but we set batch mode anyway save it and then we're going to be able to create a review as well so we're just going to take uh this we have
23:56
to set the title so I just copy this and that's basically what we need to fill in the rest is going to be filled in automatically it's going to set a rating it's going to create a screen sh
24:06
right um okay one field I forgot now is this has to be connected to the restaurant of course so you have to use a reference field in Drupal but once again that has nothing to do with the AI
24:17
interpolator that's something you can figure out yourself by watching other videos how how Drupal Works basically uh if we wanted a representative image of the restaurant
24:30
as well we could use OpenAI or or dream Studio plugin to create stuff as well so now you see it's showing the HTML body that's what I removed the last time that's not
24:42
something we actually care about so I'm just going to remove this and I'm going to show you by the way or I did show you uh that it was scraping scraping stuff right from the Sur
24:56
results so um I will just show rating and screenshot right so we just remove the links we can just we don't have to show oh sorry this is what I'm looking for so we
25:12
just going to show a thumbnail and this if you click on it we can see the file so let's reload so it got rating 85 because Berlin's best k out this is a very positive review then there's a
25:26
screenshot here you can click on and you see you have the screenshot of the website we can take another one just to show you that this is working I found another one that is far far away from
25:38
my H list so this is similar to what I said it's kind of overrated so we're just oh sorry I copied the title first which is what I wanted to do just copy that and then
25:52
copy the URL and we're going to save this and hopefully the rating should be well under 50 I guess at least it's it's not a super critical review but it's definitely not saying that it's Berlin's
26:05
best Don know that is what some people says about it right yeah so you see rating 25 pretty low uh and a screenshot that you have and then you can I just can show you in edit you can
26:20
see that it also store the actual review completely as it is right um and now of course there's a lot of steps in Drupal you need to connect this to the restaurant you need to create views
26:34
you need to design it but you already have the grounds for creating your restaurant meta critic website in this case without a lot of editorial work so yeah that's
26:51
it um yeah just feel free to start using it and happy usage thank you for watching
