0:01
hey guys Marcus here um so I'm going to Showcase a workflow today that's like workflow Inception basically what we're going to do is we're going to uh scrape our whole web page so that's something I
0:15
added to the scraping but you can actually scrape a whole web page where it starts going down deep down into the web page and get everything uh and use that as a context
0:24
and what I thought I would do today is basically to take uh the whole documentation of the AI Interpolator all the rules scrape that and then create a a form where you can ask hey I want to
0:37
create this workflow is that possible using the AI Interpolator and it hopefully will answer like yeah you can set it up using these configurations um but before I start
0:50
that I would just take a couple of minutes and talk about uh stuff that happened in my personal and uh my uh professional life so I have actually been very busy even
1:03
if you haven't seen videos in a while now you can see here in May so the last couple of weeks I've been pushing a lot there's a lot of new features uh in AI Interpolator um I've been looking into
1:17
other AI stuff as well been pushing stuff um and I I just didn't have time to actually create videos for this yet so I'm going to create videos for it um it's still a lot of it is still lacking
1:30
in documentation as well so this is coming as well uh but I just want to talk about the reason why I can be so productive now basically is these guys um so I joined a company called FreelyGive uh
1:48
and as you can see here they are basically a company that specializes in the combination Drupal and CRM so they have like a a native Drupal CRM him uh they're very focused on that and they've
2:03
been building that for uh the longest time uh it's also interesting because they they like they found Drupal as the natural platform for what they were doing uh because of its extremely
2:19
flexible nature in terms of what you can do with Fields databases structure and so on uh which is very similar to why I took a decision in my old company that we should use Drupal for uh the AI
2:35
chaining because it was pretty obvious pretty fast even if LangChain and LangChain JS existed for Python and OJs building the whole structure around something that you can do no code so you
2:50
can be a site Builder and set up uh chains that would take a lot of time even in using something like Django together with LangChain would just take a lot lot of time for me to build so
3:02
Drupal was actually the Natural Choice there uh so that's very interesting basically they they saw the same thing and they have also been looking heavily into AI you can see on the the web page
3:15
you're looking at that they are focusing a lot on AI um so so I met Jamie in the slack channels drup of slack channels for AI so that's basically how I met them Jamie
3:27
is one of the directors then at uh FreelyGive uh and they gave me this fantastic opportunity basically to uh help them with AI but also while doing this help the community in large so they are True
3:45
Believers in open source like I am I you know open source is really important for me um they they have worked a lot with AI so if you used like the search AP AI module that's from freely
4:03
give uh or partially at least from FreelyGive um and they have done a lot of you know Rag chatbots and stuff like this so they have already looked in a lot into the AI uh part and they have
4:18
customers then that are uh some of the bigger like UK train companies um and they have a case management system there so you can read more about the here uh called Journey call uh which is yeah
4:36
it's partly a delay repay system so you know the train was late you want money back and parts of this actually um there uh they they try to build AI into it so it actually can take decisions or
4:53
at least give a a certain help for the person taking the decision if money should be paid feedback and so on uh so they are actually doing this AI in real life production environment which is
5:08
extremely exciting um so yeah I I just wanted to say I'm extremely grateful for uh Jamie uh and the other director Andrew for giving me uh this opportunity uh it's I I think it's good
5:28
for me personally hopefully I can contribute back to the company so so um they have use for me outside of just contributing to the open source Community um and I will hopefully be
5:43
able to contribute a lot more to the open source Community not just AI Interpolator but in general a in AI uh this opens up a lot of opportunities so if you're interesting
5:56
interested in that you know just contact me or or Jamie would be the the person to contact if you if you want to talk about actual business uh and this also opens up the
6:09
credibility then maybe for the AI Interpolator that there is a company behind it that you can also pay for consultancy which means that it's not me doing it on my free time and me having
6:21
to figure out administrative how I can you know uh work for someone for a couple of days with all the things with bills and stuff because I never liked being a freelancer or or or having my
6:35
own company so this is a possibility now as well if you need help with some specific workflow if you need help with uh uh uh building some specific rules into AI or you know AI consultation in
6:49
general because there's a lot of smart people in this company uh and not just in in Drupal PHP but in other languages that works with AI uh you know please contact Jamie just look for FreelyGive
7:02
and you will find his contact data there [Music] um so yeah that's it once again I'm very uh grateful for this uh opportunity and let's start our workflow so before I show the example
7:23
just there is workflows of ai.com where you can filter search for all the videos I've done so far uh um so I started with doing what I said in the intro that I would do to scrape the workflows of AI
7:36
documentation that's on drupal.org drupal.org has scraping protection in line the modules I've created can bypass this I'm not going to show you how to do this uh if you ever want to do this you
7:50
have to figure it out yourself I would not recommend it the the protection is there for a reason so I'm not going to scrape drupal.org in this case uh what I'm going to scrape instead is I
8:03
just took a simple example here the Guzzle documentation right and we're going to ask a question about Guzzle uh and why is it important to scrape here by the way and why is it
8:13
important to have the Guzzle documentation before we ask the llm something so you would think with a model like OpenAI GPT-4o you know the latest coolest version it would be able
8:26
to answer any question about Guzzle uh and this it can do in 99.99% of the cases but the problem in the end is an llm an LM is not an answering machine LM is a text generator which means that it
8:49
might be not so credible in this case so as far as I know Guzzle doesn't exist as a product or as an even a name somewhere else but it might be in the token space when when it's generating uh your
9:06
sentences that you want as answers it might actually have a high temperature have high creativity and actually go somewhere else uh in this case it also makes sense
9:17
that you know as you can see we're on the latest stable version we we just want the latest stable version of the documentation so it doesn't answer something for uh Guzzle
9:29
or GLE 5 or Guzzle 4 we wanted to answer something about Guzzle version 7 and I'm sure anyone that's in Drupal might have noticed this with chat GPT that sometimes uh it gives back functions
9:44
from Drupal 7 even if you're asking it like hey in Drupal 10 how do you do this and this is the same thing it's going to a token space where maybe the word Drupal and the combination of whatever
9:56
you're asking is is weighing more than the Drupal 10 token um and that's why you have to in that case you know reminded oh this doesn't exist in Drupal 7 I'm uh sorry this doesn't exist in
10:10
Drupal 10 I meant Drupal 10 can you please answer how you do it there and then maybe it answers um and because of this always if you want a truthful answer you need
10:23
context so this uh the fancy word for this is rag retrieval augmented Generations where you you know if you create a chat but you can have embeddings which is a complex way of
10:34
searching with uh intent uh but this is a simple way of doing rag where you basically fetch a context and answer it uh and of course I will also mention if you're doing like a
10:48
chatbot or answering machine or something you you only need to get it one time usually I'm just setting this up so you can see shaning basically um so so what we're going to use in this
11:00
case we're going to use Simple crawler there's two modules there's the scraping button and there's the simple crawler uh so you get some idea what the difference between these are both of
11:12
them are for fetching web content uh ScrapingBot has the advantage first of all ScrapingBot is a service so it's actually calling API this means that if you want to run this
11:26
on some kind of data uh where you have privacy privacy concerns you should not use ScrapingBot first of all because it goes through their service I'm pretty sure it says in the terms and condition
11:37
that they are not spying on this but you know who knows you can't control that uh but what they have that's a huge Advantage is that they're running inside a browser so it opens Chrome uh and can
11:53
actually go to websites that you know are JavaScript based or headless or anything that that's not server side rendered it can actually render this it also has two other main advantages I
12:06
would say one thing you can run it from almost any country uh any country that matters for testing I would say on in terms of uh economical ma markets except China China has you know special rules
12:23
um and thirdly uh also there it can have a lot of different IPS right so so you can you can bypass yeah what I was talking about before basically if if you have rate
12:37
limiting to an IP drupal.org has more complex than this I've seen but if you just have rate limiting to IP can connect from multiple IPS um third and last Advantage when it comes to scraping
12:51
bot is that they have premium Proxes as well uh and to explain this Netflix is probably the best uh explanation here even if you're not going to scrape Netflix but uh I'm in
13:06
Germany sometimes I want to watch America Netflix show so I want to pretend I'm American right if you just get any VPN for this it's going to fail uh and the reason behind that is
13:18
that almost all of the vpns runs on a Cloud Server so that's how they how they can scale how they work right and the cloud servers have very defined IP ranges that they own which means that if
13:35
some VPN service set up like openvpn or some other service in USA I pay for this I go via this and it's set up in AVS or aure cloud or something right then my IP
13:49
will look like I'm coming from there and Netflix would see okay hey that's a server IP you're not allowed to watch that you're trying to you're trying to bypass our rules right
14:01
uh in the case with scraping but this might be interesting when it comes to LinkedIn for instance you want to scrape LinkedIn uh Twitter some certain other uh uh
14:12
Services what ScrapingBot has they have something called premium proxy it's actually 10 times as expensive or you use 10 times more credits but sometimes it might be worth it what it can do then
14:24
is basically scrape from a what looks like a home internet so it doesn't look like a server internet so that's three uh reasons to use it if you're going to use it you have this code here as well
14:37
that helps me a little bit because I get a free testing account from them so I can run automated tests basically um so you and you get 10% of the first month if you use this the other thing that
14:49
exist then is the simple crawler uh and this is very simple it's just using GLE uh it has a lot of options though so it has some some other advantages that you can do with scraping Bol so one
15:02
thing is it can byass bypass basic goth I guess scraping B can bypass basic go if you put it in the URL in theory but you don't want to do that of course uh if you care about privacy or security
15:16
because this ScrapingBot would see so you can do uh basic o you can put your own cookies in there you can pretend to be any user agent um and some other settings you have in there uh that are
15:28
pretty cool so uh what it can do then that's super important is it can and can only scrape server site rendered web pages with which this Guzzle documentation is so
15:42
that's what we're going to use in this case um lastly that might be an advantage and a disadvantage this runs from the server of course which means that if there's IP protection you know
15:54
you're not allowed to spam with from the same IP this is bad on the other hand if you want to migrate something locally you can actually do this because this runs
16:04
from your computer in that case so uh I you can see here I'm on a d local development environment if I had some old let's say I want to migrate from a WordPress site or a LEL site that is not
16:18
online but the project exists there uh so I can run it locally I can of course use this to scrape that website uh and then we're using OpenAI then so I I'm just going to start by
16:31
logging in here um everything is set up I have installed simple crawler I've installed the admin toolbar as you can see I've installed the token module and I have
16:42
installed OpenAI OpenAI have set up a a key under OpenAI settings as well so we're just going to start here by creating a content type uh that we call the Guzzle I have it there Guzzle
16:56
answering M machine uh I'm just clicking around here if you don't know droo you will have to look for information what I'm doing here somewhere else uh but soon I I will probably add a uh custom
17:10
entity for workflows a disposable one so more information comes about that later uh we're just going to start by adding The Source link so this is the link we're putting
17:22
in right this is the one that we want to scrape from external and disabled uh we save that so oh we should set that required as well and then we also want to ask a question right uh we
17:36
do that takes plain long and we call that uh Guzzle question and set that required so these are the two things we have to no let's do this because the title is going to
17:53
show let's use the title instead we just call this Guzzle question well it's 255 characters but that should be enough for my my questions then we create a format text
18:07
text format long which are the HTML Pages uh this is super important that you set unlimited here or set a high value at least what you don't want to do is that you're
18:23
oh sorry I haven't explained yet what I'm what I have added here so just so you know all these videos I'm not putting any production values into it I should probably do that but I don't
18:35
have the time to like coding more it's basically this is one part of me testing the actual module so I'm just turning on OBS and talking while I'm doing that and
18:45
that's that's that's how these videos comes to be uh but I think it's helpful at least in some degree to understand how how everything works so what I forgot to mention is there are
19:00
new rules to both these so before they could you know take a link and scrape the header or the main image or the web page what it can do now is that it can crawl so you can basically say hey I
19:15
want to craw the whole website and put this in in a lot of text Fields right so we have the entire website as um as uh the the the context because of course with the Guzzle I just think they have
19:31
like 10 15 P pages in the end uh so it's maybe possible you know I copy here click that copy there click there I could do that but if you have thousands of pages that you want to use for
19:44
context this sucks of course and because of this you have to put it unlimited this thing I added of course also adds a lot of context that you can put into the search API AI which you can do rag
19:57
chatbots and really cool stuff also there's another uh depth link rule that does the same it just doesn't store the actual content of the website instead it looks at other Links of specific
20:13
extensions that you can configure so say that you want to go to a web page and find all the PDFs that's on that web page and get the links to those and with those links you can use the extractor to
20:26
download them later so AI Interpolator extractor um that is possible as well but now we're use we're going to use uh this rule I'm going to show you so we turn on
20:37
full HTML and you understand how why unlimited is important here and we're going to use the simple depth crawler it would be ScrapingBot depth crawler than if you're using ScrapingBot uh oh one
20:49
more thing uh simple crawler has a release now with this scraping but it's still in depth because I need to fix some other stuff so if you want want to test it today in in scraping but you
21:00
have to get the de version advanced settings we're going to go we say depth two will be enough so this means that it's going to click link here and it finds another link well in
21:14
this case it's a internal link so that doesn't count but uh if it finds another link even deeper it's going to go two steps in this case uh don't set this very high unless you really want to
21:27
scrape a lot right include Source URL means that it gets this actual page as well so the first page not just the page that it scrapes host only means that if the host is docs.guzzlephp.org and it
21:41
links to something else like our cookie policy.com or something we're not going to start scraping that keep that checked unless you know what you're doing I'm going to come later you can add ring
21:53
expense stuff because there are valid reasons why not to do this because of course if you want to scrap PDFs like I talked about before and the same settings exist in this link stuff then
22:06
uh those might be on a blob storage that has a completely separate uh uh host name right so you can do that URL on top just puts the URL on top this is important if you're doing rag or
22:21
something and you want to to reference the URL where you got the answer from uh I we can keep that actually because then we can maybe ask it even to reference it in the prompt so we can see
22:34
it's correct uh body only means that it this is not talking about what it's going to scrape so the content it's going to get get it's rather that you're saying that only look for links within
22:47
the body tags so if there happens to be a link in the head tag it's not going to use that here is just your comma separated exclude pages and these are some normal stuff that you don't care
22:58
about like terms and condition cookie policies and stuff and then you have include patterns and exclude patterns and these are reg then that you can put in so in our case if you click here
23:10
there are actually links to 6.5 5.3 version five PDFs and stuff that we don't care about but it follows a pattern here where the latest version is under stable and we only want the
23:23
English version as well right so we can just copy that this should be some in the path right uh and this is then regex so I have to escape here so you need to know regex to use
23:37
this um you can leave empty if you want to include all a cool down means that it's basically waiting uh in between each uh each time it uh scrapes uh be nice there keep a cool down like half a
23:53
second or something you don't want to take down website by spamming them that's really not nice and uh if you check your logs for when Bots hit your website or non-malicious Bots you will
24:03
see that they do this they don't just spam you with if you see Google bot for instance unless you're a very big website they're not going to spam you with you know thousands of requests in a
24:14
minute or something like that uh if you want to pretend to be a user agent this is usually good because this is what many websites actually uh checks for when it comes to um is this
24:30
website uh or is this request from a browser or not right so we're just going to add the Chrome there here you see basic off custom headers I can add custom cookies we don't care about that
24:42
then we have the r crawler mode where we can say either we get the whole dump of the web page so we would get everything or or we can use HTML selector which is similar to jQuery
24:56
we're just looking for ID or classroom or something like that but in this case we use article segmentation uh which comes from the readability Library so the readability
25:05
Library comes from Mozilla Foundation uh you see it in the Firefox browser where you can basically go to our website and say hey I only want to look at the article I don't care about all the ads
25:16
and other menus and stuff I want to read the article cleaned out it's using the same thing so in this case it's just going to get this while these links and these links are menus it recognizes say
25:28
this and doesn't scrape that uh then we're going to use a weight of 95 so in this case direct could perhaps work in theory I think it actually might finish before 30 seconds uh because it's
25:48
like I said it's like 15 pages or something but of course if you're going to do some deeper scrape this will never work so instead what we're having is uh you can choose between these two qron
26:01
this is the most optimal one because there's no usually no timeout you might have PHP timeout but usually on Drush you don't have it uh batch mode it's a little bit special because what this
26:13
does it doesn't put all of the scraping into one batch it hooks into the AI interpolators batch and adds all these jobs basically where they're supposed to be so for each web page it basically has
26:27
whatever PHP your sorry web server timeout you have so we're going to try batch now because it's also very visible uh and then we're just going to have an answer so we just call that
26:42
answer uh and here is where OpenAI come in then so we're going to use OpenAI string long in this case uh we're going to use advanced mode because we have to take two different uh two different
26:56
context and one thing that's important for you to know with base mode as well so if we were to use context of the HTML Pages this means that actually it would split up each of them and run this
27:13
prompt for each of the pages which in this case would make no sense right so that's always how the base Field Works because that's how it have to work if you have four links before what we were
27:25
talking about scraping it should scrape all those four links L separately right so this is always how it works so when you want to say that I want to take all the HTML Pages as one big blobe instead
27:36
you always have to use the advanced mode and token so let's write our prompt so uh you are a Guzzle expert and have the Guzzle documentation in the context the user has a question about Guzzle you
28:04
will answer this by using the context the answer should be between two to four sentences long um please reference the [Music] URL that the that you used for the uh
28:34
answer all the URLs are visible at the top of the page uh top of each uh uh document documentation page question so this is the question then we get from node uh what do we call this we called
28:59
it Godel question I guess no question uh uh I'm completely lost now let's see if I go back here oh I didn't sorry I didn't add the question wait is this this is answer oh
29:23
man I I have a Misha so I'm going to add had the question here I thought I did Guzzle question it's even in my history let's see required field save okay that's weird so let's reload
29:43
here I have copied it luckily my um prompt uh so token question notes Guzzle question that's what we're looking for let's also put it like this this so it understand where this starts
30:00
and when this ends H context GLE documentation and we do the same here start stop yeah so regarding by the way I do test stuff of course uh before uh I'm showcasing this but these
30:27
are usually the first time I uh test it on complete clean environment that's why this is running and recording these workflows are very good actually for testing as well so I H get the field
30:40
HTML pages I set this to batch mode as well we used the new cool GPT-4o model that was released recently this weight has to be higher than the other I think I lowered it but anyway it's at 100 five
30:53
just to be sure and let's see well one thing I have to do now because I didn't add the question correct I want to ah now I remember we wanted to use the title let's re sorry I can't even keep
31:16
track of myself we remove this again we go back to this uh prompt uh sorry not HTML Pages I'm so utterly confused now answer with we go to the prompt and the question here is going to come from the
31:32
title that's what we said man it's not easy sometimes so uh we take the question here save it let's see now if we can add a gun Guzzle answering machine because then we
31:47
only need to add two things here which is good so I'm just going to copy this URL and then I will go to some page here and figure out something H how do I access headers that's a good
32:01
question how do I access the headers and I might have to you will see if if it cuts now it's because it took too long time let's see so I will talk through this I don't think it will take
32:19
long actually uh it will go through all the Links at the first level then the now it's the uh second level basically and then it goes one deep as well because the first is counted as zero if
32:32
it finds something it would now create another batch job here because of course it can't know beforehand how many jobs is in the second level at the beginning so that's why it might seem a little bit
32:47
funny because it goes back to zero and then starts again and goes back to zero and starts again but that's the way I could solve it now if someone has some good idea how to actually make this look
32:56
nicer please contacted me and what's happening now I think because it takes a long time it's actually connecting to the AI so it goes back to basically the normal AI Interpolator batch job queue
33:10
after it's finished getting all the the data when you're using batch by the way it's saves during the time so if it were to fail because batch is a little bit frail right you you're based on
33:24
JavaScript your internet can die and then it dies in The Middle and so on at least it will have stored everything up until that time then so uh oh a 404 not found let's try that
33:37
actually uh yeah this is a 44 not found so that's fine uh let's find the next one so we can see here Source it this is what it adds then yeah here's the next one
33:52
overview uh quick start and quick start is where I think we had the head stuff anyway there's a lot of content here let's do this if I go back here and go to manage displays I don't want to see
34:06
the HTML Pages here I just want to see my answer right so if I reload now uh okay you can retrieve header from the respond using has header Geth header and get headers for example response get
34:19
headers but it wasn't very good because it didn't actually answer the question it might be I'm just going to try a thing now so this this is something I've seen in general with uh gp4 do gp4 o
34:34
it's actually not great it's not as good as gp4 turbo uh and it might be connected to my prompt as well it it's a lot faster but it's not super great actually it might
34:48
be also please reference the URL you use for the URL the URL should be inside the two to four sentences so this can also be uh that it actually created two ansers so if I would have put unlimited
35:06
it was in the second there uh because basically if you're if you're asking it to generate many paragraphs it will generate those in new text Fields uh and this how it has to work
35:18
because if you're asking for instance a image generator to uh generate three images it of course has to be in three images you can't concatenate that with text there is a version where you can
35:29
always concatenate there's going to be option for this soon so this is uh a ticket I have uh to fix um or actually there's another guy that hopefully will fix it I will I will try to teach him
35:41
how everything works and maybe he can he can join and help in in the effort with the interpolator but um for now this is how it is so I'm just going to save and how it always works if
35:54
you're not happy with the answer because the prompt was not perfect you can just remove it and then it runs that uh particular job again so now it's not going to fetch the website again it has
36:04
fetched everything it's just going to run the AI job again so now we're trying with a little bit improved prompt engineering better model and see if we get the URL as
36:17
well no you can access the header from the geter meod respon if you need to check uh okay I'm going to go even harder at this because uh it's obviously not understanding the
36:31
UR should be inside the two four sentences uh the output should look like this uh uh sence one about the answer sentence two about the [Music] answer the URL wait we have to do fit
37:00
sentences it should of course be dots uh the URL to the answer from the documentation let's try this maybe this is a dead cause maybe it's uh oh no let's also do this I just reload top of
37:24
each documentation page with the title source so it does that connection in the token space as well because it's a source uh Source not URL right uh as you can see when this load
37:40
it says like this Source col uh remove and try last time and if this doesn't work I give up but you can see that the answer at least is correct and this is to get this correct this is
37:54
just prompt engineering but I don't want to B bore you with that to too much sometimes for the production stuff I do I sit um hours to get the right prompt yeah
38:07
finally uh part of the pi or seven respond object yeah but this was actually not correct why is well let's see maybe it's there as well get get header okay you can retrieve okay okay
38:25
no no no this is fine then so it's actually there as well so it could be in quick start or it wasn't maybe in quick start yeah it's explained there as well but I guess it's actually better
38:40
explained here complex headers accessing headers okay sorry the AI is right I'm wrong I just skimmed through the web page so you see AI is super good to have uh when you don't have time to actually
38:54
read something yourself uh so it did work well it gave back too much sentences actually but maybe it felt it needed to do that uh so yeah that's the workflow sorry
39:13
about the missteps but as you can see I'm just doing this as I'm trying out stuff uh and it's probably good anyway to see that it's not perfect that you will you know
39:27
follow into these problems yourself when you're setting it up probably so yeah thank you for watching
