0:01
hi there Marcus here today's workflow is about migration so what we're going to do is we're going to migrate out in this case WordPress site but you can take any uh any site basically any web page and
0:14
migrate that to Drupal without writing a single line of code and setting it up in around 30 minutes and we're not just going to migrate we're also going to improve the contact so we're what we're going to do we're going to take
0:27
something that's fairly unstructured or could be completely structure WordPress uses uh uses Gutenberg blocks so it is has some kind of structure right but you could think of this as a WYSIWYG page so it's a
0:42
uh review site about Berlin restaurant and I'm I'm going to move this now into Drupal uh in a very structured way so I'm going to show you there's a lot of field here so there's a field for opening hours uh that's using the office
0:56
hours module there's a address field that can show uh the place on a map right and we're we're going to add this during the migration as well so anyone that used uh
1:10
that has done database migration or used the migration module knows that this would be a lot of work and sometimes it's not even worth it you know might even make more sense to just copy and paste the stuff look up data so in this
1:24
case for instance the office hours it it just says like this from Tuesday to Saturday 1830 into 2300 and adding this then into the office hours module I can show you uh
1:37
via the office hours module is a lot of work because you have to fill in this right for every day and so on so what we have done here we have set up a migration in 30 minutes that moved this and put it into this uh complex field
1:52
types and fill them out so you can see it creates graphs create custom Fields creates you know telephone website uh it creates a Google map figures out exactly where it is doesn't hallucinate and fills out this opening hours gets
2:09
the images into a gallery and so on so a lot of the things that you actually need to do for a migration uh job so this is what I'm going to show you now okay so what we have here is a lot of structure field is using um some of
2:29
the core modules but it's also using external modules so this is the office hours module for instance this is the address mod uh module when we come to um ratings there
2:43
is the uh custom field modules and there's the chart modules and this this chart is a bad example you don't you're never going to show that in our review but I just wanted to show you how you
2:55
can migrate that as well uh what we are migrating for from is currently then a WordPress website but uh for this migration to work it doesn't really matter as long as it's a website that is available somehow that somehow that you
3:14
can scrap basically so of course this is just possible for simple migration not for migration where the values never show up on the website right there you have to still use the normal Drupal
3:26
migration style I'm talking about content migrations um in this case where you can even do it from static web pages right you don't even have to have a you know database to database or something
3:39
because it's just scraping for the migration in this case um what is special and what I will show you in this case then is that this is uh this is uh a WordPress as you can see
3:53
wordpress.com but this is a private post so all the posts that fictionally fiction now needs to be migrat uh migrated or private which means that we need a session to be able to see them so I'm going to show how that works as well there are three modules that we need for
4:11
this so there is the AI Interpolator simple crawler that's the one that's just going to get the content there is a scraping bot version as well so if you actually need JavaScript to render the web page that you're using then you you
4:25
have to use the scraping B but the simple crawler comes with a lot of advantages as well which is uh it's a lot faster because it's just service side rendering it's just scraping using guzzle uh you can run it from your local
4:39
machine as well right so you could have a local website you want to migrate from to a local website in Drupal and still be able to scrape it then we use the OpenAI that's going to help us with a lot of the complex field types so using AI to populate this basically but since
4:57
it's using context it's not going to help hallucinate um I will come more to that later and then the extractor which has this is nothing has nothing to do with AI but it's very helpful it can
5:10
extract stuff like images into the image field from HTML for instance uh so I have already installed this I have already set that up so it's basically under OpenAI settings you have to get the OpenAI key uh this will be gone by the way
5:29
temporary key same with Google Places config you have to put a Google Places API key which I have done and I will explain that is for one specific field right so how we should go uh about this
5:44
now is basically that we have to start with uh having a old page link which is a a link and this is basically just the only thing that we're going to have to give except for the title so that is the uh this
6:04
link which is then the link to the old website right um as you can see it's uh it's using Wordpress which uses like Gutenberg in the end so it is very uh unformatted um I mean there is like the Gutenberg blocks right but still it it
6:25
you can think of it almost like a CKEditor field so it is very un formatted and we want to make that formatted uh so we're adding this link then we just take one we say it's
6:38
required uh for now so the things that we add to migrate we're going to remove later right so we set external link we don't need link text we just save this and then we need to create a old page HTML as well and format a text text
6:56
format long right so this is basically going to scrape the website then uh and this takes full HTML and because I've installed the AI Interpolator mod modules in this case
7:09
simple crawler there's rules here and this checkbox shows up so I can enable AI Interpolator and I can choose the uh the rule simple crawler this is what we're going to do so this crawls a web page and stores it and we're running
7:25
this in base mode uh we're saying that we want the old page link I will come back to what base mode advanced mode is but we want the old page link so not the website that we're showing on the the
7:38
actual page right and uh we need some advanced settings because as I said this is behind uh uh authentication so what we're going to do now is we're going to go to network reload this and we're going to go to the main page because
7:55
that's going to have cookies here and there is a cookie called WordPress logged in that takes care of the session right so we're just going to copy this goes to custom cookies paste the value
8:07
and then I need the key as well so I just said WordPress logged in equals and this will log me in maybe I will just search Chrome user agent here to get the latest agent here to make sure that word press thinks that I'm
8:24
actual browser as well uh then this part is very important when we're talking about migration I'm just going to keep it actually at article segmentation now but uh the reason this module existed
8:38
from the beginning is not for M migration purposes it's rather for uh contextualizing for uh AI so there's a surf plugin for instance so you can search find links scrape those and use that context as truth because if you
8:55
don't have truth the AI will hallucinate it will come up with stuff that's just not true right but if you give it context like given this context answer this question it will answer true
9:07
truthfully um but it has some modes that are very uh helpful I think in this case so there's raw dump where you can basically get the whole HTML structure there is
9:19
article segmentation which if we go back to the web page it's a Firefox plugin that's open source we're using that can basically get the actual content right so it doesn't care about menus and stuff
9:32
but then what is also uh interesting I think if you're doing migrations is that there is HTML selector so if say that this page had a gallery outside of the images that's inside the article and we just wanted to move the gallery then we
9:48
could find a selector for the gallery and dump that and then use the image extractor only on those right uh but now I'm going to use everything I'm I'm going to set a lower
10:00
interpolated weight because this has to happen first before anything else right uh we have to have the website content before we can start populating other fields normally if you do actual migration I mean you would use uh cron
10:17
right uh so use the Cron job to run all the AI jobs and all the uh crawling and stuff uh but I want to make it graphically uh nice now so I'm going to use the batch API basically and then there is the direct that just saves but that can time out you know you get serve
10:36
time outs uh so this uses the JavaScript batch API instead so I'm going to save this then we're going to start adding the other thing that we need to fill out so we need address right uh and this
10:50
uses the address module uh and what we're going to do here we're going to enable the AI Interpolator we're going to use the rule text address finder base mode uh and we're going to use uh the title
11:02
because the title is the restaurant name right and then we can uh write from the context text find the restaurant we're looking for context and then we can just add Berlin because my uh my whole
11:16
um my whole web page is about restaurants in Berlin in this case if it wasn't about restaurants in Berlin what you could do is you could create a uh basically an extra field there in
11:29
between that reads the HTML page and asks it hey what are the restaurants we're looking for answer with the restaurant's name and the the city name basically then we're going to go into advanced settings we're going to do batch we're going to do GPT-4 Turbo in
11:47
this case we just save this uh and here for that we doesn't matter that much because that's just dependent on title that we have to fill out anyway next thing is charted ratings which is a
12:00
stupid thing to have but I just wanted to show how it's possible to do this right so this is using the charts uh module we're going to once again enable AI Interpolator use OpenAI charts from
12:13
text base mode we're going to use the old page HTML and then we're going to say from uh the context page uh context page use the uh dishes and their ratings as a chart so what this is going to go
12:37
in uh oh by the way context then that means basically the clean text from the base field so that's the HTML without its uh HTML TX you can use raw context if the HTML TX is important so what this will do is we'll go down here and look
12:54
at the rating here and the name of this uh dish and create a a a short from that basically uh as values probably we should write uh and this we do a batch we need to use GPT-4 Turbo preview for this or
13:16
GPT-4 anything because it's pretty complex uh we just have a higher weight than the fetching so this runs later um the thing is we probably have to set up the default type as well so I'm just
13:33
going to say that we use Highcharts and we use a bar chart in this case uh and I know that the range is from 0 to five the rating right so we set all that we just save so that set up
13:50
then cine that's a taxonomy term so we have this one I can show you the terms it's just beef alal koser vegan and so on that's of course good uh to figure out for people that are interested in something specific and this you could do any tonomy term right um we set this
14:08
unlimited because it can be uh you know beef and poultry for instance uh once again enable AI Interpolator use the O OpenAI taxonomy rule base mode we can do in this case as well take the old page HTML and then we write wait I'm
14:27
just going to copy this actually because that's a pretty good um pretty good prompt there from the placeholder so choose uh we don't have to do up to based on the context text choose category from the content okay so
14:49
PR fits the text based on the context text menu choose category from the category context that fits the text text category option value options com context is then
15:02
the text what is this is going to do is going to go down to the menu read it it's in German but that doesn't matter for OpenAI uh read through it and figure out okay here's fish then we set
15:14
fish I'm not sure if I had a fish actually but calans that that's ve uh tail right so it's going to add that to uh beef probably uh and so on so it's going to C I and it's going to figure out that it's uh if
15:32
there is a vegan and a vegetarian option as well right automatically once again batch once again we do GPT-4 Turbo preview and we save this then we have the email field and in
15:47
this case we don't have to use OpenAI or be so uh extreme we can just use the email extractor so this is basically a regex that looks for emails in the text so there is a email some somewhere here to contact them there right so it's just
16:02
going to find this and set this so we just say this is old page HTML batch higher value and that's it Gallery so what we want to do there is actually a gallery and a header image so what we want to do is that the gallery gets uh all images except the
16:21
first one so it gets all the other images here and the header images takes the first one so how we're going to do this is basic basically that first of all we're going to set uh a minimum image
16:33
Dimension and this is important because if if there happen to be an icon or something here it would be stored other way so now it will make sure that the image has to be over 500 500 otherwise
16:46
it doesn't save it right enable AI Interpolator use the image extractor on the old page HTML batch and then we're going to set offset that we start from the second
16:59
right we don't set the amount we just want everything after that and we save this and then we go to the header image we're going to do the exact same thing here uh 500 *
17:11
500 enable AI Interpolator on the old page HTML and we turn on batch uh but in this case we just get one and we don't set offset right so it just takes the first one
17:26
um and then we come to the oh something I want to do first by the way the old page HTML old page link should not be shown right and we're going to add it as to the as a tab here as well we just call it old
17:48
page because this we can basically when the migration is finished we can just remove this if we want right because after that you can start adding manually if you want of course you can still use the AI anyway so we're done with this
18:02
menu I'm going to skip for now it's pretty complex I have to think actually how to solve that uh opening hours is uh using office hours and that just is the same thing and here I'm
18:17
going to show you really the power of AI because this is going to show every day and so on right uh when it's open if you look in the text it just says fleetingly here somewhere uh there uh I fil f ctin is open Tuesday to
18:33
Saturday 18:30 to 2300 right so that's what it says that's the only context that it has so we're just going to enable this take the old page HTML uh we're just going to copy this because this is actually pretty
18:49
good I think let's see here based on the Contex can you return the opening hours that you can find Advanced we said batch which takes orable preview 100 so it happens after the text is gotten um ratings is a custom field this
19:10
also makes no sense but I just wanted to show you so like custom field uh is a module where you can have multiple values or you can create your own complex field type basically and this then has the reviews the rating and
19:24
the the um yeah what did we call the last yeah text let's see name reviews rating so this comes from here right name review uh sorry review and rating right so we just enable the AI Interpolator there is one for custom field we can use
19:48
base field here as well uh and here we have to do multiple prompts so this uses like one shot learning where you give examples on how you want it but also a over uh view like
20:04
a uh how do you call it like a you know a parent prompt basically uh So based on the context figure out the dishes that got reviewed and extract the dish name and trans to English so this I stored in German but maybe we want it uh
20:33
in English in this case uh so you see La for instance uh or or Gish there so that's um we need to translate this to English right so uh extra translate to English also also
20:54
get the numeric rating and their review text right so that's uh the [Music] uh that's the context let's see here so we're just doing that and then we have advanced settings where we actually have to give per each first
21:20
batch then give and one sentence how name should be filled out so this is the dish name uh fil trout so this is just example of a name fil trout with uh bamal right uh the textual
21:41
review this is the textual review this dish tasted awesome and was really good I don't know uh the numeric uh uh review inte rating and this we give example three we use GPT-4 Vision turbo preview in this
22:07
case save it uh yeah I can say there is Vision preview so we could even do this from a image we don't even need to do it from a an website but this in this case we're talking about migrations right but if
22:21
you ever need to read the text for instance from uh migration so here this is just just going to copy here so style is just the style of the restaurant basically so we're just going to open up this on Old page HTML do
22:40
batch uh we're just going to copy here based on the content choose category from the category based from the content text about the restant uh bad GPT-4 Turbo preview
23:00
save that's it and then this is the only thing we're going to AI generate so we might want a summary on preview pages right and this doesn't really exist here we could do what Drupal doesn't you know cut it off somewhere but this is usually never good
23:18
summary so we we're actually going to write a summary here instead so we just take AI Interpolator OpenAI text longan so this takes text to text based on a prompt and in this case what we can do
23:31
is we can take Advance mode because we want the title in here as well so what we can say here is uh for the restaurant uh and then we take node title here so in this case we can use multiple
23:51
field to fill in the prompt right no title uh with the following context write a summary that is one paragraph with two sentences up to 150 characters context and this we just then take the uh old HTML page HTML we called
24:23
it right and we're just going to take the value so we have it there do Advan settings again batch we can use GPT-3.5 it's usually good at summarization and it's cheaper and faster uh then we have a
24:42
telephone uh and in this case there's actually no telephone extractors but this is still smarter because I'm going to show you something here so from this whole context reading this text you're going to realize this is about Germany
24:55
right and then there is somewhere uh let's see if we can find a phone number there supposed to be here there's a local phone number right the phone uh field wants to store this with country
25:08
code so this can figure this out based on the context right so we're just going to copy this prompt here so based on the Contex return all telephone numbers listed with a country code at it and we're going to take the old page HTML
25:23
once again 100 and we're going to take GPT-4 Turbo previews save um and then we have text review uh well I mean yeah we can use OpenAI text long and we take the O page uh So based on
25:50
the context text only get the texture or all review without any images or the menu right I I think I hope this works I mean this of course you could do programmatically in five minutes writing two lines of code
26:18
but as I said I don't want to use any code in this migration so GPT-4 Turbo preview save and last but not least website and in this case it's the same as email uh sorry disabled enable and we
26:36
take Link extractor so there is a OpenAI link you can promp and try to figure out stuff if it's complex but in this case we're just going to extract the link basically uh and batch and we don't want
26:50
any links that links to images and stuff of course so we're just going to save that and now we're going to do a migration so I Fe Fe that's the title right and the other thing we need to fill in oh I have to reload I forgot I
27:10
and the old page is this link and we click save and this is going to start working now so the first thing it's going to do is get the HTML and then it's going to start processing all these different fields and try to fill them
27:22
out and there might be be that we need to do some fine tuning but this is basically the general idea to do the migration um and then after you're done with this you can just remove all the
27:37
helter fields that we had basically uh and that's it the migration is done so uh of course this is as you can see slower than MySQL queries but we're talking about migrations where you have
27:56
maybe up to a couple of thousand websites with 10 different entities or something up after that of course it makes sense to set up migration script and go through that work but uh up until
28:09
then I think this is actually a nice way of doing the migration um let's see when it's finished if we actually get some usable results and truthful as well is important of
28:39
course so there we go so we have a header image the actual getting of the text didn't work so we have to figure that out uh categories it said so it has poultry vegan vegetarian it's a fine dining place which is correct it has SK
28:57
Michelin stuff right uh and then you can see it has actually added the ratings on a chart uh we can just check one has four rating four right uh and here we have them on the
29:23
custom field type and you see they have translated it's leaked it's FAL so we translated to English like we told it to this doesn't show the ratings I can show you that just so you don't think the ratings are gone I just didn't want to
29:37
show them so you can see that the rating is hidden right so if I click edit there's the rating is going to be there click contact data email filled in website filled in there's a address there's a opening hour that has been set
29:52
correctly right so 183 to 2300 and then there's gallery this looks ugly I know I didn't want to install a gallery module but it is the four images that comes after you can click on them
30:04
and you have the image right uh and that's basically the idea of it all that's how you can do a simple migration for websites and even improve the content right um so what didn't work now then
30:23
was the actual uh uh the actual text here so the text review didn't really work and the problem be between uh because the problem with this is basically that it would probably have wanted to split it up in multiple Fields
30:46
due to how the interpolator works so in this case I guess uh using uh code would be better uh I will think about it and think we can get the text menu out as well it shouldn't actually be a problem uh but you get anyway a summary here as
31:04
well right to sentences what we talked about and I can just show you under ratings that the ratings are set in this custom field type as well so what we could do now is just remove these two
31:16
Fields remove this tab and the migration is done right we have moved from the old to the new oh telephone didn't work so we probably have to so here's the thing with prompts they they are with AI to
31:30
start with it's not deterministic it can answer different depending on settings and so on but there's a lot of prompt engineering going into this as well so you have to Perfect The Prompt so uh we
31:43
we can try to do that actually just to show you how it works then with a phone uh so you get a a better idea on how long time this takes to set up because I just copied all the placeholder stuff right So based on the
31:58
context text find all potential telephone numbers and return them with the country code added with a plus sign maybe we need to be very very Boer so we just save this and then you can just resave as long as this is not filled out it's going to try to get it again right so
32:25
I'm just going to try to resave and it should run one job now uh oh it didn't actually run that's maybe no here it is I maybe had this set to direct right yeah because it didn't use the batch ah okay so that's why it
32:41
failed sorry I forgot to set this to batch which meant direct runs before the batch API runs so the HTML didn't exist so there was actually nothing wrong with my prompt I just did another error there
32:55
so that was the reason so so the telephone is filled out now we have the results and that's it basically so what we spent half an hour to set up this migration and now you could just you know use the CV CSV
33:08
import a module that exists for instance a great module where you can you know create entities from this and that's it you have your migration done or also of course in the case you maybe just have 100 websites you could
33:21
even copy and paste the links just manually so that's it thank you for for watching
