--- url: 'http://workflows-of-ai.com/blog/contextualize-using-files' title: 'Contextualize using files' author: name: 'Marcus Johansson' url: 'http://workflows-of-ai.com/about/marcus-johansson' date: '2024-04-23T11:36:13+02:00' updated: '2026-10-10T12:14:04+02:00' type: article summary: 'Use the Unstructured module and AI Interpolator to turn PDF, Excel and image files into HTML, table fields or Markdown in Drupal.' tags: - 'Marcus Johansson' - Workflows - 'Drupal AI' image: 'http://workflows-of-ai.com/sites/default/files/previews/bb5822451822c398.jpg' published: true og: site_name: 'Workflows of AI' 'image:width': '1280' 'image:height': '720' 'image:alt': 'Contextualize using files' --- Drupal AI # Contextualize using files Turn PDFs, Excel sheets and photos into HTML, tables or Markdown in Drupal with the Unstructured module. Marcus Johansson April 23, 2024 ยท 1 min read ## About this video Unstructured IO is a service that takes structured files like PDF files, PowerPoints, Excel sheets, Word documents and Markdown files and generates structure into text, Markdown or HTML. It can also take unstructured files like photos of tables or text and turn those into structured content. Introducing a way to improve your site search, contextualize for AI or create webpages using the HTML mode inside Drupal. The service can be self-hosted or be set up using Unstructured SaaS. The module: The documentation: Unstructured website: *This video is from before the AI Interpolator module became AI Automators, part of the [AI module](https://www.drupal.org/project/ai). Module and menu names have changed since, and some steps look different today.* ## Watch the video [Watch on YouTube: Contextualize using files](https://www.youtube.com/watch?v=a-EK-j8zdTg) ### Video chapters - 0:00 What Unstructured IO does - 1:06 The Unstructured module - 2:43 Three ways to use it - 6:25 Installing the modules - 7:45 PDF to HTML field - 9:06 Output format and strategy - 11:37 API key and first test - 14:28 Excel to Table Field - 16:25 Receipt photo to Markdown - 17:48 Choosing a hi\_res model ## Transcript YouTube's automatic captions, lightly corrected. Click a line to jump to that moment in the video. ### Transcript - 0:00 hi there everyone Marcus here today I'm going to Showcase a new member of the AI Interpolator family called unstructured This is a helper module to contextualize files basically so - 0:14 unstructured IO is a service that you can either use their software as a service they have a free tier so you can try it out for I think it's 1,000 requests per month uh but it's also - 0:29 available then as a open source server that you can host yourself which makes it very very powerful so what it is is a server that takes unstructured files uh you can see some examples here HTML PDF - 0:43 CSV it does Excel for instance as well uh images PowerPoints markdown uh and some other stuff um and structures this then into a way uh where AI can basically uh use it for for whatever different - 1:04 request you might think of right um so the module exists here uh it's called unstructured uh it's not necessarily just for AI Interpolator there is a subm module for AI Interpolator the idea is - 1:21 later that this is of course extremely powerful for search API where you can index you know PowerPoints index uh uh images and so on based on text inside images and stuff like this uh so if - 1:38 someone wants to co- maintain and help out with that please let me know I will eventually come to it later but I don't have all the time in the world um so in this case I'm just going to showcase the - 1:51 AI Interpolator part of it then so I'm going to Showcase it in three different ways I'm not going to set up chains by the way in this case I think you can realize how powerful this is in terms of - 2:05 chains because setting up something as simple as um you might have seen the example where I uploaded a video and it generates a web page take screenshots of that right doing the same with a PDF or - 2:18 a PowerPoint is extremely simple with this and the OpenAI plugin uh so this opens up for a lot of uh powerful uh workflows but in this case I'm just going to show how to plainly use the - 2:35 module uh and I think you understand if you look at the other videos how you can actually change this into really powerful things then I'm going to Showcase it in three - 2:46 different ways uh because uh there's a lot of ways how you can use this uh first of all the actual module itself it has um it has three different rules but within those rules - 3:02 there are a lot of settings as well I will just show you here so for you if you don't know about it the AI Interpolator is fully documented how to get started how to write your own - 3:14 plugins for it all the available plugins but also all the available rules so I think we're up to like 80 rules or something now uh and if you go then to any of the plugins and in this case - 3:28 unstructured you can see the rules available uh so I'm going to Showcase two of those in this case the one I'm not going to Showcase is that you can also take a PDF or a image that has - 3:41 multiple images inside it so so to speak so if you have an image of a poster where it's text and some small images uh you can basically upload this file and get all those images out into - 3:54 image Fields then uh but this I'm not going to Showcase I'm going to showcase file to string and file to tables uh so the first showcase I'm going to do is take this - 4:07 PDF and this I'm not going to make ready for AI in this case what I want to do instead is I want to take this uh file that has that is fairly structured right it has like header one header two you - 4:20 know paragraphs uh text uh links a table and a image and I want to generate actual HTML from this so we're going to take a formatted field and generate HTML from this um the other two examples then - 4:39 is I'm going to take a a very simple Excel file this can be a lot more complex of course doesn't really matter uh and I'm going to use this files to tables and this takes the table field - 4:52 module that uh is a third party uh field type that is a structured way of sh tables it's awesome module if you want to show tables right so this structuralize things as well this is - 5:06 also probably not as useful for AI uh and then then the third thing I'm going to show is I'm going to take a image of a receipt and this is a a photo of our receipt right so it's as you can - 5:20 see it's not perfectly structured uh the text inside the receipt is of course not uh written in text text format in the jpeg uh as it is in a PDF for Excel file then so this is - 5:35 a lot harder and in this case it's going to use AI modules to extract this uh just so you know this can also run normal OCR mod models which is AI it's deep learning right but they have a - 5:50 generative AI model that's a lot more complex than I'm going to Showcase uh in this case as well uh um so before I start showcasing how to set up this I just a friendly - 6:04 reminder there is workflows Das of- ai.com uh where you can find all the videos I've produced so far uh in a searchable and filterable format of course you can use YouTube you can - 6:17 subscribe to YouTube if you like the videos uh but if you prefer to actually find them this website exists so let's get started well let's go to the module uh um so we're just going to do - 6:33 ddev in this case because I'm using ddev as the local environment and composer require same with the table field here so because um this is not the um uh just a plugin for AI Interpolator - 6:51 the the composer file in this case will not download AI Interpolator by default the core module so if you don't have it and want to do this example just click here on AI Interpolator right uh and - 7:05 then you just click here and you just get this as well right so that is getting the modules we just log into our um Clean Slate Drupal in this case as always the only thing I - 7:23 have done is install the admin toolbar we just oh sorry not appearance extend I just need to uh turn on the different modules now so we need unstructured and we need table field in this case - 7:39 right so table field is for this specific Excel example but the first example we're going to start with is uh PDF to HTML I just disable a lot of stuff here I don't care - 7:55 about uh and then we're going to remove the default body field going to create a new field and this is going to be the file we upload that we're going to use as the file that we want context from - 8:09 well in this case HTML then and this is a file that we're going to call the PDF file uh and we allow the extension PDF of course here then you could make a workflow where you take a lot of - 8:26 different types of unstructured filed and it always fills in a in this case formatted field type uh field right so that's what we're going to create now a format the text text format long uh that - 8:41 we called HTML representation so this going to be the HTML representation of this PDF we use full HTML uh and then we enable the AI Interpolator that's the important thing right we're - 8:55 going to use this rule unstructured API file to string we're going to use base mode uh and we're going to base it on the PDF file right then under advanced settings what is important then if we - 9:08 want to actually create HTML there is this output format where you can say you want pure text you want markdown or you want HTML pure text and markdown is very good for AI context HTML is not but if - 9:23 you actually want like in this case you don't care about the AI uh perspective of it you just want a representation of the PDF in HTML you just choose HTML these are the different types that - 9:38 uh unstructured uh splits up text into so you can actually choose that uh oh we just want images uh uh for instance that's possible if if you would just want that right uh then you can also - 9:53 split it into multiple Fields so if you set uh up here then to Unlimited uh so you can create multiple text fields and you want to create one text field per page you can split it up per - 10:07 page or even per element this creates a lot of fields yes so you know so this is every every paragraph every header and so on becomes its own field in that case - 10:18 uh then you can set strategy AO is very good if you have structured uh uh I mean if you have digital Fields so not image representation of a receipt like we're going to do later then AO is usually - 10:33 good enough but what you have here then is you have fast which is just trying to do it as fast as possible you can do OCR only as far as I know they use Tesseract in the background there uh or you can - 10:46 choose hi\_res and then you can choose one of the models I'm going to come to this later uh we're going to have checked uh extract images so this means that if there are images in the PDF or image - 11:00 uh and this doesn't work for word for some reason just so you know that so it's not going to extract images from word then you need to convert it into a PDF if you want images it's going to - 11:09 extract that and put it into the WYSIWYG field in this case so CKEditor right uh there is this checkbox that's just to save API credential so bypass on perfect match that means if if this file that I - 11:23 upload a Wasa HTML file there's no reason for me to send it to their API to create HTML it is already HTML same for Markdown same for text so we're just going to save this I just realized I - 11:37 forgot something so we need to go to configuration to set up unstructured as well under web services unstructured config so here there are two ways to do it then so you can use the uh software - 11:49 as a service then you need to get the API key uh and that's what I'm going to do in this case then um so I'm just G to copy and paste my API key here but of course if you self host it you can just - 12:04 reference your self host if you use API key there of course you can H use it as well so that's it then we're going to try this PDF to HTML we just call the title PDF um we have this under - 12:20 examples this PDF was what what we wanted to do and this is going to fill out the HTML representation now I we quick save that and there we go I'm just going to open - 12:35 this file as well so you actually can see it side by side here uh so you see here this is a title the title is not centered in CKEditor so this doesn't look the same but - 12:51 then there's a header one to get started paragraph header two header three right uh and then a tap able for you and in this case oh actually you know what we're going to do here you see an - 13:06 example that I used auto that actually created the fastest version in the fastest version it doesn't try to figure out how a table works it just cuts it out as a image so you can see here this - 13:17 is actually image which is okay for HTML I guess if you're not going to send it through to AI it doesn't actually matter but let's just change this let's just just retry this so we're going to go to - 13:31 our PDF to HTML and just change this setting then and we're instead going to use hi\_res uh and we use the default hi\_res model I will come to the models later we just - 13:44 go back to our content so with AI Interpolator it basically works that uh if nothing is filled out it will run again so we're just going to remove this now so it runs uh reruns uh this job and - 14:00 hopefully we're going to get a table now that is a actual yeah an actual table right so here you can see then it has basically generated A visual representation of the PDF but in CK - 14:15 editor and since it is CKEditor you can also click edit here and start you know editing it if you want to do that as well um so that was the first example then I'm just going to show you how - 14:29 table Field Works then so we take uh and I'm this is Excel file then so table Fields by itself can import CSV files but in this case you can use Excel files or you could even - 14:42 use um stuff like um oh let's call this Excel to table field uh you could use stuff like someone wrote a table on an nain because you got a smart idea during dinner or something like that you you - 14:59 can import this into an actual structured table right so once again we need a file in this case XL uh and this I it X LS X I think going to create another field and in this case - 15:18 then uh we're going to do a table field so we do table \[Music\] representation continue enable AI Interpolator in this case we use this rule unstructured API file to table uh usace mode used Excel as the - 15:37 base field and we use strategy hi\_res I I should probably put this as default on table actually because you really need it uh and save the settings then we just go to add content Excel to table field - 15:54 call it Excel and we just need to upload wait let me see if that was the right one yeah test table xlsx we just save this and this table field hopefully now will be filled out - 16:11 uh it even takes the name of the actual table then so if you have multiple fields in Excel and you set up that you can add multiple table Fields it would create this as - 16:21 well um and then let's go to the third example so we're going to uh do if image to text in this casage right and just do this again so this actually in this case - 16:49 doesn't take a image field I might add that rule later but in this case you have to use the fire field because this is just a base field used as context right but yeah uh if you need it to be - 17:02 able to create from image field add that ticket if it's in a hurry for you otherwise I will come to that sooner or later right uh so this we call the image source uh and we allow jpeg PNG - 17:17 GI jpeg like that I guess as well save that and we create a new field that we take a plain text field in this case I think uh and we're going to do a markdown representation in this case - 17:33 right so we take enable AI Interpolator on structure file to string again base mode image search Source right so that's our source uh we're going to take markdown as output format and we're - 17:46 going to use strategy hi\_res and there are different models then for doing the hi\_res strategy uh by default it uses Detectron2 uh with ONNX so this is a Facebook mod model that is - 18:02 very fast so if you're in a hurry uh this is usually good enough when you have a a actual digital file but in this case where there's a image representation usually to be honest OCR - 18:19 works even better than detect from uh then there is Yolo uh that is a lot slower it's fairly good and there's a quantized version that's a little bit better but then there is also chipper - 18:35 and chipper is a beta model that comes from uh unstructured that I would say is superior to all of them because it's not super slow it takes like depending on image size it takes 20 - 18:50 30 seconds if you use their software as a service uh if you host it locally it depends on your machine basically right uh so we're going to use shop shoer shipper in this case uh to try to get - 19:05 all the information then from this uh receipt so we're just going to upload this and receip the jpeg this is now going to take a little bit of time because uh as I said it's a little bit - 19:22 more complex let's just open this receipt so we have it here uh so the import important thing if you're going to use this now in a uh AI capacity is that all the data is there uh also - 19:38 maybe understand that merchant ID is connected to this number terminal ID to this number and so on or code um so let's see here so we get this and you can see here - 19:52 it's using markdown even to create that markdown table in this case right so merchant ID is going to this terminal ID is connected to this uh number is connected to this and so on this is a - 20:05 ag3 in this case purchase becomes right so uh this for me is awesome I worked with OCR before but it it doesn't become perfectly structured so I would say this is really awesome for this use case uh - 20:24 they have done an incredible job here and of course yeah a might have mentioned markdown is something OpenAI knows about something that the biggest uh generative llms knows about - 20:38 so you can do really really powerful things when you have gotten this structured um yeah so that's it I hope I even if I didn't show any AI chains here I hope I got some uh ideas into your - 20:55 brain how you can use it on your uh project and also it goes without saying then since you can host this yourself uh and you could use a large language model that you host yourself as - 21:08 well you could use this for internal documents without you know sending it to a third party as well so yeah thank you for watching Filed under [Workflows](/blog?category=6) [Drupal AI](/blog?category=2) Parts of this content were produced by AI. This site has not assessed whether a label is required for this content. How was AI used?The video and the introduction are Marcus's own, first published on workflows-of-ai.com. AI wrote the subtitle, the summary and the video chapters from the transcript, and corrected misheard names in YouTube's automatic captions. Marcus Johansson Tinkers with AI and Drupal [About Marcus](/about/marcus-johansson)[Drupal.org profile](https://www.drupal.org/u/marcus_johansson)[Drupal AI](https://www.drupal.org/project/ai) ## The post in three sentences The Unstructured module sends files to Unstructured, as SaaS or self-hosted, and gets back text, Markdown or HTML. A PDF becomes HTML in CKEditor, an Excel file fills a Table Field, and a photo of a receipt becomes a Markdown table with the hi\_res Chipper model. Output format, strategy, splitting and image extraction settings are explained along the way.