--- url: 'http://workflows-of-ai.com/blog/finetune-stable-diffusion' title: 'Finetune Stable Diffusion' author: name: 'Marcus Johansson' url: 'http://workflows-of-ai.com/about/marcus-johansson' date: '2024-04-18T21:16:20+02:00' updated: '2026-10-10T12:14:04+02:00' type: article summary: 'Fine-tune Stable Diffusion from Drupal with Dreambooth on Runpod, host the model on Hugging Face and generate images with the AI Interpolator.' tags: - 'Marcus Johansson' - Workflows - 'Drupal AI' image: 'http://workflows-of-ai.com/sites/default/files/previews/d1a7748d8ba541a2.jpg' published: true og: site_name: 'Workflows of AI' 'image:width': '1280' 'image:height': '720' 'image:alt': 'Finetune Stable Diffusion' --- Drupal AI # Finetune Stable Diffusion Fine-tune Stable Diffusion on your own images from the Drupal UI, then generate new images with the model through the AI Interpolator. Marcus Johansson April 18, 2024 ยท 2 min read ## About this video This video will showcase how you can use Drupal as your end-to-end shop for taking a Stable Diffusion (text-to-image) model, fine-tuning it with a concept, like your own face for instance, and reusing that for generation. All done in Drupal. No Python or scripting is needed! This opens up a whole new world of personalization for you and your company. These models can then be used either to generate via the inference endpoints on Hugging Face, or you can download them to other providers or run them completely locally. The two services you need are Hugging Face (free) for hosting the model and inference, and Runpod to host GPU machines with enough juice to fine-tune. The price for an SD 1.4 model is around $0.5 and for an SD 2 model $1. If you sign up for Runpod and like my videos, please use the following link so I can get access to more credits to play around with even more. Links: Drupal Hugging Face module: Test my generated AirPods model: Runpod to credit me: Docker Hub: Repo for Docker: Hugging Face: *This video is from before the AI Interpolator module became AI Automators, part of the [AI module](https://www.drupal.org/project/ai). Module and menu names have changed since, and some steps look different today.* ## Watch the video [Watch on YouTube: Finetune Stable Diffusion](https://www.youtube.com/watch?v=RJbnIndOfVw) ### Video chapters - 0:00 Fine-tuning without a GPU - 2:18 How it works for you - 2:54 Runpod and the credit system - 8:21 Installing the Hugging Face module - 9:47 Hugging Face and Runpod tokens - 10:54 Creating the Dreambooth fine-tune - 13:27 Uploading and cropping images - 16:47 Testing the model on Hugging Face - 18:54 Generating images with AI Interpolator - 22:20 A dedicated inference endpoint ## Transcript YouTube's automatic captions, lightly corrected. Click a line to jump to that moment in the video. ### Transcript - 0:01 hey there Marcus here um so I've been on vacation start a new job and I've been sick so it's been a while uh but now I'm going to show you something uh completely new uh that it - 0:16 has some connections to AI Interpolator but it's it's a New Concept which is this part of the Hugging Face module which makes it possible to use Drupal uh not have a GPU at all and to - 0:31 be able to fine-tune something and take that fine-tune model and use it inside Drupal to generate stuff uh and this is completely inside the GUI no coding needed uh no server - 0:45 setup knowledge or anything like that needed um so the reason that I'm it's in the Hugging Face module is basically that anything that it generates will be hosted at Hugging Face so if you if - 1:03 you're not aware of these Concepts basically Hugging Face is uh you can think of it as GitHub for AI models so it's a repository where you can host uh your private AI models uh but it also - 1:16 has a lot of public AI models it has uh open source AI models and what we're going to do is we're going to take one of these models uh in this case Stable Diffusion uh any - 1:30 of the different versions which is a text to image model and teach it A New Concept so that's what you can see that I have done here so I have fin you on images of - 1:42 myself I've called myself Marcus Johansson JJJ so it's it's it's its own token because there is a famous Marcus Johansson NHL player so I needed to basically have my own token not to - 1:56 delute with the knowledge about him and then I can write something like that a photo on me in front of the Taj Mahal and it creates this right uh and I can use the AI Interpolator to also run this - 2:09 specific model and do it inside Drupal as well uh so that's basically what we're going to uh what I'm going to Showcase today so how this works for you as a user is basically you install huging - 2:25 face model uh module sorry there is a page you go to that you describe your New Concept you describe which base model you want to base it on you upload a lot of - 2:36 images uh you crop these to focus on whatever it is you're trying to uh generate you click on start finetuning and you wait some time and then it's going to show up on hugging - 2:50 face and then you can use AI Interpolator to generate images with this um so what's happening in the background then is this that when you click generate it's going to start a VM - 3:04 in Runpod and Runpod is just a cloud provider that's specializes in AI uh so it specializes in GPU servers so servers you most likely don't have available uh and that are freaky - 3:21 freakishly expensive and that is the other reason why I chose Runpod because Runpod has a credit system which means that you have to upload credits um to be able to use it - 3:34 because even though I'm doing open source and open source clearly states I'm not responsible for how you use my code right I don't want to have this scen scenario that someone looks at this - 3:47 video starts fine-tuning something there is some bug in there that I didn't know about uh and it just dies halfway through and they don't understand the concept of cloud at all so they don't - 4:02 understand that oh there was uh the garbage handling garbage collection didn't work here so there's still a server running and these servers as I said are frequently expensive and this - 4:13 person at the end of the month all of a sudden you know get a bill of $2 $3,000 I don't want that to happen so because of the credit system it means that you upload $10 for instance first - 4:25 case you lose $10 if there's a bug like this um but the module also has a very clear checkbox that you have to check that you're aware that this might cause costs - 4:36 and that you should also manually go in and check so uh the VMS are removed automatically otherwise you should remove them manually but anyway this creates this - 4:49 pod takes this Docker image as the image um and just so you know I do everything open source there is open source uh rep for this if you want to see what the docker image is doing so it's not you - 5:04 know stealing your images or something like that um or more importantly it's going to have the Hugging Face API token on there so I don't steal that of course uh so you can see that it's not doing - 5:17 that does the fine tuning when it's finished with the fine tuning takes this mod mod uh model and pushes it to the Hugging Face reple so that's what it's doing um and the advantage with this of course - 5:33 is that you you know you don't need to Learn Python a lot of this stuff is uh it's not very user friendly so a lot of the AI scientists which are not usually developers they are very - 5:48 very very very very good mathematicians uh uses python because that's usually a language scripting language that you learn and um and because of this they might write a - 6:05 script that you know generates this but they don't think about usability at all so because of this you kind of need to know python to do this otherwise there are some UI tools for it but most of it - 6:17 is anyway based on python at best maybe you can get a Docker image which you can just start um so in this case you don't have to think about it then plus you don't have to have the GPU server then - 6:31 uh there is one more thing before I'm going to show you how to set this up then and that there there is this affiliate link um I'm not in any way trying to get twitch on this then I wouldn't be doing - 6:44 open source that would be very very stupid but when it comes to most of the AI uh modules usually I can ask for like can I get a free account here just to try out some stuff and sometimes they - 6:57 give it to me so I don't lose any money in this case I lose a lot of money testing around and I want to do a lot more uh with run pods in terms of one fine-tuning so currently it's only doing - 7:11 image models I want to be able to do large language models which cost a lot more money because they require a lot of more uh memory better Hardware uh and the other thing I have - 7:24 done privately or in the company I used to work for is working with Runpod serverless which is a way where you can take a model host it but you only pay for when you actually use it so it it - 7:37 works similar to Amazon Lambda just that it has a GPU uh so I want to build open source stuff with Drupal around this as well it would be nice to not lose too much money - 7:47 on this so if you're anyway interested in using this and you don't mind then please use this uh link because from everything that you spend I will get 3% in credits so it's not money I get I can - 8:03 take this out as money but at least I don't lose as much money uh when I'm doing this open source stuff um so that's it let's get started then on how this works so let's start this then uh this - 8:24 is a plain Drupal standard profile admin toolbar install that's all we just need the Hugging Face module um I'm running D in this case but whatever local environment you have you - 8:39 just use composer to get it right uh then the Hugging Face module itself is just API module and this comes with sub modules that can use it so there's a augmentor um plugin for it a module for - 8:54 it but what we're going to do in this case we're going to use the Runpod uh fine-tuning module that is a module that's using it and the AI Interpolator Hugging Face module that's using it and - 9:05 these have dependencies that doesn't get loaded automatically so I will just get the base AI Interpolator in this case and also for the uh uh fine-tuning I need image widget crop to be able to uh - 9:21 specify what part of the image that I want to find Jun right then I just after getting this I click extend uh then there is Runpod fine tune I need to install an AI - 9:35 interpolator Hugging Face those two modules are what I need one for to fine tune and one to generate this is going to install a lot of dependencies as you see um just - 9:47 approve that uh and the next step is we have to go to web services Hugging Face config so we have to add our access token so we're able to both well save the model oh sorry I did I click new - 10:02 organization settings is what you want but also to upload uh well create the finetune but also to read that fine tune when we're using the a interpolator access - 10:13 token uh we create a new fine tune here uh fine tune temp I'm going to remove this after so you can't copy it and use mine uh we're going to show that copy paste it here save and we need to do the - 10:29 same same them for Runpod um it's important that you have money in your Runpod otherwise this will not work so you should fill up with like $10 there that's under billing I think uh anyway - 10:40 I'm going to click settings API key create one read and write same thing here it's going to be gone by the time you see this uh and this needs to be able to write of course because it needs - 10:53 to start servers then it's just under configuration content authoring fine tune Dreambooth uh this is going to have a fine tune models later and Dreambooth llms and so - 11:04 on as a sub uh uh directory but uh yeah if you can think of a better place to put it please let me know so because I have the token it knows that I my name space is Ivan boring Swedish comedian by - 11:18 the way and then I'm just going to uh here create airpods uh so what we're going to do we're going to take a lot of images of uh this is this is getting recorded in the middle - 11:31 of April so if you see this in the future this might be completely off but there's rumors of a airpod earphone from Apple that's going to have a display on the case some people have rendered - 11:43 images of this and I want to take those and be able to generate my own images of the so-called future airpod then visibility status I said to private uh no I actually will do this public so - 11:57 if you ever go there you can actually try this afterwards to see how how great it works table diffusion 2 we're going to base it on right now it's just 1.4 and two we're going to I'm going to add - 12:08 so you can add a lot more text to image models that's based on anything that Dreambooth then can uh find you on and this includes custom models as well because if I find you in this - 12:20 airpods and I realize I also want to fine tune on a a a future iPhone that might get uh generated but I want those two things in the same model so I can have the airpods with display and the - 12:33 iPhone next to each other uh I will be able to F tune that concept on top of the model I already did right and here we have to write something then that connects to it so I - 12:46 it's already stored here a phot of the in ear earphones so this basically adds the taxonomy of what this is right this is part of the taxonomy in your earphones and I just set that custom - 12:59 token to this because airpods already exists and I could of course do airpods with display but it wouldn't know be so certain if it's the airpods with display that I'm uploading or if it's the actual - 13:13 concept of airpods with display so because of this I generate a unique token I don't think this uh token or this word has ever existed inside the training data so that's why - 13:27 then um just just going to go here and upload a lot of images because I'm using Drupal here it doesn't have to be PNG 512 or 7 768 or whatever is the very specific things you need if you run this - 13:43 as a script you can just upload any type of image then image Styles will take care of making this so this will be the next step there is advanced settings if you're using Stable Diffusion 1.4 you - 13:55 could write uh RTX 309 in this case because you don't need such an expensive model you don't need so much vram uh and you can set something else there uh if you're going to use Stable Diffusion 3 - 14:10 was just released if this will be able to fine tune it probably needs even more stuff then there's a lot of uh detailed fine-tuning uh parameters you can set here you have to read about this if you - 14:24 if you're not happy with the result start reading about this and you might get better result basically anyway we will just leave it as this is as it is click to The Next - 14:35 Step which is basically the cropping then so we will make sure we just focus on the stuff this is probably not going to be the best results because sometimes as you can see the earphones are outside - 14:48 of the display case sometimes like this I have to choose if I want some text in there or not uh so if you want a perfect mod model you should should um fix the images in Photoshop first uh and in this - 15:05 case also everyone is guessing different you see this is a monochrome display this is a color display that's a square this is rectangle so on uh so uh this will probably not be the best - 15:18 result but I will just showcase this as a example of what you can do uh and when I am finished with those there is this check box that I need to check also on so this is from my - 15:33 perspective what I talked about before that you do understand that this cost money this starts machines and if something fails during this run you have to manually go in and stop these - 15:43 machines but anyway I will cck uh click Start fine tuning now this will take forever I'm going to fast forward this I'm guessing with a Stable Diffusion 2 model takes around 30 minutes it's on - 15:57 step two now waiting for the Runpod machine to build and start so the funny thing is if we go to Runpod and click pods now all of a sudden you have a pod here so it's actually starting this pod - 16:08 now and this is then the place if you click here and click stop and then under here there's going to be terminate pod if you ever need to do that manually right uh but anyway this will take 20 30 - 16:21 minutes now but for you it's going to look like it takes 10 seconds so we're finished I will put a success message here I realized I forgot that that's not very good that you don't get - 16:55 any feedback that it's done but uh we're just going to verify two things so first we go to my pods reload and hopefully there's no pods left so it has done the garbage collection remove the Pod double - 17:09 check this anyway and remove it if it's still there for whatever reason you see it's costed me some money here uh then we're going to go to Hugging Face go to my profile and we - 17:24 have a new model called Ivan boring airpods uh and actually uh because we're using Stable Diffusion in this case we can actually use the inference API here just to try - 17:37 it out so we could try it here a photo of the uh in ear ear phones uh what did we call it it's here airpods with display uh on a table we just press here this free - 17:56 version might or might not be a available it needs to start machine sometimes uh to be there it worked in this case so see we have it here I'm just going to show you so we can compare - 18:09 so we take Stable Diffusion to Hugging Face so we're going to compare with the exact same phrase uh on the base table diffusion too let's see if this is started - 18:24 otherwise it might take some time um because then you can compare and see that it's not aware of this so you see it it kind of figures out its earphones right but it's not the the airpods that - 18:38 we generated so it has generated it here I mean it's not perfect as I said before uh you need to have better pictures to create a perfect uh model you need more pictures than images than - 18:51 10 as well so anyway we have now uh fine tuned the model so now come comes next step we want to start using this mod uh uh model as well right so I'm just going to - 19:08 create a new content type here I'm going to call it um airpods image generation right uh and uh I'm just going to disable some stuff here and I'm not going to uh do that - 19:30 much chaining if you want to know how to chain stuff uh look at my previous videos if you like AI stuff please subscribe I will release a lot of Drupal AI videos in the future hopefully as - 19:43 well but um if you want to know how to chain so you can have stuff like you know asking in natural language for uh the the leaked airpods and this becomes this uh token that we have airpods with - 19:58 display this is something OpenAI or some other LM can fix for you before you create the prompt but right now I'm just going to create a plain text uh prompt take a long - 20:11 one uh we call this image prompt and we just save this oh we should set it required maybe so like we know we have to fill this out and then I just create that image field and that's where - 20:28 we want to generate this right so we take file upload image and we call this the generated image I have it here in my history and we're going to use the AI Interpolator here so in this first case - 20:46 I'm going to show you uh how to use it for free so we're going to use Hugging Face text to image right uh we use base mode and we use the image prompt and we just going to cop copy it as it is so we - 21:00 just use context we're not going to add anything else uh we say 100 and we're going to save it directly and uh we're going to uh use the free API inference in this case and - 21:16 then we just need to basically search for our model here so Ivan boring air uh pods oh this might have a cache let's see if we can find it at least if I save here yeah so um this might actually take - 21:34 some time before it shows up and now to complete because it's cached but anyway it's there it's validating that it's there uh and then we just need to create this we need to add a title here because - 21:47 of uh we're using a node node is probably not what we should uh be using here uh and then we're just going to take this and on our table do the same thing again save so if this gets timeout now - 22:06 it's because you're using the free version and the free version uh you know if they have service available they will generate it and that happened in this case sometimes they - 22:17 don't have servers available and you will get a timeout and then you have to redo it again so say that you actually want this but you want it stable I will just show you how this works then so - 22:28 what you you can do with your private mod model now is you can take deploy and uh take a inference end point dedicated uh you have to choose a machine here and this depends on the - 22:41 model so 16 GB is okay I think should be okay for Stable Diffusion inference Stable Diffusion 2 we just yeah I just this are okay I just make it public protected is okay as well because you're - 22:56 using the tokens um and I'm just going to create this end point and you see it cost 50 cents per hour so make sure you turn this off because here there are no credits this - 23:10 will actually just start costing money uh right away and we just have to wait here and this is now going to create that end point then and the only thing that's - 23:24 different is we go back to the content types manage Fields uh uh the generated image and under the settings instead of API inference we take dedicated inference and we're just going to have - 23:39 to copy this endpoint URL whenever it starts let's see this might take some time to start Maybe e and yeah we have a generated image here um doesn't do text very well as you can - 24:57 see so uh that's it basically uh how you can fine-tune a model and use that module model to generate custom images using some completely new uh fine tune token and also how you can set up dedicated - 25:15 machines that can do it then so once again thank you for watching if you like it subscribe um and see you in next video Filed under [Workflows](/blog?category=6) [Drupal AI](/blog?category=2) Parts of this content were produced by AI. This site has not assessed whether a label is required for this content. How was AI used?The video and the introduction are Marcus's own, first published on workflows-of-ai.com. AI wrote the subtitle, the summary and the video chapters from the transcript, and corrected misheard names in YouTube's automatic captions. Marcus Johansson Tinkers with AI and Drupal [About Marcus](/about/marcus-johansson)[Drupal.org profile](https://www.drupal.org/u/marcus_johansson)[Drupal AI](https://www.drupal.org/project/ai) ## The post in three sentences The Hugging Face module starts a Runpod GPU machine that fine-tunes Stable Diffusion with Dreambooth on images uploaded and cropped in Drupal, then pushes the model to Hugging Face. Marcus trains it on rendered images of rumoured AirPods with a display and compares the result with the base model. The model is then used through the AI Interpolator, first with the free inference API and then with a dedicated inference endpoint.