0:00
Today's video is sponsored by FreelyGive. If you need help with Drupal AI, don't hesitate to get in contact with us.
0:08
So before we dive in, I want to quickly set the stage for what this is you're looking at. So this is the new AI agents debugger and the goal here is simple, full visibility into what your agents
0:20
are actually doing. So what you can see here is I asked the in this case the Drupal CMS system but it could be any agent a question and then you can actually see all the inputs and outputs
0:34
for a run. You can get detailed information about how a sub agent request is starting what is added into a AI input including things like token replacements the provider that is being
0:48
used and so on. You can also export and import runs. So debugging isn't locked to a single environment or this specific debugger.
0:58
Uh you can view the entire chat discussion that's happening from the user to the assistant but also from the assistant to different agents. So you can see how the agents are talking to
1:10
each other. Uh you can also have a simpler graph view of what's happening in each loop.
1:18
Um, and this part I think is especially uh powerful. You can actually discuss a run with an agent itself and ask this agent that is looking at this run to change the agent that you're talking
1:34
about. So you can actually start chatting with it. Uh so yeah, with this in mind, let's jump into why this needed to exist and how it fits into the whole AI agents
1:46
ecosystem. So yeah, hey everyone and welcome. Um so today I want to show you something that I been wanting to build for a long time, the AI agents debugger. I was in uh
1:58
Brussels last week for the Drupal AI hackathon and because the Drupal I was there as technical support and because the Drupal AI module seems to be fairly stable, I
2:10
didn't really have that much to do. So finally I had some time to basically focus on one thing and one thing only.
2:18
So what I did then was to do this, the AI agents debugger. So if you ever worked with AI agents in Drupal, you probably know that the debugging of agents has always been less than ideal
2:31
because we had the AI agents explorer module for a while and that's inside AI agents and that work great for the all the workflow based coded agents but once we have introduced like the ReACT agents
2:43
that we have since AI agents 1.1 the config ones that you can set up inside the guey this was just not built for it. So we needed something that is purposely built
2:55
for it. Um so you know instead of trying to keep stretching the explorer further and further I decided to do this. So this is a module that exists in beta already. Uh
3:07
you can install it like any other module. Uh there are still some things that needs to go in in terms of a road map. So the most important thing I want to do is the step through function. So
3:21
it actually works similar to how Xdebug works for instance because the agents themselves they are serializable un serializable which means that you can store them in a state that they have
3:34
after one loop which means that you could you know step through the loops of the agent but also go back maybe change some tool output or maybe change the system prompt to see if the that loop
3:48
can become better depending on how you change it. Um, so let's go through how to get started, installed, and everything it can do.
4:02
Just a side note here, there's a disclosure. Uh, I think there should always be a disclosures until the coding agents becomes defacto standard. But I will be very clear with the only reason
4:15
that I could do this in half a day this module uh was because of AI. And I would say since the end of December or sometime around December something happened both with Claude Code Codex um
4:35
they just became so much better and um yeah the hype is becoming reality let's say like this you should look into it if you haven't seen it but this is also a disclosure so you know uh that a
4:51
lot of this code was actually uh generated. So how you get started? You just install this module. Um I would recommend you to uninstall the uh AI agents explorer and
5:06
use this instead. Then it's the same as always. You go to AI agent under AI, take one of the agents. So we're going to take the CMS assistant. And now there's a debug button. So if you have
5:18
both, you actually see debug and explore. Um, and this will open the debugger and then it is working similar to the explorer. So, this is one of the uh to-dos that in Drupal AI well 2.0 but
5:36
probably 1.3.0 we have a form element that can actually show chat history. So, you should be able to put in whole history here. So, it can replicate what you're doing with
5:46
the assistant where you might talk back and forth and back and forth. Um, but anyway, you know, you just choose the agent and the prompt. That's the easiest mode. Click run the agent and then
5:57
you're going to start seeing the steps. Um, and what's important here is it's it's showing each of the step structured and you can also see very clearly when it
6:12
goes into sub agents. You can also see all the timings. So almost right away we did a AI uh uh question where it basically asked what I asked here, right? You have the
6:26
system prompt here and you can see then in the AI output what uh decisions it took. So it actually answered with a text and took the uh said that the content type agent try
6:42
age. So that's a sub agent, right? It told this prompt check if there's a content type called page and if it has an image field. Then it goes into the sub aent that starts running and you can
6:53
see the first loop here. So this is the sub aent that is then the content type agent.
6:59
Uh and you can see that this then takes a decision to use the tool get content type info to get more information about this content type. Then there's a second loop or you can also see it's using this
7:15
tool content get content type info and this is the output from this tool. So this is what's being fed back to the second loop. So it actually makes it very easy to follow along what's
7:28
happening and then based on having that in the history uh you can see uh here you can see that it has the two results. So now there's a chat history here because it's the second loop system
7:44
prompt and then an output where it writes back to the uh writes back to the orchestration agent but then the orchestration agent is on its second loop where it has to
8:00
figure out that okay I also have to ask the field agent list the fields for this basic page and now it knows that the machine name is page.
8:10
because it got that information from content type agent. And then if we just skip forward here, you can see that the it's uh using the get entity field information for node page. And this is
8:23
then if we clicking on this tool, it gives back all the fields available. Which means that it can see that there is this field feature image that is I entered the reference of media of target
8:36
bundle type image and from that it can understand then in its output that it will say uh here are all the content types and then it's going to base on this figure out then that there is the field
8:52
feature image with the media entity of type image. Now one thing you saw directly here that you can maybe improve is what's going in here uh or no actually this was good. It's
9:10
actually asking field type image. I'm not sure why it's an uh returning everything there. Yeah, here it's answering. So this is a thing we could actually improve because this is
9:21
creating a lot of tokens and the only thing the user in the end was interested in is this answer here. They're not interested in all this. Uh so what we can do then is that there is this AI
9:36
agent handler agent. It's a little bit meta but what this is it installs a agent that can create agents for you and it actually works quite well. I've tried it on creating a lot of different
9:49
agents. As long as the tools are there, it's very good at it. But if you install that, this button shows up and there it actually feeds the whole run into this agent. So you can start asking questions
10:02
like uh the field agent answered with a very long uh uh response. That was too verbose. What should I change in the system prompt of that agent to make it more concise
10:31
to save tokens. So we can actually ask stuff like this because it is a agent that actually has the uh uh possibility to change stuff. You can actually go ahead and ask it to change this as well.
10:44
and try to rerun it then. Um, so you can see here basically what it wants to do and it has all the information about that specific agent.
10:59
So it's not just giving back some generic thing here. It's basing it in this case on the field agent then. So what can you also do here? Well, you can see the actual
11:13
uh each of the loops and you can see that in this loops it goes into another agent here that also has a loop one.
11:21
Then you can see the uh tools it's using when it's ending the second loop and then it's getting a final answer. You can click on this mode where you can see exactly what is being said. So the user
11:35
is telling the Drupal CMS assistant, can you tell me if there's a content type page and uh it has some image field on it. First thing the Drupal CMS assistant is doing is contacting the content type
11:47
agent that's asking this and answering this. Second step when this comes back is that it calls the field agent with this and returns this. Right? So it's a it's a
12:02
easy way to see how the conversation is flowing. Uh there are some other cool things as well. So as you might or might not know a agent can take tokens Drupal tokens
12:18
but when you're testing those tokens might not be available right. So you can actually spoof tokens here by you know adding the token name and uh the value and that way you can test it with some
12:32
uh uh madeup token values. What you can also do is when this has run you can click export and it exports this and then you can import it. I'm going to import a lot longer
12:45
conversation here where you can see in the prompt wait let's close here that I'm telling it you know can you create a content type that's called travel blogs also create a taxonomy field that should
12:57
have the taxonomy travel places on it could you add the 20 most interesting places to visit in the world on it and then we want to add those as field to the content type so if you look in this
13:08
view you're going to see for instance it's creating a lot of taxonomic terms here, right? Uh here is probably something we could improve by doing a bulk tool or the option to add uh bulk
13:21
tools. Uh but yeah, you get all the information out here. Uh this view you have to zoom of course to be able to see what actually happens because it's using the taxonomy term. It
13:36
might be that we should put a number there. But the thing is each of these calls has different parameters and that's probably good to know.
13:45
And then you can see here how the conversation is flowing uh sequentially. Uh and the idea behind the export and import function is that uh this should also be able to hook into other things.
14:04
So with the chatbot assistant that we have, you can actually hook in and add buttons that knows uh what has happened so far. So this means that we could create a button that
14:17
is export the session and then you can open it here to see exactly what happened. Uh we should of course build it into uh the logging system. So you can choose to log certain agents, maybe even
14:31
production, export them and see them here. Um yeah so that's the reasoning behind it.
14:38
I will just go through also the rest of the road map that is important. Yeah you have the chatbot expert chat history and step through I talked about this. The last thing that should be
14:53
added before it's production is spoof tool output. So one of the things we see that basically is a a make or break for an agent is how the tool output looks like. So
15:09
the idea is if we take some tool here you can see that uh in this case it added the great wall of China.
15:20
This is probably a okayish output uh because it tells it successfully what it's doing. Uh we can see if we can find something else. Maybe this one is too long for
15:34
instance. Maybe we have to make that concise. But we don't want to go in and program and see if that makes a difference. So the idea is here that you should be able to modify the output of
15:43
that tool just for this debug session. uh go back using the debugger and run it again.
15:51
There is one thing that I didn't show on the road map there as well. Uh and that's the idea that you should be able to manipulate um the agents that's being used but not actually save them. So you
16:06
should be able to try over and over until you feel okay this is better now I actually persist this in the age.
16:13
Um yeah that is it. H one thing I can show also of course if you open one of these things there is next and previous that you can use. So you can follow that way as well
16:26
and you can use right and left button as well uh on your keyboard just to see everything there. Um so that is it. It's still in beta.
16:38
Please test it. I I'm 99% sure that I have all the inputs and outputs there. If you feel something is missing that should be there, just you know create some um create some issue in the issue
16:52
queue. Um as always, if you're uh willing to maintain uh please reach out to me. We always need maintainers for different modules.
17:05
Um thank you for watching.
