Markus from CopilotKit, hang out with Mastras, and a chat about intelligence

um I am calling from or we're having the stream solo today well there'll be other people coming but I'm still in Belgium this will be the last stream of the week um tomorrow there will be probably a US time zone stream tomorrow but uh for the EU time there will not be one tomorrow because we're headed to a new location

so next week we'll be streaming from a different location um I guess you'll see on Monday or Tuesday one of those days um cool so let me just say some crazy um happened at AI.engineer we brought 1500 books of the both the V1's and the V2s of the principles of building AI agents and I think most of them are gone so

that's just a lot of books that we uh handed out there a lot of cool social capital gains um so yeah oh we already have some a chat message the chat is non silent today already that's good du's back what up dude hey Obby how are you i'm doing great um yeah can't complain life is breezy um let me talk about what we're going to get into today i have a couple mastras that are going to come on and

demo something that they're very excited to show off then I'm going to do a little architecture diagram session about intelligence very high level and then I'm going to do a pairing session with Marcus from co-pilot kit which is kind of a extension of yesterday's stream where we're going to integrate

that working structured memory into the UI so should be a fun stream but first I will bring in our homie Paul and he wants to show you some new documentation that's coming out um so Paul I think you're in the stream i'm going to bring you in there you are hello how's it going it's going great man it's going great

let's do this there you are fantastic what are you off today so today I wanted to talk about the new workflows uh um how are we officially referring to them workflows v2 or just now they're workflows and then they're legacy workflows so we still have the legacy workflow ones in the docs but current workflows

which were the next workflows have now been uh fully documented i just wanted to walk through uh the new pages um Abby you mentioned this morning it was like a novel so I think maybe uh maybe it does require a bit of a a step through so here we go let me see if I can uh share my screen here have we got screen sharing up and

running yep cool okay so uh from the lefthand nav in the docs scroll down workflows so under this new item here overview overview um it's an introduction explain what they are if you don't know what workflows are um introduces the main uh functions and packages that you'll need to install or import to use and then we kind of dive

straight into the steps which are really like the crucial part of workflows um so within a step you can have a bunch of uh inline logic you can make API calls you could do um you know anything you like within here and then you know the output of your step is what gets passed on to the next step in the

workflow so here's a workflow with just a single step just explains how to set up a workflow you'd have your input schema and then your output schema and then if you want to run steps sequentially you can use then uh you can have as many as you like that would run sequentially and then commit after it um there's an example there of this step would run the output of this

step gets passed into the input of this step and then just to set up your workflows register them in the mastery instance and just an example of how to run it so that's the basics but there's a lot more to it because they're super powerful as I discovered so under control flow I've tried to explain as

best I can pretty much every which way you might want to use a workflow so the one I just covered there was sequential that's a fairly common one where steps run in order but you can also run them in parallel you might have two separate workflows that are going off and doing separate things you can have conditional workflows so uh you might have uh a step that has a

result that goes one way or the other and from there you can decide which step to run next you can also run loops you can run continuously do while until something is true or do until something is true this might be particularly useful again if you've got asynchronous requests going off and you're waiting for a response to come back

you've got four eaches so you could run the same logic over each step um there's an example of a run instance you can also set the concurrency so you could have uh multiple steps running at any given time and then you can run workflows themselves in parallel as steps within a workflow this is where it gets a bit meta mhm and then nested workflows so

you could have a workflow as a step that itself has steps um so this is where it kind of unlocks like I would say some really really powerful functionality and these things can get quite complex but because it's composable I think for just your um your own mind and brain I think you can abstract these things out into logical workflows themselves and then build them all up i think over a much

larger code base with multiple agents this is a really helpful and a really powerful way of doing it because it's not just a giant file with 300 lines you can actually kind of separate it up um suspend and resume is pretty useful sometimes if you're waiting for um say user input before you go on to the next step you can suspend a workflow until you have the information that you need

to continue and then you know to resume you might want to pass in some data it could be user input um that's pretty helpful I think pretty common yeah um data mapping I think is super powerful as you've probably come across multiple times where the general idea behind a step is that the output of say step one would be the input for step two

um but that's not always going to be the case and so in those scenarios you would probably want to inter in in between your two steps introduce a map that could take the input data from the first step and effectively remap it to match the input data of the next step so that's really powerful as well because it allows you to still have two very

separate steps of different inputs and outputs but within your workflow you could sort of modify it ever so slightly just to suit the needs i think again for composability that's super useful um number of ways to do this you can grab all of the information from the input data you can get the step results

from a step by using get step result you can get the init data which is the initial data from the workflow so really you've got access to absolutely everything um renaming outputs again within the map different schemas uh or change values you can rename them um I won't go through all of these the the actual one I re was really keen to show though um was how you can now

use agents and tools in steps so probably already done before many times would be to have this like logic in the step but now we've actually got agents as steps so um you can create a step and just pass an agent in and that agent could be defined as it normally would anywhere else and you could use an agent on its own if you wanted um so this you

can just pass it straight in and and that just sort of means it's reusable in multiple places and you don't have to duplicate a bunch of code um same with a tool if you have a tool um where have I done that in this code my oh here we go sorry you can create a step and just pass a tool directly in so you could have all of your tools you know abstracted nicely away in your

tools directory and rather than having them all inlined in a step just create a step from the tool um so that's my favorite feature I would say and dude that is what people are doing or before these docs and before uh V-Ex workflows people were doing this by hand which is but it's so nice now that you could just put pass primitives in and they just work i love that yeah yeah so hopefully

that's um enough to cover off um don't want to take up too much of your time but I was super excited to get those live today and hopefully it should clear everything up for anyone if they were having any issues i know there was a couple of things in Discord but if there is anything else um yeah ping me don't

Oh there's the cat sorry um don't don't ping Shane you can ping me yeah dude paul that's really good and what I meant by a novel was like um there's like a big like you know it read like a like building on context you know and you're kind of like reading through this narrative that I was like oh I know

how to do this now because as we all know the docs were kind of all over the place before so this is way cleaner and yeah yeah I mean I took it as a compliment just to be clear but yeah I'm glad you I'm glad you liked it i mean I think it does it should be something that you could work through from start to finish and uh have

a really good understanding of how workflows can be used yeah cool thanks Paul as always all right man next time all right see you later all right so that was one monster down couple to go um if you're just joining us 54 of you here that's awesome um if you if you do have questions you can put them in the chat i will answer them as we go um but uh I got I was laughing in

the beginning of that because Marvin had said that I had got a haircut okay and I did get a haircut today but it was very awkward because the barber didn't speak English and I don't speak Turkish or Dutch or anything he was speaking so we you know I had to use my intelligence which we're going to talk about later to be like number six cut on the hair you know and that's

kind of like you know primitive communication right um anyway that's just alluding to what we're going to talk about next uh we got some new homies in the chat siobhan's back what up dude um hello how are you doing i never understood the difference between tool sets and tools what would be ELI5 version of that so a tool by itself is just a single tool and you can pass that

to your agent um a tool set is a map of tools maybe from different providers technically they're all the same um tool sets is just something we named in our API where you can pass in tool sets which is just an object of a bunch of stuff probably we need to rename that and just call it tools um because

there's nothing there's not a difference the difference between tool sets though in MRA and then regular tools is in the tool set you can have a key that encapsulates tools so for example let's say I wrote three tools for my database three tools for stripe and I wanted to pass that as a tool set i would have an object with one key called stripe other

object put all my stripe tools in there and the second key which is DB put all my tools in there but it actually doesn't matter because they all become single tools at the end of the day so thanks for answering asking that question it's just something we need to change next up I'm going to bring in Kenny what up how's it going Kenny yeah good it's

going pretty good remind the people where you from yeah I'm from Lagos Nigeria and that's where I am right now though you can see Apple Park behind me dude we're global out here man we're global out here yesterday we had a big ass release for something you've been putting your heart and soul into and I would love for you to give a demo to the viewers here because it is that sick so

let's do it yeah so um yesterday we released Can you see your hand because it says my Why is it saying connection loss okay yeah I'm I'm back so yesterday we finally launched a master course which is a one-of-a-kind course that has never been done before and it's it's a course that you take with like an MCP server inside of you an aentic editor to

something like say cursor wind stuff you know or even VS code with co-pilot mode and I'm just going to share like a lot more about that today not maybe a lot more just give you like more details so you can check this out yourself and actually see how it works so I'm going to try to share my screen real quick

and I'm going to see how it goes okay one second and try to find this All right i think this is this is better all right can everybody see this okay yep yeah so the first thing we launched is university so university is just this page on our website that allows you to get access to our workshop so all of the events we're going to have the book that

Sam has written and there's like a second version which is updated with a bunch more interesting stuff but the star of the show in all of this is Master 101 which is a course that explains to you all of the basics you need to get started with Mastra and so if you go to this section of the website we have an introductory video by Shane

but the interesting part about all of this is how you actually take this course so what I'm going to do is I'm going to just walk through this website real quick and then maybe jump to the editor to show that as well so what we allow you to do is you install Mastra and then you install Mastra's MCP server

which then allows you to chat with the agents inside of cursor for example and it walks you through the course right step by step so inside of the course you're going to learn how to say build your first agent or and also like how to add tools and MCP and then finally how to like add memory to the agent and this is particularly useful if you haven't used master before but let's say you

have used master before and you just want to see how like things are done if you're doing it correctly you know if there are things that could be better right this course is also very useful i think one more thing to add to it is um regarding the MCP server right a bunch more a lot of people might run into issues especially setting up and so we

have this FAQ section down here which should answer some of the most popular questions you'll get or you'd run into when you try to install the MCP server and also just get started with the course so I'm going to just you know move over to my code editor real quick rather my terminal and just try to get

started with creating a new project you know opening the course for the first time and just walking you through how everything is just going to flow from there so we're going to run npm create master at latest which should get me started i'm going to zoom in so this is a lot easier to see and then I'm going to call this my master new

course my master new app it's going to try to create a project for me and that's going to take couple seconds and it should be off to the races so while that is happening I'm just going to say that okay we have like an introductory video from Shane which just walks you through what the course is going to be about but then we also have like videos for every section so

every section has an intro that just gives you this short walk through of what the course is going to be about and you can see all of the lessons as well so let me just check back into this it's still installing i'm just going to wait a while for for that to finish so while that is happening um Abby do you have

any questions regarding the course you want to talk yeah like or anything at all yeah so I guess what was the hardest thing about building this or maybe the better question is like what was the most interesting thing that you like what was the most thing you found interesting while building this yeah so I think one thing um that was interesting was trying to

stare the agents to actually do what we wanted it to do so because this is based on an MCP server that returns like a bunch of instructions it's it's kind of like tuning the prompt right we had to like tune the prompt to to let the agent return the things we wanted to return right and so say for instance there's an

interesting part I'm going to show which is we allow you to actually see your course progress on the master website so we're able to like connect the progress you're making in the course inside of your editor to this website where we show you how far you've come with it and so some of the time the agent wouldn't return the URL so we have to tell it okay you have to return the URL every

time the user ask for like course progress so that was that was um that was interesting as well but I think the the most interesting part was just connecting all of the pieces together which is you have like the editor side of things where we save all of your progress on your device and it will try to sync that to like this to the server on Reddus and then you know have this

website where you're able to now see everything as you go through the course which makes the law more a lot more fun so yeah that's uh that's what yeah let's let's let's see this in action let's see it in action so I'm going to I'm going to just choose all of the defaults choose OpenAI i'm going to skip for now

i don't need an example but I'm going to add like this is the interesting part the important part by the way so I'm going to add cursor MCB server inside of this and then open this up and then open my editor i have to pause for a second Kenny because I either I'm going crazy or you have a sound effect when you hit enter

i do have a sound effect my sound effect dude I thought I was going insane that's sick okay sorry anyway keep going yeah all right so so one thing we've kind of discovered is that because zoom in okay zoom in let me just zoom in some more increase the button is it better now i I think so and I'll make it full screen boom okay okay right so one thing we actually

discovered that is interesting as well is the type of responses you get is also dependent on like the agents like say maybe cursor is using so we've had a lot of successes with the clawed models clawed foret for some reason doesn't actually do what we tell it to do with the tools so I think we're also going to

have a FAQ section for like the best models to do this with you know um GPT 4.1 works okay 3.5E as well but yeah enough talking let's get started so what I'm going to do is once I have my new project set up I'm going to go to my MCP settings first of all and enable this this is the new cursor 1.0 launch so you

actually have this new UI for MCPS and things like that i'm going to wait a while for this to enable we have all of our tools you know our course tools which looks good and then I'm just going to maybe use and say start master course which you know it's going to ask me oh what's your email address i'm like

oh right so I'm going to say please use my master email use that and then it's going to call the tools also I have YOLO mode on because I don't want to confirm tools all the time so it just like auto confirms and then moves on so yeah so it's going to say "Oh I look through your directory i see that you have master installed i'm going to

help you run the master server." Like I don't have to do anything because it runs terminal commands now so I'm going to copy this real quick and go back to my browser paste this here and I see you know my server is running my playground is running locally so I don't have an agent yet and I'm going to see I want to say I don't have

an agent yet can you create one for me so I can see it in the playground and then it's probably able to you know use the MCP tools create the agent for me write it to the file maybe fix a bunch of LinkedIn errors here and there as well um I don't think there is something called system so it's hallucinating this so it's trying to check the docs to verify what's wrong

and it's going to update that to fix the error as well nice the model stuff seems to be wrong but if I wait a little bit more it should use the doc MCP to get the right model and then use that instead okay let's wait a bit for that and we're good it fixed the error so I think yeah so it's going to just create this for us i can accept it's going to export that register it to the master

class itself and then if I go back to my playground now you know and say I refresh I have my agent pretty nice so one final thing is as you go through the course you can just say um what is my course progress which you know it's going to tell you okay you've done this amount of lessons this is the amount of lessons you have left and you can check out you know

um you can check out the status on the mastery website so I'm just going to wait for that to return it so yeah I've gone through four steps i'm in the fifth step of running the playground uh just to confirm Abby can you still hear me though yep I can all right all right so so yeah so I've gone through four steps i'm on the fifth

step of running the playground we run a playground and I can just copy this link go back to Chrome paste this and then you know on the website I'm able to see how far I've come with the course i can take this I share with others they can see my progress and yeah that's dude that's pretty much it man i love that that is so cool dude

so this is master course with an agent and we hope this helps and that's awesome it's able to just teach you more that is awesome i think W's in the chat for that one for sure that's what those Gen Z folks say right w's in the chat right um uh dude I just remember learning things on um Egghehead.io back in the day on all these different websites and this is

such a different learning experience now like you're in your editor you have the course details you have the course you're not paying for you're I tweeted this yesterday mp uh knowledge is an MPX command away like for sure like also great work bro great work i think chat needs to do an applause too that thing was sick um and with that I'll let you go buddy till next time see

you dude all right yeah thank you thank you holy crap that was cool um that was just I don't know i'm kind of like speechless even though I've already knew that it existed but um it's just cool to see it like live now we've been working on it for a while um and there's a lot of intricate things like to build a course that is non-deterministic in a way right we have we have definitely have these course logic and the steps but then as you saw

to do and execute the step like AI hallucinates a lot of the time and then we have to improve our docs to make sure that uh that doesn't happen but it can self-correct given the tools it has right it can check actual mosher documentation and see if it actually works so pretty dope I got one more get or two more guests but one more thing u we were talking about agent networks on Monday which blew up on the stream in

the sense that there are a hell of people on stream watching us do this and we were kind of discussing how we want the playground to be we stumbled through showing those messages well I'm happy to say I'm bringing back Miss Monio Mvin and Hey folks how's it going back in the bid how you doing guys doing good doing good dude yesterday there's tweet came out that only cracked engineers come from uh like the Midwest

and Toronto area and by that definition like Shane and Daniel were the only people in that uh in that bubble so then I I I replied tweeted that thing and I posted every uh every teammate's uh Twitter handle and I said I guess you know I think for uh for Tafi I was like I guess they don't know talent exists in

Africa and I think what I said for yours Marvin but you guys are cracked for sure so why don't we show the folks what you've been working on yeah so uh it's been a rough I mean we had hard days you know it was like hey we need to figure out how we can handle this agent to agent this agent networks basically which is kind of crazy because as you mentioned with sp just before now

it starts to become very complex because you can have tools that are agents you can have a workflow that triggers a workflow that has a step that triggers an agent you know this kind of things so it just becoming very very uh complex and so we uh we are trying we are still trying to be honest to make something

that works and we can show you probably some stuff what you think can we show stuff yeah yeah all right let me share my screen so let's do it don't expect anything very very crazy we are uh going with very basic use cases so this is how we work right we are trying to fix simple use cases and we add on top of

that so that it's iterative and we start small and then it grows it grows it grows and if it becomes too messy at some point we refactor our code on this kind of things so we started with two simple in between codes simple uh use cases uh the first one is uh talking to an agent network and streaming back the response from an agent and the other one is triggering a workflow from an agent network so let me

show how it works um I'm having a master application running hopefully okay can you can you see my screen yeah yep yeah cool so yeah sure i need to figure out how to zoom man i'm on a weird keyboard you I'm on this weird uh split keyboard with predefined mapping i don't even know where is my my plus thing

uh yeah and then it's just like oh come on you know engineering not being able to just zoom in what the is that i don't know i don't know let's just let's just do it that way i will roll with it yeah so we have this agent network um and basically the first user experience will just be a chat and I'm going to ask it uh tell me about BMW

cars and it will be thinking for some times and then it will stream some responses so it looks like it's just an agent response like we do on the agent section but it's a bit more we can check what's going on at the at the agent network level and we see that multiple things are happening the first one is that the agent networks was able to

decide which agent it has to call so at the at the code level uh I should be able to show you example agent network vx networks okay so we have an agent agent one uh we need the code i think we only see the browser okay but but I mean in my code I just have two agents so it's not important if we don't

see the code for now in my code I have two agents um and the network was able to say hey we are talking about car i'm pretty sure I should send this message to the agent one and agent one will be the one that has to give an answer and then the agent has made some execution and has turned back some information so that's the first use case like the use

case where your agent network is able to know which agent it has to forward the request to so that it can gives you back the expected answer right so agent one is responsible for dealing with cars and we have another one so I will refresh the page and I will say hey tell me about London and we will have a thinking process

again so it will think for a long time and now it gives us back some JSON answer and right now you cannot really know what uh has happened but if we expand this one again we see that the agent network has been through a decision making process so just to know which agent it has to or which workflow it has to route the input to and in this

case instead of trying to go to the agent responsible for uh providing information about cars it has decided to go uh to the workflow number one right and then it has executed this workflow so we have the same we have yeah we have the same UI as the other one we have the input output or traces nested workflow blah blah blah if they are so it's just

working the same way as workflows uh and then it has been through the agent step one agent step two and at the end it was able to resolve the final um output so that's what we've achieved and it's way more complex than it than it looks in this example because there are so many things that can happen in this execution flow as I mentioned tools calling agents

that are calling tools again that might also be an agent you know it can be very very complex but uh as a first step we were able to resolve you know having this decision- making process at the agent network level so it has made a decision go to the agent or go to the workflow then we had the workflow execution the different steps and etc

and etc so that's where we are at the UI level uh there will still be a lot of thing I mean a lot of thing has to be done yet uh we are lacking some information at the core side for example Tony is aware of that we will iterate on this uh and yeah then it will be going with more complex workflow with nesting parallel stuff happening all these kind of things but this is basically uh a

first version of it so for people that are interested in only the output they have it and for people that are interested in having the thinking process if they want to debug stuff it's also available but it's hidden so that it does not pollute the UI Okay and yeah that's all we have for now yo that was legit can we do one more

examp or can we do one more like test like uh show me cars again or something like that i just want to with an agent yeah like just agent example there's 120 of y'all just here if you just joined us we're doing like just demos if you joined us on Monday we were talking about agent networks and trying to build

a UI well Marvin and Tafi made some freaking moves and you can see like this is kind of the EV evolution from from Monday we move fast around here y'all um this is dope dude this is dope looks like I mean a lot of things has to be done yet but yeah we will figure this out oop sorry we will figure this out i can see the path forward which is the most important thing when building these

things you know for for everyone in the audience like we didn't know what the UX should be and this may not be the UX that is correct but we wanted to make sure that we could do something to move forward that way people can because the UI let's be real the UI is a great reflection of the agent network but the network itself still has to work right

and without these two pieces it's really hard to say that the network is complete um and we're one step closer so good job dudes yeah and it's it's very interesting at the Thank you man thank you uh it's very interesting at the discovery level because this is the kind of feature that you don't know how far it can expand because when you talk to

an agent you are expecting one output which can be a stream or just a string depends it's quite easy to to think about right but in this specific situation uh with an agent network it's super complex uh you don't really know what will happen right because you you are not aware of what will happen in this decision- making process and it can

go very very very crazy and yeah like having a lot of parallel stuff and also yeah amazing amazing thanks man i will stop sharing my screen then great so well thanks for being here we'll see you next time thanks Marvin see you folks bye bye need some W's in the chat for these guys too see you team bye-bye right all right and then there was just me um

that was really cool um we're building a very special team and we all really love working together and I think you can see it in the work right like it it totally bleeds through so um get yourself a good team that's all it takes um all right we're gonna move on to the next topic and then I got to bring Marcus in here

um and then we're you know sayanara after that but we still have a long stream we have ways to go um we're going to get to the next topic which is intelligence now this is going to sound like some hippie dippy stuff okay but just bear with me because I've been thinking a lot about this and I just want to poke your mind a little bit

about artificial intelligence and everything like that if I sound like an just let me know but I'll share my screen and we will draw some stuff um and since I'm solo I can't really see the chat but I will I will try um so I'm going to do this blank screen let me make sure I can see the chat suj's back what up SUJ dude I'm gonna Dude I like having

these repeat uh viewers because it's we can we can make the same jokes every day um hey what's up you guys are literally bumbling fumbling and jumbling dude well SJ that was your job title yesterday in stream so I think you know thanks for giving me the the new title um all right so this is also going to be very meta okay so bear with me again but

um when we talk about when we talk about intelligence right let's talk about it from a human perspective first what makes you intelligent right there's many things right but let's just like kind of uh let's break it down from like the very basic thing you need to have a brain right you need to have a

freaking brain so let's say the core of intelligence is the brain and what the brain allows you to do is what like it's it has all these neural pathways that allow you have to have motor functions um and so let's like talk about these capabilities of intelligence um and I'll say some of my personal stuff later about like why this is important to me but the brain is your

reasoning engine right when you look at this like when you're a kid your brain's not developed you never seen a water bottle before but eventually your brain starts understanding that if I'm thirsty there's some impulse in my brain i'm thirsty this thing I've seen other people drink water they do it like this

except in Europe the cap doesn't come off which is hilarious but anyway and then they drink it and then you start associating like oh if I'm thirsty I just drink I drink water right now a lot of these things are innate I doubt anyone had to teach you well maybe because you know if your parents took care of you or whatever but you know some things are innate so the innate reactions in

intelligence are let's just say these innate reactions so there's just some things that are innate innate now there's also things that are like you have to reason about right so your brain is a reasoning engine for example it's cloudy outside you know your your deduction would say it's probably going to rain you know so you need to be able to reason

so reasoning is part of intelligence reasoning now back in the day I'm Indian so we my mom comes from a different era where when you call someone dumb the word dumb used to be you couldn't talk right you couldn't speak so maybe that is and now it's dumb is dumb right like you're stupid or whatever but you know

that old colloquialness of dumb means you couldn't speak you couldn't express your reasoning or what you what you learned so you do need some ability to speech or to have speech or to communicate communicate and that shit's real right if you're a mute it's tough for you to communicate you could be the smartest person in the world but how would you

express yourself right expression often comes through communication right then I'm using a computer right a computer is a tool a hammer is a tool we have dexterity in our hands to be able to do things we learn how to use these tools and those are what you have as skills right you have skills and skills are the knowledge of

using a tool right so you have skills and you know in AI world skills and tools are kind of like synonymous but I would actually take it a step I would actually separate those concepts a skill is the knowledge of how to use a tool the tool itself is just a tool and that is man-made like a hammer or this water bottle there's a tool for

me to drink water right but I know I need to that actually that that metaphor doesn't actually work but anyway I'm getting there then there's this interesting concept that intelligence has which is if I do the same thing over and over again I can well f let's actually take a step back there's this thing where if there is a step-by-step instruction on

how to do something and I can follow it that means you know I can run these playbooks right all of us maybe if you're an engineer working in infrastructure you see you see an alert and you start running a playbook we call these playbooks right what is a playbook oh you know log into this system run these commands check the output you are running a playbook so you have the

ability to run playbooks or let's call them workflows and we do workflows all day right when you make your coffee in the morning you're hitting the button you put in the thing in right when you do something long enough you develop a second re like just an innate reaction of doing it called muscle memory so these workflows essentially sometimes

you have maybe you can't remember all the steps but sometimes you done it so many times you skip steps you've done five steps but you're like I can already I can do steps one two or three all in parallel or I can do it until you know I'm tired or whatever which Paul was talking about workflows right these are very much semantic ways of thinking

about how we do right workflows if you do it enough it evolves into what I call muscle memory Okay cool but then if you have muscle memory that means you had stored the procedure somewhere in the brain right that means you had memory to begin with now the human brain is the best freaking database that exists on this planet

except it's very flawed because it does not have long-term storage forever and I can get into a whole discussion give me give me some wine and we can talk about that for a long time about memory it's one of the most fascinating pieces of intelligence but also memory is flawed too because there actually are two different types of events the way you

remember them and what actually happened you ever got into an argument with your friend or your spouse about something you said but you said "I didn't say that." That's because two people are remembering events differently and I don't know like take it who I don't know who was right in those times i like to say I was but my memory was flawed maybe

right um SJ said STM short-term memory as well yes i'm going to get into that um so we're going to break down memory some more there's different types of memory and let's make it like once again this is all human we're going to talk about this as a human obviously you can see where I'm going with this but

there's different types of memory you have short-term memory some people have short-term memory they can't remember five numbers to save their life right if I told you like 3754 and I ask you 30 minutes later you're like "What?" And but some people can remember text better or images like "Oh I saw that

flag on the on the post before." Or "I know I know where we're going because I saw the water fountain here last time we took a left here." So there are different types of memory memory is multimodal so let me also put long-term memory in here memory is multimodal right it's multimodal in the sense that

and honestly if you don't have an inner voice which I know a lot of people don't have um I know Marvin doesn't uh like multimodal maybe it's not really apply to you because you don't think in images or maybe you know your brain reaction is just just happens you know for me I think in images and music and

taste and like taste like let's talk about the modalities right you have vision you have images you have sound you have actually sound versus voice is different you can remember speeches versus you heard a cl right these are different u uh sensory things taste right and those are some examples but let's break this down even more y'all memory can even be episodic i

remember my 20s at the glory days right i was in college my 20s i can remember that it's episodic in my life i'm in my 30s now that was a chapter I don't remember much about and then you are sorry I'll get into this but these episodes of your life are how you kind of kind of like keep track of things

right you compress memories into smaller things let's talk about compression right so you have summarization memory sometimes you don't remember every detail but you remember enough you compress it into something in your mind where you can be like "Oh yeah I remember something." Like "Oh yeah I remember being there but I don't even know exactly what I did." Right and then there's

archival you know after after many years of of life right you have to essentially archive in your brain it happens automatically is beautiful but then you also have to pull it out of the archive and like that's like like I said the brain is a faster database than any database you know because you can do that in you know microscond or whatever

um so that's just memory that's a whole that's a whole can of worms that are there okay so then we kind of broken down like what intelligence is um let's talk about the sad parts of this this thing your brain can get damaged and you lose some of these abilities right you can't do anything with your hands anymore you can't learn anything anymore or you know maybe you

weren't even born with the ability to do this you're disabled um you know you you born with brain damage right or you're a mute like you can be impaired in all these ways and over time your memory can degrade based on your genetics right so okay enough of this like high in the pie scientific right I have dri

what I have written down is actually very interesting because let's map this to what AI is I know for a fact because I'm work we're working on it let's just copy this real Also let me check the chat 151 of you thanks you're coming to this experimental me talk about some random stuff hour um hope you're enjoying it um

let's just make this a parallel of what AI is trying to do and maybe what also I forgot to even write about emotions emotions are a very strong thing of intelligence because people who can control their emotions we call them very clever or sly or whatever or maybe they're just really intelligent right maybe that is a part of an intelligence

because there's emotional intelligence right okay so now I now it's all symmetrical for me so I feel better moving on so memory so let's talk about AI right let's and then let's replace these skills or like replace these bubbles with things that we know are tooling in the AI world today so this is a onetoone

thing memory everything I just said is what we are trying to make possible in our AI applications and if you follow MRA like we are trying to do um we're trying to make this whole box programmable you know if the brain is the best database well we got Postgress jokes aside well we got Postgress y so we could do this and we can do a lot of these things there are people who are

building multimodal databases too probably because they think in the same way now what is innate in AI Well I'm going to call this caching caching and stored procedures right skills well they're tools reasoning well this is an LLM workflows well those are workflows muscle memory that's a type of memory that can make a

workflow communication is STS TTS ST or real time audio or real time speech or what we call it is STS speech speed sorry my bad and you know these LLMs are starting to do EQ we we saw that the other day right on Tuesday they're trying to do emotional emotion emotional intelligence or emotional responses right

so I am going to close this segment now i hope I made my point like we are trying to recreate intelligence that's why it's artificial it's not chat GPT it's not talking on a chat that is like that is dumb to think that that's where we're going if people think AGI is like gonna take jobs it probably will because you're trying to recreate intelligence of the human brain but it's not flawed because you can have

memory forever you can throw more databases at her some servers etc um so anyway I will leave it at that i know this was very experimental but this is just something very close to why we're doing this because it's very interesting um and if you're not if you're if you're thinking that uh like if you're thinking that we're there yet we're not and anytime you want

to think that you're there you should compare it to can it do what my brain does if you have a healthy one I don't want to be an ableist but can it do what your brain does if it can't then it we ain't there yet all right so let me stop sharing i'm going to answer some chats here all right from SUJ i just want help

from you okay sure what resource would you recommend for learning prompt engineering efficiently i feel sometimes my younger brother is kind of a bad prompter he really needs it you know prompt engineering used to be really important maybe a couple months ago where you had to do all these like fancymancy things what's actually become

very interesting is how much you need to be good at just communication through text so if you're if you can if you can express your thoughts in the prompt if you can break down things actually you should challenge your brother you should ask him to prompt you and if you don't understand what the he's saying

then he's probably not good at prompting and that's where it's evolving to you should be able to prompt the agent or this or your IDE as if you were going to tell someone what to do okay next question listening to this makes me feel I'm not crazy to have had the same thought process on the mapping hey great minds think alike I guess and

I guess same for SUJ and then Du has a out of the context question yeah dude just post it um I was playing around with different sort of agents two agents have different APIs as tools and in between I'm using a third to summarize prompts etc and I'm curious what I cannot see traces in the third one this is a Monstra thing um I would definitely post

this on Discord in the help help forum and we'll we'll get to you and we can dive into your code and see all that stuff i don't have the capacity to do that on stream um maybe I should uh have a segment where we dive into customer issues live but uh please go on Discord we got you the whole team's there thank

you okay okay now I need to get our next guest in um I got to send him the link so let me do that you can ask more questions in the meantime while we wait alrighty i just sent him the link marcus from Copilot Kit is going to join us things are going to get pretty spicy in here just kidding and yeah no problem guys thank you you're welcome you never have to

thank me it's my job but that thing on prompting is interesting because prompt engineering has changed um yeah like I still like do some of those tactics like I capitalize words and stuff like that makes sense right because that's what you would do in real life so um okay my homies here yes Marcus what up dude hey guys how's it

going hey calling all good thank you are you calling from Austria oh just a second uh I had you on on the Twitter also so I was getting confused all good how you doing man you know just running the stream having fun um we have 170 of y'all in here welcome uh if you don't know Marcus why don't you give him

an intro yeah so uh my name is Marcus i'm uh working at C-Pilot Kit uh so what is Copilot Kit it's a framework for integrating uh your agents into applications uh so you can basically uh use convenient React hooks to easily uh hook up to your agent framework and make your agents work inside of your applications um we come you know with prepackaged UI components and you know

all sorts of things and uh the latest thing that we were doing is a protocol called AGUI which uh basically uh is a you know lightweight um uh independent uh protocol that uh standardizes how uh user interface can connect to AI agents um so yeah dope dope and uh so yesterday on stream we built structured memory

which essentially is agent state right this concept of agent state um and today what Marcus and I are going to try to do in our pairing session um is connected to agi and hopefully figure some out um before that Marcus you wanted to kind of set the stage right so the floor is yours yeah exactly So I wanted to show um what

uh what would be the end goal of uh what we want to build um let me share my screen um so basically what we want to build is a shared state between uh the front end and uh the agent we have a demo here which shows you know what the end state would look like uh it's a demo of a recipe builder um where you as a

user uh have this uh recipe um UI and you can use it to uh you know you can edit it yourself you can just uh I don't know check boxes here i want uh vegetarian low carb for example um and once I send this to the AI or I type anything in here the I will see what um I have chosen and it's going to take that into

account basically and and uh change the state Oh you dropped out internet connectivity problems if you ever see that show Parks and Recreation there's some internet connectivity problems i think you're back and we're back maybe nope i guess we'll see him in a second internet connectivity problems and we're back i

hope I don't think so technical issues y'all it's all good um some technical difficulties bear with us hi Marcus i will let me chat with him on the side all right third time's a charm is he back all right you're back sorry for that so we have some pretty bad internet for the last couple of days for some reason they're fixing something i don't

know okay uh so let's try again share screen let's add that oh it's super blurry though all right so it's super Yeah oh no and yeah the chat says they can't really see it because it's blurry maybe I can send link to you and you could show it yeah definitely send it send it over and you can just talk talk it over right

you got it in a sec dope let me share my screen there it is let me zoom out a little bit all right right go for it so basically what I wanted to explain is that uh you know anything you select here uh will automatically be shared with the AI and we make it really easy uh to do just that um so it gives the AI a context of

u you know what what you want to do here basically and um it can also edit the state by itself so it's kind of you and the AI working uh you know together to achieve a certain thing in this case uh making a recipe for something so if you now click proof with AI or you could also send a messagely it's going to take

whatever uh you put there and uh iterate on it right and you could uh you know give it additional instructions basically I don't know like I I want like a a white pizza instead of a red one so with cream instead of or what you know whatever you want basically uh or you can you could make these uh edits by yourself and the would be aware of that

and you know basically work with you together on the structured data to achieve a result um so yeah that's uh that's the thing that uh we we are going to build today to master I think um a carrot pizza bake yum white carrot yeah sounds delicious yeah cool all right cool so let's see what we could do let me take I'll uh I'll share

my code and let me also get into where I was yesterday uh so let me show you what happened yesterday Marcus and then we can put our brains together um and make some moves so I'm gonna share this this bada bing cool you can see my CLI right dope let me just make sure I install and hide this oh no back

here then let me just build some packages i'll just build the packages thank you and I'm going to go to this example I made and I'll open it up and in the example in my agent I have a new agent down here that is just this working memory um structure data so as you could see um with the agent has memory we can say

that the memory the working memory here is has a schema and in this schema had me and some friends and there are name and occupation like and as I talk to the agent it will remember or it'll try to fill in these details right so let me run and I'll show you the little demo of that So Mary the memory agent hi I'm Obby and my friend is

Marcus i work at uh MRA and he works at Copilot Kit so you can see it updated the working memory that's Oh wow that's really nice game time baby um so we have this right we have this ability now so but what what we don't have right now is like the client and we can build it right no problem but right now it's like

um I'm essentially giving that state via this text but I should be able to just inject that state right into um this working memory we also need to figure out how we're going to stream it and all that type of stuff so what's the right where's the right place to begin do you think in the streaming um I guess what we could do like we

could take two paths so first uh we could set up like a an interactive development environment so we can you know see things happening in real time if that's what you want to do or you know we we could just build it first and then connect to it um so what do you think is best I think because I'm in I have my AGUI example so we can

go there and I know it's not like latest of on stuff but um let me open this one and I can bring my working memory over there into this and sure I actually only want these where's my Zod action is that sod here then let me also make this full screen for people why is it tripping oh that's not what I care about sorry let's ignore the workspace

fail to create bin why is that what did I do wrong um let me ag chilling i'm just going to run it see what happens oh ma loggers that's what happened i need loggers loggers that's dumb um don't need this we have our log that's great sorry I haven't been in this module for a while this example I mean and I don't know why the types are up but I don't really care just

want to start oh yeah I got to install ignore workspace do your thing boom okay sick and then if I start the UI dev maybe oh hello there hi hey look at that we are now business okay nice so this is like the basic example let's see let's see what we can do now also don't understand this type zod issue it's probably because Let me look at it zod in this other one i don't have

it zod is right here yeah I get this all the time when the sword version is different yeah it's so annoying bro oops i need to ignore our workspace installs audio there we go maybe that makes us our lives better seems like it okay so and then because we're using AG like right now we have AGUI in coming from like the MRA register

copilot kit method so okay now that the monor repo we can make whatever moves we want right so that's cool um so the next piece I guess is what um we have to So so if I So right now I need a good way of uh demonstrating this let me do something i'm gonna do something stupid okay but it's just bear with me uh let's

make like a method register API route nope sorry need register i hate when like I hate when Typescript doesn't give me my autocomplete it pisses me off you're tripping dude okay one second i'll just go into here sorry I'm all over the place uh agi index so we have going to take this Put this in here annoying oh it's cuz I need

server i need server and I have server i'm just dumb and it just didn't autocomplete for some reason um the path let's just say like check working mem get and we don't need any of this and to check working me I'll get our agent so MRA get agent the weather agent then I'm going to get memory agent.get

memory cool then I'm going to get the working memory working mem memory get working memory and can I force a thread ID Marcus on the front end um you can do it on the front end yes I can force it so you you could set a Fred ID yeah you can go to the front end and uh just pass it to Copilot Kit uh the

top level you know component the provider that's what I'm going to do and then I'm going to just C.JSON this and now we have a custom API route on the server the reason I added that is I just want to check like in this intermediate stage where we don't have going on i just want to make sure that the working memory is getting set in this

context so like weather agent and instead of this object let's say recent weather reports and it's an array of zod object where we have the location and let's say the weather right so I refresh i'm sure all this stuff refreshed nope because I'm not running my server anymore like a dummy and the UI is going this is going oh that's because I have another

playground somewhere running maybe yeah it's because I'm not in my playground all right dude so let's just test it here then let's also test it in agi um what's the weather in Brussels it didn't call it tool um but that's good invalid arguments i must have made something what did I do wrong here what did I do wrong let me just do something simple to debug this this is a

new thing so there probably some kinks here a recent call that's the weather in Brussels okay and so you can see recent here is that and I have this thread ID so technically I should be able to curl i should be able to just curl local host what did I name this route dude something like check work man i don't know but I think you also hardcoded it

to a certain thread ID yeah oh that's true yeah damn it oh I'll just put this here just put this yeah yeah and then it's check check working man oops um oh that's because I I restarted the server oopsie okay let's try this again um what's the weather in Austria oh okay what's a city in Austria uh Vienna vienna austria what's the

weather in Vienna dude where's my weather tool go do I not even have one oh I'm so stupid i don't even have a weather tool on here god actually then I don't care i'm just going to tell it yo the weather today is sunny it is but then invalid type expected object h i must have broken the thread actually let me try this again the weather is

sunny in Belgium i have to force it to do this let me see um recent let's say recent location all right I'm dumbing this even down um okay um I am in Belgium okay cool okay and then you know the problem though one problem and let me fix this problem is persistence problem i'm going to do this let's not do memory let's let's

sorry memory in the sense of in memory okay have consistent so I am in Belgium okay now if I go to my API route here I guess I could have made it a header so this this drama doesn't exist but now let's curl it see we have our recent location so that's cool um let's go into the HUI piece now in the app and I I can put a thread here yeah you can build a Fred ID exactly put that

as It's not Oh okay you want to do a different one right because the other Okay let's go back here and make it one again just so we have that bada bing bada boom and let's not I am in Finland oh it just blew up there okay well error executing tool execution what tool am I what model am I using am I doing some Let's just use mini

And also you you catalog where I have been cataloged i think it got it like it gets his directive all weird um okay hi I'm in Belgium i spelled that wrong anyway okay then I Okay perfect we're in business now okay oh wait that's odd make sure you catalog me in Belgium did it though no h it didn't why is

that my location um make sure to update working memory when I tell you to I tell you where I have been yesterday was a lot smoother but I think that was just like you know and Tyler was telling me that I think we got lucky yesterday in some ways um but also we'll see um I am in Vienna h what school called did it still say Is it still saying

string i didn't uh see it yeah it still says string it's weird yeah it was working in Let's put Let's try it in the playground again i am in Vienna huh but the location is a string oh but the memory here is not this is odd but this is probably a bug on our end somehow um this is fine i'm going to switch to doing something else or I'm going to

switch this to not use the JSON format and what this will do is give me a string and I can just JSON parse this you know and it exists or let's say a string for now and let's curl it again there we go i'll fix that other later but we can we can roll with this so then if I did again I am in France okay all right dude we're Okay we're in motion we're back in motion we're back in motion um so now I need to

send state deltas right like that's the whole point of this right well you can uh make it easy first so you could just send send a state snapshot which is the full state oh six okay let's do that also let me check the chat real quick because I haven't been looking at it is it B is it blurry even on my Oh

that was that was before okay I think we're good now and 203 people in here thanks for being here um if you're just getting here me and Marcus are making moves on agent state in AGUI and uh yeah let's keep going so full state snapshots I hear you loud and clear dude let's get into the code of AGUI not that one and not that one

either and it's this one agi source index and I guess here in the run function right Marcus was where we where the magic happens right that's where the magic happens and uh that's where we probably maybe at the end of the whole thing we would want to set send a state um so that would mean on message finished what do you think yeah that's right on on finished message part

here yeah finished message part yep so right before run finished we could do subscriber do next and then uh we have a type of event type dot uh me something with snapshot state snapshot I think messages snapshot or no not not a messages snapshot but uh state snapshot yeah and we need a value that's just a Java yeah and the key is called value I believe Nice

yeah let me have a look at the talks but make sure and state has to be a structured thing right it can't just be some markdown yeah yeah sick well it could actually theoretically it could also be a string or a number or nothing oh okay but I guess it makes the most sense to just uh send a JSON object and we have our thread ID somewhere in

here right so that comes input thread ID oh sorry so actually a state snapshot event takes a snapshot key and that snapshot uh is is the value snapshot oh does not exist it looks like um What do we have let's see we have raw event and time stamp um so I guess uh maybe maybe it needs to maybe Yeah you you have to type it you

have to type it so I guess if you could say like convent uh column uh state snap event exactly and then equals so then you get auto completes and everything yeah and let's say if we have working memory we will send it i'm just going to say it's a string actually it should say format string but that doesn't mean it could still be a string so I'm just going to

say it is a string but why is it blowing up at me you know sometimes TypeScript pisses me so much and that's okay happens but um let's see working memory oh cuz I have to await this cuz I'm a dumbass there we go oh but we need to Can this be async or no i think so give it a shot let's do it okay okay so technically I'm going to console log here because that's the right thing to do

um we'll put this here we have that i'm going to rebuild AGUI the the module bro if this works that nicely I'm going to be I'm going to give you a big hug when I see you dude works the first try which I don't think this will work the first try but I have a very sneaking okay so then this will restart which I believe it did and that's cool let's

refresh i am in Paris cool so I don't Oh wait the log where's the log at oh I don't see the log let me restart this i'll try it again also I did build this right i did okay and let me just double check it's null because we just restarted the server i am in Paris bro Paris sick do we have our log we do

great so what what how do I show what do I do now yeah so actually it should already be uh you know on the front end but we you know we can't see it because uh you need to have a we need to hook it up to the front front end so what we're going to do is go to that page yes sir and on that page right that's the one so

I guess the best uh position would be in weather agent chat let's just log something okay so we would say uh use co-agent um use co-agent that would be the name of the hook use co- agent and you would give it a name which would be weather agent like anything does it matter um it does matter it should be the same name uh of

of the agent so uh yeah exactly this one camo case sweater agent and I guess that would be enough so we don't want initial state we don't care about this stuff right now we just want to say uh like um const um con state before before the use co agent sick console it and then console it complain no no yeah me too me too I am in

Berlin oh boy I'm so excited for this console log dude in the chat yeah dude that was easy that's amazing i'm sure there's more to go okay so what's what's our next move oh I guess we could display it right somewhere um you know just uh put it above the above the chat um yeah oh dude this will be sick okay okay okay

okay uh I'm not a front end that you are in state dot what did that say um my location right and by the way you can also uh type the use co agent book we're not going to do it right now but you can basically the type can be shared between master and uh in that case you want to have type safety there so I'm in Berlin

but now I am in Belgium oh I love that i love that let me make it strong that much I love it um cool okay what's our next move so I guess next we want to send back uh you know any changes to the state um to master if if that's possible we'll make it possible we'll make it possible yeah so uh basically the our uh AGUI uh

integration is going to receive an input and the input is going to have a state property and we somehow want to send that to master and then uh ideally have it end up in the in the prompt yeah okay let's do that so um where can I where do we do the set state part and all that and send it off or where would we do that so uh there would be a uh set state that

is returned by useco agent okay set state here nice right and you could just use that basically to update the state on on change or something like that yeah exactly so this changes that but then it's not in the back end yet right right so it wouldn't be in the back end until you interact the next time with your agent

right so that allows the user to edit something you know to work on it and only if they're finished you know they can send it to the LM and say you know that's what I want basically improve it further or something so now that we changed it on the client to say New York I say "Do you know where I am now?" It's not going to know it's not going to know will it know no did I

change okay so now we have to bridge that gap exactly exactly and it's sending us Belgium back again because it still remembers that right and we just built it to basically send the memory every time which is the right thing to do mhm so how do I update or like how do I get this value across the wire

yeah so if you go to the AGUI integration um yes sir we will have that run method right and it's going to get um it's input of type agent input oops and that's going to be that's going to have a state uh property nice so input maybe we can we can start by logging that just to make sure um is this is this updating live Abby uh what do you mean like uh if you if we console

log and then run again is it going to Oh it yeah yeah getting state from the client which is state input state and then it's rebuild agi let's just restart the server and then let's do our same thing actually we refresh the page here i am in France sick what about now and we should see our boom okay okay okay great that's

what we want okay that's tight now we got to do something with this um what is it that we need to do i could it like we could update it like we can update it directly right we could do that but it's is it more so that we want to pass this state into the prompt and then you know or is it more so that you

just update the state directly so I think uh it would make sense to update the state directly because you are already passing that memory to the to the prompt automatically I guess right yes yeah so if we update the state from from that I think u that should do the trick will it and then like it'll always have

this client side if I did it again right um will this inputstate just be the same or is it only for this run it takes the client side state value like if I did another run here actually like hey hi um Oh so it does continue to send state which is fine it does nice so and and and basically the idea would be if it reconnects to an existing thread which we didn't implement yet but

we're going to be doing that really soon it's going to get the state also right so it's going to reload the messages it's going to reload the state so it's exactly at the same spot where the user left it you know even displaying that state in other places in the user interface okay um so and and then

everything should be fine right so the user will always see the latest state of the agent and uh it will they will be able to edit it they will be able to you know send it to the uh agent so if we have input.state um and then you know we're not going to worry about all the details right now because like it has to match the schema and all that right um we we'll

do like some zod parsing and stuff but I'm going to blindly just update the state um so I can do memory.update update I think wonder what the API forgot what it is let me look at the tool that we made um save thread we already have a thread yep this is what we want essentially um here we have also I need to get the

thread um get thread by ID can this be async uh no this cannot be async but let's chill um we're just figuring it out as we go so if there's a thread which there will be hopefully oh yeah if memory and that we'll just take the current thread data take the current metadata and I want to JSON stringify i'll just JSON stringify the whole state because I don't give a

right now and then I'm going to do this but then oops I don't want the agent to run I have to do like a dot I have to chain this den right because I I don't want this I guess I I guess you would need you know for now to run the agent inside of the den yeah or can I make it work can I make an async function here maybe for running run stuff yeah and use

closures to my advantage because then I could do whatever I want here and then we just do run stuff do whatever so like here let me rewrite this a little bit so I can do thread and grab that and then I can do that that's cool that's cool then also Yep and then this all all this can just go in there right

uh and then here oh yeah but this needs to be a fat arrow function right yeah a big fat one um async uh here then we'll return here and then that looks good get rid of all this think I have an extra i just want to run stuff not the prettiest but it'll get the job done hopefully okay let's see what happens

dude let's see me too and while we check let me just check on the chat 231 we have some shout outs here from Ulie shout out Marcus that's his Twitter if anyone you know wants to know back to business so we built that i'm going to restart the server man we're efficient right now Marcus i feel like we're very efficient we're doing efficiently um hi i am

in Belgium okay up to the update the location where am I now no that's okay we probably have some issues let me add some logs and stuff too uh what does it say in the UI did it switch back to Belgium or did it leave the I don't think the thread existed it left it uh it left it as Belgium for some reason maybe this is not working so let's just say thread

It could also be a prompting issue maybe it's uh it gets the right information but it just doesn't you know look at it it looks at the threat history and thinks that it's uh it's fuming is it possible that's could be true but then we fetch working memory and inject it into the system prompt before doing anything

um and we can actually if I put I'll put debug logs actually you know what i should put some debug logs on and we'll see our our business so let me go in here and to the logger i put debug on we'll see a bunch of now and we'll just let me also just put some logs in here as well so here's the thread thread to save and I guess I could fetch the thread

after just to double check huh and we'll console log that as well okay that's cool and I need to rebuild this and I need to do this okay and the thread ID is one i am in France pull France but let's see did anything happen so you see how this is my working memory right now like memory sir yeah and then we update it with France that's cool and here's our state

snapshot but I don't see our log huh for a thread oh the thread is null so that at that time right because we just started the chat so maybe now where am I now but this is the New York one but that snapshot happened lower where is the start of it here's my thread and it does have New York in in the So so this is this is the system

prompt uh that's what what goes into the prompt right no uh sorry i mean the the thread metadata for its working memory is New York now as you can see here so that's cool but it's Let's look at the system prompt it's not using that meth it's not using that data to then assess you know because it's not in the this is a bug actually in the the I'm

not injecting this into the context at all um right now so that's the problem it's not being injected like there should be a there should be like a context between these right that would state that I like I'm now in you know I'm now in this new location it changed you know um let's So I got to figure that out let me figure that out actually I can look at the PR and Tyler

had reviewed it already and we can see because he he did say this so it's not like I'm surprised um so let's look at this is the structured memory PR there's a ton of review on here um no that's not it that's not it that's something we already know um okay that's fine that's okay that's nothing that doesn't matter yep yeah it was it doesn't work quite well but it's

very close you need to inject the stored data back into the system prompt each time it only looks like it's working properly because the agent is calling the tool and can see the full conversation history it can't actually see the structured memory and context only the schema of it oh yeah this is exactly what we running into right

that's why it thinks we're intention so I need to add that to the system prompt huh into the memory system prompt that's what we need to do okay I know what we need to do i may be wrong but for the the sake of the drama of this stream I know what to do um so this our memory module and you can see this is the working memory tool instructions right

so what if I just add here current working memory and I don't know if this is correct but we are going to make it work so I have what is in here config i have a thread ID let's get the working memory content let's pass that in um and go here here and let's just place it in here and see what happens bro let's see

what happens that's what we do around here we just see what happens um CD memory let's build this yesterday Ward taught me some Oh i have to Oh yeah i have to do this dumbass i have to do that casting because I know it's a string it knows it's a string too let's try that again okay let's restart this also I'm sorry if I'm not monitor chat because I'm a single screen so I don't know what's happening let me just

double check real quick this guy's still going yep that's me ward i was just mentioning Ward me and Ward were doing this yesterday um okay there's 252 of you thanks for being here if you're just joining us now you're seeing uh inception on the screen so let me just go back to this if you're just joining us today we started with showing

a bunch of demos we rewrote our workflow documentation it reads very well um then we talked about the MSRA course which is an MCP course that you can download you know just do in your IDE and then lastly we talked about agent networks and like our initial UX for that um which I'm sure will land in AGUI at some point as

well and then we brought Mark or then I talked about intelligence and I just rambled and about what I think about the human brain and AI and stuff so if you were not there for that maybe that's a good thing and then Marcus came on and we're making moves right now on bringing yesterday's stream for structured memory into AGUI and we're

getting we're getting somewhere so that's good all right back to business so I restarted everything i am in France okay update it where am I now okay let's look at the system prompt um let's see memory structure here's the current working memory new York so I guess uh you know it's not giving that uh enough importance

uh I think it's a prompt issue you you we would have to say something different there yes let's go back and let's say um oops important if you receive um data in current working working memory emphasize if you receive data in current working memory and a user is requesting information from it um use it use it i don't know that's a

bad prompt but we'll see let's try it again and I need to rerun this we're so close dude so close where am I now you man you but double check did my system prompt get updated yeah so maybe the importance there maybe you would have to use h you already put important you must use it as your single source of truth or something

like that in capitals maybe maybe I'll do some this is our current uh state of I don't need all that um Um uh if you receive data and the user is requesting information you must use it as context i'm sure when I tell show this to Tyler he'll have the answer but we're just going to bumble and stumble our way to victories

also Marcus if you have like a cut off just let me know because I know Oh no i'm I'm good okay so what if I do uh I am in Paris where am I now can you ask it if it's if it has any other information i mean what is in But really where am I cuz maybe it's a second call thing no interesting interesting what What What memories do you have what What does your memory say

about where I am or something like that and what does your working memory say where I am but it's not it's not there h what if I added a user message actually like show me or summarize your working memory that's not true though because if it was true this this state would not be this way right so it it gets updated every time so yeah

and we are also logging that right we are correct yeah like where is that still New York so both the database actually let's look at the thread too thread thread oh why are the threads null it's probably because it's really old let me just do another call here um look at your working memory but here is I mean it's here right here that's what it is

um so it's definitely in there but actually here let me try one other thing in agi like I don't want the whole state i want um messages rest in the input boxes and I just want the rest in there i don't know if that's going to change anything but it's worth a shot actually that could be it cuz there's all those things in there exactly i mean

we're telling it to look at the working memory and it still has the conversation history in there so that's true huh h where am I now oh what is my location interesting it's looking at the user right like okay I'm going to can you can can you tell it to print the working uh memory as JSON print our working memory I don't think

it has a tool to do that but it should but we can always we can always figure that out well well I guess uh you know to to answer that or something I mean it's injected in the prompt so it's wrong it's you it's getting that from somewhere in the thread Right not the state um all right let's do the hack i'm going to hack this a little bit so we

have this um converted messages where do we convert those somewhere converted messages right here what if I added a message um if there's state right maybe like let's convert the messages after and input do messages so like initial messages uh input do messages and then here we save it that's all fine and dandy but we

push uh was it roll user h we could also do system but yeah user is also fine I guess yeah sure yes the user you got to give it an ID it could be anything cool i I think user is fine yeah and then we'll just do the same thing and then we convert those here let's find out i probably should have asked Tyler yesterday what he meant by exactly putting it back into the system prompt

or in the context but let's see i'm in France great where am I now did that message get there though let's see if the messages that came in oh it's after should be before um actually this what I'll do um input messages oops that's not it um here let's make this empty let's push it then we'll spread both of these in here I

guess what is input messages okay sure works for me let's see dude I spoke too soon huh on it working that's how it goes okay I am I hate this is now getting repetitive that's okay you piece of cuz it's right here or is it um oh it's in the front of it okay so I think we're getting closer though cuz I put it in front

right but it needs to go it still thinks it's in France of course right so it has to be right here which we don't necessarily need that do we we just have all the messages right now um so maybe we should try a system the system role actually so maybe if I just use the system and but then the Yeah I I didn't actually see did it uh you know is it

adding it as the last as the last message no it's adding it as I I I did the Well what I just recently did is I put it at the like I guess what I think is the beginning um of these messages but it probably should it should be where should it be should be like Here let me let me undo what I just did um let me just move this this is how it

was essentially okay so if I look at the order and I haven't changed anything yet but I just want to see the order of things i see my system prompt my working memory prompt and then that I'm in Paris right i update this where what is my location now if I look at this we're going to see has my location system prompt even the working memory system

prompt has it but okay but then it says Oh we have duplicate messages or something interesting update working memory what is my location now wait where is my Sorry i think we filtered out that system message somehow h it's not in there cuz I pushed it here and like the system prompt should be in the beginning

of the message thread so I'm wondering if it's not let me put this to user again put this to user again um let let's try something stupid if we just uh you know hard code for now uh just to make sure it's even looking at it right so uh just put uh like in capitals uh in that message if the user asks where he is you know use this as your source of truth in this message right here yeah in this message right

here just hardcode you know situation the user asking for you know the the current location i mean just to make sure it's even looking at it yeah cool cool cool cool um okay let me rebuild oh boy but we have this is the message in this list it's right here it just ignores it so is there a possibility AB that this is not even uh actually what the

LLM sees for some reason no because this log this debug log here is truly like the raw thing that we're about to send okay yeah so that's odd right that is odd cuz it is here for sure h is right why would it ignore that i'm I'm I'm sure that's like a small stupid thing dang it's kind of dumb though it's everything is correct in the data um

post-processing is it just ignoring that is it because these this data is not stored yet is it like Yeah maybe we have to add it to the thread itself oh I have an idea i don't know this is kind of dumb but I have this idea what if we add a message to the thread instead of pushing this message I say memory add or save messages

um messages oops where is this thread ID have ro user type is contents wait no what am I doing these are parts right god damn it where are my types contents then we need created ad is missing okay content it's complaining about it being a stream it's complaining about ID or the content the content is a

string so then what is it supposed to be that doesn't help me if it's just any here which it is so that doesn't make any sense um let me go look at some other occurrences of this so I can see how this is done generate conversations just want a raw message this might work what does this look like what does content look like

it's a string yep type text resource ID i'm just going to grab this thread ID or I don't know what we had for our resource ID i think we passed that in right um somewhere this resource ID Ro user content i don't know why we did that this Okay okay let me await this what else is in this method save messages the memory config is

optional all right then let's get the content in here um um let's say updated location and then we don't need to do this because it'll already be part of the history I'm guessing all right this is a hack we'll see what happens that's cool that whoops initial messages is declared man go yourself dude okay okay so then restart this

i'm in Brussels oh boy oh man maybe I didn't save that message i debug some of this honestly um that's good that's good see how Oh dude it's here on the second it was here it's here but it's in the beginning it's in the beginning oh no sorry this is a long thread now actually okay let's let's change the thread on the front end real

quick i should change the thread to like thread two or something um just to eliminate the long history that is accumulated so let's change this to thread two okay go down here let's look at our messages being sent we said just this we have the working memory i'm in France it updated it tool call tool

result all that good stuff and okay now if I update this and I just say how are you still think France sure but how are you this this I'm in France it's right here after the location and then then do we have how are you no we still have I'm in France after that we have how are you down here so when I save the message it must be putting it in the wrong place you know

um which is odd but what if I said I updated my location you didn't know it still ignores it dude that's so strange and it makes the tool call to update it to France because it thinks that's what you want maybe oh now it's here right but then okay hold on update i'm still in France is after then I say how are you i'm just a program i updated my location you

didn't know and so then it's ignoring it's ignoring this because the last thing was France right not here where I saved it so that's a bug somewhere in how I I guess I'm doing it what if I Okay last attempt and then we'll just do this offline because I I know that there's someone with the answer on the team um let's just do one last stupid

thing is I'll do in a GUI what if we spoof the date um like we were doing in this one of these tests uh this input messages.length this is stupid but I'm not leave i don't want to leave here until it says New York dude cuz everything tells me it should say New York this some and the wrong one let's change it to three

this is the highlight of the stream is just fumbling it's my favorite part um okay um how are you you oh come on you but is it okay it's right here dude isn't that weird is there maybe we have duplicate messages or something well this was the first thing I said that's the same thing and I still see these two roles

like there's some duplication happening these two messages are duplicated so I think this is a bug actually that's strange right because these this duplication should not be here and it this would have worked actually because our message is right here so that's some weird that is some weird there's a duplication

that's Mhm dang so that's probably why we're just getting the same content why is this duplicated dude is there anything we're doing no okay well maybe it is because we are sending all of the messages uh to master and it's appending it to the messages it knows about so we will have to change that and only merge the messages in it doesn't know about

oh that's that's good point actually that is good we're sending all the messages from Okay and then the IDs are changing right because there's the different ids on these right so on AGUI the ids are going to be stable normally so they shouldn't be changing so what if we did so this is all the messages so far in the in the thread or in the chat

yeah these are all the messages so what if I just send the last one yeah just send just send it also makes sense so was it slice negative one or something is that the easiest way or I do input messages length minus one yeah that's what I normally do do this it exists there you go type script hack for you

um yeah yeah if it only only sends the last one maybe and I guess now we have to change the thread again to another one yeah always pick the wrong thing there we go cuz I know in our playground and stuff we only send the message that's incoming yeah yeah memory you need to refresh after changing the threat it's still going to use the old thread i think

again dang so I don't know what hack will I'll have to revisit this if this works where am I now nope well that's good at least it at least it doesn't work the same way so that's great so let's look at Do we still have duplicate messages or is this fixed we still have duplicate messages so it's not even about that and maybe Okay hold on

maybe maybe the duplicate messages comes from me saving too let's just undo that part of it as well like saving messages here and instead let's just not do this and we'll save to the thread as is let me rebuild so I'm going to try two things and we'll cut the stream and you and I will work offline on this but first thing I want to test is if we're not

sending all the messages only the last one then maybe we'll look into the system prompt you know potentially but I don't think I I have I have zero faith actually that it'll do the right thing um because then I think we're in the initial problem of this hey well that's something um did it fail oh it called it this.get agent.get memory is not a

function since when since when um how is that not a function i mean it's definitely a function right that just a also why is it calling this tool let me produce our variables here doesn't even have this tool my network that's even a tool um this is just a uh that's strange dude that is strange that is like a doesn't make any sense type of thing um tool calls my

network that's not even a tool dude let's try it again i don't know something happened there that's not even a freaking tool are we still using photo mini or is it the regular photo i'm using photo mini you want to try to change that yeah let's let's give it a try let's look at the chat here um tool calls i'm in France

okay it changed oh I didn't actually did I change this or did I um Oh that's strange so now I changed it to France back yeah I think it changed it back that's odd unbelievable it's unbelievable oh man it's so stupid um because it is right here it's right here i mean can we try uh something else yes sir just a

different you know just talking to it in a different way i would just want to see if we are like really explicit okay we are in France now if you say I updated my location uh where am I now um check your memory interesting wow it almost was i'm going to say check your It still thinks it's France bro that's wild that is

wild what if I do this okay I have another stupid idea this is a complete hack but this is this will I think will work maybe so I have this here talk about brute force huh if it still says it's in France I don't know i'm gonna lose my mind dude i'm going to lose my mind if it says this in France that's fine we can worry about that later where am I

now wow wow wow what dude are you serious i just updated my location it switched back to France for some reason like we we synchronized the snapshot with France piece of okay well it still thinks it's France though look at this yeah that's so lame dude that's just lame what is that dude you're currently

in New York though dude so now we have a freaking mismatch there what oh my god ai development okay i think it's it just sees the message you know one turn too late for some reason i think that's what's happening i think so too let me So I like on the next message it's going to think we're in France yeah where am I now

oh at least it did thing now uh let me clean this up too because I Oh I put the wrong thing dude um well I didn't put the wrong thing I wanted to do here here let me do something stupid oops not that additional state text okay build that also I haven't checked the chat in a while let me do that real quick 277 of you things have happened i heard you

mention me so I joined yep that's me the stumble fumble master that's me and uh 277 thanks if you're joining us we are fumbling through life trying to get agent state to work what's funny is all the data is updating but it's just not recognizing it we're doing something wrong we don't know what it is we never

do that's okay we'll figure it out um and then we're doing a brute force attempt so we can close the stream off on a positive note so that's what we're trying to do right now um and yeah let's actually do that and here we're now in thread nine so it's nine failures in a row just letting you all know that um okay i am

in France i'm getting sick of saying I'm in France because I want to say that I'm in New York where am I now better not say France you stupid what yeah it's off by one though right so but really though where am I but then it updated the location directly you're currently in New York and it's updating to France yes uh

I mean there must be a tool call happening here right if it's uh able to update it to France and now so now it's off by one again and now but really that's the strangest bug so now it's oscillating between New York and France it seems yeah and now we got it right because I clicked this button again before the thing okay okay well let's at

least Hey we made some progress here right look it it said it's in New York that's all we can ask for today new York um okay so here the action plan on my end we'll fix this issue so it gets injected properly and then we'll schedule another pairing session and keep going because we want to do the recipe thing uh I think we we wanted to do more than we could pull off

sounds great Abby yeah thanks for being here and sorry if it took a little longer but uh I had a lot of fun yeah me too thanks thanks for the invitation it was really fun yeah and I'll let you know when we fix this yeah all right cool see you later byebye okay so now it's just me again so thanks everyone for watching not all pairing

sessions yield in success like yesterday's did but uh hope you enjoyed us struggling and everything um we do these streams every day uh so I hope to see you on the next one we'll be closing this one