Disclaimer: I work at Microsoft at the time of writing this, but the perspectives and experiences shared here are my own.
I’ve been spending time with Microsoft Azure AI Foundry, and one thing became clear very quickly: if you don’t get the mental model right, everything feels confusing.
I built a Pizza-Ordering Agent guided by this tutorial from Global AI Community Vancouver Chapter to anchor my exploration with Agents and Azure AI Foundry. This combined system prompts, document retrieval (RAG), custom tool calling, and real-time integrations.
So while I love pizza, I want to instead discuss my findings on how to think about agents and this ever evolving AI landscape.
Azure AI Foundry is a single pane of glass from Microsoft for building and operating generative AI systems. I used GPT-4o model and while deploying the model itself was straightforward, it’s the nuances that was the most interesting. For example, when you “deploy” a model you are not actually deploying anything, it’s mostly a routing you are creating to already running instances of the model.
As for the concept of agents, I was already familiar with what is an agent but I was more interested to get into the developer experience of building Agents leveraging Azure AI Foundry.
One of the most important ideas I had to wrap my mind around in Foundry is that state belongs to threads (not to be confused with classical OS terminology) and not the agent.
Threads hold messages, actions, and tool calls. Agents are stateless as they define behaviour and not memory. I like this separation between state and behaviour as it makes it easier to understand how multiple agents can interact on one thread of activity.
The next mental model to get comfortable with is that a ‘Run’ is a unit of work i.e. a single execution of a thread by an agent using the toolset it has available to reason and produce an output. Easy!
Another mental model that is needed for Foundry is how domain knowledge using RAG design pattern is made operational in Foundry. In Azure AI Foundry, this is done by a combination of Vector Stores which is the embedding of the domain specific information and File Search which is the capability of pulling information from the store. This capability is attached to the agent as a ‘tool’ it can leverage. So the agent isn’t given the knowledge upfront, it’s given the ability to retrieve knowledge when needed. This concept of tools is very fundamental with using Agentic frameworks using FunctionTools can give a deterministic lever that a fundamentally non-deterministic workflow i.e. the agent can utilize.
While doing all of this wiring and leveraging of these functionalities, I was using Azure AI Foundry’s Python SDK. I found it intuitive to use and in my limited use, the only area that was not as intuitive for me was how Content Blocks are structured in thread messages.
Another aspect I explored deeper in this exercise was leveraging tools via MCP. I will discuss this more in a future blog.
To wrap up, as a software engineer who has spent my career hard-coding flows, designing systems around the concept of outsourcing reasoning to an Agent required new mental models. Once you get a few of those in place, the clarity flows.
The only downside here was that since this was not a production system, no pizza showed up at my door once I was done!