I recently organized and presented a session at our Nashua Cloud .NET User Group on “Who Runs the Loop? From Chat Completions to Responses API”.
The main question I wanted to explore was simple: When building AI agents, who should manage the execution loop — our application or the AI runtime?
With Chat Completions, we typically manage tool calls, conversation history, and the execution loop within our application. Responses API takes a different approach, where the runtime can handle some of these responsibilities, especially with hosted tools and MCP integration.
During the session, I walked through the architectural differences between the two APIs, conversation state, reasoning continuity, Hosted MCP vs application-managed MCP, and how Microsoft Agent Framework helps us work with both APIs using C# and Python.
I also shared a demo comparing three approaches:
- Chat Completions with application-managed MCP
- Responses API with application-managed MCP
- Responses API with Hosted MCP
One interesting observation from the demo was that Responses API significantly reduced the amount of data our application sends and the number of calls it needs to manage. However, Hosted MCP was not necessarily faster. In some of our tests, it actually took longer because of the additional work happening within the runtime.
For me, the takeaway is not that Chat Completions is outdated or that everyone should immediately move to Responses API. Both have their use cases. But if I were starting a new agentic application today, I would lean towards Responses API and use Microsoft Agent Framework to keep the application independent of the underlying API or model provider.
The bigger shift is not just about choosing an API. It is about deciding where orchestration, tool execution, and state management should live, and how much of that responsibility we want to keep within our application.
If you missed the session, I’ve shared the recording and demo code below.
🎥 YouTube Recording: Watch the Session
💻 GitHub Demo Code: Chat Completions vs Responses API
Would love to hear how others are approaching this, especially when it comes to Hosted MCP vs managing tool execution within the application.