MCP Dev Summit North America

At Duolingo, we built an AI Slackbot using Claude's AI Agent SDK and the Model Context Protocol, connecting 180+ tools across 15+ MCP servers—GitHub, Jenkins, Sentry, Grafana, PagerDuty, Honeycomb, and more.

This talk covers:
1. Building and Standardizing MCP Servers: Patterns for stdio and HTTP transports with consistent configuration schemas.
2. Authentication and Role-Based Access Control: YAML-based tool allowlists gating sensitive tools by user role.
3. Evaluation Framework: A Python script runs parallel evaluation tests validating MCP tool usage, anti-hallucination checks, and response quality.
4. Real-World Challenges:
* Testing in production without synthetic environments
* Preventing customer PII from reaching AI models
* Performance optimization: Tradeoff between fast answer vs good answer
* AI hallucinating answers instead of calling MCP tools
* Wrong MCP parameters causing slow queries and failures
5. Live Examples: PagerDuty triage, Jenkins debugging, and incident investigation—all from Slack.