Morgan Stanley AI Integration and Evaluation Framework

Morgan Stanley has partnered with OpenAI to integrate GPT-4 and Whisper into its financial services workflows, enabling financial advisors to retrieve information faster and automate repetitive tasks. This deployment is centered on a robust evaluation (eval) framework that ensures AI performance meets the strict quality and reliability standards required for financial services.

Rigorous Evaluation Framework for AI Deployment

Morgan Stanley utilizes an evaluation framework to test every AI use case before deployment, using expert feedback to measure model performance against real-world scenarios. This framework allows the firm to refine prompts and improve output quality through specific testing methodologies:

  • Summarization Evals: Used to test the model's ability to condense intellectual capital and process-driven content into concise, accurate summaries.
  • Translation Evals: Implemented to support multilingual clients.
  • Regression Suites: Daily testing with sample questions is used to identify weaknesses and ensure the system delivers compliant outputs.

Through this approach, Morgan Stanley expanded its internal knowledge retrieval capabilities from answering 7,000 questions to effectively answering any question from a corpus of 100,000 documents.

AI-Powered Tools for Financial Advisors

Morgan Stanley has deployed two primary AI solutions to enhance advisor productivity and client engagement:

AI @ Morgan Stanley Assistant

An internal chatbot that allows financial advisors to seamlessly retrieve information from the firm's knowledge base. Over 98% of advisor teams in wealth management actively use this tool.

AI @ Morgan Stanley Debrief

Powered by Whisper and GPT-4, this tool converts Zoom recordings (with client consent) into actionable outputs. It automatically integrates client notes into CRM systems and generates draft follow-up emails summarizing key action items for advisor review and refinement.

Security, Compliance, and Data Privacy

To maintain financial services compliance, Morgan Stanley integrated quality assurance directly into its evaluation framework and worked with OpenAI to optimize retrieval methods for higher accuracy.

Data privacy is managed through OpenAI's zero data retention policy, which ensures that Morgan Stanley's proprietary data is not used to train public ChatGPT models.

Impact on Advisor Productivity and Firmwide Adoption

The implementation of AI tools has resulted in significant operational improvements:

  • High Adoption Rates: Over 98% of advisor teams in wealth management use the AI tools daily.
  • Increased Document Access: Access to internal documents has increased from 20% to 80%, significantly reducing search time.
  • Efficiency Gains: Follow-ups that previously took days are now completed within hours, allowing advisors to spend more time on client relationships.

Future Scaling and Institutional Expansion

Morgan Stanley views these platforms as a "super app" for employees, creating a foundation for further expansion. The firm is currently scaling the Assistant functionality for its institutional securities group and exploring the application of the Debrief tool for other roles, such as investment bankers communicating with CFOs.

Sources