The enterprise software maker SAP held its annual Sapphire event in Orlando, Fla. this week, drawing 15,000 attendees over three days to the Orange County Convention Center. There were the usual keynotes, a massive trade show floor, and media-only previews and demos. I also got a chance to sit down with the Global Head of AI for SAP, Dr. Walter Sun.
The conference opened with a fast-paced 90-minute series of talks and demos from SAP leadership, and case studies from spotlight customers like NBCUniversal and Standard Chartered. The star of the show was “Joule,” SAP’s GenAI copilot, which debuted this February and took center stage in every demo throughout the conference. Joule is intended be used across the full ecosystem of applications, acting as a just-in-time assistant in both SAP and non SAP applications. And it ability to access the full web, thanks to a just-announced partnership with Perplexity.
Here are the highlights of my interview with Sun, who joined SAP in 2023 after serving as a Vice President of AI at Microsoft.
• • •
Let’s start with your current portfolio at SAP. What aspects of AI are you overseeing, and how does your work ladder up to the broader corporate strategy?
I’m responsible for researching and testing a full suite of AI capabilities, from narrow AI for domain-specific predictive analytics and those kinds of applications all the way up to the fast evolving space of GenAI and LLMs.
I work closely with my colleague, Muhammad Alam. In his role as Head of SAP Product & Engineering, he is responsible for product innovation and the design of AI-powered products like Joule, and I am principally looking outwards at what’s coming, helping make smarter strategic bets on these emerging technologies, and looking for many different reusable skills.
An example of reusable capability would be something like our “Document AI,” which can process receipts from hotels or dining expenses stored in Concur (an SAP owned enterprise expense management platform), and that skill can be re-used or applied to extracting key data from invoices in, say, a transportation/logistics use case.
How do you stay on top of this fast moving space of AI, and especially GenAI?
It’s an incredibly fast moving target, for sure. We do leverage our internal SAP workforce to help us with evaluating many of the generative possibilities. We created an internal Generative AI hub where our users can interact with over 30 different LLMs, to experiment with various prompts, using our internal datasets, and importantly, providing feedback. It’s a space for experimentation that lets us really see what’s working and what needs adjustment. It’s a place for any of our executive teams to quietly experiment and learn and also build up their knowledge base around GenAI.
To give a sense of what we are testing, it includes a range of small, medium, and large offerings from the big frontier vendors such as Anthropic’s Claude, OpenAI’s ChatGPT, Google’s Gemini, Meta’s Llama and even Mistral, which is a unique partnership where we can host their LLMs in our own SAP data centers.
I have a sort of mental model or set of categories for thinking about the space of LLMs. I want to explore both the open and closed LLM ecosystems, and I’m tracking the very large generic models, and the potential for creating very small distilled instances from those large models for domain-specific use cases.
To use a car metaphor, there are high-end luxury vehicles that set the bar for performance, and then car manufacturers will scale those down for other segments like mid-range and commodity vehicles. We are similarly looking at the most high-performance LLMs, and then the scaled-back versions of these top line systems. Essentially, we’re asking the question, “How can we deliver the same capability with a smaller version of a model at one-tenth the cost?”
In your experience, what’s the first mistake most companies make when deploying AI? Is it technical debt, lack of clarity on use cases, or something more human — like organizational fear, or hype-driven investment?
The human and social dimensions are very important. We are always looking to see how we can empower teams to access the tools and experiment. In addition to our standing Generative AI hub, we have lots of “AI Days” — basically hackathon events. As a general rule, we want to put the latest stuff in front of our employees first, before our customers even see them, because we know that we will get trusted, honest internal feedback.
You might say SAP favors a “bottom-up innovation” approach when it comes to our AI and GenAI product development. To go back to the Document AI case I mentioned before, that reuse idea came out of a hackathon session. We were able to blend the best of narrow AI skills that covered 98 percent of the printed receipts, and then we were able to use GenAI to cover the long tail of receipts that were perhaps handwritten or less logical. So we built this premium document extraction ability to solve that issue, and now we can cover 100 percent of the expense receipts.
During the keynote, three key pillars stood out: Joule, Joule Agents, and the AI Foundation. Can you give us a simple explanation of what sits under those three pillars?
Joule is our SAP-tuned and native co-pilot. Customers use it to access any SAP data across the full suite of applications covering many lines of business. And as we showed in the demos, leveraging our acquisition of WalkMe, we are able to let Joule access non-SAP application data in order to integrate across the full ERP value chain.
Agents are specialized tools that are unique to each line of business. For example, we created a highly-tuned Concur agent for expense management, and we have a supply chain agent, and so on. The role of these under-the-hood software systems is to transform long, asynchronous, manual business processes composed of back-and-forth email chains into simple automations that are like a supercharged version of robotic process automation, or RPA.
The AI Foundation is our full corporate operating system for enabling and supporting our Business Technology Platform (BTP) customers, granting them full access to our Generative AI hub, where they can experiment and no special knowledge or skills are required. Our BTP customers also know we are running the hub in a safe data sandbox, because we take security and privacy very seriously. This is an enterprise-grade experimentation center.
SAP CTO Philipp Herzig said “prompt engineering is dead” during his talk, which got a chuckle from the audience. The new model for SAP is “prompt optimization.” What’s the difference?
Many people have discovered the hard way that designing or engineering prompts is all well and good, until you migrate to another LLM. And firms might and often do need to migrate for many good reasons, like data hosting in particular geographies, or to access a new capability available in one system and not in another. And when they switched out the underlying model, their prompts all broke — [they] just didn’t perform the same.
Prompt optimization is about being able to strategically design prompts for talking to many different LLMs successfully. The term today is “evals,” systems that can evaluate prompt strategies and help people make genuine apples-to-apples comparisons between models. The bottom line is, prompt engineering is too limited a view. We believe one must have a holistic perspective across the whole ecosystem and optimize for that reality.
This morning, your colleague Muhammad Alam talked about an apps/data/AI flywheel, and to get it spinning, SAP’s role is to simplify the AI infrastructure and plumbing, so customers just focus on apps and value. Do you have a particular customer story that highlights how SAP drove innovation and value creation with AI?
Team Liquid. They are a leader in the e-sports space. They’ve been an SAP customer for a long time, and this week we’re showcasing what they’re doing with Joule agents. The AI agents give them instant access to game statistics, player performance trends, and strategic comparisons using natural language. It’s democratizing these insights across their team, simply because the user interface is just natural language. Users access insights through AI-driven queries. The real-time data comparisons allow them to make precise, data-backed decisions, which provides a competitive edge in super-competitive and dynamic gaming environments.
You must be logged in to post a comment.