Building LLM products looks easy, but looks can be deceiving
10 challenges product teams must master for LLM success
Leveraging LLMs is easy.
Large Language Models (LLMs) like ChatGPT, Bard, and Claude make AI look easy.
Sign in, access the API, write a few prompts and you are off to the races.
No wonder, product teams are eager to incorporate them into new applications.
The hard part is everything else.
Unfortunately, the hard part is everything that surrounds the LLM.
Here are the 10 things product teams must get right in order to launch a successful LLM product:
Building the right use case. LLMs can do a lot. So it is tempting to do a little bit of everything and see what sticks. However, this scattergun approach rarely works. Your better bet is to ruthlessly prioritize and focus only on a handful of ideas that solve a customer problem, deliver value, align with business objectives, and are technically feasible. This foundational alignment sets your LLM efforts up for success.
Assembling the right technology stack. LLMs require an extensive tech stack for deployment, security, monitoring, etc. A typical stack may include APIs, app servers, databases, user authentication, hosting infrastructure, CI/CD pipelines, monitoring tools, and more. Managing all these components is complex. LLM is the easiest part (unless you decide to go build one yourself from scratch).
Generating consistent high-quality responses. Hallucinations and incorrect outputs are common failure modes for LLMs. Too little context yields generic answers, while excessive information confuses the model. Finding the right balance requires diligent tuning of prompts, models, and other variables plus rigorous experimentation.
Acquiring initial users. Driving adoption for any new product is hard enough. With LLMs, you have the extra challenge of overcoming skepticism, fear, and lack of know-how. Simply showcasing capabilities is insufficient - you need to put together a comprehensive GTM market plan that can gradually overcome hesitancy and prove LLMs' merits.
Finding the right resources. LLM experience is still relatively new. If you are hoping to “hire” this talent, think again. In the near term, training and education are your best bet. And not just for your engineers. You will need to bring the rest of the cross-functional product team up to speed as well.
Creating the right team structure. LLMs are inherently ambiguous. A few engineering resources are not going to cut it. What you need is a true cross-functional, empowered, and agile team. At a minimum a mix of product management, engineering, design, content creation, and QA resources.
Following the modern product development process. I teach a class on LLMs, and everyone there wants to just start building. The hard part is having the discipline to go through the modern product development steps - journey mapping, user interviews, prototyping, iterating, etc. - so that you are building the right product.
Deploying and managing the application. LLM applications are inherently brittle. First, it requires careful tracking of code, prompts, model versions, temperature, embeddings, etc., to maintain performance. Second, the parameters are constantly changing. Abstracting and modular architecture help, but maintaining production code (vs. conventional software) is expensive, as of now.
Providing security and privacy. This is a hard one. LLMs work best when they have access to the right data for the right user. But that level of access control, especially, for large enterprises does not exist yet. For now, teams must rely on traditional security controls to manage access, minimize exposure, and prevent breaches.
Controlling costs. The LLM API may be cheap, but everything around the tech stack is not. Additional resources for security, ops, engineering, content creation/curation, etc. add up. Poorly written code and infrastructure make it worse. To develop responsibly, track model usage and system resources to manage and allocate expenses.
By focusing on the above 10 challenges, product teams can systematically address the multifaceted challenges of building next-generation LLM products.
Happy building!!
If you enjoyed this, please turn that ♡ into ❤️. This will help others find this newsletter. Thank you so much 😁