AIfication of Everything

The AIfication of everything makes one think what is actually generating long-term values and what’s just hype. In this post, I’ll show you how to add AI, ML, magic, whatever you want to call it, into SaaS apps. We’ll go over what you can do with traditional machine learning models and what would require the use of an LLM.

LLM as the De Facto Default

Forget it, let’s just throw everything at an LLM!

Well, that works if you assume you have little data, few users, or zero-cost inference. Cost-wise, it’s a non-starter for most applications. LLMs seem easy to plug in, but they’re fraught with landmines like limited context windows and hallucinations necessitating complex RAG pipelines for ingesting and querying data.

We can think of LLMs as either the first mile or last mile interface. For many applications, you can use the LLM to understand the intent of the user, or to generate text. While you can use it as the engine for everything in and out, it may make more sense to keep it at the periphery for cost, latency, and security reasons.

AI and ML Patterns: LLM vs Classical AI

Classical AI/ML already powers many recommendation engines and features like collaborative filtering, but it’s the LLM that really makes natural language functionality shine. What was previously the domain of semantic search can now be done by LLMs, or can it?

Natural-language interface to improve UX (the LLM sprinkle):

  • Crafting complex filter expressions using natural language instead of manually writing filter expressions
  • Generating database or warehouse queries
  • Navigation inside complex hierarchies (take me to a specific record)
  • Auto-fill forms based on natural language

Text generation:

  • Summarize long customer support threads and customer notes
  • Generate drafts for customer support messages
  • Summarize commit history

Classification:

  • Automatically tagging or categorizing items
  • Predict support ticket priority
  • Spam detection, sentiment analysis, prediction

Forecasting:

  • Generate usage alerts (e.g., customer will exceed plan limits soon)
  • Churn prediction (e.g., user hasn’t logged in for a week)

Recommendations:

  • Suggested next actions (e.g., customers like you did X next)
  • Context-aware autocomplete for form fields and drop-downs

Personalization:

  • Adaptive sorting (e.g., show frequently-contacted customers first)

Paradox of AI

Developing with AI is like having lived your entire life in the digital realm, and discovering analog. Real-world data is messy, and needs cleaning, massaging, and domain-specific intuition to make AI work reliably. Regardless, at some extremes, it can still break unpredictably. If your data has no semblance in the AI’s training dataset, the solution may occasionally fall apart. And that’s the paradox of AI: promise of digital magic, but in a deeply analog reality.

Best Practice: Combine Structured Predictions with GenAI

The key insight is to combine structured predictions from classical ML models with generative AI. Often, one or the other isn’t enough, especially with higher expectations from modern AI. Today, an LLM isn’t accurate enough or just doesn’t fit the dataset no matter how hard you try.

Let’s say you want to add natural language filters to your app, in which a user can type a query like unpaid invoices > $10k from vendors in EU and have the UI automatically apply the necessary filters in your search UI. A first-pass at the problem might be to use an LLM and populate the context with:

  • User-visible filter names, allowed values, and operands
  • User’s query
  • Desired output format such as a filter expression or URL query string

This solution quickly becomes untenable if you have thousands of filters, complex rules (e.g., can’t use certain filters together, data-dependent filters, etc.), or latency requirements. It may not be cost-effective to have a general-purpose LLM in the hot path for a feature where each query might cost $0.01. It definitely adds up.

Instead of jumping to an LLM first, consider designing such an AI-assisted feature as a pipeline of structured and fuzzy stages:

  • Structured parsing: extract keywords like unpaid, invoices, and vendors, operators like >, and data values like $10k and EU
  • Canonicalization: map synonyms like clients → customers or Europe → EU countries
  • Rule validation: check allowed fields, operands, and business rules
  • Tiebreaker: use an LLM only when intent is ambiguous

You’ll inevitably need to capture data and test it against your pipeline. In such a setup, the LLM is only used in the end, when the query doesn’t cleanly map to expected patterns, or to resolve user intent. Another option is to fine-tune a smaller model to answer these kinds of queries.

In practice, you might be able to bypass an LLM entirely, or use it in 5% of cases instead of 50%, a huge win for cost and latency.

Tips for CxOs and Implementers

Right now, many companies are focusing on AI, especially generative AI, as the entry point of a product feature. I don’t know about you, but I’m over chatbots. I’m beyond fatigued. They’re fine for capturing user intent, but shouldn’t be the main way people use your product. Agents are here, but driving agents via a chat interface isn’t the final destination either.

Don’t start with “how can I add AI?” but rather “where’s the friction?”. The best AI features are invisible and address pain points without being front and center.

I think of AI as the new Perl. In the 1990s, it was everywhere, powering websites and at the heart of text processing workflows. But over time, the magic faded as other programming languages gained similar features, and Perl lost its limelight. I’m not saying AI will go the same way, but the euphoric hype will inevitably shift to the next shiny object. Let’s try to avoid making the mistake of blindly jumping on the bandwagon and understand what it is we’re supposed to be doing with AI in the first place.

AI will disappear into the workflow. The sooner we stop talking about it, the sooner we’ll recognize that it’s just another tool in our massive toolbox.