Hi there, my name is Jakub and today I’m going to tell you a little about potential problems of using LLMs in areas that they were not originally intended to be used in. This presentation was prepared for DEVAI conference by Academinc Partners Foundation, that took place on 4 and 5 December 2025
We’ve all seen the tutorials. You import a library, write five lines of code, and suddenly you have a chatbot that can explain set theory or write a poem. It feels like magic.
But then, you try to build a production system. You point that same “magic” code at a directory of messy PDF invoices, and the illusion shatters. The PDF headers are invalid. The “reasoning engine” hallucinates the total tax sum—giving you $551.67 one minute and $510.67 the next. When you ask for a list of invoice numbers, it returns a JSON array once, a comma-separated string the next time, and a polite sentence the third time.
This is the “5-Line Code” Illusion. While “magic” AI frameworks are incredible for prototyping, relying on them for production is a disaster waiting to happen.
Here is why we need to embrace “boring” engineering patterns to build GenAI projects that actually work.
The Problem with “Magic” Frameworks
When we use high-level abstractions to interact with Large Language Models (LLMs), we treat the LLM as a brain. In reality, it is just a weird function.
Frameworks often act as a black box. Deep inside that box, there might be a hidden system prompt telling the model “You are a helpful pirate,” or an auto-injected history that eats up 4,000 tokens of overhead that you didn’t ask for.
When the output inevitably fails, you are left with a “Format Error” and no way to debug it because you lack information on how the result was generated. You cannot fix what you cannot see.
Solution 1: Explicit Prompt Management
If you can’t see the prompt, you can’t control the output. We need to move away from hidden framework prompts and toward Explicit Prompt Management.
One effective way to do this is using template engines like Jinja2. Instead of letting a framework guess what you need, you explicitly render the system prompt. This allows you to:
- Inject Safety Rules: Dynamically insert rules like “Only propose flights within the EU”.
- Enforce Prerequisites: Check for missing details (like dates or budget) programmatically before the LLM is even allowed to answer.
- Avoid Guessing: Instruct the model to ask clarifying questions rather than hallucinating an answer.
Solution 2: Validation and Parsing (The Pydantic Way)
An LLM might promise to return JSON, but it will often return garbage. To fix this, we need structured data extraction.
We can combine LLMs with tools like Pydantic to enforce strict schemas. For example, if we are building a pizza ordering bot, we shouldn’t just ask for an order; we should demand a specific JSON structure where the pizza type is a Literal (e.g., “margherita”, “pepperoni”) and the quantity is a strictly validated integer.
If the LLM outputs a quantity of “-5” or a pizza type that doesn’t exist, the validation layer catches it before it breaks your backend.
Solution 3: Taming Chaos with State Machines
Agents acting in an open loop are prone to getting stuck or drifting off-topic. To solve this, we can use the “good old” State Machine pattern.
We should treat the LLM as a router and a parser, not a calculator or a manager.
- Intent Classification: Use the LLM to understand what the user wants.
- Structured Extraction: Use the LLM to convert text to data.
- Workflow Enforcement: Use a Finite State Machine (FSM) or graph library (like LangGraph) to control the process.
- Calculations: Use Python. Never ask an LLM to do math; use NumPy or standard code.
A Better Workflow: The Pizza Order
Instead of a single prompt, a robust flow looks like this:
- Parse: The LLM extracts the order into a Pydantic object.
- Validate: Python code checks business rules (e.g., “Too many different pizzas”).
- Price: Python code calculates the total based on a deterministic price table.
- Confirm: The LLM generates a friendly confirmation message using the validated data.
Conclusion: Observability > Abstraction
The Golden Rule for GenAI production is simple: If you can’t easily debug it, don’t deploy it.
We must stop chasing the “magic” of one-liner code and start applying solid engineering principles. Logging exactly what you send to the API—full prompts, tokens, and costs—is infinitely more valuable than an elegant abstraction chain you cannot introspect.
In the GenAI era, the winner isn’t the developer using the most magic. It’s the one applying the most solid engineering.