Taking off the blinders: LLMEx for machine learning explainability

In the modern enterprise, stakeholders acknowledge the value of predictive models, but they struggle to trust outputs that they cannot interpret.

Blindly trusting opaque numbers is error fraught, especially in high-stakes sales pipelines where a misunderstood signal can lead to a missed opportunity or wasted effort. Data always has “blind spots,” and without visibility, our stakeholders are operating in the dark.

The solution: LLMEx

To bridge this gap, we developed LLMEx, a framework designed to make complex model outputs human-readable without sacrificing mathematical integrity.

Our core philosophy is simple: by articulating the rationale behind a predictive score in plain English, we provide stakeholders with the context necessary to fill in the blind spots. When users understand what tipped the model in a specific direction, they are no longer just consumers of a prediction; they become active partners in the process. This transparency empowers them to apply human intuition to adjust, refine, or even reject the model’s advice on a case-by-case basis.

Under the hood

At its heart, LLMEX is about extracting meaningful signals from model output and converting them into actionable narratives.

Building on feature importance techniques like SHAP which provide scores on its decision factors, LLMEX translates those scores into intuitive, plain-English rationales. However, generating natural language is not enough—it must be accurate.

To ensure integrity, LLMEX employs a dual-model training architecture:

1. Rationale Generation: The first model call generates a human-readable explanation based on the underlying feature ranking.
2. Reconstruction Verification: The second model call attempts to reconstruct the feature rank ordering solely from that rationale. Crucially, this second model cannot see the raw feature rank inputs.

This “double-check” process acts like an autoencoder pattern: it encourages the rationale generator to encode the ranking faithfully. If the explanation is hallucinated or lacks depth, the second model (call it the Judge) will fail to reconstruct the feature rankings. This ensures the output is both trustworthy and grounded in the actual feature importance scores.

Engineering for confidence: DSPy

Building complex LLM systems can be brittle and time consuming, especially when relying on traditional “prompt engineering.” To iterate more quickly and ensure our explanations are reliable, we optimize our LLMEx system with DSPy.

DSPy provides the software abstractions we need to organize multi-module LLM systems and jointly optimize their prompts. This approach is often as powerful as fine-tuning but offers more flexibility, for example, by removing the need for complex GPU infrastructure. Moreover, it is bitter-lesson pilled: To benefit from the latest frontier intelligence, we just “re-compile” LLMEx using the newer LLM model.

The bottom line, however, is that by testing our explanations over a large, representative test set using DSPy, we can deploy LLMEx with statistical confidence. We are no longer guessing whether our model is hallucinating; we are measuring its performance against ground truth.

Experiment

Below I describe experiments I did to illustrate the power of this method to increase trust in the English outputs. This is a simplified setup compared with our production system with many variables removed. A discussion follows below about those differences. What I show is that prompt optimization can drastically improve over the use of a naive prompt.

Setup

I began with a sample of around 1,900 accounts from Demandbase’s own data. We run our proprietary Pipeline Predict model across all accounts on a daily basis. The results of that prediction include feature importance structures stored in our database systems. We only look at the Likely and Highly Likely accounts for LLMEx. I first selected a holdout of 50 accounts at random for the test set.

As a baseline, I ran the LLMEx simulation program without DSPy optimization. Then, starting at 200 training samples, I trained the model (i.e., optimized the prompt and in-context examples). After each training session, I saved that model and then doubled the amount of training examples until I reached the full training pool at around 1,600 samples. I also provide a fixed validation set to the optimizer of 180 samples.

The metric for grading the explanations (Figure 1) is a simple pairwise accuracy augmented by a coverage and purity score. These are analogous to recall and precision and check that the Judge found all and only the feature names provided. These are combined into an F1 score and multiplied times the pairwise accuracy for a final metric. I use this metric in DSPy’s MIPROv2 optimization (a.k.a., compilation) and use it to compute test set accuracy after training (or before in the baseline case).

Figure 1. Breakdown of the evaluation metric.

Results

As you can see below, with each training round the ability of the Judge to determine the feature ranking from the text increases for a total increase of +14 from 73% to 87%. This confirms the hypotheses that (1) optimized prompts with few-shot examples indeed improve the power of the model to communicate the required information, assuming the LLM is a suitable proxy for human readers, and that (2) increased training increases that accuracy similar to traditional machine learning.

Figure 2. Progressive accuracy as training sample size is increased.

Cheating mitigation

One reality that has set in during the new era of LLM evals is the specter of cheating. Simpler models like SVM or Random Forest don’t have a motivation to cheat or any motivations at all. But LLMs, whether we call it a motivation or not, have a documented history of cheating with the clear intention of boosting their apparent accuracy.

While this phenomenon typically crops up during agentic evaluations where LLMs have the ability to observe and change their environment, our simpler task can still fall victim. During the product development cycle, I noticed blurbs where the encoder model would insert the feature importance numbers directly next to the name of the feature, giving the Judge what it needed to perform a boosted evaluation.

While I can imagine other ways a model could cheat, I took the most straightforward approach to mitigating this behavior. Whenever a feature importance value is detected in the text, that sample is immediately given a score of zero.

The quality of these texts is highly important. I want our customers to be sharpened over time by providing accurate and insightful glimpses of the impact of their go-to-market motions. Cheating mitigation is an important tool in that endeavor. Cheating mitigation is effectively hallucination mitigation, because it ensures our statistical rigor isn’t hindered by cover-ups.

Simulation vs production

The simulation above is meant to demonstrate the power of prompt optimization to increase the faithfulness of our predictive explainability. Much more goes into our process on the platform. Additional measures we take during the training procedure, for example, include penalties for extra long blurbs and markdown quality misses. Moreover, since Pipeline Predict is focused on timely market activity, firmographic information about the target account, despite often being important, is included towards the bottom of the blurbs.

Conclusion

Ultimately, transparency isn’t just a byproduct of good design; it is the catalyst for AI adoption. By using frameworks like DSPy to rigorously optimize our rationale generation, we have moved beyond simple interpretability to verifiable trust—demonstrating that even in complex systems, we can quantify and improve human-readable outputs by a significant margin. This doesn’t just demystify the ‘black box’; it transforms AI from an inscrutable oracle into an active partner. By merging the statistical rigor of traditional machine learning with the generative fluency of LLMs, we are not just improving a tool—we are redefining the partnership between data and human intuition, ensuring that our decisions are grounded in evidence rather than blind faith.