Researchers have developed a new method called FERN to improve how computer systems identify relevant database tables when answering questions posed in natural language. This step—retrieving the right tables—is crucial for text-to-SQL systems, which translate everyday language queries into structured database commands. The new approach aims to match or surpass existing methods that depend on large generative language models, but without using costly or complex AI generation. This could make text-to-SQL tools faster, more efficient, and more accessible in practice.
Key Takeaways
- FERN combines three fixed (“frozen”) retrieval models to select up to 100 candidate tables for a given question, then ranks them using a machine learning model trained on features describing each candidate.
- On the popular Spider benchmark, FERN’s candidate selection alone matches the recall of a leading generative model-based system (RASL) when looking at 15 tables, and its ranking significantly improves the chance that all relevant tables appear in the top 5 results.
- FERN performs competitively on other benchmarks like BIRD and BEAVER, even without being specifically trained on those datasets, though it falls short on the Fiben dataset, indicating room for improvement in some contexts.
- The approach achieves fast response times (median 208 milliseconds per question) on powerful GPUs, without requiring calls to hosted generative AI services.
Text-to-SQL systems help users query large databases using natural language instead of complex coding. However, these systems first need to figure out which tables in a large collection are relevant to the user’s question—a challenging retrieval problem. Many recent methods rely on generative large language models (LLMs), which generate keywords or summaries to guide table selection. While effective, these models require substantial computational resources and can be costly to run at scale.
FERN (Feature-Engineered Ranking with No generative LLM) takes a different approach. Instead of generating new text or keywords, it leverages three “frozen” retrievers—pretrained models that do not update during training—to independently propose candidate tables. These candidates are then described using up to 120 handcrafted features, which capture various signals about the match between the question and each table. A LightGBM ranker, a type of gradient boosting machine learning model, is trained once on a set of questions to learn how to reorder these candidates effectively.
The term “frozen retriever” means the underlying retrieval models are fixed and not fine-tuned during this process, which simplifies training and reduces computational overhead. LightGBM helps by efficiently combining the multiple features to prioritize tables most likely relevant to the query. This two-step process—candidate generation followed by learned ranking—enables FERN to operate without the need for generating new text or keywords, setting it apart from systems like RASL that use Claude 3.5 models for keyword extraction.
FERN’s performance was evaluated across several benchmarks, including Spider, BIRD, BEAVER, and Fiben, which vary in database complexity and question style. On Spider, a widely used benchmark, FERN’s candidate pool already matches the recall (ability to include all relevant tables) of RASL when considering 15 tables. Its learned ranking further boosts the share of questions where all correct tables appear in the top 5 results from 83.9% to 95.6%, a substantial improvement. On BIRD, with additional evidence features, FERN’s recall is close to RASL’s, and non-retrieval features improve ranking accuracy notably. On BEAVER, despite no training on that dataset, FERN’s recall is competitive, though the completeness of its top results lags behind specialized retrievers. On Fiben, the method shows weaker performance, highlighting potential limits in generalization.
Importantly, FERN achieves these results with a median latency of just over 200 milliseconds per question on a high-end NVIDIA A100 GPU, without needing to call external generative AI services. This suggests it could offer a more efficient and cost-effective solution for real-world applications where quick and accurate table retrieval is needed.
Looking ahead, this research opens the door to building robust text-to-SQL systems that rely less on heavy generative models and more on smart combinations of retrieval and ranking techniques. Future work may explore improving generalization to diverse databases and questions, integrating additional features, or optimizing speed further. For organizations seeking practical, scalable ways to enable natural language querying of complex databases, approaches like FERN represent a promising direction.
Based on research published on arXiv by Arjun Rao, Nishant Gopinath.