RAG Large Data Pipeline with 1000 Models

Show HN: RAG Large Data Pipeline with 1000 Models

RAG Large Data Pipeline with 1000 Models is active in the show hn launch space.

Profile

Hi, I'm a PhD in data + LLMs. I'm building an LLM chatbot for dbt.The challenge isn't LLMs but dbt pipelines, which are too large (e.g., >1K models) to fit in the context window.Traditional vector RAG only works well for texts but poorly for SQLs.To solve this, we built a novel RAG using lineage.I tested it on dbt projects with 1000+ models, and it works very well.Some cool use cases the such chatbot does well: - Models discovery (I have a high-level question, which tables to use?)- Safe model edits (I want to edit the models, what downstream models are affected?)- Model debugging (This column looks wrong, how is it computed upstream?)- New pipelines prototyping (I want to add a new metric,

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

It is recorded at the Show HN launch stage.

Website: cocoon-rag.github.io/page/pipeline