Contributors: Robert Corwin, CEO and Ahmed Ata, Data Scientist and Kahlil Wehmeyer, Senior Data Scientist
Intelligent Chatbots & LLM Orchestration Platforms
Austin Ai has been creating intelligent chatbots that understand, summarize, and talk about our clients' internal and external documents. LLM orchestration platforms “glue together” parts of the chat and LLM architecture and simulate things like conversation memory and reasoning.
Open Source Tools
Many of the tools we employ are open source, allowing us to deliver a customized application exceptionally quickly for a fixed price – with no recurring SaaS fees. Components include vector DBs, large language models (LLMs), and orchestration platforms, which “glue together” various parts of the chat and LLM architecture and simulate things like conversation memory and reasoning.
LangChain vs. Haystack
Our researchers have evaluated the LangChain and Haystack orchestration platforms, two of the most popular. We made six applications with each: a chatbot, a search app, a web scraper, an OCR app, and some simple NLP apps, in addition to production apps for our clients. In this article, we share our resultant impressions of those two frameworks.
In brief, both frameworks function well, and LangChain has more capabilities but is more complicated than Haystack. FWIW, we are making several enterprise-wide chat apps in LangChain, but Haystack remains an option for lighter-weight applications or quick proofs of concepts. The table below shows the details of our comparison across various aspects.
NB: Both frameworks, like the entire LLM space, are new and rapidly changing, and our comments are based on our own particular experience which has necessarily been short, but nonetheless oriented in the real business world. Other notable orchestration platforms include LlamaIndex and Griptape. We have briefly looked at these and the former appears to be more similar to LangChain, and the latter to Haystack. Another tool is Rasa, an open source conversational framework which we have experience with. We continue to experiment with all and will report any further conclusions.
Detailed Comparisons:
| Aspect |
LangChain |
Haystack |
| Website |
https://www.langchain.com |
https://haystack.deepset.ai |
| Cost and model |
Open source; business model appears to be value-added services on top of open source to large enterprises. |
Open source; presumably supports parent company deepset’s other products like deepset Cloud |
| Funding |
$10 MM |
$45.2 MM (deepset, parent company) |
| Out-of-the-box integrations and tools |
Many, e.g., AWS Lambda, APIFY, Huggingface, YouTube |
Fewer, but allows user to create custom tools / pipelines |
| Community support |
Very Good |
OK |
| Complication |
Rather complicated. Abstracts almost every concept into a class, e.g. even a simple Python f string is abstracted into a “Prompt Template”. Your opinion of this may depend on your affinity for very object-oriented code bases. |
Simpler to open, read, understand and use out of the box. No shortage of classes, but they seem slightly more intuitive in design and in how you are supposed to use them together. |
| Simplified workflow description |
Prompts and other Model I/O components feed into Chains. Agents may route the conversation to different Tools or ToolKits. |
Each Node performs a task; multiple Nodes make up a Pipeline. Agents may route the conversation to different Tools. |
| Data connectors and tools |
Robust set of tools like loaders, transformers, embedding models, vector DB interfaces, retrievers. |
Set of many tools like converters, classifiers, retrievers with seemingly slightly less bells and whistles. |
| Conversation "memory" retention |
Several different options including conversation history or summary, knowledge graphs, token length, different flushing options, storage in a DB and targeted retrieval |
Fewer options; handled less explicitly. Integrates with REDIS to store memory across conversations |
| Output parsers |
Is more flexible in structuring the model response; parser can be a List, Datetime, Enum, Pydantic, Auto-fixing, Retry or Structured Output Parser |
Limited parser options; parser can be either the default BaseOutputParser or AnswerParser which parses the model output to extract the answer into a proper Answer object using regex patterns |
| Debugging |
Proprietary debugging framework, LangSmith, currently in beta. Stack trace during normal IDE debugging is inflated by everything being a class. |
No special tools outside of normal IDE debugging |
| Other features |
Callbacks to hook into various stages of the application. Asynchronous support. Examples of autonomous agents. Moderation of response fallacies. |
OCR support. Pre-built REST API to interface with it. Examples of integration with Rasa. Annotation tool. |
| Other comments |
More tools, but some may crash; entire app may crash when query is meaningless. |
Can’t retrieve data after 2021. Seems to handle meaningless queries better. |
