We no longer use LangChain for building our AI agents

Chat GPT is just GPT version 3.5. OpenAI released many other versions of GPT before that. In fact, Open AI became really popular around the time of the GPT 2 which was a fairly good chat model.

Also, the Transformer architecture was not created by OpenAI so LLMs were a thing way before OpenAI existed :)

Chat models were not invented with ChatGPT. Conversational search and AI was a well-established field of study well before ChatGPT. It is remarkable how many people unfamiliar with the field think ChatGPT was the first chat model. It may be the first widely-popular chat model but it certainly isn’t the first

I am not sure what you mean by "turn these one-shot APIs into Markov chains." To me, langchain was mostly marketed as a framework that makes RAG easy by providing integration with all kinds of data sources(vector db, pdf, sql db, web search, etc). Also older models(including initial chatgpt) had limited context lengths. Langchain helped you to manage the conversation memory by splitting it up and storing the pieces in a vector db. Another thing langchain did was implementing the react framework(which you can implement with a few lines of code) to help you answer multi hop problems.

Chat models worked great for everything, including what we used instruct & completion models for

In 2022, I built and used a bot using the older completion model. After GPT3.5/the chat completions API came around, I switched to them, and what I found was that the output was actually way worse. It started producing all those robotic "As an AI language model, I cannot..." and "It's important to note that..." all the time. The older completion models didn't have such.

We use instruct models extensively as we find smaller models fine tuned to our prompts perform better when general chat models that are much larger. This lets us run inference that can be 1000x cheaper than 3.5, meaning both money saving and much better latencies.

I feel like most of this complaint is about OOP, not java.

Well. I'm working on a product that relies on both AI assistants in the user-facing parts, as well as LLM inference in the data processing pipeline. If we let our LLM guy run free, he would create an inscrutable tangled mess of Python code, notebooks, Celery tasks, and expensive VMs in the cloud.

I know Pythonista's regard themselves more as artists than engineers, but the rest of us needs reliable and deterministically running applications with observability, authorization, and accessible documentation. I don't want to drop into a notebook to understand what the current throughput is, I don't want to deploy huge pickle and CSV files alongside my source to do something interesting.

LangChain might not be the answer, but having no standard tools at all isn't either.

Id just like to point out the source of the Gorilla Banana problem is Joe Armstrong. He really had an amazing way of explain complex problems in a simple way.

https://www.johndcook.com/blog/2011/07/19/you-wanted-banana/

I’ll use this to explain why typescript is bad

Langchain was my first real contact with Python development, and it felt worse than Enterprise Java. I didn't know that OOP is so prominent in Python libraries, it looks like many devs are just copying the mistakes from Enterprise Java/.NET projects.

I feel this too, I think it's because Java is an artifact of layers of innovation that have accumulated over time, which weren't present at its inception. Langchain is similar, but has been developing even more rapidly than Java did.

I still find LC really useful if you stick to the core abstractions. That tends to minimize the dependency issues.

It's funny because I was using Langchain recently and found the most confusing part to be the inheritance model and what type was meant to fill which function in the chain. Using Java would make it impossible to mistype an object even while coding. I constantly wonder why the hell the industry decided Python was suitable for this kind of work.

Oh my! I've been looking for this comment Will be using it in the future to explain my feelings about Java and Python

Holy moly this was _exactly_ my impression. It seems to really be proliferating and it drives me nuts. It makes it almost impossible to useful things, which never used to be a problem with Python - even in the case of complex projects.

Figuring out how to customize something in a project like LangChain is positively Byzantine.

The point is that you don’t need a framework for that; the APIs are already similar enough that it should be obvious how to abstract over them using whatever approach is natural in your programming language of choice.

LiteLLM.

https://www.litellm.ai/

Openrouter maybe?

Using Llama Index for this via the `llama_index.core.base.llms.base.BaseLLM` interface. Using config files to describe the args to different models makes swapping models literally as easy as:

  chat_model:
    cls: llama_index.llms.openai.OpenAI
    kwargs:
      model: gpt-4

  chat_model:
    cls: llama_index.llms.gemini.Gemini
    kwargs:
      model_name: models/gemini-pro

The strategy design pattern would be suitable for this.

Vercel AI SDK[1] shines in this aspect in JS ecosystem.

They have the concept of providers [2] and switching between them is easy as changing parameters of a function[3]

[1]:https://sdk.vercel.ai/docs/introduction

[2]: https://sdk.vercel.ai/docs/foundations/providers-and-models

[3]: https://sdk.vercel.ai/docs/ai-sdk-core/overview#ai-sdk-core

Use a consistent argument structure and make a simple class or function for each provider that translates that to the specific API calls. They are very similar APIs. Maybe select the function call based on the model name.

Use openrouter. One OpenAI like api but lots of models

I appreciate that you're taking feedback seriously, and it sounds like you're making some good changes.

But frankly, all my goodwill was burnt up in the days I spent trying to make LangChain work, and the number of posts I've seen like this one make it clear I'm not the only one. The changes you've made might be awesome, but it also means NEW abstractions to learn, and "fool me once..." comes to mind.

But if you're sure it's in a much better place now, then for marketing purposes you might be better off relaunching as LangChain2, intentionally distancing the project from earlier versions.

Thanks Harrison. LangGraph (eg graph theory + Networkx) is the correct implementation of multi-agent frameworks, though it is looking further into, and anticipating a future, then where most GPT/agent deployments are at.

And while structured output and tool calling are good, from client feedback, I'm seeing more of a need for different types of composable agents other then the default ReAct, which has distinct limitations and performs poorly in many scenarios. Reflection/Reflextion are really good, REWOO or Plan/Execute as well.

Different agents for different situations...

[deleted]

using LangGraph for a month, every single "graph" was the same single solution. The idea is cool, but it isn't solving the right problem.... (and the problem statement shouldn't be generating buzz on twitter. sorry to be harsh).

You could borrow some ideas from DSPy (which borrows from pytorch) their Module: def forward: and chain LM objects this way. LangGraph sounds cool, but is a very fancy and limited version of basic conditional statements like switch/if, already built into languages.

Could someone point me towards a good resource for learning how to build a RAG app without llangchain or llamaindex? It's hard to find good information.

I had a similar experience when LangChain first came out. I spent a good amount of time trying to use it - including making some contributions to add functionality I needed - but ultimately dropped it. It made my head hurt.

Most LLM applications require nothing more than string handling, API calls, loops, and maybe a vector DB if you're doing RAG. You don't need several layers of abstraction and a bucketload of dependencies to manage basic string interpolation, HTTP requests, and for/while loops, especially in Python.

On the prompting side of things, aside from some basic tricks that are trivial to implement (CoT, in-context learning, whatever) prompting is very case-by-case and iterative, and being effective at it primarily relies on understanding how these models work, not cargo-culting the same prompts everyone else is using. LLM applications are not conceptually difficult applications to implement, but they are finicky and tough to corral, and something like LangChain only gets in the way IMO.

I wish I was this pragmatic as an intern.

Wise perspective from an intern. The type of pragmatism we love.

you are heading the right direction. It's amazing to see seasoned engineers go through the mental gymnastic of justifying installing all those dependencies and arguing about vector db choices when the data fit in ram and the swiss knife is right there: np.array

Any tutorials you follow?

impressive to decide against something as shiny as langchain as intern

I admire what the Langchain team has been building toward even if people don’t agree with some of their design choices.

The OpenAI api and others are quite raw, and it’s hard as a developer to resist building abstractions on top of it.

Some people are comparing libraries like Langchain to ORMs in this conversation, but I think maybe the better comparison would be web frameworks. Like, yeah the web/HTML/JSON are “just text” too, but you probably don’t want to reinvent a bunch of string and header parsing libraries every time you spin up a new project.

Coming from the JS ecosystem, I imagine a lot of people would like a lighter weight library like Express that handles the boring parts but doesn’t get in the way.

Way to follow your instinct.

I ran into similar limitations for relatively simple tasks. For example I wanted access to the token usage metadata in the response. This seems like such an obvious use case. This wasn’t possible at the time, or it wasn’t well documented anyway.

Groupthink is really common among programmers, especially when they have no idea what they are talking about. It shows you don't need a lot of experience to see the emperor has no clothes, but you do need to pay attention.

It was the first pass at solving the common problems when building with LLMs. People jumped on it because it was trendy and popular.

But it quickly became obvious that LangChain would be better named LangSpaghetti.

That’s nothing against the authors. What are the chances the first attempt at solving a problem is successful? They should be commended for shipping quickly and raising money on top of it to keep iterating.

The mistake of LangChain is that they doubled down on the bad abstraction. They should have been iterating by exploring different approaches to solving the problem, not by adding even more complexity to their first attempt.

Ah, the halcyon days of March 2023, we were a while loop away from AGI. I remember there was something that was It for like a month, to the point that whoever built the framework was treating a cocktail napkin on which they scribbled, whatever, "act, evaluate, decide next action, repeat", as if it was a historical talisman. And I wasn't sure! Maybe it was!

Every time I approached LangChain, contrary to the attitude of my colleagues, I could never figure out what the point of it was other than to fetishize certain design patterns. Interacting with an LLM in a useful way requires literally none of what LangChain has to offer, yet for a time it was on its way to being the de facto way to do anything with LLMs. It reminds me a lot of the false promise of ORMs, which is that if you trust the patterns then you can swap out the underlying engine and everything will still just work, and is more or less a fantasy.

Langchain seems to have been made just for the tutorial business on Udemy and Youtube.

never understood the "chain" in langchain.

I think it was great at first when llms were new and prompting required more strategy. Now the amount of of abstractions/ bloat they have for essentially string wrappers makes no sense

This seems to be a universal sentiment.. we took a short look at langchain and determined it was doing really trivial string manipulation/string templating stuff but inside really rigid and unnecessary abstractions. It was all stuff that could be implemented by any competent programmer in hours in any language without all the crap, so that's what we did. shrug

GraphQL actually holds value in my view as it gives custom SQL-like functionality instead of basic JSON APIs. With it, you can do fewer calls and retrieve only the attributes you need. Granted, if SQL were directly an API, then GraphQL wouldn't hold too much value.

Langchain has no such benefit.

Evaluating technology based on its "cool kid usage" and a vague sense of complexity is likely not the best strategy. Perhaps instead you could ask "what problems does this solve/create?"

I don't know a thing about LangChain so this is a real digression, but I often wonder if people who are critiquing GraphQL do so from the position of only having written GraphQL resolvers by hand.

If so, it would make sense. Because that's not a whole lot of fun. But a GraphQL server-side that is based around the GraphQL Schema Language is another matter entirely.

I've written several applications that started out as proofs of concept and have evolved into production platforms based on this pairing:

https://lighthouse-php.com https://lighthouse-php-auth.com

It is staggeringly productive, replaces lots of code generation in model queries and authentication, interacts pretty cleanly with ORM objects, and because it's part of the Laravel request cycle is still amenable to various techniques to e.g. whitelist, rate-limit or complexity-limit queries on production machines.

I have written resolvers (for non-database types) and I don't personally use the automatic mutations; it's better to write those by hand (and no different, really, to writing a POST handler).

The rest is an enormous amount of code-not-written, described in a set of files that look much like documentation and can be commented as such.

One might well not want to use it on heavily-used sites, but for intranet-type knowledgebase/admin interfaces that are an evolving proposition, it's super-valuable, particularly paired with something like Nuxt. Also pretty useful for wiring up federated websites, and it presents an extremely rapid way to develop an interface that can be used for pretty arbitrary static content generation.

GraphQL is very powerful when combined with Relay. It’s useless extra bloat if you just use it like REST.

The difference between the two technologies is that LangChain was developed and funded before anyone know what to do with LLMs and GraphQL was internal tooling using to solve a real problem at Meta.

In a lot of ways, LangChain is a poor abstraction because the layer it’s abstracting was (and still is) in it’s infancy.

What's the difference? I thought "agents" was just a fancier word for sequential prompts.

What does semantic kernel do for you? It isn't immediately obvious from the Readme.

Don’t waste your time, it’s been around since GPT3, and had no results so far. Also notice how no frontier lab is working on it.

In practice this means function calling - the LLM chooses the function to call (and its parameters). Usually in a loop with a 'finish' function that returns the control to the outside code.

You can do that without function calling - as did the original ReAct paper - but then you have to write your own grammar for the communication with the LLM, a parser for it, and also you need to teach the LLM to use that grammar. This is very time consuming.

Also, how much success people have or had with automating the E2E tests for their various apps by stringing such agents themselves together?

There’s a few startups in the space doing this like QA Tech in Stockholm, and others even in YC (but I forgot the name). I’m skeptical of how successful they’ll be, not just from complex test cases but things like data management and mistakingly affecting other tests. Interesting to follow just in case though, E2E is a pain!

I want to learn about agents too!

Fundamentally, "Agent" refers to anything that operates in an "observe-act" loop. So in the context of LLMs, an agent sees an observation (like the code base and test output) and produces an action (like a patch), and repeats.

For most of what people are doing with AI you don't need Python because you don't need the ML ecosystem. You're either going to be talking to some provider's API (in which case there are wrappers aplenty and even if there weren't their APIs are simple and trivial to wrap yourself) or you're going to self-host a model somewhere, in which case you can use something like ollama to give yourself an easy API to code against.

All of the logic of stringing prompts and outputs together can easily happen in basically any programming language with maybe a tiny bespoke framework customized to your needs.

Calling these things "AI agents" makes them sound both cooler and more complicated than they actually are or need to be. It's all just taking the output from one black box and sticking it into the input of another, the same kind of work frontline programmers have been doing for decades.

prompt engineering requires the ability to see what is happening at various steps and langchain makes that harder if not impossible.

honestly I don't need that much abstraction.

Disclamer: I work for Octomind.

I think the reading is more "It's hard to find a good abstraction in a field that has not settled yet on what a good abstraction is. In that case, you might want to avoid frameworks as things shift around too much."

Very much depends on the framework. I'm currently building a GitHub App with the Probot framework, which mostly just handles authentication boilerplate and some testing niceties, then just gives you an authenticated GitHub API client (no facade/abstraction).

Then of course there's the many web application frameworks, because nobody in their right mind would want to implement http request parsing themselves (outside of academic exercises).

In fact, I would argue that most popular frameworks exist precisely because it's often more time efficient to forget about underlying details. All computer software is built on abstraction. The key is picking the right level of abstraction for your use case.

It often took quite a long time for those historic frameworks to get the abstraction right. Survivorship bias sees us forget all the failed attempts.

I'm unconvinced there is no room for a framework here because LLMs are somehow special. LangChain just missed the mark. Unsurprisingly so, it being an early attempt, not to mention predating general availability of the LLM chatbots that have come to define the landscape.

I've only ever played around with it and not built out an app like you have, but in my experience the second you want to go off script from what the tutorials suggest, it becomes an impossible nightmare of reading source code trying to get a basic thing to work. LangChain is _the_ definition of death by abstraction.

LangChain got its start before LLMs had robust conversational abilities and before the LLM providers had developer decent native APIs (heck, there was basically only OpenAI at that time). It was a bit DOA as a result. Even by last spring, I felt more comfortable just working with the OpenAI API than trying to learn LangChain’s particular way of doing things.

Kudos to the LangChain folks for building what they built. They deserve some recognition for that. But, yes, I don’t think it’s been particularly helpful for quite some time.

Which alternatives have you been introduced to?

    > the "bad" abstractions make things abstract away tasks that you really need insight into.

Yeah, I was kind of surprised. The premise of the article started as "LangChain abstractions are off" and then the complaint was about... just a very simple pipeline?

I honestly don't care about the syntax (as long as it's sane enough), and `|` operator overloading isn't the worst one. Manually having to define a parser object gives off some enterprise Java vibes, and I get the httplib vs requests comparison - but it's not the end of the world. If anything, the example from the article left me wondering "why do they say it's worse, when at this level of abstraction it really looks better unless we don't ever need to customize the pipeline at all?" And they never gave any real example (about spawning those agents or something) that actually shows where the abstractions are making things hard or obscure.

Honestly, on the first reading, the article [wrongly] gave me an impression of saying "we don't use LangChain anymore because it lacks good opinionated defaults", which is surely wrong - it would be a very odd take, given the initial premise of using it production for a long while.

(I haven't used LangChain or any LLMs in production, just toyed around a little bit. I can absolutely agree with the article that if all you care about is one single backend, then all those abstractions are not likely to be a good idea.)

I can already use multiple backends by writing different code. The value-add langchain would need to prove is whether i can get better results using their abstractions compared to me doing it manually. Every time I’ve looked at how langchain’s prompts are constructed, they went wayyy against LLM vendor guidance so I have doubts.

Also the downside of not being able to easily tweak prompts based on experiments (crucial!)

And not to mention the library doesn’t actually live up to this use case, and you immediately (IME) run into “you actually can’t use a _Chain with provider _ if you want to use their _ API”, so I ultimately did have to care about whats supposed to be abstracted over

How does langgraph stack up against the alternatives?

LangGraph is the primary reason I use LangChain - being able to express my flow as a state machine has been a boon to both the design of my platform as well as my own productivity.

As the article points out, the difference between frameworks for building a website vs building an LLM agent is that we have decades more industrial experience behind our website-building opinions. I’ve used heavyweight frameworks before, and would understand your defense in the context of eg complaints about Spring Boot—but Langchain isn’t Spring; it really does kinda suck, for reasons that go beyond the inherent trade offs of using any framework.

I think that yes, there is a better way. You have a function that calls the API, then take the output and call another function that calls the API, inserting the first output into the second one's prompt using an f-string or whatever. You can have a helper function that has defaults for model params or something.

You don't need an abstraction at all really. Inserting the previous output into the new prompt is one line of code, and calling the API is another line of code.

If you really feel like you need to abstract that then you can make an additional helper function. But often you want to do different things at each stage so that doesn't really help.

Looks like they've parlayed it into some kind of business https://www.langchain.com/

A heuristic that I use when judging code quality is a search for "datas" or "metadatas".

I found it weird as well to see that. I didn't know LangChain overrode Python syntax.

But, if you're familiar with Linux/Unix, this should be familiar. You are piping the output of one function as the input of another function.

From memory, I recall Vulture might do something like that!

We've had success with non-developers using some of the visual tools built on top of LangChain to build exploratory models in order to prove a concept. LangChain does seem well suited to providing the "backend" for that type of visual node-based modelling.

Of course, once the model is proven it is handed off to developers to build something more production-worthy.

It makes it simple (and uniform) to switch providers.

Have you considered LiteLLM?