Consistency LLM: converting LLMs to parallel decoders accelerates inference 3.5x

Nor is it how I work, I think that's normal enough. I do have an idea of what I'm going to say before I say it, I think that's closer to what they meant. I think and speak in increments of ideas, not words.

You might not have an internal monologue. A lot of us don't, and the ones that do are equally shocked every time they find out. For what it's worth, I'm in the same boat—can form sentences, but why would I? It'd slow me down.

People who don't have inner monologues tend to assume that all that stuff is some form of analogy or metaphor. It's not. It's entirely literal.

Are you sure. It might not be the whole sentence, but I would find it hard to believe that in practice the way you speak or write is like

hello <think> May <think> be <think> I'll <think> go <think> get <think> break <think> fast

In some conversations, maybe it's easier to form complete sentences. In some others, the best we can do is: have a rough draft about what to say in mind and then refine it word by word while speaking.

"Rem tene, verba sequentur" (you hold the matter, then words come) is largely "how it works".

You form logical ideas as you speak, as you speak your speech develops, so the translation is from ideas to sentences. It is not clear in which phase one would mentally form a complete sentence, nor why it should be relevant. You "see something [that makes sense]", then you describe it - iteratively.

Probably.

You are probably pretty far from the LLM extreme, though, of thinking one token at a time.

They probably do not mean people form entire sentences before expressing them, I am not aware of anybody doing that. I assume it refers to people first coming up with a global outline of what they want to say before they start speaking.

Hi we are CLLM authors and thanks for sharing your experience and insights! I can see this drawing skill refining process echoes with the training process in CLLM, the only thing is at this point stressor in CLLM training is not getting progressively demanding.

For example, while drawing, you can set very specific time limit on how long you are allowed to draw in each trial and make the time progressively shorter. In CLLM, maybe we can make this the learning process more and more difficult by mapping more and more distant states in Jacobi trajectory to its final state.

We are using the term "consistency" because we draw parallelism between consistency LLM and the consistency model in diffusion image generation where the training processes are analogous.

Systems generally become more efficient when under stress. They are also forced into local optima - everything has upsides and downsides.

I had an interesting experience in an Invertebrate Zoology lab class one summer.

We students were brought into a lab, given specimens to draw, and the only instructions we received were 'You have 30 minutes to draw this. Go.'

There was no "here's how to draw. here's what to do and not to do". It was just basically "We don't care about any insecurities you might have. We don't care if you think you can't draw. No excuses, just fucking draw it. Now."

Not only did we draw, but we (all of us) improved enormously over the course of the class as more animals were brought in and the exercise was repeated over and over and over again throughout the summer.

What it taught us is that everyone, and I mean everyone, can draw. Our collective attitude shifted from "don't know if this is even possible" to "of course we can do this. this is easy. routine. trivial."

Highly recommended approach.

It was the most freeing and amazing class I had in college.

The optimization does not affect the result of LLM, it's guaranteed to produce equivalent results as decoding directly. Let's not treat that LLM as some magic that resembles our mind, it's just another program that produces sentences that happens to make sense.

What is the geometric interpretation?

That assumption might be useful in this context, but I think it's pretty clearly not true. Ask anyone to tell you about a complex past event with a lot of parallel branches and you'll quickly see them add bits, pieces and tangents midsentence to cover the full range of events. I don't think I've seen the sentence granularity hypothesis in any serious scientific context before.

It's because an LLM is essentially a probability matrix. You type a prompt, then it calculates what's the probability of getting a next word and so on, eventually forming a sentence. The probability learned is based on the training data.

Because of the underlying probability model, it's not going to be 100% deterministic. Plus a model like ChatGPT purposefully have "temperature" parameter that will further add randomisation to the whole process.

My answer is based on this paper if you're interested to read more: The Matrix: A Bayesian learning model for LLMs, https://arxiv.org/abs/2402.03175

I want to ask, for example, how is it that an LLM when given the same prompt does not respond in the same deterministic way?

You can control that in most systems with an inference-set parameter called "temperature". But setting the temperature as low as possible tends to lead to very low-quality answers - the system can't crawl out of some local optimum and ends up repeating itself over and over. Such answers may be "deterministic" but they're also not good.

how is it that an LLM when given the same prompt does not respond in the same deterministic way?

In software (not in the model) here's literally a random number generator that picks from a weighted set of "next-token" choices that the model spits out. The selection process can have a series of knobs to manipulate the responses. If you want it to be deterministic (if you have direct access to the software) you can tell it to set "top-k = 1" or "temperature = 0.0" (depending on your software) and it will be deterministic.

Usually the default settings are not for determinism, because for whatever reason the quality of the results tends to not be that good when you go fully d.

For this particular question, ask chatgpt how temperature affects llm softmax sampling.

For other things, study using Karpathy's videos.

For that answer, you can refer to the 3blue1brown videos

The llm model outputs a vector of probabilities for tokens, and the llm user picks a token from the most likely list using a random number

There are better and worse algorithms. I'm not sure "there is no free lunch" always applies in a particularly meaningful way. Some things aren't on the pareto frontier.

Groq is serving an LLM from (100s of chips worth of) SRAM, so the effective bandwidth thus token generation speed is an order of magnitude higher than HBM. This would 3.5x their speed as well, it is orthogonal.

My understanding is that theirs is a pure hardware solution. The hardware is flexible enough to model any current NN architecture.

(Incidentally, there are black box optimization algorithms, so a system as good as grok at inference might be useful for training even if it can't support gradient descent)

According to someone I talked to at groq event I was invited to (I did not sign an nda), They are putting ~8 racks of hardware per llm. Of course coordinating those racks to have exact timings between them to pull tokens through is definitely "part of the hard part".

LLMs can easily produce data not in training dataset.

LLMs do not navigate stored data. An LLM is not a DB of the training data.

But indexing *is* training. It's just not using end-to-end gradient descent.

You might like, the Infinigram paper then. It was discussed recently:

https://news.ycombinator.com/item?id=40266791

Similar or greater inference wins are achieved with speculative decoding which is already widely used, so while this is really interesting (and was tried before with less success AFAIK), it's not yet clear how impactful it would be.

Thanks for interesting in our work! Yes we found training with consistency loss + AR loss on even a subset of a dataset results in significant speedup (0.01% pre-training cost). Training on more data permits even further speedup: the model is able to learn from more frequently-appearing collocations and phrases.

For more details, please check out our paper and you can also see speedup saturates as the size of training data grows.

Yes, seems like a huge important result for LLM performance.

I’m not aware of any other paper that has offered to increase inference LLM performance to this degree. Has there ever been one before?

At least while also:

- Maintaining output quality. The benchmarks used were somewhat narrow but so far so good.

- Improving not just query latency but also global throughput

- Not requiring more compute

- Having a relatively practical implementation and not adding big challenges and complexity

You could argue the insight is incremental, as it builds on what’s been done with parallel/jacobi decoding. Those previous results were necessary and important, but this may be the one that finally extracts real world value from the promise of parallel decoding.

Speculative decoding requires you to load the smaller model into memory and run inference on it.

`the previous tokens are absolutely needed before predicting the next token'

Maybe this is the key contribution of this paper: demonstrating that LLMs can predict the next n-tokens even if there are incorrect guesses in previous tokens through consistency training?

On the other hand, while mathematically it is true that p(x_t|x_1,...,x_t-1) depends on all x_1 to x_t-1, in practice, it is possible that predicting x_t only requires x_1 to x_t-2, and the attention to x_t-1 is minimal. Thus, predicting x_t with x_1 to x_t-2 and inaccurate x_t-1 is possible.

For all of these "how can we batch predicting the next n tokens?" the intuition is basically that it takes a buttload of math to predict some of the tokens, but that most tokens are actually easy to guess. For example, if I asked "What was that phone number from that 80's song?" as soon as a model generates 867- it shouldn't take that much math at all to finish predicting 5309.

"Try to fix all the words in a sentence at once. Keep iterating until you don't think it needs fixing."

The lab is tied to the vLLM project. I would say it might get picked up sooner by vLLM than other inference frameworks.

Ollama doesn't have their own inference engine, they just wrap llama.cpp. But yes, it will be awesome when it's more generally available.

Yes this is a great question! We are actively working on supporting other sampling strategies other than greedy sampling. In the context of CLLM training, instead of mapping to a static fixed point obtained from Jacobi decoding as the training ojbective, we term it dynamic fixed point. You can keep an eye on our github repo for new progress.

Agreed. It's straightforward to check that a token was the argmax, but it seems difficult to check that a token appeared with the probability you wanted it to. You could still do the fine-tuning step I guess, where you train the trajectories to approach n-token completions with the statistics you want, but I can't see how you can replace the "check for a fixed point" step. Maybe "check the result was above this fixed threshold for likelihood".

Yep. My roommate and I were talking about this a year ago. You can also do something similar for LLM steering.

I think any next token predictor will benefit. Iiuc mamba is a next token predictor.

I just skimmed the gradient article, but if their only change is swapping out the transformer block for the mamba block, I don't think it's already using this optimization

The only similarity between Medusa and CLLM is both train and adapt LLMs for fast inference. But they use completely different training technique, decoding technique and as you pointed out CLLMs don't need extra parameters or configuring attention mask for tree-based verification.