Gemini Flash

With 1M tokens you can dump 2000 pages of documents into the context windows before starting a chat.

Gemini's strength isn't in being able to answer logic puzzles, it's strength is in its context length. Studying for an exam? Just put the entire textbook in the chat. Need to use a dead language for an old test system with no information on the internet? Drop the 1300 page reference manual in and ask away.

I tried to use the 1M tokens with Gemini a couple of months ago. It either crashed or responded ___very__ slowly and then crashed.

I tried a half dozen times and gave up, I hope this one is faster and more stable.

I think that's a bit like asking why would someone need a 1gb Gmail when 50mb yahoo account is clearly enough.

It means you can dump context without thinking about it twice and without needing to hack some solutions to deal with context overflow etc.

And given that most use cases most likely deal with text and not multimodal the advantage seems pretty clear imo.

There's no way it's Llama 3 70b quality.

I've been trying to work Gemini 1.5 Pro into our workstream for all kinds of stuff and it is so bad. Unbelievable amount of hallucinations, especially when you introduce video or audio.

I'm not sure I can think of a single use case where a high hallucination tiny multimodal model is practical in most businesses. Without reliability it's just a toy.

GPT-3.5 has 0.5s average first-token latency and Claude3 Haiku 0.4s.

1m is great for multimodal agentic workflows where you need to keep track of history

I guess it depends on what you want to do.

E.g. I want to send an entire code base in a context. It might not fit into 128k.

Filtering down is a complex task by itself. It's much easier to call a single API.

Regarding quality of responses, I've seen both disappointing and brilliant responses from Gemini. Do maybe worth trying. But it will probably take several iterations until it can be relied upon.

With the release of GPT-4o I'm not clear on why an organization not bound to GCP would pick Gemini.

Price for anything, particularly multimodal tasks that with OpenAI GPT-4o is the cheapest model, that doesn't need GPT-4 quality. GPT-3.5-Turbo — which itself is 1/10 the cost of GPT-4o, is $0.5/1M tokens on input, $1.50/1M on output, with a 16K context window. Gemini 1.5 Flash, for prompts up to 128K, is $0.35/1M tokens on input, and $0.53/1M tokens on output.

For tasks that require multimodality but not GPT-4 smarts (which I think includes a lot of document-processing tasks, for which GPT-4 with Vision and now GPT-4 are magical but pricy), Gemini Flash looks like close to a 95% price cut.

Price.

Limitations of single point in vector space of what dimension?

I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make the embedding dimension.

My point is just that there’s no limitation in principle, it’s just a matter of how they design it and resource constraints.

1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means.

Multimodality in a model That's between 4-7% the cost per token of OpenAI’s cheapest multimodal model is an important feature when you are talking about production use and not just economically unsustainable demos.

My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.

We are dealing with multi-headed attention, therefore we have multiple points per token. You can always increase the number of heads or the size of the key vector.

My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.

Is it possible to explain what this means in a way that somebody only roughly familiar with vectors and vector databases? Or recommend an article or further reading on the topic?

Yeah it's not very good in practice, you can get a halfway decent demo out of it ("look I gave it 6.5 harry potters and it made an SVG map connecting characters with annotations!!"...some of the characters...spare annotations...cost $20). Just good enough to fool you a couple times when you try to make it work 10 times.

Price competition isn't limited to commodities.

Google figured it can't beat OpenAI technically, but they sure know they can beat them financially and infrastructurally.

But the race to the bottom has an opposition right?

So people expect to see a return of investment which will create the bottom of pricing (at least as soon as the old money ran out)

I'm also curious if AI is a good example because ai will become fundamental. This means if you don't invest you might be gone therefore it's more like a fee in case the investment would not pan out.

Google is building on top of and integrated with their cloud offerings. Having first party solutions like this gives big cloud customers an easy way to integrate. For Google it’s just another tool in the chest that gets sold to these big enterprises. Many go all in on all the same cloud products. Also the models are only the building blocks. Other cloud products at Google will be built with this and sold as a service

Not so sure about Open AI though…

Is this race to the bottom or just Googles new TPUs being extremly efficient?

You're saying the quiet part out loud here.

On a character level this should be trivial.

Characters are not the semantic components of words—these are syllables. Generally speaking, anyway. I've got to imagine this approach would yield higher quality results than the roman alphabet. I'm curious if this could be tested by just looking at how LLMs handle English vs Chinese.

The attention mechanism is vastly more efficient to train when it can attend to larger, more meaningful tokens. For inference servers, a significant amount of memory goes into the KV cache, and as you note, to build up the embedding through attention would then require correlating far more tokens, each of which is "less meaningful".

I think we may get to this point eventually, in the limit we will want multimodal LLMs that understand images and sounds down to the pixel and frequency, and it seems like for text, too, we will eventually want that as well.

In AI music generation we have much better results with large vocabulary sizes of 10^6 order, my uneducated guess is that's because transformers are not universal pattern recognizers, they can catch patterns on a certain granularity level only.

I would say 2 big problems are:

1. latency, which would get worse if you have to sequentially generate more output

2. These models very roughly turn tokens -> "average meaning" on the embedding layer, followed by attention layers that combine the meanings, and feed forward layers that match the current meaning combination to some kind of learned archetype/prototype almost. When you move from word parts to characters all of that becomes more confusing (what's the average meaning of a?) and so I don't think there are good enough techniques to learn character-based models yet

Does a monkey know that it is a monkey?

The models didn't exist when their training data was collected.

But... that's not really an excuse any more. Model vendors should understand now that the most natural thing in the world is for people to ask models directly about their own abilities and architecture.

I think models should have a final layer of fine-tuning or even system prompting to help them answer these kinds of questions in a useful way.

Ask a human how many neurons they have. Hell, over history humans haven't even consistently understood that the brain is where cognition happens.

Gemini Ultra is 1.0 with 8k window. This is 1.5 with 1m window. Your feeling is based on incorrect assumption.

Anyone who uses both products regularly will tell you that Gemini Advanced is far behind GPT-4 and Claude 3 Opus.

Pretending that they have a model internally that's on par but they're not releasing it is a very "my girlfriend goes to another school" move and makes no sense if they're a business that's actually trying to compete.

OpenAI desperately needs a marketing consult.

"GPT4o"? Seriously?

Even "GPT4 Omni" is easier in conversation, and that's what the "o" stands for!

They severely underestimate the number of casual users they have.

"ChatGPT" is like "Google". "Gemini" is never replacing that.

Last I checked you could disable the safety triggers as an API user with gemini (which doesn't alleviate your obligation to follow the TOS as to the uses of the model).

So many high-performing, yet poorly-named OpenAI models in that list.

Interestingly, until your comment I hadn't made any connection with old Flash, even though I spent hundreds of hours making Flash games.

This suggests names don't stick around for long and can be re-used. Perhaps Google could bring back "Buzz" and "Wave" since enough time has passed!

FAIR really swung for the fences with Llama3. It's a very impressive model, but the 8K context size is quite limiting for most use-cases.

Apple made a deal with OpenAI for GPT4o, the stakes are indeed high, can't be caught with pants down. iPhone needs to remain the premium brand.

Presumably we are defining smartness as doing more with less, so this indicates they have something going on in the latent space which will scale.

How does it compare to others products in your use case?