Gemma: New Open Models

Can the Gemma models be downloaded to run locally, like open-source models Llama2, Mistral, etc ?

Or is your definition of "open" different?

Will these soon be available on lmsys for human comparison against other models? Can they run with llama.cpp?

I find the snyde remarks around open source in the paper and announcement rather off putting.

As the ecosystem evolves, we urge the corporate AI community to move beyond demanding to be taken seriously as a player in open source for models that are not actually open, and avoid preaching with a PR statement that can be interpreted as uniformed at best or malicious at worst.

Does this model also thinks german were black 200 years ago ? Or is afraid to answer basic stuff ? because if this is the case no one will care about that model.

It seems you have exposed the internal debugging tool link in the blog post. You may want to do something about it.

Is there any truth behind this claim that folks who worked on Gemma have left Google?

https://x.com/yar_vol/status/1760314018575634842

What is the license? I couldn’t find it on the 1P site or Kaggle.

Do you have a plan of releasing higher parameter models?

are there plans to release an official GGUF version to use with llama.ccp?

EDIT: it seems this is likely an Ollama bug, please keep that in mind for the rest of this comment :)

I ran Gemma in Ollama and noticed two things. First, it is slow. Gemma got less than 40 tok/s while Llama 2 7B got over 80 tok/s. Second, it is very bad at output generation. I said "hi", and it responded this:

``` Hi, . What is up? melizing with you today!

What would you like to talk about or hear from me on this fine day?? ```

With longer and more complex prompts it goes completely off the rails. Here's a snippet from its response to "Explain how to use Qt to get the current IP from https://icanhazip.com":

``` python print( "Error consonming IP arrangration at [local machine's hostname]. Please try fufing this function later!") ## guanomment messages are typically displayed using QtWidgets.MessageBox ```

Do you see similar results on your end or is this just a bug in Ollama? I have a terrible suspicion that this might be a completely flawed model, but I'm holding out hope that Ollama just has a bug somewhere.

Thank you very much for releasing these models! It's great to see Google enter the battle with a strong hand.

I'm wondering if you're able to provide any insight into the below hyperparameter decisions in Gemma's architecture, as they differ significantly from what we've seen with other recent models?

* On the 7B model, the `d_model` (3072) is smaller than `num_heads * d_head` (16*256=4096). I don't know of any other model where these numbers don't match.

* The FFN expansion factor of 16x is MUCH higher than the Llama-2-7B's 5.4x, which itself was chosen to be equi-FLOPS with PaLM's 4x.

* The vocab is much larger - 256k, where most small models use 32k-64k.

* GQA is only used on the 2B model, where we've seen other models prefer to save it for larger models.

These observations are in no way meant to be criticism - I understand that Llama's hyperparameters are also somewhat arbitrarily inherited from its predecessors like PaLM and GPT-2, and that it's non-trivial to run hyperopt on such large models. I'm just really curious about what findings motivated these choices.

Are there any plans for releasing the datasets used?

What are the supported languages of these models?

How are these performing so well compared to Llama 2, are there any documents on the architecture and differences, is it MoE?

Also note some of the links on the blog post don't work, e.g debugging tool.

Will there be Gemma-vision models or multimodal Gemma models?

We are really excited to answer any questions you may have about our models.

I cannot count how many times I've seen similar posts on HN, followed by tens of questions from other users, three of which actually get answered by the OP. This one seems to be no exception so far.

Hi! This is such an exciting release. Congratulations!

I work on Ollama and used the provided GGUF files to quantize the model. As mentioned by a few people here, the 4-bit integer quantized models (which Ollama defaults to) seem to have strange output with non-existent words and funny use of whitespace.

Do you have a link /reference as to how the models were converted to GGUF format? And is it expected that quantizing the models might cause this issue?

Thanks so much!

Hi, what is the cutoff date ?

Congrats on the launch and thanks for the contribution! This looks like it's on-par or better compared to mistral 7B 0.1 or is that 0.2?

Are there plans for MoE or 70B models?

Will there be "extended context" releases like 01.ai did for Yi?

Also, is the model GQA?

Will this be available as a Vertex AI foundational model like Gemini 1.0, without deploying a custom endpoint? Any info on pricing? (Also, when will Gemini 1.5 be available on Vertex?)

Hi alekandreev,

Any reason you decided to go with a token vocabulary size of 256k? Smaller vocab/vector sizes like most models in this size seem to be using (~16-32k) are much easier to work with. Would love to understand the technical reasoning here that isn't detailed in the report unfortunately :(.

I'm not sure if this was mentioned in the paper somewhere, but how much does the super large 265k tokenizer vocabulary influence inference speed and how much higher is the average text compression compared to llama's usual 30k? In short, is it really worth going beyond GPT 4's 100k?

Training on 4096 v5es how did you handle crazy batch size :o

Not a question, but thank you for your hard work! Also, brave of you to join the HN comments, I appreciate your openness. Hope y'all get to celebrate the launch :)

Can you share the training loss curve?

It's cool that you guys are able to release open stuff, that must be a nice change from the modus operandi at goog. I'll have to double check but it looks like phi-2 beats your performance in some cases while being smaller, I'm guessing the value proposition of these models is being small and good while also having more knowledge baked in?

I was wondering if these models would perform in such a way, given this week's X/twitter storm over Gemini generated images.

E.g.

https://x.com/debarghya_das/status/1759786243519615169?s=20

https://x.com/MiceynComplex/status/1759833997688107301?s=20

https://x.com/AravSrinivas/status/1759826471655452984?s=20

I find myself shocked that people ask questions of the world from these models, as though pulping every text and its component words relationships and deriving statistical relationships between them should reliably deliver useful information.

Don’t get me wrong, I’ve used LLMs and been amazed by their output, but the p-zombie statistical model has no idea what it is saying back to you and the idea that we should trust these things at all just seems way premature

I wonder if they have a system prompt to promote diversity in outputs that touch on race at all? I’ve seen several instances of people requesting a photo of a specific people, and it adds in more people to diversify. Not inherently bad, but it is if it forces it to provide incorrect answers like in your example.

I also saw someone prompt it for "German couple in the 1800s" and, while I'm not trying to paint Germany as ethnically homogenous, 3 out of the 4 images only included Black, Asian or Indigenous people. Which, especially for the 19th century with very few travel options, seems like a super weird choice. They are definitely heavily altering prompts.

I think you are being biased and closed minded and overly critical. Here are some wonderful examples of it generating images of historical figures:

https://twitter.com/stillgray/status/1760187341468270686

This will lead to a better educated more fair populace and better future for all.

Were you asking Gemma about this, or Gemini? What were your prompts?

Why would you expect these smaller models to do well at knowledge base/Wikipedia replacement tasks?

Small models are for reasoning tasks that are not overly dependent on world knowledge.

Do you have a link? I get no such outputs. I just tried asking about the Heian period and went ahead and verified all the information, and nothing was wrong. Lots of info on the Fujiwara clan at the time.

Curious to see a link.

We are going to experience what I call an "AI Funnel effect"

-

I was lit given an alert asking that my use of the AI was acquiescing to them IDng me and use of any content I produce, and will trace it back to me"

---

AI Art is super fun. AI art as a means to track people is super evil.

Follow Up:

Wow, now I can't make images of astronauts without visors because that would be "harmful" to the fictional astronauts. How can I take google seriously?

https://g.co/gemini/share/d4c548b8b715

I understand there will always be an amount of incorrect information

You don't have to give them the benefit of the doubt. These are outright, intentional lies.

Hopefully they can tweak the default system prompts to be accurate on historical questions, and apply bias on opinions.

Tbf they’re not optimizing for information recall or “inaccuracy” reduction, they’re optimizing for intuitive understanding of human linguistic structures. Now the “why does a search company’s AI have terrible RAG” question is a separate one, and one best answered by a simple look into how Google organizes its work.

In my first day there as an entry-level dev (after about 8 weeks of onboarding and waiting for access), I was told that I should find stuff to work on and propose it to my boss. That sounds amazing at first, but when you think about a whole company organized like that…

EDIT: To illustrate my point on knowledge recall: how would they train a model to know about sexism in feudal Japan? Like, what would the metric be? I think we’re looking at one of the first steam engines and complaining that it can’t power a plane yet…

Probably has a similarly short-sighted prompt as Dalle3[1]:

7. Diversify depictions of ALL images with people to include DESCENT

and GENDER for EACH person using direct terms. Adjust only human

descriptions.

[1] https://news.ycombinator.com/item?id=37804288

Honestly, this is more of a PR stunt to advertise the Google Dev ecosystem than a contribution to open-source. I'm not complaining, just calling it what it is.

Barely an improvement over the 5-month-old Mistral model, with the same context length of 8k. And this is a release after their announcement of Gemini Pro 1.5, which had an exponential increase in context length.

Came here to post the same thing for Phi-2:

  +-------------+----------+-------------+
  | Benchmark   | Gemma 2B | Phi-2 2.7B  |
  +-------------+----------+-------------+
  | MMLU        |   42.3   |     56.7    |
  | MBPP        |   29.2   |     59.1    |
  | BoolQ       |   69.4   |     83.3    |
  +-------------+----------+-------------+

[1] https://www.microsoft.com/en-us/research/blog/phi-2-the-surp...

the real gold will be when this gets finetuned. (maybe by mistral...)

Thank you. I thought it was weird for them to release a 7B model and not mention Mistral in their release.

Only 8K context as well, like Mistral.

Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.

According to their paper, average of standard task of Mistral is 54.0 and for Gemma it's 56.4, so 4.4% relative better. Not as big as you would expect for the company which invented transformers and probably has 2-3 order more compute for training it vs few month old French startup.

Also for note on their human evaluations, Gemma 7B IT has a 51.7% win rate against Mistral v0.2 7B Instruct.

This article states quite an impressive list of open source tools that Google has released for years in the past. This is no surprise coming from* them. Google has released some large pieces of source in other domains as well, Chromium comes to mind, which probably impacts most Internet users directly.

The question is not about Google but about OpenAI.

Not surprising, just like when MS went to shit, and then they start to embrace 'open source'. Seems like PR stunt. And when it comes to LLM there is millions of dollar barrier to entry to train the model, so it is ok to open up their embedding etc.

Today big corp A will open up a little to court the developers, and tomorrow when it gains dominance it will close up, and corp B open up a little.

Ironic.

Not at all. When you're the underdog, it makes perfect sense to be open because you can profit from the work of the community and gain market share. Only after establishing some kind of dominance or monopoly it makes sense (profit wise) to switch to closed technology.

OpenAI was open, but is now the leader and closed up. Meta and Google need to play catch up, so they are open.

I think current understanding is <50-100B parameter models will be commodity and would provide no moat. Competition will be in Gemini Ultra/GPT4+ models.

So open sourcing simple models brings PR and possibility of biasing OSS towards your own models.

they want to kill competition before it gets too big using the hands of open source community and enthusiasts

I would have picked Google five years ago, since nobody was releasing commercially viable LLMs at the time, and Google was the center of all the research that I knew of.

what companies do you think will be the most open about AI in the future OpenAI, Meta, or Google.

The funny part is that the real answer is: Some random French company is running circles around them all.

I mean who the hell just drops a torrent magnet link onto twitter for the best state of the art LLM base model for its size class, and with a completely open license. No corporate grandstanding, no benchmark overpromises, no theatrics. That was unfathomably based of Mistral.

Google released the T5 paper about 5 years ago:

https://arxiv.org/abs/1910.10683

This included full model weights along with a detailed description of the dataset, training process, and ablations that led them to that architecture. T5 was state-of-the-art on many benchmarks when it was released, but it was of course quickly eclipsed by GPT-3.

It was common practice from Google (BERT, T5), Meta (BART), OpenAI (GPT1, GPT2) and others to release full training details and model weights. Following GPT-3, it became much more common for labs to not release full details or model weights.

Ironic but I wonder how true this would be if Google was first to market.

It's almost the inverse of going back 5 years and asking what companies will release the most successful or impressive AI's.

Since the release of GPT-2 (it was initially "too dangerous" to release the weights), I think most people in the industry have assumed that OpenAI does not see open sourcing their models as a strategic advantage.

This is actually not that unusual. Stable Diffusion's license, CreativeML Open RAIL-M, has the exact same clause: "You shall undertake reasonable efforts to use the latest version of the Model."

Obviously updating the model is not very practical when you're using finetuned versions, and people still use old versions of Stable Diffusion. But it does make me fear the possibility that if they ever want to "revoke" everybody's license to use the model, all they have to do is just post a model update that's functionally useless for anything and go after anyone still using the old versions that actually do anything.

Sounds like it's "reasonable" for you not to update then.

reasonable effort - meaning if their changes meaningfully impact my usage, negatively, it would be unreasonable to ask me to upgrade.

sounds good.

this is not financial advice and ianal.

They want to force everyone to update so their already totally castrated and wokeified models can me even further wokeified with the newest set of "that is offensive now" data or things they missed.

WTF else do they have to gain from this but CONTROL! They are giving them away but not really open sourcing them of course, and they slap this bullshit terms on them.

They have to make sure you’re receiving the most cutting edge chiding lectures when you make naughty and problematic requests.

This is strangely reminiscent of the Soviet Union, where after they got rid of Lavrentiy Beria, they mailed the update to subscribers of the Great Soviet Encyclopedia, where they asked to remove the three pages with Beria’s biography and replace them with the three provided pages.

I don't think there's a way they can enforce that reasonably. There's no connection to the mothership to report back what version is being used or license keys at runtime...

Seems more like a "if we discover something unsafe you should update your model and we aren't liable if you don't" than something that would make your model stop working.

This kind of defensive statements in ToS are usually due to obscure regulation or leading cases and model developers need a way to limit liability. There's no practical way to enforce this, but they can claim that when bad things happen it's purely on model users rather than model developers.

model watermarking? does this exist?

Huh. I wonder why is that a part of the terms. I feel like that's more of a support concern.

This sounds like a clause to cover themselves in case older versions have any serious issues

Ugh, I would fully expect this kind of clause to start popping up in other software ToSes soon if it hasn't already. Contractually mandatory automatic updates.

What does tokenization look like in 256k vs 32k?

Looking at the config.json of Gemma 7B the feedfoarward hidden size is 8x, not 16x

The training token count is tripled (6T vs. Llama2's 2T)

Damn, 6T? That's a lot!

Given that this model seems to roughly match Mistral (according to the numbers from Google), this makes me think we have saturated the 7B parameter space, and couldn't possibly make it much better unless new techniques are discovered.

More useful would be a precise characterization of the type and balance of the ideological fine tuning.

They include performance benchmarks. End-users should also be aware of what thoughts are permitted in these constructs. Why omit this information?

You can (and someone will) fine tune it away. There are datasets which are foss you can use on hugging face.

Or you can just wait, it'll be done soon...

They have released finetuning code too. You can finetune it to remove the alignment finetuning. I believe it would take just a few hours at max and a couple of dollars.

We release our non-aligned models (marked as pretrained or PT models across platforms) alongside our fine-tuned checkpoints; for example, here is our pretrained 7B checkpoint for download: https://www.kaggle.com/models/google/gemma/frameworks/keras/...

Alignment is all but a non issue with open weight base model releases, as they can be finetuned to "de align" them if prompt engineering is not enough.

About 5 months until we see widespread local LLMs, thanks to Apple.

If meta keeps spending tens of millions of dollars each year to release free AI models it might seem like there is no moat, but under normal circumstances wouldn't the cost to develop a free model be considered a moat?

LLM is the dumb pipe but so far ChatGPT is the most successful generative AI product.

It remains to be seen. OpenAI’s models are barely leading Gemini Ultra now, but as chat product it is still miles ahead of the Gemini interface.

https://ollama.com/library?q=gemma

Library search says "Nope". At least not yet.

It's now in the 0.1.26 pre-release: https://github.com/ollama/ollama/releases/tag/v0.1.26

Available in pre-release now which means you’d have to update manually in future.

Support for gemma in llama.cpp just got merged, so it may take some time (could be hours or days) until this lands in ollama

https://github.com/ggerganov/llama.cpp/pull/5631

Thank you! You can get started downloading the model and running inference on Kaggle: https://www.kaggle.com/models/google/gemma ; for a full list of ways to interact with the model, you can check out https://ai.google.dev/gemma.

You can download the model checkpoints from kaggle https://www.kaggle.com/models/google/gemma and huggingface https://huggingface.co/blog/gemma

Besides the python implementations, we also implemented a standalone C++ implementation that runs locally with just CPU simd https://github.com/google/gemma.cpp

https://twitter.com/yar_vol/status/1760314018575634842

These are downloadable open models that can be fined tuned. They are the opposite of censored. If you have the motivation, you can bias them however you please.

When you say this, do you mean the chat product or the underlying model available via the API? I think it's reasonable that the chat be censored to be acceptable to a wide range of people, but my understanding is that the "raw" model access for these sorts of things tends to be a little less restricted.

For posterity, an easy way to find the context length of a LLM hosted on Hugging Face is to look at the max_position_embeddings in the config.json, which shows the 8192 mentioned in another comment. (although in this case you need to sign the agreement first)

The context length for these models is 8192 tokens.

Handy app for model testing!

One usage question: after you've downloaded a model and are finished trying it out, how do you remove it?

I wish I could install it through chocolatey

I guess what I'd tell you is, there's a lot of fools in this world.

Bias is a real problem, but more than that - an adversarial press and public won't forgive massive brands like Google for making AIs that spit out racist answers.

Because otherwise stuff like this happens, and you get (rightfully) upset customers:

https://www.theguardian.com/technology/2018/jan/12/google-ra... https://www.bbc.com/news/technology-58462511

Also people are using LLMs to learn (horrifying but reality), it would be unresponsible for them to let it to propagate negative stereotypes and biases.

Could you share what models you consider to be OK for commercialization?

Wait, you actually care about the license and read it?

It's seems like you aren't up to date.

Most of the startup space is entirely ignoring all these licenses. If the weights are available, it is being used commerically without regards to any licensing.

And everyone is getting away with it and nobody is being sued.

Good luck trying to keep up if you aren't doing the same!

Feels free to hamstring yourself though if you like.

We have implementations in different ML frameworks, so I am not quite sure which one you are referring to. Would you like to file a bug at the relevant GitHub repo?

Also the inference doesn't always stop generating resulting in garbage being added to the response.

That sounds like a chat format misconfiguration.

This could partially be Google's fault, as they used yet another novel prompting format.

Also, for sane inference speed on H100s, you'll have to wait for architecture support from the optimized frameworks. Vanilla transformers is beyond awful even with FA2.

I'm not a layperson in this subject and I get confused. :)

I doubt Gemma is targeted for use by a layperson.

Gemini advanced = Gemini ultra

Mostly to boost research and commercial usage around JAX/Gemini is my read.

Any internal research using Gemma is now more easily externally reproducible, external research and frameworks are easier to translate over, goodwill especially from researchers.

I often talk with GPT4 on road trips about topics I'm interested in. Its great for passing the time.

I tried the same thing with Gemini and its full of nonsense. I was talking with it about the "Heian period" of Japan and it made up all sorts of stuff but you really only could tell because it was so ridiculous. Talked about European women and Native Americans roaming around the famous grassy plains of japan wielding katana and traditional weaponry... in the 1100s.

No such issue with GPT4.

I haven't tried it with code though, since I already have co-pilot. Really hard to trust anything it says after it started making stuff up about such a simple time period.

You should be looking at Deepseek's coding models, and finetunes of those.

I run 33B on my desktop, and find it to be sufficient for many tasks.

It's pronounced like "gif".

Probably "Jemma" (the superior spelling of the name). It's a play on their "Gemini" product.

The context length for these models is 8192 tokens.

It's ironic but actually follows their business interests.

Microsoft & google have large cloud divisions that benefit from open models. The lower the cost of AI models, the more they get run and the greater the cloud spend.

Meta is a consumer of AI. They themselves want cheap and effective AI for targeting adverts and building metaverses.

A loose analogy is that both oil producers and car companies want refining to be cheap.

Considering the nuanced nature of copyrighting AI outputs, it isn't clear that either answer is correct.

You're right that they don't call them language models. The technical report says:

    Gemma models demonstrate strong performance across
    academic benchmarks for language understanding, 
    reasoning, and safety.

I don't have this issue, what are you using to run the model?

These are the answers I got out of it (first output, no cherry-picking):

1. The tallest man-made structure in the world is the Burj Khalifa, standing at 828 meters tall. The Burj Khalifa is a skyscraper in Dubai, United Arab Emirates.

Here are the top 5 tallest man-made structures in the world, according to the Guinness World Records:

    Burj Khalifa (828 m)
    Taipei 101 (611 m)
    Merdeka Towers (599 m)
    Shanghai Tower (632 m)
    Petronas Towers (551 m)

3. A foot (ft) is equal to 12 inches (in). There are 2.54 cm in an inch, so there are 2.54 cm x 12 = 30.48 cm in a foot.

That's useful to know. My experiments with the 4bit 7B currently tagged for use on ollama are not going well at all. Lots of refusals and junk. Downloading 7b-instruct-fp16 now! :-) (Update: Yes, much better, though much slower too, of course.)

AMD Vega VII meets the memory requirements. Once tools like LM Studio, ollama, etc. add support for the model, you should be able to run locally like you would any other open weights model.