Grok

We don't know since no one is releasing their data.

Calling these models open source is like calling a binary open source because you can download it.

Which in this day and age isn't far from where were at.

that's a subtle dig at the fact that they have all of Twitter as a training corpus to use, but we don't know how they weight tweets. which, we know they're not gonna be weighted evenly.

Or even how much it was trained on this dataset, the amount of FLOPs.

I've always thought of 4K as "4x FullHD". In that way it makes sense.

GitHub have a soft repository size limit of 5GB, documented here: https://docs.github.com/en/repositories/working-with-files/m...

Soft size limit means "If your repository excessively impacts our infrastructure, you might receive an email from GitHub Support asking you to take corrective action." - I know people who have received such emails.

Most model releases happen through Hugging Face which does not have such a size limit.

No git would be impossible. I’ve never seen a repo even a few GB in size, if you are uploading non code files you really should not be using git. Git is a version management software for code. I often see repos which images and even videos checked in, please don’t, there are so many far better and more performant solutions out there.

The other approach would be to use AWS S3 or other cloud providers which would cost them money every time someone downloads their code, which is not their prerogative to pay for when they are releasing something for free. Torrents seems like the only good solution, unless someone hosts this on the cloud for free for everyone.

Others have pointed out that GitHub doesn't allow that, but

Torrents can unfortunately die after a period of time if no one continues seeding it or if they don't use a permanent web based seeder, which doesn't appear to be the case.

So to can web links, especially when they are 300 GB and egressing out of AWS at $0.09/GB or worse (in non-US regions). Each full download would cost $27 at that rate. 10,000 downloads would cost $270,000.

Sure you could go for something with a better cost model like R2, but you can't beat using one or two unmetered connections on a VPN to constantly seed on Bittorrent, your pricing would be effectively free and reliability would be higher than if you just exposed a HTTP server on the Internet in such a way.

This is not some crappy DVD rip on The Pirate Bay. It will be seeded as long as its relevant.

Twitter/X has their own massive infrastructure and bandwidth to seed this indefinitely.

The great thing about torrents is that you (or anyone else who cares) can single-handedly solve the problem you're complaining about by seeding the torrent.

GitHub may choose to throttle downloads or remove the files simply because they're taking up too much bandwidth.

A torrent is less likely to go down in the short term.

In what way is the bittorrent protocol criminalized?

Twitter is valued at 40 billion

WAS vaulued at 44B.

Now?

Maybe 5 billion.

xAI is a separate entity, and not a X/Twitter subsidiary.

The data pipeline isn't included in this release, and we already know it is a pretty simple RAG pipeline using qdrant, https://twitter.com/qdrant_engine/status/1721097971830260030.

Nothing about using data in "real time" predicates that the model parameters need to be this large, and is likely quite inefficient for their "non-woke" instructional use-case.

Isn't that... the same thing as search?

Why is that relevant to the size?

Post search on X is done as it is with any other data from any other source, you use RAG and function calling to insert the context.

< 7B open source models can function call very well. In fact, Nous Hermes 2 Pro (7B) is benchmarking better at that then GPT-3.5.

Not related to the size, if I'm not mistaken.

Disagree. It took more than 12 months from the release of GPT-4 to someone else producing a model of equivalent quality, and that definitely wasn't due to a shortage of investment from the competition.

There's a huge amount of depth in training a really good LLM. Not helped by the fact that iteration is incredibly expensive - it might take several months (and millions of dollars) before you can tell if your new model is working well or if there was some mistake in the pipeline that lead to a poor quality result.

Almost all of the world-class LLMs outside of OpenAI/DeepMind have been trained by people who previously worked at those organizations - giving them invaluable experience such that they could avoid the most expensive mistakes while training their new models.

i like some of claudes answers better, but it doesnt seem to be a better coder imo

Has it, though? LMSys Arena Leaderboard (blind ranking by humans) [0] positions Opus just below GPT-4 with a negligible ELO gap.

[0] https://chat.lmsys.org/

I don't know if Claude is "smarter" in any significant way. But its harder working. I can ask it for some code, and I never get a placeholder. It dutifully gives me the code I need.

citation needed (other than 'vibes')

The more disregard a company has for intellectual property rights, the more data they can use.

Google had far more to lose from a "copyright? lol" approach than OpenAI did.

Edited, I did mean the Grok in the article not the inference chip.

Indeed, Groq is a company building inference accelerators. Grok is completely unaffiliated.

It is known

I use these models regularly, and Claude is dumb as a rock compared to GPT-4.

A big catch here is that you can't slap an open source license on a bunch of copyrighted training data, and to date no-one has created a truly convincing LLM exclusively trained on public domain data. It might happen soon though - there are some convincing effort in progress.

Come on, that's not reasonable to expect from a company, or useful for indie hackers. Having weights that can be used however you like is enough for most people, even large companies.

Agreed. It's ridiculous people have to resort to saying their question dumb to avoid being attacked by toxic commenters.

If you release that instead of the binary weights you can be both more open and less useful for users. Fun

https://youtu.be/WyTzRnGSlcI?t=88

I consider the weights a binary program and the source code is the training data. The training algorithm is the compiler.

I agree this isn't standard terminology, but it makes the most sense to me in terms of power dynamics and information flow.

We know from interpretability research that the weights do algorithms eg sin approximation etc. So they feel like binary programs to me.

If you can't rebuild it, then how can you be considered to have the "source code" ?

The training data isn't a dataset used at runtime - it's basically the source code to the weights.

Not sure it really matters here though (who has the GPUs and desire to retrain Grok?), but just as a matter of definition "open weights" fits better than "open source".

I agree, medical history is probably not the sexiest reason to have AI ask questions. I think there are many more reasons; I think the Turing Test is the best metric to evaluate AIs, and current models come nowhere close. When people first meet they ask questions about their background. It would be nice if a model replicated that

?? gpt4 does this for me regularly

Not sure what the snark here is for: It would be trivial to produce a dataset where the model asked you questions then fine-tune on that.

People already do it with chain-of-thought and you could get away with a few dozen examples if you wanted to try this.

I'd bet a synthetic data set could do the job effectively.

Try Pi by inflection. It asks a lot of questions.

What's the difference?

Oh no, I only want _pure_ intentions for anything I use. Which is why I reject all for profit medicine.

It doesn't matter why he did it. What matters is that he did it.

This makes no sense to me for two reasons:

- He pointed out that his understanding was that it would be open source in some way

- The name OpenAI implies an open source endeavor. I dont know many things named Open that are in fact close sourced.

Engineers who joined Twitter pre-Musk days who live and work in the US on an H1-B visa can't just quit.

You can criticize Elon Musk without criticizing people who would have their lives upended if they quit or were fired.

"Good" cars is a real stretch.

Tesla is likely making good cars because the CEO is 'problematic'

The phrase merely means dont just overlook something because someone else who did not even labour over the end result.

I'm a little out of touch, is there a way to see how Grok measures up to other models?

One of the interesting things when weights are open sourced is the community can often improve the results. See all the bugs fixed in Gemma for an example.

You're right, this model is going to be too big for most people to play around with. But to answer your question I have a 128GB of RAM in my M3 MacBook Pro, so I can use most of that for GPU inferencing. But still, this model is going to need to be heavily quantized for me to be able to use it. (fwiw, I probably wont try this one)

In the next week or two I expect we'll see a GGUF version of the weights (might need to wait for a patch to llama.cpp first), and someone will release super small quantizations of it. I suspect my computer might be able to run a 3 bit quant, but it might need to go down to 2 bits to have any kind of reasonable context length. But with quants that small I'd expect the model's performance to degrade well below that of Mixtral, so it probably isn't really even worth using. But we'll see; quantization is weird, some models perform better than others when quantized.

It's easier to get a trademark on an altered word than a plain dictionary word. Just acquiring the easier one to acquire doesn't mean you now have rights over the harder one to acquire, though eventually after enough market recognition you might be given some control over other people using the common one. I wouldn't think groq is there yet.

Generally agree, though I would say "knowing a subject at a much deeper level than any LLM is capable of", as AI more broadly also includes specialist models that are wildly super-human in narrow domains like chess and Go.

Six is plural.

He's started a company in the UK: https://suite.endole.co.uk/insight/company/15361921-thebloke...

Interestingly registered just around the corner from where one of my relatives used to live.

Was any .gguf file hosted on HuggingFace found to be crafted in a way to exploit this?

what exactly are you implying here?

GPT-4 Turbo: 31.0

Claude 3 Opus: 27.3

Mistral Large: 17.7

Mistral Medium: 15.3

Gemini Pro 1.0: 14.2

Qwen 1.5 72B Chat: 10.7

Claude 3 Sonnet: 7.6

GPT-3.5 Turbo: 4.2

Mixtral 8x7B Instruct: 4.2

Llama 2 70B Chat: 3.5

Nous Hermes 2 Yi 34B: 1.5

The interesting part is the large improvement from medium to large models. Existing over-optimized benchmarks don't show this.

- Max is 100. 267 puzzles, 3 prompts for each, uppercase and lowercase

- Partial credit is given if the puzzle is not fully solved

- There is only one attempt allowed per puzzle, 0-shot.

- Humans get 4 attempts and a hint when they are one step away from solving a group

I hoped to get the results of Gemini Advanced, Gemini Pro 1.5, and Grok and do a few-shot version before posting it on GitHub.

I'd expect more configuration issues in getting it to run on them than from a tested llama.cpp version, since this doesn't seem like a polished release. But maybe.

That’s actually more valuable. Twitters data of small format text is awful for training. Best to just exclude it.

There are hundreds of millions of people on Twitter, and a few of them are very smart. I don’t see how that helps here though.

I don’t see difference here.

Userbase and their social networks and interactions is the data.

They don’t have much value from advertising point of view anymore.

What do you mean? The entire model and architecture and executables are fully open source.

The training methods are nothing secret, right? The architecture is well known.

Expecting the entire training dataset to be fully open is delusional.

One very compelling argument against the idea that current gen LLMs have personal beliefs etc is that they don't have a feedback loop, so they don't really "see" themselves in the way that we can inspect our own thoughts and actions and the consequences of such.

Interesting. That point didn't come across in your original comment. I recommend you state it next time at the end. Often times stuff that seems obvious to us / yourself / people who know about something -- can go unstated in stuff you say that otherwise references specific points at hand -- and omits these general, but enlightening/useful perspectives/priors, which it would be good to share.

This is not only for you specifically just a general reminder for all of us including me.

https://pjmedia.com/paula-bolyard/2018/10/24/tech-community-...

Yes. So say the same thing about them Open source has a definition and abusing that hurts all of us except the billionaires.

I've been in the open source community for about 25 years so I doubt it.

For what it's worth I would say a model should be fully reproducible to be open source, but that's not a decided consensus -- and AI models are sufficiently different than the source code / binary code distinction as to invoke discussion around defining it.

Sounds like they did what they said they would.