Mistral: Our first AI endpoints are available in early access

Does the 8x7B model really perform at a GPT-3.5 level? That means we might see GPT-3.5 models running locally on our phones in a few years.

FWIW - I need to remeasure but - IIRC my system with a 4090 only uses ~500w (maybe up to 600w) during inference of LLMs, the LLMs have a lot harder time saturating the compute compared to stable diffusion I'm assuming because of the VRAM speed (and this is all on-card, nothing swapping from system memory). The 4090 itself only really used 300~400w most of the time because of this.

If you consider 600w for the entire system, that's only 6kWh/1M token, for me 6kWh @0.2USD/kWh is 1.2USD/1M tokens.

And that's without the power efficiency improvements that an H100 has over the 4090. So I think 2$/1M should be achievable once you combine the efficiencies of H100s+batching, etc. Since LLM's generally dwarf the network delay anyway, you could host in places like washington for dirt cheap prices (their residential prices are almost half of what I used for calculations)

Depends how and where you source your energy. If you invest in your own solar panels and batteries, all that energy is essentially fixed price (cost of the infrastructure) amortized over the lifetime of the setup (1-2 decades or so). Maybe you have some variable pricing on top for grid connectivity and use the grid as a fallback. But there's also the notion of selling excess energy back to the grid that offsets that.

So, 10kwh could be a lot less than what you cite. That's also how grid operators make money. They generate cheaply and sell with a nice margin. Prices are determined by the most expensive energy sources on the grid in some markets (coal, nuclear, etc.). So, that pricing doesn't reflect actual cost for renewables, which is typically a lot lower than that. Anyone consuming large amounts of energy will be looking to cut their cost. For data centers that typically means investing in energy generation, storage, and efficient hardware and cooling.

The 4090 is considerably more power-hungry compared to e.g. an A100, however.

Is $0.2-0.4/kWh a good estimate for price paid in a data center? That’s pretty expensive for energy, and I think vPPA prices at big data centers are much lower (I think 0.1 is a decent upper bound in the US, though I could see EU being more expensive by 2x).

Well the 4090 is certainly less efficient on this. They are using H100's or better no doubt. If they optimize for TPUs, it'll be even better.

I get 40 tok/sec on my M3 Max on various 34B models, I gather a desktop 4090 would be at least 80?

Batching changes that equation a fair bit. Also these cards will not consume full power since llm are mostly limited by memory bandwidth and the processing part will get some idle time.

Mistral-small explicitly has inference costs of a 12.9b, but more than that, it's probably ran with batch size of 32 or higher. They'll worry more about offsetting training costs than about this.

Here's how it works in reality:

https://docs.mystic.ai/docs/mistral-ai-7b-vllm-fast-inferenc...

I pay for hosted databases all the time. It’s more convenient. But those same databases are popular because they are open source.

I also know that because it’s open source, if I ever have a need to, I can host it on my own servers. Currently I don’t have that need, but it’s nice to know that it’s in the cards.

You may be fine with shipping your data to OpenAI or Mistral, but worry about what happens if they change terms or if their future models change in a way that causes problems for you, or if they go bankrupt. In any of those cases, knowing you can take the model and run it yourself (or hire someone else to run it for you) mitigates risk. Whether those risks matter enough will of course differ wildly.

The big advantage of a hosted open model is insurance against model changes.

If you carefully craft and evaluate your more complex prompts against a closed model... and then that model is retired, you need to redo that process.

A lot of people were burned when OpenAI withdrew Codex, for example. I think that was a poor decision by OpenAI as it illustrated this exact risk.

If the hosted model you are using is open, you have options for continuing to use it should the host decide to stop offering it.

Same reason why you would use GPT-4. Plenty of people pay for that, some pay really good money.

If I'm happy with my infrastructure being built on top of the potential energy of a loadbearing rugpull, I'd probably stick with OpenAI in the average use case.

It's safe to assume they are confident it's better than 3.5. But people can be confident and wrong.

What's your evidence for that claim?

All they have to do to beat Anthropic's Claude is to skip having a permanent waitlist and let the credit cards get charged.

No. Mistral uses sentencepiece and the GPT use tiktoken.

The only vendor lock-in to GPT3.5 is the absence (perceived or real) of competitors at the same quality and availability.

It does look like Google will be the Yahoo of 2030

Perhaps even sooner, given the pace tech is evolving nowadays comparing to when Google took over Yahoo's place.

Google is conflicted about AI. Having AI do searches for you and summerize the results is both way more expensive than normal searches and means way fewer ads are shown.

Google can't do anything about it either - consumers will prefer the chat approach and will go where it is offered. Google has to offer it, or people will go to ChatGpt and Bing, but at the same time they lose a lot of money each time people opt for the chat summary.

It is way expensive for Bing too, but they don't care because their market is growing and since the only thing people used to search Bing for was how to download Chrome and how to change the default browser, they aren't losing a lot this way.

Google in general looks like they're lost their touch and are kinda on a downward spiral. Trying to squeeze out as much as they can from Youtube is I think a major sign of that, usually attempts like that start to pop up in burnt-out companies that are heading to their end.

As for AI they have a lot of world-class talent but they somehow cannot transform that into a coherent product even though they did amazing stuff on the research side. It has been brought up many times on HN on how Google has a product-design problem and it seems to be true.

Google was once a small company beating some big players with a fraction of their resources.

They already did, weights on Twitter.

Or it could help them be more globally competitive because they’ll have built their product to accommodate the EU rules and will be ready if/when other countries introduce something similar.

Maybe in 1970 someone was arguing that Volvo is handicapped because Swedish safety regulations have become so stringent that their cars can’t compete on price elsewhere. Turns out it was exactly Volvo’s strength to be ahead of the curve on regulation.

Have to see the final text, of course, but open source models apparently got some breaks in the final agreement last Friday.

The EU makes in it’s proposed law a lot of exceptions for open models. It might be a big selling point for mistral (or Facebook) but maybe tricky to deal with for closed models like OpenAI.

In some ways I see this as a sign that LLMs possibly aren’t that hard to iterate on now that the cat is out of the bag. A small team can push the state of the art in just 6 months? It feels like this is more a circumstance of the state of LLMs presently and less a statement on Mistral. I wouldn’t it be surprised if we see 10+ other new $1B+ companies spring up because of how much room for improvement there is in the space.

Yes, I'm aware there's been a ton of recent chatter. Most of these are dupes though so it's really just Mixtral, their funding, and now their commercial offering.

My argument is opening up the commercial offering is perhaps the biggest story yet simply because of all excitement this company has been able to generate.

So 14400 H100 for GPT-4, but that's just a fraction of the new system that Azure is building for OpenAI.

FWIW, I most enjoyed the 29TB machine demo at the end.

I found this discussion on the local llama subreddit that digs a little bit more into what effects the sliding window might have, in case you or anyone else reading this comment thread finds it interesting: https://old.reddit.com/r/LocalLLaMA/comments/17k2mwq/i_dont_....

It refers to the original Mistral 7B though not the new Mixtral fwiw

Real world testing and experience. If you only need the llm to retain the "gist" of the input tokens in order to return a "related" answer, the sliding window design is fine. But if you need actual technical analysis or tasks that involve verbatim referencing, quoting, recomposing, etc based off parts of the input documents, it doesn't work.

I tried using it for "business document" use cases but have ran into this with code as well; the latter might be a better explanation given where we're having this discussion. If you only need the llm to retain the general shape of your inputs so it can reuse them to influence the output, the sliding context is fine. But if you need it to actually reuse code verbatim from the input that you fed it (or to remember the api calls and their surrounding context verbatim to recall from a sample of just one that this api must be called before that api, when the prompt includes instructions to that effect) the "decomposition" of the input tokens with the sliding model is insufficient and the llm completely fails at the assigned task.

The sliding window does not lose the data but it does "decompose" it so that it can't be recalled verbatim. For analyzing code (feed it n classes and ask it to create a class using all of them to accomplish a task) that isn't good enough. It's also not good enough for some of the "corporate business" use cases we tried putting Mistral and other sliding window models to use on, where you need it to re-use verbatim or reference specific portions of one of n documents fed into it as input tokens.

Again, sufficient training can overcome these limitations. But that's only for cases where the corpus of input documents is static or at least contains significant reuse.

By the same submitter, minutes apart and both on the homepage.

They are two different news about one thing.

Then they'd be differing opinions?

Tiny and small, those two have been released, yes.

Yeah. This naming is interesting and speaks of lofty goals indeed. :)

hell yea!