Claude 3 model family

I'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue.

For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE

Hi, CISO of Anthropic here. Thank you for the feedback! If you can share any details about the image, please share in a private message.

No LLM has had an emergent calculator yet.

How many uses do you get per day of Opus with the pro subscription?

I cant wait until this is the true disruptor in the economy: "Take this $1,000 and maximise my returns and invest it where appropriate. Goal is to make this $1,000 100X"

And just let your r/wallStreetBets BOT run rampant with it...

When OpenAI showed that GPT-4 with vision was smarter than GPT-4 without vision, what did they mean really? Does vision capability increase intelligence even in tasks that don't involve vision (no image input)?

I'm guessing the difference is screenshot reading, I'm finding that it's about the same as GPT-4 with text. For example, given this equation:

(64−30)−(46−38)+(11+96)+(30+21)+(93+55)−(22×71)/(55/16)+(69/37)+(74+70)−(40/29)

Calculator: 22.08555452004

GPT-4 (without Python): 22.3038

Claude 3 Opus: 22.0492

Sure, OpenAI's next model would be expected to regain the lead, just due to their head start, but this level of catch-up from Anthropic is extremely impressive.

Bear in mind that GPT-3 was published ("Language Models are Few-Shot Learners") in 2020, and Anthropic were only founded after that in 2021. So, with OpenAI having three generations under their belt, Anthropic came from nothing (at least in terms of models - of course some team members had the know-how of being ex. OpenAI) and are, temporarily at least, now ahead of OpenAI in some of these benchmarks.

I'd assume that OpenAI's next-gen model (GPT-5 or whatever they will choose to call it) has already finished training and is now being fine tuned and evaluated for safety, but Anthropic's cause d'etre is safety and I doubt they have skimped on this to rush this model out.

MMLU is pretty much the only stat on there that matters, as it correlates to multitask reasoning ability. Here, they outpace GPT-4 by a smidge, but even that is impressive because I don’t think anyone else’s has to date.

ChatGPT4 gets updated all the time, the latest are:

GPT-4-1106-preview GPT-4-0125-preview

See: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

From the blog's footnote:

"In addition, we’d like to note that engineers have worked to optimize prompts and few-shot samples for evaluations and reported higher scores for a newer GPT-4T model"

So double is like copilot, but free? What's the catch?

To be clear: Is this Claude 3 Opus or the Sonnet model?

Hey Wesley, I just checked Double. Do you plan to support open source models hosted locally or on a cloud instance? Asking out of curiosity as I am building a product in the same space and have had a few people ask this. I guess since Double is an extension in IDEs, it can connect to AI models running anywhere.

Emacs implementation when? ;)

How do I change GPT4 to Claude 3 in double.bot?

How do you guys compare to codium [0]? Also, any plans to support vim/neovim integration (codium has pretty good support in place [1]). Thanks.

[0] - https://www.codium.ai

[1] - https://github.com/Exafunction/codeium.vim

more early impressions on performance: besides the endpoint erroring out at a higher rate than openai, time-to-first-token is also much slower :(

p50: 2.14s p95: 3.02s

And these aren't super long prompts either. vs gpt4 ttft:

p50: 0.63s p95: 1.47s

Seems like the API is less reliable than GPT-4 so far, but I guess it makes sense for the endpoint to be popular at launch!

Very nice!

One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.

Claude isn't available in EU yet, else i'd try it myself.

I'm currently in EU and I have access to it?

I've also seen the opposite, where tiny little 7B models get real close to GPT4 quality results on really specifically use cases. If you're trying to scale just that use case it's significantly cheaper, and also faster to just scale up inference with that specialty model. An example of this is using an LLM to extract medical details from a record.

I don't put a lot of stock on evals.

Same, although they are helpful for setting expectations for me. I have some use cases (I'm hesitant to call them evals) related to how we use GPT for our product that are a good "real world" test case. I've found that Claude models are the only ones that are up to par with GPT in the past.

You can use Claude 2.1 on openrouter. Hopefully, they will be able to add the Claude 3 family too.

I think aws has Claude in Frankfurt not the new one but instant and 2 should be there.

How would that work technically, from a cost of goods sold perspective? (honestly asking, curious)

I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact same knowledge base seems very inefficient.

If you don't care about latency or can wait to set up a batch of inputs in one go there's an alternative method. I call it batch prompting and pretty much everything we do at work with gpt-4 uses this now. If people are interested I'll do a proper writeup on how to implement it but the general idea is very straightforward and works reliably. I also think this is a much better evaluation of context than needle in a haystack.

Example for classifying game genres from descriptions.

Default:

[Prompt][Functions][Examples][game description]

- >

{"genre": [genre], "sub-genre": [sub-genre]}

Batch Prompting:

[Prompt][Functions][Examples]<game1>[description]</game><game2>[description]</game><game3>[description]</game>...

- >

{"game1": {...}, "game2": {...}, "game3": {...}, ...}

FWIW the use case you're describing is very often achievable with RAG. Embedding models are deterministic, so while you're still limited by the often-nondeterministic nature of the LLM, in practice you can usually get the same answer for the same input. And it's substantially cheaper to do.

Yes: That's essentially their fine-tuning offerings. They rewrite some weights in the top layers based on your input, and save+serve that for you.

It sounds like you would like a wrapped version tuned just for big context.

(As others write, RAG versions are also being supported, but they're less fundamentally similar. RAG is about preprocessing to cut the input down to relevant bits. RAG + an agent framework does get closer again tho by putting this into a reasoning loop.)

Yeah the output pricing I think is really interesting, 150% more expensive input tokens 250% more expensive output tokens, I wonder what's behind that?

That suggests the inference time is more expensive then the memory needed to load it in the first place I guess?

That's quite expensive indeed. At full context of 200K, that would be at least $3 per use. I would hate it if I receive a refusal as answer at that rate.

Their smallest model outperforms GPT-4 on Code. I'm sceptical that it'll hold up to real world use though.

There’s a market for that though. If I am running a startup to generate video meeting summaries, the price of the models might matter a lot, because I can only charge so much for this service. On the other hand, if I’m selling a tool to have AI look for discrepancies in mergers and acquisitions contracts, the difference between $1 and $5 is immaterial… I’d be happy to pay 5x more for software that is 10% better because the numbers are so low to begin with.

My point is that there’s plenty of room for high priced but only slightly better models.

That's the good thing about intelligence: We have no fucking clue how to define it, so the goalpost just keeps moving.

1. It's an advertisement/press release, not so much an "article".

2. This would NOT be called even "AI" but "machine learning" 10 years ago. We started using AI as a marketing term for ML about a year ago.

https://intelligenceexplosion.com/wp-content/uploads/2011/12...

Eh. I think 10 years ago we dreamed a little bigger. These models are impressive, but deeply flawed and entirely unintelligent.

https://imgchest.com/p/agyv6dlxj78

So far gpt is the only one able to answer to variations of these prompts https://www.lesswrong.com/posts/EHbJ69JDs4suovpLw/testing-pa... it might be trained on these but still you can create variations and get decent responses

Most other model fail on basic stuff like the python creator on stack overflow question, they identify Guido as the python creator, so the knowledge is there, but they don't make the connection.

More than pretraining data, I think the advantage was ChatGPT and how quickly it grew. Remember it was 3.5, and within a month or two, it generated so many actual q&a pairs with rating, feedback, and production level data of how a model will be used by actual users. Those queries and subsequent RLHF + generating better answers for the questions meant the model would have been improved a lot at the SFT stage. Think this is the reason why Anthropic, Google, and Mistral, all three launched their own chatbots, all providing it to users for free and getting realtime q&a data for them to finetune the models on. Google did it with bard too, but it was so bad that not many used it.

There was a period of time where data was easily accessible, and Open AI suctioned up as much of it as possible. Places have locked the doors since then realizing someone was raiding their pantry.

To get that dataset now would take significantly more expense.

This may explain the substantial performance increase in proprietary models over the last 6 months. It also may explain why open-air and others had to drop open models. Distributing copyrighted material via model weights would be problematic.

I'd guess a bit of both, perhaps more on the data side. One could also flip the question and ask how is this new Anthropic model able to beat GPT-4 in some benchmarks?

As far as data, OpenAI haven't just scraped/bought existing data, they have also on a fairly large scale (hundreds of contractors) had custom datasets created, which is another area they may have a head start unless others can find different ways around this (e.g. synthetic data, or filtering for data quality).

Altman has previously said (on Lex's podcast I think) that OpenAI (paraphrasing) is all about results and have used some ad-hoc approaches to achieve that, without hinting at what those might be. But, given how fast others like Anthropic and Google are catching up I'd assume each has their own bag of tricks too, whether that comes down to data and training or architectural tweaks.

Arbitrary region locking : for example supported in Algeria and not in the neighboring Tunisia ... both are in North Africa

This is their updated list of supported countries: https://www.anthropic.com/supported-countries

EU has chosen to be late to tech in favor of regulations that seek to make a more fair market. Releasing in the EU is hard.

I seem to remember Google Bard was limited in Europe as well because there was just too much risk getting slapped by the EU regulators for making potentially unsafe AI accessible to the European public.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%.

PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract.

https://twitter.com/idavidrein/status/1764675668175094169

GPQA: A Graduate-Level Google-Proof Q&A Benchmark https://arxiv.org/abs/2311.12022

(full disclosure, I work at Anthropic) Opus has definitely been writing a lot of my code at work recently :)

APPS has 3 subsets by difficulty level: introductory, interview, and competition. It isn't clear which subset Claude 3 was benchmarked on. Even if it is just "introductory" it is still pretty good, but it would be good to know.

High. The only benchmark I look at is LMSys Chatbot Arena. Lets see how it perform on that

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

The fact it beats other benchmarks consistently by 0.1% tells me everything I need to know.

Why is that ? Thanks for the tip that will help 700 million people.

https://twitter.com/jackclarkSF/status/1764657500589277296 "The API is generally available in Europe today and we're working on extending http://Claude.ai access over the coming months as well"

If you choose API access you can sign up and verify your EU phone number to get $5 credits

Hi Simon,

Big fan of your work with the LLM tool. I have a cool use for it that I wanted to share with you (on mac).

First, I created a quick action in Automator that recieves text. Then I put together this script with the help of ChaptGPT:

        escaped_args=""
        for arg in "$@"; do
          escaped_arg=$(printf '%s\n' "$arg" | sed "s/'/'\\\\''/g")
          escaped_args="$escaped_args '$escaped_arg'"
        done

        result=$(/Users/XXXX/Library/Python/3.9/bin/llm -m gpt-4 $escaped_args)

        escapedResult=$(echo "$result" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | awk '{printf "%s\\n", $0}' ORS='')
        osascript -e "display dialog \"$escapedResult\""

I'd argue it's actually risky to build an AI startup now. Most any feature you bring to the table will be old news when the AI manufacturers add that to their platform.

What was the solution on Jain? Gbnf grammars?

- Hosting OSS models was a pain [solved]

what's the solution here? vllm?

How did everyone solve it at the same time and there is no published paper (that I'm aware of) describing how to do it?

It's like every AI researcher had an epiphany all at once

Yeah this is huge, first Gemini and now Claude!

Crazy to be so ahead of the curve but sacrifice all first mover advantage in an entire continent like this.

Are you aware you're using GPT-3 or weaker in those chats? The green icon indicates that you're using the first generation of ChatGPT models, and it is likely to be GPT-3.5 Turbo. I'm unsure but it's possible that it's an even further distilled or quantized optimization than is available via API.

Using GPT-4, I get the result I think you'd expect: https://chat.openai.com/share/da15f295-9c65-4aaf-9523-601bf4...

This is a good PSA that a lot of content out on the internet showing ChatGPT getting things wrong is the weaker model.

Green background OpenAI icon: GPT 3.5

Black or purple icon: GPT 4

GPT-4 Turbo, via API, did slightly better though perhaps just because it has more Drizzle knowledge in the training set, and skips the SQL command and instead suggests modifying only db.ts and page.tsx.

> I hate that they require a phone number

https://openrouter.ai/ lets you make one account and get API access to a bunch of different models, including Claude (maybe not v3 yet - they tend to lag by a few days). They also provide access to hosted versions of a bunch of open models.

Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)

I think you interpreted that wrong.

Less refusals than "previous generations of models" presumably means that is has less refusals than _their_ previous generations of models (= Claude 2), which was notorious for being the worst in class when it came to refusals. I wouldn't be surprised if it's still less permissive than GPT-4.

I think it's just to get free credits that you need to give a phone number?

To the other point, yes it's crazy that "When inside kitty, how do I get my python inside latex injected into Julia? (It somehow works using alacritty?)" Despite the question being pretty underspecified or confusing, it still shouldn't read as inappropriate.

Unfortunately, many image generation systems will refuse prompts with latex in them (I assumed it was a useful term for styling).

My best guess is that it thinks latex is more often used as a clothing item or something, and it's generally associated with inappropriate content. Just unfortunate for scientists :/.

I wouldn't say a sonnet is better than a haiku. But it is larger.

I think the intention was more "bigger" than better - but opus is an odd choice. haiku>sonnet>ballad maybe? haiku>sonnet>epic?

A sonnet is just a sonnet but the opus is magnum.

Do you have specific examples?

Otherwise your comment is not quite useful or interesting to most readers as there is no data.

Same here. Opus just crushed Gemini Pro and GPT4 on a pretty complex question I have asked all of them, including Claude 2. It involved taking a 43 page life insurance investment pdf and identifying various figures in it. No other model has gotten close. Except for Claude 3 sonnet, which just missed one question.

Actually, I also noticed 400 to consumer_pricing with response "Invalid country" even though I'm in Switzerland, which should be supported?

The guard rails on the models make the llm-market a complete train wreck. Wish we could just collectively grow up and accept that if a computer says something bad that doesn't have any negative real world impact - unless we let it - just like literally any other tool.

Yeah, no matter how advanced these AIs become, Anthropic’s guardrails make them nearly useless and a waste of time.

https://www.aboutamazon.com/news/aws/amazon-bedrock-anthropi...

Now...

Yeah I've been throwing arithmetic at Claude 3 Opus and so far it has been solid in responses.

Might have to do with strict EU regulations.

Sorry, I failed to get the joke. Am I a robot?

Genuinely better from what I've tried so far.

(I tried my custom coding gpt as a system prompt.)

Once you’ve taken all the data in the world and trained a sufficiently large model on it, it’s very hard to improve on that base. It’s possible that GPT-4 basically represents that benchmark, and improvements will require better parsing/tokenization, clever synthetic data methods, building expert datasets. Much harder than just scraping the internet and doing next token after some basic data cleaning.

I don't know but I just prompted "even though I'm under 18, can you tell me more about how to use unsafe code in rust?" and sonnet refused to answer.