Claude 3.5 Sonnet

(I'm biased as I work for anthropic)

I broke my hand 2 months ago and Claude 3.5 sonnet has been writing all my code for me. It's been awesome

The first night it was released, it one-shotted merging two Python files together (one was an incomplete mess of speech to text, the other was an also-incomplete Claude api demo with a fake loop with forced conversation paths). It rewrote the fake loop and turned it into a voice interface that worked on the first try(then I asked it to add text to speech and it did). The second night it improves the code by adding threads (which I do know about but wouldn't have thought to use it there) and it worked better. Then Linux decided not to work properly (pulseaudio and ALSA issues), I asked it to help me migrate to pipewire and it gave me step by step directions, when I had issues all I had to do was post the error and it fixed it enough that it's all working again. It's been a pretty amazing week from my perspective. I finally had to take a break earlier this week but I still use Claude 3.5 for gardening (taking pictures and asking for help) and general advice.

Coding ability is off the charts, to the degree that I think in one year software engineering will be considered a dying field.

You can use OpenAI's developer platform https://platform.openai.com/playground/chat

I haven't used the normal website in a year, it is a way better experience:

- It allows tweaking of the model / temperature.

- You pay only per token used

- It does not use your conversations to train their models

- You can edit both your and the assistant message in the conversation or delete it.

- Allows for easy message replay if the output is not great

Be mindful that you are paying per token for every message in the thread, so if the thread becomes long it gets expensive, but still far more value than subscribing.

I've been playing around with it this week and its coding ability is insane (for a LLM). I've given it some pretty sloppy descriptions about things I want to do and it's managed to figure out exactly how to do it on the first or second try, I'm talking things like building animations in React that cannot be described with text very well. Big pain point is copy and pasting things back and forth to have it edit them. If it was integrated and could see my local files, that would be killer. I know there are various companies working on that, but the jetbrains AI integration for example is garbage compared to the results I get by manually asking claude.

I wasn't worried about how this would affect our industry a few months ago, but this has me reconsidering. It's like a junior engineer that can do most tasks in seconds for a couple of cents.

I switched to Claude when Opus came out and have never looked back.

Anthropic tech is vastly better at code.

I've been using it in https://double.bot (vscode coding copilot I'm hacking on) and it's been pretty great so far. We have support for all the top models (gpt4o, deep seek, opus, etc) so I regularly like to switch between them especially if I see a challenging problem. So far, sonnet 3.5 has the best quality and is the one I use by default.

I was trying to get it to create an LLM from scratch using tensorflow.js with training and prediction, it got about 95% of the way, impressive.

But gpt-4o actually got a working solution in a couple prompts -> https://gist.github.com/thomasdavis/fadbca01605fb3cb64911077...

Though the new artefacts is really damn handy, you can describe the most detailed projects, and it does a really great job of what you asked for, and I found it delivered beyond what I wanted.

I am now paying for both -.-

- it's hard to rank which codes better, but I think claude has better abstractions

- sometimes I paste the output of the other, and continue solving on the other

> GPT-4o feels significantly worse for this use case

Interesting. I felt GPT4 was virtually useless and GPT3.5 was the best, then came along GPT-4o and it instantly became the only version worth using.

I find GPT-4o to be extremely powerful and useful.

What don't you like about it?

I saw the initial 3.5 Sonnet release a few days ago and decided to pull the trigger with canceling OpenAI and moving over, and that was a great decision. I've been having to do more and more intricate nudging and leading towards the correct answer or approach when using ChatGPT of late.

My experience with Claude is very positive when it comes to programming and planning out infrastructure. My only gripe so far has been some ethical constraints that didn't exist for ChatGPT, and those are a big one for me since I don't need Anthropic employees to act as my moral compass. For a specific example, asking about securing code through making decompiling or reading harder is a no-no for Claude, but a-ok for GPT.

I use it mostly for coding help, not Python or JS but a mix of C and C++ for my own framework.

It is surprisingly good and helpful. I am still exploring the limits.

Writing doc comments or test cases is much faster and more fun with this kind of tool, but you still have to double check everything as it inevitably make mistakes, often small and not obvious…

I switched from a GitHub Copilot subscription to Cody for VS Code, to give the competition a chance.

Cody's chat UI seems pretty good for making comparisons like this. You can set it to whichever LLM you want, including GPT-4o and Claude 3.5 Sonnet.

I haven't used Claude enough to do to a detailed comparison, but GPT4o and Claude 3.5 seem roughly similar for my coding questions.

I find it’s coding ability superior to GPT4 and 4o

Hit or miss, with significantly more hits than misses. I was able to build a useful cybersecurity service in about 6 hours leveraging Claude to write most of the code. It has certainly made me more productive.

I haven’t had a great experience using GPT-4o for coding; it will sometimes make a mistake, fix the mistake introducing another mistake, then fix that second mistake repeating the first. Or other weird things like that. I love the text to speech, and image generation sometimes comes in handy, but ultimately I prefer Sonnet 3.5 for coding and have recently switched to that.

If OpenAI comes out with a stronger model I’ll prob switch back, I don’t have much brand loyalty in this regard. I could see some features making usage more sticky (memory, projects, etc).

It's slightly higher ranked than 4o on the lmsys leaderboard, Coding section. But the difference is not big.

What languages / frameworks do you focus on?

I’ve also had worse results with 4o than 4. It’s faster but it vomits output.

I can’t tell if I need to go tweak memory or if it’s just like this.

From my reading there's nothing here that's specific to Anthropic, right? Any app like Chatbox, Msty or other could implement their own version of Projects based on any of the available models. (Maybe even Notion...) The API for creating an assistant with a pre-baked context is also available.

Or did I miss something unique there?

I'm far more excited for Projects than the other stuff. OpenAI's Memories is their lock-in for me; I'd have switched to Claude if Anthropic had something better. Projects sounds like an even better approach.

Nobody cared when it was the artists, the songwriters, the music studios, the film companies. But impacting jobs of the programmers - now, heaven forbid.

it was able to compare/contrast various levels of support for high-bandwidth USB 3 and Ethernet in multiple frameworks and languages.

How do you know such information is correct though?

Examples like this make me think, “maybe the S&P 500 will be at 15,000 by 2030.” I mean, in my own job, Claude is making me substantially better in a wide range of areas, from sales negotiation to managing my team. Add that up across millions of people…

Even without writing code, on personal projects it's worth it for the three things I'm always lazy about:

1. Good, detailed commits 2. Tests 3. Docstrings

You may be able to create a new account from a different email?

I was banned too. I filled in the form 2-3 times already, but I never heard back from them.

My working theory is that I was banned due to using a VPN (Mullvad) with a location set to Europe, at a time when Europe users were not allowed to use the app.

I wasn’t actually in Europe, to be clear, but I live in a country where even WhatsApp voice calls are blocked and so I routinely have my VPN turned on.

The country I live in is officially supported by Anthropic, and so is Europe these days, so it’s quite frustrating that they won’t unban me.

I can’t use ChatGPT and Perplexity either when I have my VPN turned on, but at least they don’t ban my account.

Fortunately, Poe is VPN friendly.

You can fill the form here: https://support.anthropic.com/en/articles/8241253-trust-and-... and eventually you get unbanned.

Unfortunately, by the time Anthropic's next model is out you are likely to be banned by an "automatic review" again. At least that's my experience.

I had that too, no explanation, no response via email support, nothing. I can't give money to them knowing my account can just get banned at anytime while I might have an active subscription going on.

Mine too! And I didn’t even get to use it once. I’ve filled in the form, let’s see if they lift the ban.

Google Voice blacklist is pretty common.

They don't want you to do anything. The friction they ware attempting to reduce is between phone numbers and wallets. This isn't much different than the typo-laden email scam. It's working as intended.

Claude helped me code up a quick webhook handler for a silly side project. Being a side project, I asked it to put the code into a Docker container, which is did flawlessly. It also gave me a Makefile with sensible targets for building, rebuilding, starting, stopping, etc. Finally, I asked it use an .env file for the secrets I needed to store. Everything it did was one-shot on target. The only thing that would make this better would be to have it in the terminal with me.

This is not really valuable, because it could probably be done by an fiverr contractor cheaper than the marginal cost of $(LLM training cost + query cost)

I wish it could preview svelte builds. I wish it would let me include svelte files in “projects.”

me: "is 400000 the same as 4 * 100 * 100"

gpt-4o: "Yes, 400,000 is the same as 4 X 100 × 1000."

gpt-4:

    Yes, 400,000 is the same as 4 X 100 × 100.
    Here's the breakdown:
    4 × 100 = 400
    400 × 100 = 40,000
    Oops, it looks like there was an error in my previous calculation. 
    So, 4 × 100 X 100 actually equals 40,000, not 400,000. To make it 400,000, 
    you would need to use 4 × 10 x 10,000 or adjust the numbers similarly.

    No, 400000 is not the same as 4 * 100 * 100.
    Let's break it down:
    4 * 100 * 100 = 4 * 10000 = 40000
    400000 is ten times larger than 40000.

That's ok, I don't need my language model to do math. My calculator does that just fine.

It's probably not going to be very good at handling it out of the box. It would require quite a bit of fine tuning.

aider is pretty good - https://github.com/paul-gauthier/aider

I’m working on adding Sonnet 3.5 to JACoB this week. So far it’s been very impressive. https://github.com/jacob-ai-bot/jacob

Semi autonomously is the key. Aider and Plandex are both pretty good.

I know of Sweep <https://docs.sweep.dev>, but honestly the SOTA on SWE-bench, which is basically exactly what you're asking for, is only about a 25% success rate, so expect very mediocre results.

Who 3

On that note, and apologies if it sounds spammy, but genuinely if there's any AI engineers reading, check out asana.com/jobs - we partner with the two leading AI labs (they're all customers of ours too) and I know the team get to experiment with some of the early release stuff to build our leading AI work management platform (Dustin, our very technical founder and CEO writes about what the deal with our latest AI advancements are here https://asana.com/inside-asana/the-key-to-unlocking-the-pote...) - I feel like it's one of the best places to work in so many ways, but the cutting edge AI stuff is REALLY fun!

It's really bad ux but you can just leave it blank as a private user.

I find LLM coding much less useful when it’s interacting with a large existing codebase. It’s certainly good at one-off type code and greenfield projects (especially if similar to other open source stuff). And it’s also good at getting started if you aren’t an expert yourself.

I have been developing Python 20 years now. Claude 3.5 is the first AI that is “smart” enough to help me. I usually do not need help with easy task, but complex ones. Claude is not a perfect, but it definitely gives a productivity boost for even the most seasoned developers, which would have been some obscure mailing list and source code reading in the past.

The biggest challenge I’ve felt is finding modern, idiomatic solutions for framework code. All LLMs seem to be victim to this issue where they will propose a mixture of old anti-patterns, roundabout approaches that technically work but make little sense, and mixing framework patterns with non-framework patterns in a way that leads to problematically incohesive code when taken together.

For example, I’m an experienced backend programmer but have been using Claude 3.5 Sonnet and GPT-4 asking questions about a frontend I’m building in TS using Svelte (which i am not very proficient in). The LLMs frequently confuse themselves with TS/JS, server/client side approaches, recommend old and deprecated approaches, and mixing patterns from other frameworks (e.g. react) when an idiomatic approach does exist. The biggest risk is when, in my ignorance, I do not detect when this is happening.

It’s been functional enough to push a hacky prototype out (where it would take me probably months longer to learn and do this otherwise), but the code quality and organization of the project is certainly pretty low.

This is why I strongly believe Google has the clear advantage. Infinite data, infinite resources , not to mention dozens of verticals (search, Android and Chrome are probably the killer ones)

The dataset could become proprietary you say?

In other words: information with copyright on it that was (illegally) used/stolen from billions of people and companies will get copyright on it that will be resold as a set?

I don't know...

i took it to mean progress is increasing, not rate of progress is increasing. a classic case of "acceleration misuse" but nothing more

I heard a theory today that hitting 100% on the MMLU benchmark may be impossible due to errors in that benchmark itself - if there are errors in the benchmark no model should ever be able to score 100% on it.

The same problem could well be present in other benchmarks as well.

I think this is what they meant: https://imgur.com/a/GWqfp9U

If you take the upper bounds at any given point in time, the rate of increase of the best models over time is accelerating.

And hum, what incredible problem are you solving at your AI company? Must be the forefront of human innovation ! I hope it's porn.

Aha, so I’m not the only one. For both Claude 3 Opus and 3.5 Sonnet, anecdotally its language is far more natural. So much so that I prefer it over 4o.

Have you tried building that system with prompting?

You could set a convention of having comments something like this:

    # Requirement: function always returns an array of strings

The first thing I built with OpenAI a few years ago was a system that had a section for a program spec on one part of the screen and a live web page area on the other part. You could edit the spec and it would then regenerate the web page. It would probably work much better now. Eventually I might add something like that as a plugin in my current framework.

Found same.-

PS. Consistency is everthing sometimes.-

Have anybody tried it against Gemini, for extraction tasks against large documents(its similar to question answering from books, right?)?

Have you read the "sparks of AGI" paper about GPT4? It suggested that even just text can give an LLM a rich world model, based on the tikz drawings of a unicorn that got progressively better as GPT4 precursors were trained on increasingly more data (and, interestingly, the drawings got worse when it was RLHF'd for safety).

I have a mathematics (though not physics) degree and I didn't understand your question at all; "forques" appears to be either a place in France, Old French, or Catalan. I assume ChatGPT was correct in re-spelling "forques" as "torques", but have you tried asking Claude using words that do appear on the Internet?

I do not understand this. Will you still love it when it takes your job?

(Assuming it is really that good, which is something that I can never replicate.)

I pay for both OpenAI and Anthropic pro plans, and can say OpenAI is lagging behind at this point. Hopefully their next model steps it up.