Launch HN: Retell AI (YC W24) – Conversational Speech API for Your LLM

[stub for offtopicness]

Wow this is incredible. I've worked a bit in the conversational LLM space and one of the hardest problems we were struggling with was human interruption handling. From the demo it seems like you guys have it down. Can't wait to see where this goes :) BTW I don't think the demo works on mobile, tried it on safari on IOS and got no response.

I would feel deceived if I were a customer of any company or office that uses this. If I take the trouble to call by phone, it's because I want to speak with a person. If I wanted to talk to a machine, I would send an email, talk to a chatbot, or even try to communicate with the company through social media. Calling by phone implies that I am investing time and effort, and I expect the same from the other side.

This is incredible and terrifying at the same time. Does it support long context? As in, can I voice chat with an instance of an agent, and then later in a different chat refer to items discussed in the previous chat? Can I also type / text with the agent and have it recall items from a previous session?

Is there different language support too?

What's the difference between your product and Gridspace's? I get the sense that your offering is more developer-focused, but I'm curious if there are any technical differences.

With respect to Alignment, it should be a fundamental requirement that an speech AI is ___REQUIRED___ to honestly inform a Human if its speaking to an AI.

Can you please ensure, going forward, you have the universal "truth" as it were, to have your system always identify if its AI when "prompted" (irrespective of what app/dev has built - your API should ensure that if "safeword" is used it shall reveal its AI)

--

"trust me, if you ask them if they are a cop, they legally have to tell you they are a cop" (court rules its legal for cops to lie to you) etc....

(it should be like those tones on a hold-call, to remind you that youre still on hold... but instead its a constant reminder that this bitch is AI) -- there should be some Root-level escape word to require any use of this tool to contact a Human. That word used to be "operator" MANY times, but still...

Maybe if a conversation with an elderly Human goes on with too many "huh? I cant hear you" or "i dont understand, can you repeat that" questions, your AI knows its talking to a non-tech Human, and it should re-MIND the Human that youre just an AI. (meaning no sympathy, emotion, it will not stop until you are dead) etc...

Guardrails, motherfucker, Do you speak it!"

I was skeptical but the demo is incredible (https://beta.retellai.com/home-agent)

The demo is nice but it makes me wonder: why would a company have a fully automated voice line rather than a booking interface? As a customer I'm never happy to call a company to make a reservation. I'd be extra annoyed if an AI picked up and I had to go through the motions of a conversation instead of doing two clicks in a Web UI.

Can you buy/remove phone numbers programmatically? I can't find anything in the docs.

Until this demo the most impressive conversational experiences I've seen were Pi and Livekit's Kitt demo (https://livekit.io/kitt). I do not think kitt was quite as fast in response time (as retell) but incredibly impressive for being fully opensource and open to any choice of api's (imagine kitt with groq api + deepgram's aura for super low latency).

Retell focusing on all of the other weird/unpredictable aspects of human conversation sounds super interesting and the demo's incredible.

Things are moving so fast, wow.

Can you share what you're doing for TTS? Is it a proprietary fully pretrained in-house model, a fine tuned open source one, or a commercially licensed one?

I forgot that we're playing the demo games now.

Well I'll see your demo, and raise you another demo

https://x.com/sean_moriarity/status/1760435005119934862

Except that demo was done by a single person, with a day job and no cap table.

This is very interesting. One thing I wondered about the per-minute pricing is how to keep a phone agent like this from being kept on the phone in order to run up a bill for a company using it. It'd be very inexpensive to make many automated calls to an AI bot like the dentist receptionist in the demo, and to just play a recording of someone asking questions designed to keep the bot on the phone.

As a customer of a service like Retell (though of course not specific to Retell itself), how might one go about setting up rules to keep a phone conversation from going on for too long? At 17¢ per minute, a 6-minute call will cost just over $1, or about $10 per hour. Assuming the AI receptionist can take calls outside of business hours (which would be a nice thing for a business to offer), then such a malicious/time-wasting caller could start at closing time (5pm) and continue nonstop until opening time the next day (8am), with that 15 hour span costing the business $150 for billable AI time. If the receptionist is available on weekends (from Friday at 5pm until Monday at 8am), that's a 63-hour stretch of time, or $630. And if the phone system can handle 10 calls in parallel, the dentist could come in Monday morning to an AI receptionist bill of over $6,300 for a single weekend (63 hours × $10 per hour × 10 lines).

This is in no way a reflection on Retell (I think the service is compelling and the usage-based pricing is fair, and with that being the only cost, it's approachable and easy for people to try out). The problem of when to end a call is one I hadn't considered until now. Of course you could waste the time of a human receptionist who is being paid an hourly wage by the business, but that receptionist is going to hang up on you when it becomes clear you're just wasting their time. But an AI bot may not know when to hang up, or may be prevented from doing so by its programming if the human (or recording) on the other end of the line is asking it not to hang up. You could say it shouldn't ever take more than five minutes to book a dentist appointment, but what if the person has questions about the available dental procedures, or what if it's a person who can't hear well or a non-native speaker who has trouble understanding and needs the receptionist to repeat certain things? A human can handle that easily, but it seems difficult to program limits like this in a phone system.

Congratulations! How do you position yourselves against Google Duplex/Dialogflow and other competitors? https://cloud.google.com/dialogflow

This is interesting but having a piece like my speech engine tied to a specific model provider is a non-starter. I'll probably become a customer at some point if you guys just make a cheap API for streaming natural voice from LLM text ouput, if open source tools don't solve that problem conclusively before then.

It's really good, but the AI cracks still show up. Trying the demo therapist, I mentioned I'm not finding a job. It suggested finding a career counsellor and said "it would get back as soon as possible"... yeah, no it didn't. It claimed to be "working on it" but would say "I'm here if you want to speak...". It clearly doesn't understand what it's saying, it feels like bing's ai would be "better" at not claiming to do a task it can't.

Excellent product and fantastic demo!

Would it be possible to integrate Retell with a RAG pipeline?

the actual conversational flow is awesome. 800ms is only a little worse than internet audio latency (commonly 300-500ms on services like Discord or even in-game audio for things like Valorant)! Also cool that you can bring your own LLM and audio provider. awesome product!

How does this compare to vocode, another YC company?

Just tried, its impressive but needs work - trying to book an appointment out in 3 weeks, it acked that but could not confirm exact date time. Still impressed.

Awesome product. We would love to use this for our app, but it wouldn't make sense economically. At $0.10 per minute it would cost significantly more than our existing TTS and SST solution. We've manually added a VAD and will have to add a way of handling interruption. All-in-all it roughly costs us $0.01 per minute and we just can't afford a 10X increase in costs.

Guessing you guys have found a use case with higher margins than ours which'll explain the price. Great work. Hope we can afford this one day.

Just tried the dental appointment example. Voice sounds great! But I found two issues with sharing: - I told it I wasn't available until next year. We confirmed a date. It said Feb 4th, next year. I asked it when next year was and it gave me the definition. On further prying, it told me the current year was 2022, so next year was 2023. For a scheduling use case, it should be date/availability/time zone aware. - At the end, it got into a loop saying "I apologize for the confusion. Let me double check your records...". After staying silent, it said "it looks like we've been disconnected". I said "no, I was waiting for you to check my records". The loop repeated. I eventually asked how long it would take to check my records and it told me "a few minutes" but still went through the "disconnected" message.

Wonder what the justifications for the different voice prices are...

With a little more tweaking and training, the voice AI will sound like her in https://en.wikipedia.org/wiki/Her_(film).

Very cool! Have you looked into call center agents use cases?

Is there societal value that this product is harming?

One use case that I'd be interested in for this is training it to use my voice a la Descript. It's been really nice that our Head of Marketing can iterate on video voiceovers using my voice and all I had to do was read 30 seconds of copy.

Any plans for something like that? It'd be really interesting to have prospects what with an AI version of me to answer some basic questions.

Congrats on launching. It feels very natural and the demo call was good.

Nice work. You seem to have addressed some of the challenges that arise in teaching computers to speak.

This blog breaks it down well:

https://www.papercup.com/blog/realistic-synthetic-voices

This is super cool, congrats on the launch!

I'm always curious for things like this where people get training data.

Commercially, I struggle to see how this fits in to the most natural application, a contact center.

For example, take Amazon Connect pricing:

There is an Amazon Connect service usage charge, based on end-customer call duration. At $0.018 per minute * 7 minutes = $0.126

There is an inbound call per minute charge for German DID numbers. At $0.0040 per minute * 7 minutes = $0.0280

If I were to bring retell into the loop, I'm changing my self-service per minute cost from .018 + .004 = .0184 per minute to .1184 per minute at the cheapest setting. And from that, I don't have a clear case for the impact on deflection, and because of that I don't have clear ROI.

This isn't saying that the product isn't great -- but at the current price I struggle to see how anyone at scale can use it without eating the double whammy of LLM + Retell costs.

Could you add Partykit/Cloudflare Durable Objects integration docs?

If I record screen and audio when playing video games with my friends, will I be able to fine-tune an LLM+audio model on that dataset?

It'll be like that San Junipero episode of Black Mirror - immortality in a dark way.

First thought is can’t you set up calls with prospects and have the product _literally_ sell itself?

In the "Select Voice" dialog, all the DeepGram clips end in a loud click. Might want to fix that.

I tried the demo, and it got confused and disconnected, but it's a cool proof of concept. Suggest bumping up the happiness emotion on the agent, and a Calendly integration would immediately unlock a lot of use cases. Good luck!

The demo was incredible, and this seems perfect for my current project. I am going to try to integrate this as soon as I can to see if it works for me. How responsive can I expect the support to be?

My friend group and I have been playing with LLMs: https://news.ycombinator.com/item?id=39208451. We tend to hang out in multi-user voice chat sometimes, and I've speculated that it would be interesting to hook an LLM to ASR and TTS and bring it into our voice chat. Yeah, naive, and to be honest, I'm not even sure where to start. Have you tried bringing your conversational LLM into a multi-person conversation?

Curious what model the dentist bot is running on? Tried it out, was surprisingly good, though eventually it contradicted itself (booked a slot it said previously was not available). (I get that’s the programming but am curious especially given the latency is really great).

This is absolutely wild - I got chills when I thought about the fact I’m talking to a computer. Congratulations on flying straight over uncanny valley.

I'm assuming you're targeting this mainly at enterprises and business use-cases such as callcenters, but are you planning to make this usable for personal use cases as well? For example, having a bot to bounce ideas off while coding. Pretty much "just" the TTS / STT layer to talk to my finetuned LLM in a natural manner while you handle interruptions and such.

I think the main issue right now for personal use would be cost (and I'm guessing STT / TTS are the most expensive parts..)

Sad that people think working on crypto is a waste of time yet here we are making antiquated contact methodologies even harder to prune out.

Wow, this is sweet! With a little better latency and less perfection, it'd be well over the uncanny valley (not that it wouldn't fool many people as-is). Are you planning to add more "human" elements like filler words or disfluencies? If anything it feels too perfect to be human. Awesome stuff!

P.S: I tried to fool the Dental Office demo trying to book on Sunday or outside of the slots it had indicated, and it did a better job than many humans would have :)

How is this different from vocode.dev

backchanneling phrases are annoying in human conversation

Please make a free community self host option available like danswer from your own YC batch..

https://news.ycombinator.com/item?id=39467413

What languages are you supporting?

It's pretty amazing but can get a bit lost/into a bugged state.

- Did not work on firefox for me (start conversation and nothing would happen, I would not hear the voice) - on chrome it would not allow to change my microphone so had to open my macbook. - Also on firefox, when I first logged in, clicking on he "try now" button would send me back to the landing page, I had to go and click on the playground

With that out of the way, it's really interesting. The challenge I suppose is how narrow / wide the API made by the developer is. A narrow case, like "booking a dentist appointment", might feel like a step down compared to an online form, and most likely would fail short of satisfying someone calling, because if they're calling they had some deeper needs.

On the other hand, a wide API, like if you gave access to pricing information of the dental practice, info about the doctors, a way to reach an actual human being, health advice and post op advice, generate documents from past appointments etc, and you have a higher chance of hallucination, misclassification of what the user wants etc.

I'm still not sure if the best approach is just to leave the AI to deal with that, or to represent the user intent etc. with some sort of State Machine. Maybe a case by case etc. But the more you add in that logic, the slower the machine is.

is possible training add knowledge with a pdf? thanks in advance is all magic

Unfortunately, based on my experience, GPT-3.5 is simply not up to snuff in phone conversations in terms of style and reasoning. And GPT-4T latency is far too large for real-time conversation with straightforward STT-TextGen-TTS pipeline.

This was impressive have you integrated this with something like Amazon Connect, or Twilio? Ultimately looking how I could bring it to my customers via what they use for their call centers.

This really makes me wonder if we can go full circle at fast-food restaurants like McDonalds. Ordering at a counter (10 seconds) has been replaced in recent years with touching a giant screen (2+ minutes).

This demo really shines light on how you could "talk" to place your order in a comparatively fast amount of time, similar to speaking with a human. Yes it can't handle all the million corner cases, and it will get it wrong a bunch of the time, but at least you can verify your order on a screen before tapping your phone to pay.