A copied voice is now easy to make, so a familiar voice on the phone is no longer proof of who is calling. Copy a voice only with the person’s written consent. When a call asks for money or secrets, hang up and call back on a number you already know.
Say your phone rings and it sounds exactly like your manager. They need a payment sent to a new supplier in the next hour, and they ask you to keep it quiet until the deal is announced. The voice, the way they say your name, and the background noise all seem right. It could still be a copy built from a few minutes of a public interview.
This post explains how voice copying works in plain words, what consent and the law require, four steps to take when a call asks for money, what each defense can and cannot do, and a short program that applies a safe call policy.
How a voice gets copied
Older systems needed hours of studio recordings of one speaker. Many current tools need only a short clip. A part of the model called a speaker encoder turns that clip into a list of numbers that captures how the person sounds. The model then speaks any new sentence in that voice.
That short clip can come from anywhere a person talks in public: a podcast, a conference video, or a social media post. Phone calls make copies harder to spot, because phone lines carry only part of the sound and smooth away small details a listener might otherwise notice.
What consent and the law require
If you copy a real person’s voice for your own work, get written consent that says exactly what the voice will be used for, where, and for how long. Pay the person fairly, and delete the voice model when the agreement ends. Union performers have their own contract terms on digital replicas, so check those before you record anyone who belongs to a union.
Laws are catching up. Tennessee’s ELVIS Act (Ensuring Likeness Voice and Image Security), passed in 2024, added a person’s voice to the rights the state protects against unauthorized commercial use. In the European Union, Article 50 of the AI Act has applied since August 2, 2026, and requires anyone who publishes a deepfake, including realistic AI audio of a real-seeming person, to disclose that it is AI-made, according to the European Commission. These are general points, not legal advice.
Four steps when a call asks for money
The U.S. Federal Trade Commission (FTC), the government agency that protects consumers from fraud, says it plainly in a consumer alert from March 20, 2023: “Don’t trust the voice.” It advises calling the person back on a number you know is theirs, and it warns that requests to wire money, send cryptocurrency, or buy gift cards can be signs of a scam.
- Pause: Urgency and secrecy are the scammer’s main tools. A real request can wait ten minutes.
- Hang up and call back: Use a number you already have, not the one that called and not one the caller gives you.
- Confirm another way: Ask for a code word agreed in advance, or confirm in a chat or email account you trust.
- Report it: At work, tell your security team. At home in the U.S., report it at ReportFraud.ftc.gov.
At work, the strongest fix is a rule that no single phone call can move money. Payments to new accounts need a second person and a confirmation outside the call.
What each defense can and cannot do
Some tools that generate voices hide a watermark in the audio, a pattern people cannot hear but software can detect. Meta published research on one such method, AudioSeal, and Google uses SynthID in its own products. Content Credentials, built on the C2PA standard, attach a signed record of how a file was made.
These help, but they have limits. A scammer can use a tool that adds no watermark, and a missing watermark does not prove a voice is real. Software that tries to spot fake voices can miss good copies, which is one reason the FTC ran a Voice Cloning Challenge in 2024 to encourage better defenses. Treat these tools as hints. The call-back habit works even against a perfect copy.
A call policy you can run
The short Python program below applies that policy to three made-up calls. Any high-risk request needs a call back on a known number and a code from a second channel, no matter whose voice it is. It needs nothing beyond Python.
# A call policy that never treats a voice as proof of who is calling.
# The calls below are made up for practice.
HIGH_RISK = {"wire money", "change bank details", "reset password", "buy gift cards"}
RED_FLAGS = {"urgent", "secret", "gift cards", "crypto"}
calls = [
{"sounds_like": "your CFO", "asks_to": "wire money", "says": "urgent, keep this secret",
"called_back_on_known_number": False, "code_from_second_channel": False},
{"sounds_like": "your CFO", "asks_to": "wire money", "says": "pay the approved invoice",
"called_back_on_known_number": True, "code_from_second_channel": True},
{"sounds_like": "a coworker", "asks_to": "share the meeting notes", "says": "when you have a minute",
"called_back_on_known_number": False, "code_from_second_channel": False},
]
for call in calls:
flags = sorted(f for f in RED_FLAGS if f in call["says"].lower() or f in call["asks_to"])
if call["asks_to"] in HIGH_RISK:
checked = call["called_back_on_known_number"] and call["code_from_second_channel"]
decision = "go ahead" if checked else "stop: call back on a known number and confirm a code"
else:
decision = "low risk, carry on"
note = f" (red flags: {', '.join(flags)})" if flags else ""
print(f"Sounds like {call['sounds_like']}, wants to {call['asks_to']}: {decision}{note}")Running it prints:
Sounds like your CFO, wants to wire money: stop: call back on a known number and confirm a code (red flags: secret, urgent)
Sounds like your CFO, wants to wire money: go ahead
Sounds like a coworker, wants to share the meeting notes: low risk, carry onIn plain words, the program keeps a list of high-risk requests and common warning words. The first call sounds like your CFO, wants money wired, and uses “urgent” and “secret,” so it is stopped. The second asks for the same thing but has been checked by call-back and code, so it can go ahead. The third is a low-risk request. Notice that how the caller sounds never changes the decision.
Quick recap
- A short public clip can be enough to copy a voice, so a voice alone proves nothing.
- Copy a voice only with written consent that sets the use, the place, and an end date, because consent without limits invites misuse later.
- When a call asks for money, pause, call back on a known number, and confirm another way.
- Watermarks and detectors help, but the call-back habit is the defense that always works.
Your next step this week
Agree on a code word with your family and, if you handle payments, with your team. Keep it off social media. Then write down the rule that no single call can move money, and share it with anyone who could be targeted.
Series notes
This is Part 4 of Voice models from scratch. Previous: the speech AI product map. Next: voice quality engineering.
Sources
- U.S. Federal Trade Commission, Scammers use AI to enhance their family emergency schemes, March 20, 2023.
- U.S. Federal Trade Commission, The FTC Voice Cloning Challenge.
- Tennessee General Assembly, HB 2091, the ELVIS Act, 2024.
- European Commission, Transparency obligations under Article 50 of the AI Act, checked October 3, 2026.
- Robin San Roman and others, Proactive Detection of Voice Cloning with Localized Watermarking (AudioSeal), 2024.
- Google DeepMind, SynthID.
- Coalition for Content Provenance and Authenticity, C2PA.
Keep going
Same lessons in your feed
Short diagrams, hooks, and weekly tutorials on Substack, Instagram, X, and Facebook.
