How Judy Smith — the Real Olivia Pope! — Told President George H. W. Bush About 'Scandal' 's Plotline

Scandal wound up being a major hit that is still referenced within popular culture to this day — but it wouldn't have been a reality without Judy Smith.

Smith, 63, previously served as deputy press secretary in President George H. W. Bush's administration. In 1991, she became the first Black woman to deliver a White House press briefing. She eventually left her position to start the crisis management firm, Smith & Company — and then her real-life work inspired Scandal, which was created by Shonda Rhimes.

The series featured Kerry Washington as Olivia Pope, who was a fictionalized version of Smith. But the big twist audiences learned in the show's 2012 premiere was that Olivia is having an affair with President Fitzgerald "Fitz" Thomas Grant III, played by Tony Goldwyn.

For the ABC drama's 10-year anniversary, Smith tells PEOPLE exclusively about how the late president reacted to the news of Scandal's creation — and the central character's steamy love story.

"As a good crisis person, you need to quickly reach out to stakeholders — and one of those key stakeholders was the president. I'm like, 'Okay, I got to tell him first. I got to call him up,' " Smith, who was a co-executive producer on the show, says of having to tell Bush about Scandal.

"So after I talked to Shonda about it, I called up the president, and I talked to his chief of staff. And I said, 'Look, I've got to tell him about the show, and I just wanted to make sure he's in the loop on it.' " she recalls, adding, "I remember the chief of staff saying, 'Oh, no. He's so excited about the show. He's so proud of you and everybody.' I said, 'No, there are other little things I need to talk to him about.' "

Bush did eventually call Smith back to discuss the series further — and he left a hilarious message while doing so.

"I remember he left a message on the cell phone because he's always a joker. It was like, 'I love you. I want you. You left me. By the way, I'm the former leader of the free world. Call me,' " she says. "I called him up. I said, 'See, this is why I'm calling you now. Let's stop joking about this stuff.' And so I told him what the storyline was going to be, and he said, 'Oh, yeah. No, yeah. I remember that.' I said, 'No, there's nothing! There's no relationship. Don't even be joking that any of that stuff is true!'

"I had to tell him. I said, 'It's not good. It's not going to be good. People are not going to take your joking,' " she continues. "He was making jokes about out how people, some of his staff members were saying, 'Oh, no, it'd be good for my reputation.' And I'm like, 'No, it is not.' "

Smith had told Bush she'd send over talking points for him to use when discussing the series and gave him a strong warning if he went against them.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"I said, 'If you do not go along with those talking points, I'm going to tell your boss. Don't play around!' And he's like, "You wouldn't,' " she recalls. "And I said, 'I will call up [your wife] Barbara in a second. I'm not fooling with you.' Because he was going, 'Remember that time we went to this place? We'd meet at that place?' I said, 'No, there's no time. Do not make up anything. I'm not playing.' "

Smith adds, "Because he was known as a jokester, he had a reputation of that. I was like, 'This is nothing to play with now.' "

Scott Foley, Bellamy Young, Katie Lowes, Guillermo Díaz, Darby Stanchfield, Jeff Perry, Joshua Malina, Joe Morton and George Newbern also starred in Scandal, which ran for seven seasons on ABC.

During its time on air, Smith recalls receiving phone calls from Bush raving about the series. "He watched it. He would call me on an episode like, 'Oh, wow! Where did you get that from?' " she says.

"Then he would call me one time. He said, 'See, I told you that's a good character. I'm playing a good president,' " remembers Smith. "So we'd laugh about it. And he was very gracious."

Smith says the former president "wanted to be supportive" of her during that time, even hosting a dinner for her at his family's Texas home. "He said, 'Come on down. I'm going to throw a shindig for you,' " she says, adding that it was "very thoughtful of him."

Scandal's influence extends past Bush as Smith says it "definitely raised the profile" of Smith & Company. The renowned firm has represented several high-profile clients, including Angelina Jolie, Monica Lewinsky and Wesley Snipes.

Smith also notes how Scandal helped change the landscape of television. At the time, Washington's casting made her the first Black woman to lead a network television drama in 38 years.

"I think it's amazing because it opened the door for what you see happening right now in television: more black women are cast in lead roles," Smith says. "For so many decades, there seemed to be, obviously, people who were hesitant to cast women in lead roles, and so I think it showed to the industry and to people that that could be successful. I think the other thing that it also showed is a different way to galvanize an audience."

Smith adds that Scandal "was able to demonstrate to an industry that thought that this wasn't possible, that it actually was — and it was a hit."