Prince Charles Quips About 'Enormous Amount of People … Waiting for Me to Get Out' in Self-Effacing Speech

Prince Charles isn't above making a joke at his own expense.

During his speech at Thursday's Prince's Trust Awards Trophy Ceremony at St. James's Palace in London, the future king was taking a moment to thank the events sponsors when he got in a little quip that seemed to throw a little shade at his odd position as heir to the British throne.

Telling a story about a visit he made to a local T.K. Maxx store a few years back, he said, "I remember there were an enormous number of people who were trying to get in, as they were waiting for me to get out!"

He added drolly, "Story of my life…."

Royal watchers can't help but read into those four words — in the years before her death in 1997, Princess Diana had publicly hinted (and reportedly explicitly told friends) that she believed her older son, Prince William, should bypass Charles and ascend the throne to be the next King.

And it's not unusual for the British royals to make wry, self-effacing remarks. During a June visit with Australian Prime Minister Scott Morrison, Queen Elizabeth was told she was "quite the hit" when the royals had hosted at a G-7 Summit reception — so much so that the ultra-exclusive club of world leaders was still talking about her the next evening.

Without missing a beat, the 95-year-old monarch replied, "Oh, Lord. Were they really?"

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

At Thursday's ceremony, which was attended by Oscar nominee Richard E. Grant and Vogue U.K. Editor-in-Chief Edward Enninful, Charles's witty asides were woven into the story of decades of work toward a mission very close to his heart.

"When I created The Prince's Trust in 1976 to help improve the lives of disadvantaged young people, it was because I was so acutely aware of the challenges that they faced," shared the prince, 72. "Over the years some of the challenges have changed, but the overall mission of giving people self-confidence, self-esteem and better opportunities, remains the same."

He noted with pride that "we have helped over one million young people transform their lives and The Prince's Trust now works in 18 countries across the Commonwealth and beyond."

Of course, some dark humor crept in as he mentioned the Trust's expansion into the U.S. and the global relationships they've been cultivating from Jamaica to Jordan as he added, "And before I drop dead, I want to make sure we can get even further!"

Speaking of the "deeply concerning" job statistics for young people in the U.K. in the wake of the COVID-19 pandemic, he also pointed to areas of opportunity and aspiration — including the passion he shares with William, 39, for finding practical solutions to climate change.

"The Prince's Trust is determined to help support young people into employment and enterprise in the rapidly changing global market, including of course the fast-growing green economy," he said.

He wrapped up his speech, telling the recipients, "I am enormously proud of all of you and cannot possibly congratulate you enough. I hope today's experience hasn't been too much of a shock to your system, so that you can recover with a stiff drink at the end!"