Jensen Ackles Recalls Sibling-Like 'Bickering' with Jessica Alba on 'Dark Angel' Set: 'I Got Picked On'

Jensen Ackles is opening up about working with Jessica Alba on the set of Dark Angel.

During a Wednesday appearance on Inside of You: Michael Rosenbaum, the host asked Ackles, 43, what it was like to work with Alba, 40, on the Fox series, which aired from 2000 to 2002.

"Horrible," Ackles said. "I'm not kidding. I've told this to her face."

Ackles clarified: "I love Jess, which I know kind of contradicts what I've just said ... She was under an immense amount of pressure on that show. She was young, she was in a relationship with [costar Michael Weatherly], and that was rocky and causing some undue stress I believe on set."

Ackles, who joined the series in its sophomore season, pointed out, "I was the new, fresh face on the set who wasn't really there in season one. I came in for one episode but then they wrote me in as a series regular for season two. I was the new kid on the block, and I got picked on by the lead."

He described his connection with Alba as "the worst kind of bickering a brother and sister could do."

"She had it out for me," the Supernatural star added.

When asked by Rosenbaum if Alba simply "didn't like" him, Ackles shared, "It wasn't that she didn't like me. She just was like, 'Oh here's the pretty boy that network brought in for some more window dressing, 'cause that's what we need.' "

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

He confirmed the Honest Company founder called him out in front of the crew and that he decided to fight "fire with fire," ultimately leading to a "mutual respect" between the stars.

Ackles recalled how Alba was supportive after his grandfather died while they were filming the series. "She literally just walked into my trailer and held me for half an hour," Ackles shared. "It was that kind of a relationship."

He said of their friendship today, "If she walked in, we'd be all hugs."

RELATED VIDEO: Jessica Alba "Always Wanted to Be Treated the Way I Saw Men Be Treated" on Set

A rep for Alba did not immediately respond to PEOPLE's request for comment.

In 2012, Jessica Alba shared her experience on the set, revealing she felt "objectified."

"I had a show [Fox's Dark Angel] that premiered when I was 19," Alba told Marie Claire. "And right away, everyone formed a strong opinion about me because of the way I was marketed. I was supposed to be sexy, this tough action girl. That's what people expected."

" I felt like I was being objectified, and it made me uncomfortable," she explained. "I wanted to be chic and elegant!"