Henry Cavill opens up to PEOPLE about being teased about his weight as a kid

Growing up in the Channel Islands in the U.K., the Man from U.N.C.L.E. star was bullied for being overweight as kid while attending St. Michael’s Preparatory School in Saint Saviour, Jersey.

“I was teased and it was hard,” he tells PEOPLE.

While the muscle-bound action star has certainly gotten the last laugh, he says he doesn’t harbor any resentment toward the kids who picked on him.

“Even though for some it may be malicious, I think it’s just kids being kids, and I don’t hold any grudges,” he says. “People have had it far worse than me.”

As Cavill got older, he learned how to use his social experiences to his advantage. “Someone who is on the outskirts of popularity is someone who gets to look in,” he explains. “So I became very much an observer, and that taught me an awful lot about people.”

He adds, “It helped me read people very well, which has helped me in this industry. If I can understand the inner workings of a human being, then I can probably be a better actor.”

Getting teased in school has also helped the chiseled star stay grounded despite his success and sex-symbol persona.

“You take everything in stride,” he says of all the attention paid to his good looks. “Of course it’s flattering when people say that, but you’ve got to keep yourself aware of the fact that it’s all perspective. There’s one group of people saying that and there’s one group of people who may not feel the same way.”

In the upcoming Batman v Superman: Dawn of Justice, Cavill stars alongside another dashing star, former PEOPLE’s Sexiest Man Alive Ben Affleck.

While Affleck’s onscreen persona, the Batman, isn’t known to be a barrel of laughs, Cavill says the Gone Girl star was hysterical to work with.

“Ben was fantastic, he’s a really funny guy,” he says. “I had to try quite hard not to laugh in between takes and stuff because he is really, really funny.”

He adds that Affleck’s “wonderfully dry sense of humor” helped the cast get through some long hours on set. “It’s good to be around that kind of thing on long, long jobs because you want to be able to take the levity out of a situation.”

Cavill says that Affleck – who announced in June that he and Jennifer Garner are divorcing – is a “wonderfully talented actor” and that “he did a wonderful job as Batman.”