Sum 41 frontman Deryck Whibley is reflecting on his relationship with Avril Lavigne — and how hostile responses from both of their fans got to his head.

In his new memoir Walking Disaster: My Life Through Heaven and Hell, the pop-punk musician recalled the toll of heightened media scrutiny during his romance with the "Complicated" singer. "I felt unsafe and harassed everywhere I went, and now with all the extra attention on my relationship with Avril, it had gotten even worse," he wrote. "I wanted all of it out of my life."

Chris Polk/WireImage

Whibley said that the intense disapproval from Sum 41 fans weighed heavily on him during the relationship. "While we loved being together, it seemed our fans didn’t feel the same way," he wrote. "I got a lot of flack for dating a pop star. Sometimes at our shows, people would bring signs that read, 'Avril Sucks!' When I signed autographs after the show people would flat-out say, 'Why are you with her? Her music is s---.'"

Lavigne's listeners weren't much better. "Her fans thought I was ugly and not good enough for her," Whibley remembered. "All of this made us retreat into our own relationship. Us against the world, f--- ’em all."

The disapproval became so dramatic that Whibley feared for his safety — with good reason. "Our management had been notified by the FBI that I was receiving credible death threats for being with Avril," he said. "Normally, they ignore these things, but these letters were a bit more sophisticated, and had no fingerprints on them. They suggested that the band bring on professional security that dealt with this kind of stuff."

Whibley said that the media also treated him with immense cruelty. "If I was with Avril, they either looked down on me or ignored me entirely," he remembered. "Whenever online press used a photo of us, they didn’t bother mentioning Sum 41, and snarky writers started calling me 'Mr. Lavigne' or 'Avril’s troll of a husband.' Paparazzi would taunt me with Avril’s success, trying to get me to react on camera. As if my life could get any worse!"

Kevin Mazur/WireImage

The musicians' relationship also faced resistance from Whibley's Sum 41 bandmates. "I knew the band hated the fact that Avril and I were dating, so I was hesitant to talk about my personal life with them, too," he wrote. "I had heard from friends that they were s--- -talking us behind my back, and whenever Avril came around they acted weird and uncomfortable and didn’t make much of an effort to make her feel part of our club."

Whibley acknowledged that his ex-wife "can be strange and shy" around people whom she doesn't know well, noting that her small-town upbringing affected her social skills. That didn't stop her from picking up the negative vibes, though. "Avril was savvy enough to pick up on the fact that my band didn’t seem to like her, and she would often ask me if they hated her," he wrote. "I just said, 'Of course the guys like you! They’re just tough to get to know at first, but they’ll come around.'"

Sign up for Entertainment Weekly's free daily newsletter to get breaking TV news, exclusive first looks, recaps, reviews, interviews with your favorite stars, and more.

The band's negativity toward Lavigne was so intense, Whibley said, that former Sum 41 drummer Stevo Jocz called her out for the group faltering. "We all knew how much he hated Avril and that relationship," the singer wrote. "He had outright blamed it for a lot of the failure of the Underclass Hero album. I don’t remember what we were arguing about, but in a moment of anger, and drinking, he once said, 'We’re not doing as well anymore because you dated Paris Hilton and now are with Avril Lavigne.'"

After marrying in 2006, Whibley and Lavigne grew apart and ultimately divorced in 2010, though they continued to collaborate on the soundtrack for Tim Burton's Alice in Wonderland after their separation.

Walking Disaster: My Life Through Heaven and Hell is out now.