"Not to take anything away from you..."

When you insult someone's character, integrity and intelligence, I feel you should own up to the insult. Don't follow up with "not to take anything away from you," because that's exactly what you meant to do. Rather, follow up with, "not that it's your fault you lazy piece of shit."