"When my wife Caroline was dying, seven years ago, she said, 'You should tell people you're leaving Parliament to devote more time to politics.'"