Do Not Throw Away Your Confidence