A permit to carry a concealed handgun is just that—a permit to carry it. It’s not a permit to use it. Use of that handgun is covered by other laws dealing with self-defense, and those laws would apply whether you were carrying a handgun or not. But, if you are carrying a handgun under a legally issued permit, you will be held to a higher standard of behavior; if you have to use that gun, your every action leading up to its use will be put under a microscope.
A concealed-carry permit holder who becomes abusive, threatening or argumentative, or whose behavior is viewed by the investigating authorities as initiating the confrontation, might be seen as the aggressor. In that case, a claim of self-defense might be denied. The aggressor loses the right to be the victim in a self-defense claim in most jurisdictions. The fact that you’re carrying a gun does not mean that you don't have to “take stuff” from anybody. It actually means that you have to take more “stuff” than if you were not carrying.
A gun on the hip carries with it the power of life and death. The responsible concealed carrier will comport himself with greater calm and wisdom than an unarmed man. A simple argument resulting in a fistfight when unarmed can escalate far beyond that when a gun is present.
The responsible concealed carrier will avoid any confrontations. The gun is only there as a last resort.