Condemning people to death and bankruptcy is violence.

That said, we consider vigilante justice wrong because we believe there is supposed to be actual justice from a functional system. When the system does not function, extrajudicial attempts at justice will become more common. Just like people will poach more in a famine.