Yeah that's where everything gets complicated, obviously people's natural rights can conflict and that's where you need to set up some sort of rules based system, but the point is that you don't infringe on someone's natural rights unless there's a direct conflict with someone else's natural rights. For example your natural rights to bear a weapon cause a conflict if you are bearing them in someone's house in order to rob them after you just broke in. Conflicts with their right to have property, etc.
It seems like your position is "natural rights should guide or at least influence societal rights", and you consider that to be self-evident or maybe axiomatic
I don't see why natural rights would be part of the discussion altogether
