There is a strong correlation between the behavior an animal will exhibit against a human and the behavior it will exhibit against other animals of its own species.

The animals which do not tolerate other animals of their own species and which will attack them and fight with them are also very likely to attack humans when they believe that humans have invaded their personal space. For example, an adult bear will never be truly friendly with a human, even with a human that has raised it as a cub, because adult bears are never friendly with other bears, but they attack any intruders. On the other hand, a wolf raised by a human can become tame and attached to the human, like a wild wolf would behave towards its real parents.

Similarly, male sperm whales fight viciously with any other male sperm whales and they also do not hesitate to attack any boats with humans that harass them.

AFAIK, intra-specific fighting is not frequent among orcas, but they are used to have good relations between them, even with some from different pods. This may explain their lack of aggressivity against humans, as long as they are not perceived as a possible prey.

I'm interested in those examples of sperm whales attacking humans. I believe those might have been defending and not actively attacking. It is said that Mocha Dick was docile until attacked. And I think that an animal that defends itself when attacked, is a different game. I haven't heard of cases of people hunting down and fishing Orcas, like we did with Sperm Whales. Perhaps we would have had Orcas attack then?