I'll pick my battles, it's part risk appetite and part expected attacker model. It also depends on what I'm trying to protect.

If it's in a standard library of an open source programming language, I'm less inclined to fully check the implementation.