Sure you should be suspicious. You should always be suspicious. Especially if it's free. And you can do something to calm your suspicions. Like checking out Signlas Open Source code.
What's preventing them from serving a backdoored version? xz was open source as well, that didn't stop the backdoor. There might be reproducible builds on android, but you can't even inspect the executable on iOS without jailbreaking.
Signal designs their systems from the ground up to deliver verifiable trust mechanisms (via remote attestation) along with data minimization/zero-access encryption techniques.
It is neither against the signal software’s license, nor it is against the signal service’s terms of service.
This is a false meme spread because the Signal founder (who is no longer with the company) didn’t like people making forks without changing the API server URL and running their own servers.
Open source software doesn’t work like that, however.
How would that calm suspicion if you're not arr/ign-orant and understand that continuous security audit is practically impossible at an individual level?