I have not read it but it looks like a comprehensive reference. For a more applied treatment see Foundations of Deep Reinforcement Learning. https://slm-lab.gitbook.io/slm-lab/publications-and-talks/in...

Neither cover LLMs. I don't follow the literature closely so I can only suggest you read papers: https://github.com/WindyLab/LLM-RL-Papers