I wouldn't say understand. But your answers is patterns. Formalism is mostly definition (axioms) and inference rules (theories). If we take programming languages, most grammars (which describe these two elements) are only a few pages long. With LLM being patterns seeker at its core, I guess it would be easy to extract the rules from a sample of programs, as the structure is so rigid.

You won't get the Turing machine evaluation mechanism and determinism, but you will have a generator. Although the viability of what is generated is is question. Because the other part of formalism, semantics, is almost always missing.