I noticed this while reviewing #5478. Apparently it's safe enough, but it makes me uneasy anyway. If this is desired, I'm happy to rebase this after #5478's merge.
Since uint256 has .begin() and .size(), I don't see why we should trust that a pointer to the object lines up with the data we're after rather than using those functions.
I did a naive grep to find these, there are likely others as well.