It is unsigned in Bitcoin Core, so the tests should match it:
Large positive values for the block height are too difficult to hit in tests, but it still seems fine to fix this.
The bug was introduced when the code was written in 6ccc8fc067bf516cda7bc5d7d721945be5ac2003.
(Lowercase i
means signed, see https://docs.python.org/3/library/struct.html#format-characters)