Consensus-wise we already treat it as an unsigned integer (the only rules around it are in CSV/locktime handling), but changing the underlying data type means touching consensus code for a simple cleanup change, which isn't really worth it.
See-also, https://github.com/rust-bitcoin/rust-bitcoin/pull/299