The lift_x function in the BIP and the reference implementation expect an integer to be passed rather than a byte array.
Can be tested with:
$ python3 test-vectors.py > expected.csv
$ diff test-vectors.csv expected.csv
On master, the first call fails with
Traceback (most recent call last):
File "/home/thestack/bips/bip-0340/test-vectors.py", line 267, in <module>
vector0(),
^^^^^^^^^
File "/home/thestack/bips/bip-0340/test-vectors.py", line 30, in vector0
pubkey_point = lift_x(pubkey)
^^^^^^^^^^^^^^
File "/home/thestack/bips/bip-0340/reference.py", line 72, in lift_x
if x >= p:
^^^^^^
TypeError: '>=' not supported between instances of 'bytes' and 'int'