There are a couple of errors in setban()
which return incorrect or
misleading errors. This commit fixes those and updates nodehandling.py
to test that the correct error codes and messages are being returned.
Also add a testcase for trying to setban on an invalid subnet.
This PR builds on top of https://github.com/bitcoin/bitcoin/pull/9707