[tests] Fix RPC failure testing (again) #10853

1. 
   jnewbery commented at 3:55 pm on July 17, 2017: member

   I did this a few months ago (here: #9707), but a few new examples have crept back in.

   When testing RPC failures, the test case should always assert the error value and message, to ensure that the failure was for the correct reason. Not doing that can hide bugs in the test code and mean that the test is not testing the correct behaviour.

   RPC failure testing should use the utility function assert_raises_jsonrpc() (renamed in the final commit of this PR to assert_raises_rpc_error().

   This PR does the following:

   • changes all remaining instances of tests directly testing on JSONRPCException to calls to assert_raises_jsonrpc()
   • prevents assert_raises_message() from being called with JSONRPCException
   • scripted-diff changes assert_raises_jsonrpc() to assert_raises_rpc_error()
2. laanwj added the label Tests on Jul 17, 2017
3. jnewbery force-pushed on Jul 24, 2017
4. jnewbery force-pushed on Jul 24, 2017
5. 
   jnewbery commented at 3:55 pm on July 24, 2017: member

   Rebased and scripted-diff Travis check fixed.
6. 
   MarcoFalke commented at 1:42 pm on July 30, 2017: member

   Not sure if it is worth to go through the rebases due to renaming of the method. Mind to drop the last commit for now?
7. 
   jnewbery commented at 11:17 pm on July 30, 2017: member

   I wanted the name change for a couple of reasons:

   • assert_raises_jsonrpc() doesn’t make much sense. It’s raising an error, so it should include that in the function name. Hopefully changing it to assert_raises_rpc_error() will mean people notice the function and use it instead of assert_raises_message() or try to roll their own.
   • with --usecli (see #10798), I’m adding a new mode to the test framework, where RPCs are sent via bitcoin-cli rather than using JSON over HTTP. In that context, assert_raises_rpc_error() makes more sense.

   It’s a scripted diff changing one function name, so rebasing shouldn’t be too difficult.

8. MarcoFalke added this to the milestone 0.16.0 on Jul 31, 2017
9. jnewbery force-pushed on Aug 14, 2017
10. jnewbery force-pushed on Aug 14, 2017
11. jnewbery force-pushed on Sep 1, 2017
12. jnewbery force-pushed on Sep 1, 2017
13. jnewbery force-pushed on Sep 12, 2017
14. 
    jnewbery commented at 6:25 pm on September 12, 2017: member

    rebased
15. jnewbery force-pushed on Sep 14, 2017
16. 
    jnewbery commented at 3:04 pm on September 14, 2017: member

    rebased
17. 
    MarcoFalke commented at 6:53 pm on October 2, 2017: member

    Needs rebase
18. jnewbery force-pushed on Oct 2, 2017
19. jnewbery force-pushed on Oct 2, 2017
20. 
    jnewbery commented at 8:34 pm on October 2, 2017: member

    rebased
21. 
    MarcoFalke commented at 12:47 pm on October 5, 2017: member

    Needs silent merge conflicts solved.
22. [tests] remove direct testing on JSONRPCException from individual test cases 5864e9c161
23. [tests] do not allow assert_raises_message to be called with JSONRPCException 677d893ff7
24. scripted-diff: rename assert_raises_jsonrpc to assert_raises_rpc error

    -BEGIN VERIFY SCRIPT-
    sed -i 's/assert_raises_jsonrpc/assert_raises_rpc_error/g' test/functional/*py test/functional/test_framework/*py
    -END VERIFY SCRIPT-

    47ba8cf71e
25. jnewbery force-pushed on Oct 5, 2017
26. 
    jnewbery commented at 1:59 pm on October 5, 2017: member

    Thanks. scripted-diff rerun and repushed.
27. 
    MarcoFalke commented at 6:47 pm on October 9, 2017: member

    utACK 47ba8cf71e9a3a6f7452488fc9ab3b350f0fae36
28. MarcoFalke merged this on Oct 9, 2017
29. MarcoFalke closed this on Oct 9, 2017

    ---

30. MarcoFalke referenced this in commit c63364610f on Oct 9, 2017
31. jnewbery deleted the branch on Oct 9, 2017
32. MarcoFalke referenced this in commit 5b728c8e98 on Nov 1, 2017
33. MarcoFalke referenced this in commit 623de0acbb on Nov 1, 2017
34. MarcoFalke referenced this in commit d94fc336c4 on Nov 1, 2017
35. DrahtBot locked this on Sep 8, 2021

Contributors
jnewbery MarcoFalke

Labels
Tests

Milestone
0.16.0