This blog will serve as a project report for my Summer of Bitcoin project with LDK. My project focuses on integrating LDK which is lightning network implementation with the Lnprototest test suite.

Lnprototest

Lnprototest is a set of test helpers written in Python3, designed to make it easy to write new tests when you propose changes to the lightning network protocol, as well as test existing implementations.

Tests are simply a list of Events, like Msg to send a message and ExpectMsg to receive a response. But the tool goes much further: TryAll indicates multiple different variants should all be attempted, such as running the test with different features or trying reconnection in the middle of the test. It also has Events to create commitment transactions, add and remove HTLCs, and other things you’d expect.

Testers can also write their own custom Events for more specialized testing and Events can be conditional on what features were negotiated or other runtime behavior.

LDK

Lightning Development Kit is a library that allows you to build a lightning node without worrying about implementing low-level lightning logic correctly.

LDK Logo



LDK is based on Rust-Lightning, a full-featured but also incredibly flexible lightning implementation designed to be modular and customizable. It allows you to build a lightning node without having to worry about all of the low-level lightning logic like the state machine or routing.

The project

In order to integrate LDK with Lnprototest, few things should be kept in mind.

  • LDK is not a node, its a lightning implementation that can be used to create a node.
  • In order to integrate with Lnprototest, we need a node implemented using LDK.
  • For these reasons, I will be using LDK-Sample.

LDK-Sample is a sample node implementation which uses LDK.

In order to integrate it with Lnprototest, some changes are required in LDK-Sample and a ldk.py script is needed in Lnprototest. This script will contain a ldk runner class which will act as a middleman between Lnprototest and LDK-Sample.

This class consists of various functions which need to be implemented for ensuring proper functionality.

Progress Made

  • Implementing features.rs in LDK-Sample. This file simply displays the features of LDK-Sample node. These features are then used by the ldk.py script inside Lnprototest.
  • Adding an exit function in LDK-Sample. Having an exit function ensures that the port used by LDK-Sample is unoccupied after the node is exited and can be used for other purposes.
  • Implementing myKeysManager struct in LDK-Sample. This struct manages the node secrets and uses a key_seed to generate different keys. The advantage of using this instead of the default KeysManager is that it allows me to use my own set of channel secrets(funding_key, revocation_base_key, payment_key, delayed_payment_base_key, htlc_base_key) instead of the randomly generated ones. This was useful for setting up the configuration required by Lnprototest.
  • Implementing KeysInterface trait for the myKeysManager struct. This was essential for myKeysManager as it has a lot of useful methods.
  • Implementing openchannel_test method in LDK-Sample. This method allows opening a fundchannel without providing the host or port.
  • Adding the ldk.py script inside Lnprototest. This script does all the heavy lifting in the integration process.
  • Implementing the LDKConn class inside the script. This class is used to connect to the node using pyln.proto.wire module and intercept all the messages that the node initiates and receives.
  • Implementing the LDK Runner class. This class contains all the functions used to start/stop bitcoind, start/stop LDK-Sample node, adding a connection, connecting to a peer, carrying out transactions, receiving output messages etc. The tests are run against this class and it basically tells the node to do a particular task(like connecting to a peer) and the LDKConn class intercepts the messages sent by the node. The various methods I have implemented are:
    • __init__
    • __init_sandbox_dir
    • get_keyset
    • get_node_privkey
    • get_node_bitcoinkey
    • is_running
    • start
    • shutdown
    • stop
    • def restart
    • connect
    • getblockheight
    • trim_blocks
    • add_blocks
    • recv
    • fundchannel
    • get_output_message
    • check_error
    • check_final_error
    • expect_tx
    • expect_tx
    • has_option
  • I have also started some testing with the BOLT tests. The LDK implementation follows all the BOLT specifications but BOLT#1 and BOLT#2 tests are failing because of a bug in Lnprototest for which I opened an issue.

I haven’t made any pull requests to the main repos as I would like to finish the entire project before merging it. However, my work can be found in my forked repos. Lnprototest
LDK-Sample

Remaining work

  • Finishing the ldk.py script. There are some functions in the LDK runner class that need to be implemented. These include:
    • init_rbf
    • invoice
    • accept_add_fund
    • addhtlc
  • Testing against the BOLT Specification tests and improving the script as required.

The end goal of this project is to ensure that the script works perfectly and all the tests pass. The LDK community has been really helpful to me during this time. All the developers are friendly and have supported me extensively through my journey. I would like to thank my mentor Conor Okus for always being there for me and guiding me. I would also like to thank Matt Corallo, Jeffrey Czyz and Vincenzo Palazzo for entertaining my silly questions.