Skip to content

DeFiChain (DFI) Rosetta API Implementation

License

Notifications You must be signed in to change notification settings

DeFiCh/rosetta-defichain

 
 

Rosetta

Rosetta DeFiChain

ROSETTA-DEFICHAIN IS CONSIDERED ALPHA SOFTWARE. USE AT YOUR OWN RISK! COINBASE ASSUMES NO RESPONSIBILITY NOR LIABILITY IF THERE IS A BUG IN THIS IMPLEMENTATION.

Overview

rosetta-defichain provides a reference implementation of the Rosetta API for DeFiChain in Golang. If you haven't heard of the Rosetta API, you can find more information here.

Features

  • Rosetta API implementation (both Data API and Construction API)
  • UTXO cache for all accounts (accessible using /account/balance)
  • Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address

Usage

As specified in the Rosetta API Principles, all Rosetta implementations must be deployable via Docker and support running via either an online or offline mode.

YOU MUST INSTALL DOCKER FOR THE FOLLOWING INSTRUCTIONS TO WORK. YOU CAN DOWNLOAD DOCKER HERE.

Install

Running the following commands will create a Docker image called rosetta-defichain:latest.

From Source

After cloning this repository, run:

make build-local

Docker

You also may build your docker image by invoking the following command:

docker build -t rosetta-defichain:latest https://github.com/DeFiCh/rosetta-defichain.git

If you want to build a certain version of the image, you may add tag, branch and other supported references to the end of the command above. On how to to this, please, see this details

Run

Running the following commands will start a Docker container in detached mode with a data directory at <working directory>/defichain-data and the Rosetta API accessible at port 8080.

Mainnet:Online

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/defichain-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 rosetta-defichain:latest

If you cloned the repository, you can run make run-mainnet-online.

Mainnet:Offline

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 rosetta-defichain:latest

If you cloned the repository, you can run make run-mainnet-offline.

Testnet:Online

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/defichain-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 rosetta-defichain:latest

If you cloned the repository, you can run make run-testnet-online.

Testnet:Offline

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 rosetta-defichain:latest

If you cloned the repository, you can run make run-testnet-offline.

General information about ports

  • Online API port is 8080
  • Offline API port is 8081

Requirements

Machine

The following results may be expected by running rosetta-defichain on a machine with characteristics comparable to the following:

  • x8 CPU i5-8250U @ 1.60GHz
  • ~8 GB RAM & ~4-12 SWAP

Mainnet:

  • about 40 GB of disk space usage
  • about 4 hours to sync

Testnet:

  • about 20 GB of disk space usage
  • about 2 hours to sync
NOTE: presented values may change, since blockchain data is growing up continuously

Network Settings

To increase the load rosetta-defichain can handle, it is recommended to tune your OS settings to allow for more connections. On a linux-based OS, you can run the following commands (source):

sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_max_syn_backlog=10000
sysctl -w net.core.somaxconn=10000
sysctl -p (when done)

We have not tested rosetta-defichain with net.ipv4.tcp_tw_recycle and do not recommend enabling it.

You should also modify your open file settings to 100000. This can be done on a linux-based OS with the command: ulimit -n 100000.

Memory-Mapped Files

rosetta-defichain uses memory-mapped files to persist data in the indexer. As a result, you must run rosetta-defichain on a 64-bit architecture.

If you receive a kernel OOM, you may need to increase the allocated size of swap space on your OS. There is a great tutorial for how to do this on Linux here.

Architecture

rosetta-defichain uses the syncer, storage, parser, and server package from rosetta-sdk-go instead of a new DeFiChain-specific implementation of packages of similar functionality. Below you can find a high-level overview of how everything fits together:

                               +------------------------------------------------------------------+
                               |                                                                  |
                               |                 +--------------------------------------+         |
                               |                 |                                      |         |
                               |                 |                 indexer              |         |
                               |                 |                                      |         |
                               |                 | +--------+                           |         |
                               +-------------------+ pruner <----------+                |         |
                               |                 | +--------+          |                |         |
                         +-----v----+            |                     |                |         |
                         |   defid  |            |              +------+--------+       |         |
                         +-----+----+            |     +--------> block_storage <----+  |         |
                               |                 |     |        +---------------+    |  |         |
                               |                 | +---+----+                        |  |         |
                               +-------------------> syncer |                        |  |         |
                                                 | +---+----+                        |  |         |
                                                 |     |        +--------------+     |  |         |
                                                 |     +--------> coin_storage |     |  |         |
                                                 |              +------^-------+     |  |         |
                                                 |                     |             |  |         |
                                                 +--------------------------------------+         |
                                                                       |             |            |
+-------------------------------------------------------------------------------------------+     |
|                                                                      |             |      |     |
|         +------------------------------------------------------------+             |      |     |
|         |                                                                          |      |     |
|         |                     +---------------------+-----------------------+------+      |     |
|         |                     |                     |                       |             |     |
| +-------+---------+   +-------+---------+   +-------+-------+   +-----------+----------+  |     |
| | account_service |   | network_service |   | block_service |   | construction_service +--------+
| +-----------------+   +-----------------+   +---------------+   +----------------------+  |
|                                                                                           |
|                                         server                                            |
|                                                                                           |
+-------------------------------------------------------------------------------------------+

Optimizations

  • Automatically prune defid while indexing blocks
  • Reduce sync time with concurrent block indexing
  • Use Zstandard compression to reduce the size of data stored on disk without needing to write a manual byte-level encoding

Concurrent Block Syncing

To speed up indexing, rosetta-defichain uses concurrent block processing with a "wait free" design (using channels instead of sleeps to signal which threads are unblocked). This allows rosetta-defichain to fetch multiple inputs from disk while it waits for inputs that appeared in recently processed blocks to save to disk.

                                                   +----------+
                                                   |   defid  |
                                                   +-----+----+
                                                         |
                                                         |
          +---------+ fetch block data / unpopulated txs |
          | block 1 <------------------------------------+
          +---------+                                    |
       +-->   tx 1  |                                    |
       |  +---------+                                    |
       |  |   tx 2  |                                    |
       |  +----+----+                                    |
       |       |                                         |
       |       |           +---------+                   |
       |       |           | block 2 <-------------------+
       |       |           +---------+                   |
       |       +----------->   tx 3  +--+                |
       |                   +---------+  |                |
       +------------------->   tx 4  |  |                |
       |                   +---------+  |                |
       |                                |                |
       | retrieve previously synced     |   +---------+  |
       | inputs needed for future       |   | block 3 <--+
       | blocks while waiting for       |   +---------+
       | populated blocks to save to    +--->   tx 5  |
       | disk                               +---------+
       +------------------------------------>   tx 6  |
       |                                    +---------+
       |
       |
+------+--------+
|  coin_storage |
+---------------+

Testing with rosetta-cli

To validate rosetta-defichain, install rosetta-cli and run one of the following commands:

  • rosetta-cli check:data --configuration-file rosetta-cli-conf/testnet/config.json
  • rosetta-cli check:construction --configuration-file rosetta-cli-conf/testnet/config.json
  • rosetta-cli check:data --configuration-file rosetta-cli-conf/mainnet/config.json

Future Work

  • Add CI test using rosetta-cli to run on each PR

Development

  • make deps to install dependencies
  • make test to run tests
  • make lint to lint the source code
  • make salus to check for security concerns
  • make build to build a Docker image from the remote repository
  • make build-local to build a Docker image from the local context
  • make coverage-local to generate a coverage report

License

This project is available open source under the terms of the Apache 2.0 License.

About

DeFiChain (DFI) Rosetta API Implementation

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 95.1%
  • Shell 2.4%
  • Makefile 1.3%
  • Dockerfile 1.2%