TN: Gatling - Support price feeds

[theoreticalbts]

Price feeds must either be ported from the main network #2, or set to a constant value. Even if price feeds are set to a constant value, the witnesses will need to periodically refresh price feeds since they expire. Since price feeds require transactions to be issued during the ongoing life of the network, we can't implement a constant-valued feed by simply adding the feed value to the testnet initialization.

AC

• # Results comment with details of how to set price-feeds on a TN and a link to/from the execution story.

[theoreticalbts]

So I think maybe I need to better explain what needs to happen here.

From the point of view of the backend developer, what needs to happen is very trivial and straightforward: The testnet witnesses need to publish feed_publish_operation, and do so on a regular basis, since too-old values are ignored by the code.

From the point of view of the deployment engineer, what needs to happen is fraught with complexity.

• Who are the witnesses?
• What is the data source for price information?
• What script does the feed_publish_operation creator run?
• How is the feed_publish_operation deployed on a machine?

Who are the witnesses?

The first question that needs to be answered is, who are the witnesses? There is a separate ticket for this #22

Only witnesses can publish a price feed. Ipso facto, if we aren't running the testnet witnesses, price feeds aren't our problem. So in this ticket, I will confine myself to considering testnets where we (Steemit) run the witnesses, option (a) in #22.

What is the data source for the information?

For MVP, the data source can be a constant value supplied as a command line option to the script.

We could also port data from the blockchain, basically have a script that maps witnesses on the main blockchain to testnet witnesses. Then whenever a witness on the main blockchain publishes price_feed_operation, the corresponding testnet witness issues a price_feed_operation with the same value, perhaps with a configurable constant of proportionality. This script would take a blockchain API endpoint and poll interval as command line or config file parameters.

What script does the feed_publish_operation creator run?

In keeping with tinman design principles, tinman should have a script whose CLI is something like tinman pricefeed constant --value=1.25 --realtime-interval=1h, which creates a list of actions on stdout which can be piped to tinman keysub and uses time.sleep(3600) to wait for an hour.

If you want to change the constant, you can simply Ctrl+C the running script and re-run with a different value.

A different approach with a more complicated data flow would be to give the tinman pricefeed script an API endpoint on the command line or in a config file. Then the script could output whenever the existing value is more than halfway to expiring, or whenever the specified value is different from the existing value (or perhaps within some margin of error). A lot of features are possible for this script.

How is the feed_publish_operation node deployed on a machine?

Just install tinman and set up a shell script that does tinman pricefeed | tinman keysub | tinman submit, adding appropriate command-line parameters to each stage's configuration. This node doesn't need to run steemd, all it needs is an API endpoint, so it can use a very lightweight instance. #23 is relevant here.

[inertia186]

Looks like this might come in handy after all:

#92 (comment)

... perhaps with a configurable constant of proportionality ...

Emphasis mine

[relativityboy]

We've decided to make this story a spike, to iron out what the AC should actually be.

[inertia186]

As mentioned, one option is to use mainnet as the source of the price feeds, replacing the price with a proportional price and the witness with a testnet witness. Say we just run tinman gatling -o - with the following config:

{
 "transaction_signer" : "tnman",

 "transaction_source" :
 {
  "node" : "https://api.steemit.com",
  "appbase" : "True"
 },

 "min_block_number" : "0",
 "max_block_number" : "0",

 "ported_operations" :
 [
  {"type":"feed_publish_operation","roles":["active"]}
 ]
}

What we're saying here is, just port feed_publish_operation. The results I get are:

["submit_transaction",{"esc":"z","tx":{"operations":[{"type":"feed_publish_operation","value":{"exchange_rate":{"base":{"amount":"1023","nai":"@@000000013","precision":3},"quote":{"amount":"1000","nai":"@@000000021","precision":3}},"publisher":"jrswab"}}],"wif_sigs":["zprivatekey:active-tnmanz"]}}]
["submit_transaction",{"esc":"j","tx":{"operations":[{"type":"feed_publish_operation","value":{"exchange_rate":{"base":{"amount":"1023","nai":"@@000000013","precision":3},"quote":{"amount":"1000","nai":"@@000000021","precision":3}},"publisher":"steempress"}}],"wif_sigs":["jprivatekey:active-tnmanj"]}}]
["submit_transaction",{"esc":"j","tx":{"operations":[{"type":"feed_publish_operation","value":{"exchange_rate":{"base":{"amount":"1021","nai":"@@000000013","precision":3},"quote":{"amount":"1000","nai":"@@000000021","precision":3}},"publisher":"blockbrothers"}}],"wif_sigs":["jprivatekey:active-tnmanj"]}}]
["submit_transaction",{"esc":"j","tx":{"operations":[{"type":"feed_publish_operation","value":{"exchange_rate":{"base":{"amount":"1023","nai":"@@000000013","precision":3},"quote":{"amount":"1000","nai":"@@000000021","precision":3}},"publisher":"windforce"}}],"wif_sigs":["jprivatekey:active-tnmanj"]}}]
["submit_transaction",{"esc":"j","tx":{"operations":[{"type":"feed_publish_operation","value":{"exchange_rate":{"base":{"amount":"1026","nai":"@@000000013","precision":3},"quote":{"amount":"1000","nai":"@@000000021","precision":3}},"publisher":"thecryptodrive"}}],"wif_sigs":["jprivatekey:active-tnmanj"]}}]

If we were to pipe these results through tinman amountsub --floor-satoshi 1 --ratio -0.187, we would get:

["submit_transaction", {"esc": "z", "tx": {"operations": [{"type": "feed_publish_operation", "value": {"exchange_rate": {"base": {"amount": "191", "nai": "@@000000013", "precision": 3}, "quote": {"amount": "187", "nai": "@@000000021", "precision": 3}}, "publisher": "jrswab"}}], "wif_sigs": ["zprivatekey:active-tnmanz"]}}]
["submit_transaction", {"esc": "j", "tx": {"operations": [{"type": "feed_publish_operation", "value": {"exchange_rate": {"base": {"amount": "191", "nai": "@@000000013", "precision": 3}, "quote": {"amount": "187", "nai": "@@000000021", "precision": 3}}, "publisher": "steempress"}}], "wif_sigs": ["jprivatekey:active-tnmanj"]}}]
["submit_transaction", {"esc": "j", "tx": {"operations": [{"type": "feed_publish_operation", "value": {"exchange_rate": {"base": {"amount": "190", "nai": "@@000000013", "precision": 3}, "quote": {"amount": "187", "nai": "@@000000021", "precision": 3}}, "publisher": "blockbrothers"}}], "wif_sigs": ["jprivatekey:active-tnmanj"]}}]
["submit_transaction", {"esc": "j", "tx": {"operations": [{"type": "feed_publish_operation", "value": {"exchange_rate": {"base": {"amount": "191", "nai": "@@000000013", "precision": 3}, "quote": {"amount": "187", "nai": "@@000000021", "precision": 3}}, "publisher": "windforce"}}], "wif_sigs": ["jprivatekey:active-tnmanj"]}}]
["submit_transaction", {"esc": "j", "tx": {"operations": [{"type": "feed_publish_operation", "value": {"exchange_rate": {"base": {"amount": "191", "nai": "@@000000013", "precision": 3}, "quote": {"amount": "187", "nai": "@@000000021", "precision": 3}}, "publisher": "thecryptodrive"}}], "wif_sigs": ["jprivatekey:active-tnmanj"]}}]

Next, we need a way to re-write the which witness. For this, I think gatling could be enhanced. We could add a replace_witness option to ported_operations of the .conf, such as:

{
 "ported_operations" :
 [
  {"type":"feed_publish_operation","roles":["active"],"replace_witness":true}
 ]
}

Or something similar. When gatling sees an operation that has this option set to true, it will look at a list of witnesses (also in .conf) to pick randomly.

[relativityboy]

We want to be able to disconnect Testnet from Mainnet. Having Gatling pipe from Mainnet is interesting and perhaps an option to support, but we must also support an artificial price feed that can run "in a vacuum".

Moreover, since we need to support a stand alone Testnet, perhaps the price feed utility should be written so that it can be run separate from Gatling.

[inertia186]

Well, separate from gatling can just mean that we have an actions file with a single price feed transaction (or multiple from various producers) that we just pipe to the testnet, which yes, is a lot more simple.

[relativityboy]

Why an . actions file specifically?

Is your thought to wrap submit in a wrapper that loops with a configurable sleep & authentication?

[inertia186]

Correct.

[relativityboy]

Results

For MVP on price feed, we'll create a module that can be invoked on it's own, or as a flag within gatling.

There is an assumption made that a valid signing key for all accounts will be provided to the price feed as needed.

The behavior will be customizable in the following ways.

1. Price feed "interval" will be a param or configuration
2. The actions file location will be a param or configuration, but will have a default location, and default file that can be used
3. there will be a param or configuration to use actions file accounts as-stated, or to cycle through existing witnesses.