Indexer: Add Indexer and Hasura GraphQl to movement #295

musitdev · 2024-08-02T15:53:29Z

Summary

RFCs: Link to RFC, Link to RFC, or $\emptyset$.
Categories: any of protocol-units, networks, scripts, util, cicd, or misc.

Add the indexer and Hasura GraphQl engine to process compose.

This PR depends on these one:

Indexer: Remove all unwrap and panic for missing Tx WriteSet data. aptos-core#53
Indexer: Update to be comptible with movement's Aptos-core aptos-indexer-processors#2

Changelog

Create an Indexer crate under network/suzuka.
Update process compose to run Indexer and Hasura.
Remove the indexer started with Suzuka node.
Create 2 process-compose script: indexer and Hasura
The Postgres Db is now started by the indexer and not Suzuka-node.

Testing

If there's some issue for building (dieser error) you need to clean the repo with cargo clean
If you have the issue with fixed crate that need rustc version 1.79.0 execute this command: cargo update -p fixed --precise 1.27.0
To run everything:

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just suzuka-full-node native build.setup.eth-local.celestia-local.indexer.hasura --keep-tui"

When the Hasura server is started, open a browser at this url: http://localhost:8085/console
To add the Suzuka Tx db to Hasura:

go in Data (top of the windows)
Connect database
Connect existing database
Put a name to Database name
Select database Url
set Database URL to postgresql://postgres:password@database:5432/postgres
Validate with connect dabase.
Under Datamanager select your database-> public
All the indexer table appears. Select track on the transactions table for example.
Go under API
Type this query

query MyQuery {
 transactions {
   hash
   success
   type
   version
 }
}

To see all Tx in the Db.

To run the indexer e2e test:

CELESTIA_LOG_LEVEL=FATAL nix develop --extra-experimental-features nix-command --extra-experimental-features flakes --command bash  -c "just suzuka-full-node native build.setup.eth-local.celestia-local.indexer.hasura.indexer-test --keep-tui"

Outstanding issues

I start only one indexer processor: dedfault_processor. I'm going to see how to start more processor to populate more data in the db.
Available processors are:

    AccountTransactionsProcessor,
    AnsProcessor(AnsProcessorConfig),
    CoinProcessor,
    DefaultProcessor,
    EventsProcessor,
    FungibleAssetProcessor,
    MonitoringProcessor,
    NftMetadataProcessor(NftMetadataProcessorConfig),
    ObjectsProcessor(ObjectsProcessorConfig),
    StakeProcessor(StakeProcessorConfig),
    TokenProcessor(TokenProcessorConfig),
    TokenV2Processor(TokenV2ProcessorConfig),
    TransactionMetadataProcessor,
    UserTransactionProcessor,
    ParquetDefaultProcessor(ParquetDefaultProcessorConfig),
    ParquetFungibleAssetProcessor(ParquetFungibleAssetProcessorConfig),

l-monninger · 2024-08-02T16:44:19Z

May need: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/libelf/default.nix

l-monninger · 2024-08-02T17:00:37Z

docker/compose/indexer/docker-compose.hasura.yml

+ image: hasura/graphql-data-connector:v2.40.0
+ restart: always
+ ports:
+ - 8081:8081


process-compose/suzuka-full-node/process-compose.hasura.yml

l-monninger · 2024-08-02T17:02:10Z

process-compose/suzuka-full-node/process-compose.yml

@@ -40,15 +40,6 @@ processes:
 initial_delay_seconds: 3
 exec:
 command: grpcurl -plaintext 0.0.0.0:30730 list


I don't see why the test of the grpc port of the lightnode should be removed?

I was trying to comment on the removals below.

Below it's the suzuka-full-node entry. I still don't see.
I see there's a postgres entry in the suzuka-full-node/docker-compose.yml. Postgres is not needed for the Suzuka-full-node. Perhaps it should be removed and added to the indexer docker-compose or just removed. Depends on how postgres db is managed.

scripts/movement/run

l-monninger · 2024-08-02T17:02:58Z

scripts/postgres/start-dev

@@ -7,6 +7,10 @@ rm -rf ./.data
 # Initialize the database cluster
 initdb -D ./.data --no-locale

+# allow docker connection to the db.
+sed -i "/listen_addresses/c\listen_addresses = \'*\'" ./.data/postgresql.conf


Nvm, getting invalid command code .

sed 1: "./.data/postgresql.conf": invalid command code .

I correct Mac specific errors. I should work now on mac.

l-monninger · 2024-08-02T17:04:42Z

protocol-units/execution/opt-executor/src/executor/indexer.rs

@@ -31,24 +28,23 @@ impl Executor {
 self.maptos_config.chain.maptos_chain_id.clone(),


We should probably add parameters to the config to turn this on and off.

I don't understand what to turn on/off? The chain_id is mandatory.

Sorry, it looks like I didn't select the range properly. I meant the run_indexer_grpc_service.

Yes run_indexer_grpc_service is only needed for the indexer. It's part of the Executor config. I don't see how to turn it off. Perhaps I should move it to a specific indexer config struct that is only defined for the indexer process?

l-monninger · 2024-08-02T17:06:15Z

networks/suzuka/indexer/src/main.rs

+ // };
+
+ //create config file
+ let indexer_config_content = format!(


l-monninger

Overall, looks very nice! But, I need a fix to run a play around with.

musitdev · 2024-08-06T13:50:00Z

I've added an e2e test. For Indexer, I query the DB to see if there's some Tx. For Hasura, I just test if the console is available. I do some search to see how to configure Hasura and declare the indexer table to be able to graphQl query the transaction table, but I didn't see an obvious way, so I test the DB and the presence of Hasura.

l-monninger · 2024-08-07T08:29:58Z

process-compose/suzuka-full-node/process-compose.indexer-test.yml

+
+ indexer-test:
+ command: |
+ sleep 5


Reasonable for now.

l-monninger · 2024-08-07T08:30:34Z

process-compose/suzuka-full-node/process-compose.yml

@@ -40,15 +40,6 @@ processes:
 initial_delay_seconds: 3
 exec:
 command: grpcurl -plaintext 0.0.0.0:30730 list


I was trying to comment on the removals below.

l-monninger · 2024-08-07T08:31:20Z

protocol-units/execution/opt-executor/src/executor/indexer.rs

@@ -31,24 +28,23 @@ impl Executor {
 self.maptos_config.chain.maptos_chain_id.clone(),


Sorry, it looks like I didn't select the range properly. I meant the run_indexer_grpc_service.

l-monninger · 2024-08-07T09:38:49Z

docker/compose/indexer/docker-compose.hasura.yml

+ ports:
+ - "8085:8085"
+ restart: always
+ environment:


So, does the existing Docker compose work for the rest of the indexer. That is, as long as we provide the same postgres connection here and there, are we good? Or, are we still missing the indexer-processor?

I think it's better to provide 2 docker scripts, one for Hasura and one for the indexer. I can use the same Ip env variable for the indexer docker script.

l-monninger · 2024-08-07T11:24:11Z

@musitdev it looks like the indexer test has a legitimate error: https://github.com/movementlabsxyz/movement/actions/runs/10267756524/job/28452717854?pr=295

musitdev · 2024-08-07T12:43:51Z

Ah yes, I didn't see. It's caused because Postgres db is not started, but I don't see why. The indexer depends on it in process-compose, but I don't see the logs of its start. I try to restart it.

musitdev · 2024-08-07T15:30:10Z

The test pass after restart.

musitdev added 2 commits August 1, 2024 10:08

Remove indexer from Suzuka node. Keep gRpc stream

8e75b17

Add indexer and Hasura to movement

05c6726

musitdev requested review from l-monninger, andyjsbell, 0xmovses and mzabaluev as code owners August 2, 2024 15:53

fix: try libelf.

18dc50f

l-monninger reviewed Aug 2, 2024

View reviewed changes

process-compose/suzuka-full-node/process-compose.hasura.yml Outdated Show resolved Hide resolved

l-monninger reviewed Aug 2, 2024

View reviewed changes

scripts/movement/run Outdated Show resolved Hide resolved

l-monninger reviewed Aug 2, 2024

View reviewed changes

networks/suzuka/indexer/src/main.rs

// };

//create config file

let indexer_config_content = format!(

Copy link

Collaborator

l-monninger Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, nice

l-monninger requested changes Aug 2, 2024

View reviewed changes

musitdev added 3 commits August 5, 2024 14:24

Correct sed compatibility issue between linux and mac

8bfa89c

correct Mac compatibility issue with linux cmd

7906f0c

add e2e test of indexer and hasura

c103916

l-monninger requested changes Aug 7, 2024

View reviewed changes

musitdev and others added 6 commits August 7, 2024 17:45

develop indexer docker script

89586b1

Correct and test docker compose script

8dcdf14

remove unnecessary var definition

674b9b7

Remove indexer from Suzuka node. Keep gRpc stream

2a21125

Add indexer and Hasura to movement

4aed0f2

fix: try libelf.

ac37e19

musitdev and others added 24 commits August 21, 2024 13:47

Change postgres parameters

f502a68

Add some log in postgres start

7ac38c3

Change pg_ctl option

8ec3398

do not use hard coded IP for postgres in docker compose setup

6c58565

clean some

fe25aef

more cleanup

3ffe589

try to make a flexible config for process compose and docker compose

2e086fa

update container version and flexible config

dc6ee6c

json fix syntax

eedaab4

correct process compose and postgres execution

4600e16

correct indexer test script issue

f568034

update compose indexer for prod

bcd4dbe

fix: toolchain.

2276d82

correct test script

dcda605

correct Aptos rev issue

cb53f90

update to latest changes of indexer and update docs

634c663

add docs on how to connect local indexer to remote suzuka grpc node

6d39d18

add persistent storage for postgres db

1c43a9f

update aptos indexer rev for Tx skip

e3e8a0c

change process compose health timing

cc32688

rebase main and desactivate indexer test in CI

3498574

correct conflic after pull

9966bb0

correct main merge conflict

b74428b

desactivate suzuka local indexer start

1f61d49

radupopa369 approved these changes Aug 21, 2024

View reviewed changes

musitdev added 2 commits August 21, 2024 15:59

reactivate the indexer

d760e9d

Correct test

08deab5

radupopa369 merged commit f308ed2 into main Aug 21, 2024
29 checks passed

musitdev mentioned this pull request Aug 30, 2024

Factor indexer support out of maptos-opt-executor #241

Open

musitdev deleted the musitdev/activate_indexer branch September 13, 2024 11:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indexer: Add Indexer and Hasura GraphQl to movement #295

Indexer: Add Indexer and Hasura GraphQl to movement #295

musitdev commented Aug 2, 2024 •

edited

Loading

l-monninger commented Aug 2, 2024

l-monninger Aug 2, 2024

l-monninger Aug 2, 2024

musitdev Aug 6, 2024

l-monninger Aug 7, 2024

musitdev Aug 8, 2024 •

edited

Loading

l-monninger Aug 2, 2024

l-monninger Aug 2, 2024

l-monninger Aug 2, 2024

musitdev Aug 6, 2024

l-monninger Aug 2, 2024

musitdev Aug 6, 2024

l-monninger Aug 7, 2024

musitdev Aug 8, 2024

l-monninger Aug 2, 2024

l-monninger left a comment

musitdev commented Aug 6, 2024

l-monninger Aug 7, 2024

l-monninger Aug 7, 2024

l-monninger Aug 7, 2024

l-monninger Aug 7, 2024

musitdev Aug 7, 2024

l-monninger commented Aug 7, 2024

musitdev commented Aug 7, 2024

musitdev commented Aug 7, 2024

		@@ -31,24 +28,23 @@ impl Executor {
		self.maptos_config.chain.maptos_chain_id.clone(),

Indexer: Add Indexer and Hasura GraphQl to movement #295

Indexer: Add Indexer and Hasura GraphQl to movement #295

Conversation

musitdev commented Aug 2, 2024 • edited Loading

Summary

Changelog

Testing

Outstanding issues

l-monninger commented Aug 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

musitdev Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

l-monninger left a comment

Choose a reason for hiding this comment

musitdev commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

l-monninger commented Aug 7, 2024

musitdev commented Aug 7, 2024

musitdev commented Aug 7, 2024

musitdev commented Aug 2, 2024 •

edited

Loading

musitdev Aug 8, 2024 •

edited

Loading