Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot perform a listdir command to a recreated volume #29

Open
iychoi opened this issue Jun 7, 2017 · 0 comments
Open

Cannot perform a listdir command to a recreated volume #29

iychoi opened this issue Jun 7, 2017 · 0 comments
Labels
bug

Comments

@iychoi
Copy link
Member

iychoi commented Jun 7, 2017

Cannot perform a listdir command to a recreated volume. I observed the issue while running syndicatefs.
The volume I tried to mount was recreated one. It was created then deleted. I guess duplicated root directories caused the issue.


6017248.263945167 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.263975146 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from libsyndicate.cpp:2390)
6017248.263991273 00584:00007f27cffff700: [libsyndicate.cpp:2395] ms_entry_verify: DEBUG: Check signature of B9962681FC303901 from gateway 7092660688083880344
6017248.264019963 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:822)
6017248.264039432 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:835)
6017248.264115492 00584:00007f27cffff700: [libsyndicate.cpp:2430] ms_entry_verify: DEBUG: Verify:
type: 1
file_id: 13372918483246921985
ctime_sec: 1496814636
ctime_nsec: 13782273
mtime_sec: 1496814636
mtime_nsec: 13782273
manifest_mtime_sec: 1496814636
manifest_mtime_nsec: 13782433
owner: 1215882033360548424
coordinator: 7092660688083880344
volume: 8593192222588951184
mode: 493
size: 14232283
version: 1
max_read_freshness: 300000
max_write_freshness: 300000
name: "POV_L.Spr.I.2000m_reads.fa"
write_nonce: 1
xattr_nonce: 0
generation: 1
signature: "LYXpJk3siEFBIgHwTe3SXGV6B73JPPauX57pU8AmGNiwTtut7ssV2Nv/NwmkUl/CGfJcyrtPqNSQk7bkawksYb53t1D4KGFfxxGfavvRkUUvgrHDJLZMej5LsKQpaOaWQN0ww0RwmM2NG4iATgvJi4uAnFWo9ImXgNsZM/fi0q19ByJo1c+Q/odYTPSUfKzQQmeF2Tvuien+l51oHSMhsGcTOJWTHQ1mL6ZNqS0mpOWFg0/DhiGs1lOdJimj8k1GiMg4ZCnPng3jLpXjKJC3oHxf8v80X2xkiXEwkBWqGcQ1yd9ccWVtCrG5dcDxPNsWNMowL5jo27k3WzPa9jwo+pg5PeDpGZURwrOW6GaCQoPF8ZSpPMcjMbm/vM7vLwLS7poZZ8sJWGPPMOVTG6FXfk7AUol7Qh/rmJ8s72NX549tQxObcNHsboOI9fdd+Xg9Wa1ufkRa9zDZ2s5hdNozGECPEYiDIgVkNIKy7LWgPn20wrAz1Db21BO/hpRslTyeNgkpYNYgz36A/cqMLfA1tAbhlcVSQgLRbNkq+UOZGeT6sNDurI9MZKyFfaOCNyoJ3MaeghDsLhYi6oFEWeb4EXis8swlksM/kojKvIqZ4cRq/OYemHYzvXk/MqPMm54NBiQHEMhJHSOo/daiTX5DB9Uxis8fIeIoREoe5hbxciI="
parent_id: 0
num_children: 0
capacity: 16

6017248.264968963 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.265004214 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from libsyndicate.cpp:2390)
6017248.265021223 00584:00007f27cffff700: [libsyndicate.cpp:2395] ms_entry_verify: DEBUG: Check signature of 732B1409ACCC923C from gateway 7092660688083880344
6017248.265036391 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:822)
6017248.265052639 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:835)
6017248.265127246 00584:00007f27cffff700: [libsyndicate.cpp:2430] ms_entry_verify: DEBUG: Verify:
type: 1
file_id: 8298748770146619964
ctime_sec: 1496814638
ctime_nsec: 997577324
mtime_sec: 1496814638
mtime_nsec: 997577324
manifest_mtime_sec: 1496814638
manifest_mtime_nsec: 997577637
owner: 1215882033360548424
coordinator: 7092660688083880344
volume: 8593192222588951184
mode: 493
size: 21844560
version: 1
max_read_freshness: 300000
max_write_freshness: 300000
name: "POV_L.Spr.I.500m_reads.fa"
write_nonce: 1
xattr_nonce: 0
generation: 1
signature: "SyJJPReNrNJoMjAdG3L8a6PEqtXM/TzHpefm2WUXav8mB+Q5Z8aFAd7ni0KYjX1pPxZsT2m5NGOYaXfZ9jowhlpJCXwMHrvVbx7OUtTC92gBvi+kt1kgDXI1dIxlRPJ2ngIu1agfgYpxDMLoLKdbGC7Qi8Acruhp5TVIfjilMIkPNdbX1XekVYAb6WFh/6fRTmIkAz7qPQMZyEAlDDTCrrxmcvfhS38YribzLugHd+aWfglNX9+G8Tkq6ti8Jvhq4FEYqdokPMXNrifNhF7nRol26mf/q5mP38CemDVQfAP7LuBup2yuv56LIm5grOi39DxRIJXXxCT38LoaRNLhU0e1fUaV6QUGWrKlrPXGm1KdIhfasHqmClkWXRjiMGVdx0Sm8h+tXKV3hzaa87FKXiaNDV36wxkaxq1l0zE3qfi9ZMOgRo7aZKiLFsLzV/OZDYDVMd2KR3ELyWcXwS4RgJrNgqnl4o8kSHXFDWrmxRaVsc7msIYqm3wOUlzfp3qe/ulh9yUWijxJRBh2HEeIj1bt8vrYXsEu4epC5W2k2QEQECfZrVei3+a6QNXlbqmCxKu3YHr7fOEQVSHZ7G5gMrwBbrjYuxVhQuDs4SXRmnswajao1wVneBXRie33LrAnxvK9sDmAqaU61pxVmipb8g4dpQyHloD15io8PWg4tUk="
parent_id: 0
num_children: 0
capacity: 16

6017248.265955811 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.266010280 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003290 ref 0 (from download.cpp:917)
6017248.266029766 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003290, from download.cpp:920 (refcount 0)
6017248.266071914 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: DEC73519649A5F27
6017248.266118960 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 679551BC1A676707
6017248.266141230 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: AD902D7A83DE15D5
6017248.266161459 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 4B39F96631B6E11B
6017248.266182488 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 97369BC237B0D261
6017248.266214852 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 9440B9648343B033
6017248.266236831 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: B9CCA73717C50FFA
6017248.266257012 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 2C2DEF886F78F2B
6017248.266277625 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: B9962681FC303901
6017248.266298548 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 732B1409ACCC923C
6017248.266320434 00584:00007f27cffff700: [  ms/listdir.cpp:0453] ms_client_get_dir_metadata: DEBUG: Fetched 10 (20 downloaded total)
6017248.266338975 00584:00007f27cffff700: [  ms/listdir.cpp:0459] ms_client_get_dir_metadata: DEBUG: Downloaded 20 children (29 given by inode); l.u.g. is now 14968146718936
6017248.266364040 00584:00007f27cffff700: [  ms/listdir.cpp:0417] ms_client_get_dir_metadata: DEBUG: Finished 1 downloads (rc = -11)
6017248.266383242 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:639)
6017248.266399300 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:643)
6017248.266414850 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:629)
6017248.266429524 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:632)
6017248.266444156 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:619)
6017248.266458641 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:622)
6017248.266528761 00584:00007f27cffff700: [    download.cpp:0753] md_download_context_init: DEBUG: Initialize download context 0x7f27c8003290
6017248.266564990 00584:00007f27cffff700: [    download.cpp:1108] md_download_set_add: DEBUG: Add download context 0x7f27c8003290 to download set 0x7f27c8002f28
6017248.266583253 00584:00007f27cffff700: [    download.cpp:0881] md_download_context_ref2: DEBUG: download 0x7f27c8003290 ref 1 (from download.cpp:1181)
6017248.266602803 00584:00007f27cffff700: [    download.cpp:0418] md_downloader_insert_pending: DEBUG: download 0x7f27c8003290 ref 2
6017248.266618832 00584:00007f27cffff700: [    download.cpp:0427] md_downloader_insert_pending: DEBUG: Start download context 0x7f27c8003290
6017248.266634928 00584:00007f27cffff700: [    download.cpp:1037] md_download_context_wait_any: DEBUG: Wait on download set 0x7f27c8002f28 (3 contexts)
* Found bundle for host syndicate-ms-datasets.appspot.com: 0x7f27d0001930
* Re-using existing connection! (#0) with host syndicate-ms-datasets.appspot.com
* Connected to syndicate-ms-datasets.appspot.com (172.217.2.244) port 80 (#0)
> GET /FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936 HTTP/1.1

User-Agent: Syndicate-Gateway/1.0

Host: syndicate-ms-datasets.appspot.com

Accept: */*



< HTTP/1.1 500 Internal Server Error

< Cache-Control: no-cache

< Content-Type: text/plain

< X-Cloud-Trace-Context: 61888446c0f04ec45ee66a0698112996;o=1

< Date: Wed, 07 Jun 2017 07:42:29 GMT

* Server Google Frontend is not blacklisted
< Server: Google Frontend

< Content-Length: 0

< 

* Connection #1 to host syndicate-ms-datasets.appspot.com left intact
6017248.336420057 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c8003340 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936)
6017248.336450543 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c8003340 ref 1
6017248.336491408 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.336570261 00584:00007f27cffff700: [  ms/listdir.cpp:0185] ms_client_get_dir_metadata_end: ERROR: ms_client_download_parse_errors( 0x7f27c8003340 ) rc = -121
6017248.336630336 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003340 ref 0 (from download.cpp:917)
6017248.336648097 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003340, from download.cpp:920 (refcount 0)
6017248.336683921 00584:00007f27cffff700: [  ms/listdir.cpp:0430] ms_client_get_dir_metadata: ERROR: ms_client_get_dir_metadata_end rc = -121
6017248.336699433 00584:00007f27cffff700: [  ms/listdir.cpp:0467] ms_client_get_dir_metadata: DEBUG: Breaking loop on rc = -121
6017248.336731114 00584:00007f27cffff700: [    download.cpp:0443] md_downloader_insert_cancelling: DEBUG: Cancel download context 0x7f27c8003290
6017248.336757865 00584:00007f27cffff700: [    download.cpp:0484] md_downloader_insert_cancelling: DEBUG: download 0x7f27c8003290 ref 3
6017248.336776701 00584:00007f27cffff700: [    download.cpp:1233] md_download_context_cancel: DEBUG: Wait for cancelling download 0x7f27c8003290 (set 0x7f27c8002f28)
6017248.336795219 00584:00007f27cffff700: [    download.cpp:1011] md_download_context_wait: DEBUG: Wait on download context 0x7f27c8003290 (18446744073709551615 millis)
* Closing connection 0
6017248.341727263 00584:00007f27d7dc9700: [    download.cpp:0618] md_downloader_end_all_cancelling: DEBUG: download 0x7f27c8003290 ref 2
6017248.341799253 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c8003290 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936)
6017248.341817500 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c8003290 ref 1
6017248.341831904 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.341863707 00584:00007f27cffff700: [    download.cpp:1241] md_download_context_cancel: DEBUG: cancelled 0x7f27c8003290
6017248.341919451 00584:00007f27cffff700: [    download.cpp:0443] md_downloader_insert_cancelling: DEBUG: Cancel download context 0x7f27c80033f0
6017248.341944955 00584:00007f27cffff700: [    download.cpp:0484] md_downloader_insert_cancelling: DEBUG: download 0x7f27c80033f0 ref 3
6017248.341961165 00584:00007f27cffff700: [    download.cpp:1233] md_download_context_cancel: DEBUG: Wait for cancelling download 0x7f27c80033f0 (set 0x7f27c8002f28)
6017248.341976331 00584:00007f27cffff700: [    download.cpp:1011] md_download_context_wait: DEBUG: Wait on download context 0x7f27c80033f0 (18446744073709551615 millis)
* Closing connection 2
6017248.347023387 00584:00007f27d7dc9700: [    download.cpp:0618] md_downloader_end_all_cancelling: DEBUG: download 0x7f27c80033f0 ref 2
6017248.347059215 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c80033f0 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?page_id=2)
6017248.347081636 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c80033f0 ref 1
6017248.347096141 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.347146344 00584:00007f27cffff700: [    download.cpp:1241] md_download_context_cancel: DEBUG: cancelled 0x7f27c80033f0
6017248.347210963 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003290 ref 0 (from download.cpp:917)
6017248.347228096 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003290, from download.cpp:920 (refcount 0)
6017248.347280436 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c80033f0 ref 0 (from download.cpp:917)
6017248.347297703 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c80033f0, from download.cpp:920 (refcount 0)
6017248.347323661 00584:00007f27cffff700: [    download.cpp:2115] md_download_loop_cleanup: DEBUG: Clean up download loop 0x7f27c8002f10 (set 0x7f27c8002f28)
6017248.347339828 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c8003290 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.347365402 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c8003340 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
...
6017248.349023811 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c80075f0 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.349039730 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c80076a0 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.349078149 00584:00007f27cffff700: [    download.cpp:1074] md_download_set_free: DEBUG: Free download set 0x7f27c8002f28
6017248.349095154 00584:00007f27cffff700: [  ms/listdir.cpp:0514] ms_client_get_dir_metadata: DEBUG: Downloaded 20 children (out of 29)
6017248.349183568 00584:00007f27cffff700: [ consistency.cpp:2125] UG_consistency_dir_ensure_fresh: ERROR: ms_client_listdir('/') rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from path.c:825
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from path.c:855
6017248.349255956 00584:00007f27cffff700: [          fs.cpp:0296] UG_fs_open: ERROR: UG_consistency_dir_ensure_fresh('/') rc = -121
00584:00007F27CFFFF700: [fskit          route.c:0640] fskit_route_dispatch ERROR: fskit_safe_dispatch(3) rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from opendir.c:98
00584:00007F27CFFFF700: [fskit        opendir.c:0103] fskit_opendir ERROR: fskit_run_user_open(/) rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from opendir.c:104
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from path.c:825
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from path.c:855
00584:00007F27CFFFF700: [fskit     fskit_fuse.c:0671] fskit_fuse_opendir DEBUG: opendir(/, 0x7f27cfffec10) rc = -121
@iychoi iychoi added the bug label Jun 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@iychoi