Questions about AMReX Grid Creation/Initiation within PeleC

[Nadexterbrown]

Hello, I am looking into imposing a predetermined grid (from a pltFile) onto the current simulation grid for implementing simulation restart and domain expansion capabilities. Currently, numerous problems are occurring with mismatching of the boxes when imposing the pltData onto the AMReX generated grid for levels higher than zero. I was wondering if would be possible to, using AMReX, create a grid using the pltFile grid information? I have already been able to load the pltData and extract the necessary parameters but cannot quite seem to find where the grid is initially defined/created and any direction would be extremely helpful. Thank you for your assistance and have a wonderful day.

Best,
Nolan Dexter

[marchdf]

If I understand your question correctly... You can specify "exactly" how to refine by using a grids file. There is an example in the TG case and it looks like this: https://github.com/AMReX-Combustion/PeleC/blob/development/Exec/RegTests/TG/grids_file. You need to tell the input file to use that amr.initial_grid_file = nameoffile. Hopefully this helps. If not, please provide more information about what you are trying to do (input files, case files, etc) and we can help more.

[Nadexterbrown]

I did not know about this functionality of amr, thank you for bringing this to my attention. I will look into this, as I believe I will need to copy and modify the header file to collect the appropriate grid boxes. I will post back with updates as I make them. Thank you again and have a wonderful day.

[baperry2]

I think domain expansion could end up involving some pretty subtle/tricky features of Pele and AMReX, so I'd definitely encourage you to continue updating on your approach and progress so we can provide pointers.

[Nadexterbrown]

I was wondering if it would be possible to ask for some follow up advice. Currently, I have created a python script to extract the pltFile boxes to ensure the gridfile is exactly what is desired. However, when I go to restart with the required grid, PeleC reads the file but doesn't implement the gridfile. We can see that for my Level 1 grid the imposed (gridfile/pltfile boxes) should be the following:

((1840,0) (1999,7) (0,0))
((2000,0) (2127,7) (0,0))
((2128,0) (2383,7) (0,0))
((2384,0) (2639,7) (0,0))
((2640,0) (2895,7) (0,0))
((2896,0) (3039,7) (0,0))
((3040,0) (3167,7) (0,0))
((3168,0) (3423,7) (0,0))
((3424,0) (3559,7) (0,0))
((3560,0) (3687,7) (0,0))
((3688,0) (3943,7) (0,0))
((1840,8) (1879,15) (0,0))
((1880,8) (1919,23) (0,0))
((1920,8) (2007,31) (0,0))
((2008,8) (2047,55) (0,0))
((2048,8) (2095,103) (0,0))
((3920,8) (3943,111) (0,0))
((2008,80) (2047,111) (0,0))
((1984,88) (2007,111) (0,0))
((1960,96) (1983,111) (0,0))
((1840,104) (1959,111) (0,0))
((1816,112) (1975,127) (0,0))
((1976,112) (2103,127) (0,0))
((2104,112) (2359,127) (0,0))
((2360,112) (2503,127) (0,0))
((2504,112) (2631,127) (0,0))
((2632,112) (2887,127) (0,0))
((2888,112) (3047,127) (0,0))
((3048,112) (3175,127) (0,0))
((3176,112) (3431,127) (0,0))
((3432,112) (3687,127) (0,0))
((3688,112) (3943,127) (0,0))

however, the grid utilized in PeleC.cpp for the PeleC::initData() function somehow has the following grid:

Simulation Box: ((3680,0) (3999,15) (0,0))
Simulation Box: ((4000,0) (4255,15) (0,0))
Simulation Box: ((4256,0) (4767,15) (0,0))
Simulation Box: ((4768,0) (5279,15) (0,0))
Simulation Box: ((5280,0) (5791,15) (0,0))
Simulation Box: ((5792,0) (6079,15) (0,0))
Simulation Box: ((6080,0) (6335,15) (0,0))
Simulation Box: ((6336,0) (6847,15) (0,0))
Simulation Box: ((6848,0) (7119,15) (0,0))
Simulation Box: ((7120,0) (7375,15) (0,0))
Simulation Box: ((7376,0) (7887,15) (0,0))
Simulation Box: ((3680,16) (3759,31) (0,0))
Simulation Box: ((3760,16) (3839,47) (0,0))
Simulation Box: ((3840,16) (4015,63) (0,0))
Simulation Box: ((4016,16) (4095,111) (0,0))
Simulation Box: ((4096,16) (4191,207) (0,0))
Simulation Box: ((7840,16) (7887,223) (0,0))
Simulation Box: ((4016,160) (4095,223) (0,0))
Simulation Box: ((3968,176) (4015,223) (0,0))
Simulation Box: ((3920,192) (3967,223) (0,0))
Simulation Box: ((3680,208) (3919,223) (0,0))
Simulation Box: ((3632,224) (3951,255) (0,0))
Simulation Box: ((3952,224) (4207,255) (0,0))
Simulation Box: ((4208,224) (4719,255) (0,0))
Simulation Box: ((4720,224) (5007,255) (0,0))
Simulation Box: ((5008,224) (5263,255) (0,0))
Simulation Box: ((5264,224) (5775,255) (0,0))
Simulation Box: ((5776,224) (6095,255) (0,0))
Simulation Box: ((6096,224) (6351,255) (0,0))
Simulation Box: ((6352,224) (6863,255) (0,0))
Simulation Box: ((6864,224) (7375,255) (0,0))
Simulation Box: ((7376,224) (7887,255) (0,0))

It is appearing as though the simulation grid bounds (x_lo, y_lo) (x_hi, y_hi) are twice that of what they should be. Now this might have something to do with the fact that I am expanding my domain by 2, from 32 cm to 64 cm in the x-direction while keeping the y direction constant. Since the grid is dictated by integer values, I wouldn't think that this would cause a mismatch as the "location" of the points should remain constant.

I am currently imposting my pltdata and populating all "empty" cells due to domain expansion using a slightly modfied pc_initrestartdata function from prob.cpp and then calling IO.cpp to load the pltData. This is working correctly when only Level 0 is considered, with the pltdata being correctly imposed on the new domain. Currently my major issue is originating from this mismatch in the MultiFab boxes between the pltFile and current simulation data.

My current suspicion is that either the domain is being refined from the gridfile, despite regard_on_restart and regrid_init both being set to 0 (off) or something is changing between loading the gridfile and when it is used.

Thank you for your any input you can provide.

EDIT:

I tested the hypothesis regarding the current simulation boxes being double of the imposed pltfile boxes. I did this by changing the python script to divide the extracted pltfile boxes by 2. When this was done the simulation boxes directly overlayed with the pltfile boxes. Meaning that something is occurring when the gridfile is loaded and the domain imposed is not the same physical size, despite the domain being scaled such that all cells between the smaller and larger domain overlap (e.g., 32 cm with 2048 cells and 64 with 4096).