Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Particle Patches: Fallback #8

Open
ax3l opened this issue Jun 4, 2020 · 27 comments
Open

Particle Patches: Fallback #8

ax3l opened this issue Jun 4, 2020 · 27 comments
Labels
bug Something isn't working

Comments

@ax3l
Copy link
Member

ax3l commented Jun 4, 2020

Hi @KseniaBastrakova,

just a minor issue discovered by @NeilZaim: "particle patches" are an optional feature in openPMD:

SCALAR = openpmd_api.Mesh_Record_Component.SCALAR
num_particles = particle_species.particle_patches["numParticles"][SCALAR].load()
series_hdf.flush()

Although PIConGPU always writes them, these patches might not exist in other data sets such as the examples here:
https://github.com/openPMD/openPMD-example-datasets/
This currently leads to a segfault when using the scripts with those examples.

Can we please add the following simple fallback logic? If particle species does not declare particle patches, just chunk the particle read by a fixed number of particles, e.g. 1e7 particles at a time.

@ax3l ax3l added the bug Something isn't working label Jun 4, 2020
@KseniaBastrakova
Copy link
Collaborator

Thanks for pointing out, I will try to fix this now

@KseniaBastrakova
Copy link
Collaborator

Fixed. Tested on openPMD example datasets. Now it seems to work correctly

@ax3l
Copy link
Member Author

ax3l commented Jun 8, 2020

Thanks a lot, this is great!
@NeilZaim does this work for you, too? :)

@NeilZaim
Copy link
Contributor

NeilZaim commented Jun 8, 2020

Thanks a lot! It goes further in the code but I still observe a couple of issues.

I'm trying to use the Voronoi algorithm using the following command:
python reduction_main.py -algorithm=voronoi -hdf=../simData_0008000.h5 -hdf_re=../reduced.h5 -momentum_tol=0.1 -momentum_pos=0.1 (these are random values for the tolerances)

The first issue I encounter is that I think the dimensions attribute of the algorithm is never set, which, for the Voronoi algorithm, results in a failure of the check_needs_subdivision function.

Since I'm testing the code with data from a 2D simulation, I guess that I can temporarily fix this by adding algorithm.dimensions = Dimensions(2,3) approximately anywhere in the code right?

Now if I do this and try the above command line, I obtain the following error:

Traceback (most recent call last):
File "reduction_main.py", line 891, in
base_reduction_function(args.hdf, args.hdf_re, "voronoi", parameters)
File "reduction_main.py", line 133, in base_reduction_function
process_iteration_group(algorithm, current_iteration, series_hdf, series_hdf_reduction, reduction_iteration)
File "reduction_main.py", line 160, in process_iteration_group
process_patches_in_group_v2(iteration.particles[name_group], series_hdf,
File "reduction_main.py", line 737, in process_patches_in_group_v2
write_draft_copy(reduced_weight, relative_result,
File "reduction_main.py", line 537, in write_draft_copy
particle_species["weighting_copy"][SCALAR][previos_idx:current_idx] = reduced_weight
TypeError: setitem(): incompatible function arguments. The following argument types are supported:
1. (self: openpmd_api.Record_Component, tuple of index slices: tuple, array with values to assign: array) -> None
2. (self: openpmd_api.Record_Component, slice: slice, array with values to assign: array) -> None
3. (self: openpmd_api.Record_Component, axis index: int_, array with values to assign: array) -> None

Invoked with: <openPMD.Record_Component of dimensionality '1'>, slice(0, 1, None), [9120000000.0]

Do you know where this is coming from? I can run more tests if needed.

Thanks again!

@KseniaBastrakova
Copy link
Collaborator

@NeilZaim thanks. i reproduced it
working on a fix

@KseniaBastrakova
Copy link
Collaborator

@NeilZaim now it seems to work. I also add some information about algorithms to a command arguments description

@NeilZaim
Copy link
Contributor

Thanks a lot @KseniaBastrakova !

I am now able to produce h5 output that I can read (tested with the Voronoi algorithm) so I'll start toying around and see what I can get :) .

I've encountered a few issues, listed below:

  • On the reduced dataset, the mass and charge still have the shape of the old dataset. (If I had 1000 particles before and 10 particles now, reading the mass on the reduced dataset still results in a vector of size 1000, at least using openPMDviewer). This causes openPMDviewer to crash when reading the momentum during the renormalization step. This is however easy to work around by removing the momentum normalization in openPMDviewer and doing it manually afterwards.

  • I still encounter some random segmentation faults (core dumped) during some openpmd-api calls. More specifically this can occur during the line reduction_series.flush() at the very end of copy_meshes in reduction_main.py. If I comment out copy_meshes (which is ok for me, I only need particle data in the reduced dataset, not grid data) I can still have a segfault during the call series_hdf_reduction.flush() in line 536 of reduction_main.py (in the function write_draft_copy). These segfaults do not occur everytime so I guess it's harder to debug but I can provide more info if needed. When there is no segfault the code runs until the end.

  • I think that we can remove line 895 in reduction_main.py (base_reduction_function(args.hdf, args.hdf_re, "voronoi", parameters)). The base_reduction_function is already called in the function voronoi_algorithm so this makes the code run twice.

Thanks again!

@KseniaBastrakova
Copy link
Collaborator

@NeilZaim Thank you very much for the detailed description of the errors. Working on reproduction and fix

@KseniaBastrakova
Copy link
Collaborator

@NeilZaim, I fixed the first issue and, probably, the the secound one(I found a possible problem that could cause it), so now it seems work for all algorithms

@sbastrakov
Copy link
Member

And I assume the third one was fixed with PR by @NeilZaim ?

@KseniaBastrakova
Copy link
Collaborator

KseniaBastrakova commented Jun 24, 2020

And I assume the third one was fixed with PR by @NeilZaim ?

yes, I merged it already

@NeilZaim
Copy link
Contributor

Thanks a lot! I've just given it a try but I get the following error message

Traceback (most recent call last):
File "reduction_main.py", line 746, in
voronoi_algorithm(args.hdf, args.hdf_re, args.momentum_tol, args.position_lol)
File "reduction_main.py", line 708, in voronoi_algorithm
base_reduction_function(hdf_file_name, hdf_file_reduction_name, "voronoi", parameters)
File "reduction_main.py", line 203, in base_reduction_function
process_iteration_group(algorithm, current_iteration, series_hdf, series_hdf_reduction, reduction_iteration)
File "reduction_main.py", line 222, in process_iteration_group
process_patches_in_group_v2(iteration.particles[name_group], series_hdf,
File "reduction_main.py", line 654, in process_patches_in_group_v2
write_draft_copy(relative_data, dict_data_indexes, series_hdf_reduction, particle_species_reduction,
File "reduction_main.py", line 463, in write_draft_copy
write_record_copy(current_record, values, series_hdf_reduction, previos_idx, current_idx)
File "reduction_main.py", line 435, in write_record_copy
current_reduced_data = values_to_write[:, pos_vector_in_reduction_data].astype(current_type)
IndexError: index 1 is out of bounds for axis 1 with size 1

I don't have time to look at it in more details today but if you want I can run more tests/provide more details tomorrow.

@KseniaBastrakova
Copy link
Collaborator

@ NeilZaim Do you use 1-d coordinates? I think my problem in improper handling this dataset. (only testing on 2-d and 3-d)
working on a fix

@NeilZaim
Copy link
Contributor

No this is data from a 2D simulation. Although in this case the coordinates are named x and z. (there is no y coordinate) Could this be causing the issue?

@NeilZaim
Copy link
Contributor

On a completely unrelated note, what is the current status of the Vranic algorithm? I see that there is a file in the Algorithm folder but the algorithm is not included in reduction_main.py. Is the algorithm working except for IO? Or is there still work to do on the algorithm itself? In both cases I can try to make it work, I'd like to test it.

@KseniaBastrakova
Copy link
Collaborator

KseniaBastrakova commented Jun 25, 2020

No this is data from a 2D simulation. Although in this case the coordinates are named x and z. (there is no y coordinate) Could this be causing the issue?

could you provide a file? I made a file only with x and z. coordinates, but, sadly, still can't reproduce the bug

@NeilZaim
Copy link
Contributor

Ok thanks. I think the data is a bit large but I'll see if I can produce smaller data with the same error message.

@KseniaBastrakova
Copy link
Collaborator

On a completely unrelated note, what is the current status of the Vranic algorithm? I see that there is a file in the Algorithm folder but the algorithm is not included in reduction_main.py. Is the algorithm working except for IO? Or is there still work to do on the algorithm itself? In both cases I can try to make it work, I'd like to test it.

I still work on the algorithm itself, so it works for a few simple cases, but I did not test it completely and did not measure the metrics

@NeilZaim
Copy link
Contributor

No this is data from a 2D simulation. Although in this case the coordinates are named x and z. (there is no y coordinate) Could this be causing the issue?

could you provide a file? I made a file only with x and z. coordinates, but, sadly, still can't reproduce the bug

Hello Ksenia. I attach below a somewhat small h5 (~3MB) file which reproduces the issue.
simData_0000040.h5.zip

The error appears when record_name in write_draft_copy is E so maybe it comes from the fact that the particle data I use contains the E and B field at the particles position?

In any case, I've also noticed that the line values = data[particle_index_range[0]:particle_index_range[1]][0] in write_draft_copy seems to be equivalent to values = data[particle_index_range[0]], which could be different from what you initially wanted no? (maybe you meant values = data[particle_index_range[0]:particle_index_range[1],0]?) If this is the case, could this cause the issue? (if this means that values does not have the expected shape)

@KseniaBastrakova
Copy link
Collaborator

@ NeilZaim, thank you for the small example. Yes, I think, it can cause this problem. I working on a small fix

@KseniaBastrakova
Copy link
Collaborator

KseniaBastrakova commented Jul 2, 2020

@NeilZaim , i fixed it in #19. Now, your file working correctly in my case, could you try once again?

@NeilZaim
Copy link
Contributor

NeilZaim commented Jul 8, 2020

So I've tested it and it works now, thanks a lot!

I still have some random segfaults during some of the calls to series_hdf_reduction.flush(), but I can live with that.

I've also noticed that the code feeds many values to the merging algorithms (all the components in particle_species.items(), for instance the E and B values at the particle positions) which are for the most part unused by the algorithms. Typically I've added something like

if record_component_name != "momentum":
            continue

in get_data and it can make the scripts much faster when working on larger data arrays.

I also now have a working version of the Vranic algorithm, I'll write a PR soon.

@sbastrakov
Copy link
Member

Thanks for your feedback @NeilZaim .

Regarding the loading, I agree that many things are not needed in the algorithms. Unfortunately, they still need to be loaded from and written back to the file, as the structure changes with some macroparticles being deleted. Also, when macroWeighted = 1 the values need to be changed when the weight changes. However, it can be that the implementation does some more data shuffling than necessary. Could you look into that @KseniaBastrakova ?

@NeilZaim
Copy link
Contributor

NeilZaim commented Jul 9, 2020

I see, thanks. (although I guess that in the general case values other than momentum, position and weight are undefined anyways when we create new particles (with Voronoi for instance)). Note that what I have currently written for the Vranic algorithm just returns position/momentum/weight, I can modify it to include other values if needed.

In any case you don't need to worry too much about further optimizing the code as far as I am concerned, what is currently here is fine for me.

@NeilZaim
Copy link
Contributor

NeilZaim commented Jul 9, 2020

Actually Vranic currently crashes if we don't add the lines

if record_component_name != "momentum":
            continue

😬
I'll write a fix

@sbastrakov
Copy link
Member

@NeilZaim will you provide this as another commit to your PR #20 ?

@NeilZaim
Copy link
Contributor

NeilZaim commented Jul 9, 2020

Yes, just pushed it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants