Draft: Valloc Interface for New Reducer

[rchen20]

Summary

• This PR is a refactoring

• It does the following:

  • Refactors new reduction interface to have consistent operator semantics
  • Introduces type safety to reduction operators
  • Will require the use of additional types for reduction targets

• sequential

• CUDA

• HIP

• SYCL

• OpenMP

[MrBurmark]

Do we have an option to change the loc type?

[rchen20]

Do you mean you want something like ValLocOp<double, int, minimum>, where int represents the loc type? I don't think we've had that in the past.

[MrBurmark]

yep, and we always should have had it.

[MrBurmark]

This reminds me that we should specialize SoAPtr on ValLoc, etc.

[MrBurmark]

We should figure out why using Op<ValOp<ValLoc<T>,OP>, ...>::identity() works here but Op<ValOp<T,OP>, ...>::identity() doesn't work below.

[rchen20]

After some discussion with @MrBurmark, we've both concluded that the GPU backends for a valloc interface have enough complications that will require a re-design of valloc. The problem lies in the fact that the GPU backend implementation of grid_reduce (e.g. policy/cuda/reduce.hpp) relies on the reduction object and the val-x type (T). In the current valloc design, the val-x type can become a complex type (ValLoc, ValOp, ValOp), which was not the intended use for grid_reduce which primarily should handle basic data types. To remedy this, I will change the valloc interface requirements, and attempt to hide the various val-x types within RAJA, rather than exposing them to the user. Here is an example of the new valloc we have in mind:

Original Valloc Design -

using REF_INT_SUM = RAJA::expt::ValOp<int, RAJA::operators::plus>;
using REFLOC_INT_MIN = RAJA::expt::ValLocOp<int, RAJA::operators::minimum>;
REF_INT_SUM rcuda_sum(0);
REFLOC_INT_MIN rcuda_minloc(std::numeric_limits<int>::max(), -1);
RAJA::forall<EXEC_POL3R>(cuda_res, arange,
  RAJA::expt::Reduce(&rcuda_sum),
  RAJA::expt::Reduce(&rcuda_minloc),
  [=] RAJA_DEVICE (int i, REF_INT_SUM &_rcuda_sum, REFLOC_INT_MIN &_rcuda_minloc){ // lambda body } );

New Valloc Design -

int rcuda_sum(0);
RAJA::expt::ValLoc<int> rcuda_minloc(std::numeric_limits<int>::max(), -1);
RAJA::forall<EXEC_POL3R>(cuda_res, arange,
  RAJA::expt::Reduce<RAJA::operators::plus>(&rcuda_sum),
  RAJA::expt::Reduce<RAJA::operators::minimum>(&rcuda_minloc),
  [=] RAJA_DEVICE (int i, RAJA::expt::reduceref &_rcuda_sum, RAJA::expt::reduceref &_rcuda_minloc){ // lambda body } );

Notes on new design -
It looks like the first new reducer design, except that the lambda data types are references to the underlying data. The lambda data type RAJA::expt::reduceref will internally generate the various val-x types we need, depending on the Reduce policies and concrete data types passed in. reduceref will ensure the type safety and interface consistency that we wanted in the beginning.

[artv3]

@rchen20 @MrBurmark Will we update all the other reducers as well?

[rchen20]

@rchen20 @MrBurmark Will we update all the other reducers as well?

No, I'm fairly certain this interface change can be contained to the new reductions. Although, later on when we unify the old and new reducers, we can consider how that interface will look, and how we want to implement it. For this implementation, I will probably use some of the principles from @MrBurmark's multi-reducer design.

[artv3]

@rchen20 @MrBurmark Will we update all the other reducers as well?

No, I'm fairly certain this interface change can be contained to the new reductions. Although, later on when we unify the old and new reducers, we can consider how that interface will look, and how we want to implement it. For this implementation, I will probably use some of the principles from @MrBurmark's multi-reducer design.

Right that is what I meant, like the SumReducer etc...

[rchen20]

Closing in favor of the more updated version here #1732.