Add Dilated Sliding Window mask_mod #12
Conversation
facebook-github-bot
added
the
CLA Signed
|
|
||
| def dilated_sliding_window(b, h, q_idx, kv_idx): | ||
| diff = q_idx - kv_idx | ||
| in_window = (diff >= 0) & (diff < window_size * dilation) |
There was a problem hiding this comment.
hmm from the paper its not clear to me that its always causal
what about torch.abs(diff) < window_size ?
There was a problem hiding this comment.
One other nit I think that its clearer if we keep the window_size and dilation separate
e.g. to recreate the paper (if we didnt have the and_causal mask)
we would set window_size = 8 and dilation = 2
There was a problem hiding this comment.
what about torch.abs(diff) < window_size ?
I thought it would be good to make this implementation consistent with attn_gym/masks/sliding_window.py.
However, seems reasonable to follow the non-causal way the paper described. I will update the generate_dilated_sliding_window() function.
There was a problem hiding this comment.
One other nit I think that its clearer if we keep the window_size and dilation separate
e.g. to recreate the paper (if we didnt have the and_causal mask)
we would set window_size = 8 and dilation = 2
Maybe, I missed something. Can you please explain what does it mean by "if we keep the window_size and dilation separate"?
Did you mean setting window_size = 8 and dilation = 2?
There was a problem hiding this comment.
Ohh I just meant that the dilation factor doesnt have any impact on the absolute size of the window.
window_size * dilation -> window_size
So the "potential" size of the window is 16 elements (8 forward, 8 backward ) but a dilation factor knocks out half and we end up up with 4 on both sides. We dont extend the window so as to capture more elements
|
Hi @drisspg, thanks for the review! |
Summary
Visualization