Query regarding layer-wise analog conversion

[IannoIITR]

I had a model with grouped convolutions(which is not supported in aihwkit). So I tried converting individual layers from the model dict to analog.
for eg.

model.backbone.input_stem.op_list[0] = convert_to_analog(model.backbone.input_stem.op_list[0],rpu_config)
model.backbone.input_stem.op_list[1].main.point_conv = convert_to_analog(model.backbone.input_stem.op_list[1].main.point_conv ,rpu_config)
model.backbone.stages[0].op_list[0].main.inverted_conv = convert_to_analog(model.backbone.stages[0].op_list[0].main.inverted_conv ,rpu_config)
model.backbone.stages[0].op_list[0].main.point_conv = convert_to_analog(model.backbone.stages[0].op_list[0].main.point_conv ,rpu_config)

so basically replacing the existing floating point layer in the model with an equivalent one converted to analog.

I wanted to know if this is a valid approach or is there any problem in it as it is working for some models but in others there is a large drop in accuracy. (single layer drops accuracy of pre-trained model to 0, even when rpu config is near ideal [32 bit resolution and no noise values])

[maljoras]

Hi @IannoIITR ,
many thanks for the question. Correct, grouped convolution is currently not supported. Typically you want to just use convert_to_analog(model, rpu_config) for the full model, it will look for the Conv2D layers to be replaced.

In your approach, is the e.g. point_conv still a grouped convolution? Then it will not work. The problem with grouped convolution is that they map poorly to crossbar tiles, as they result in block-diagonal kernel matrices. One could put the blocks onto individual tiles, but that would become inefficient very quickly. Or one could use a full tile for the non-grouped conv kernel matrix and then "program" approx zeros in the off-diagonal values. For inference it would be fast, but for training it would need some sequential masking of input and gradients for each group.

So while problematic for large number of groups, for small number of groups it could be reasonable. Handling conv groups could be a good feature to add to the toolkit in future.

[IannoIITR]

Hi @maljoras
When I try to convert this model it just throws an exception that groups >1 are not supported. Hence I resorted to convert the layers individually. specifically the layers with no groups.

        self.depth_conv = ConvLayer(
            in_channels,
            in_channels,
            kernel_size,
            stride,
            groups=in_channels,
            norm=norm[0],
            act_func=act_func[0],
            use_bias=use_bias[0],
        )
        self.point_conv = ConvLayer(
            in_channels,
            out_channels,
            1,
            norm=norm[1],
            act_func=act_func[1],
            use_bias=use_bias[1],
        )

Here the depth conv has 'in_channels' as number of groups but the point conv is a normal convolution with kernel size 1 and also group size 1.
Converting this single layer to analog is also significantly impacting the accuracy and I used the above command to 'insert' the analog converted layer into the original model dictionary. inplace of the layer that is present in full precision
model.backbone.input_stem.op_list[1].main.point_conv = convert_to_analog(model.backbone.input_stem.op_list[1].main.point_conv ,rpu_config)

It was working for other albeit simpler models, I wanted to verify if this is a valid approach to testing 'hybrid' models where part of the layers are analog and part of them are in full precision.

[maljoras]

I see, yes, it should be possible to convert single layers in the way you suggest. Might be worthwhile to add an option in the convert_to_analog script to just skip group>1 convolution in the conversion. In the long run, we want to support group convolution in some way.

Accuracy might drop if you just convert it. Hardware-aware training might be needed to make the model more robust against analog noise and non-idealities. See e.g. this paper, where the direct mapping approach (Table 2) impact accuracy quite severely for some DNNs (especially CNNs). Some of the accuracy drop can be recovered by hardware-aware training (Table 3).