Bad allocation/Segmentation Faults

[jphoplun]

What are the CPU/Memory requirements for using scikit-learn Intel library?

I am trying to use Intel library (SVM) and getting bad allocation/segmentation faults. I am using Azure VM (2 core 8GB). When I use smaller training data set (less then 100K) it works but when I give more then that, I am getting the issue, I am expecting training set to around 300K.

Also, it consumes almost all the CPU/Memory so VM is not usable any mechanism or setting which would avoid it.

Thanks & Regards
Jakir

[masdevas]

Hi @jphoplun!
Did you try to process your data with original sklearn? Did it work?
If it worked, I would like to clarify about your data and soft. Could you please share number of features of your dataset? Also I would like to understand your environment (sklearnex version, sklearn version, OS).
If you can share your code with SVM usage it will be really helpful for reproduction on my side.

Thank you,
Andrey

[masdevas]

I will analyse your SVM case in nearest time and contact to you.

Thank you so much,
Andrey

[masdevas]

Hi Jakir @jphoplun !
I've just analyzed your SVM case. I found out that in single-thread mode both Scikit-learn and Intel(R) Extension for Scikit-learn consumes ~2.5GB RAM on your case with default SVC's parameters. But in our default multithread mode with X threads algorithm in our extension will consume in ~X times more memory. So, if you have host with 2 cores + enabled HyperThreading (4 threads at all) and 8 GB RAM you can catch "out of memory". I would suggest you to reduce threads count available to your application. You can do that with daalinit or Linux tools like taskset.

Thank you,
Andrey

[jphoplun]

Thanks Andrey.

I tried below couple of things

1. Tried using daalinit which you mentioned but it didn't worked. I have 2 core each core has 1 HT. I had tried daalinit(2) and daalinit(1) but still got the same issue.
2. I downgraded scikit-learn from 0.24.2. to 0.22.2 and ran the same code with same distribution of train/test and it worked. It completed the training and test with the Intel 3.0 library. So not sure if it has some incompatibility.

Thanks & Regards
Jakir

[masdevas]

Jakir, could you please check number of used cores when you run with daalinit(1)? It should be 1.

Thank you,
Andrey

[jphoplun]

When I am using daalinit(1), I see only one processor being used (almost 100%).

Thanks & Regards
Jakir