EntityLinker architecture

[Joozty]

Hi there, would it be possible to extend the documentation about EntityLinker architecture? I studied the source code a little bit and from what I can tell there is some residual network and linear network? Is it correct? What is the rationale behind it? Is it based on some research paper?

Many thanks

[polm]

Not sure if there was a specific paper it was based on, but the presentation from spaCy IRL might be helpful.

[svlandeg]

Hi! The IRL presentation is probably the best reference indeed. At the time, we took inspiration from several research papers, including https://arxiv.org/pdf/1808.07699.pdf, but it's definitely not a 1-1 mapping.

[Joozty]

Thanks I will read it 🙂

[Joozty]

I saw the presentation several times but it doesn't tell much about the architecture. It's more like a general overview and it seems to be outdated. For example, I don't think NER type is a part of the input vector to the EL model based on the source code I studied. Maybe I overlooked it 🤔.

I am trying to understand what is behind this arrow:

[svlandeg]

Ah, apologies, you're right! At the time we were experimenting with two different architectures and decided for a final more simple version:

(this slide was in backup but I think it wasn't actually presented)

[Joozty]

Thanks a lot. And what about the actual architecture. Could you please also shed some light on this? I mean what types of neural networks are used and why?

[svlandeg]

You've linked the code so that really is the main source of truth, and I sent you one of the articles where we've gotten inspiration from. I'm not sure what to add to that ;-)

We can definitely envision other/more complex architectures in the future, but we'd need a really good gold-standard dataset to benchmark properly on first. You can easily swap in different custom architectures with the v3 config system though!

[Joozty]

Fair enough... The thing is that I am kind of new to all this ML stuff so I was just seeking more help to like understand why for example residual network and linear network is there (if I get it from the code correctly). I will get through the article though and see if there is something that helps me understand this.

[Joozty]

I got through the article you provided as well as spaCy documentation and source code and this is the diagram I created. Sharing it so somebody in the future can benefit from it too. Could you please @svlandeg confirm it is correct?

Obviously I draw only context-sensitive model (context=true, prior=false). With the prior set to true, it is just another layer on top of it that adds prior probabilities in similarity computation.