Feature Request: Adding "add_jmes" and "replace_jmes" method to ItemLoader

[IAlwaysBeCoding]

So, currently the ItemLoader class has 6 methods for loading values:
add_xpath()
replace_xpath()
add_css()
replace_css()
add_value()
replace_value()

Could we add another 2 more methods for loading data through JmesPath selectors. Currently, I have to use the SelectJmes processor to do this. Eventually, it looks really ugly and ends up taking so much line real estate.
I did a hack where I extended the ItemLoader to include those 2 extra methods that are desperately needed.

When there is json data to parse instead of html, JmesPath selectors are the best way to go for parsing, so there should be support for JmesPath selectors in the ItemLoader class as well.

[voith]

I had done something similar in parsel. scrapy/parsel#27

[IAlwaysBeCoding]

@voith that is really nice! You went the step further and got json out of html using xpath then using jmespath to get the stuff you want. However, my request is more for pure Json data that contains no html what so ever. I am scraping a lot of sites using the React framework and all of the data I need from these sites is in the Json. Plus, apis that consume Json is also another need to use jmespath. Just like xpath/css selectors are favor over regular expressions, jmespath are also favored over turning the json data into a python dict and access directly in loops and using key indexes. Jmespath provides a query string that can be used the same way as xpath/css selectors.

[voith]

@IAlwaysBeCoding I understand your requirement. The above PR can also work only with Jmespath too.
My requirement was such that I needed chain selectors. I've scraped sites which give json with some embedded html in the response. So it would be nice to have nested selectors that would solve everyones need.
But the problem is that building such nested selectors is not that easy. My PR too is not a full proof solution. Its just a POC to take the idea ahead(I suspect that there will be several bugs in my implementation). There was quite a discussion held in scrapy/parsel#25 because of the complexity.

[MaxValue]

I think this feature should go to the ItemLoader repository: https://github.com/scrapy/itemloaders/

But I want it very much! The current workaround is probably defining a separate Item type for that (json) data item and then setting SelectJmes as input_processor.
That way, the item unfortunately is specific to the page.

[Gallaecio]

I have this working in a project, and the path forward requires 4 changes in 3 repositories, 2 of them in Scrapy:

• Remove the use of parsel.Selector._default_type scrapy#4961
• Support JMESPath now parsel#181
• Add JMESPath support to itemloaders
• Add JMESPath support to Scrapy (response.add_jmespath, otherwise you would need to use response.selector.add_jmespath)

[Gallaecio]

I’m moving this to itemloaders since we have scrapy/scrapy#5894 for the remaining Scrapy work.