-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: Adding "add_jmes" and "replace_jmes" method to ItemLoader #67
Comments
I had done something similar in |
@voith that is really nice! You went the step further and got json out of html using xpath then using jmespath to get the stuff you want. However, my request is more for pure Json data that contains no html what so ever. I am scraping a lot of sites using the React framework and all of the data I need from these sites is in the Json. Plus, apis that consume Json is also another need to use jmespath. Just like xpath/css selectors are favor over regular expressions, jmespath are also favored over turning the json data into a python dict and access directly in loops and using key indexes. Jmespath provides a query string that can be used the same way as xpath/css selectors. |
@IAlwaysBeCoding I understand your requirement. The above PR can also work only with Jmespath too. |
I think this feature should go to the ItemLoader repository: https://github.com/scrapy/itemloaders/ But I want it very much! The current workaround is probably defining a separate Item type for that (json) data item and then setting |
I have this working in a project, and the path forward requires 4 changes in 3 repositories, 2 of them in Scrapy:
|
I’m moving this to itemloaders since we have scrapy/scrapy#5894 for the remaining Scrapy work. |
So, currently the
ItemLoader
class has 6 methods for loading values:add_xpath()
replace_xpath()
add_css()
replace_css()
add_value()
replace_value()
Could we add another 2 more methods for loading data through JmesPath selectors. Currently, I have to use the
SelectJmes
processor to do this. Eventually, it looks really ugly and ends up taking so much line real estate.I did a hack where I extended the ItemLoader to include those 2 extra methods that are desperately needed.
When there is json data to parse instead of html, JmesPath selectors are the best way to go for parsing, so there should be support for JmesPath selectors in the ItemLoader class as well.
The text was updated successfully, but these errors were encountered: