Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DO NOT MERGE Next #27

Draft
wants to merge 36 commits into
base: master
Choose a base branch
from
Draft

DO NOT MERGE Next #27

wants to merge 36 commits into from

Conversation

vringar
Copy link
Contributor

@vringar vringar commented Apr 6, 2021

This PR only aims to illustrate how far the actual development has progressed vs what is currently in master.
Each of these changes should be structured as their own PRs.

vringar and others added 30 commits June 15, 2020 11:41
@vringar
Added mode parameter to read_table
d230939
@vringar
Added mode to PySparkS3Dataset.read_table
24df34c
@vringar
updated get_option_dict
348a819
@vringar
Verified mapping
84c13b7
@vringar
Getting blockers to work without files
0e275a9
@vringar
Wrapped get_matching rules to bind blockers in scope
4c182a9
@vringar
Removed filtering from S3Datasets
fabb2f7
@vringar
Removed udfs
f63989e
@vringar
Moved comment to the appropriate place
505bcf6
@vringar
We shouldn't try to follow OpenWPMs versions
14e279d
@vringar
Backported fixes in get_worst_status_per_visit_id
d07eed7
@vringar
Fixed reduce_to_best_command_status
a0be666
@vringar
Added display_crawl_results
be6aa26
@vringar
Rewrote crawlhistory.py
5b24f6f
@vringar
Removed dataquality.py
937285b
@vringar
Used typeannotations
d8eb4bb
Fixing display_crawl_history
5280711
Added docstrings
ae05d1f
Made PySparkS3Dataset forward to S3Dataset
c89dd3a 
It turns out that opening a file via boto is
a lot faster than spark_content.textFile

Closes #23
Made PySparkS3Dataset a subclass of S3Dataset
798ea1e
Merge branch 'master' into update_option_dict
880c4f4
Returning None if something goes wrong during get_matching_rules
f92da0c
Merge branch 'master' into display_crawl_history
e9d2fc2
Added demo file
8cf5823
Merge branch 'master' into update_option_dict
1b65030
Merge branch 'display_crawl_history' into next
e1af7ce
Merge branch 'fixing_collect_content' into next
16e016b
GcsDataset implementation
5b5157d
Merge branch 'display_crawl_history' into gcsDataset
4aeee1c
Changing scheme from gcs to gs
a8003ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vringar