- Officially drop support for Python 3.6. Sparkmagic will not guarantee Python 3.6 compatibility moving forward.
- Re-format all code with
blackand validate via CI
- Fixed, simplified, and sped up the sparkmagic Docker images for local development and testing
- Adds support for
ipython>=8 - Adds support for
ipykernel>=6
- Pins
ipython<8.0.0because of breaking changes. Thanks @utkarshgupta137
- Pins
ipykernel<6.0.0because of breaking changes
- Migrate from Travis to Github Actions for CI
- Added one internal magic to enable retry of session creation. Thanks @edwardps
- New
%%prettymagic for pretty printing a dataframe as an HTML table. Thanks @hegary - Update Endpoint widget to shield passwords when entering them in the ipywidget. Thanks @J0rg3M3nd3z @jodom961
- Officially drop support for Python 2. Sparkmagic will not guarantee Python 2 compatibility moving forward.
- Update Spark, SparkR, and PySpark kernels to include language info and file extension for Jupyterlab LSP compatibility. Thanks @skakker
KeyboardInterruptcancels the Livy statement. Thanks @hanyucui- Add user to %%info output. Thanks @tprelle
- Fix missing url passing when using a custom authenticator #684. Thanks @gthomas-slack
- Customizable Livy authentication methods #662. Thanks @alexismacaskilll
- Fix Dockerfile.jupyter build #672
- Fix ContextualVersionConflict in Dockerfile.spark. Thanks Linan Zheng, @LinanZheng
- Fix Info Subcommand in RemoteSparkMagic. Thanks Linan Zheng, @LinanZheng
- Fix to ignore unsupported spark session types whilst fetching the session list. Thanks, Murat Burak Migdisoglu, @mmigdiso
cleanup_all_sessions_on_exitconfiguration to cleanup all registered livy sessions regardless of whether the process exits gracefully. Thanks Juho Autio, @juhoautio- Add configuration options to for the default
HTTPKerberosAuthconstructor. Thanks Pedro Gonçalves Rossi Rodrigues, @PedroRossi
- Respect the
all_errors_are_fatalflag and raise an exception if the session fails to start. Thanks Devin Stein, @devstein - Use
requests.Sessionto avoid negotiating Kerberos tickets in every request. Thanks Pedro Gonçalves Rossi Rodrigues, @PedroRossi
- Enabled heartbeat by default, so long-running tasks don't time out. Thanks to John Pugliesi for the bug report.
- The PySpark kernel uses Python 3 lexer, instead of Python 2. Python 2 support is going away in the near future.
- Fixed papermill support; there's a different option now you need to use,
all_errors_are_fatal. See the README for details. Thanks to Devin Stein for the patch.
- Fixed silly mistake in the tests.
- Added two configuration options that make it easier to run Sparkmagic notebooks with Papermill. Thanks to Michael Diolosa for the patch.
- Support
text/htmlmessages from the Livy server; currently Livy/Spark itself don't really do this, but some experimental tools can benefit from this. Thanks to Steve Suh.
- Support server-side rendering of images, so you don't have to ship all the data to the client to do visualization—see the
%matplotusage in the example notebook. Thanks to wangqiaoshi for the patch. - Progress bar for long running queries. Thanks to @juliusvonkohout.
- Work correctly with newer versions of the Jupyter notebook. Thanks to Jaipreet Singh for the patch, Eric Dill for testing, and G-Research for sponsoring Itamar Turner-Trauring's time.
- Switch to Plotly 3.
- Updated code to work with Livy 0.5 and later, where Python 3 support is not a different kind of session. Thanks to Gianmario Spacagna for contributing some of the code, and G-Research for sponsoring Itamar Turner-Trauring's time.
- Fixed
AttributeErroronNone, thanks to Eric Dill. recoveringsession status won't cause a blow up anymore. Thanks to G-Research for sponsoring Itamar Turner-Trauring's time.