release-notes.md

Release notes

Version 0.4.20 (2022-01-19)

• Docker container provided

Version 0.4.19 (2021-10-15)

• Figure generation bug fix

Version 0.4.17 (2021-06-10)

Main updates

• Now supports Python 3.9
• Requires CUDA 11.2 and cuDNN 8.1

Version 0.4.16 (2021-06-10)

Main updates

Added data export to abc4d.

Fixed

Pinned to latest version of cellfinder-core to prevent issues when upgrading.

Version 0.4.15 (2021-05-05)

Fixed a bug where the analysis step may not run.

Version 0.4.13 (2021-05-04)

Main updates

The cell detection algorithm is now also implemented as a napari plugin. This is also the preferred method for generating training data.

Fixed

Fixed a bug where cellfinder crashes at the analysis step if no cells are detected in that channel.

Version 0.4.12 (2021-04-19)

Fixed a bug where cellfinder would not continue after only cell detection (no classification). Thanks to Nicole Vissers (SWC) for spotting this!

Version 0.4.11 (2021-03-11)

Core cell detection computation moved to the cellfinder-core package.

Version 0.4.8 (2021-01-11)

Updated required versions of tensorflow and numpy. cellfinder now requires CUDA 11.0.

Version 0.4.7 (2020-11-25)

Added

• A csv of every detected cell is now exported with coordinates and brain region

Fixed

• Fixed a bug causing cell distribution analysis to be rerun if already completed.
• Updated cell export to be compatible with brainrender v0.2.0

Developers

• Testing and deployment has moved from TravisCI to GitHub actions

Version 0.4.6 (2020-11-06)

Fixes a bug that prevents cells from being detected if the background intensity of the image is zero.

Version 0.4.5 (2020-10-29)

Fixed a bug in cellfinder_curate

Version 0.4.4 (2020-10-27)

Updated data export to be compatible with brainrender 1.1

Version 0.4.3 (2020-10-13)

Fixed a bug in the final data export

Version 0.4.2 (2020-10-13)

Minor bug fixes

Version 0.4.1 (2020-10-13)

Minor bug fixes

Version 0.4.0 (2020-10-13)

Main Updates

• cellfinder is now part of the BrainGlobe suite of computational neuroanatomy tools, and the documentation has moved here.
• The registration in cellfinder has moved from amap to brainreg, which uses the BrainGlobe atlas API to suppport many more atlases.
• amap is now deprecated, and will not be updated (in favour of brainreg)
• The default atlas is now the 25um Allen Mouse Brain atlas, but these can be chosen by using the --atlas flag. Run brainglobe list to see which atlases are available.
• cellfinder has now adopted the bg-space convention for inputting the orientation of your data, using the --orientation flag. For more details see Image definition
• The -x, -y, -z and --metadata flags have now been removed. See Image definition for how to enter this information.
• manual_seg has now been replaced with a new (brainreg-based) tool, brainreg-segment.
• Support for Python 3.6 has been dropped, and support for Python 3.8 added (this is now the recommended Python version).
• Data visualisation has now moved to napari plugins, see Visualisation.

Added

• --save-progress flag added for training. Saves training progress to a .csv file (output_directory/training.csv).
• Added an easy way to find the currently installed version (cellfinder --version)

Changed

• During training, the model will be saved after each epoch by default. Use --no-save-checkpoints to suppress this behaviour and save disk space. Each model file can be large, and if you don't have much training data, they can be generated quickly.

Removed

• Removed many standalone tools due to the behaviour being included within the main cellfinder program (or no longer have a use):
  • cellfinder_cell_standard
  • cellfinder_count_summary
  • cellfinder_region_summary
  • cellfinder_xml_crop
  • cellfinder_xml_scale
  • cellfinder_gen_region_vol
  • cellfinder_cells_to_brainrender

Version 0.3.14 (2020-06-12)

Main updates

• New combined cell detection and registration viewer (see Visualisation)
• There is a new and improved GUI for manually segmenting linear tracks and volumes in standard space (see Manually segment in standard space).

Added

• Registration can now deal with data in a non-standard orientation, see Image orientation
• Detected cell positions can now be manually curated and saved (see Visualisation).

Fixed

• Bug causing data to occasionally be loaded in the incorrect order for registration is fixed

Removed

• amap_vis has been replaced with cellfinder_view
• cellfinder_view_cells has been replaced with cellfinder view

Version 0.3.13 (2020-05-13)

Main updates

• Support is now officially for Windows and Linux. macOS should generally work, but there may be issues.
• Updated to amap v0.1.21 (with an updated NiftyReg version, so registration should now be slightly faster).

Fixed

• Bugs left in cellfinder_cell_standard after refactoring main code are fixed.
• Compatible with napari v0.3.0

Removed

• cellfinder_view_3D has been removed

Version 0.3.12 (2020-04-30)

Main updates

• Windows bug fix thanks to @EmanPaoli
• Prebuilt wheels now released for Linux, Windows and macOS (untested)

Developers

• Update dependencies & refactor usage of amap api

Version 0.3.11 (2020-04-24)

Update dependencies

Version 0.3.10 (2020-04-16)

Minor refactoring

Version 0.3.9 (2020-03-11)

Added

Training can now be saved every N epochs using --checkpoint-interval

Fixed

• Bug fixed in cellfinder_xml_crop
• Pandas dependencies updated thanks to @amedyukhina

Version 0.3.8 (2020-02-18)

Fixed

• Fix memory leak in training

Version 0.3.7 (2020-02-02)

Changed

• Update curation interface

Version 0.3.5 (2020-02-03)

Main updates

• New all python training data generation interface, cellfinder_curate. Details here.

Developers

• Heatmap and converting cells to brainrender format moved to neuro
• Lots of utility functions moved to imlib
• Automatic deployment with Travis

Version 0.3.2a (2020-01-10)

Main updates

• Very basic Windows support (allthough some issues remain.)
• Pixel sizes are now defined in um (rather than mm). Users specifying a metadata file will not be affected. Those specifying the pixel sizes manually will need to change their usage. Flags such as --x-pixel-mm are now --x-pixel-um.
• All cell detection parameters (e.g. --soma-diameter) are now defined in um or um3 (rather than mm)
• Registration parameters are now defined via command-line arguments (e.g. --freeform-use-n-steps) rather than a config file.
• Installation is simplified (no need to install tensorflow separately)

Changed

• Updated default parameters to better work on noisy data.

Fixed

• Memory consumption during inference reduced

Developers

• Registration code is now in amap
• Compatibility with tensorflow 2.1.0

Version 0.3.1a (2019-10-31)

Main updates

• Cube extraction is now carried out on the fly during classification (and not saved to disk). This prevents file system issues due to the generation of (potentially) hundreds of thousands of cells.

Changed

• Likelihood during training of each individual augmentation transformation reduced to 10% (from 50%)

Fixed

• Bug fix in cellfinder_view_cells to deal with Napari API change

Developers

• New viewer to inspect batches of generated cubes

  (cellfinder.classify.tools.batch_viewer)

• Removed some old, unnecessary tests

• Fixed a bug in tests/tests/test_integration/test_detection.py

Version 0.3.0a (2019-10-29)

Main updates

• Removed NiftyNet dependency. Classification is now implemented in tf.keras.

  THIS MEANS YOU NEED TO TRAIN A NEW MODEL. OLD MODELS ARE INCOMPATIBLE!

Added

• New cellfinder_view_cells command which uses napari to view cells overlaid on raw data.
• New cellfinder_gen_region_vol command line tool to generate images of specific brain regions (for figures etc)
• roi_transform now supports ROIs generated in the original, full-size images
• New cellfinder_cells_to_brainrender to allow classified cells to be viewed in BrainRender

Removed

cellfinder_view_cells2D function is removed, and replaced with cellfinder_view_cells.

Changed

• Cube scaling is now 3D (theoretically leading to better classification

  performance)

• cellfinder_view_3D now uses napari.

Developers

• Brain image IO moved to brainio

• Reading a slurm environment (if present) moved to

  slurmio.

• Logging moved to fancylog

Version 0.2.12a (2019-09-24)

Added roi_transform command line tool to transform ImageJ ROI sets into images in standard space.

Version 0.2.11a (2019-09-23)

• Fixes a bug causing heatmaps to be scaled incorrectly

Version 0.2.10a (2019-09-20)

Only minor changes to documentation and testing.

Version 0.2.9a (2019-09-20)

Added

• New --max-ram to specify a maximum amount of RAM to use (in GB). Currently only affects processes that will scale to use as much RAM as is available (such as cube extraction).
• New cellfinder_xml_scale command line tool to rescale the cell positions within an XML file. For compatibility with other software, or if your data has been scaled after cell detection.

Version 0.2.8a (2019-09-17)

THIS VERSION HAS NOT BEEN FULLY TESTED

Added

• Instructions for installing CUDA 9 and cuDNN via conda added.
• cellfinder_region_summary now has a --sum-regions flag combine child regions.

Fixed

• Bug which caused heatmaps to be calculated incorrectly is fixed.
• Bug which causes --regions-list input to cellfinder_region_summary to be parsed incorrectly is fixed.
• Bug which crashed summary csv generation if a cell is transformed incorrectly due to rounding errors is fixed.
• Bug which crashed volume calculation in registration if a region is missing is fixed.
• Bug causing registration to crash if a custom registration file without a base_folder entry is fixed.
• urllib3 version is specified to prevent atlas downloading from failing.

Version 0.2.7a (2019-09-05)

Fixed

• A bug which caused the atlas not be downloaded when using cellfinder_register is fixed.
• Registration now respects the --n-free-cpus flag

Version 0.2.6a (2019-09-05)

Main updates

• Python 3.5 is no longer supported.
• Detected cell positions can now be transformed into standard space. By default, if cell classification and registration has been performed, an additional cells_in_standard_space.xml file will be generated. Use --no-standard_space to prevent this behaviour. This functionality can also be run using the standalone script cellfinder_cell_standard.

Added

• Pixel sizes must still be specified, but instead of using the -x, -y, and -z flags, a --metadata flag can be used instead. Supported formats include BakingTray recipe files, mesoSPIM metadata files or cellfinder custom metadata files (see below). If both pixel sizes and metadata are provided, the command line arguments will take priority.

Fixed

• An installation bug (due to new requirements of gputools is fixed.)

Deprecations

• The main command-line interface to cellfinder is now cellfinder, cellfinder_run has been removed (and may call an old version of the software). Please use cellfinder from now on.

Version 0.2.5a (2019-07-31)

Fixed

• Bug in cellfinder_gen_cubes has been fixed

Deprecations

• The main command-line interface to cellfinder will be cellfinder, cellfinder_run will stop working soon (and may call an old version of the software). Please use cellfinder from now on.

Version 0.2.4a (2019-07-30)

THIS VERSION HAS NOT BEEN FULLY TESTED Use Version 0.2.3a if you don't need the new features

Main Updates

• Pixel sizes must now be specified with -x, -y, and -z.

Added

• cellfinder.IO.cells.get_cells() can now read the .yml files output from the cell counting plugin of MaSIV.
• Registration will now calculate a volumes.csv file which contains the volumes of each brain area in the atlas, in the sample brain.
• The --summarise flag will now save more information, including:
  • Cell counts combined across hemispheres
  • Cell densities in cells per mm3.
• Registration will also calculate the reverse transforms (i.e. sample -> atlas)
• A new --figures flag will create a subdirectory of 3D images in the coordinate space of the downsampled raw data (and registered atlas) that can be used to generate figures e.g. in FIJI. Currently three files will be generated by default.

  • heatmap.nii - This is a 3D histogram, showing cell densities

    throughout the brain. Likely to be more useful than visualising cellpositions for dense areas. By default, this is smoothed, and masked (so that the smoothing doesn't "bleed" outside of the brain).

  • glass_brain.nii This is a binary image, just showing the brain surface. Plotting this in 3D along with heatmap.nii gives a nice sense of cell density throughout the brain. ClearVolume works well for this.

  • boundaries.nii Another binary image, but showing the divisions between each region in the atlas. More useful for smaller, 3D figures to delineate region boundaries. Also useful to assess registration by overlaying on the raw, downsampled data.

Fixed

• Bug on certain systems where cube extraction could use too much RAM, and crash the system has been fixed.

Changed

• Output directory structure has been reorganised, and simplified slightly.

Developers

• Code coverage increased to 58%.
• A lot of the code has been refactored, and (hopefully simplified).
• Log files will now print more information (e.g. information about the git repository, and internal parameters).

Version 0.2.3a (2019-07-01)

Main Updates

• Registration can now handle data in any orientation. The default is coronal, and in the NifTI standard orientation (origin is the most ventral, posterior, left voxel). If your data does not match this, use the --orientation, --flip-x, --flip-y and --flip-z tags.

Added

• There is now a standalone registration command, cellfinder_register. This will be the maintained python version of aMAP going forward.
• There is now an option --remove-intermediate to remove all intermediate files generated by cellfinder. Use with caution.

Fixed

• All downsampled data, registered atlases and cell locations are in the same coordinate space as the raw data (although possibly scaled). This fixes issues #22 and #23, and allows overlay of the cell positions on the downsampled data from the registration step.

Changed

• Cube extraction will now use as many CPU cores as available, slightly speeding up this step. However, if you don't have much RAM (or a lot of CPU cores) see #36.
• Input data paths no longer need to end in the channel number. For channel referencing purposes, this will be set automatically, or can be overridden with --signal-channel-ids and --background-channel-id (or --channel-ids for cellfinder_gen_cubes).
• By default, registration will now save downsampled versions of all channels. if you don't want these files, use --no-save-downsampled.
• Cube extraction will now default to extracting cubes of 50um x 50um x 100um. This can be overridden with --x_pixel_mm_network and --y_pixel_mm_network. Currently there is no rescaling in z.
• Cube extraction now supports the output of ROI sorter, rather than just xml.

Developers

• Code coverage increased to 32%.
• Logging now includes diagnostic information.

Version 0.2.2a (2019-06-17)

Main Updates

• Cellfinder can now be installed with a single command:

    pip install git+https://github.com/adamltyson/cellfinder

• The atlas will now no longer be downloaded automatically upon installation of cellfinder, but on the first use of the atlas. To download in advance, cellfinder_download can be used.

Added

• New function to load .nrrd files (in cellfinder.tools.brain.io.load_any).
• Data can now be passed as a directory of 2D .tif files (rather than just a text file of file paths).
• If a network was trained with a cubes of a different pixel size to that of the input data, then the pixel spacing of the output cubes can be changed (e.g. using: --x-pixel-mm-network).
• Cell (and candidate) positions can be saved as .csv (as well as .xml), by using --save-csv.
• cellfinder_xml_crop has been added. This curates an input .xml file and outputs a file with only those positions within given anatomical areas. Currently, the cells, and the atlas need to be in the same anatomical space.

Fixed

• Bug which caused cellfinder to hang when using part of a node managed by

  SLURM is fixed.

• Bug fixed which prevented cellfinder_run from progressing if a cube directory was

  present, but empty. Thanks to ablot.

• Bug fixed which prevented the checkpoint file from a previously trained network being copied to the output directory. Thanks to ablot.

Changed

• If a cube cannot be extracted for any reason (such as proximity to the edge of an image), the default is now to not save it (rather than save an empty cube in it's place). This can be overridden with --save-empty-cubes.
• Error messages when a cube cannot be extracted (due to proximity to the edge of the image) have been clarified.

Developers

• Code is now automatically formatted with Black, which requires Python >= 3.6 to run.
• Testing has been added (although currently there is only one test).
• All commits & pull requests will be build by Travis.
• For development, cellfinder should be installed using pip install -e .[dev]in the repository.

Version 0.2.1a (2019-05-09)

Main Updates

• Simple summary information can be generated with the --summarise flag. This will run registration (if it wasn't run already) and associate each cell output from the cell detection step with a brain region. This info is saved as a csv file.

• The mouse brain atlas can be downloaded to any directory when installing cellfinder using the --atlas-install-path flag. This flag can also be used to point to an existing atlas installation (e.g. from aMAP).

  This will update the default config file so the correct atlas is sourced (and a second isn't downloaded.)

• cellfinder should know what parts have already been run, so if you re-run a cellfinder_run command, it won't repeat any parts of the pipeline.

Added

• PDF documentation
• cellfinder_gen_cubes function to generate tiff cubes from an xml file independent of the rest of cellfinder. Useful for generating training data.
• cellfinder_count_summary function to combine a brain registered to the allen atlas with cell counts. Generates a csv file of cells per region.
• cellfinder_region_summary function to organise (align by brain area) csv files of summary cell counts from multiple brains.
• cellfinder_view_cells2D function to view cells overlaid on raw data. Uses hardlyany RAM, but I recommend ROI sorterinstead.
• cellfinder_view_3D function. A very rudimentary 3D viewer, but can load any file type in use by cellfinder.
• Some API docs

Fixed

• Logging that interfered with multiprocessing when the --verbose tag was used has been emoved. This was an issue in case cellfinder was stopped in the middle of cell classification as it could cause GPU memory to not be released (and not show up in nvidia-smi).

Changed

• Documentation moved from pydocmd to sphinx.
• Installation requires configobj (as well as Cython).
• Install with --dev if you want to build the documentation yourself.

Version 0.2.0a (2019-04-30)

Main Updates

• Can run cell detection on N-channels
• Streamlined training of cell classification network via cellfinder_train
• Includes (rudimentary) integration of aMAP for registration to the Allen brain atlas

Added

• Further options to only run parts of the analysis
• Option --tensorboard to launch tensorboard automatically during training
• Entry point cellfinder_view to launch the (very simple) image/cell viewer
• Docs generated via pydocmd

Fixed

• Output of cell candidate detection step cells.xml can now be opened in Roi Sorter

Changed

• Docs generated by pydocmd means that installation info is currently here

Version 0.1.0a (2019-04-12)

Main Updates

First version compatible with NiftyNet 0.5.0, TensorFlow 1.12.0 and therefore CUDA 9/10.

Added

• Option to specify external configuration file for cell candidate classification
• All results saved to a single directory
• Option to keep or discard bright spots classified as artifact during cell candidate detection
• Log file

Fixed

• Z positions of cell candidates when analysing a z-subset of the data fixed in the resulting .xml file
• Cell candidate detection .xml file only provides positive candidate types <Type>1</Type> for compatibility with Roi Sorter

Changed

• No longer relies on NiftyNet model zoo (although this may return)
• Much simplified installation (no manual cp/mv)
• Moving towards a single API, calls to n_count_python etc. broken
• Multiprocessing compatible with shared resources (e.g. HPC job schedulers)

Notes

• Developed with Python 3.5, but should work with 3.6/3.7. no Python 2 compatibility
• HPC compatibility only tested with SLURM, but should work with LSF & SGE

Version 0.0.1a (2019-01-23)

Final version compatible with niftynet 0.2.2 and tensorflow 1.4.1 (and therefore CUDA 8).