- allows dataset registration on errored task
- ks2task import conflict fix
- Ks2 task does not depend on ephys pulses
- Ephys mtscomp
- ks2 task registers first probe even if one failing
- mtscomp task register .ch and .meta even if .cbin doesn't exist
- move ibllib tests to tests_ibllib
- brainbox atlas plot functions
- Ephys extraction:
- synchronisation between probes computed in the ephysPulses job
- spike sorting resync done directly after KS2 output
- unit-based metrics have their own task
- create local server tasks only on raw_session.flag
- ephys alignment QC
- hotfix: ibl errors inherit Exception, not BaseException
- hotfix: partial qc task extractor keeps FPGA stim times
- create tasks looks for create_me.flags
- add Karolina's optogenetics tasks for extractions
- histology probe QC pipeline and final locations dataset export
- numpy needs upgrading >= 1.18
- session creation skips alyx procedure for unknown task protocol (custom projects)
- task extraction:
- Habituation QC
- ephys extraction StimOffTimes fix
- ibllib.atlas
- allen csv Atlas part of package is not installed in dev mode
- improved slicing performance
- ephys extraction: mtscomp registers ch file on run and re-runs bis
- mtscomp registers ch file on run and re-runs
- ffmpeg nostdin option as jobs were stopped in background on a server
- QC base class
- Support for task QC on FPGA data
- TaskQC run during task extraction
- local server: catches error when subject is not registered in Alyx
- ibllib.atlas.AllenAtlas
- re-ordered the volumes in c-order contiguous ml-ap-dv efficient coronal shapes
- top/bottom surface extraction
- Ephys extraction SyncSpikeSorting: specify different dir for ks2 ouput and raw ephys data
- Ephys extraction ks2: mkdir for scratch more robust
Ephys extraction bugfixes:
- RawEphysQC: No object "ephysTimeRmsAP" found
- EphysMtsComp,RawEphysQC, EphysPulses : ValueError: mmap length is greater than file size
- Ks2: ks2 job cleans-up temp dir
- ONE offline mode and cache dataset table to speed up reloading of large datasets (Olivier)
- ALF io naming conventions on loading objects (Miles)
- KS2 Matlab ephys pipeline tasks (Olivier)
- Support for running QC on biased and training sessions (Nico)
- metrics_df and passed_df properties in BpodQC obj for qcplots (Nico)
- Added missing unittest to stim_move_before_goCue metric (Nico)
- ibllib.pipes.training_preprocessing.TrainingAudio
- ibllib.pipes.training_preprocessing.TrainingAudio: returns files for registration and proper status. (Olivier)
- ibllib.atlas: compute nearest region from probe trajectory (Mayo)
- Local server jobs:
- fix wheel moves size mismatch extractor error
- only look for raw_session.flag for ephys extraction to avoid race conditions
- Ephys extraction:
- spike amplitudes in Volts
- added waveforms samples dataset to use Phy from Flatiron datasets
- ONE performance:
- Metaclass implementation of UniqueSingletons for AlyxClient
- Multi-threaded downloads
- Added JSON fields methods to AlyxClient
- QCs: Bpod and ONE QC features, basic plotting, examples
- Added permutation test, comparing a metric on two sets of datasets, by shuffling labels and seeing how plausible the observed actual difference is Sped up calculation of firing_rate
- ephys extraction: updated extracted metrics, including a new contamination estimate and drift metrics.
- ibllib.io.spikeglx
- Hotfix: rig transfer - create probes. One variable used before assignation.
- ONE.load overwrites local file if filesizes different or hash mismatch
- ephys extraction:
- registration sets the session.procedure field to acute recording
- bugfix synchronization on re-extraction: always recompute spike.times from spike.samples
- ephys registration sets the session.procedure field to acute recording
- training extraction:
- added biasedVisOffChoiceWorld as training extractor
- wheel data
- dropping support for wheel velocity, not extracted anymore
- bugfix: Include sessions data files for ephys mock
- bugfix: Single probe 3B gets synchronized
- bugfix: Include sessions data files in pip package
- Big brainbox merge and release
- bugfix: clusters.metrics spiking rates accurate
- probability left for ephys choice world contain generative probabilities, not outcomes
- ONE Light Windows fixes
- Installation documentation separates conda and virtualenv options
- Conda yaml environement file for ibllib
- ONE Light for behaviour paper data release
- ONE() standard syntax matching the one light examples
- Alyx registration: add md5, version and filesize to the pipeline registration
- Data Patcher: allows to register data from anywhere through FTP/SSH/GLobus
- ONE Light for behaviour paper data release
- Ephys extraction: left probability bug when sequence was 0 - fixed
- Ephys extraction:
- left probability extracted properly
- add robustness to audio fronts extraction in FPGA
- Passive stimulus: raw data registered in pipeline
- Training extraction: microphone extraction for habituation sessions
- ALF: specific to_dataframe method for Bunch
- Ephys extraction: handle fringe case where recording is interrupted in the middle
- Wheel extraction: if rotary encoder version is outdated and stores data in the wrong unit, auto-detect and output in seconds even for new versions
- FPGA/bpod events synchronization performed even when their counts do not match
- Updated requirement versions for mtscomp and phylib
- wheel extraction outputs a timestamps attribute, not times
- make the wheel extraction more robust
- Ephys extraction:
- un-synchronized spike sortings not uploaded on flat-iron
- reaction times extracted
- valve-open times bugfix
- Wheel extraction:
- training wheel position and timing are now correct
- ephys & training: units: radians mathematical convention
- ONE:
- Alyx client handles pagination seamlessly
- cross-platform get of session folder for rig computer copy to server
- spikeglx.verify_hash() method to check file integrity after transfers/manipulations
- create wirings settings files on ephys computer transfer to server
- Ephys Extraction:
- duplicate probe.trajectories bugfix
- extraction works with unoperational fram2ttl at beginning of ephys session
- clusters.metrics.csv has consistent size with npy cluster objects
- ephys transfer: create ephys extraction flags after the transfer is complete
- Rename spike.times on failed sync to reflect the clock as per ALF convention
- sync 3A fails if first cam event whithin 200ms of start
- compress ephys goes through a tempfile to not interfere with transfers/globbing
- Ephys Extraction (phylib)
- convert ks2 amplitudes to volts for spikes.amps, clusters.amps. templates.waveforms, clusters.waveforms to get uV
- generates Cluster UUIDs file
- individual spike depths computed from PC features
- Ephys Synchronization
- use frame2TTL split for 3A by default, if not found look for right_camera
- output individual probe sync in ALF timestamps format
- registration ignores ks2alf probes subfolders
- fix typo in raw qc dataset types
- Alyx registration adds the relative path to session root as dataset.subcollection
- Ephys extraction:
- split probe folders output alf/probe00 and alf/probe01 instead of merge
- outputs templates.waveforms and clusters.waveforms in sparse arrays
- outputs probes.description and probes.trajectory
- renamed the raw ephys QC output
- outputs clusters.metrics
- Bugfixes:
- 3B raw ephys QC output ap.file not found on nidq object
- 3A sync probe threshold set to 2.1 samples
- transfer scripts from ephys/video/rig computers to local servers
- bugfix spigeglx.glob_ephys_files when metadata file without ap.bin file
- Transfer rig data takes into account session type (ephys/training) to create flags
- Ephys video compression in pipeline
- Ephys audio compression in pipeline
- Ephys extraction: provide full 3B default wirings if files do not exist.
- Ephys extraction: merge sync ephys in the pipeline overwrites ks2_alf directory if it already exists
- Added
biasedScanningChoiceWorldtask to biased extractor for Zador lab
spikeglx.Readersupports mtscomp ephys binaries- server pipeline for compression of ephys files
brainbox.singlecell.pethswith testsbrainbox.processing.bincount2Dsupports aggregation on fixed scale- simple examples script and notebook
- examples/brainbox/plot_peths.py: by Matt. W.
- examples/brainbox/rasters by Michaël S.
- Allen Atlas framework and probe registration base functions
- server pipeline for audio extraction of training sessions