* docs: adding draft QC docs

* docs: adding more qc docs

* feat: add myst_parser and qc to index

* refactor: replace .rst file with .md

* docs: adding information about reference URLs

* docs: separating portal information into an AIND section

* docs: cleaning up documentation for qc

* docs: more cleanup on qc docs

* docs: small updates to docs based on discussion about credentials

* docs: updating qc docs based on discussion

* docs: cleaning up docs and clarifying in a few spots

* feat: adding optional ResourceUsage to DataProcess

* tests: adding example

* refactor: adding ResourceTimestamped

* chore: remove timeunit

* tests: adding tests

* fix: missing Field()

* chore: lint

* chore: lint

* tests: examples

* chore: lint

* refactor: date->datetime and usage_unit

* chore: update processing example

* chore: lint

* fix: datetime -> AwareDatetime

* refactor: rename memory to system_memory

* chore: lint

* tests: force timezones to utc for CI

* fix: timezone issue for tests

* refactor: remove MemoryValue and add validators for Optional unit fields

* refactor: remove file_io

* chore: lint

* chore: lint

* refactor: remove file_io

* refactor: clean up validators into a single root validator

* chore: lint

* tests: fix examples, also allow float memory sizes

* tests: update example file

* chore: lint

…sed on metric states (#1074)

* refactor: changing references -> reference

* feat: add an overall evaluator name

* feat: adding QCStatus and lists of status for versioning

* chore: lint

* tests: updating tests for QCStatus

* feat: add QCStatus list to metrics

* feat: adding a function to automate setting the evaluation_status field

Coverage not at 100% yet
Also replaced date with datetime in QCStatus

* test: test coverage for overall_status

* test: add coverage for properties

* refactor: date -> datetime

* chore: lint

* doc: missing docstrings

* tests: remove unnecessary test

* feat: auto-set status when never previously evaluated, and tests

* fix: fixing an issue with private attributes

You can't do what I wanted (hide the _evaluation_status list) because pydantic doesn't serialize the field. So we have to keep the history exposed but we'll have to encourage people not to access it directly

* chore: example and lint

* fix: removing default_factory, trying to allow timestamp pass through

* test: fixing examples test timestamp issue

* chore: lint

* refactor: datetime -> AwareDatetime and remove default

* tests: add a test that confirms that order is preserved

* fix: missing timezones for example

* feat: add option to allow failures in metrics

* feat: add a convenience function to get metrics that are failing

* tests: update qc.json example

* feat: ensure metric_status_history not empty with validator

* chore: docstring

* test: improving examples

* chore: lint

* feat: add asset_id field to qc

* tests: fix examples

* tests: fix examples

* docs: extended desc for allowing failed metrics

* feat: removing prefixes to clean up fields

* feat: remove property prefixes

* chore: lint

* chore/tests: lint and fix examples

* tests: fix refactor issues in tests

* docs: fix typo

* chore: unused import snuck in?

* fix: allow none to return from failed_metrics

* ci: fix for python 3.8 failures

* fix: remove the confusing "Automated" evaluator

* tests: fix qc example

* refactor: removing redundant status_history from QualityControl and QCEvaluation

* fix: forgot to remove redundant fields

* tests: fix examples

* build: updates version number

* feat: adds qc schema json

* fix: updates schema version check to handle new core schemas

* ci: adds service account token to github actions

* bump schema version [skip actions]

* fix: updates broken github action

* bump schema version [skip actions]

* build: increments version to 1.0.0

* build: sets core metadata schema to v1.0.0

* chore: version bump

* bump schema version [skip actions]

* chore: version bump for schemas (#1094)

* chore: version bump for schemas

* tests: example updates

* bump schema version [skip actions]

---------

Co-authored-by: Jon Young <104453205+jtyoung84@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

…xpected modalities (#1084)

* feat/refactor: add validator for required/excluded files

Also refactors CORE_FILES into a separate variable

* refactor: remove redundant validators

* fix: properly handle modalit*ies* list, fixing tests

* chore: lint

* tests: adding excluded file test

* feat: adding support for multi-modal records

Prioritize required > optional > excluded for files

* fix: partial fix for modalities with underscores

* fix: fixing underscore issues and adding test coverage

* tests: fixing underscore test

* tests: Value->ValidationError

* fix: modality.abbreviation tracked correctly

Added modality into requirement_dict to track it for throwing errors

* chore: lint

* chore: lint

example file generated in the root folder by accident

* chore: add default= to schema_version

* chore: add default= for URL

…1106)

* feat: adding provenance

* chore: lint

* feat: adding multi-asset stage and typo

* chore: examples

* refactor: adding validator and renaming

tests are still todo

* tests: adding tests, minor fixes

* tests: examples and line length

* fix: multi_session->multi_asset

* fix: session suffix to asset

* fix: typo

* fix: more typos