{"name":"napari-training-assistant","display_name":"Training Assistant","visibility":"public","icon":"","categories":[],"schema_version":"0.2.1","on_activate":null,"on_deactivate":null,"contributions":{"commands":[{"id":"napari-training-assistant.make_training_widget","title":"Training Assistant","python_name":"napari_training_assistant._workflow_widget:TrainingAssistantWidget","short_title":null,"category":null,"icon":null,"enablement":null}],"readers":null,"writers":null,"widgets":[{"command":"napari-training-assistant.make_training_widget","display_name":"Training Assistant","autogenerate":false}],"sample_data":null,"themes":null,"menus":{},"submenus":null,"keybindings":null,"configuration":[]},"package_metadata":{"metadata_version":"2.4","name":"napari-training-assistant","version":"0.3.0","dynamic":["license-file"],"platform":null,"supported_platform":null,"summary":"Persistent SAM3-to-U-Net training project workspace for napari.","description":"# napari-training-assistant\n\nA napari plugin that treats a Training Project Folder as the persistent workspace\nfor a SAM3-to-U-Net training loop.\n\nThe project folder owns accepted image/mask pairs, dataset manifests, training\nsettings, auto-configuration decisions, checkpoints, benchmark history,\nprediction outputs, and logs.\n\n## Current Scope\n\nVersion 0.3.0 implements the persistent project workspace, SAM3 preview\nannotation flow, Model Task management, task-scoped dataset storage, U-Net\narchitecture configuration, working PyTorch U-Net training, checkpoint history,\nimported starting-weight tracking, prediction output storage, and dataset mask\npreparation.\n\nThe `Train U-Net` action now runs the project training pipeline, builds patch\ndatasets from the active Model Task, trains a U-Net with PyTorch, saves the best\nmodel checkpoint, and records run metadata, metrics, benchmark history, and\ntraining history in the project folder.\n\n\n## Installation\n\nFor the full SAM3-to-U-Net workflow, first install and validate SAM3 support by following\nthe `napari-sam3-assistant` installation guide:\n\n```text\nhttps://github.com/wulinteousa2-hash/napari-sam3-assistant\n```\n\nUse that README as the reference for installing:\n\n- `napari-sam3-assistant`\n- the SAM3 Python backend\n- local SAM3 model weights\n- compatible CUDA/PyTorch dependencies for SAM3 and SAM3.1 multiplex workflows\n\nAfter `napari-sam3-assistant` is working in your napari environment, install\n`napari-training-assistant` into the same environment:\n\n```bash\ngit clone https://github.com/wulinteousa2-hash/napari-training-assistant.git\ncd napari-training-assistant\npip install -e .\n```\n\nStart napari and open the plugin from the napari Plugins menu:\n\n```bash\nnapari\n```\n\nInside `napari-training-assistant`, select the SAM3 model folders in the\n**SAM3** tab:\n\n- 2D SAM3 modes require a SAM3.0 image model folder containing `sam3.pt` or\n  `model.safetensors`.\n- SAM3.1 3D/multiplex mode requires a SAM3.1 model folder containing\n  `sam3.1_multiplex.pt` and CUDA.\n\nU-Net training from existing image/mask pairs can run without using SAM3 during\nthe workflow. However, for the full SAM3-assisted mask-generation workflow, this\nplugin expects `napari-sam3-assistant` and a working SAM3 installation to already\nbe available in the same environment.\n\n## Dependencies\n\nFor the full workflow, this plugin should be installed into the same environment\nwhere `napari-sam3-assistant` and SAM3 already work. The SAM3 installation guide\nis intentionally not duplicated here because SAM3 setup depends on OS, CUDA,\nPyTorch, and model version.\n\nThe training assistant itself uses:\n\n- `napari` and `qtpy` for the dock widget UI\n- `numpy` for array handling\n- `tifffile` and `Pillow` for image I/O\n- `dask`, `zarr`, and `ome-zarr` for OME-Zarr loading\n- `torch` for U-Net training and inference\n- `napari-sam3-assistant` for SAM3-assisted prompt collection, SAM3.1 multiplex\n  handoff, and napari layer writing\n- a working SAM3 backend and local SAM3 model folders for SAM3 preview and\n  propagation\n\nPractical rule:\n\n- Follow the `napari-sam3-assistant` README first.\n- Confirm SAM3 works there.\n- Then install `napari-training-assistant` in the same environment.\n\n## User Interface\n\nThe dock widget is organized as a tabbed workflow with a compact project status\nbar at the top.\n\nAlways visible:\n\n- Project selector\n- Short project path\n- Project state\n- Dataset count\n- Latest checkpoint\n- Latest benchmark summary\n\nTabs:\n\n- **SAM3**: configure SAM3 model folders, choose a prompt mode, auto-create\n  prompt layers, prepare preview labels, and accept preview masks into the\n  persistent dataset.\n- **Dataset**: choose image/mask layers, prepare masks, add accepted pairs,\n  reopen selected pairs, and inspect the compact dataset table.\n- **Train**: choose training mode, dataset source, starting point, and core\n  training parameters.\n- **Checkpoints**: inspect checkpoint history and choose a checkpoint for\n  continued training.\n- **Predict**: save prediction layers into the project.\n- **Advanced**: edit U-Net architecture, import pretrained weights, and update\n  project notes.\n\nTraining-related actions are disabled until the user selects or creates a\nTraining Project Folder.\n\n## Model Task Workflow\n\nA Model Task defines one segmentation target inside a Training Project. Examples\ninclude `myelin + background`, `axon + background`, or a multiclass task such as\n`background + myelin + axon`.\n\nEach Model Task keeps its own dataset manifest, copied image/mask pairs,\ncheckpoints, predictions, benchmark history, and training-run history. This keeps\nseparate segmentation goals from mixing their data or model outputs.\n\nThe Model Task bar supports:\n\n- creating a fresh task\n- duplicating a task configuration\n- renaming a task\n- switching the active task\n- importing an existing paired image/mask dataset into the active task\n\nThe active Model Task controls which dataset pairs are used for training and\nwhere new checkpoints and predictions are saved.\n\n## Training Project Folder\n\nThe Training Project Folder is the persistent workspace. Users should select or\ncreate it before adding masks, training U-Net, or saving predictions.\n\nExpected structure:\n\n```text\ntraining_project/\n    project_config.json\n\n    architecture/\n        architecture_config.json\n\n    sam3/\n        sam3_config.json\n\n    tasks/\n        tasks.json\n\n        default_binary/\n            task_config.json\n\n            dataset/\n                images/\n                masks/\n                manifest.json\n\n            checkpoints/\n                checkpoints.json\n                latest.pt\n\n            predictions/\n                prediction_001.tif\n                prediction_002.tif\n\n            history/\n                training_runs.json\n                benchmark_history.csv\n                unet_runs/\n                    unet_run_001_YYYYMMDDTHHMMSSZ/\n                        best_model.pt\n                        config.json\n                        summary.json\n                        history.csv\n\n    models/\n        imported/\n        model_registry.json\n        starting_weights_config.json\n\n    checkpoints/\n        latest.pt\n\n    logs/\n        training.log\n```\n\nThe root-level project files preserve global settings. The `tasks/` folder owns\nthe active training data, task-specific checkpoints, predictions, and run\nhistory.\n\nReopening the same project restores Model Tasks, dataset history, training\nsettings, U-Net architecture settings, starting-weight choice, checkpoint\nhistory, latest checkpoint pointer, benchmark history, and previously selected\nlayer names.\n\n## SAM3 Annotation Tab\n\nThe SAM3 tab is task-driven. The user selects an image layer and a prompt mode;\nthe plugin creates or reuses the expected napari prompt layer automatically.\n\nSupported prompt modes in the compact UI:\n\n- **2D box**: creates/selects `SAM3 boxes` as a Shapes layer in rectangle mode.\n- **2D points**: creates/selects `SAM3 points` as a Points layer.\n- **Live points**: creates/selects `SAM3 live points` for immediate\n  point-driven preview.\n- **2D exemplar**: creates/selects `SAM3 exemplar boxes` as a Shapes layer.\n- **3D / multiplex**: creates/selects `SAM3 3D prompts` for SAM3.1 multiplex\n  propagation.\n\nModel folder expectations:\n\n- 2D modes use a SAM3.0 image model folder containing `sam3.pt` or\n  `model.safetensors`.\n- 3D / multiplex mode uses a SAM3.1 model folder containing\n  `sam3.1_multiplex.pt`.\n- CPU is only treated as valid for 2D mode. SAM3.1 multiplex requires CUDA.\n\nThe SAM3 tab prepares layers, validates model folders, runs SAM3 previews, and\nwrites results to `SAM3 preview labels` for 2D modes or `SAM3 propagated labels`\nfor SAM3.1 multiplex propagation. SAM3.1 multiplex uses CUDA and delegates the\nvideo propagation flow to `napari-sam3-assistant`; frame results are queued and\nwritten back to napari labels through a Qt timer so layer refreshes do not\nthrottle propagation. The **Accept preview to Dataset** button routes the\npreview through the same persistent mask-preparation path as the Dataset tab.\n\n## U-Net Architecture\n\nThe trainable model is a U-Net-family segmentation model. SAM3 is treated as an\nannotation source for generating candidate masks, not as the trainable model.\n\nThe default backend is:\n\n```json\n{\n  \"backend\": \"basic_unet\",\n  \"spatial_dims\": \"2d\",\n  \"preset\": \"standard_unet\",\n  \"depth\": 4,\n  \"base_channels\": 32,\n  \"normalization\": \"batch\",\n  \"upsampling\": \"transpose\",\n  \"input_channels\": 1,\n  \"output_mode\": \"binary\",\n  \"num_classes\": 2,\n  \"output_channels\": 1,\n  \"activation\": \"sigmoid\",\n  \"loss\": \"bce_dice\",\n  \"threshold\": 0.5\n}\n```\n\nFor the default 2D U-Net:\n\n- Feature channels are `[32, 64, 128, 256, 512]`.\n- There are 4 encoder levels, 1 bottleneck, and 4 decoder levels.\n- Each block uses 2 convolution layers.\n- The architecture has 18 main convolution layers, 4 upsampling layers, and 1\n  final projection layer.\n\nThe schema already records dimensionality so future 3D U-Net support can be\nadded without changing the project folder structure. The UI currently marks 3D\nU-Net options as future/unsupported.\n\n## Binary And Multiclass Masks\n\nBinary mode is the default:\n\n- `0` means background.\n- nonzero source labels can be merged into `1` foreground.\n- model output has 1 channel.\n- prediction uses sigmoid plus threshold.\n\nMulticlass mode is supported in the project schema:\n\n- masks should contain integer labels from `0` to `num_classes - 1`.\n- model output channels equal `num_classes`.\n- prediction uses channel argmax.\n\n## Mask Preparation Quick Fix\n\nSAM-style masks may contain multiple instance IDs for the same semantic object:\n\n```text\n0 = background\n1 = object instance A\n2 = object instance B\n3 = object instance C\n```\n\nFor binary U-Net training, the default quick fix is:\n\n```python\nmask = (mask > 0).astype(\"uint8\")\n```\n\nThis saves the training mask as:\n\n```text\n0 = background\n1 = foreground\n```\n\nThe dataset manifest records the source labels, saved labels, target class name,\nand label transform so the conversion is auditable.\n\n## PyTorch Training Output\n\nThe `Train U-Net` action runs PyTorch training for the active Model Task. The\ntraining runner builds a patch dataset from selected image/mask pairs, trains\nwith the configured U-Net settings, saves the best model state, and registers a\ncheckpoint in the task history.\n\nFor each U-Net run, the plugin writes:\n\n- `best_model.pt` for the best validation checkpoint\n- `config.json` for the resolved run configuration\n- `summary.json` for image count, patch count, best epoch, Dice, and IoU\n- `history.csv` for per-epoch loss and metric history\n- task checkpoint metadata and benchmark history\n\nCurrent supported production path: 2D U-Net training. 3D U-Net code paths are\nreserved for future expansion and should be treated as experimental until the\nfull UI workflow is validated.\n\n## Starting Weights\n\nSupported starting-weight choices in the UI:\n\n- Train from scratch\n- Continue from latest project checkpoint\n- Continue from selected project checkpoint\n- Start from imported pretrained U-Net\n\nImported `.pt` and `.pth` files are copied into `models/imported/` and tracked\nin `models/model_registry.json`. A PyTorch checkpoint alone is not considered\nself-describing; the project also stores the architecture config used with that\ncheckpoint.\n\nCheckpoint compatibility is checked against:\n\n- backend\n- dimensionality\n- preset\n- depth\n- base channels\n- normalization\n- upsampling\n- input channels\n- output mode\n- number of classes\n- output channels\n\n## Checkpoint Metadata\n\nEvery successful training run creates a new numbered checkpoint. Previous\ncheckpoints are not overwritten by default. `latest.pt` is only updated after a\nsuccessful checkpoint registration.\n\nEach checkpoint records:\n\n- checkpoint ID\n- parent checkpoint ID\n- training mode\n- dataset pair IDs used\n- image and patch counts\n- train/validation split\n- loss metrics\n- Dice/IoU metrics, when available\n- timestamp\n- checkpoint path\n- architecture snapshot\n- starting weights snapshot\n- architecture summary\n\n## Development Notes\n\nThe default U-Net descriptor and lazy PyTorch builder live in\n`src/napari_training_assistant/unet.py`. The project-level training runner lives\nin `src/napari_training_assistant/unet_backend/project_runner.py` and connects\nthe active Model Task to patch dataset creation, PyTorch training, run-output\nfiles, checkpoint registration, and benchmark history.\n\nSAM3.1 multiplex behavior is intentionally delegated to `napari-sam3-assistant`\nfor prompt collection, adapter behavior, video-session semantics, and napari\nlayer writing. `napari-training-assistant` uses SAM3 as an annotation source and\nkeeps U-Net training as the persistent trainable-model workflow.\n","description_content_type":"text/markdown","keywords":null,"home_page":null,"download_url":null,"author":"Wulin Teo","author_email":null,"maintainer":null,"maintainer_email":null,"license":"MIT","classifier":["Framework :: napari","Intended Audience :: Science/Research","License :: OSI Approved :: MIT License","Programming Language :: Python :: 3","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Topic :: Scientific/Engineering :: Image Processing"],"requires_dist":["napari>=0.5.0","qtpy","numpy","tifffile","pillow","zarr","ome-zarr","dask[array]","torch"],"requires_python":">=3.10","requires_external":null,"project_url":null,"provides_extra":null,"provides_dist":null,"obsoletes_dist":null},"npe1_shim":false}