{"name":"napari-zenapi-streaming","display_name":"ZEN API Streaming","visibility":"public","icon":null,"categories":[],"schema_version":"0.3.0","on_activate":null,"on_deactivate":null,"contributions":{"commands":[{"id":"napari-zenapi-streaming.start_streaming","title":"Start ZEN Streaming","python_name":"napari_zen_streaming._widget:create_zen_widget","short_title":null,"category":null,"icon":null,"enablement":null}],"readers":null,"writers":null,"widgets":[{"command":"napari-zenapi-streaming.start_streaming","display_name":"ZEN API Streaming","autogenerate":false,"default_dock_area":"right","tabify":false}],"sample_data":null,"themes":null,"menus":{},"submenus":null,"keybindings":null,"configurations":{}},"package_metadata":{"metadata_version":"2.4","name":"napari-zenapi-streaming","version":"0.0.1","dynamic":["license-file"],"platform":null,"supported_platform":null,"summary":"ZEN API streaming plugin for napari","description":"# napari-zenapi-streaming\n\n`napari-zenapi-streaming` is a napari plugin and Python app that streams live\nmicroscopy image data from ZEISS ZEN Blue into napari via the ZEN API (gRPC).\n\nIt supports three workflows selectable from the plugin UI:\n\n- **Display: watch ZEN experiment** – passively displays an experiment\n  started in ZEN Blue. Channel choices are All or 1-16, independent of the\n  selected setup; its expected frame count is unknown.\n- **Display: start selected experiment** – starts the selected setup from\n  napari and displays its frames. Channel choices match the setup's XML\n  channel count.\n- **OME-ZARR only** – writes frames directly to an OME-ZARR file on-the-fly\n  using the standalone writer.\n\nDisplay workflows use the same live reader and restructure received frames\ninto per-channel scene mosaics. Channel labels are 1-based; the ZEN API filter\nuses zero-based indices. The filter defaults to All channels when a\npreviously selected channel is not available in the new mode or experiment.\n\nA **standalone CLI** (`ZEN_stream2omezarr.py`) is also available to stream\ndirectly to OME-ZARR without napari.\n\n## Current status\n\n- Project phase: alpha / MVP\n- Primary OS target: Windows for ZEN Blue and the ZEN API Gateway\n- Python target: `>=3.12, <3.15`\n\n## Known limitations\n\n- ZEN Blue and the ZEN API Gateway are required for live acquisition.\n  Automated tests use simulated responses; they do not establish end-to-end\n  throughput or frame completeness on real hardware. ZEN Blue and its gateway\n  run on Windows.\n- Running napari on a separate Linux host may be possible, but this remote\n  topology is experimental and not officially supported. It requires a fast,\n  reliable connection to the Windows gateway and expertise configuring gateway\n  certificates and trust for the remote client; it has not been validated\n  here.\n- The OME-ZARR writer accepts only 2D `GRAY8` and `GRAY16` frames. Color\n  pixel types are not supported by that output path.\n- ZEN-started Display mode monitors the global pixel stream without a known\n  experiment ID; run only one acquisition at a time in this mode.\n- Display mode retains received frames in memory for final scene assembly.\n  Very large or long acquisitions may exceed available RAM; the standalone\n  unconfigured `--experiment` CLI mode also buffers the acquisition.\n- Out-of-order OME-ZARR frames use a bounded pending buffer. If the stream\n  stays incomplete until the inactivity timeout, missing positions are\n  skipped rather than recovered from ZEN. The configured writer retains the\n  partial store, writes `acquisition_status.json`, and reports failure\n  instead of success. Inspect the report before using partial output.\n- Display mode currently assumes `uint16` input and normalizes every frame\n  for preview. Its restructured layers are not raw quantitative image data.\n  Use configured OME-ZARR output to retain raw GRAY8/GRAY16 intensities.\n\n## Features\n\n- Live frame ingestion through `ExperimentStreamingService` (gRPC).\n- **Display mode:** Buffered async pipeline (`reader → queue → processor`)\n  for smooth UI updates, sparse frame storage, automatic post-acquisition\n  restructure from a 7D streaming layer to 5D `S,T,Z,Y,X` scene mosaics\n  per channel (merging the `M` tiles spatially).\n- **OME-ZARR mode:** Dedicated gRPC channel, tight read loop, on-the-fly\n  writes via `ome-writers`. Identical code path for both plugin and CLI.\n- Acquisition dimensions and Z spacing are derived automatically from the\n  selected experiment XML. Changing the selection refreshes these values.\n- OME-ZARR dtype is derived from the runtime frame `PixelType`; the raw\n  payload size is checked before decoding.\n- Incoming frames are assigned from live stream metadata. In particular,\n  `FramePosition.s` and `FramePosition.m` select the scene and tile; frame\n  arrival order and XML TileRegion order are not used for assignment.\n- Thread-safe napari updates via Qt signals/slots.\n- Configurable startup and timeout behavior from `.env` or CLI.\n\n## Repository layout\n\n```text\nsrc/napari_zen_streaming/\n  main.py              # App entrypoint and orchestration\n  ZEN_config.py        # Runtime config dataclasses + env parsing\n  ZEN_init.py          # gRPC connection + experiment lifecycle helpers\n  ZEN_pipeline.py      # Async reader/processor pipeline (Display mode)\n  ZEN_ui.py            # Viewer UI, layer lifecycle, mode switching\n  ZEN_utils.py         # Metadata + image processing functions\n  ZEN_omezarr.py       # OME-ZARR helpers, ExperimentConfig, viewer launchers\n  ZEN_stream2omezarr.py # Standalone OME-ZARR streaming (CLI + plugin backend)\n  _widget.py           # Napari plugin widget entrypoint\n  misc.py              # gRPC channel initialization from config.ini\n  napari.yaml          # Plugin manifest\n  .env                 # Editable-checkout fallback only; not packaged\n\nexperiment_streaming_config.ini  # Experiment config for OME-ZARR streaming\n\ndocs/\n  *.mmd               # Mermaid flowchart and sequence sources\n  *.png               # Rendered diagram preview\n```\n\n## Requirements\n\n- ZEISS ZEN Blue with ZEN API Gateway installed\n- A valid ZEN control token\n- Access to the ZEN API certificate (`ZenApiPersonalSigningRootCA.pem`)\n- Python 3.12-3.14 with Qt >=6.7; PyQt6 is used in CI. PyQt5 is not\n  compatible with napari 0.10 or newer.\n- Authorized, compatible `zen_api` and `zen_api_utils` wheels from\n  [ZEISS OAD](https://github.com/zeiss-microscopy/OAD/tree/master/ZEN-API)\n  (neither package is distributed on PyPI)\n\n## Installation\n\n> **Required before a normal PyPI installation:** `zen_api` and `zen_api_utils`\n> are required dependencies, but pip cannot discover their GitHub wheels on\n> PyPI. On a fresh environment, `pip install napari-zenapi-streaming` will fail\n> with a resolver error such as `No matching distribution found for zen_api`.\n> Install both compatible wheels as shown below, then retry the plugin install.\n> This happens before plugin code runs, so the plugin cannot display a custom\n> installation error or GUI message.\n\n### Install the required ZEN wheels\n\nUse the same Python environment for the wheels, the plugin, and napari. The\ncurrent dependency requirements are:\n\n- `zen_api>=2026.5.1,<=2026.11.1`: choose a version compatible with your ZEN API\n  Gateway.\n- `zen_api_utils>=0.1.0,<0.2`: utilities have their own version number, separate\n  from the API release.\n\nFor example, install API `2026.7.1` and utilities `0.1.0` from OAD:\n\n```powershell\n$base = \"https://github.com/zeiss-microscopy/OAD/releases/download\"\npython -m pip install `\n  \"$base/v2026.07.1/zen_api-2026.7.1-py3-none-any.whl\" `\n  \"$base/zen-api-utils-v0.1.0/zen_api_utils-0.1.0-py3-none-any.whl\"\n```\n\nThe backtick is PowerShell's line-continuation character. Alternatively, pass\nthe paths to your downloaded authorized wheels:\n\n```powershell\npython -m pip install `\n  \"C:\\path\\to\\authorized\\zen_api-<version>-py3-none-any.whl\" `\n  \"C:\\path\\to\\authorized\\zen_api_utils-0.1.0-py3-none-any.whl\"\n```\n\nReplace the example paths and API version with the wheels supplied for your\ngateway. Do not commit wheels, tokens, or certificates to this repository\nwithout confirming redistribution rights.\n\n### Install the plugin\n\nFor a published PyPI release:\n\n```powershell\npython -m pip install napari-zenapi-streaming \"napari[pyqt6]\" \"PyQt6>=6.7,<6.10\"\n```\n\nFrom this checkout, install the plugin:\n\n```bash\npython -m pip install .\n```\n\nFor a downloaded release wheel, pass that wheel path instead of `.`. For\ndevelopment use `python -m pip install -e \".[dev,testing]\"`.\n\nIf multiple Qt bindings are installed, select PyQt6 before launching napari:\n\n```powershell\n$env:QT_API = \"pyqt6\"\n```\n\nThe two ZEN wheel URLs or paths can also be supplied alongside the plugin in\none `python -m pip install` command; installing them beforehand is not required\nwhen they are provided explicitly. Keep the required version constraints in\nthe plugin metadata rather than using `--no-deps` to bypass them.\n\nIf pip still reports a missing ZEN dependency, confirm that both wheels satisfy\nthe version ranges above and were installed with the same `python` executable\nused for the plugin. Run `python -m pip check` after installation to check for\nmissing or incompatible dependencies.\n\n### Conda environment\n\nAlternatively, create the supplied Conda environment first, then install the\ntwo authorized wheels and this plugin as above:\n\n```bash\nconda env create -f python_env/env_zenapi_napari.yml\nconda activate zenapi-napari\n```\n\nThe Conda manifest intentionally does not download the ZEN wheels; supplying\nboth `zen_api` and `zen_api_utils` is required for both installation routes.\n\n## Configuration\n\n### 1) `config.ini` (required)\n\nCreate a user-owned gateway configuration:\n\n```bash\npython -m napari_zen_streaming.ZEN_config --init\n```\n\nThe command prints the new path and never overwrites an existing file. Edit\nthe host, certificate path, and control token before opening the plugin.\nThe default directory is `%APPDATA%\\napari-zenapi-streaming` on Windows and\n`$XDG_CONFIG_HOME/napari-zenapi-streaming` (or\n`~/.config/napari-zenapi-streaming`) elsewhere. Configuration does not belong\ninside `site-packages`.\n\nSet `ZEN_CONFIG_FILE` to use an existing gateway INI instead. The standalone\napp also accepts `--config-file`. Explicit relative paths resolve from the\nworking directory. For source-checkout compatibility, an existing\n`config.ini` in the working directory or checkout root is used when no user\nconfiguration exists. User configuration takes priority over these fallbacks.\n\n```ini\n[api]\nhost = 127.0.0.1\nport = 5000\ncert_file = C:\\ProgramData\\Carl Zeiss\\ZenApiGateway\\Certificates\\ZenApiPersonalSigningRootCA.pem\ncontrol-token = your-control-token\n\n[image_streaming]\nhost = 127.0.0.1\nport = 5280\n```\n\nThe loopback addresses above assume napari runs on the same Windows host as\nZEN Blue. A remote client needs reachable control and image-streaming hosts\nand a certificate configuration appropriate to the gateway hostname. The\ncertificate path shown above is a Windows-local example, not a Linux path.\n\n### 2) Environment variables (optional)\n\nDefine settings in the process environment or a `.env` beside the user-owned\n`config.ini`. Set `ZEN_ENV_FILE` to load a different dotenv file; an explicit\nmissing file is an error. Editable checkouts also retain the package-local\n`.env` as a fallback when no user settings file exists.\n\n- `ZEN_EXP_NAME` – initially selected experiment setup. Default:\n  `ZEN_API_overview`.\n- `ZEN_UI_START` – `true` initially selects napari-started Display; `false`\n  selects passive ZEN watch. Default: `true`.\n- `ZEN_CONFIG_FILE` – path to the gateway configuration. Default: user\n  directory.\n- `ZEN_ENV_FILE` – explicit optional settings file. Default: user `.env`.\n- `ZEN_CHANNEL_INDEX` – gRPC channel index filter (integer). Leave empty or\n  unset to receive **all channels**.\n- `ZEN_RESTRUCTURE_TIMEOUT` – idle timeout in seconds before final\n  restructure in ZEN-started mode. Default: `10.0`.\n- `ZEN_SHOW_ONLY_LAST_FRAME_DURING_LIVE` – use the lightweight 2D\n  latest-frame preview in Display mode. Default: `true`.\n- `ZEN_OUTPUT_DIR` – default output directory for OME-ZARR files. Default:\n  unset.\n- `ZEN_LOG_DIR` – directory containing the rotating `zen_streaming.log`\n  file. Default: platform-specific, see [Logging](#logging).\n- `ZEN_LOG_LEVEL` – `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`.\n  Default: `INFO`.\n\nThe optional `.env` supplies runtime settings such as the\nexperiment name, startup mode, channel filter, display behavior, and the path\nto `config.ini`. It does not contain the ZEN API control token or certificate;\nthose remain in the required `config.ini`.\n\nBoth plugin and CLI preserve existing process environment values. Precedence\nis explicit CLI flags (where supported), process environment, `.env`, then\ncode defaults. Only the selected dotenv file is loaded; files are not merged.\n\nTo place the logfile in a custom directory, set an absolute path in the\nuser `.env`, for example:\n\n```dotenv\nZEN_LOG_DIR=C:\\Microscopy\\logs\nZEN_LOG_LEVEL=DEBUG\n```\n\nLeave `ZEN_LOG_DIR=` empty or remove the line to use the platform-specific\ndefault documented in [Logging](#logging). Restart napari after changing the\nfile because logging is configured when the plugin widget opens.\n\n### Default OME-ZARR output directory\n\nSet the plugin's initial output folder with `ZEN_OUTPUT_DIR` in the user\nsettings file. On Windows this is\n`%APPDATA%\\napari-zenapi-streaming\\.env`. The settings directory may not exist\nyet; create it and open the file from PowerShell:\n\n```powershell\n$settingsDir = Join-Path $env:APPDATA \"napari-zenapi-streaming\"\nNew-Item -ItemType Directory -Force $settingsDir | Out-Null\nnotepad (Join-Path $settingsDir \".env\")\n```\n\nIf Notepad asks to create the file, accept. Add or update the following entry\nwithout removing any existing settings, then save it with the exact filename\n`.env`, not `.env.txt`:\n\n```dotenv\nZEN_OUTPUT_DIR=F:\\Zen_Output\n```\n\nCreating this `.env` does not require creating a user `config.ini`. If your\ngateway configuration already works, leave it in place. Do not create an empty\n`config.ini` in the user settings directory: it takes priority over working\ndirectory and checkout fallbacks. For a new gateway configuration, use the\n[configuration initializer](#1-configini-required) and populate its template\nbefore connecting.\n\nThe user `.env` replaces the editable checkout's fallback `.env`; the files\nare not merged. Include any checkout settings you still need, such as\n`ZEN_CONFIG_FILE`. Existing process environment variables override `.env`\nvalues. Restart napari after changing the file.\n\nAlternatively, set the output default for one PowerShell session before\nlaunching napari:\n\n```powershell\n$env:ZEN_OUTPUT_DIR = \"F:\\Zen_Output\"\nnapari\n```\n\nThe folder field in the plugin overrides this initial value. Without\n`ZEN_OUTPUT_DIR`, it defaults to your Documents directory. The standalone\nexperiment INI's `[output] directory` is separate and does not configure the\nplugin.\n\nBefore configured streaming starts an experiment or opens pixel streams, the\nwriter creates the output folder if needed and probes temporary-directory and\nfile creation, then removes the probe. It also rejects the Windows read-only\nattribute because the native writer rejects such parent folders even when\nPython can write to them. Windows Documents may have that attribute; choose a\nwritable subfolder or another output directory instead. This preflight does\nnot guarantee sufficient disk space or that access will remain available.\n\n## Running\n\n### Option A: Run as a napari plugin\n\n1. Start napari:\n\n   ```bash\n   napari\n   ```\n\n2. Open the plugin widget:\n   `Plugins > ZEN API Streaming > ZEN API Streaming`.\n\n### Option B: Run as a module\n\n```bash\npython -m napari_zen_streaming.main\n```\n\nWith CLI overrides:\n\n```bash\npython -m napari_zen_streaming.main --ui-start true \\\n    --exp-name ZEN_API_overview --config-file config.ini \\\n    --channel-index 0 --restructure-timeout 10\n```\n\n## Runtime behavior\n\n### Operating modes\n\nThe plugin UI exposes three stream modes. Connecting the plugin never starts\nan acquisition; only clicking Start in a start-capable mode does so.\n\n#### Display: watch ZEN experiment (index 0)\n\nThe plugin waits for an experiment started independently in ZEN Blue. No\nselected setup or known frame count is assumed, and there is no Start button.\nThe channel filter offers All channels or labels 1-16 (ZEN indices 0-15).\n\n#### Display: start selected experiment (index 1)\n\nThe Start button loads and triggers the selected experiment via the ZEN API.\nIts XML dimensions are shown in the widget, and the channel filter offers All\nchannels or exactly the channels available in that setup. Until metadata has\nloaded, only All channels is available. Changing the selected setup reloads\nboth its dimensions and channel choices. Click **Update ZEN Experiment Data**\nbelow the selector to reload the current setup without changing the selection.\n\nBoth Display modes use the full async streaming pipeline (`ZEN_pipeline.py`):\n\n1. **Streaming phase** – Frames are received via `monitor_all_experiments()`,\n   processed in `ZEN_utils.process_frame()`, and stored in a sparse dict.\n   The sparse key `(S, T, M, Z, C)` comes directly from each frame's\n   `FramePosition` metadata, including its S and M indices. By default,\n   **Show only last frame during live** displays a fixed 2D YX layer and\n   replaces it with the newest frame without rebuilding acquisition history.\n   Every frame remains in sparse storage for final assembly. Uncheck the\n   option to restore the more expensive 7D frame-history preview. The option\n   is frozen during acquisition and re-enabled after scene assembly.\n2. **Channel filter** – Changing the filter immediately restarts the pipeline\n   reader. Labels are 1-based, while the gRPC request uses zero-based indices.\n   The initial value comes from `ZEN_CHANNEL_INDEX` in `.env`; leave it empty\n   to receive all channels. An out-of-range choice resets to All channels.\n3. **Completion detection** –\n   - Napari-started experiments: the status monitor watches\n     `is_experiment_running`. Because ZEN can report completion before all\n     gRPC pixel data arrives, the plugin then waits for the XML-derived unique\n     frame count and uses pixel-stream inactivity as a safety fallback.\n   - ZEN-started experiments: inactivity timeout (`ZEN_RESTRUCTURE_TIMEOUT`).\n4. **Post-processing** – The streaming layer is removed and each scene's M\n   tiles are placed from their streamed stage coordinates into per-channel\n   5D layers (`S, T, Z, Y, X`). In overlap regions, pixels from the tile with\n   the higher M index replace pixels from lower-M tiles.\n\n**Experiment identity:** Display mode keeps one\n`monitor_all_experiments()` subscription open because the targeted ZEN stream\ncan close when experiment status changes to finished, before trailing pixel\npayloads have arrived. For a Napari-started acquisition, the plugin loads the\nexperiment first, arms a client-side filter for its `FrameData.experiment_id`,\nand only then starts acquisition. Frames from other experiments are ignored\nwhile that filter is active, including during trailing-frame drain. The filter\nis cleared after final restructure so later ZEN-started acquisitions remain\ndiscoverable. A ZEN-started run has no experiment ID known in advance and\ntherefore continues to accept the global stream; keep only one such\nacquisition active at a time.\n\n#### OME-ZARR only (index 2)\n\nBypasses the pipeline entirely.  When **Start** is clicked:\n\n1. A fresh `ExperimentConfig` is built from the selected experiment, the\n   OME-ZARR controls, the main ZEN API gateway config, and fixed plugin\n   defaults. XML-derived T/C/Z/M/S and Z spacing replace its planning\n   placeholders before writing. No experiment INI is selected by the plugin.\n2. **Auto-trigger checkbox** — \"Start experiment via ZEN API\":\n   - **Checked (default):** clicking Start immediately triggers the experiment\n     via the ZEN API and calls `stream_to_omezarr_with_config(ecfg)`.\n   - **Unchecked:** clicking Start arms the writer\n     (button shows **\"Writer Armed — start ZEN Experiment\"**)\n     and waits for the user to start the experiment manually in ZEN Blue.\n     The writer activates as soon as the first frame arrives.\n3. `stream_to_omezarr_with_config(ecfg)` — the **same function** used by the\n   standalone CLI — creates its own dedicated gRPC channel, opens the pixel\n   stream, optionally starts the experiment, and queries live status when\n   the first frame arrives. Positive live T/C/Z/S/M counts define the valid\n   coordinate ranges; XML-derived values remain the fallback if status is\n   unavailable.\n4. Each frame is routed by `(T, S, M, C, Z)`. T/S/M/Z come from that frame's\n   `FramePosition`; C comes from the explicit channel subscription because\n   all-channel mode merges one gRPC stream per source channel. The tuple is\n   mapped to the linear append order required by `ome-writers`. Frames may\n   arrive in any order and wait in a `pending` buffer. Duplicate tuples and\n   tuples outside the expected live/configured ranges are skipped; only\n   consecutive linear slots are appended to the Zarr stream.\n   The first frame's runtime `PixelType` selects `uint8` for `GRAY8` or\n   `uint16` for `GRAY16`; payload length is validated before decoding. ZEN\n   width is stored on X and height on Y without a rotation or transpose.\n   Streamed X/Y stage positions and pixel scales retain the same axis names.\n   Auto-triggered runs accept only their returned experiment ID. Manually\n   armed runs pin to the first received experiment ID; ensure that first frame\n   belongs to the intended acquisition. Later unrelated frames are ignored\n   before decoding and frames without an experiment ID are rejected.\n5. The loop ends after every unique expected coordinate has arrived, or on\n   inactivity timeout. On timeout, buffered frames are drained and truly\n   missing slots are emitted with `stream.skip()` so the array stays\n   rectangular. The configured writer records expected/received counts and\n   missing `(t, s, m, c, z)` coordinates in `acquisition_status.json` inside\n   the output store. Missing frames raise `IncompleteAcquisitionError`;\n   the CLI exits nonzero and the plugin shows a warning containing the partial\n   output path. Partial stores are retained, but are not auto-opened and do\n   not receive post-acquisition pyramids. Other writer errors also fail the\n   run; a status report is guaranteed only after successful finalization.\n6. For generic acquisitions with multiple scenes or tiles, the raw tile\n   streams are finalized into one TCZYX mosaic per scene. Streamed XYZ\n   coordinates and physical pixel scaling determine placement; M identifies\n   the raw tile within S but does not imply a row or column. In overlaps, the\n   highest M index wins, and gaps in irregular scene outlines remain at the\n   array fill value. The `OME/series` list advertises only the scene mosaics.\n   **Keep individual source tiles** controls whether the raw tile groups remain\n   after successful mosaic assembly. It is unchecked by default to avoid\n   duplicated pixel storage. For HCS acquisitions, one frame from every M tile\n   first establishes each scene canvas. The writer then buffers only the tiles\n   for each `(T, S, C, Z)` plane, merges them in memory, and writes the final\n   well image directly. Raw HCS tile arrays are never created, so the store is\n   not reread or rewritten after acquisition. The source-tile control is hidden\n   for HCS output because it does not apply.\n7. **Button state sequence:**\n   `Start Selected Experiment` → *(click)* →\n   `Writer Armed — start ZEN Experiment` *(auto-trigger off)* or\n   `Experiment Running...` *(auto-trigger on)* →\n   `Experiment & Stream Running` *(first frame arrived)* →\n   back to `Start Selected Experiment` *(done)*.\n8. Worker-to-Qt progress notifications are limited to approximately 10 Hz.\n   This keeps napari responsive without throttling frame reception or\n   OME-ZARR writes; first-frame, final-frame, and completion updates are\n   always delivered.\n\nThe plugin and the configured CLI share the same writer. Both use a dedicated\ngRPC channel while the display pipeline reader is suspended, avoiding frame\nloss from competing consumers. The\nunconfigured `--experiment` CLI mode instead buffers the full acquisition.\nThat legacy buffered mode cannot certify a complete expected coordinate set;\nuse configured mode for acquisition completeness reporting.\n\nStarting a new acquisition removes only layers created or opened by this\nplugin. Unrelated user layers and other plugins' dock widgets are retained.\n\n### Plugin OME-ZARR defaults\n\nThe plugin exposes output directory, compression, auto-trigger, HCS layout,\nand **Create pyramid after acquisition** in its UI. When enabled, choose how\nmany 2x Y/X downsampled levels to append after acquisition completes. Napari\nopens the OME-Zarr only after pyramid generation finishes. Existing level `0`\nis reused and is never rewritten.\nGeneric scene mosaics also expose **Keep individual source tiles**; unchecked\nremoves source groups only after mosaic creation succeeds. Advanced values\nintentionally use fixed defaults declared at the top of `ZEN_omezarr.py`:\n\n- `PLUGIN_DEFAULT_CZI_NAME = \"zenapi_stream\"`\n- `PLUGIN_DEFAULT_OVERWRITE_CZI = True`\n- `PLUGIN_DEFAULT_OVERWRITE_ZARR = True`\n- `PLUGIN_DEFAULT_SPATIAL_SHARD_SIZE_CHUNKS = 2`\n- `PLUGIN_DEFAULT_PYRAMID_LEVELS = 0`\n\nChange those constants to alter plugin behavior globally. Plugin OME-ZARR\ndtype is not a fixed default: it is derived from the first runtime frame's\nZEN `PixelType`. Color pixel types are rejected because the current writer\nmodels each streamed frame as a two-dimensional grayscale plane.\n\n### Standalone experiment config INI\n\nThe standalone CLI config mode accepts `experiment_streaming_config.ini`. The\nplugin does not load this file. Before opening pixel subscriptions, the CLI\nloads and exports the named experiment through ZEN API. Exported T/C/Z/M/S,\nZ spacing, and scene-to-well positions replace the INI's dimension defaults;\nthis ensures all active channels are subscribed and explicit wells can use\nthe HCS plate layout. The INI supplies the other run settings:\n\n- `[experiment]` – name, CZI output name, start mode, overwrite settings.\n- `[dimensions]` – legacy dimension fallbacks for callers of\n  `load_experiment_config()`. The configured CLI replaces them with exported\n  experiment metadata before opening the writer.\n- `[output]` – directory, dtype fallback, compression, overwrite,\n  `keep_source_tiles`, and optional `spatial_shard_size_chunks` for Zarr v3\n  Y/X sharding. Set the shard size to `0` or leave it empty to disable\n  sharding. `pyramid_levels` controls post-acquisition Y/X pyramid creation.\n- `[stream]` – optional source channel filter and `max_pending_frames` safety\n  bound for out-of-order frames.\n\nBefore OME-ZARR streaming, both the plugin and the configured CLI load and\nexport the selected ZEN experiment to derive T, C, Z, M, and S from its XML.\nActive rectangular\n`TileRegion` grids describe scene-specific tile layouts; for example, two\nactive 3-by-2 grids produce S=2 and M=6. The total frame count is recalculated\nfrom the exported metadata. These values plan the acquisition and provide a\nfallback, but they do not assign incoming pixels. The first live status\nsnapshot replaces positive T/C/Z/S/M counts before the writer layout is\nopened. Every frame is then assigned by its live `FramePosition`, including\n`FramePosition.s` and `FramePosition.m`; per-frame XYZ and scaling remain\nauthoritative for physical tile placement.\n\n### Stream metadata authority\n\nThe writer deliberately separates acquisition planning from frame routing:\n\n- **Selected experiment XML** – planned T/C/Z/S/M and Z spacing; fallback\n  when live status is unavailable.\n- **First live `get_status()` response** – positive T/C/Z/S/M counts used to\n  build the valid coordinate space and expected frame set.\n- **Per-frame `FramePosition`** – T/S/M/Z assignment for that incoming\n  image. C is the merged subscription index.\n- **First frame `pixel_data.pixel_type`** – OME-ZARR NumPy dtype, validated\n  against the raw payload byte count.\n- **Per-frame stage XYZ and pixel scaling** – native X/Y physical placement\n  and scale of each M tile inside its S scene.\n\nConsequently, ZEN may deliver channel-major or otherwise out-of-order frames\nwithout changing their destination. Scene membership is always the streamed S\nindex, tile identity is always the streamed M index, and spatial placement is\nalways based on streamed physical coordinates rather than an assumed grid.\n\n### Standalone CLI\n\nFor a Python API example, see\n[examples/stream_to_omezarr.py](examples/stream_to_omezarr.py). It loads the\nexperiment INI, exports acquisition metadata, and awaits\n`stream_to_omezarr_with_config()`. Use the same environment and gateway\nconfiguration described above. From the repository root:\n\n```powershell\npython examples/stream_to_omezarr.py\n```\n\nWith no argument, it uses `experiment_streaming_config.ini` from the repository\nroot, located relative to the script rather than the working directory. Pass\nanother INI path as the positional argument to override this default.\n\nUse `--experiment NAME` to override the INI's experiment name for both metadata\nexport and streaming. It does not start acquisition unless you also pass\n`--start-experiment`:\n\n```powershell\npython examples/stream_to_omezarr.py --experiment \"MyExperiment\" --start-experiment\n```\n\nEdit the INI's experiment name and output directory before running. By default,\nthe example waits for you to start the acquisition in ZEN Blue. Add\n`--start-experiment` to trigger it via the API. The example disables CZI and\nOME-ZARR overwriting, honors the INI's pyramid settings, and lets errors\npropagate so incomplete acquisitions exit nonzero and retain partial output.\n\n`ZEN_stream2omezarr.py` can be run directly without napari:\n\n```bash\n# Config mode (export metadata first, then write on-the-fly):\npython -m napari_zen_streaming.ZEN_stream2omezarr \\\n    --experiment-config experiment_streaming_config.ini\n\n# CLI mode (unknown dimensions, buffered write):\npython -m napari_zen_streaming.ZEN_stream2omezarr \\\n    --experiment MyExp --output-dir ./output --start-experiment\n\n# Wait for user to start from ZEN UI:\npython -m napari_zen_streaming.ZEN_stream2omezarr \\\n    --experiment-config experiment_streaming_config.ini --no-start-experiment\n\n# Open viewer after acquisition:\npython -m napari_zen_streaming.ZEN_stream2omezarr \\\n    --experiment-config experiment_streaming_config.ini --viewer ndv\n\n# Override the INI and append three pyramid levels after acquisition:\npython -m napari_zen_streaming.ZEN_stream2omezarr \\\n  --experiment-config experiment_streaming_config.ini --pyramid-levels 3\n```\n\n## Logging\n\n- Log file on Windows:\n  `C:\\Users\\<username>\\AppData\\Local\\napari-zenapi-streaming\\logs\\zen_streaming.log`\n  (`%LOCALAPPDATA%\\napari-zenapi-streaming\\logs\\zen_streaming.log`).\n  `AppData` is hidden by default in File Explorer.\n- Log file on Linux/macOS:\n  `$XDG_STATE_HOME/napari-zenapi-streaming/logs/zen_streaming.log`\n  (or `~/.local/state/napari-zenapi-streaming/logs/zen_streaming.log`)\n- Set `ZEN_LOG_DIR` to override the log directory.\n- Logs rotate at 5 MB and retain three backups.\n- Messages are also written to the terminal that launched napari after the\n  plugin widget is opened.\n- Set `ZEN_LOG_LEVEL=DEBUG` in `.env` to include connection, pipeline, frame\n  processing, layer-update, and restructure details. The default `INFO` level\n  records only milestones, warnings, and errors.\n- Logging format: `%(asctime)s - %(name)s - %(levelname)s - %(message)s`\n\nTo see live terminal records, launch napari in the configured environment and\nkeep that terminal open:\n\n```powershell\nnapari\n```\n\nLogging starts when the **ZEN API Streaming** plugin widget is opened. A napari\nprocess launched from the Start menu or another detached GUI launcher cannot\nwrite records into an existing PowerShell terminal.\n\nOn Windows, print the exact logfile location for the current user:\n\n```powershell\n$logFile = \"$env:LOCALAPPDATA\\napari-zenapi-streaming\\logs\\zen_streaming.log\"\n$logFile\n```\n\nOpen its directory in File Explorer:\n\n```powershell\nexplorer.exe \"$env:LOCALAPPDATA\\napari-zenapi-streaming\\logs\"\n```\n\nIf `ZEN_LOG_DIR` is not set, follow the default Windows logfile in a separate\nPowerShell terminal:\n\n```powershell\nGet-Content \"$env:LOCALAPPDATA\\napari-zenapi-streaming\\logs\\zen_streaming.log\" -Wait\n```\n\nWhen `ZEN_LOG_DIR=C:\\Microscopy\\logs`, use the configured location instead:\n\n```powershell\nGet-Content \"C:\\Microscopy\\logs\\zen_streaming.log\" -Wait\n```\n\n`Get-Content` prints the existing records and `-Wait` keeps the command\nrunning so new records appear as they are written. Press `Ctrl+C` to stop.\nIf the 5 MB logfile rotates while this command is running, restart the command\nto follow the new active `zen_streaming.log` file.\n\n## Architecture\n\nThe latest-frame preview avoids rebuilding a dense history array during\nacquisition. Sparse frames remain available for final scene assembly; disabling\nthe preview option restores full-history live display.\n\n### Display mode data flow\n\nThe global pixel stream feeds the pipeline reader, queue and processor.\nQt signals update sparse frame storage and the latest 2D preview, or the\noptional 7D history layer. For napari-started runs, the experiment ID filter\nstays armed through the trailing-frame drain after status completion.\nFinal restructure creates one STZYX scene-mosaic layer per channel, then\nclears the ID filter. Only plugin-owned layers are removed.\n\n### OME-ZARR mode data flow\n\nThe plugin suspends the Display reader before the shared configured writer\nopens its dedicated channel and per-source-channel pixel streams. Experiment\nIDs are checked before decoding. Runtime GRAY8/GRAY16 metadata selects the\ndtype; positive live dimensions override XML fallback counts. Unique\n`(T,S,M,C,Z)` keys drive bounded buffering and ordered writes, either as raw\ngeneric tiles or directly merged HCS scene planes.\n\nAfter completion or inactivity, the writer drains pending slots, finalizes\nthe layout and writes `acquisition_status.json`. Complete stores may receive\npyramids and open in a viewer. Missing coordinates instead raise\n`IncompleteAcquisitionError`, preserve partial output and suppress pyramids\nand auto-opening. The plugin warns on failure and resumes the Display reader\nin `finally`; the configured CLI exits nonzero for incomplete acquisitions.\n\nBoth flows use the same `monitor_all_experiments()` gRPC call.\nThe OME-ZARR path creates its own channel so there is no contention.\n\n## Diagrams and docs\n\n- [Mermaid flowchart](docs/napari_zenapi_streaming_flowchart.mmd)\n- [Mermaid sequence diagram](docs/napari_zenapi_streaming_sequence.mmd)\n\nThe diagrams focus on the three plugin workflows and their shared configured\nwriter. The standalone buffered CLI mode is described separately above.\n\nFlowchart preview:\n\n![Napari ZEN streaming flowchart](docs/napari_zenapi_streaming_flowchart.png)\n\nRender the Mermaid sources with Mermaid CLI, or preview `.mmd` files in the\nMermaid VS Code extension. From the repository root:\n\n```bash\nnpx --yes --package @mermaid-js/mermaid-cli -- mmdc \\\n    -i docs/napari_zenapi_streaming_flowchart.mmd \\\n    -o docs/napari_zenapi_streaming_flowchart.png\n```\n\n## Troubleshooting\n\n- Missing `config.ini`: run `python -m napari_zen_streaming.ZEN_config --init`,\n  populate the template, or set `ZEN_CONFIG_FILE` to an existing file.\n- TLS or certificate issues: verify the `cert_file` path and that the file is\n  readable.\n- **Qt5 incompatibility:** install `\"PyQt6>=6.7,<6.10\"` and select\n  `QT_API=pyqt6`. For pytest, also set `PYTEST_QT_API=pyqt6`.\n- No experiments in dropdown: confirm ZEN API Gateway is running and token\n  is valid.\n- Viewer not updating: check `zen_streaming.log` in the user log directory\n  described above for pipeline errors.\n- Restructure too early/late for ZEN-started runs: adjust\n  `ZEN_RESTRUCTURE_TIMEOUT`.\n- **Progress bar stuck on \"Streaming frames\":** if ZEN stops an experiment\n  without the plugin detecting it (e.g. network glitch), click **Stop** to\n  dismiss the bar.  Clicking **Start** on the next experiment also clears any\n  stale bar automatically.\n- **Incomplete acquisition:** inspect `acquisition_status.json` and the\n  warning's partial-store path. Check frame counts, gateway logs, and\n  inactivity timeout before retrying; a rectangular output array does not\n  imply complete data.\n- **Channel filter not taking effect:** ensure no experiment is currently\n  running; the reader restarts on combo box change, but an in-flight gRPC\n  stream may buffer a few straggler frames.\n\n## Development\n\n- Install development extras:\n\n```bash\npython -m pip install -e \".[dev,testing]\" build\npython -m pytest -q src/napari_zen_streaming/_tests\npython -m ruff check src/napari_zen_streaming\npython -m ruff format --check src/napari_zen_streaming\npython -m build --sdist --wheel\n```\n\n- Ruff lints and formats the code using the settings in `pyproject.toml`\n  (line length 79). In VS Code, set\n  `\"ruff.configurationPreference\": \"filesystemFirst\"` so user-level Ruff\n  settings do not override the repository configuration.\n- Tests mock the gateway and cover routing, partial acquisitions, mosaics,\n  metadata refresh, dtype handling, progress, configuration, and layer\n  ownership.\n- A clean Windows environment passed the full test suite with Python 3.12.15,\n  napari 0.10.0, PyQt6 6.9.1, ome-writers 0.3.2, ngff-zarr 0.51.0,\n  zen-api 2026.7.1, and zen-api-utils 0.1.0. This records one tested\n  environment, not a claim that every gateway version is compatible.\n- CI checks Python 3.12-3.14 on Windows, Linux, and macOS. Pushes and pull\n  requests target `main`; version tags trigger publishing only after tests pass.\n- CI supplies the public OAD release wheel URLs for `zen_api==2026.7.1` and\n  `zen_api_utils==0.1.0` in the same pip command as the plugin installation.\n  No ZEN wheel URL secret is required, including for forked pull requests.\n- The testing extra installs `PyQt6>=6.7,<6.10`. Both CI workflows set\n  `QT_API=pyqt6` and `PYTEST_QT_API=pyqt6` to avoid selecting another binding.\n\n### Before tagging a release\n\n- Confirm the pinned OAD release wheels remain reachable and the full CI\n  matrix passes. Local tests do not replace this check.\n- Choose the release version, update `pyproject.toml` and `CHANGELOG.md`, and\n  verify the tag matches. No release version is selected by the cleanup.\n- Install the built wheel in a clean environment, initialize user configuration,\n  and verify plugin discovery without relying on this checkout.\n- On a real ZEN gateway, test watch/start Display modes, manual/auto-triggered\n  OME-ZARR, multiple channels and tiles, HCS wells, and pyramid generation.\n- Interrupt a real acquisition: confirm a visible warning, non-success status,\n  preserved partial output, and correct missing-coordinate reporting.\n- Verify Stop, disconnect, viewer close, repeated runs, and preservation of\n  unrelated layers and docks. Measure throughput and memory at production size.\n\n## Authors\n\n- Krijn H. van der Steen (UMC Utrecht)\n- Sebastian Rhode (ZEISS)\n\n## License\n\nBSD 3-Clause (see `LICENSE`).\n","description_content_type":"text/markdown","keywords":"napari,plugin,microscopy,ZEN,streaming,ZEN-API","home_page":null,"download_url":null,"author":null,"author_email":"\"Krijn H. van der Steen\" <k.h.vandersteen@umcutrecht.nl>, Sebastian Rhode <sebastian.rhode@zeiss.com>","maintainer":null,"maintainer_email":null,"license":null,"classifier":["Development Status :: 3 - Alpha","Intended Audience :: Science/Research","Framework :: napari","Topic :: Scientific/Engineering :: Image Processing","Programming Language :: Python :: 3","Programming Language :: Python :: 3.12","Programming Language :: Python :: 3.13","Programming Language :: Python :: 3.14","Operating System :: OS Independent"],"requires_dist":["napari>=0.9.1","magicgui>=0.7.0","qtpy>=2.3.0","numpy>=1.21.0","qasync>=0.24.0","grpclib","pydantic","python-dotenv","ome-writers[acquire-zarr]","ngff-zarr[dask-image]","zarr>=3","zen_api<=2026.11.1,>=2026.5.1","zen_api_utils<0.2,>=0.1.0","pytest>=6.0; extra == \"dev\"","pytest-qt; extra == \"dev\"","pytest-asyncio; extra == \"dev\"","ruff; extra == \"dev\"","mypy; extra == \"dev\"","tox; extra == \"testing\"","pytest; extra == \"testing\"","pytest-cov; extra == \"testing\"","pytest-qt; extra == \"testing\"","pytest-timeout; extra == \"testing\"","napari; extra == \"testing\"","pyqt6<6.10,>=6.7; extra == \"testing\""],"requires_python":"<3.15,>=3.12","requires_external":null,"project_url":["Homepage, https://github.com/sebi06/napari-zenapi-streaming","Bug Tracker, https://github.com/sebi06/napari-zenapi-streaming/issues","Documentation, https://github.com/sebi06/napari-zenapi-streaming#readme","Source Code, https://github.com/sebi06/napari-zenapi-streaming"],"provides_extra":["dev","testing"],"provides_dist":null,"obsoletes_dist":null},"npe1_shim":false}