Skip to content

Faster, more reliable first-time setup: drop unused TF/StarDist, pin deps; restore folder-table sorting #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aaturk2 wants to merge 3 commits into
base: main
Choose a base branch
from
+107 −149
Open

Faster, more reliable first-time setup: drop unused TF/StarDist, pin deps; restore folder-table sorting #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f0152cd
Remove unused tensorflow/stardist deps; support Python 3.10-3.13
aaturk2 Jul 20, 2026
1bb10fb
Folder analysis: restore sortable batch ledger table
aaturk2 Jul 20, 2026
2f606f6
Pin dependency versions to a known-good set (Python 3.10-3.12)
aaturk2 Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,8 @@ Group website: https://murphy-group.chemistry.illinois.edu/

## Notes:
- DO NOT double-click on RodSizer_CLEANER_MacOS.command unless you are sure that you want to clean ALL local history of data and reports.
- First-time launching may take some time, like 5-10 mins. Most of that is
installing the deep-learning packages (`tensorflow` + `stardist`, ~1 GB). These
are kept for possible future ML-based detection but are **not used** by the
current pipeline (K-means + watershed), so the wait is one-time setup only —
later launches are fast. See `backend/requirements.txt` to remove them if you
want a leaner install.
- First-time launching takes a few minutes (typically 2-3) while Python
packages are installed. This is one-time setup only — later launches are fast.
- Try ask a coding agent if there's an issue with environment setup.
- MacOS is more recommended.
- (Windows) If Windows Defender asks, click `More Info` -> `Run Anyway`.
Expand All @@ -88,4 +84,4 @@ Group website: https://murphy-group.chemistry.illinois.edu/

## Requirements
- macOS or Windows
- Python 3 installed (standard on most Macs, or downloadable from `python.org`)
- Python 3.10-3.12 (the Mac launcher installs it automatically if missing; on Windows download from `python.org`)
2 changes: 2 additions & 0 deletions RodSizer_DEBUG_Windows.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@ REM Auto-Detect
if exist "C:\Python39\python.exe" set "PY_EXE=C:\Python39\python.exe" & goto FOUND_PYTHON
if exist "C:\Python310\python.exe" set "PY_EXE=C:\Python310\python.exe" & goto FOUND_PYTHON
if exist "C:\Python311\python.exe" set "PY_EXE=C:\Python311\python.exe" & goto FOUND_PYTHON
if exist "C:\Python312\python.exe" set "PY_EXE=C:\Python312\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python39\python.exe" set "PY_EXE=C:\Program Files\Python39\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python310\python.exe" set "PY_EXE=C:\Program Files\Python310\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python311\python.exe" set "PY_EXE=C:\Program Files\Python311\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python312\python.exe" set "PY_EXE=C:\Program Files\Python312\python.exe" & goto FOUND_PYTHON

:ASK_USER
echo.
Expand Down
143 changes: 31 additions & 112 deletions RodSizer_Launcher_MacOS.command
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,65 +74,47 @@ if command -v tesseract &>/dev/null; then
fi

# =============================================================================
# STEP 2 — Python 3.11
#
# TensorFlow 2.x officially supports Python 3.8–3.11.
# Python 3.12+ does NOT have stable TensorFlow wheels yet.
# We therefore require Python 3.11 specifically.
# STEP 2 — Python 3.10-3.12
# (the pinned dependency set in backend/requirements.txt targets these
# versions; 3.13+ lacks wheels for the pinned numpy)
# =============================================================================
step "Step 2/4: Checking Python 3.11"
step "Step 2/4: Checking Python"

PYTHON_CMD=""

# Helper: find python3.11 by checking all known locations
find_python311() {
# 1. Homebrew-managed path (most reliable — works regardless of PATH)
local brew_prefix
brew_prefix="$(brew --prefix python@3.11 2>/dev/null)"
if [ -n "$brew_prefix" ] && [ -x "$brew_prefix/bin/python3.11" ]; then
echo "$brew_prefix/bin/python3.11"; return
fi

# 2. Direct binary in PATH
if command -v python3.11 &>/dev/null; then
command -v python3.11; return
fi

# 3. Common Homebrew Cellar fallback (Apple Silicon)
local cellar_py
cellar_py=$(ls /opt/homebrew/Cellar/python@3.11/*/bin/python3.11 2>/dev/null | sort -V | tail -1)
if [ -x "$cellar_py" ]; then
echo "$cellar_py"; return
fi
# Intel
cellar_py=$(ls /usr/local/Cellar/python@3.11/*/bin/python3.11 2>/dev/null | sort -V | tail -1)
if [ -x "$cellar_py" ]; then
echo "$cellar_py"; return
fi

# 4. pyenv
if command -v pyenv &>/dev/null; then
local pyenv_ver
pyenv_ver=$(pyenv versions --bare 2>/dev/null | grep "^3\.11\." | sort -V | tail -1)
if [ -n "$pyenv_ver" ]; then
local pyenv_py="$(pyenv root)/versions/$pyenv_ver/bin/python3.11"
[ -x "$pyenv_py" ] && echo "$pyenv_py"; return
# Helper: find a suitable Python (3.10-3.12), preferring newer versions
find_python() {
local candidate
for candidate in python3.12 python3.11 python3.10 python3; do
if command -v "$candidate" &>/dev/null; then
if "$candidate" -c 'import sys; sys.exit(0 if (3, 10) <= sys.version_info[:2] <= (3, 12) else 1)' &>/dev/null; then
command -v "$candidate"; return
fi
fi
fi
done

# Homebrew-managed fallback (works regardless of PATH)
local brew_prefix ver
for ver in 3.12 3.11 3.10; do
brew_prefix="$(brew --prefix python@$ver 2>/dev/null)"
if [ -n "$brew_prefix" ] && [ -x "$brew_prefix/bin/python$ver" ]; then
echo "$brew_prefix/bin/python$ver"; return
fi
done
}

PYTHON_CMD="$(find_python311)"
PYTHON_CMD="$(find_python)"

# If still not found, install via Homebrew then retry
if [ -z "$PYTHON_CMD" ]; then
warn "Python 3.11 not found. Installing via Homebrew (needed for TensorFlow)..."
brew install python@3.11
PYTHON_CMD="$(find_python311)"
warn "Python 3.10-3.12 not found. Installing 3.12 via Homebrew..."
brew install python@3.12
PYTHON_CMD="$(find_python)"
fi

if [ -z "$PYTHON_CMD" ] || [ ! -x "$PYTHON_CMD" ]; then
error "Could not find or install Python 3.11."
error "Please install it manually: brew install python@3.11"
error "Could not find or install Python."
error "Please install it manually: brew install python@3.12"
echo "Press Enter to close..."; read -r; exit 1
fi

Expand Down Expand Up @@ -170,84 +152,21 @@ needs_python_packages() {
"$VENV_PYTHON" - <<'PY' &>/dev/null
import importlib

for name in ("fastapi", "uvicorn", "tensorflow", "numpy", "cv2", "stardist"):
for name in ("fastapi", "uvicorn", "numpy", "cv2", "skimage", "tifffile", "h5py"):
importlib.import_module(name)
PY
}

install_tensorflow_with_fallback() {
local tf_spec="tensorflow==2.21.0"

info "Installing TensorFlow..."
if "$VENV_PYTHON" -m pip install "$tf_spec"; then
return 0
fi

warn "pip could not resolve tensorflow from the package index."
warn "Retrying TensorFlow install directly from PyPI..."
PIP_CONFIG_FILE=/dev/null "$VENV_PYTHON" -m pip install --index-url https://pypi.org/simple "$tf_spec" && return 0

return 1
}

# Check whether a first-time install is needed
if ! needs_python_packages; then
info "Installing Python packages — this may take 5–10 minutes on first run."
info "Installing Python packages — this may take 2–3 minutes on first run."
info "Please do NOT close this window."

# Upgrade pip / setuptools silently
"$VENV_PYTHON" -m pip install --upgrade pip setuptools wheel --quiet

# ── TensorFlow first (sets numpy version constraint everything else must follow) ──
#
# tensorflow-macos was deprecated at 2.13 and forces numpy<2, which conflicts
# with opencv-python-headless >=4.9 and ncempy >=1.15 (both require numpy>=2).
#
# tensorflow >= 2.16 ships a universal wheel that runs natively on Apple
# Silicon (M-series) without a separate macos fork, and is compatible with
# numpy 2.x. tensorflow-metal is still the GPU-acceleration plugin for M-chips.
if ! install_tensorflow_with_fallback; then
error "TensorFlow installation failed."
error "This is usually a package-index or network issue, not a RodSizer code issue."
if [ "$IS_APPLE_SILICON" = true ]; then
error "If needed, try this manually inside backend/.venv:"
error " python -m pip install --index-url https://pypi.org/simple tensorflow==2.21.0"
else
error "Try rerunning later, or install TensorFlow manually inside backend/.venv."
fi
echo "Press Enter to close..."; read -r; exit 1
fi

# Note: tensorflow-metal is NOT installed because v1.2.0 is incompatible
# with tensorflow >=2.16 (dlopen fails on _pywrap_tensorflow_internal.so).
# Modern TensorFlow already runs natively on Apple Silicon without it.

if ! "$VENV_PYTHON" -c "import tensorflow" &>/dev/null 2>&1; then
error "TensorFlow import failed after installation."
error "Try deleting backend/.venv and relaunching, or check network connectivity."
echo "Press Enter to close..."; read -r; exit 1
fi

# ── Remaining packages (numpy version is now fixed by TF above) ─────────────
TMP_REQ=$(mktemp /tmp/rodsizer_req_XXXX.txt)
grep -v "^tensorflow" "$BACKEND_DIR/requirements.txt" > "$TMP_REQ"

if ! "$VENV_PYTHON" -m pip install -r "$TMP_REQ" --quiet; then
if ! "$VENV_PYTHON" -m pip install -r "$BACKEND_DIR/requirements.txt" --quiet; then
error "Failed to install some packages. Check the output above."
rm -f "$TMP_REQ"
echo "Press Enter to close..."; read -r; exit 1
fi
rm -f "$TMP_REQ"

# ── Verify tensorflow is importable ───────────────────────────────────────
if ! "$VENV_PYTHON" -c "import tensorflow" &>/dev/null 2>&1; then
error "TensorFlow installation failed."
error "Try running this manually inside the venv:"
if [ "$IS_APPLE_SILICON" = true ]; then
error " python -m pip install --index-url https://pypi.org/simple tensorflow==2.21.0"
else
error " python -m pip install tensorflow==2.21.0"
fi
echo "Press Enter to close..."; read -r; exit 1
fi

Expand Down
2 changes: 2 additions & 0 deletions RodSizer_Launcher_Windows.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ REM --- 3. Try Auto-Detect Common Paths ---
if exist "C:\Python39\python.exe" set "PY_EXE=C:\Python39\python.exe" & goto FOUND_PYTHON
if exist "C:\Python310\python.exe" set "PY_EXE=C:\Python310\python.exe" & goto FOUND_PYTHON
if exist "C:\Python311\python.exe" set "PY_EXE=C:\Python311\python.exe" & goto FOUND_PYTHON
if exist "C:\Python312\python.exe" set "PY_EXE=C:\Python312\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python39\python.exe" set "PY_EXE=C:\Program Files\Python39\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python310\python.exe" set "PY_EXE=C:\Program Files\Python310\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python311\python.exe" set "PY_EXE=C:\Program Files\Python311\python.exe" & goto FOUND_PYTHON
if exist "C:\Program Files\Python312\python.exe" set "PY_EXE=C:\Program Files\Python312\python.exe" & goto FOUND_PYTHON

REM --- 4. User Input Fallback ---
:ASK_USER
Expand Down
46 changes: 23 additions & 23 deletions backend/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
fastapi
uvicorn
python-multipart
numpy
opencv-python-headless
scikit-image
ncempy
matplotlib
pandas
Pillow
pytesseract
tifffile
openpyxl
h5py
# Pinned, known-good dependency set (tested on Python 3.12, July 2026).
# Pinning exact versions keeps every computer running identical code —
# unpinned installs were a major source of "works here, not there" bugs.
# Python 3.10-3.12 required (numpy 1.26 has no wheels for 3.13+).
fastapi==0.122.0
uvicorn==0.38.0
python-multipart==0.0.20
numpy==1.26.4
opencv-python-headless==4.12.0.88
scikit-image==0.25.2
ncempy==1.14
matplotlib==3.10.7
pandas==2.3.3
Pillow==12.0.0
pytesseract==0.3.13
tifffile==2025.10.16
openpyxl==3.1.5
h5py==3.15.1

# --- Heavy deps kept for possible future ML detection (NOT used right now) ---
# stardist + tensorflow are the main reason the FIRST-TIME setup is slow (~5-10
# min) and large (~1 GB+): they pull in a full TensorFlow build. The current
# detection pipeline is K-means + distance-transform watershed and does NOT use
# them (they are imported nowhere in the hot path). They are kept on purpose in
# case StarDist-based detection is added later. Remove these two lines if you
# want a much faster, smaller install and don't plan to use StarDist.
stardist
tensorflow
# NOTE: stardist + tensorflow were removed (July 2026). Nothing in the current
# pipeline (K-means + distance-transform watershed) uses them, and they were the
# main reason first-time setup was slow (~5-10 min, ~1 GB) and failed on some
# machines (no TensorFlow wheels for newer Python versions). If StarDist-based
# detection is ever added, re-add: stardist and tensorflow
53 changes: 46 additions & 7 deletions frontend/folder_analysis.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,11 +278,11 @@ <h2>Batch Ledger</h2>
<table id="dataTable">
<thead>
<tr>
<th>Source Image</th>
<th>ID</th>
<th>L (nm)</th>
<th>W (nm)</th>
<th>AR</th>
<th onclick="sortTable('source_image')" title="Click to sort" style="cursor:pointer; user-select:none;">Source Image <span class="sort-arrow" data-col="source_image"></span></th>
<th onclick="sortTable('id')" title="Click to sort" style="cursor:pointer; user-select:none;">ID <span class="sort-arrow" data-col="id"></span></th>
<th onclick="sortTable('length_nm')" title="Click to sort" style="cursor:pointer; user-select:none;">L (nm) <span class="sort-arrow" data-col="length_nm"></span></th>
<th onclick="sortTable('width_nm')" title="Click to sort" style="cursor:pointer; user-select:none;">W (nm) <span class="sort-arrow" data-col="width_nm"></span></th>
<th onclick="sortTable('aspect_ratio')" title="Click to sort" style="cursor:pointer; user-select:none;">AR <span class="sort-arrow" data-col="aspect_ratio"></span></th>
</tr>
</thead>
<tbody></tbody>
Expand All @@ -308,6 +308,8 @@ <h2>Batch Ledger</h2>
dlBtn.href = `/folders/${encodeURIComponent(folderName)}/export_aggregate`;

let lengthChart, widthChart;
let currentData = [];
let currentSort = { column: null, direction: 'asc' };

async function loadAggregateData() {
try {
Expand Down Expand Up @@ -338,14 +340,41 @@ <h2>Batch Ledger</h2>
document.getElementById('meanAR').textContent = data.stats.mean_ar || "-";

// Charts
const items = data.data;
currentData = data.data || [];
const items = currentData;
const lengths = items.map(d => d.length_nm);
const widths = items.map(d => d.width_nm);

lengthChart = createHistogram('lengthChart', 'Length Distribution', lengths);
widthChart = createHistogram('widthChart', 'Width Distribution', widths);

// Table
renderTable();
}

function renderTable() {
let items = currentData.slice();

if (currentSort.column) {
const col = currentSort.column;
const dir = currentSort.direction === 'asc' ? 1 : -1;
items.sort((a, b) => {
const va = a[col], vb = b[col];
if (typeof va === 'string' || typeof vb === 'string') {
return String(va ?? '').localeCompare(String(vb ?? '')) * dir;
}
return ((va ?? 0) - (vb ?? 0)) * dir;
});
}

// Update header arrows
document.querySelectorAll('#dataTable .sort-arrow').forEach(span => {
if (span.dataset.col === currentSort.column) {
span.textContent = currentSort.direction === 'asc' ? '▲' : '▼';
} else {
span.textContent = '';
}
});

const tbody = document.querySelector('#dataTable tbody');
tbody.innerHTML = '';
items.forEach(item => {
Expand All @@ -366,6 +395,16 @@ <h2>Batch Ledger</h2>
});
}

function sortTable(column) {
if (currentSort.column === column) {
currentSort.direction = currentSort.direction === 'asc' ? 'desc' : 'asc';
} else {
currentSort.column = column;
currentSort.direction = 'asc';
}
renderTable();
}

function createHistogram(canvasId, label, data) {
const ctx = document.getElementById(canvasId).getContext('2d');
if (!data.length) return;
Expand Down