Update agents.md to prevent agents from running multiple concurrent remote jobs

AGENTS.md

@@ -0,0 +1,28 @@
+# Instructions for AI coding agents
+
+## ClearML jobs: only one running remotely at a time, across ALL agents
+
+Never have more than one ClearML job running **remotely** at once (submitted
+to a queue and executed by a `clearml-agent` worker) — this applies across
+every agent session the user has open, not just the current one. This is
+about contention for shared queue/GPU resources.
+
+This restriction does **not** apply to jobs run locally (`queue_name: local`/
+`locally`, i.e. never enqueued) — run as many of those concurrently as you
+like, even if they're tracked as ClearML tasks.
+
+Before submitting any job remotely (e.g. via `silnlp/nmt/experiment.py`,
+`silnlp/nmt/train.py`, `silnlp/nmt/translate.py`, alignment scripts under
+`silnlp/alignment/`, or anything using `silnlp/nmt/clearml_connection.py`
+with a non-local `queue_name`):
+
+1. Run `python scripts/check_clearml_jobs.py`. It uses the ClearML API
+   credentials in the environment to list any remote jobs already running
+   or queued under the current user's account (a non-zero exit code means
+   one exists) — this works even though you (the agent) can't browse the
+   ClearML web UI.
+2. If a remote job is already active, don't submit another yet — and don't
+   just tell the user to wait. Keep the job in your own backlog of work to
+   submit, and handle the scheduling yourself: check again (e.g. after other
+   work, or by polling) and submit it as soon as the check shows no active
+   remote job. Only involve the user if something looks stuck or wrong.

scripts/check_clearml_jobs.py

@@ -0,0 +1,54 @@
+"""Report remote ClearML jobs currently running or queued under the authenticated user.
+
+Intended for agents to run before submitting a new remote ClearML job, to
+check whether one is already in flight (across any of the user's agent
+sessions). Jobs run locally (queue_name "local"/"locally", never enqueued)
+are not included, since they don't contend for shared queue/GPU resources.
+
+Usage: python scripts/check_clearml_jobs.py
+Exit code: 1 if any remote job is running/queued, 0 otherwise.
+"""
+
+import base64
+import json
+import sys
+
+from clearml.backend_api.services import tasks as tasks_service
+from clearml.backend_api.session import Session
+
+
+def _current_user_id(session: Session) -> str:
+    token = session.send_request(service="auth", action="login", method="GET", json={}).json()["data"]["token"]
+    payload = token.split(".")[1]
+    payload += "=" * (-len(payload) % 4)
+    return json.loads(base64.urlsafe_b64decode(payload))["identity"]["user"]
+
+
+def main() -> int:
+    session = Session()
+    user_id = _current_user_id(session)
+
+    result = session.send(
+        tasks_service.GetAllRequest(
+            user=[user_id],
+            status=["in_progress", "queued"],
+            only_fields=["name", "status", "started", "execution.queue"],
+            order_by=["-started"],
+        )
+    )
+    # tasks only get a queue when enqueued for remote execution (Task.execute_remotely);
+    # locally-run tasks never have one, so this excludes them.
+    active = [t for t in result.response.tasks if t.execution and t.execution.queue]
+
+    if not active:
+        print("No remote ClearML jobs currently running or queued under your account.")
+        return 0
+
+    print(f"{len(active)} remote ClearML job(s) already running/queued under your account:")
+    for task in active:
+        print(f"  [{task.status}] {task.name} (started: {task.started})")
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())