Building an Agentic AI Night Operator for My Remote Observatory

A clear, moonless night at the observatory yields about six usable hours of integration time. Losing half of that window because a sequencer continued exposing into thick haze9 usable hours out of 12. The rest went to haze. is not an equipment failure, but a situational blind spot. I ran into the same issue on M81 and M82The first night, scrapped over a camera angle., where an entire night of data was scrapped because the camera angle did not match the earlier sessions.

Both times, the hardware functioned normally. The sequencer simply did not know the plan was no longer useful because it had no awareness of its surroundings, so it continued executing the original instructions until morning. The goal of this night operator is to bridge that gap by supervising telemetry, respecting hard mathematical boundaries, and adapting the session dynamically overnight.

The Three Tiers of Observatory Operation

To understand where true autonomy begins, it helps to distinguish how remote observatories operate across three distinct tiers:

• Remote Operable: Extends physical reach over a network using tools like Tailscale, IoT power switches, and remote desktop. You eliminate physical travel to the pier, but you still sit at a monitor at 02:00 to inspect frames and manually click buttons.
• Automated: Executes pre-programmed, deterministic instruction trees through software like NINA and PHD2. The setup runs unattended until an unhandled edge case occurs, at which point it either halts safely or wastes dark sky by executing bad instructions into poor conditions.
• Agentic: Operates against high-level project intent under shifting constraints. Rather than executing a static queue, an agent evaluates live telemetry, checks project quotas, and dynamically adjusts the schedule to maximize seasonal progress.

My setup in Bhagalpur currently operates solidly at the Automated tierFirst light, January 2026. The telescopes stay mounted on permanent piers with persistent polar alignment, routed cabling, and dedicated mini-PCs that boot on power restoration.

Current automated infrastructureIoT switches, Tailscale mesh VPN, NUC auto-boot

Reliable baseline systems already run on site: power sockets run through IoT switches, mini-PCs boot on power restoration, and Tailscale provides encrypted mesh networking. These deterministic foundational layers do not need an AI model.

The observatory still relies on an on-site caretaker to unlock and roll back the roof, though adding rain sensors and a motorized roof driver will eventually automate that physical link once the software supervisor is proven. The true bottleneck is not mechanical. It is the absence of context during execution.

Architecture: Deterministic Inner Loops, Probabilistic Outer Loops

When bringing AI into an automated setup, it is easy to fall into two common traps:

• Treating a Chatbot as True Autonomy: Setting up an AI to send alerts and answer questions is a convenient interface, but it is not automation. If an operator still has to sit at a screen typing prompts to diagnose an issue and keep the night moving, you have not built an autonomous system. You have simply replaced clicking a button with typing an expensive, token-burning message.
• Forcing AI into Tasks That Already Work: Wrapping routine actions (such as taking an exposure, slewing to coordinates, or rotating a filter wheel) inside an AI loop is poor engineering. It takes basic tasks that standard software executes with 100 percent reliability and makes them slow, costly, and vulnerable to random model errors.

Connecting NINA's API to a Model Context Protocol (MCP) server lets a language model turn "Point to the Moon and expose 5ms" into a slew command. That is a useful testing tool, but it remains a basic natural language interface, acting as little more than a conversational remote control.

Real operational autonomy requires a clean separation between two distinct layers:

• The Execution Plane (Inner Loop): Low-level hardware tasks like slewing, plate solving, autofocus routines, filter changes, exposure counts, and PHD2 guiding corrections belong entirely to NINA and PHD2. They run locally and reliably without any token overhead.
• The Supervisory Plane (Outer Loop): The operating environment is always changing, whether that means weather forecasts, sudden haze, seeing shifts, or multi-night project quotas. The supervisory agent operates only at this higher level, evaluating qualitative trade-offs, handling unexpected interruptions, and passing pre-validated sequence queues down to the execution engine.

Alongside NINA and PHD2, an automated session recorder aggregates real-time target data, filter progress, guiding RMS, and environmental telemetry into structured records.

Hermes manages the overall session lifecycle. Built on the open-source agent framework from Nous Research, Hermes monitors workflow state, checks pre-authorized operational boundaries, and coordinates external alerts over Telegram.

Continue

Follow the pre-approved sequence without intervention, covering standard imaging runs.

Ask

Query the model for qualitative decisions, such as ranking standby targets during an unexpected clear window.

Escalate

Alert me directly if an issue falls outside pre-approved parameters and requires manual intervention.

Dispatch

Send an automated dispatch to on-site support in Bhagalpur when a physical action is required.

If a primary target finishes early, the language model might suggest switching to NGC 6888 to complete a missing filter set, even if the object has already set below the local horizon.

PROPOSAL    resume NGC 6888, 1h Ha still needed to close
CHECK       TARGET      ALTITUDE   LIMITS   TIME LEFT   VERDICT
            NGC 6888    -13.0      n/a      48 min      FAIL
RESULT      Target is below the horizon; rejected before reaching sequencer.

Because Hermes evaluates physical constraints before passing any plan to NINA, impossible suggestions are rejected instantly.

Supervised Desk Work via MCP: The Proving Ground

Before deploying Hermes as an unattended background daemon, the underlying toolchain is validated interactively through the Model Context Protocol (MCP). Exposing NINA's API through an MCP server lets a language model inspect system state and test complex multi-step routines during supervised desk work.

MCP translates conversational commands into structured JSON tool calls, letting the model query project databases and coordinate hardware commands while NINA handles execution. The raw video files and sub-exposures remain untouched on network storage until manual post-processing on the workstation. The stacked lunar terminator close-up in the galleryThe best twenty percent of the frames, stacked in AutoStakkert shows the result of a run like this.

This MCP setup is the engineering testbed: it validates that tool definitions, API serialization, and hardware feedback loops work flawlessly. But it is strictly a stepping stone. Hermes packages these proven tool integrations into an autonomous daemon that monitors telemetry across hours, evaluates contingencies, and runs the entire observing session without human presence.

Walkthrough: An Unattended Prototype Night

To show how telemetry logging, constraint checks, and the decision engine operate together in a production scenario, here is the chronological breakdown of an automated observing session planned for the upcoming winter season.

The project queue holds two primary targets: NGC 7000, which needs a final 3h20m of Hα before setting in the west, and IC 40534.5 hours on the Auriga complex, March 2026, which requires several hours of OIII data once the 32% Moon sets at 21:51. High clouds are forecast after 02:00.

IC 405, the Flaming Star Nebula34.5 hours collected across multiple sessions

Multi-filter projects like the Auriga complexThe OIII master channel, 140 frames of 300s require filter tracking across months of variable weather, where automated scheduling prevents wasted acquisition time.

The inputs for the schedule are fully deterministic: ephemeris tables provide altitude and moonset timings, the forecast provides cloud trends, and the local session database tracks remaining filter requirements. The operator synthesizes these variables into an organized plan.

The Target Selection Dashboard on this site already performs the mathematical side of this calculation. It computes visibility windows, checks horizon limits, and ranks reachable targets. The night operator automates the final step: executing and supervising that plan within verified boundaries.

EARLIER BUILD

Explore the Target Selection Dashboard

[ LIVE TOOL ]

16:45: Automated Plan Formulation

Hermes initiates ten minutes before sunset, querying ephemeris data, filter requirements, and forecasts. It discards targets that violate horizon limits and formats an initial schedule for Telegram review.

16:48: Defining Operational Boundaries

I approve the schedule, specify a plate-solve rotation check on IC 405 to guarantee framing alignment with March data, and set an approved standby list in case weather interrupts the primary plan.

BACKGROUND

Handling framing errors from rotation mismatches

[ FEB 2025 ]

Checking the plate-solve rotation angle against previous session logs prevents multi-night integration errors. Setting the standby list in advance gives the agent clear, bounded options if conditions change overnight.

17:45: Pre-Flight Hardware Initialization

Hermes runs the startup routine. NINA connects to hardware, cools the camera sensors, homes the filter wheel, and verifies available storage.

18:05: Physical Ground Dispatch in Bhagalpur

Because the roll-off roof is manually operated, the automated schedule still interfaces with a physical human on site. The operator sends the opening request directly to the caretaker over Telegram in Hindi.

With the roof confirmed open, Hermes updates the session log and starts the sequence on plan.

18:20 to 02:30: Routine Imaging

NINA manages active exposures, plate solving, autofocus routines, meridian flips, and PHD2 autoguiding.

IMAGINGNGC 7000 · Hαframe 34 of 40RUNNING SINCE 18:20

Exposure

300s

HFR

2.28

Stars

412

Guiding

0.52 arcsec

Cooler

−10.0 C

Ambient

12.4 C

Dew point

10.8 C

Next

Filter change 21:55

Telemetry snapshot during standard acquisition with NINA running the active sequence.

During standard data acquisition, the supervisor remains quiet, recording telemetry and monitoring equipment health without adding computational overhead to the imaging sequence.

02:30: Weather Interruption and Assessment

At 02:30, high clouds cause the Half-Flux Radius (HFR) to spike while guide-star SNR drops sharply, prompting NINA to pause the active sequence.

Hermes matches the HFR pause against weather telemetry, verifies that the hardware is functioning normally, and waits for conditions to clear rather than generating false alarms.

03:05: Contingency Rescheduling in a Narrow Window

IC 405 reached its six-hour integration target at 01:40, so the frames captured before the cloud pause were bonus depth beyond the goal. By 03:05, forecasts show a remaining 45-minute clear window before dawn. Hermes filters the pre-approved standby list against altitude and horizon limits.

WINDOW      03:20 to 04:05, approx 45 min usable
CHECK       TARGET      ALTITUDE   LIMITS   VERDICT
            M42         53.8       ok       PASS
            M45         47.6       ok       PASS
            NGC 2244    69.0       ok       PASS
RESULT      All 3 targets clear physical constraints; querying model for project priority.

With all three standby targets physically reachable, Hermes evaluates their project utility:

• M42: High altitude; 30-second exposures will complete the unclipped Trapezium core data needed for the existing deep integration15 hours on M42, with a saturated core in longer subs.
• M45: Reachable, but gains little value from a 45-minute broadband run under changing transparency.
• NGC 2244: Reachable, but too short an interval to add meaningful narrowband depth.

Hermes selects the appropriate sequence, loads it into NINA, and resumes imaging automatically.

04:05: Morning Park and Dispatch

When the standby window closes, Hermes parks the mount, warms the sensors, and sends the morning roof closure message to the caretaker.

The morning report confirms that all acquisition targets, hardware resets, and watchdog checks completed without requiring manual intervention.

Expanding from Prototypes to Seasonal Portfolios

Selecting between standby targets across a single evening is an effective prototype, but true autonomy means moving from nightly fallback logic to seasonal portfolio management.

In amateur astrophotography, the core challenge is not picking an object for an isolated hour; it is balancing multi-night projects across a four-month winter season. A complete portfolio might include a faint broadband reflection nebula, a widefield four-panel narrowband mosaic, and an opportunistic comet pass.

This is where the supervisory layer demonstrates its value:

• Intent-Based Scheduling: Instead of building a fixed sequence by hand each afternoon, you define seasonal goals (e.g., "Finish the 40-hour Auriga mosaic and bank 25 hours on M81 IFN by March"). The agent evaluates moon phase, seeing forecasts, and current integration depths to decide which project to load on any given night.
• Dynamic Seeing Allocation: The supervisor monitors real-time sub-exposure quality. If sky transparency or seeing degrades, it dynamically shifts from delicate broadband targets to high-signal narrowband sequences, or safely parks the mount to conserve hardware life.
• Deterministic Guiding Diagnostics vs. High-Level Safety: Guiding loop corrections in PHD2 remain strictly deterministic math. However, the supervisory agent monitors tracking trends. If persistent wind gusts or mechanical slip push guiding RMS beyond an acceptable threshold for three consecutive frames, the supervisor decides whether to re-calibrate, switch to a shorter focal length rig on Pier 2, or safe the telescope.
• Targets of Opportunity: For fast-moving events like bright transiting comets or newly alerted novae, the supervisor can cross-reference orbital ephemeris, verify horizon clearance, and automatically allocate an unpredicted observing window before dawn without manual reconfiguration.

Implementation Roadmap: From Scripts to Seasonal Autonomy

Developing this architecture follows a phased engineering progression, ensuring deterministic reliability at each level before adding autonomous complexity:

The jump from Phase 2 (MCP Tooling) to Phase 3 (Nightly Prototypes) proves the mechanics: can an agent observe live state, respect horizon math, and make a safe decision without crashing the telescope?

The real destination, however, is Phase 4 (Seasonal Intent). Once bounded decisions work reliably over an isolated night, we remove the concept of manually building nightly queues altogether. The human operator defines the high-level seasonal portfolio; the supervisor manages the nightly trade-offs to bring that portfolio across the finish line by spring.

Failure Modes and Observability

Operating an unattended observatory requires accounting for failures in both hardware communication and automated reasoning:

• Stale or Corrupted Telemetry: If the session recorder misreads a stalled sequence as active, downstream decisions will fail. Accurate timestamps and independent hardware watchdog heartbeats are essential for validating system state.
• Valid but Inefficient Decisions: An agent can strictly adhere to constraints while selecting a low-priority target. Logging decision transcripts makes it straightforward to refine ranking prompts over time.
• Outdated Standby Lists: Fallback queues must be maintained to reflect current seasonal targets, preventing the system from using clear sky on completed projects.
• Silent Process Hangs: An unresponsive script looks identical to a quiet, normal session. The morning report must confirm continuous watchdog pings rather than assuming success from the absence of error logs.

To keep debugging straightforward, raw equipment telemetry is logged separately from agent reasoning transcripts. NINA and PHD2 record physical hardware performance, while Hermes logs the contextual decisions and prompt interactions that triggered each action.

The core architectural balance remains consistent: keep deterministic control within NINA and PHD2, enforce hard mathematical boundaries, and use the supervisory model only when conditions require an adaptive decision.

PREVIOUS LOG

Building the observatory this sits on top of

[ JAN 2026 ]