During the night of Sunday the 8th of December 2013 a sudden storm at the Boyden observatory caused a power outage that resulted in Watcher taking a rather unwanted shower. Since then Watcher has been undergoing maintenance to several different components in order to bring it back online.
The telescope software (RTS2) is, by default, a very paranoid piece of code. It contains a lot of fail-safes in order to protect the telescope from a variety of sources. One of these fail-safes is the list of required devices. If communication to one of the required devices is severed then the dome closes and observations stop.
The three required devices for Watcher are the rain sensor, cloud sensor and the uninterruptible power supply (UPS). If the telescope software loses contact with any of these systems, or if the sensors trigger for rain or cloud, the dome will close until everything is safe to observe again.
The Gathering of the Clouds
During the summer of 2013, the communication system (polling) between the telescope and the UPS was continually failing. In order to prevent unnecessary dome closures, the UPS was removed from the required devices list. Despite this, the UPS remained functional and provided to the entire system during the frequent power outages that the observatory undergoes.
The polling frequency on Watcher has been set to 300 seconds, which is also intended to prevent unwarranted dome closures due to temporary outages in polling the rain sensor.
The Clouds Burst
Everything happened very quickly on Sunday the 8th, all in less than 2 minutes. The UPS was not fully working, and due to being removed from the required devices this failure went unnoticed. Up until the very last moment before the failure, the cloud sensor detected standard amounts of cloud cover for the time of year, which was well within observing requirements.
The approaching storm caused a power outage in the observatory, and 14 seconds before we lost power completely, the rain sensor detected the first drops and initiated a dome close. The dome close procedure was not completed and Watcher was left exposed to the elements as the storm rolled in.
The Return Journey
As it stands, the majority of Watchers systems are back in working order. Despite taking a bath, only a few of the components were damaged. The Andor CCD has been sent back to the manufacturers to be serviced and have the cooling system replaced.
The power supply for the mount also required some repair, but thankfully it was able to be fixed on site.
We have reattached the original Apogee CCD to Watcher, which requires some work to allow it to work with the RTS2 software, and the telescope will also need to be re-focused to allow clean images to be taken.
Due to the extended outage and the changes to the system, a new Astrometry model will need to be developed to allow the telescope to detect its pointing direction and correct its position.
We still have a way to go, but we are well on way to having Watcher back in action.