General harp clock synchronization failure

[glopesdev]

Over the weekend we noticed a generalized drift across all Harp devices of both AEON3 and AEON4 rigs. Upon further monitoring it seems like all devices continued operation, but were not being synchronized by the timestamp generator / clock synchronizer devices.

We are currently unsure of the reason for the failure, or whether there is any relationship between the almost simultaneous occurrence of the issue on both machines with different synchronizer devices. We are not sure also if this could be related to the update to the Harp 1.13 core which was done at the beginning of the experiments.

Complementary debugging strategies to try next:

• Split the clock signal to an input expander (using BNC adapter) and log the raw clock stream
• Consider using the new timestamp generator gen3 functionality to report connected devices:
  Implement and expose connected devices register in device.yml harp-tech/device.timestampgeneratorgen3#9
• If the issue happens again, try to debug live with @filcarv by replacing the clock generator with a replacement device while leaving the affected generator in place

[aspaNeuro]

To be able to understand better the problem a procedure as been agreed.
Whenever this happens again we need to replace the timestamp generator on the rig with the problem while keeping the old one connected to the computer.
There is a timestamp generator in the desk to be used has a replacement.

The procedure is:

1. AEON4 Unplug the sound card from the USB hub
2. Connect the new timestamp generator gen 3 to the available port in the USB hub.
3. Connect all the clock minijacks that where connected to the previous clock generator to the newly installed one
4. Go to the computer device manager, expand the Ports (COM & LPT), open the COM5.
5. Remap the current COM5 to COM30, open the port settings tab, click advanced and on the Com port number drop-down select COM30, then press OK to close both windows.
6. Remap the new timestamp generator to COM5, for this first is needed to discover the port number that was added when the new clock was connected, so open the bonsai shortcut, and create a new workflow.
7. Add a device(harp) to that workflow, on the properties panel select a COM number, it should be the highest (except 30) in the drop-down.
8. On the command line window should appear TimestampGeneratorGen3, if not repeat step 7 with different port numbers (except 30) until it does.
9. repeat step 4 but with the newly discovered COM port.
10. Repeat step 5 but rename the new device to COM5.
11. Is now needed to synchronize the new device, so close the "SystemStartup if is running", press the windows key, and type run, then in the text box type "shell:startup"
12. Double click on the "SystemStartup-AEONx"
13. The device should now be synchronized so run experiment like usually by double click on the desktop shotcut.

Further actions may be needed to support @filcarv in remote debugging.