Q:
When I switch the HF clock to run from one of the low frequency oscillators I lose the debug connection. What could be wrong?

A:
When switching the HF clock to the one of the LF clocks, the debug interface will also start clocking at this lower frequency. However, by default the IDEs are set up to communicate at a higher baud rate. After switching to a slower clock, the MCU is no longer able to communicate with the higher baudrate coming from the PC side and the debug connection is broken. To solve this you need to manually configure the debug baudrate in your IDE to 32 kHz or less.

Keil:

Go to the Debug tab in the project options and click the Settings button next to the pulldown menu that should be set to Cortex-M/R J-LINK/J-Trace. Set Max Clock for SW below 32 kHz.

. 

IAR:

In the project options go to General Options->Debugger->J-Link/J-Trace and select the Setup tab. Fix SWD speed to 32 kHz or lower.