Normal UPS' does not have any special filters so won't help with these kind of issues.
Adding a seperate common mode choke to the input power cable of the computer/speaker might help, but usually computer power supplies has a common mode choke, since the OP has said that does not filter, adding a new one won't make a much of a difference.
My assumption is, there is a high frequency leakage in somewhere in the circuit,(It may be in the AC or most probably in a LED lamp), and somehow it picks up by Audio circuits.
If OP has a laptop, he can see if this happens even when the laptop is running on the battery. then we can figure out if the interference is wireless or coming from the power line.
My guess is: High frequency leakage from a cheap LED lamp or CCTV power supply, these cheap LED lamps or CCTV power supplies has cheaply made circuits which leak wide range of high frequencies that are not filtered by standard common mode chokes.