You happen to have, by any chance, some updating services running? Scheduled tasks, any?

Apart from the garbage collect, which is inavoidable the slow downs are most likely either some software running in the background (can be as simple as Messenger) suddenly waking up or system measures against overheating/overflowing workload.