rsync speed goes down over time

What could be reasons for my rsync, which is syncing two remote servers through ssh, to slow down over time like this? It keeps happening. How to check what is the bottleneck?