Behringer Guitar 2 USB Building Latency

I'm using a Behringer Guitar 2 USB cable to connect my electric guitar to a Mac mini, running GarageBand in monitor mode. All software is up to date. As soon as I start playing all is good, but after a few minutes the system starts building a sensible amount of delay. The only way to reset this effect is by removing the cable from the USB port and then plugging it again or quitting GarageBand and running it again.

How can I fix this? Is there a way to prevent this delay from building up?