Sorry about the formatting of this message, the WYSIWYG editor wasn't working for me. Message was edited by: Joe Krukosky
The first thing that comes in mind that you dont really stop/end debugging sessions, may be some even got stuck and consume RAM/CPU? try investigating on server with ps/netstat or other sutiable tools. May be just start from perfmon.
If you sayings that slowness lasts for about 1 hour after debug is over .. then xdebug cookies may still be present and xdebug still trying to connect to debug client on every page view (but the delay should be around 1 second only and it is not configurable, unless xdebug is built from sources).
Those browser extensions/bookmarklets that used to initiate debug session (that I was trying/using) they set such cookie lifetime to 1 hour .. or until browser is restarted. Clearing cookies in conjunction with browser restart should solve the problem completely.
In any case: after you finish debugging .. try using "Stop" functionality of the tool used to initiate it in first place (e.g. "Stop debugger" bookmarklet or deactivate/stop debug session if browser extension is used -- obviously, it needs to be initiated when you working with such site, as cookies are set/removed on per domain base).