I needed to delete some files from source control and noticed that it was taking Visual Studio/TFS about 30 seconds to delete a file (move it to the delete pending list). Since I needed to delete a lot of files, you can imagine how annoying this was. So, I fired up good old Process Monitor and started watching devenv.exe. Then I went into Source Control explorer and deleted a file. Suddenly Process Monitor was full of folder accesses. Dozens of folders were being scanned. After some research, it turned out that when I deleted a file for some reason Visual Studio was looking through all the folders in the same workspace as the file.
So, I created a new workspace and put my project and only that project in the workspace. Now when I try to delete a file it happens very quickly.