I had a look inside my decoding cache folder, and there are numerous multiple copies of the same file in there. When I open a new set Ableton is making new copies of all the files that it decoded. The folder has also exceeded the limit set in the preferences.
I'm definitely seeing a strong case for pre-converting any mp3 files. Evidently when you allow Ableton to handle them behind the scenes things can get messy. Not to mention, if your files are on an external audio drive (as they should be) and you open Ableton without that drive hooked up Ableton will reset the cache folder back to your internal drive, effectively creating another copy. Then when you switch back it has to rebuild the entire thing over again. So you wind up with Ableton decoding these files multiple times, and it takes a long time.
-Wesley