It could be one of the Oblivion problem, hdd caching. The cached files get fragmented as oblivion really use the hdd, alot.
The official way to solve this is when you put in the disc, hold A until the bethesda logo loads up. It should clear the cache files. (then start of the game you can hear constant hdd reading sound from xb360, game slow-down for awhile...recache perhaps)

I tried it...it does 'feel' faster but it could be my imagination