Apparently back when the current sultan's ancestors were just temenggung (in the 19th century), they were in control (on behalf of the sultan) of the islands around riau and singapore. Due to war, they had to move to Singapore, then when the brits came they started to dispossess the temenggung from Singapore, eventually leaving him with some parcels of land in Singapore and recognition (i.e. promotion) as sultan.

Johor itself was technically not part of what the johor temenggung controlled. It was empty jungle with very little population. Left with no land, they developed the coast of johor and expanded - eventually leveraging their influence with the brits to become sultan themselves.