Hi Djole,
what map provider are you using?
The time it takes to open the map is dependent on the provider - plus of course how much of the map is already in your cache. (It may go faster for other points near your first one.)
I don't thik the slow-down is on the server side, or the client side, I suspect it is on the map-provider side. (You could use FireBug, Network tab, to verify that.)
cheers
Bruce