You can get to Two Harbors from the San Pedro terminal (limited schedules) and from Two Harbors you can rent boats on either side of the island.

Maybe that is a more centrally located spot to start your tour from???