Put the potatoes in a large saucepan with enough water to cover by 2 inches; season with salt. Bring to a boil over high heat, and then reduce the heat to maintain a steady simmer. Simmer until a paring knife just pierces through the potatoes, 20 to 25 minutes.
Meanwhile, combine the oil and rosemary in a small skillet. Heat over medium heat until the rosemary sizzles; cook until fragrant, about 1 minute, and then remove from the heat and let stand.
Drain the potatoes. When cool enough to handle, use a thin-bladed sharp knife to cut each in quarters. Very gently toss with the rosemary oil and olives. Season with salt and pepper. Serve warm.