Yep, just rebatch it and add the olive oil.
-If your soap is hard, you're going to need to cut it up. Make sure you wear gloves since I'm guessing your soap's going to be lye heavy.
-If it's not hard, you can just mush it up and put it in a double broiler, crockpot, or boiling bag.
-Heat it on low heat, add the olive oil. You can add a few TBSPS of liquid (water, milk, etc) if you think your soap needs it, but if it's new soap, it most likely won't need it. Keep in mind that rebatched soap will not become liquidy like it does when originally making it.
-Cook until it reaches a mashed potato-like consistency.
-Glop it into the mold. Bang the mold down on the counter to help release air pockets. Then you can use a spatula to help press the soap down and smooth it out.
Here's Anne-Marie's video on it. She explains it much better than I.
http://www.youtube.com/watch?v=2YcCV0AWutw