Do you have to fight all those ghosts when you're bringing the tome back, or can you run right through them to replace the book?
My group got completely owned trying to fight our way through them, and hadn't even broken a sweat up until that point.
Its easier to fight them I think. It's possible to run through and place the book, but I dunno how effective that'd be, because they have slow skills that slow down the already slowed runner.
Just pull 3-4 at a time, and it shouldn't be a problem.