I have always done a lot of comparative book testing. Playing one book against another with the same engine and settings. Not only has this provided me with a good way to subjectively judge relative strength of a book, it also is an excellent way to expose weakness inherent in one. If a player can get access to the books being used by his opponents this is an very good way to gain a competitive advantage in rated play.
We all come up against what I call the out of the box books. Fritz’s opening book, Rybka’s book and others that are included with the chess engine software. These books are not updated by the software producers and become out of date quickly in online play. They also have a tendency to be statically based books that lend themselves to utilizing the learn function of the GUI.
A good hand made book must be ready for these out of the box books by knowing their weak lines and exploiting them from the start. There is no better way to find the holes in other book than to do this kind of comparative testing.
There is nothing difficult or mystical about this just sit up your machine with a Fritz GUI client to play engine matches with identical engines and settings. The only variant being the book. And let it play 40-50 games. If the book you are improving is designed for blitz play then simulate depths commonly arrived at in online play by the quad core CPUs. Do this by adjusting the time controls of the match. (a dual core CPU may need to play at 15-0 to 20-0 min. to get the right depth in the middle game) The modern Quad-core CPUs can get a depth of 15 to 21 Ply. in this part of the game.
I recommend that much of this kind of testing be done in your books initial development stage. (when you are still adding games to the main database) Take the games that your fledgling book won and add them directly to the books databases, both main and learn. As for the loses set them aside and use them as the first games to be worked on once the book goes into the hand tuning phase.
In a search for unique content for my website and in the vague hope that it may be of some help. I will began posting the results of my current comparative tests. At this time I am working on blitz books for 3-0 times. I keep one of my machines an old Dell 650 Work Station with two Dual Xenon CPUs busy 24-7 doing this kind of book testing.
By looking at the games generated you may be able to discern some holes and or weakness in books that are commonly available for download on the net.