"Is it each of the books is or each of the books are?"

The formal rule is that any noun phrase beginning with each has to take a singular verb, so technically each of the books is is correct and each of the books are is incorrect. The same rule holds for phrases beginning with every: formally, you have to say each/every (one) of the books is, not each/every (one) of the books are.

But this is one of the many cases where our perceptions of reality clash with the formal rules of grammar. Naturally, when we say each of the books we are talking and thinking about all of them, hence the desire to use a plural verb (compare all the books are). Another problem is that in English we tend to want to make the verb agree with the noun that's closest to the verb whether or not it's really the grammatical subject of the sentence. That's why it might seem strange to put is after books in each of the books is, but for the reasons given this is the "correct" way to say the phrase.