Justin Fox has a good, balanced article on the market for college textbooks (an interest of mine, for obvious reasons). Below is a noteworthy figure from it.
These aggregate data reflect my own experience: While the market share of my books is as robust as ever, my royalty income is down in both real and nominal terms. (Don’t worry. I’ll be fine.) Students are increasingly switching to buying less expensive options, such as digital or loose-leaf versions rather than the traditional bound book. For example, for my intermediate macro text, the loose-leaf version can be purchased for about half the price of the hardcover book.
Open-source teaching materials may also explain part of this trend, though I suspect it is not the main force at work here. At least for economics courses, the available open-source options appear not to have much of a market share. I do not expect that to change. As Justin suggests in the article, relying on funding from foundation grants is probably not a sustainable model.
One possible implication of the shrinking pie is that fewer professors should enter the market by writing their own textbooks. I don’t know of any data to gauge to what extent this is occurring. In any event, it is not a good time to be entering the market with a new textbook.