If society were structured that way (or if it had been), then probably. But there are notably fewer people who are against matriarchy for, I reason, the fact that matriarchal society has not been prevalent in the history of civilization and it isn't today. Most philosophers who are against patriarchy would argue that patriarchy is bad because of its power relations, not some mystical "innate badness" of the male sex. As such, their criticisms would likely apply to matriarchy too.