The Strawman Cometh
Not only is it unfair, it's extremely dishonest and intellectually lazy. It's your typical, garden variety strawman which is only meant to distract from legitimate issues involved in a subject or debate. Something that is akin to saying "feminist issues don't need to be addressed, because they are wrong in their attitude toward men." The obvious fallacy lies in the fact that feminism's legitimacy should be based on the actual validity of their positions. If we assume, for the sake of argument, that feminists don't care about violence against men, then that doesn't invalidate such issues as equal property and contract rights, equal voting rights, opposition to forced marriages, sexual and reproductive rights, economic and employment equality, and, of course violence toward women.
All of these, and many more, are issues that feminism has dealt with and none of them are dependent on feminist's attitudes toward men in order to have value as an issue. In fact, every one of them is an issue that could be debated, independent of feminism's advocacy for them, as a stand alone subject. You certainly don't have to be apathetic toward violence against men to believe that women should have a right to vote, receive equal pay, and not be sexually harassed or physically abused.Men Are Big Boys
The reason that feminism is associated closely with those issues is because they are issues that profoundly affect women. For obvious reasons, feminists have a primary interest in issues that effect women. It really shouldn't be surprising that a group chiefly populated by women would take up matters that relate mostly to women. By the same token, why shouldn't men be the ones advancing uniquely male issues, such as violence (exclusively) aimed at men? It generally follows that the person or group most effected by discrimination drives the discourse surrounding that topic. All the more so, being that feminist issues generally involve women's treatment by, or in relation to, men.
Furthermore, men simply don't need feminists to run to their rescue, because they are very much capable of standing up for their own rights. Men are overwhelming represented within every segment of society, including government, business, and the media. There is no shortage of access to venues for men to address issues that predominantly effect men. By contrast, most feminist issues would receive little, if any, attention if not for the fact that a group devoted to issues effecting women exists.
Having established that men are the ones responsible for men's issues, yet another reason this isn't a fair question is because it assumes and/or falsely implies that feminists aren't in fact involved with issues that relate to violence against men or other non-female groups. Truth is that activists are rarely involved in one, singular group or issue. There are many subjects and political disputes that effect individual activists that might not necessarily be universally connected to women's issues. Thus, while they might appeal to certain feminists, they would not be a part of feminism itself. For instance, a vegetarian feminist is likely to be involved in animal rights groups; an Asian feminist is likely to be involved in anti-racism groups; a feminist, who is also a pacifist, is likely to be involved in anti-war groups, etc.
It's rather naive or simple-minded (if not both) to believe that people involved in one group are incapable of being involved with or having sympathetic feelings toward other groups and issues that might be outside the realm of the first group. In regards to violence, it would be unlikely that you would find a peace rally that did not have its share of feminist attendees. While that might not relate specifically to violence toward men, it certainly would include them. So, not only is it unfair to make such a statement, it is an expression of ignorance and probably willful ignorance at that.