Sometimes the best option is to tell a potential client that you are unavailable to shoot, or to explain that this isn't the type of work you do. It's just so crazy that it actually works and can improve your business!
Photography is an art form, and if you haven't noticed yet, artists specialize in certain styles and subjects. There are even artists that have made entire careers out of drawing an outline of one shape, like a triangle! So how is this possible? How can someone ever become famous if they only do one thing?
By specializing your photography (you can read more in this article about specializing) you can truly become the best at what you photograph. When you focus on one aspect, you become the expert on it and will be highly demanded for that type of work. The benefit is that you get to shoot exactly what you want, and don't have to waste your time on jobs that have nothing to do with your aspirations. Trust me, those jobs won't help you much aside from making a few bucks and becoming known as a "do-it-all" photographer who produces "OK" work, not "GREAT".
When a potential client comes up to you that isn't in your field, let them know! By telling them you don't shoot that genre, and that your focus is "action photography" or whatever it may be, you are informing them of your expertise in that area. Then you won't be wasting time on a job that you don't specialize in, and the client will keep you in mind and mention you to anyone who is looking for your specialized work.
It may be tough weeding out the clients at first when you don't have enough money to get buy, but in the long haul it will pay off tremendously. By doing this, I've become known as an "action photographer" in my hometown. This helps me get athletes to shoot with, companies to shoot for, and lets me keep doing what I love!
This article came about after reading a great article from "The Great Business Project" magazine. There's also an interesting article here about firing clients that aren't good for business: The Great Business Project Blog: Firing Clients