If your not breaking any laws, then you are not required to explain yourself, your hobbies, or how you spend your time or money to anyone. Who cares what they think? Do what makes you happy.
If they think it's silly and are ignorant enough to say something, then just nod your head yes in aggreance, smile and say, Yea it probably is, but I like it anyway.

Make note, that is one thing you wont have incommon with those people, so avoid attmepting conversation with them about it. Only becasue they are ignorant about it and you will lose thier interest quickly. I'm sure you can likely find other subjects you are interested in that they are too that would make for a good conversation between you.