LOL yes, I wouldn’t travel with any sort of white powder in a ziploc bag! 
I think with gemstones you’ll be fine, especially if they are or look like jewelry. I’ve travelled with sea salt and it was fine. They did stop me for bringing candy once because they looked suspicious in the x-ray machine. They ask me if it was candy and that was that. I came from Argentina with palo santo, and kilos of yerba mate, they don’t care about that either.
As long as it’s not in the TSA blacklist you shouldn’t worry!