If you're making dust then you should definitely wear protection no matter what kind of wood you're working with. Some woods like ebony are outright are toxic but even dust from innocuous woods has negative effects when it ends up in your breathing passages and lungs.

Sensitivities seem to be somewhat personal. I figured out that birches cause me some minor issues. But it's easy enough to wear a mask.