This is the exact reason that I changed how specialties work at my table. I basically ask them the question "Could any of your specialties provide a benefit to this roll?" If yes then they get a +1e. If they then fail the roll then I check the parent skill they rolled and if they had 3 dots or more in the skill then they gain a additional momentum. I also let my players buy specialties for skills that are below 3 dots to show limited knowledge in a focused topic.

This lets the guy with a specialty in swords get his 1e on an attack as well as the guy who has a specialty in...