That's in addition to the actual disorder that makes math difficult, because we guess this wasn't convoluted enough.
Recent studies have suggested that there's some small genetic component to all this, as some participants were intrinsically more likely to be anxious about math, spatial reasoning, navigation, and presumably life in general. But that can be overcome, and there are also huge environmental factors. Having a demanding teacher or an archnemesis who makes you solve elaborate mathematical problems to earn your freedom can kick-start the fear, but it doesn't help that the Western world's method of teaching math is deeply flawed.
Students are told that math is vital, but also that it's an inherent talent. You're either born with it or you're not, like Jedi powers. (Remember, that genetic component is small and easily defeated, like our favorite street-fighting opponents, babies.) After filling kids' heads with these fears, we then tell them they have to ace some difficult timed tests or be held back. And if they are held back, that means they're just plain bad at math. It's like telling teenagers, "You failed your first driving test, so you can't be taught to drive. You'll have to walk forever. Sorry, but this crucial skill simply isn't for you."