Language is cultural (ethnically) bound.
The language itself is parsimonious, if it had too much complexities no one would be able to understand it, or speak it. After all, what good is a language that only one or two people can understand how to use? All languages are easy enough to learn given that the individual is constantly immersed, and has a willingness to acquire it.
The way I observe the multiple languages utilized in the world is a hierarchy like a phylogenetic tree. Older languages derive and new languages branch off it, just like how old korean is not the same as modern hangul, chinese and cantonese are not exactly the same, UK english, american english *even yorker vs midwestern (standard) english) have varying regional differences, exactly the same regional difference given in american sign vs say Australian sign. What you learn in the language is exactly part of that culture, which, you will see the world in that view. We have slang for people of non-american race, so do those races have identifier words for americans.
Your reason for why mathematics is different is simple to explain. It's not a language per say - we learn what numbers and variables LOOK like - 1, 2, 3, 4, 5, pi, Θ, α, dx, whatever else you want to toss in, the rest is history. This is all arithmetic, a brain processes operations differently compared to how it will process a question. I daresay you may realize other species in the animal kingdom can process simple math, too. s1 s2, s3 thus, from my reasoning (biological rhetoric) should not be considered a primary language, it is more of a conceptual area that is mutual between species of higher functioning.
