This is all down to the fact that English is a language of acutely mongrel lineage. It has substantive roots in Celtic, Romance and Germanic languages (to name a few) and a grammar that lends itself well to the adoption of "loan words" (non-native words adopted into the native tongue.)
The "standard" means of pluralising a noun is to append -s, with some conventional variations (eg -f becomes -ves, -y becomes -ies) for convenience in spelling and pronunciation.
However latin-based words tend to pluralise in the latin fashion, so for example bacterium becomes bacteria, and cactus becomes cacti. Similarly greek-based words will adopt the equivalent pluralisation appropriate for the original root.
Still other words of Saxon or earlier origin have lovely, earthy plurals that defy the "conventions" due to their traditional forms being maintained. Geese, Mice and Children owe their unusual conjugations to their ancient roots, and to the fact that they are common words whose everyday repetition keeps them from slipping into bland conformity.
In my experience, words which do not pluralise are those which relate to herding, hunting and the counting of animals. These words tend to be saxon (germanic) or celtic in origin owing to the presence of farming and hunting in Britain long before the Norman invasion. This can be inferred by the fact that sheep, cattle and game do not pluralise, while whales, sparrows and elephants (seldom hunted or farmed in Britain!) definitely do.
I suspect these tend to be a contraction of the traditional counting forms for such cases ("head" of cattle, "brace" of partridge, "shoal" of fish) but this doesn't really answer the question of why such plurals take the same form as the singular. It could be that when counted in such a way, the animals being counted were considered an uncountable, continuous quantity (similar to water or money) that could only be "counted" when quantified with their associated counter, so cattle would be rendered an uncountable noun by its quantifying counter head. It's interesting however to note that bird pluralises to birds, while aircraft does not pluralise.
Sadly for the non-native speaker, this makes learning the "rules" of English an arbitrary and frustrating affair. However, spare a thought for the Japanese, who do not have plurals for any but a few unique nouns, and must instead learn a separate counting-suffix and corresponding character (kanji) for almost every class of noun imagineable. There are in fact entire volumes of the things, and it would be nigh-on impossible for any person to learn them all. Wikipedia lists a choice selection.
I initially thought it was French, but actually it's a draw between French and Latin. There have been many studies on this topic, which came up as you would expect, with different percentages, but the conclusion remains the same.
There is a paragraph about this topic in the Wikipedia article, and another dedicated Wikipedia article that lists English loanwords by country and language of origin. One interesting conclusion in the first article is that if you consider only very common words, the Anglo-Saxon proportion is more important, which is a clear indication of English's Germanic substrate.
You know, if you had asked the question a couple of centuries ago, it would have been French alone. The reason why Latin has caught up is because many scientific and medical terms are coined after Latin. Remarkably, the second largest contribution of Latin is actually the result of the introduction of Christianity from the 7th Century onwards and the Roman occupation comes only in third place.
The French influence really began with the Norman invasion of 1066; but that was only a start, as it was followed by a continuous influx of French immigrants in England throughout the Middle Ages and even after. Indeed, a few of the early Norman kings rarely set foot in their English kingdom and couldn't speak English. Saxon English nearly became an underground language. However, many of these immigrants came without women, and once they'd intermarried with the locals, after a few generations their descendants could not speak French any more.
All considered, this was only too fair because the Norman people were actually of Scandinavian origin and they had made French their language for the same reason: Norman armies had landed on French coasts with very few women. The date of the establishment of the French Dukedom of Normandy is 911. Compare this to 1066. A mere 155 years had turned the North men into the Normans.
Other notable contributions to English are Celtic (see below), Old Norse (due to the Viking invasions), Greek, Dutch (with William and Mary and the Dutch wars) and a variety of other languages especially from the former colonies.
Language origin specialists theorise that the proportion of loanwords in a language is a function of both the length of exposure to and the social status of the borrowed language. For this reason, there are comparatively very few Celtic words — whiskey being one of the most prominent ones. One clue to this apparent oddity is the double meaning of Wilisc in Old English, the Saxon word for Celt (⇒ Welsh): it means both "foreigner" and "slave".
Best Answer
Would you also insist that Japanese speakers pluralize English words when used in the plural?
Once a word has been borrowed into a language, it adheres to the grammar for normal words in that language. We don't borrow Japanese grammar, just words, so there is no need to use a zero plural with borrowed Japanese words.
It is true there is a small set of animal nouns in English that have a zero plural, but they are not borrowed and are a special case.
This is borne out by the results in the Corpus of Contemporary American English, which has 68 incidences of "ninjas". Of the 496 incidences of "ninja", there are a handful of uses of "ninja" as a zero plural, as OP suggests, from the script for the 2003 film The Last Samurai, but the vast majority are singular (or attributive) usage. So it does appear that zero-plural "ninja" is used, albeit uncommonly so.
So yes, ninja meaning ninjas is a usage that gets some use, but regular pluralized ninjas is more common, and perfectly grammatical.