To my mind, Grave of the Fireflies has nothing to do with which nation was ideologically right or wrong in the conflict. It has everything to do with the suffering of innocents, as you point out in your second paragraph. It isn't a political film in any way. It's a human film. So yes - there absolutely were victims on the Japanese side as much as the US side. The same was true in Germany.