I don't think you need to worry about political correctness, as the Philistine race is extinct. I've certainly used this insult a great deal. :)

An aside: If you study the Bible carefully, it is likely the Philistines contained some DNA passed on from the giant race the Nephilim, hence Goliath. Goliath is actually pretty short in comparison with some of the other giants in the Bible; a mere 9 feet tall compared with, say, Og of Bashan (over 13 feet tall and six feet wide).