It seems that adding that many polygons to the model does not have a hit on frame rate (weird)
That'd depend on the framework. Rembrandt has a very simple geometry stage and does all the work per pixel, i.e. it can crunch models with high vertex counts incredibly efficiently but loses performance as compared to other strategies on scenery where polygons are large. Dependent on the precise set of effects running for the model, a vertex-shader dominated strategy might execute rather slowly.
Also, it depends on how many 'that many' are - my old GPU could crunch about 500.000 vertices in terrain and models without large performance deterioration, then it weakened significantly (with the new one, I haven't ever been able to measure any performance hit due to terrain, it largely goes with clouds only).