American nazi party wasn't that weak. Roosevelt walked a fine line because so many people supported the Germans. If I remember my history correctly, the country was pretty evenly divided, though most wanted to keep out of the war. That all changed after Pearl Harbor. Had Hitler not declared war in support of Japan, things might have gotten interesting.
You're confusing opposition to war with support for the Nazis.
http://staff.imsa.edu/socsci/jvictory/wwii_antiwar/time_pub_opinion39.html
The breakdown would have been about
30% for entering on the side of the allies.
10% for entering on the side of the axis.
45% for neutrality, but allies if necessary.
15% for neutrality, but axis if necessary.
Keep in mind this is before the fall of france, the battle of britain, and the invasion of the soviet union. On December 6th, 1941, I would estimate public support as about
50% for entering on the side of the allies.
2% for entering on the side of axis.
40% for neutrality, but allies if necessary.
8% for neutrality, but axis if necessary.
Obviously after Pearl Harbor, nearly everyone supported war with the axis.