The "different form of life" comment from GRRM confirms that Old Nan is wrong in her statement that they are "cold dead things".
Here's Leafs quote about the old races that were native to Westeros. You will not find any mention of White Walkers.
Gone down into the earth … Into the stones, into the trees. Before the First Men came all this land that you call Westeros was home to us, yet even in those days we were few. The gods gave us long lives but not great numbers, lest we overrun the world as deer will overrun a wood where there are no wolves to hunt them. That was in the dawn of days, when our sun was rising. Now it sinks, and this is our long dwindling. The giants are almost gone as well, they who were our bane and our brothers. The great lions of the western hills have been slain, the unicorns are all but gone, the mammoths down to a few hundred. The direwolves will outlast us all, but their time will come as well. In the world that men have made, there is no room for them, or us.
Also as magically transformed humans they are not even a species anymore than Melisandre or Moqorro who have also been magically transformed are.
From the Worldbook.
Regardless, the children of the forest fought as fiercely as the First Men to defend their lives. Inexorably, the war ground on across generations, until at last the children understood that they could not win. The First Men, perhaps tired of war, also wished to see an end to the fighting. The wisest of both races prevailed, and the chief heroes and rulers of both sides met upon the isle in the Gods Eye to form the Pact. Giving up all the lands of Westeros save for the deep forests, the children won from the First Men the promise that they would no longer cut down the weirwoods.
So after failing in their second attempt at shattering the continent with the Hammer of the Waters, the Children realized they could not win. They gave up ALL of the lands of Westeros except for the deepest woods and all they got in return was a promise that men would not cut down Weirwoods.
That is not an even truce, that is a defeat.
Drift apart?
Only here, where the King in the North thre back every army that tried to cross the Neck, did the rule of the First Men endure. The Andals burnt out the Weirwood groves, hacked down the faces, slaughtered the Children where they found them, and everywhere procliamed the triumph of the Seven over the old gods. So the Children fled north.
The Andals were slaughtering them and more importantly breaking the pact by cutting down Weirwoods, that is not "drifting apart". Note also that the Children didn't flee to "the North", they simply fled "north".
"But the Children of the Forest are all gone now you said."
"Here, they are", said Osha as she bit off the end of the last bandage with her teeth. "North of the Wall, things are different, that's where the Children went, and the Giants, and the other old races.