Although a judge at Taunton County Court earlier found there was no evidence to support the accusation, he backed the decision to take the boy and girl into local authority care because of the risk of future harm.
Earlier this year Judge O'Malley agreed with an expert who reported that 'parental deficiencies' in both the mother and father - who have learning difficulties - could have grave consequences for the children in the future.
Richard Hickmet, for the parents, said: 'There was no evidence, nor was the expert able to say, that the children were in fact suffering from significant harm at the date they were taken into care.'
Mr Hickmet added that judge had 'delegated' a fact finding exercise about the abilities of the parents to the expert rather than making findings himself and that was 'wholly inappropriate'.
However Susan Campbell, for the local authority, said there was ample evidence that the children would be at risk if allowed to remain with their parents.
She said the youngsters had witnessed domestic violence between their parents, and the girl had been sexually abused by a friend of the father, although that matter was reported to police and a conviction followed.
Ms Campbell added the father 'openly denigrated' the mother in front of the children, had 'lax sexual boundaries' and a 'lack of understanding' of the children's needs, while the mother's learning difficulties were so severe that she cannot parent on her own.
The barrister also said there had been a 'marked change' for the better in the behaviour of the children since being placed in foster care.
The court heard the children were taken into care following an allegation made by a third party that the father had whipped them, but the children themselves never made a complaint or had physical injuries consistent with being beaten.
Recognising the importance of the case, Lord Justice Keene, Lord Justice Wall and Lord Justice Wilson reserved their decision in the case and did not indicate when it would be handed down.
STRAND NEWS SERVICE