Orsdxy

This society traces its lineage through matrilineal lines, with witchcraft being exclusive to the female line. Ritual magic involves using the mana of the world and directing it in a way to suit you .It requires chanting, numerous ingredients, and a group of people depending on the spell. Society has evolved to raise boys in bulk for their unskilled labor, and are considered "sons of society". Girls are kept at home by their mother for proper training in witchcraft and are referred to as "daughters of their mother".

Family units are important because they provide for the needs of indiviuals and introduces them to the broader social structure of their world (sex and companionship between adults, raising and rearing of kids, protection from threats, etc). The classic "two parents + children" is common because it naturally organizes around human reproductive patterns and fulfills these requirements. This setup is unique in that it raises boys and girls differently due to magical ability. I need to make this system culturally viable over the long term to meet these needs. How can I do this?

This is also similar in some ways to the Spartan system.

http://www.historywiz.com/didyouknow/spartanfamily.htm

Children were children of the state more than of their parents. They were raised to be soldiers, loyal to the state, strong and self-disciplined.

It began in infancy. When a Spartan baby was born, soldiers came to the house and examined it carefully to determine its strength.The baby was bathed in wine rather than water, to see its reaction. If a baby was weak, the Spartans exposed it on the hillside or took it away to become a slave (helot). Infanticide was common in ancient cultures, but the Spartans were particularly picky about their children. It was not just a matter of the family, the city-state decided the fate of the child. Nurses had the primary care of the baby and did not coddle it.

Soldiers took the boys from their mothers at age 7, housed them in a dormitory with other boys and trained them as soldiers. The mother's softening influence was considered detrimental to a boy's education. The boys endured harsh physical discipline and deprivation to make them strong. The marched without shoes and went without food. They learned to fight, endure pain and survive through their wits. The older boys willingly participated in beating the younger boys to toughen them. Self-denial, simplicity, the warrior code, and loyalty to the city-state governed their lives.

This setup is not unique. Levi-Strauss described very similar tribes where boys were raised by whole village in common huts while girls were with mothers and aunts who teached them skills unique for their gender.

You present matriarchal system where women, through magic, are the stronger gender. If you take "Pride and Prejudice" you have exactly your setup. Boys who are send to school and given jobs, males being only heirs and women raised in bulk with no particular skill above the ones needed to accompany their husbands.

Society has evolved to raise boys in bulk for their unskilled labor

This sounds like a society where males' only role is to impregnate a woman, she being the core and value bearer.

Therefore I would also expect that a family could be defined as "the group of women related by birth relationship", with no place for males, who could be just considered members of the "extended family", as they were just picked up to provide the semen on that particular day.

The males would then have to struggle and prove their value to be picked for mating, no serious and respectable woman wanting to mingle with a man of dubious reputation.