A kinship system is the structure of the social relationships that constitute kinship in a particular culture. Kinship includes the terminology that is used to define those social relationships and the reciprocal cultural obligations that are associated with the terminological categories. The kinship roles are culturally determined and therefore imposed on the individuals by reason of their cultural and linguistic background and unless an individual consciously changes his or her culture and language it is highly unlikely that the individual will escape the restrictions of the dominant kinship system.
Genealogists and genealogy as a study come primarily from a historical and not an anthropological or sociological background so there is little or no mention of kinship systems in the context of family histories. This is likely due, at least in major part, to the fact that most people are unaware of their own cultures kinship system as a system. In a sense they are too close to the forest to see anything but trees. Likewise, kinship studies of very large complex societies are, for the most part, conspicuously absent from the literature.
Familial relationships are for the most part so integrated into the genealogical studies that it is only when there has be an historical shift in the meaning of a relationship terms, that the issue is even discussed. The issue is the tendency that we will all have to impose our own understanding of the familial relationships on cultures that are entirely foreign to our own. We take it for granted, that the terms we use apply universally. That a Grandparent is always the parent our own parent, when in some cultures the relationship may be much more complex than that. Since genealogy is essentially the names of individuals and their kin relations to one another, ignorance of these relationships and the terminology used in any particular cultural context may contribute to completely misleading conclusions about relationships.
Every person belongs to both a biological family of orientation (i.e. mother, father, and if present, brothers and sisters) as well as a procreative family (i.e. spouse, wife, husband, children) In all of these cultural contexts we have a culturally defined terminology for referring to various iterations of the family unit, such as step-father, step-mother, half brother or sister, etc.
From the genealogical standpoint, it is important to be aware of these systems so that we can accurately assess relationships. Some of the factors that go into a kinship system is in the terminology. For example, a female spouse is called a wife, a male spouse is called a husband. English also has words for male and female siblings, i.e. brother and sister. Some subcultures within the U.S. use these terms for other purposes. For example, a Catholic Priest may be called Father, or a member of the same religion might be referred to as brother or sister. The use of these terms is external to the kinship system.
The dominant kinship systems in the U.S. can be defined as open, multilineal, conjugal systems. In English we have the word family that refers to the conjugal unit, no matter how constituted, and the term "relative" that refers to anyone who is considered a kinsman. Our relations are constituted almost exclusively of interlocking conjugal families. As such English speaking Americans are always members of two interlocking families and unless the brothers of one family marry the sisters of another each individual is the unique member of the intersection of both familial groups.
Kinship is an extensive study. Lack of awareness of basic kinship relationships can cause genealogists to make invalid conclusions. If you would like to have a greater understanding of how kinship system terminology may affect your genealogical conclusions, I might suggest starting with Parsons, Talcott, The Kinship System of the Contemporary United States. The complete article is available from the American Anthropologist for a free download.