The terms "naming conventions" or "naming customs" refer to the culturally based method of naming individuals. In some Western European cultures, a personal name may consist of a family name and one or more given names. In some cultures this is name given to a child at birth, but in others, the name the child is called at birth may change at different stages in the individual's life. A person may also acquire a different name either due to marriage, events in the person's life or for religious or ceremonial reasons.
Naming conventions are different than naming patterns. The pattern of naming children is also culturally determined but deals with the sequence and use of names within a family and not with the overall method of forming a name in any given culture. Unfortunately, the terms used to describe the cultural-based system of names is not uniform. To some extent the words convention, custom, practice and patterns are used interchangeably.
It does not take some genealogists very long to become aware of different naming conventions. One commonly encountered variation in European research is the use of patronymics, that is, giving a child the name of the father with the suffix -son or -daughter. For example, in Denmark, the father's may be Jens Andersen, or the son of Anders. If this person were to get married and have a son, the son's name would then be the given name plus Jensen or Jens' son (in English). As the countries became more industrialized, they dropped the use of patronymics but there are still countries that use patronymics such as Iceland.
Most of the commercially available genealogy programs do not handle unusual or non-English naming conventions very well.
Here are a few links describing different naming conventions from the FamilySearch Research Wiki:
Sweden Names, Personal
Denmark Names, Personal
Mexico Names, Personal
Netherlands Names, Personal
Swedish naming practices
Jewish Names Personal
There are quite a few more articles in the Research Wiki, just search for articles on the country of your interest.
One of the difficulties of using a generic, European-based genealogy program is maintaining a consistent way of dealing with different naming conventions. The online program FamilySearch.org Family Tree has approached that issue by allowing the user to choose a language for the Family Tree program. The link to the different languages is located on the bottom of the FamilySearch.org startup page. Also, once you enter the Family Tree program, there is a further choice when you edit the name of a person to select a naming standard. There are presently 10 different naming standards. Changing the standard then reflects the language based system of naming convention, to some extent. The system is not perfect but it is a step towards allowing different naming conventions within the same program.
Here is a screen shot showing the list of naming conventions available from the name edit screen of any individual in the program.
It is not clear how the names are cataloged by the program, but it is likely by the first letters of the the surname category regardless of the naming pattern.