Here is an interesting article by Rabbi H. Shachter of Yeshiva University
http://www.torahweb.org/torah/2006/parsha/rsch_vaeschanan.html
In it , he discusses the prohibition of intermarriage, and how Jewishness is determined.
He mentions several opinions in the Talmud (without proper citations) , the minority opinons seem to be also held by Karaites - namely, that Jewishness is determined either by both parents being Jewish, or either of them (father or mother) being Jewish.
Below is a copy of the article:
Intermarriage
In one possuk at the end of parshas Va'eschanan (7:3) the Torah prohibits both forms of intermarriage: a Jewish man may not take a non-Jewish woman, nor may a Jewish woman marry a non-Jewish man. In Shulchan Aruch (Yoreh Deah 157:61) the opinion of the Ramban (Milchamos, Sanhedrin 74) has been adopted, that there is a big difference between the two aforementioned cases. Because in the case of a Jewish man taking a non-Jewish wife the children will not be Jewish, this prohibition is considered more serious; it is considered as if the man had become a "mechuttan" with the avodah zarah. This is the end of the line! The tradition of Jewishness transmitted from Mt. Sinai from generation to generation will not be able to continue. But when a Jewish woman marries a non-Jewish man, the children will be Jewish; the transmission of Jewishness will continue. The woman has violated a serious aveira, but this is not a case of yehoreig ve'al ya'avor.
In Europe the common practice was that when a Jewish man would marry a non-Jewish woman, this was considered equivalent to his converting to another religion (shmad). However when a Jewish woman married a non-Jewish man, the custom was not necessarily so. This aveira was not considered the equivalent of shmad.
Whenever there is a "mixed" marriage between two Jews, for example when a Kohein or a Levi marries a girl who is not a Kohenes or a Leviah, the status of the children is determined by the father. The same is true when there is a "mixed" marriage between two non-Jews. Amaleki, Edomi, Mitzri, and Canaani each have a special status according to the halacha. When there is a mingling between two nationalities, the halacha declares that all the children follow the nationality of the father. This halacha is based on the possuk in Parshas Bamidbar (1:2) "l'mishpechosom l'beis avosom", which implies that in cases of a conflict, the mishpacha of the father is to be followed. The only exception is where there is a mixed marriage between Jew and non-Jew. In Talmudic times none of the rabbis felt that in these cases the status of the children should be determined solely by the father. One opinion felt that in order to be Jewish one must have both a father and a mother who are Jewish. A second opinion held that with either parent being Jewish, all the children would be considered Jewish. And the accepted opinion is that the issue is determined solely by the mother[1]. This position was arrived at based on the Rabbi's careful reading of the pesukim (7:3-4) at the end of our parsha. The Reform movement's renunciation of this position was a rejection of a tradition that has been accepted for over 1,500 years.
It is interesting to note that in a marriage between a Jew and a non-Jew none of the rabbis felt that the status of the children should be determined by the father. If in the other two types of mixed marriages (where both parents are Jewish or where the parents come from two different non-Jewish nations) the halacha established that everything is determined by the father, what motivated the rabbis to assume that the same should not be the case when a Jew and non-Jew marry?
The answer lies in the wording of the possuk in Bamidbar (ibid). The status of the children is determined solely by the father when we're dealing with an issue of "mishpacha". Being a Kohein or Levi is an issue of mishpachas kehuna or mishpachas leviah. The same is true regarding Amaleki, Edomi, etc. we colloquially refer to these groups as "nationalities", but strictly speaking (halachically) they are merely "mishapchos". In order to be a member of a certain mishpacha, you must have yichus (genealogical lineage) of ben achar ben through your father. Being Jewish, however, is not a function of which mishpacha one belongs to. This is illustrated by the institution of geirus (conversion). After conversion, a ger belongs to no mishpacha, but nonetheless is just as Jewish as all the other Jews. Being Jewish is a function of belonging to the Jewish people (Am Yisroel). The Jewish people are the only ones called a nation as such! "Umi ke'amcha Yisroel goy echad ba'aretz" (Shmuel II 7:27)[2].
The rabbis apparently assumed that since "mishapacha" and "am" are fundamentally different, it must be that inclusion in each one will be determined by different factors in the case of a mixed marriage. A major difference between a mishpacha and a nation is that a mishpacha consists of a collection of individuals who relate to each other in a special way, while the term "goy" (nation) comes from the word "geviah" (body). Klal Yisroel is considered "one body". We must adopt this attitude and act accordingly.
[1] Tosafos Yevamos 16b, s.v. oveid kochavim, and Rabbi Akiva Eiger (Gilyon Hashas ad loc.)
[2] See "Chilul Hashem" where we explained in a similar vein why the actions of one Jew are seen as a reflection on all Jews, as opposed to other nations where the actions of an individual are not understood as such.
He then tries to point out that there was not a rabbinical opinion which claimed that Patrlineal descent is what determines Jewishness.
however, he as admitted that it can be either patrilineal of Matrilineal, according to one of the unnamed rabbis.
In any case these minority views have never been accepted as Rabbinic halacha, but nevertheless it is interesting that such views existed.