• 14.2 MovieLens 1M数据集
    • 计算评分分歧

    14.2 MovieLens 1M数据集

    GroupLens Research(http://www.grouplens.org/node/73 )采集了一组从20世纪90年末到21世纪初由MovieLens用户提供的电影评分数据。这些数据中包括电影评分、电影元数据(风格类型和年代)以及关于用户的人口统计学数据(年龄、邮编、性别和职业等)。基于机器学习算法的推荐系统一般都会对此类数据感兴趣。虽然我不会在本书中详细介绍机器学习技术,但我会告诉你如何对这种数据进行切片切块以满足实际需求。

    MovieLens 1M数据集含有来自6000名用户对4000部电影的100万条评分数据。它分为三个表:评分、用户信息和电影信息。将该数据从zip文件中解压出来之后,可以通过pandas.read_table将各个表分别读到一个pandas DataFrame对象中:

    1. import pandas as pd
    2. # Make display smaller
    3. pd.options.display.max_rows = 10
    4. unames = ['user_id', 'gender', 'age', 'occupation', 'zip']
    5. users = pd.read_table('datasets/movielens/users.dat', sep='::',
    6. header=None, names=unames)
    7. rnames = ['user_id', 'movie_id', 'rating', 'timestamp']
    8. ratings = pd.read_table('datasets/movielens/ratings.dat', sep='::',
    9. header=None, names=rnames)
    10. mnames = ['movie_id', 'title', 'genres']
    11. movies = pd.read_table('datasets/movielens/movies.dat', sep='::',
    12. header=None, names=mnames)

    利用Python的切片语法,通过查看每个DataFrame的前几行即可验证数据加载工作是否一切顺利:

    1. In [69]: users[:5]
    2. Out[69]:
    3. user_id gender age occupation zip
    4. 0 1 F 1 10 48067
    5. 1 2 M 56 16 70072
    6. 2 3 M 25 15 55117
    7. 3 4 M 45 7 02460
    8. 4 5 M 25 20 55455
    9. In [70]: ratings[:5]
    10. Out[70]:
    11. user_id movie_id rating timestamp
    12. 0 1 1193 5 978300760
    13. 1 1 661 3 978302109
    14. 2 1 914 3 978301968
    15. 3 1 3408 4 978300275
    16. 4 1 2355 5 978824291
    17. In [71]: movies[:5]
    18. Out[71]:
    19. movie_id title genres
    20. 0 1 Toy Story (1995) Animation|Children's|Comedy
    21. 1 2 Jumanji (1995) Adventure|Children's|Fantasy
    22. 2 3 Grumpier Old Men (1995) Comedy|Romance
    23. 3 4 Waiting to Exhale (1995) Comedy|Drama
    24. 4 5 Father of the Bride Part II (1995) Comedy
    25. In [72]: ratings
    26. Out[72]:
    27. user_id movie_id rating timestamp
    28. 0 1 1193 5 978300760
    29. 1 1 661 3 978302109
    30. 2 1 914 3 978301968
    31. 3 1 3408 4 978300275
    32. 4 1 2355 5 978824291
    33. ... ... ... ... ...
    34. 1000204 6040 1091 1 956716541
    35. 1000205 6040 1094 5 956704887
    36. 1000206 6040 562 5 956704746
    37. 1000207 6040 1096 4 956715648
    38. 1000208 6040 1097 4 956715569
    39. [1000209 rows x 4 columns]

    注意,其中的年龄和职业是以编码形式给出的,它们的具体含义请参考该数据集的README文件。分析散布在三个表中的数据可不是一件轻松的事情。假设我们想要根据性别和年龄计算某部电影的平均得分,如果将所有数据都合并到一个表中的话问题就简单多了。我们先用pandas的merge函数将ratings跟users合并到一起,然后再将movies也合并进去。pandas会根据列名的重叠情况推断出哪些列是合并(或连接)键:

    1. In [73]: data = pd.merge(pd.merge(ratings, users), movies)
    2. In [74]: data
    3. Out[74]:
    4. user_id movie_id rating timestamp gender age occupation zip \
    5. 0 1 1193 5 978300760 F 1 10 48067
    6. 1 2 1193 5 978298413 M 56 16 70072
    7. 2 12 1193 4 978220179 M 25 12 32793
    8. 3 15 1193 4 978199279 M 25 7 22903
    9. 4 17 1193 5 978158471 M 50 1 95350
    10. ... ... ... ... ... ... ... ... ...
    11. 1000204 5949 2198 5 958846401 M 18 17 47901
    12. 1000205 5675 2703 3 976029116 M 35 14 30030
    13. 1000206 5780 2845 1 958153068 M 18 17 92886
    14. 1000207 5851 3607 5 957756608 F 18 20 55410
    15. 1000208 5938 2909 4 957273353 M 25 1 35401
    16. title genres
    17. 0 One Flew Over the Cuckoo's Nest (1975) Drama
    18. 1 One Flew Over the Cuckoo's Nest (1975) Drama
    19. 2 One Flew Over the Cuckoo's Nest (1975) Drama
    20. 3 One Flew Over the Cuckoo's Nest (1975) Drama
    21. 4 One Flew Over the Cuckoo's Nest (1975) Drama
    22. ... ... ...
    23. 1000204 Modulations (1998) Documentary
    24. 1000205 Broken Vessels (1998) Drama
    25. 1000206 White Boys (1999) Drama
    26. 1000207 One Little Indian (1973) Comedy|Drama|Western
    27. 1000208 Five Wives, Three Secretaries and Me (1998) Documentary
    28. [1000209 rows x 10 columns]
    29. In [75]: data.iloc[0]
    30. Out[75]:
    31. user_id 1
    32. movie_id 1193
    33. rating 5
    34. timestamp 978300760
    35. gender F
    36. age 1
    37. occupation 10
    38. zip 48067
    39. title One Flew Over the Cuckoo's Nest (1975)
    40. genres Drama
    41. Name: 0, dtype: object

    为了按性别计算每部电影的平均得分,我们可以使用pivot_table方法:

    1. In [76]: mean_ratings = data.pivot_table('rating', index='title',
    2. ....: columns='gender', aggfunc='mean')
    3. In [77]: mean_ratings[:5]
    4. Out[77]:
    5. gender F M
    6. title
    7. $1,000,000 Duck (1971) 3.375000 2.761905
    8. 'Night Mother (1986) 3.388889 3.352941
    9. 'Til There Was You (1997) 2.675676 2.733333
    10. 'burbs, The (1989) 2.793478 2.962085
    11. ...And Justice for All (1979) 3.828571 3.689024

    该操作产生了另一个DataFrame,其内容为电影平均得分,行标为电影名称(索引),列标为性别。现在,我打算过滤掉评分数据不够250条的电影(随便选的一个数字)。为了达到这个目的,我先对title进行分组,然后利用size()得到一个含有各电影分组大小的Series对象:

    1. In [78]: ratings_by_title = data.groupby('title').size()
    2. In [79]: ratings_by_title[:10]
    3. Out[79]:
    4. title
    5. $1,000,000 Duck (1971) 37
    6. 'Night Mother (1986) 70
    7. 'Til There Was You (1997) 52
    8. 'burbs, The (1989) 303
    9. ...And Justice for All (1979) 199
    10. 1-900 (1994) 2
    11. 10 Things I Hate About You (1999) 700
    12. 101 Dalmatians (1961) 565
    13. 101 Dalmatians (1996) 364
    14. 12 Angry Men (1957) 616
    15. dtype: int64
    16. In [80]: active_titles = ratings_by_title.index[ratings_by_title >= 250]
    17. In [81]: active_titles
    18. Out[81]:
    19. Index([''burbs, The (1989)', '10 Things I Hate About You (1999)',
    20. '101 Dalmatians (1961)', '101 Dalmatians (1996)', '12 Angry Men (1957)',
    21. '13th Warrior, The (1999)', '2 Days in the Valley (1996)',
    22. '20,000 Leagues Under the Sea (1954)', '2001: A Space Odyssey (1968)',
    23. '2010 (1984)',
    24. ...
    25. 'X-Men (2000)', 'Year of Living Dangerously (1982)',
    26. 'Yellow Submarine (1968)', 'You've Got Mail (1998)',
    27. 'Young Frankenstein (1974)', 'Young Guns (1988)',
    28. 'Young Guns II (1990)', 'Young Sherlock Holmes (1985)',
    29. 'Zero Effect (1998)', 'eXistenZ (1999)'],
    30. dtype='object', name='title', length=1216)

    标题索引中含有评分数据大于250条的电影名称,然后我们就可以据此从前面的mean_ratings中选取所需的行了:

    1. # Select rows on the index
    2. In [82]: mean_ratings = mean_ratings.loc[active_titles]
    3. In [83]: mean_ratings
    4. Out[83]:
    5. gender F M
    6. title
    7. 'burbs, The (1989) 2.793478 2.962085
    8. 10 Things I Hate About You (1999) 3.646552 3.311966
    9. 101 Dalmatians (1961) 3.791444 3.500000
    10. 101 Dalmatians (1996) 3.240000 2.911215
    11. 12 Angry Men (1957) 4.184397 4.328421
    12. ... ... ...
    13. Young Guns (1988) 3.371795 3.425620
    14. Young Guns II (1990) 2.934783 2.904025
    15. Young Sherlock Holmes (1985) 3.514706 3.363344
    16. Zero Effect (1998) 3.864407 3.723140
    17. eXistenZ (1999) 3.098592 3.289086
    18. [1216 rows x 2 columns]

    为了了解女性观众最喜欢的电影,我们可以对F列降序排列:

    1. In [85]: top_female_ratings = mean_ratings.sort_values(by='F', ascending=False)
    2. In [86]: top_female_ratings[:10]
    3. Out[86]:
    4. gender F M
    5. title
    6. Close Shave, A (1995) 4.644444 4.473795
    7. Wrong Trousers, The (1993) 4.588235 4.478261
    8. Sunset Blvd. (a.k.a. Sunset Boulevard) (1950) 4.572650 4.464589
    9. Wallace & Gromit: The Best of Aardman Animation... 4.563107 4.385075
    10. Schindler's List (1993) 4.562602 4.491415
    11. Shawshank Redemption, The (1994) 4.539075 4.560625
    12. Grand Day Out, A (1992) 4.537879 4.293255
    13. To Kill a Mockingbird (1962) 4.536667 4.372611
    14. Creature Comforts (1990) 4.513889 4.272277
    15. Usual Suspects, The (1995) 4.513317 4.518248

    计算评分分歧

    假设我们想要找出男性和女性观众分歧最大的电影。一个办法是给mean_ratings加上一个用于存放平均得分之差的列,并对其进行排序:

    1. In [87]: mean_ratings['diff'] = mean_ratings['M'] - mean_ratings['F']

    按”diff”排序即可得到分歧最大且女性观众更喜欢的电影:

    1. In [88]: sorted_by_diff = mean_ratings.sort_values(by='diff')
    2. In [89]: sorted_by_diff[:10]
    3. Out[89]:
    4. gender F M diff
    5. title
    6. Dirty Dancing (1987) 3.790378 2.959596 -0.830782
    7. Jumpin' Jack Flash (1986) 3.254717 2.578358 -0.676359
    8. Grease (1978) 3.975265 3.367041 -0.608224
    9. Little Women (1994) 3.870588 3.321739 -0.548849
    10. Steel Magnolias (1989) 3.901734 3.365957 -0.535777
    11. Anastasia (1997) 3.800000 3.281609 -0.518391
    12. Rocky Horror Picture Show, The (1975) 3.673016 3.160131 -0.512885
    13. Color Purple, The (1985) 4.158192 3.659341 -0.498851
    14. Age of Innocence, The (1993) 3.827068 3.339506 -0.487561
    15. Free Willy (1993) 2.921348 2.438776 -0.482573

    对排序结果反序并取出前10行,得到的则是男性观众更喜欢的电影:

    1. # Reverse order of rows, take first 10 rows
    2. In [90]: sorted_by_diff[::-1][:10]
    3. Out[90]:
    4. gender F M diff
    5. title
    6. Good, The Bad and The Ugly, The (1966) 3.494949 4.221300 0.726351
    7. Kentucky Fried Movie, The (1977) 2.878788 3.555147 0.676359
    8. Dumb & Dumber (1994) 2.697987 3.336595 0.638608
    9. Longest Day, The (1962) 3.411765 4.031447 0.619682
    10. Cable Guy, The (1996) 2.250000 2.863787 0.613787
    11. Evil Dead II (Dead By Dawn) (1987) 3.297297 3.909283 0.611985
    12. Hidden, The (1987) 3.137931 3.745098 0.607167
    13. Rocky III (1982) 2.361702 2.943503 0.581801
    14. Caddyshack (1980) 3.396135 3.969737 0.573602
    15. For a Few Dollars More (1965) 3.409091 3.953795 0.544704

    如果只是想要找出分歧最大的电影(不考虑性别因素),则可以计算得分数据的方差或标准差:

    1. # Standard deviation of rating grouped by title
    2. In [91]: rating_std_by_title = data.groupby('title')['rating'].std()
    3. # Filter down to active_titles
    4. In [92]: rating_std_by_title = rating_std_by_title.loc[active_titles]
    5. # Order Series by value in descending order
    6. In [93]: rating_std_by_title.sort_values(ascending=False)[:10]
    7. Out[93]:
    8. title
    9. Dumb & Dumber (1994) 1.321333
    10. Blair Witch Project, The (1999) 1.316368
    11. Natural Born Killers (1994) 1.307198
    12. Tank Girl (1995) 1.277695
    13. Rocky Horror Picture Show, The (1975) 1.260177
    14. Eyes Wide Shut (1999) 1.259624
    15. Evita (1996) 1.253631
    16. Billy Madison (1995) 1.249970
    17. Fear and Loathing in Las Vegas (1998) 1.246408
    18. Bicentennial Man (1999) 1.245533
    19. Name: rating, dtype: float64

    可能你已经注意到了,电影分类是以竖线(|)分隔的字符串形式给出的。如果想对电影分类进行分析的话,就需要先将其转换成更有用的形式才行。