Assignment 5¶
In this assignment, you'll scrape text from The California Aggie and then analyze the text.
The Aggie is organized by category into article lists. For example, there's a Campus News list, Arts & Culture list, and Sports list. Notice that each list has multiple pages, with a maximum of 15 articles per page.
The goal of exercises 1.1 - 1.3 is to scrape articles from the Aggie for analysis in exercise 1.4.
import numpy as np
import nltk
from nltk import corpus
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import NearestNeighbors
from matplotlib import pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
import requests
import requests_cache
import requests_ftp
from urllib2 import Request, urlopen
import json
Exercise 1.1. Write a function that extracts all of the links to articles in an Aggie article list. The function should:
Have a parameter
urlfor the URL of the article list.Have a parameter
pagefor the number of pages to fetch links from. The default should be1.Return a list of aricle URLs (each URL should be a string).
Test your function on 2-3 different categories to make sure it works.
Hints:
Be polite to The Aggie and save time by setting up requests_cache before you write your function.
Start by getting your function to work for just 1 page. Once that works, have your function call itself to get additional pages.
You can use lxml.html or BeautifulSoup to scrape HTML. Choose one and use it throughout the entire assignment.
requests_cache.install_cache('cache')
from bs4 import BeautifulSoup
import lxml.html as lx
# Scrapes a page to get all article URLs
def get_links(url, page):
urllist = []
for i in range(1, page + 1):
num = str(i)
urllist.append(url + 'page/' + num)
for i in range(1, len(urllist) + 1):
request = requests.get(urllist[i - 1])
html = request.text
text = BeautifulSoup(html, 'html.parser')
sets = text.find_all(name='h2',attrs={'class':'entry-title'})
links = []
for j in range(0, len(sets)):
links.append(sets[j].find_all('a'))
if i == 1:
linklist = []
else:
linklist = linklist
for j in range(0, len(links)):
linklist.append(links[j][0]['href'])
return linklist
get_links('https://theaggie.org/campus/', 2)
Exercise 1.2. Write a function that extracts the title, text, and author of an Aggie article. The function should:
Have a parameter
urlfor the URL of the article.For the author, extract the "Written By" line that appears at the end of most articles. You don't have to extract the author's name from this line.
Return a dictionary with keys "url", "title", "text", and "author". The values for these should be the article url, title, text, and author, respectively.
For example, for this article your function should return something similar to this:
{
'author': u'Written By: Bianca Antunez \xa0\u2014\xa0city@theaggie.org',
'text': u'Davis residents create financial model to make city\'s financial state more transparent To increase transparency between the city\'s financial situation and the community, three residents created a model called Project Toto which aims to improve how the city communicates its finances in an easily accessible design. Jeff Miller and Matt Williams, who are members of Davis\' Finance and Budget Commission, joined together with Davis entrepreneur Bob Fung to create the model plan to bring the project to the Finance and Budget Commission in February, according to Kelly Stachowicz, assistant city manager. "City staff appreciate the efforts that have gone into this, and the interest in trying to look at the city\'s potential financial position over the long term," Stachowicz said in an email interview. "We all have a shared goal to plan for a sound fiscal future with few surprises. We believe the Project Toto effort will mesh well with our other efforts as we build the budget for the next fiscal year and beyond." Project Toto complements the city\'s effort to amplify the transparency of city decisions to community members. The aim is to increase the understanding about the city\'s financial situation and make the information more accessible and easier to understand. The project is mostly a tool for public education, but can also make predictions about potential decisions regarding the city\'s financial future. Once completed, the program will allow residents to manipulate variables to see their eventual consequences, such as tax increases or extensions and proposed developments "This really isn\'t a budget, it is a forecast to see the intervention of these decisions," Williams said in an interview with The Davis Enterprise. "What happens if we extend the sales tax? What does it do given the other numbers that are in?" Project Toto enables users, whether it be a curious Davis resident, a concerned community member or a city leader, with the ability to project city finances with differing variables. The online program consists of the 400-page city budget for the 2016-2017 fiscal year, the previous budget, staff reports and consultant analyses. All of the documents are cited and accessible to the public within Project Toto. "It\'s a model that very easily lends itself to visual representation," Mayor Robb Davis said. "You can see the impacts of decisions the council makes on the fiscal health of the city." Complementary to this program, there is also a more advanced version of the model with more in-depth analyses of the city\'s finances. However, for an easy-to-understand, simplistic overview, Project Toto should be enough to help residents comprehend Davis finances. There is still more to do on the project, but its creators are hard at work trying to finalize it before the 2017-2018 fiscal year budget. "It\'s something I have been very much supportive of," Davis said. "Transparency is not just something that I have been supportive of but something we have stated as a city council objective [ ] this fits very well with our attempt to inform the public of our challenges with our fiscal situation." ',
'title': 'Project Toto aims to address questions regarding city finances',
'url': 'https://theaggie.org/2017/02/14/project-toto-aims-to-address-questions-regarding-city-finances/'
}Hints:
The author line is always the last line of the last paragraph.
Python 2 displays some Unicode characters as
\uXXXX. For instance,\u201cis a left-facing quotation mark. You can convert most of these to ASCII characters with the method call (on a string).translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })If you're curious about these characters, you can look them up on this page, or read more about what Unicode is.
# I think it's better if I separate this into 4 functions
# This one gets the text of the function from strong and span attrs
def get_content(url):
request = requests.get(url)
html = request.text
text = BeautifulSoup(html, 'html.parser')
strong = text.find_all(name = 'strong')
try:
strong = strong[0].text.strip()
strong = strong.translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })
except IndexError:
strong = ''
sets = text.find_all(name='span',attrs={'style':'font-weight: 400;'})
content = []
for i in range(0, len(sets)):
content.append(sets[i].text)
content = "".join(content)
content = content.translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })
body = strong + ' ' + content
return body
# Gets author, I think the names are always on the third to last p observation
def get_author(url):
request = requests.get(url)
html = request.text
text = BeautifulSoup(html, 'html.parser')
author = text.find_all(name='p')
try:
author = author[len(author) - 2].text.strip()
except IndexError:
author = "Unknown"
return author
# Gets title and then strips the part that says 'The Aggie'
def get_title(url):
request = requests.get(url)
html = request.text
text = BeautifulSoup(html, 'html.parser')
title = text.find_all(name='title')
title = title[0].text.strip()
title = title.translate({ 0x2018:0x27, 0x2019:0x27, 0x201C:0x22, 0x201D:0x22, 0x2026:0x20 })
title = title.rstrip(' | The Aggie')
return title
# Extracts title, text, url, and author by finding corresponding places, and appending them to one observation
def extract(url):
author = get_author(url)
body = get_content(url)
title = get_title(url)
result = 'author: ' + author + '\n' + 'text: ' + body + '\n' + 'title: ' + title + '\n' + 'url: ' + url
return result
extract('https://theaggie.org/2017/02/14/project-toto-aims-to-address-questions-regarding-city-finances/')
Exercise 1.3. Use your functions from exercises 1.1 and 1.2 to get a data frame of 60 Campus News articles and a data frame of 60 City News articles. Add a column to each that indicates the category, then combine them into one big data frame.
The "text" column of this data frame will be your corpus for natural language processing in exercise 1.4.
# get all 120 links
campus = get_links('https://theaggie.org/campus/', 4)
city = get_links('https://theaggie.org/city/', 4)
links = campus + city
# get column of type of article
camplist = ['campus'] * 60
citylist = ['city'] * 60
types = camplist + citylist
# gets author, title, and content
authorlist = []
bodylist = []
titlelist = []
for i in range(0, len(links)):
bodylist.append(get_content(links[i]))
titlelist.append(get_title(links[i]))
authorlist.append(get_author(links[i]))
import pandas as pd
articles = pd.DataFrame({'type':types, 'title':titlelist, 'author':authorlist, 'content':bodylist, 'url':links})
articles.head()
Exercise 1.4. Use the Aggie corpus to answer the following questions. Use plots to support your analysis.
What topics does the Aggie cover the most? Do city articles typically cover different topics than campus articles?
What are the titles of the top 3 pairs of most similar articles? Examine each pair of articles. What words do they have in common?
Do you think this corpus is representative of the Aggie? Why or why not? What kinds of inference can this corpus support? Explain your reasoning.
Hints:
The nltk book and scikit-learn documentation may be helpful here.
You can determine whether city articles are "near" campus articles from the similarity matrix or with k-nearest neighbors.
If you want, you can use the wordcloud package to plot a word cloud. To install the package, run
conda install -c https://conda.anaconda.org/amueller wordcloudin a terminal. Word clouds look nice and are easy to read, but are less precise than bar plots.
stemmer = nltk.stem.porter.PorterStemmer()
tokenize = nltk.word_tokenize
# stem and lemma functions
def stem(tokens,stemmer = PorterStemmer().stem):
return [stemmer(w.lower()) for w in tokens]
def lemmatize(text):
return stem(tokenize(text))
def dictionaries(df):
textd = {}
for n in range (0, len(articles.index)):
s = set(lemmatize(articles['content'][n]))
try:
toks = toks | s
except NameError:
toks = s
for tok in s:
try:
textd[tok].append(n)
except KeyError:
textd[tok] = [n]
# ids for each article
artids = {}
N = len(articles.index)
for i in xrange(N):
artids[articles['content'][i]] = i
# word ids
tokids = {}
tok_list = list(toks)
m = len(tok_list)
for j in xrange(m):
tokids[tok_list[j]] = j
return textd, artids, tokids
textd, artids, tokids = dictionaries(articles)
# smoothed idf
numd = {key:len(set(val)) for key,val in textd.items()}
logN = np.log(len(articles.index))
idf_smooth = {key:logN - np.log(1 + val) for key, val in numd.items() if val > 1}
plt.hist(idf_smooth.values(),bins=20)
# appends indexes with highest values
lowval = []
for i in range(0, len(idf_smooth.values())):
if idf_smooth.values()[i] < 0.5:
lowval.append(i)
for i in range(0, len(lowval)):
print(list(tokids.keys())[list(tokids.values()).index(lowval[i])])
I found the words that have the lowest idf values because by the definition of inverse document frequency, I believe that those are the words with the highest importance in the collection of articles. Of those words, it seems like they cover a lot of issues having to do with the university, competitions, and possibly health and crime.
# splits dfs
campusdf = articles.loc[articles['type'] == "campus"]
citydf = articles.loc[articles['type'] == "city"]
textd, artids, tokids = dictionaries(campusdf)
numd = {key:len(set(val)) for key,val in textd.items()}
logN = np.log(len(articles.index))
idf_smooth = {key:logN - np.log(1 + val) for key, val in numd.items() if val > 1}
plt.hist(idf_smooth.values(),bins=20)
lowval = []
for i in range(0, len(idf_smooth.values())):
if idf_smooth.values()[i] < 0.5:
lowval.append(i)
for i in range(0, len(lowval)):
print(list(tokids.keys())[list(tokids.values()).index(lowval[i])])
textd, artids, tokids = dictionaries(citydf)
numd = {key:len(set(val)) for key,val in textd.items()}
logN = np.log(len(articles.index))
idf_smooth = {key:logN - np.log(1 + val) for key, val in numd.items() if val > 1}
plt.hist(idf_smooth.values(),bins=20)
lowval = []
for i in range(0, len(idf_smooth.values())):
if idf_smooth.values()[i] < 0.5:
lowval.append(i)
for i in range(0, len(lowval)):
print(list(tokids.keys())[list(tokids.values()).index(lowval[i])])
I split the dataframes based on the types of articles and found the inverse document frequency words for both. They both have similar words so the topics they cover may be similar but I think it's very hard to tell because most of the words that differ from the two doesn't seem relevant to any particular topic.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(tokenizer=lemmatize, stop_words = "english", smooth_idf = True, norm = None)
tfs = vectorizer.fit_transform(articles['content'])
sim = tfs.dot(tfs.T)
campuslabel = [1 for i in range (0,60)] + [0 for i in range (0,60)]
citylabel = [0 for i in range (0,60)] + [1 for i in range (0,60)]
# knn
k_max = 20
nbrs = NearestNeighbors(n_neighbors=k_max).fit(tfs)
err_list = []
for k in xrange(1,k_max+1,2):
neighmat = nbrs.kneighbors_graph(n_neighbors=k)
pred_lab = (neighmat.dot(campuslabel) > k/2.)*1 pred
err = np.mean(pred_lab != campuslabel)
err_list.append(err)
plt.plot(range(1,k_max + 1,2),err_list)
plt.xlabel('k')
plt.ylabel('Error Rate')
values = []
for i in range(0, len(pred_lab)):
if pred_lab[i] == 1:
values.append(i)
values
# Find similar article titles
# Title list correspond with observations
for i in range(0, len(values)):
print titlelist[values[i]]
The most similar articles are 2017 ASUCD Winter Elections - Meet the Candidates articles, Former chanecellor turns down feminist leadership role in UC Davis, UC-Wide walkout, teachins on Trump's inauguration day, Student regent recruitment for the 2018-2019 school year begins, and Nov. 8 2016: An Election Day many may never forget.
I think that some parts of the Aggie may represent the corpus I got because I think that the Aggie should cover issues that are school related such as health and crime which benefits and alerts students and election and school official topics that informs students of what goes on in their school. The corpus is implying that what the Aggie covers is mostly about election and school official issues along with other health, crime, and safety issues which benefits students. A lot of the words I see are used in a lot of everyday articles so this may require further analysis of the corpus.