Linear Regression Homework

 Now the teacher wants to discuss a few words about the homework. The task will be to:

1. Execute the word cloud, linear regression, and the social networks. 

We will be running a word cloud program without knowing what it does

numpy is a counting program for Python and it deals with counting, matrices, high dimension matrices. Finally, we import WordCloud which depends on text. Matplotlib is a graphic program that we use quite often and then we finally get the Wordcloud, which depends on the previous 2. We have text as a variable, then specify parameters of display and specify the word. Another thing we can do is have a program that calls nltk which teaches how to do basic natural language processing. It comes with some datasets and is preinstalled with Google Colab, whcih can be an extension of this exercise. For example, if an article explains about vest, uniform, New York, men, and shirt, obviously, the article is about clothing.

Here's the first part of the code: 

import numpy as np

import matplotlib.pyplot as plt

from wordcloud import WordCloud

text = "square, repeat, true, true, true, true, true"

x, y = np.ogrid[:300, :300]

mask = (x - 150) ** 2 + (y - 150) ** 2 > 130 ** 2

print(mask)

mask = 255 * mask.astype(int)

print(mask)

wc = WorcCloud(background_color = "red", mask = mask)

wc.generate(text)

plt.axis(off)

plt.imshow(wc, interpolation = "bilinear")

plt.show()

import nltk

from nltk import corpus

nltk.download('gutenberg')

nltk.download('genesis')

nltk.download('inaugural')

nltk.download('nps_chat')

nltk.download('webtext')

nltk.download('treebank')

from ntlk.book import FreqDist

from nltk.book import (text1, text2, text3, text4, text8)

#text1.tokens

#text=FreqDist(text1)

#text=text1.tokens

text=""" THE FLEECE ZIP-UP VEST, the capstone of a new corporate uniform, lurks in air-conditioned corporate cubicles across America. It covers the sweating backs of nervous interns ordering supersize coffees at Starbucks . It’s worn by silver-haired executives in the elevator, heading up to their corner suites. It appears in myriad shades of gray and blue, on men of all shapes and sizes who earn all kinds of salaries. It has become as ubiquitous as the take-out salad in humdrum workplaces, and is slowly supplanting the suit and tie as essential office wear.Typically, the vest is worn over a button-up shirt and paired with chinos and brown dress shoes of any flavor. “The uniform” is how this ensemble has been branded around the office of one 36-year-old working in capital markets in Pittsburgh, a past practitioner of the much-mocked look who asked that his name be withheld. During a recent trip to New York City this month, he observed scores of men wearing gray fleece vests even as temperatures touched the mid-90s.The trend is so pervasive that an Instagram account with nearly 40,000 followers, The Midtown Uniform (@midtownuniform), has sprung up to savagely document these corporate clones in cities like New York, Toronto and Washington D.C. The anonymous account adds pithy captions to crowdsourced photos, riffing on the omnipresence of this particular outfit. “Money isn’t really yours unless you’re fully vested,” read a caption on a recent post showing two men in matching pink shirts and blue vests. Despite the implicit ridicule, the comment sections on these photos are littered with friends tagging each other and saying things like, “Bro, this is so you.”Will Crowley, a 25-year-old investment banker who lives in Hoboken, N.J., is well aware of The Midtown Uniform account, yet it hasn’t inspired him to give up the ubiquitous outfit. “There are times where I can’t just wear a shirt. I kind of have to wear a vest,” said Mr. Crowley. He owns six vests, which he calls a “staple of people who work in finance,” mostly from Patagonia or Vineyard Vines. He rotates through the vests during the week, unless he’s meeting with a client, in which case he wears a suit.The midtown uniform appears to have taken hold post-2008, when many financial firms loosened their once-strict suit-and-tie dress code. The message was: We know your salary is down, but at least you get to dress casual on Friday! “The payouts regressed, so just like every industry that has payment difficulties, they find other ways to satisfy employees and dress is one of the easiest ones,” said a 35-year-old stock trader in New York City, who also asked to be anonymous. He was on the floor during the 2008 recession, and described how the sport coats and wool slacks gave way to vests and cotton chinos in its aftermath.Though midtown New York has now become especially associated with this new dress code, the vest’s roots lie in Silicon Valley. “If you go to the Whole Foods here you’re going to see [the vest] everywhere,” said Christina Mongini, the costume designer for HBO’s parodic sitcom “Silicon Valley” and a Bay Area native. Jared Dunn, the show’s type-A COO, wears a fleece vest over a button-down in nearly every scene in which his character appears. Said Ms. Mongini, “Jared’s style is really perceived as the normal basic understated business-casual attire.”Outdoorsy fleece vests matched the youthful, countercultural Silicon Valley spirit in a way suits and ties never did. Bay Area C-suite executives such as Apple’s Tim Cook, Facebook’s Mark Zuckerberg and PayPal cofounder Max Levchin have boasted of cycling or hiking during the work week. As Ms. Mongini said, “You can hop on your bike and throw your vest on and head over to Santa Cruz on your lunch break if you wanted to.”Across the nation, the vest was an easy sell. “It’s very difficult to just sit there and work in a suit jacket,” said the 35-year-old stock trader. In a vest, “I can sit at my desk and feel a little bit more comfortable.” Aiding and abetting the trend toward sleevelessness, a couple of years ago brokerage houses and trading platforms shrewdly started giving away the vests as a freebie to entice traders. The vests’ low cost was a way around financial regulations (which cap gifts to traders at $100) and the wearable promos were more functional than the giveaways they replaced such as candy tins and Nerf Footballs. These promotional vests, with “Equifax” or ”Merrill Lynch” embroidered along the chest, are now a common sight in New York.The trend has become self-perpetuating: People wear the vest because it’s what people wear. “Now it’s the new thing: It’s not suspenders and a bengal-striped shirt,” said Mr. Crowley. “It’s a Patagonia vest and a button-down shirt.” He added that the “bro-culture” of finance has helped reinforce this look, with its scores of men following the same path from prep school to the Ivy League to a job in finance. Looking like your peers is part of the package. Said Mr. Crowley, “If you want to be successful, part of it is wanting to fit in.”Though young professionals do typically want to fit in, the fleece vest can also help them tackle another insecurity. The Pittsburgher who’s mostly given up the vest part of the uniform admits that it had its uses: “As I’m aging, I would say it hides your belly.” Who says fashion isn’t functional?"""

wc = WordCloud(background_color = "white") 

wc.generate(text)

plt.axis("on")

plt.imshow(wc)

plt.show()

text = pastes US CONSTITUTION here LOL

wc = WordCloud(background_color = "white")

wc.generate(text)

plt.axis("on")

plt.imshow(wc)

plt.show()

Frequencies of these word clouds reflects the type of documents that we are dealing with. We can use that corresponding information to classify information. Classifiers use vectors to establish the similarities between 2 documents, for new vectors. 

The next thing to run is the linear regression library. We import certain opertaing systems, mount Google Drive using the file from the shared directory of data and then allow Google to mount Google Drive into the research Colab script. We also use pandas, which is a library for data analysis, which puts different types of data into data frames, working more or so like spreadsheets. 

There was a drive mount error, so we need to force the remount of a drive. This solved the problem, and %ls shows the drive and it helps a lot. .polyfit has 11 parameters. The question is why do we have a big array?

We use a program called pandas to deal with the numerical data, we are computing the parameters, and we have to remember that the linear regression computes the line, or how much the price depends. 

import os

cwd = os.getcwd()

print(cwd)

from google.colab import drive

drive.mount('\gdrive')

#mount the drives this time, if this doesn't work, perform a force mount.

drive.mount("\gdrive", force_remount = True)

Now, we want to get into the task of linear regression. 

Univariate linear regression just uses one feature in one case one variable. We will use the "area" of the house to train the given linear model here

import numpy as np #scientific computing

import pandas as pd #data analysis and dataframes

%ls

#use pandas to read the excel documents 

data = pd.read_csv('/gdrive/MyDrive/6100Shared/Data/ex1data2.txt', header=None)

The first column is the size of the house, followed by the number of bedrooms, and the price

data.head()

then we make a copy of the data

and remove one feature since we are during univariate linear regression. 

Then we can drop certain data, an attempt to attempt to remove the bedroom feature. 

The idea of linear regression is to fit a line to s et of points, given by 

f(x) = y = βx + α where α is the y intercept and β is the slope or has a more general form 

h(x) = θ0x0 + θ1x1 + …. + θnxn.

We then get α = y - βx 

and receive the correlation equation as follows. 

it's actually pretty easy to get the covariance and variance of sets. 

*explain the covariance matrix here)

#get covariance

data2.cov()

data2.var()

data2.var().Size

And here's how we find a value in the data frame:

data2.cov().iloc[1][0] 

beta_hat = (data2.cov().iloc[1][0]/data2.var().Size)

"""Computing the means"""

x_bar = data2.Size.mean()

y_bar = data2.Price.mean()

print(x_bar, ' ', y_bar)

alpha_hat = y_bar - beta_hat * x_bar

alpha_hat

#the third plot

plt.plot(data2.Size, data2.Price, 'b.')

plt.show()

plt.plot(data2.Size, data2.Price, 'b^') 

plt.show()

And here's the final burial plot field: 

plt.plot(data2.Size, data2.Price, 'b.') 

slope = beta_hat

intercept = alpha_hat

line_values = [slope * v + intercept for v in data2.Size] 

plt.plot(data2.Size, line_values, 'g-')

plt.show() 

And here's another way to represent the data. 

#find the slope and intercept of the best fit line. 

slope, intercept = np.polyfit(data.Size, data.Price, 1)

print(slope, intercept)

#incrementally test slope values for each slope value and intercept. 

line_values = [slope * v + intercept for v in data.Size]

plt.plot(data.Size, data.Price, 'b.')

plt.show()

plt.plot(data.Size, line_values, 'r')

plt.title("full data")

plt.show

#fit with multiple polynomials. 

np.polyfit(data.Size, data.Price, 11)