Big Data: Lecture 1

 

This is the first lecture for graduate school.

kaggle.com is a good competition website. There is also a link to the newest machine learning code, allows to see what people are publishing, and using the newest github repositories in the experiments. There are also lists of datasets, and interviews with Kaggle masters, and the quality of the submissions depends a lot of the number of experiments you perform. Understanding your data and performing many diverse experiements are extremely important. Take a look of it, and run some of the notebooks and create some quetions about them. All the videos are also posted. You can add stuff from your drive using Colab. 

There are short quizzes in every class, as well as presentations of 2 kinds: project progress and individual presentations. In class assignments, Homework assignments with lecture, and conclude with a discussion. 

We are covering some data cleansing, and visualization and numerical data. Then train neural networks, learn about probability, statistics, etc, and talk about algorithms and the cloud, and the roles of data scientists, etc. An example of model is to train a model to predict the next volcano eruption. And then we talk about the role of data scientists and where data science jobs will be and how to best prepare for them, using books, blogs and github repositories. Then social networks, etc. We want to talk about the organization strategies and learn how to effectively communicate. The topics of vector spaces, linear and logistic regression, neural networks, dimensionality reduction, sentiment analysis, among more and hadoop/cloud computing, etc. Maybe we run some projects on Hadoop since we will have some relatively big data sets. We will talk about business, real-time analytics, and organizational strategies. Homework is 12%, in-class is 15%, midterm is 30% (15% each), and Group projects is 23% and final is 20%. Appear in every class, only once in a while excused but the expectation will be in every class, then the lowest quiz is dropped. All quizzes will be open book but have limited time. 

Read the covid test accuracy and participate in a discussion assignment and post answers to the question. Post on the discussion. The article here is to evaluate the chance of a positive and negative test based on COVID. 10 minutes to read an article and 5 minutes to type an answer, then continue discussion. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013705/full

Given a positive test, according to the article, the chances that I have COVID-19 are 90% since there are 10 false positive in 105 tests. Given a negative test, the chances that I have COVID-19 are 1% since there were 5 false negatives out of 895 negative results. Of course, the numbers could be greater or lesser due to bias, and the fact that this was the rapid molecular assays test and little information was provided on false results on the antigen and RT-PCR(control) tests. That's my thinking. Here are other ideas:

1. 

According to the article, If 1000 people were tested and only 100 people had the COVID-19, then:

105 people would test positive and out of that 10 people would result in false positive.

895 people would test negative and out of that 5 people would result in false negative.

Based on the above details provided in the article, a person has roughly 89% chance of testing negative and 11% chance of testing positive for the COVID-19.

2. For this answer, I'll just focus on Rapid Testing and its sensitivities/specificities. 1. Given a positive test, I would like to focus on the sensitivity percentage (95.2%). This states that given a positive test, I would have a 95.2 percent of ACTUALLY having COVID. 2. Since the specificity for the test is 98.9%, the chances that I am positive would then be 1.1% (100-98.9).

3. 

1. There is a 10% chance that the person given +ve may not be infected with COVID-19.

-> Out of 105 people given +ve, 10 (~10%) of them were not infected with COVID-19. This can be inferred from the false positive results shown in the section - main results.

2. There is a 1% chance that a person given -ve might be actually a victim of COVID-19.

-> Out of 895 people given -ve, 5 people (~1%) are having COVID-19. This can be inferred from the false negative results shown in the section - main results.

Even in decision tree, you go from the root node to the parent node, and the accuracy is being analyzed. 

We used Baye's Theorem to solve this problem. We want to figure out how the prevalence that a person who tests positive really is a user?

P(U|P) = P(P|U) P(U) / P(P) where P(P) = P(P|NU) + P(P|U) 

Very often our intuitions are wrong and we often have to think hard about the math that constitutes this problem. 

Machines don't have abstraction probabilities, so here's what we can do: 

We have flaws in weak external validity, but humans always face a decision with limited information. We always have to make decisions based on limited data and neural networks allow us to create a model better than intuition, but we have to make a decision based on imperfect information. 

https://www.statnews.com/2020/04/24/coronavirus-hospitals-use-ai-to-predict-patient-decline-before-knowing-it-works/

Go to wikipedia information have information and how it relates/correlates to COVID testing information. Go online for Python and R programming as well, R is good for visualizing data, but Python is better for building models.

Kaggle also has many notebooks for these datasets. Most of the cases we just experiment with many different models. Get to know the normal distributions and central limit theorems. The class is a beginner class, and there will be different distribution of skills in groups and randomness. Maybe we can somewhat specialize in the group so we can perform experiments faster. 

Wait.. so do we need to answer the question in the discussion per group, or should we look for ideas in machine learning, or both? We need to work on Google Colab, and run the notebook. Click and run linear regression and allow Google to access your drive and make sure the path is written correctly.

Baye's theorem is one of the most important tools in data science, and will be widely used when analyzing data and probabilities of events. 

The first concept that we need is the concept of conditional probability. P(A | B) = P(A∩B)/P(B). The conditional probability of even A given B is the probability of A if B is known to occur. From the definition of conditional probability, we will derive Baye's theorem, and it's used everywhere and its amazing that we get so much out of so little. 

This is known as Baye's Theorem. Why care though? Let's look at some applications.

1. A drug test will produce 99% true positive results for drug test users and 99% true negative results for non-drug users. Suppose 0.5% of people are users of the drug. If a randomly selected individual tests positive, what is the probability that he or she is a user? 

If 1000 individuals are tested we can expect to have 995 non-users and 5 users. So there should be false and true positives. So we want to figure out the probability that someone actually uses the drug give that he is tested positive. 

Now we can use this to solve the COVID test problem. 

What makes the cases? Pick your birthday as the % of drug users for the false positives and false negatives, respectively, and use Baye's Theorem to solve this. (8% false positive, 10% false negative). We need to find a reliable article on the accuracy of drug testing. The second homework is to look at the COVID test accuracy and reanswer the problems. What is the chance you have COVID, given u tested positively for COVID, and then do the same thing for the negative tests. This theorem is used a LOT in data mining. 

Here's my response: 

Upon Baye's Theorem, We want to find out the chances that COVID appears given a positive test. Use Baye's Theorem to determine that 

P(COVID|positive) = P(positive|COVID) * P(COVID) / P(positive).

If there are 10 false positives out of 105 test and 5 false negatives, then there are 95 + 5 = 100 positive results in 1,000 tests, which means that the probability of having COVID is 10%, and the probability of testing positive is 105/1000 = 10.5%. 

The probability of testing positive given that a user has COVID is 95% (5 false negatives, 95 true positive tests). 

0.95 * (0.1)/0.105 = 0.90476 or 90.476%.

Finally we use Bayes Theorem and calculate the rest of the calculation. 

P(COVID|negative) = P(negative|COVID) * P(COVID) / P(negative) 

P(negative | COVID) is 1 - P(positive|COVID) which is equal to 5%. 

P(negative) has 895 negative results which is 89.5%.

P(COVID) is still 10%.

0.05 * 0.1 / (0.895) = 0.00558659 or 0.558659%. 

So the probability of having COVID-19 given a negative test is 0.558659%.