Final Tutorial - The Data Science Process¶
Jade Wilkins Marcellus Davenport Allen Lai
Introduction¶
The goal of this tutorial is to teach you how to use the entire data science pipeline in a real life situation. The data science pipeline consists of 5 steps: Data collection, Data processing, Exploratory analysis and data visualization,Analysis, Hypothesis testing and machine learning and lastly Insight and decision. We will use python to carry out the various techniques of predicting financial market movements, specifically stock prices.
Our first technique will be using linear regression to predict the closing prices of stocks based on closing prices of the stocks in the past. We will predict the closing price of Microsoft's stock (ticker: MSFT) based on historical financial data from May 16, 2015 to April 16, 2017.
Project Overview¶
Step 1: Data Collection¶
Our data will be retrieved from Yahoo Finance. The csv file is named as "MSFT.csv".For some background on the stock market to better understand how it actually works check out this link which explains the stock market with a cool infographic http://www.timothysykes.com/blog/stock-market-basics-tutorial-infographic/
In order to collect this data with iPython notebook. We will begin the project by importing the necessary libraries. We will use Python Data Analysis Library (Pandas) and NumPy, a fundamental package for scientific computing with Python.
Next we will load and manage data about stocks which we will use for our analysis. Then we will use the data and various libraries to perform exploratory data analysis.Last to predict the stock market we will be using Linear Regression. Linear Regression is a way of modeling a relationship between a independent and dependent variable by creating a linear equation.
import pandas as pd
import numpy as np
from sklearn import linear_model
import scipy.stats as stats
import matplotlib.pyplot as plt
from decimal import *
Step 2: Data processing¶
Download the Microsoft Corporation (MSFT) Historical Prices csv file from Yahoo Finance: https://finance.yahoo.com/quote/MSFT/history?p=MSFT. Then upload the csv file into the same folder as this file.Finally you can then load the data using panda's built in read_csv function which will read in the csv file and return a data structure we can perform analysis one.
Since we are interested in predicting the closing price we will isolate the Open and Close column into two separate arrays using numpy’s built in array feature. Luckily, our data is already tidy. We do not need to worry about how to deal with missing data
df = pd.read_csv('MSFT.csv')
size = len(df.index)
data_range = np.arange(size)
train_x = np.array(df["Open"]).reshape(size,1)
train_y = np.array(df["Close"]).reshape(size,1)
df.head()
Step 3: Exploratory Data Analysis and Data Visualization¶
What is Exploratory Data Analysis¶
Exploratory data analysis is a way of analyzing a set of data in order to gain knowledge about its characteristics, such as basic summary statistics, distribution, and obvious trends to see if there are any problems with the data. You can also use it to prepare the data for modeling as we will be doing in the case of this project.This can be accomplished by creating visualizations, getting basic descriptions of the data and identifying patterns.For a more in depth explanation of Exploratory Data Analysis check out this site which explains the basics of exploratory data analysis. http://www.itl.nist.gov/div898/handbook/eda/section1/eda11.htm
Now that our data is imported we can begin to explore your data efficiently so that we then use the information gained from our analysis in order to start modeling our data.
Now that our data is imported we can begin to explore your data efficiently so that we then use the information gained from our analysis in order to start modeling our data. A great way to start analyzing data is by finding averages such as the mean and median of the opening and closing prices of Microsoft Stock so we can learn where the majority of our data lies.Another measure of central tendency we should find is the the variance.We will find the variance of our data so we can learn the range of the data.We can find all of our central tendencies using panda's built in mean, median ,and variance function on our dataframe.
opening_price_mean = df["Open"].mean()
print("The mean of the opening price: " + str(opening_price_mean))
opening_price_median = df['Open'].median()
print("The median of the opening price: " + str(opening_price_median))
open_price_variance = pd.DataFrame.var(df['Open'])
print("The variance of the opening price: " + str(open_price_variance))
print("The max of the opening price: " + str(df.xs('Open', axis=1).max()))
print("The min of the opening price: " + str(df.xs('Open', axis=1).min()))
opening_price_mean = df["Close"].mean()
print("The mean of the closing price: " + str(opening_price_mean))
opening_price_median = df['Close'].median()
print("The median of the closing price: " + str(opening_price_median))
open_price_variance = pd.DataFrame.var(df['Close'])
print("The variance of the closing price: " + str(open_price_variance))
print("The max of the closing price: " + str(df.xs('Close', axis=1).max()))
print("The min of the closing price: " + str(df.xs('Close', axis=1).min()))
After gathering the mean, median and variance of our data we can see that there is a large variance. This indicates indicates that the data points are very spread out from the mean.In order to understand exactly how spread out our data is we should visualize the data. A simple way to do this is to create a plot using matplotlib.pyplot as plt. Since the goal of our project is to predict the closing price we will focus on plottinng the closing prices.
correlation_fig = plt.figure()
correlation_fig.suptitle('Distribution of Closing Prices Across Time', fontsize=12)
plt.plot(data_range,train_y)
plt.title('May 16, 2015 to April 16, 2017')
plt.ylabel('Closing Price', fontsize=14)
plt.xlabel('Date Range', fontsize=14)
plt.show()
Our visualization does show us that the data is very spread out.Although its hard to learn anything else about the data because there does not seem to be a correlation between Microsoft stock closing prices and time.A different type of visualization may help us to gain more information. We will next plot the normal distribution of MST closing prices across time. A normal distribution plot is useful because if the data is normally distributed we can easily predict unknown values.
y_mean = np.mean(train_y )
y_std = np.std(train_y)
correlation_fig = plt.figure()
pdf = stats.norm.pdf(train_y,y_mean, y_std)
plt.plot(train_y, pdf)
correlation_fig.suptitle('Distribution of MSFT Closing Prices Across Time', fontsize=12)
plt.xlabel('Closing Price', fontsize=11)
plt.ylabel('Distrubtion', fontsize=11)
plt.show()
The distribution of MSFT closing prices shows that the closing prices are not normally distributed. We will have to analyze our data more in order to determine how to predict closing prices.
A well-known visualization for displaying financial data is a candle stick chart. Candle Stick charts shows all known information know about the price of the stock. The green candlesticks represent data with a increasing value and the red candle sticks represent days with a decreasing value. For information on how to interpret Candle Stick Charts check out this website: http://stockcharts.com/school/doku.php?id=chart_school:chart_analysis:introduction_to_candlesticks. In order to use plot.ly api you must install plot.ly in your terminal using the line "$ sudo pip install plotly".
from plotly import __version__
import plotly.graph_objs as go
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode()
from datetime import datetime
trace = go.Candlestick(x=df.index,
open=df.Open,
high=df.High,
low=df.Low,
close=df.Close)
data = [trace]
layout = {
'title': 'Microsoft Stock Prices From May 16, 2015 to April 16, 2017 ',
'yaxis': {'title': 'Microsoft Stock'},
'shapes': [{
'x0': '2015-05-16', 'x1': '2017-4-16',
'y0': 0, 'y1': 1, 'xref': 'x', 'yref': 'paper',
'line': {'color': 'rgb(30,30,30)', 'width': 1}
}],
'annotations': [{
'x': '2015-05-16', 'y': 0.05, 'xref': 'x', 'yref': 'paper',
'showarrow': False, 'xanchor': 'left',
'text':''
}]
}
fig = dict(data=data, layout=layout)
iplot(fig, filename='aapl-recession-candlestick')
We can also find great insight from Microsoft's historic financial data by plotting moving averages based on close prices. We're re-introducing the graph earlier that visualized the distribution of close prices across time. Take a look.
fifteen_day = pd.rolling_mean(df["Close"], 15)
fifty_day = pd.rolling_mean(df["Close"], 40)
twohundred_day = pd.rolling_mean(df["Close"], 120)
plt.plot(data_range,train_y)
plt.plot(data_range,fifteen_day)
plt.plot(data_range,fifty_day)
plt.plot(data_range,twohundred_day)
plt.title('Close Prices Along w/ Moving Averages from May 16, 2015 to April 16, 2017')
plt.ylabel('Closing Price', fontsize=14)
plt.xlabel('Date Range', fontsize=14)
plt.show()
In this illustration, I'm using a cross strategy with 3 different interval of days for moving averages: 15, 40 and 120. Notice that the 40-day and 120-day moving average is slower to adjust to price changes because it uses more data points in its calculation while the 15-day is quicker to respond due to each value having a greater weight in the calculation from the shorter time horizon.
As you can see from this cross strategy, you would watch for the 15-day (orange) average to cross below the 40-day (green) moving average as an entry for a short position.
Another insight derived from this cross strategy is momentum. Whenever the shorter term moving average is above the longer term, it confirms upward momentum. We can see there's an upward momentum after the 350 date range mark. This is a great indication to hold on to your stocks as it's a good chance the prices would continue to rise.
Moving averages is a excellent tool to spot trends because noise can be eliminated by properly adjusting the day intervals. You can learn more about analyzing moving averages here: http://www.investopedia.com/articles/technical/052201.asp
Another cool feature worth plotting is the average change of closing prices each month. This is calculated by grouping each day's closing price by their respective months and averaging them. Here's how we've done so. I would be using the financial data from May 16, 2015 to April 16, 2017.
from dateutil import parser
datetime_range = df["Date"].apply(lambda x: parser.parse("{} 11:00PM".format(x))) # create datetime-like objects from Date column
df.set_index(datetime_range, inplace=True)
monthly = df.resample('BM').mean() # group rows by month and average them
returns = monthly.xs(('Close'), axis=1).pct_change() # calculate return percentages from these averages for each month
correlation_fig = plt.figure()
plt.scatter(np.arange(len(monthly)),returns)
correlation_fig.suptitle('Returns of MSFT Closing Prices Across Months', fontsize=12)
plt.xlabel('Month Range', fontsize=11)
plt.ylabel('Return on MSFT Stock', fontsize=11)
plt.plot(np.arange(len(monthly)),returns)
plt.show()
With this kind of illustration you can see how Microsoft is doing at a month-to-month basis. Microsoft's close price took a substantial positive turn at the end of September, 2015 into October, 2015. Conversely, it began to take a negative turn during the months November to January in 2015.
Step 4: Linear Regression¶
What is Linear Regression¶
Linear Regression is a form of regression analysis.The goal of Linear Regression is to find a relationship between a set of independent and dependent variables in order to create a Mathematical linear function that can be used to predict dependent values for any given independent value.To learn about linear regression more in depth check out the the following page which explains the basics of linear regression.
http://www.statisticssolutions.com/what-is-linear-regression/
We will first create our Linear Regression model using linear_models built in LinearRegression function. Next we will we train our model using the open prices from May 16, 2015 to April 16, 2017 as our independent variable and their closing prices as our dependent variable.
linear_mod = linear_model.LinearRegression()
linear_mod.fit(train_x,train_y)
Now I will use this trained model to predict the close prices from April 16, 2017 to present.
df2 = pd.read_csv('MSFT2.csv')
size = len(df2.index)
actual_close = np.array(df2["Close"])
#test_X = df2.as_matrix()
test_X = np.array(df2["Open"]).reshape(size,1)
predicted_price = linear_mod.predict(test_X)
data_range = np.arange(size)
plt.plot(data_range, predicted_price)
plt.plot(data_range, actual_close)
plt.legend(['predicted closing', 'actual closing'], loc='upper left')
plt.title('Close Price Forecast from April 16, 2017 to Present')
plt.ylabel('Closing Price', fontsize=14)
plt.xlabel('Date Range', fontsize=14)
plt.show()
#print "predicted closing price: {}".format()
This is for analysis purposes only. This isn't really a useful tool because, in reality, we aren't given open prices up front. This only predicts close prices based on open prices. It would be more convenient if we can predict these open prices so that we can forecast the closing prices of later dates.
Another insight: high and low prices play a strong role in predicting the closing price. It would be nice to have a program that constantly checks the highs and lows of the day to recalculate the prediction of the closing price. The training set would be a matrix of open, high and low as features, and close as the expected output.
df3 = df[["Open","High","Low"]]
df3.head(10)
train_X = df3.as_matrix() # turns into 482 x 3 multi-dimensional numpy array
linear_mod.fit(train_X,train_y) # training data is from May 16, 2015 to April 16, 2017
Now let's predict the closing price for April 17, 2017 by using the open, high and low prices for that day.
test_x = np.array([df2["Open"][0],df2["High"][0],df2["Low"][0]]).reshape(1,3)
predicted_close_price = linear_mod.predict(test_x)
print ("Predicted close price: {}\nActual close price: {}".format(predicted_close_price,df2["Close"][0]))
The moral of the story is that there is no linear correlation to closing stock prices. You may have better luck when measuring large indexes like the S&P 500. One of the main reasons why there isn't a linear correlation is because stocks are deeply impacted by the news. Just this past Wednesday, Snap release a report about their not-as-anticipated slow user growth. Their stock fell as much as 25% hours later. Therefore, past stock prices can not predict future prices because news events have different magnitudes of effect on stock and they happen at arbitrary dates.
There are two solutions. We can use a tool that does a sentiment analysis on present-day news to predict the closing stock price or we can predict news events in the future and their respective magnitudes of influence. Both will not be covered in this tutorial.