{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"___\n",
"\n",
" \n",
"___\n",
"
*Copyright Pierian Data 2017*\n",
"*For more information, visit us at www.pieriandata.com*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Time Shifting\n",
"\n",
"Sometimes you may need to shift all your data up or down along the time series index, in fact, a lot of pandas built-in methods do this under the hood. This isn't something we won't do often in the course, but its definitely good to know about this anyways!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"df = pd.read_csv('time_data/walmart_stock.csv',index_col='Date')\n",
"df.index = pd.to_datetime(df.index)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2012-01-03 | \n",
" 59.970001 | \n",
" 61.060001 | \n",
" 59.869999 | \n",
" 60.330002 | \n",
" 12668800 | \n",
" 52.619235 | \n",
"
\n",
" \n",
" 2012-01-04 | \n",
" 60.209999 | \n",
" 60.349998 | \n",
" 59.470001 | \n",
" 59.709999 | \n",
" 9593300 | \n",
" 52.078475 | \n",
"
\n",
" \n",
" 2012-01-05 | \n",
" 59.349998 | \n",
" 59.619999 | \n",
" 58.369999 | \n",
" 59.419998 | \n",
" 12768200 | \n",
" 51.825539 | \n",
"
\n",
" \n",
" 2012-01-06 | \n",
" 59.419998 | \n",
" 59.450001 | \n",
" 58.869999 | \n",
" 59.000000 | \n",
" 8069400 | \n",
" 51.459220 | \n",
"
\n",
" \n",
" 2012-01-09 | \n",
" 59.029999 | \n",
" 59.549999 | \n",
" 58.919998 | \n",
" 59.180000 | \n",
" 6679300 | \n",
" 51.616215 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2012-01-03 59.970001 61.060001 59.869999 60.330002 12668800 52.619235\n",
"2012-01-04 60.209999 60.349998 59.470001 59.709999 9593300 52.078475\n",
"2012-01-05 59.349998 59.619999 58.369999 59.419998 12768200 51.825539\n",
"2012-01-06 59.419998 59.450001 58.869999 59.000000 8069400 51.459220\n",
"2012-01-09 59.029999 59.549999 58.919998 59.180000 6679300 51.616215"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2016-12-23 | \n",
" 69.430000 | \n",
" 69.750000 | \n",
" 69.360001 | \n",
" 69.540001 | \n",
" 4803900 | \n",
" 69.032411 | \n",
"
\n",
" \n",
" 2016-12-27 | \n",
" 69.300003 | \n",
" 69.820000 | \n",
" 69.250000 | \n",
" 69.699997 | \n",
" 4435700 | \n",
" 69.191240 | \n",
"
\n",
" \n",
" 2016-12-28 | \n",
" 69.940002 | \n",
" 70.000000 | \n",
" 69.260002 | \n",
" 69.309998 | \n",
" 4875700 | \n",
" 68.804087 | \n",
"
\n",
" \n",
" 2016-12-29 | \n",
" 69.209999 | \n",
" 69.519997 | \n",
" 69.120003 | \n",
" 69.260002 | \n",
" 4298400 | \n",
" 68.754456 | \n",
"
\n",
" \n",
" 2016-12-30 | \n",
" 69.120003 | \n",
" 69.430000 | \n",
" 68.830002 | \n",
" 69.120003 | \n",
" 6889500 | \n",
" 68.615479 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2016-12-23 69.430000 69.750000 69.360001 69.540001 4803900 69.032411\n",
"2016-12-27 69.300003 69.820000 69.250000 69.699997 4435700 69.191240\n",
"2016-12-28 69.940002 70.000000 69.260002 69.309998 4875700 68.804087\n",
"2016-12-29 69.209999 69.519997 69.120003 69.260002 4298400 68.754456\n",
"2016-12-30 69.120003 69.430000 68.830002 69.120003 6889500 68.615479"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## .shift() forward"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2012-01-03 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" 2012-01-04 | \n",
" 59.970001 | \n",
" 61.060001 | \n",
" 59.869999 | \n",
" 60.330002 | \n",
" 12668800.0 | \n",
" 52.619235 | \n",
"
\n",
" \n",
" 2012-01-05 | \n",
" 60.209999 | \n",
" 60.349998 | \n",
" 59.470001 | \n",
" 59.709999 | \n",
" 9593300.0 | \n",
" 52.078475 | \n",
"
\n",
" \n",
" 2012-01-06 | \n",
" 59.349998 | \n",
" 59.619999 | \n",
" 58.369999 | \n",
" 59.419998 | \n",
" 12768200.0 | \n",
" 51.825539 | \n",
"
\n",
" \n",
" 2012-01-09 | \n",
" 59.419998 | \n",
" 59.450001 | \n",
" 58.869999 | \n",
" 59.000000 | \n",
" 8069400.0 | \n",
" 51.459220 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2012-01-03 NaN NaN NaN NaN NaN NaN\n",
"2012-01-04 59.970001 61.060001 59.869999 60.330002 12668800.0 52.619235\n",
"2012-01-05 60.209999 60.349998 59.470001 59.709999 9593300.0 52.078475\n",
"2012-01-06 59.349998 59.619999 58.369999 59.419998 12768200.0 51.825539\n",
"2012-01-09 59.419998 59.450001 58.869999 59.000000 8069400.0 51.459220"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(1).head()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2016-12-23 | \n",
" 71.239998 | \n",
" 71.239998 | \n",
" 69.209999 | \n",
" 69.589996 | \n",
" 12106800.0 | \n",
" 69.082042 | \n",
"
\n",
" \n",
" 2016-12-27 | \n",
" 69.430000 | \n",
" 69.750000 | \n",
" 69.360001 | \n",
" 69.540001 | \n",
" 4803900.0 | \n",
" 69.032411 | \n",
"
\n",
" \n",
" 2016-12-28 | \n",
" 69.300003 | \n",
" 69.820000 | \n",
" 69.250000 | \n",
" 69.699997 | \n",
" 4435700.0 | \n",
" 69.191240 | \n",
"
\n",
" \n",
" 2016-12-29 | \n",
" 69.940002 | \n",
" 70.000000 | \n",
" 69.260002 | \n",
" 69.309998 | \n",
" 4875700.0 | \n",
" 68.804087 | \n",
"
\n",
" \n",
" 2016-12-30 | \n",
" 69.209999 | \n",
" 69.519997 | \n",
" 69.120003 | \n",
" 69.260002 | \n",
" 4298400.0 | \n",
" 68.754456 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2016-12-23 71.239998 71.239998 69.209999 69.589996 12106800.0 69.082042\n",
"2016-12-27 69.430000 69.750000 69.360001 69.540001 4803900.0 69.032411\n",
"2016-12-28 69.300003 69.820000 69.250000 69.699997 4435700.0 69.191240\n",
"2016-12-29 69.940002 70.000000 69.260002 69.309998 4875700.0 68.804087\n",
"2016-12-30 69.209999 69.519997 69.120003 69.260002 4298400.0 68.754456"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You will lose that last piece of data that no longer has an index!\n",
"df.shift(1).tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## shift() backwards"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2012-01-03 | \n",
" 60.209999 | \n",
" 60.349998 | \n",
" 59.470001 | \n",
" 59.709999 | \n",
" 9593300.0 | \n",
" 52.078475 | \n",
"
\n",
" \n",
" 2012-01-04 | \n",
" 59.349998 | \n",
" 59.619999 | \n",
" 58.369999 | \n",
" 59.419998 | \n",
" 12768200.0 | \n",
" 51.825539 | \n",
"
\n",
" \n",
" 2012-01-05 | \n",
" 59.419998 | \n",
" 59.450001 | \n",
" 58.869999 | \n",
" 59.000000 | \n",
" 8069400.0 | \n",
" 51.459220 | \n",
"
\n",
" \n",
" 2012-01-06 | \n",
" 59.029999 | \n",
" 59.549999 | \n",
" 58.919998 | \n",
" 59.180000 | \n",
" 6679300.0 | \n",
" 51.616215 | \n",
"
\n",
" \n",
" 2012-01-09 | \n",
" 59.430000 | \n",
" 59.709999 | \n",
" 58.980000 | \n",
" 59.040001 | \n",
" 6907300.0 | \n",
" 51.494109 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2012-01-03 60.209999 60.349998 59.470001 59.709999 9593300.0 52.078475\n",
"2012-01-04 59.349998 59.619999 58.369999 59.419998 12768200.0 51.825539\n",
"2012-01-05 59.419998 59.450001 58.869999 59.000000 8069400.0 51.459220\n",
"2012-01-06 59.029999 59.549999 58.919998 59.180000 6679300.0 51.616215\n",
"2012-01-09 59.430000 59.709999 58.980000 59.040001 6907300.0 51.494109"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(-1).head()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2016-12-23 | \n",
" 69.300003 | \n",
" 69.820000 | \n",
" 69.250000 | \n",
" 69.699997 | \n",
" 4435700.0 | \n",
" 69.191240 | \n",
"
\n",
" \n",
" 2016-12-27 | \n",
" 69.940002 | \n",
" 70.000000 | \n",
" 69.260002 | \n",
" 69.309998 | \n",
" 4875700.0 | \n",
" 68.804087 | \n",
"
\n",
" \n",
" 2016-12-28 | \n",
" 69.209999 | \n",
" 69.519997 | \n",
" 69.120003 | \n",
" 69.260002 | \n",
" 4298400.0 | \n",
" 68.754456 | \n",
"
\n",
" \n",
" 2016-12-29 | \n",
" 69.120003 | \n",
" 69.430000 | \n",
" 68.830002 | \n",
" 69.120003 | \n",
" 6889500.0 | \n",
" 68.615479 | \n",
"
\n",
" \n",
" 2016-12-30 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2016-12-23 69.300003 69.820000 69.250000 69.699997 4435700.0 69.191240\n",
"2016-12-27 69.940002 70.000000 69.260002 69.309998 4875700.0 68.804087\n",
"2016-12-28 69.209999 69.519997 69.120003 69.260002 4298400.0 68.754456\n",
"2016-12-29 69.120003 69.430000 68.830002 69.120003 6889500.0 68.615479\n",
"2016-12-30 NaN NaN NaN NaN NaN NaN"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shift(-1).tail()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shifting based off Time String Code \n",
"### Using tshift()"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Open | \n",
" High | \n",
" Low | \n",
" Close | \n",
" Volume | \n",
" Adj Close | \n",
"
\n",
" \n",
" Date | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" 2012-01-31 | \n",
" 59.970001 | \n",
" 61.060001 | \n",
" 59.869999 | \n",
" 60.330002 | \n",
" 12668800 | \n",
" 52.619235 | \n",
"
\n",
" \n",
" 2012-01-31 | \n",
" 60.209999 | \n",
" 60.349998 | \n",
" 59.470001 | \n",
" 59.709999 | \n",
" 9593300 | \n",
" 52.078475 | \n",
"
\n",
" \n",
" 2012-01-31 | \n",
" 59.349998 | \n",
" 59.619999 | \n",
" 58.369999 | \n",
" 59.419998 | \n",
" 12768200 | \n",
" 51.825539 | \n",
"
\n",
" \n",
" 2012-01-31 | \n",
" 59.419998 | \n",
" 59.450001 | \n",
" 58.869999 | \n",
" 59.000000 | \n",
" 8069400 | \n",
" 51.459220 | \n",
"
\n",
" \n",
" 2012-01-31 | \n",
" 59.029999 | \n",
" 59.549999 | \n",
" 58.919998 | \n",
" 59.180000 | \n",
" 6679300 | \n",
" 51.616215 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Open High Low Close Volume Adj Close\n",
"Date \n",
"2012-01-31 59.970001 61.060001 59.869999 60.330002 12668800 52.619235\n",
"2012-01-31 60.209999 60.349998 59.470001 59.709999 9593300 52.078475\n",
"2012-01-31 59.349998 59.619999 58.369999 59.419998 12768200 51.825539\n",
"2012-01-31 59.419998 59.450001 58.869999 59.000000 8069400 51.459220\n",
"2012-01-31 59.029999 59.549999 58.919998 59.180000 6679300 51.616215"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Shift everything forward one month\n",
"df.tshift(periods=1,freq='M').head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That is it for now!"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python [conda root]",
"language": "python",
"name": "conda-root-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}