The shift() method for a pandas series helps shift values in a column up or down. This is similar to using the SQL window functions for LAG() and LEAD(). You can learn about these SQL window functions via Mode's SQL tutorial.

1. Shift Downtime
2. Shift Download
3. Shift Down Words
4. Excel Ctrl Shift Down

For the raven method, you'll need to lie down in the starfish position when you're half asleep and count down from 100. You then have to imagine your desired reality, or if you have a prewritten script about how you want your shifted reality experience to pan out prepared, recite it. Move selected team down. Command+Shift+Down arrow key. Open the History menu. Go to the previous section. Go to the next section.

In this tutorial, I'll walk through an example of using the shift() pandas series method for analyzing bike rides.

In this example, I assume a service in which I lend out a single bike for people to ride throughout the day in San Francisco. Each record in the dataset is the start and end time for a ride.

The code below creates a list of start and end times of rides.

Create a pandas dataframe given the lists ride_start_times and ride_end_times.

For example, the first ride ended around 4:24PM and was next used for a ride at 5:52PM. I want to calculate a new column that states there was approximately 86 minutes, equivalent to approximately 5160 seconds, of idle time between these rides.

I want to do a column-by-column comparison. I use the shift() method to create a new column in df_bike_rides that's a shift of value in ride_end_time down one period.

The shift() method for a pandas series is similar to a window function in SQL using LAG() and LEAD(). The same operation above would look like the following in SQL:

Given this new column for previous_ride_end_time, I can subtract the time between a new bike ride's start time and the previous ride's end time. The result is the duration the bike was idle between rides.

The new column duration_bike_idle_between_rides shows the duration of idle bike time between rides in the format HH-MM-SS. The value of 01:02:00 is equivalent to saying 1 hour and 2 minutes. Below, I convert that timedelta format into a single numerical value of minutes. I utilize the dt accessor and total_seconds()method to calculate the total seconds a bike is idle between rides. Then I divide this value by 60 to get a value in minutes.

View df_bike_rides below with a new column for the minutes_bike_idle_between_rides.

I calculate the mean minutes_bike_idle_between_rides value as 63 minutes.

This example below is similar to the one above. However, I assume I now operate a fleet of 2 bikes and rent them out for people to ride to specific stations in the city of San Francisco.

Below I create a pandas dataframe with details on bike ride times, the bike id and the start and end station.

I sort df_bike_sharing first by the bike_id column and then the ride_start_time column.

For the bike_id column, I shift down values by 1 to create a new column called previous_bike_id. I do this so I can easily compare a bike id to the previous ride's ID to identify the last ride by a bike id for a day.

Similar to LAG()

In SQL, this operation would be:

For the ride_end_time column, I shift down values by 1 to create a new column called previous_ride_end_time. I do this so I can later find the idle bike time between rides.

Shift Downtime

We only want to calculate a duration the bike was idle in a row if it's a comparison for the same bike id. Below I show the code to replace all values in a row with NaN or NaT if the condition is false.

With the above change, I calculate ride_start_time minus previous_ride_end_time where the above condition holds True. With that calculation, I create a new column called duration_bike_idle.

I drop previous_bike_id and previous_ride_end_time since they were intermediary outputs used for calculations. They're not necessary for a final presentation of the critical details.

The new column duration_bike_idle_seconds shows the duration of idle bike time between rides in the format HH-MM-SS. The value of 01:26:00 is equivalent to saying 1 hour and 26 minutes. Below, I convert that timedelta format into a single numerical value of minutes. I utilize the dt accessor and total_seconds()method to calculate the total seconds a bike is idle between rides of the same bike id. Then I divide this value by 60 to get a value in minutes.

Here is the average seconds each bike_id is idle during the day in seconds between the first and last ride. I group by the bike_id column and calculate the mean of the duration_bike_idle_seconds_time column. I reset the index and rename the columns so this final output is easier to understand.

Bike id of 22 was left idle longer between rides than the bike id of 1.

Some functions (like Sine and Cosine) repeat forever
and are called Periodic Functions.

Shift Download

The Period goes from one peak to the next (or from any point to the next matching point):

The Amplitude is the height from the center line to the peak (or to the trough). Or we can measure the height from highest to lowest points and divide that by 2.

The Phase Shift is how far the function is shifted horizontally from the usual position.

The Vertical Shift is how far the function is shifted vertically from the usual position.

All Together Now!

We can have all of them in one equation:

y = A sin(B(x + C)) + D

• amplitude is A
• period is 2π/B
• phase shift is C (positive is to the left)
• vertical shift is D

And here is how it looks on a graph:

Note that we are using radians here, not degrees, and there are 2π radians in a full rotation.

Example: sin(x)

This is the basic unchanged sine formula. A = 1, B = 1, C = 0 and D = 0

So amplitude is 1, period is 2π, there is no phase shift or vertical shift:

Example: 2 sin(4(x − 0.5)) + 3

• amplitude A = 2
• period 2π/B = 2π/4 = π/2
• phase shift = −0.5 (or 0.5 to the right)
• vertical shift D = 3

In words:

• the 2 tells us it will be 2 times taller than usual, so Amplitude = 2
• the usual period is 2π, but in our case that is 'sped up' (made shorter) by the 4 in 4x, so Period = π/2
• and the −0.5 means it will be shifted to the right by 0.5
• lastly the +3 tells us the center line is y = +3, so Vertical Shift = 3

Instead of x we can have t (for time) or maybe other variables:

Example: 3 sin(100t + 1)

First we need brackets around the (t+1), so we can start by dividing the 1 by 100:

3 sin(100t + 1) = 3 sin(100(t + 0.01))

Now we can see:

• amplitude is A = 3
• period is 2π/100 = 0.02 π
• phase shift is C =0.01 (to the left)
• vertical shift is D = 0

And we get:

Frequency

Frequency is how often something happens per unit of time (per '1').

Example: Here the sine function repeats 4 times between 0 and 1:

So the Frequency is 4

And the Period is 14

In fact the Period and Frequency are related:

Frequency = 1Period

Period = 1Frequency

Shift Down Words

Example from before: 3 sin(100(t + 0.01))

Excel Ctrl Shift Down

The period is 0.02π

So the Frequency is 10.02π = 50π

Some more examples:

When frequency is per second it is called 'Hertz'.

The faster it bounces the more it 'Hertz'!

Animation