Working with Date & Time

Learn pandas’ datetime features to parse dates, extract components, and work with time-series data.

10 minutesVideo LessonPDF notes

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Introduction to Time Series — The lesson introduces the powerful time series functionality available in the Pandas library.
String Slicing Pitfall — The instructor demonstrates using string slicing to extract the hour, noting that this approach is brittle and easily broken.
Converting to Datetime — The 'time' column is converted using `pd.to_datetime()`, changing the column's data type to `datetime64`.
Using the .dt Accessor — The `.dt` namespace is used to easily extract attributes like `.hour`, `.weekday_name`, and `.dayofyear` from the datetime Series.
Creating a Timestamp — A single date string is converted into a `Timestamp` object for use in subsequent comparisons.
Datetime Comparisons — The DataFrame is filtered using a comparison operator between the datetime Series and the newly created Timestamp object.
Timedelta Calculation — Subtracting the minimum time from the maximum time yields a `Timedelta` object representing the total duration of the dataset.
Plotting Time Series Data — The year is extracted, value counts are calculated, the index is sorted, and the results are plotted chronologically.

PDF notes

Bottom line: You will learn how to convert string-based date and time data into the specialized Pandas datetime format. This conversion unlocks powerful time series functionality, allowing you to easily extract components like year or hour and perform comparisons and mathematical operations. You will also learn how to aggregate and visualize time series data.

This lesson on Working with Date & Time is hands-on and example-driven. You will learn how to convert string-based date and time data into the specialized Pandas datetime format. This conversion unlocks powerful time series functionality, allowing you to easily extract components like year or hour and perform comparisons and mathematical operations. You will also learn how to aggregate and visualize time series data.

What You'll Be Able To Do

Convert a Pandas Series of date strings into the datetime64 format using pd.to_datetime().
Extract specific time components (e.g., hour, day of year) using the .dt accessor.
Perform logical comparisons between a datetime Series and a single Timestamp object.
Calculate the difference between two Timestamps, resulting in a Timedelta object.
Aggregate and plot time series data by extracting and counting time components like the year.

Detailed Concept Walkthrough

1. Converting to Datetime Format

Pandas requires dates and times to be in the specialized datetime64 format to enable time series analysis. The pd.to_datetime() function automatically parses common date string formats.

Mechanism: pd.to_datetime() takes a Series of strings and attempts to infer the format for each entry, converting them into Timestamp objects within a datetime64 Series.
Best Practice: Avoid brittle string slicing for extraction; always convert the column first to leverage built-in, robust datetime functionality.
Under the Hood: The resulting Series has a dtype of datetime64[ns], which stores the date/time as a large integer representing nanoseconds since the epoch (Jan 1, 1970).

import pandas as pd
# Assuming UFO['time'] is a Series of date strings
UFO['time'] = pd.to_datetime(UFO['time'])
# The dtype is now datetime64[ns]

Key Takeaway: Convert date strings using pd.to_datetime() to unlock all time series features in Pandas.

2. Extracting Time Components

Once a column is in the datetime format, the .dt accessor provides a namespace to easily pull out specific attributes like the hour, day name, or year.

Mechanism: The .dt accessor is applied directly to a datetime Series and exposes properties (like .hour, .year) and methods (like .weekday_name) that return new Series containing the extracted values.
Best Practice: Use .dt.weekday_name or .dt.weekday to analyze data by day of the week without writing custom mapping or conversion code.
Syntax Rule: The accessor must be chained: Series.dt.attribute. It cannot be used on non-datetime Series.

# Assuming UFO['time'] is datetime64
hour_series = UFO['time'].dt.hour
print(UFO['time'].dt.weekday_name.head())

Key Takeaway: Use the .dt accessor to quickly and reliably extract any component from a datetime Series.

3. Datetime Math and Comparisons

Pandas allows direct comparison and mathematical operations between datetime objects, enabling filtering by time range and calculating durations.

Mechanism: A single date string converted by pd.to_datetime() results in a Timestamp object, which can be compared directly against an entire datetime Series.
Execution Flow: Comparisons (e.g., >) return a boolean Series used for filtering rows via .loc[] based on time thresholds.
Under the Hood: Subtracting one Timestamp from another (e.g., Max - Min) yields a Timedelta object, which represents a duration and has its own attributes like .days.

TS = pd.to_datetime('1/1/1999')
recent = UFO.loc[UFO['time'] >= TS, :]
duration = UFO['time'].max() - UFO['time'].min()

Key Takeaway: Datetime objects support standard mathematical and comparison operators for powerful time-based filtering and duration calculation.

Topics Covered in Working with Date & Time

Introduction to Time Series (0:20 - 0:40) — The lesson introduces the powerful time series functionality available in the Pandas library.
String Slicing Pitfall (0:50 - 1:30) — The instructor demonstrates using string slicing to extract the hour, noting that this approach is brittle and easily broken.
Converting to Datetime (1:35 - 2:40) — The 'time' column is converted using pd.to_datetime(), changing the column's data type to datetime64.
Using the .dt Accessor (2:45 - 3:45) — The .dt namespace is used to easily extract attributes like .hour, .weekday_name, and .dayofyear from the datetime Series.
Creating a Timestamp (3:55 - 4:45) — A single date string is converted into a Timestamp object for use in subsequent comparisons.
Datetime Comparisons (4:45 - 5:25) — The DataFrame is filtered using a comparison operator between the datetime Series and the newly created Timestamp object.
Timedelta Calculation (5:30 - 6:10) — Subtracting the minimum time from the maximum time yields a Timedelta object representing the total duration of the dataset.
Plotting Time Series Data (6:15 - 7:45) — The year is extracted, value counts are calculated, the index is sorted, and the results are plotted chronologically.

Python Cheat Sheet

pd.to_datetime(series) — Converts string Series to datetime format
```
UFO['time'] = pd.to_datetime(UFO['time'])
```
Series.dt.attribute — Accesses specific time components
```
UFO['time'].dt.year
```
Series.dt.weekday_name — Returns the name of the day of the week
```
UFO['time'].dt.weekday_name.head()
```
Timestamp comparison — Filters data based on a time threshold
```
UFO.loc[UFO['time'] >= TS, :]
```
Timestamp subtraction — Calculates the duration between two points
```
UFO['time'].max() - UFO['time'].min()
```
Timedelta.days — Extracts the total number of days from duration
```
duration.days
```
Series.sort_index().plot() — Orders counts by index and generates a plot
```
counts.sort_index().plot()
```
pd.to_datetime('1/1/1999') — Creates a single Timestamp object
```
TS = pd.to_datetime('1/1/1999')
```

Comparison Table

Method	Data Type	Flexibility/Robustness
String Slicing	`object` (string)	Brittle; breaks easily with format changes.
`pd.to_datetime()`	`datetime64[ns]`	Robust; infers formats automatically.
Datetime Attributes	`datetime64[ns]`	Allows extraction of hour, year, weekday name.

Common Pitfalls

Mistake: Trying to use .dt accessor on a string column. Avoid: Ensure the column dtype is datetime64 before using .dt.
Mistake: Relying on string slicing to extract time components. Avoid: Use pd.to_datetime() for robust, format-independent extraction.
Mistake: Comparing a datetime Series to a plain date string. Avoid: Convert the comparison date into a Timestamp using pd.to_datetime().
Mistake: Forgetting to sort the index before plotting counts by year. Avoid: Chain .sort_index() before .plot() to ensure chronological order.

FAQs

What if pd.to_datetime() doesn't automatically figure out my date format? The pd.to_datetime() function accepts a format argument where you can explicitly specify the expected date string structure using standard format codes.
What is the difference between a Timestamp and a Timedelta? A Timestamp represents a specific point in time (e.g., June 1, 1930). A Timedelta represents a duration or difference between two points in time (e.g., 25,781 days).
Why is string slicing considered brittle for date extraction? String slicing relies on fixed character positions; if the source format changes, the slice will extract the wrong characters.
How do I see all available attributes under the .dt accessor? Consult the official Pandas API reference documentation for the Series .dt properties and methods to see all available extraction options.

Common pitfalls

Mistake: Trying to use `.dt` accessor on a string column. Avoid: Ensure the column `dtype` is `datetime64` before using `.dt`.
Mistake: Relying on string slicing to extract time components. Avoid: Use `pd.to_datetime()` for robust, format-independent extraction.
Mistake: Comparing a datetime Series to a plain date string. Avoid: Convert the comparison date into a `Timestamp` using `pd.to_datetime()`.
Mistake: Forgetting to sort the index before plotting counts by year. Avoid: Chain `.sort_index()` before `.plot()` to ensure chronological order.

hello and welcome back to my QA video series about the pandas library in Python and the question for today is how do I work with dates and times in pandas okay great question there's a lot of powerful time series functionality in pandas and in fact a pandas series is named after the time series so I'm just going to show you the basics today okay so we're going to start by importing pandas as PD and then our example data set will be UFO reports so PD read CSV I need that as a string and bitly slash UFO reports okay and let's take a look at the head all right so each row represents a UFO reported sighting and what if I wanted to analyze the sightings by year or by time of day how would I do that so let's take a look at the D types and check those out and we'll see that the time column is an object which in this case means it's stored as a string so if I wanted to analyze the hour for example I might think well I could do some string slicing okay so let's try like UFO time dot stir dot slice and we'll slice from position negative five so five characters from the end to negative three which is three characters from the end okay and that does work it outputs a string and we probably want it like this is a two character string we probably want as an integer so we'll do dot as type int animal thrown head and now we've got the hour and this works in this case but this feels like an approach that would easily break it's very brittle okay so let's use a better methodology Kai and the solution here is to convert the time column two pandas special date-time format okay so I'm going to overwrite the time column and I'm going to say UFO bracket time equals PD to date/time this is a top level function and I'm going to pass it UFO dot time okay and if we look at the head we'll see that the data seems to be the same as before the formatting looks like it's different but the real thing that's changed is the D type is now date time kinds of special date-time format now before we move on I want to point out I did not have to specify two pandas with this date time function the format of this I did not have to tell it this is a month then there's a slash character and then there's a day then there's a four-digit year then there's a space etc it just figured it out and it and it did that automatically now if you try this on your own data and it does not work automatically there are a lot of options you can use with PD to daytime in order to get it to work okay so what are some benefits that using the date-time format gives us well the big one is it exposes some really convenient attributes like UFO time dot DT so this is a little DT namespace dot our okay that pulls out the hour for us or dot weekday name it understand it actually knows that June 1st 1930 was a Sunday you don't have to write custom code if you're for instance analyzing something by day of the week and there's also just a number version of the weekday ok you can do you can do other things like day of year and it will tell us that June 1st 1930 was the 150 second day of that year ok so there's a ton of attributes like this if you want to see them all go google pandas api reference get to this page and then search for dot DT dot and you'll click on date time like properties and you'll see all of these series properties under series DT dot whatever okay so um let me show you a couple more things and let's do dot heads so this is not taking over the screen and um let's pass PD dot to date time uh date time let's pass it a string instead of a series and I'm just going to say one one 19.99 okay it outputs what's known as a timestamp um and one thing I want to point out is note that I change the format of how I passed I passed it month/day/year in my case and it figured it out um I did not have to specify uh you know that this was the month this was the day this was the year ok so this is called a timestamp and we're going to save this as TS and the reason I save that is I'm going to use that in a comparison so one trick you can do with timestamps is you can use them as part of comparison so if I say you fo dot Lok I can say UFO dot time greater than or equal to time stamp : so I'm saying which Rose do I want to see in the UFO data frame I want to see the ones where the time is greater than or equal to this time stamp and because UFO time is date time and because T s is a time stamp it can do this comparison with greater than or equal to and then the : just means show me all columns so we'll run that and we'll see that it's only showing us rows in which the time is greater than meaning into the future from January 1st 1999 okay pretty cool all right let me throw that head on here and another trick I want to show you is that you can do kind of like mathematical operations with the date-time format so if I say UFO time max it will tell me the latest timestamp in the time series okay I can even say UFO time dot max - UFO time min so you can do math with these and it tells you it outputs in a special object called a time delta object and it tells you that the difference between the earliest row and the latest row is twenty-five thousand seven hundred eighty one days etc okay so this is called a time delta object and time delta objects also have attributes like dot days and you can pull things out like that so you can get really fancy with the stuff this is really just the basics as always I'm going to end with a bonus and the bonus for today is I'm going to do a little plotting of the number of UFO reports by year okay so to plot in the Jupiter notebook we need to do plot live in line okay that allows plots to appear and then I'm going to store I'm going to create a new column called UFO year and I'm actually just going to store UFO time dot DT dot year in it okay and we can see that that worked you go for that head okay so we've got this year column alright now if I wanted if I want to analyze how many reported sightings by year one way to do it it's not the only way would be UFO year dot value counts how many rows how many times does each year appear well this is great except it's sorted by value arm so let's sort it by the index instead so dot sort index okay and now it's in the order of the index and all we have to do to get a plot is to just say dot plot and it does a line plot by default and here we go here is our plot of UFO reported sightings by ear okay so that's it for today thank you so much for joining me as always please click Subscribe if you'd like to see more videos like this please leave me a comment below if you have a comment for me or a question I'd love to hear from you but that's it so I hope to see you again soon

Frequently asked questions

What if `pd.to_datetime()` doesn't automatically figure out my date format?

The `pd.to_datetime()` function accepts a `format` argument where you can explicitly specify the expected date string structure using standard format codes.

What is the difference between a Timestamp and a Timedelta?

A Timestamp represents a specific point in time (e.g., June 1, 1930). A Timedelta represents a duration or difference between two points in time (e.g., 25,781 days).

Why is string slicing considered brittle for date extraction?

String slicing relies on fixed character positions; if the source format changes, the slice will extract the wrong characters.

How do I see all available attributes under the `.dt` accessor?

Consult the official Pandas API reference documentation for the Series `.dt` properties and methods to see all available extraction options.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.