Many data projects start with a simple hypothesis - “We have all this data. Can we use the data to X”, where X is a goal that leads to increased revenue or reduced costs. The success of these initiatives hinges on the availability of high-quality data, which is often not available because of historical reasons and/or legacy data sources.

In this presentation, I will delve into the real-world challenges posed by bad data, offering concrete examples to illustrate their annoyances and complexities. Some examples:

• Large Excel repositories with years of historical data
• VARCHAR gotchas in databases
• Joins across tables when there are "slight" mismatches
• Problems with incorrect categorical data
• And everyone's favorite - datetimes

I explore the underlying causes and hypotheses behind the prevalence of poor data. I go over some simple tools, techniques, and patterns necessary to identify and correct data quality issues. For data validation, verification, and automation, I will introduce simple workflows to pinpoint and mitigate bad data using readily accessible tools from the PyData and Python ecosystems. Examples:
* pandas, sqlite, datasette
* Fuzzy string matching with thefuzz
* read_excel patterns
* Power of assert statements
* Makefiles for workflows

Finally, I go over prescriptive strategies for navigating data projects when confronted with less-than-ideal conditions - both technical and organizational.

This is a beginner/intermediate talk for domain experts trying to make better use of data in their organization. Basic knowledge of the PyData ecosystem is helpful but not required. The audience will find some comfort in the fact that "they're not alone" and leave with techniques that can help them overcome these challenges.