Data Cleaning with ECDF Statistics in Python (Explanation and Example)

This time I decided to demonstrate how I use ECDF (Empirical Cummulative Distirbution Function) for Data Cleaning with Real Data (real estate data scrapped by my own scrapper: https://www.youtube.com/watch?v=pqAdxZWFkTM

For this I use Numpy, Pandas and Matplotlib Python modules.
The main idea is ECDF helps us to detect data outliers, in other words - data noise that must be removed from original (raw) dataset. This called as an iteration.
After each iteration data get more and more clear. This is one of most technique for Data Preproccessing (Data Cleaning) based on Statistics mostly.

For making code more structured I divided the whole algorithm to separate sections. Important procedures (ECDF, calculate percentiles, plot the data) are defined as Python definition with input arguments.

I think that ECDF graph is the best way to plot the data distribution for data exploration purposes.

Example showed in the end of video: https://www.linkedin.com/feed/update/urn:li:activity:6390315794722537472

Whole code is written in Python programming language on framework of Jupyter Notebook.

Our task in here is get as a real data that less or more correspond the normal distribution. This type of observation distribution will works with further analyses.

After Data Cleaning by using ECDF Statistics method you can easily apply Machine Learning (ML), Deep Learning (DL), Exploratory Data Analysis (EDA, Data Exploration) and other algorithms for your data analysis. Hope this will be useful for Data Analyst, Data Scientist and for all who are in passion about data world.

Also, I explain why almost all the time better use statistical Median tha Average. It is based on my calculations.

Vytautas
https://www.linkedin.com/in/bielinskas/