This is a Macchiato Post [Middle Espresso with a touch]:

Data manipulation, commonly referred to as “Data Wrangling”, is the process of transforming and structuring data from a raw format to a desired format in order to improve its quality and make it more consumable and useful for analysis or for use, for example, in machine learning.

The goal of data manipulation is to ensure the quality and usefulness of the data. It is well known that if any raw data source is deficient, the analysis will also be deficient. Therefore, data scientists often dedicate more time to this process than to the actual data analysis.

The data manipulation process can include post-processing, data visualization, data aggregation, training a statistical model, and many other well-documented techniques.

¿How does data manipulation work?

Data management involves six (6) main steps: explore, cleanse, transform, enrich, validate, and store. We explain them below:

1.- Explore: Data exploration or discovery is a way to identify patterns, trends, and missing or incomplete information in a dataset. Most exploration is done before creating reports, data visualizations, or training models, but it’s also common to discover surprises and insights in a dataset during analysis.

2.- Clean: Data often contains errors as a result of manual entry, incomplete data, data automatically collected by sensors, or even faulty equipment. Data cleaning corrects these input errors, removes duplicates and outliers (if applicable), eliminates missing data, and imputes null values ​​according to statistical or conditional models to improve data quality.

3.- Transform: Transforming or structuring data is a crucial step. Failure to do so from the outset can compromise the remaining processes. Data transformation involves converting raw data into the correct format for use in reports, data visualizations, or analytical and modeling processes. This may include creating new variables (also known as features) and performing mathematical operations on the data.

4.- Enrich: Enrichment, or combining, increases the usefulness of a dataset by integrating additional sources, such as reliable demographic or census data. The enrichment process can also help extract additional insights from data within an organization or generate new ideas for collecting and storing additional customer information in the future. This is an opportunity to think strategically about what additional data could contribute to a report, model, or business process.

5.- Validate: Rules are repetitive programming sequences that check the consistency, quality, and security of data. Some examples of validation include ensuring a uniform distribution of attributes that should have a normal distribution (e.g., birth dates) or confirming the accuracy of fields through data checking. This is a vital step in the data manipulation process.

6.- Storage: The final part of the preparation process consists of storing or preserving the final product, along with all the steps and transformations carried out, so that it can be audited, understood and used in the future.