PRODUCT GUIDE • METHODS • FORMULAS • EXAMPLES

DataForge Documentation

DataForge is a personal project created to practice and demonstrate a complete data-analysis workflow in one interface: preparing messy datasets, exploring them, producing visual evidence, applying statistical tests, and experimenting with machine-learning models. This page documents what each control means, what happens mathematically, and what kind of values a user should select in the application.

PythonFastAPIPandasNumPySciPy MatplotlibSeabornScikit-LearnVercel Blob HTMLCSSJavaScript
01 • WORKFLOW

How the application is intended to be used

1. Load data

Select CSV, XLS, XLSX, or XLAM data. DataForge inspects the columns and makes them available to later controls.

2. Inspect

Browse every row page-by-page and use Quick Summary to understand types, missing values, duplicates, spread, and central tendency.

3. Prepare

Handle missing values, remove duplicates, control outliers, or normalize numeric features. Each transformation becomes the current working dataset.

4. Explore

Select a chart. Only parameters required by that chart appear, which reduces invalid combinations and visual clutter.

5. Test

Use statistical tests when the question is about evidence, differences, association, or group means.

6. Model

Use supervised models for prediction or unsupervised models for grouping. Choose model-specific parameters only when they are relevant.

02 • DATASET BROWSER

Viewing the complete dataset without freezing the browser

The dataset table is a paged browser rather than one giant HTML table. A user can move from the first row to the final row, while the browser renders only a controlled number of records at once. This keeps memory usage and DOM work bounded even when the dataset is large. The Rows per page selector supports 50, 100, 250, or 500 records.

Example: A dataset has 84,500 rows. With 250 rows per page, the application exposes all 338 pages, but only the active 250 rows are rendered at one time.

The Export CSV and Export XLSX actions export the current working version of the dataset, including transformations already applied.

03 • QUICK SUMMARY

Dataset profiling metrics

Mean

x̄ = Σxᵢ / n

The arithmetic average of a numeric column. Useful for typical values when extreme values are not dominating the data.

App example: salary → Mean.

Median

middle value after sorting

The 50th percentile. More robust than the mean when the distribution contains large outliers.

Example: house_price with a few luxury properties.

Mode

most frequent value

The most common value. Works naturally for categories and can also be used with numeric values.

Example: district → most common district.

Variance

s² = Σ(xᵢ − x̄)² / (n − 1)

Measures squared spread around the sample mean. Large values mean observations are more dispersed.

Example: compare variability of monthly_sales.

Standard Deviation

s = √s²

Square root of variance, expressed in the original unit of the variable.

Example: age SD = 8.2 years.

Missing Values

count(is null)

Counts empty/NaN values per column so the user can decide whether to fill or remove them.

Example: income has 143 missing rows.

Duplicates

repeated row count

Counts repeated records. Duplicates can bias totals, frequencies, model training, and reports.

Example: repeated transaction records.

Data Types

numeric • text • date-like

Shows how Pandas interpreted each column. Type matters because many statistical and ML techniques require numeric values.

Example: age should normally be numeric.
04 • DATA PREPARATION

Cleaning and transformation controls

Handle Missing Values

Select one column, then choose a strategy. DataForge shows strategies that make sense for that column type.

Remove Null ValuesDeletes rows where the selected column is null. Use when missing rows are few or cannot be responsibly imputed.Example: column = latitude, strategy = Remove Null Values
MeanFills missing numeric values with x̄. Fast, but can reduce natural variance.Example: age → 31.7
MedianFills with the 50th percentile. Better when numeric data is skewed or has outliers.Example: income → median income
ModeFills with the most frequent observed value. Common for categorical fields.Example: gender → most frequent category
Minimum / MaximumUses the smallest or largest observed value. Useful only when that choice has domain meaning.Example: priority score where boundary values are intentional

Remove Duplicates

With no column selected, entire rows must match to be considered duplicates. If a column is selected, only that column is used as the duplicate key.

Example: choose Ref number when it is supposed to uniquely identify one record. Duplicate Ref numbers will be reduced to the first occurrence.

Handle Outliers — IQR Clipping

Q1 = 25th percentile • Q3 = 75th percentile • IQR = Q3 − Q1 Lower = Q1 − 1.5×IQR • Upper = Q3 + 1.5×IQR

Values outside the lower/upper boundary are clipped to the nearest boundary rather than deleting the complete row. This limits extreme influence while preserving record count.

App example: choose monthly_income. If Upper = 250,000, a value of 900,000 is replaced with 250,000.

Normalization

Min-Max

x′ = (x − min) / (max − min)

Maps numeric features approximately to 0–1. Useful for distance-based algorithms such as KNN and K-Means.

Example: age 18–80 becomes a 0–1 scale.

Z-Score

z = (x − x̄) / s

Centers around 0 and scales by standard deviation. Useful when features have very different units.

Example: salary and age before SVM/KNN.

Max Absolute

x′ = x / max(|x|)

Scales by maximum absolute value while preserving sign and zero.

Example: features containing both negative and positive values.
05 • VISUALIZATIONS

Charts, parameters, and when to use them

Histogram

Shows the distribution of one variable by dividing its range into bins.

Use: age distribution • X=age • Bins=20 • KDE=On

KDE overlays a smoothed density estimate. Stat controls count/frequency/probability/percent/density. Multiple controls how Hue groups layer, dodge, stack, or fill.

Box Plot

Summarizes median, quartiles, spread, and possible outliers.

Use: salary by department • X=department • Y=salary

Show Outliers toggles individual fliers. Hue adds another categorical grouping.

Scatter Plot

Examines the relationship between two numeric variables.

Use: X=age • Y=income • Hue=gender • Alpha=0.6

Hue maps color, Size maps marker size, Style maps marker shape, and Alpha controls transparency.

Heatmap

Displays correlations among numeric columns.

r = cov(X,Y) / (sₓsᵧ)

Correlation ranges from −1 to +1. Show Values prints coefficients when the matrix is not excessively large. Color Map changes the visual palette.

Pair Plot

Creates pairwise scatter plots plus univariate distributions for several numeric columns.

Use: age + income + household_size + score

Select 2–8 numeric columns. Hue can separate classes. Corner Layout removes mirrored duplicates to reduce visual load.

Bar Chart

Compares the mean of a numeric Y variable across X categories.

Use: X=district • Y=income • Hue=gender

DataForge pre-aggregates the mean before plotting for faster rendering.

Count Plot

Counts how many records occur in each category.

Use: X=district • Hue=gender

Best for frequency comparisons rather than numeric magnitudes.

Line Chart

Shows progression or trend across an ordered X dimension.

Use: X=month • Y=sales • Hue=region • Marker=Circle

DataForge aggregates repeated X/Hue combinations by mean before plotting.

Violin Plot

Combines a distribution-density shape with summary information.

Use: X=department • Y=salary • Hue=gender

Inner Display can show quartiles, a mini box, points, sticks, or nothing. Split Violins require exactly two Hue groups.

Pie Chart

Shows category shares of a whole.

Use: Category=gender

Best with a small number of categories. For high-cardinality columns DataForge groups smaller categories into Other.

06 • STATISTICS

Statistical tests

Independent T-Test

t = (x̄₁ − x̄₂) / [sₚ √(1/n₁ + 1/n₂)]

Tests whether the means of two numeric samples differ more than expected from sampling variation. The output includes the t-statistic and p-value.

App values: Sample A = income_group_A, Sample B = income_group_B. A small p-value (commonly < 0.05) is evidence against equal means, assuming the test assumptions are appropriate.

Z-Test

One sample: z = (x̄ − μ₀) / SE Two sample: z = (x̄₁ − x̄₂) / SE

One-Sample: compare one sample mean with a hypothesized value. Two-Sample: compare two sample means.

One-sample example: Test Type = One-Sample, Primary Sample = delivery_days, Hypothesized Mean = 5.

Chi-Square Test of Independence

χ² = Σ (O − E)² / E

Checks whether two categorical variables are associated. O is an observed frequency and E is the frequency expected under independence.

App values: First Category = gender, Second Category = product_choice.

One-Way ANOVA

F = MSbetween / MSwithin

Compares three numeric samples in DataForge and tests whether at least one group mean differs from the others.

App values: Sample A = score_group_A, Sample B = score_group_B, Sample C = score_group_C.
07 • MACHINE LEARNING

Models and parameters

Linear Regression

ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ

Predicts a continuous numeric target. Coefficients describe the fitted linear contribution of each feature.

MSE: Σ(y−ŷ)²/n — lower is better. R²: 1 − SSres/SStot — closer to 1 indicates more explained variation.

App values: Target = house_price; encode categorical predictors such as district; Test Data = 20%.

Logistic Regression

p = 1 / (1 + e^(−z))

Classification model that converts a linear score into a probability-like value and learns class boundaries.

App values: Target = churn containing classes such as Yes/No; encode text predictors; Test Data = 20%.

Random Forest

An ensemble of decision trees. Each tree learns from randomized data/features and the forest combines their outputs, reducing dependence on one unstable tree.

Gini = 1 − Σpᵢ²Entropy = −Σpᵢ log₂(pᵢ)

Learning Type: Classification or Regression. Number of Trees: more trees usually improve stability but cost more CPU/time. Criterion: controls how splits are evaluated.

Classification example: Target = approved, Trees = 200, Criterion = Gini. Regression example: Target = sales, Trees = 200, Criterion = Squared Error.

K-Nearest Neighbor (KNN)

d(x,y) = √Σ(xᵢ − yᵢ)²

Finds the K closest training samples. Classification uses neighbor voting; regression averages nearby target values.

Neighbors (K): small K reacts strongly to local data; larger K smooths decisions. Feature scaling is highly recommended because distance is scale-sensitive.

App values: Normalize with Z-Score first; Target = risk_class; Learning Type = Classification; K = 7; Test Data = 20%.

Support Vector Machine (SVM)

decision boundary: w·x + b = 0

SVM seeks a separating boundary with a large margin. Kernel functions allow nonlinear relationships.

KernelLinear for roughly linear boundaries; RBF for flexible nonlinear boundaries; Polynomial for polynomial interactions; Sigmoid for a neural-like transform.
CRegularization trade-off. Larger C penalizes training errors more strongly and can fit more tightly.
GammaFor nonlinear kernels, controls how local each training point's influence is. Scale is usually a sensible default.
App values: Target = fraud, Learning Type = Classification, Kernel = RBF, C = 1, Gamma = Scale, Test Data = 20%.

K-Means Clustering

minimize Σ ||xᵢ − μc(i)||²

Unsupervised algorithm that assigns observations to K clusters around learned centroids. Inertia is the within-cluster squared-distance objective; lower values mean tighter clusters for the same K.

Lloyd: standard general algorithm. Elkan: can be faster on some Euclidean datasets by using triangle-inequality bounds.

App values: Normalize first; Clusters = 4; Algorithm = Lloyd; encode category columns only if they are intentionally part of clustering.

Hierarchical Clustering

Agglomerative clustering starts with individual observations and repeatedly merges the closest groups until the requested cluster count remains.

WardMerges groups to minimize the increase in within-cluster variance. Requires Euclidean distance.
CompleteUses the farthest pair between clusters; tends to create compact groups.
AverageUses average pairwise distance between clusters.
SingleUses the closest pair; can create chain-like clusters.
Distance MetricEuclidean, Manhattan/L1, L2, or Cosine depending on linkage compatibility.
App values: Clusters = 3, Linkage = Ward, Metric = Euclidean.
08 • ENCODING & EVALUATION

Preparing predictors for machine learning

Label Encoder

Red→0, Blue→1, Green→2

Maps categories to integer labels. Compact and useful when an algorithm can tolerate the numeric representation, but the numbers do not automatically mean true ordinal distance.

One-Hot Encoder

color_Red • color_Blue • color_Green

Creates binary indicator columns and avoids artificial category ordering. High-cardinality selections are guarded because thousands of generated columns can exhaust memory.

Test Data (%)

train = 100% − test%

Reserves unseen records for evaluation. A common starting point is 20%, although suitable ratios depend on dataset size and task.

09 • VALIDATION & SAFE FAILURE

What happens when inputs are not valid

DataForge is designed to reject invalid operations with a clear message instead of allowing the user interface to collapse. Important cases include:

  • Unsupported or empty uploads are rejected before analysis.
  • Oversized uploads are stopped by a safety limit before consuming excessive memory.
  • Required parameters remain disabled or are validated before requests are sent.
  • Numeric-only techniques reject text columns with an explanation.
  • Missing values are detected before machine-learning training.
  • One-hot encoding blocks extremely high-cardinality selections that could cause a memory explosion.
  • Pair plots cap the number of selected variables and sample very large datasets for responsive rendering.
  • Bar/count visualizations reject combinations that would create hundreds of unreadable bars.
  • Split violin plots require exactly two Hue groups.
  • Ward hierarchical linkage automatically requires Euclidean distance.
  • Cluster count cannot exceed the number of available observations.
  • Long-running browser requests time out safely and explain what the user can change.
  • Unexpected server errors are caught, logged with a reference ID, and returned as a controlled error response.
Important: No software can guarantee that every possible dataset, third-party library failure, hardware limit, or operating-system condition will never fail. DataForge therefore focuses on predictable validation, bounded operations, and graceful recovery rather than pretending crashes are mathematically impossible.
10 • PRACTICAL RECIPES

Example sequences to try in the app

Clean a survey dataset
  1. Upload survey.xlsx.
  2. Quick Summary → inspect missing values.
  3. Handle Missing → income → Median.
  4. Handle Missing → district → Mode.
  5. Remove Duplicates → Ref number.
  6. Export XLSX.
Explore demographic patterns
  1. Histogram → age → 20 bins.
  2. Count Plot → district → Hue gender.
  3. Bar Chart → district → household_income.
  4. Heatmap → inspect numeric correlations.
Classification experiment
  1. Handle all missing values.
  2. Normalize numeric columns if using KNN/SVM.
  3. Select Logistic Regression or SVM.
  4. Target = approved.
  5. Encode text predictors.
  6. Test Data = 20%.
  7. Run Analysis and inspect Accuracy.
Customer segmentation
  1. Choose useful numeric behavior columns.
  2. Normalize with Z-Score.
  3. K-Means → Clusters = 4 → Lloyd.
  4. Inspect labels, centers, and inertia.
  5. Try another K and compare domain usefulness.