DataFrame
- pandas-like data-frame library
- Column built on typed arrays
- tries to be memory efficient
- extensions to arrays
- great for tabular data
- reads data in various formats: CSV, JSON, array of rows, array of columns, JS object, JS Map
- work in progress
See JSDoc-generated API docs see docs.
For more human-friendly docs keep reading.
Installation
npm install --save dataf
Human-Friendly API
Preliminaries
Run the node REPL.
node
Import the library (make sure it's installed).
const DF = require('dataf')
Toy Datasets (shipped with the library)
DF.dataSets
[ 'alcohol.csv', // alcohol consumption math students
'countries.csv', // geopolitical data for all countries
'diabetes.csv',
'food.csv', // food choices
'got.csv', // game of thrones deaths
'happiness.csv', // world happiness 2017
'iris.csv',
'mushrooms.csv',
'pokemon.csv', // stats for all from all generations
'superheros.csv'
... ]
All have been placed in the public domain.
Load the Iris DataSet
let iris = new DF('iris') // use `let`, you will be re-assigning a lot
NOTE
the lookup of datasets happens by recursive search of each directory in DF.opts.DATASETS
. You can use this and simply df.opts.DATASETS.push(yourDir)
and your dataset will be discoverable. You don't need to specify the extension. .csv
and .json
extensions are appended if not provided (e.g. iris is actually stored in iris.csv
). Dataset files must be in either CSV or JSON formats.
Selecting / Slicing Rows
Head / Tail
iris.head().print()
// .head(20) for the first 20 rows
// .tail() for last rows
# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 1 5.09 3.50 1.39 0.20 Iris-setos...
1 2 4.90 3.00 1.39 0.20 Iris-setos...
2 3 4.69 3.20 1.29 0.20 Iris-setos...
3 4 4.59 3.09 1.50 0.20 Iris-setos...
4 5 5.00 3.59 1.39 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
5B 20B 20B 20B 20B NaN
NOTE the data types next to column names and memory indicators for every column.
Slicing
iris.slice(10, 20).print() // can be .slice(5) for .slice(5, end)
# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 11 5.40 3.70 1.50 0.20 Iris-setos...
1 12 4.80 3.40 1.60 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
2B 8B 8B 8B 8B NaN
NOTE the library will try to compute the width of each column
Getting a Column (Column)
We know that there are 6 columns (try running iris.nCols
). To get all column names run:
iris.colNames.print(100) // make sure it prints all
Column s [Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species]
If you want to extract a column (Column, see the Column API below) from a data frame try:
iris.Species.print(5) // last column
Column s[Iris-setosa, Iris-setosa, Iris-setosa, Iris-setosa, Iris-setosa, ... 145 more]
Here s
stands for STRING. You may also see: f64
, f32
, i32
, i16
, i8
, u32
, u16
and u8
.
NOTE some column names will have spaces or/and will clash with the API and you will have to use iris.col(2)
OR iris.col('SepalWidthCm')
.
Columns can always be referred to by their index OR name.
Selecting columns (Data Frame)
Suppose you only want the first couple of columns:
iris.select(0, 1, -2).print(5) // the 1st, 2nd and the 2nd to last
This show the first 5 rows of the new data frame with only: Id
, SepalLength
and PetalWidth
.
# u8 Id f32 SepalLe... f32 PetalWi...
--- ----- -------------- --------------
0 1 5.09 0.20
1 2 4.90 0.20
2 3 4.69 0.20
3 4 4.59 0.20
4 5 5.00 0.20
... ... (145 more) ...
--- ----- -------------- --------------
150B 600B 600B
If you want to select a range of column: e.g. from the 1st to the 3rd try:
iris.sliceCols(0, 2).print(3)
# u8 Id f32 SepalLe... f32 SepalWi...
--- ----- -------------- --------------
0 1 5.09 3.50
1 2 4.90 3.00
2 3 4.69 3.20
... ... (147 more) ...
--- ----- -------------- --------------
150B 600B 600B
This is the same as:
iris.sliceCols('Id', 'SepalWidthCm').print(3)
Only Numeric Columns (remove string columns)
iris.numeric // all BUT the "Species" column (getter)
Only String Columns (remove numeric columns)
iris.nominal // just the "Species" column (getter)
Removing (Dropping) Columns
If you want to remove the 2nd and the second to last columns:
iris.drop(1, -2).print(3)
# f32 SepalLe... f32 SepalWi... f32 PetalLe... s Species
--- -------------- -------------- -------------- -------------
0 5.09 3.50 1.39 Iris-setos...
1 4.90 3.00 1.39 Iris-setos...
2 4.69 3.20 1.29 Iris-setos...
... ... (147 more) ... ...
--- -------------- -------------- -------------- -------------
600B 600B 600B NaN
NOTE those operations are not in-place meaning dropping produces a new data frame without specified columns.
Selecting Rows
With Specific Value
Signature: iris.where(val, colId, op)
. Where op is one of {"=" (default), ">", "<", ">=", "<="}
.
iris.Species[0]
'Iris-setosa'
iris.where('Iris-setosa', -1) // -1 for last col
// ... DataFrame with subset of rows with just Iris-setosa
Matching Predicate (Test)
Signature: iris.filter(rowTestFunc)
.
Signature: iris.filter(valTestFunc, colId)
.
iris.where('Iris-setosa', -1)
// OR
iris.filter(species => species === 'Iris-setosa', -1)
Accessing Values (preferred way)
iris.val(10, 'Species') // val(rowIdx, colId)
'Iris-setosa'
Accessing Rows
One Row
Accessing a single row:
const row = iris.row(20) // 21st row
[ 21,
5.400000095367432,
3.4000000953674316,
1.7000000476837158,
0.20000000298023224,
'Iris-setosa' ]
Iterating Over Values in a Single Row
const irow = iris.irow(10);
Array.from(irow)
[ 5.400000095367432,
3.700000047683716,
1.5,
0.20000000298023224,
'Iris-setosa' ]
Iterating Over Rows
If you want to iterate over all the rows (not efficient) try:
const rowIt = iris.slice(0, 3).rowsIter // (getter)
for (const r of rowIt) {
console.log(r)
}
// you may also iterate over the dataframe (equivalent method)
for (const r of iris) {
console.log(r)
}
[ 1,
5.099999904632568,
3.5,
1.399999976158142,
0.20000000298023224,
'Iris-setosa' ]
[ 2,
4.900000095367432,
3,
1.399999976158142,
0.20000000298023224,
'Iris-setosa' ]
[ 3,
4.699999809265137,
3.200000047683716,
1.2999999523162842,
0.20000000298023224,
'Iris-setosa' ]
Manipulation
In-Place Modification of Columns
Just assign:
// 2nd col
iris[1] = iris[1].map(s => s >= 5 ? 0 : 1)
// equivalent to:
iris.SepalLengthCm = iris.SepalLengthCm.map(s => s >= 5 ? 0 : 1)
NOTE this might have to be dataset[' Col With Spaces'] = newCol
.
Mapping Columns
Apply function to each element is selected column:
iris.map(-1, label => {
// there is an easier way to do this (see `DataFrame.labelEncode()`)
if (label === 'Iris-versi') {
return 0;
} else if (label === 'Iris-virgi') {
return 1;
} else {
return 2;
}
});
NOTE use iris.map(null, f)
to apply to all columns.
Mapping Shortcuts
null
means it will be applied to all.
.trunc(colId | null)
.floor(colId | null)
.ceil(colId | null)
.round(colId | null)
.abs(colId | null)
.sqrt(colId | null)
.cbrt(colId | null)
.square(colId | null)
.cube(colId | null)
.add(colId | null, n)
.sub(colId | null, n)
.mul(colId | null, n)
.div(colId | null, n)
It's smart enough to know not to apply them to string columns if they don't
make sense (e.g. .abs()
). String columns are ignored.
Rename Columns
iris.rename(0, 'First').rename(-2, 'Second to Last')
// or just
iris.rename(0, 'First', -2, 'Second to Last')
Merging Data Frames
iris.concat(iris) // append all rows (axis 0)
iris.concat(iris, 1) // append all columns (axis 1)
NOTE this library will manage duplicate column names.
iris.concat(iris, 1).colNames
[ 'Id',
'SepalLengthCm',
'SepalWidthCm',
'PetalLengthCm',
'PetalWidthCm',
'Species',
'Id2',
'SepalLengthCm2',
'SepalWidthCm2',
'PetalLengthCm2',
'PetalWidthCm2',
'Species2' ]
Appending a Column
iris.appendCol(iris.Id, 'Id2') // .appendCol(col, colName)
Shuffle, Reverse
iris.shuffle()
iris.reverse()
Both are safe in that the won't modify in place.
Sort
Signature: iris.sort(colId, 'asc' (default) | 'des' )
.
iris.sort('SepalWidthCm') // default is iris.sort(colId, 'asc')
iris.sort('SepalWidthCm', 'des') // descending sort
NOTE
constants such as 'des'
are defined in the constants
module which you can import:
const { DataType, LoggingLevel, PrintingPreset, SortingOrder, What } = require('dataf/constants')
Statistics & Math
Aggregate operations, each is DataFrame -> DataFrame
MATH
.add()
.sub()
.mul()
.div()
STATS
.min()
.max()
.range()
.mean()
.var()
variance.stdev()
standard deviation.median()
.Q3()
.Q1()
.IQR()
inter-quartile range.skewness()
.kurtosis()
.mad()
mean absolute deviation
E.g.:
iris.IQR()
# s column f32 IQR
- ------------- -------
0 Id 75.00
1 SepalLengt... 1.30
2 SepalWidth... 0.50
3 PetalLengt... 3.50
4 PetalWidth... 1.50
- ------------- -------
NaN 20B
Sample (get a random subset of rows)
Signatures:
Signature | Description |
---|---|
`.sample(0.15)` | for random 15% of the dataset |
`iris.sample(30)` | for random 30 sample of the dataset |
`iris.sample(0.5, true)` | with replacement (default) |
`iris.sample(100, false)` | **without** replacement |
Summary
iris.summary() // this will produce a summary data frame with info for every column
# s column s dtype f32 min f32 max f32 range f32 mean f32 stdev
- ------------- ------- ------- ------- --------- -------- ---------
0 Id u8 1.00 150.00 149.00 75.50 43.30
1 SepalLengt... f32 4.30 7.90 3.59 5.84 0.82
2 SepalWidth... f32 2.00 4.40 2.40 3.05 0.43
3 PetalLengt... f32 1.00 6.90 5.90 3.75 1.75
4 PetalWidth... f32 0.10 2.50 2.40 1.19 0.76
5 Species s NaN NaN NaN NaN NaN
- ------------- ------- ------- ------- --------- -------- ---------
NaN NaN 24B 24B 24B 24B 24B
Aggregates
Counts (of unique values)
This is particularly useful for nominal / discrete attributes that take on a
small amount of values. E.g. Gender
is one of {M, F}
or Salary
is one of {Low, Med, High}
.
iris.counts(-1) // for the last column
// iris.ps(-1) // for normalized values
# s Species u8 count
- ------------- --------
0 Iris-setos... 50
1 Iris-versi... 50
2 Iris-virgi... 50
- ------------- --------
NaN 3B
Correlations (A Matrix Operation)
For a correlation of each column with each other column (matrix):
iris.corr(/* `false` to *not* print the first column */)
# s column f64 Id f64 SepalLe... f64 SepalWi... f64 PetalLe... f64 PetalWi...
- ------------- ------ -------------- -------------- -------------- --------------
0 Id 1.00 0.71 -0.39 0.88 0.89
1 SepalLengt... 0.71 1.00 -0.10 0.87 0.81
2 SepalWidth... -0.39 -0.10 1.00 -0.42 -0.35
3 PetalLengt... 0.88 0.87 -0.42 1.00 0.96
4 PetalWidth... 0.89 0.81 -0.35 0.96 1.00
- ------------- ------ -------------- -------------- -------------- --------------
NaN 40B 40B 40B 40B 40B
Other matrix operations:
iris.cov()
iris.dot()
iri.distance()
Pre-Processing
Remove NaN / Infinity / other
To remove all rows that have some value:
// from all cols i.e. remove all rows where any of the value is NaN
iris.removeAll(NaN)
// from 1th and 3rd cols and from col 'PetalLengthCm'
iris.removeAll(NaN, 0, 2, 'PetalLengthCm')
Discretize (Bin)
iris.kBins('SepalLengthCm'); // 5 bins for this column
iris.kBins(null); // 3 bins for all columns
iris.kBins(2) // 3rd (2 idx) col, 3 bins
.col(2) // select ONLY 3rd column (index is 2), which is of type Column
.print(10)
Column u8[2, 1, 2, 1, 2, 2, 2, 2, 1, 1, ... 40 more]
NOTE this is smart enough only to target numeric attributes so string columns will be ignored (no need to run .numeric
).
Feature (Column) Selection
Feature selection (i.e. select best columns, by default uses "var"
-- variance):
Signature: iris.nBest(n, metric)
where metric is one of:
"var"
"stdev"
"mean"
"mad"
"IQR"
"median"
"Q1"
"Q3"
"skewness"
"min"
"range"
"max"
OR a function from Column (one column) to a number (Column -> Num
).
iris.drop('Id') // `Id` column is not useful
.numeric // select all numeric cols
.nBest(2) // best 2 features using variance as score
.print(3) // show first 3 rows
// try: iris.drop('Id').numeric.nBest(2, 'mad').print(3)
# f32 PetalLe... f32 SepalLe...
--- -------------- --------------
0 1.39 5.09
1 1.39 4.90
2 1.29 4.69
... (147 more) ...
--- -------------- --------------
600B 600B
Normalization
Using .nBest()
in this way is naive and you might want to normalize (scale to the same range) the values:
iris.drop('Id') // `Id` column is not useful
.numeric // select all numeric cols
.normalize() // bring them to range [0, 1]
.nBest(2) // best 2 features using variance as score
.print(3)
As you can see you might get different results:
# f32 PetalWi... f32 PetalLe...
--- -------------- --------------
0 0.04 0.06
1 0.04 0.06
2 0.04 0.05
... (147 more) ...
--- -------------- --------------
600B 600B
Label Encoding
It's a bit awkward to constantly have to drop the 'Species'
column because it's a string column...
You can easily convert it to a numeric column:
From:
iris.select(-2, -1).print(48, 52)
# f32 PetalWi... s Species
--- -------------- -------------
... (48 more) ...
48 0.20 Iris-setos...
49 0.20 Iris-setos...
50 1.39 Iris-versi...
51 1.50 Iris-versi...
... (98 more) ...
--- -------------- -------------
600B NaN
To:
iris.select(-2, -1).labelEncode().print(48, 52)
# f32 PetalWi... u8 Species
--- -------------- ----------
... (48 more) ...
48 0.20 0
49 0.20 0
50 1.39 1
51 1.50 1
... (98 more) ...
--- -------------- ----------
600B 150B
By default all string columns will be label encoded (numeric columns will be ignored). You may specify the colIds
e.g. df.labelEncode(0, -3, 'Target')
.
One-Hot Encoding
Signature: iris.oneHot(colId)
// expects the column to be unsigned int
iris.labelEncode('Species')
.oneHot('Species')
.print(48, 52)
# u8 0 u8 1 u8 2
--- ---- --------- ----
... ... (48 more) ...
48 1 0 0
49 1 0 0
50 0 1 0
51 0 1 0
... ... (98 more) ...
--- ---- --------- ----
150B 150B 150B
Clipping (ensuring value is in range)
For demonstration let's make a 1-col data frame:
iris.select(1).print(3)
# f32 SepalLe...
- --------------
0 5.09
1 4.90
2 4.69
- --------------
12B
To clip:
iris.select(1)
.clip(null, 4.88, 5) // null == all cols
.print(3)
# f32 SepalLe...
--- --------------
0 5.00
1 4.90
2 4.88
... (147 more)
--- --------------
600B
Notice that 5.09
got clipped to 5.00
!
Outliers
To remove outliers (outside of Q1 to Q3) run:
iris.dropOutliers() // consider all cols
iris.dropOutliers(0, -2) // consider just 1st and second to last cols
Advanced Human-Friendly API
Data Types
Data Type | String |
---|---|
string | s |
32-bit signed integer | i32 |
16-bit signed integer | i16 |
8-bit signed integer | i8 |
32-bit unsigned integer | u32 |
16-bit unsigned integer | u16 |
8-bit unsigned integer | u8 |
32-bit float (single precision) | f32 |
64-bit float (double precision) | f64 |
If you want to get the data type for all columns try:
iris.dtypes
[ 'u8', 'f32', 'f32', 'f32', 'f32', 's' ] // read-only
Or for a prettier output make a meta data frame with information about the previous data frame!
iris.dtype() // note difference between `iris.dtype()` (method) and `iris.dtypes` (getter)
SIDENOTE .dtype()
is an aggregate! This means it produces a data frame from applying a Column -> *
operation to all columns.
# s column s dtype
- ------------- -------
0 Id u8
1 SepalLengt... f32
2 SepalWidth... f32
3 PetalLengt... f32
4 PetalWidth... f32
5 Species s
- ------------- -------
NaN NaN
You can force-cast columns:
iris.cast(2, 'u8') // passing `null` instead of `2` would run cast on all cols
Down-Casting
You can also run iris.downcast()
and let the library figure out the most efficient data type for each column so that data is not lost.
This is especially useful after truncating (floats are converted to integers).
Default:
# u8 Id f32 SepalLe... f32 SepalWi... f32 PetalLe... f32 PetalWi... s Species
- ----- -------------- -------------- -------------- -------------- -------------
0 1 5.09 3.50 1.39 0.20 Iris-setos...
1 2 4.90 3.00 1.39 0.20 Iris-setos...
2 3 4.69 3.20 1.29 0.20 Iris-setos...
- ----- -------------- -------------- -------------- -------------- -------------
3B 12B 12B 12B 12B NaN
Now see how much memory can be saved:
iris.trunc().downcast().head(3)
# u8 Id u8 SepalLe... u8 SepalWi... u8 PetalLe... u8 PetalWi... s Species
- ----- ------------- ------------- ------------- ------------- -------------
0 1 5 3 1 0 Iris-setos...
1 2 4 3 1 0 Iris-setos...
2 3 4 3 1 0 Iris-setos...
- ----- ------------- ------------- ------------- ------------- -------------
3B 3B 3B 3B 3B NaN
Memory
Although this information is by default printed, you may produce a data frame with information about memory consumption of each column.
iris.memory()
# s column u16 memory
- ------------- ----------
0 Id 150
1 SepalLengt... 600
2 SepalWidth... 600
3 PetalLengt... 600
4 PetalWidth... 600
- ------------- ----------
NaN 10B
NOTE it's not calculated for string columns (notice that "Species" is missing).
To figure out how much your data frame is taking in total try:
iris.memory()
.col(-1)
.add()
2550 // bytes
Copies
Deep Copy
If for some reason you need a deep-copy try (expensive):
iris.clone()
Shallow Copy
Shallow copies are cheap:
iris.copy()
Generalized Row Slicing
Sometimes you may want to get rows from 10th to 20th and e.g. 50th to 65th:
// [F, T],[F, T] // FROM - TO
iris.slice(9, 19, 49, 64)
Generalized Column Slicing
The same applies to column slices:
iris.sliceCols(-3, -2, 0, 2)
# f32 PetalLe... f32 PetalWi... u8 Id f32 SepalLe... f32 SepalWi...
- -------------- -------------- ----- -------------- --------------
0 1.39 0.20 1 5.09 3.50
1 1.39 0.20 2 4.90 3.00
2 1.29 0.20 3 4.69 3.20
- -------------- -------------- ----- -------------- --------------
12B 12B 3B 12B 12B
Exporting
HTML
iris.head(2).toHTML(/* optional file name */)
<table>
<tr>
<th>Id</th>
<th>SepalLengthCm</th>
<th>SepalWidthCm</th>
<th>PetalLengthCm</th>
<th>PetalWidthCm</th>
<th>Species</th>
</tr>
<tr>
<td>1</td>
<td>5.099999904632568</td>
<td>3.5</td>
<td>1.399999976158142</td>
<td>0.20000000298023224</td>
<td>Iris-setosa</td>
</tr>
<tr>
<td>2</td>
<td>4.900000095367432</td>
<td>3</td>
<td>1.399999976158142</td>
<td>0.20000000298023224</td>
<td>Iris-setosa</td>
</tr>
</table>
JSON
iris.head(2).toJSON(/* optional file name */)
{
"Id": [1, 2],
"SepalLengthCm": [5.099999904632568, 4.900000095367432],
"SepalWidthCm": [3.5, 3],
"PetalLengthCm": [1.399999976158142, 1.399999976158142],
"PetalWidthCm": [0.20000000298023224, 0.20000000298023224],
"Species": ["Iris-setosa", "Iris-setosa"]
}
CSV
iris.head(2).toCSV(/* optional file name */)
Id,SepalLengthCm,SepalWidthCm,PetalLengthCm,PetalWidthCm,Species
1,5.099999904632568,3.5,1.399999976158142,0.20000000298023224,Iris-setosa
2,4.900000095367432,3,1.399999976158142,0.20000000298023224,Iris-setosa
SQL Table
iris.head(2).toSQLTableDef('MyIrisTable', /* optional file name */)
CREATE TABLE IF NOT EXISTS MyIrisTable (
Id INT,
SepalLengthCm REAL,
SepalWidthCm REAL,
PetalLengthCm REAL,
PetalWidthCm REAL,
Species TEXT
)
SQL Updates
iris.toSQLUpdates('MyIrisTable', /* optional file name */)
UPDATE MyIrisTable SET Id = 1, SepalLengthCm = 5.099999904632568, SepalWidthCm = 3.5, PetalLengthCm = 1.399999976158142, PetalWidthCm = 0.20000000298023224, Species = Iris-setosa;
UPDATE MyIrisTable SET Id = 2, SepalLengthCm = 4.900000095367432, SepalWidthCm = 3, PetalLengthCm = 1.399999976158142, PetalWidthCm = 0.20000000298023224, Species = Iris-setosa;
UPDATE MyIrisTable SET Id = 3, SepalLengthCm = 4.699999809265137, SepalWidthCm = 3.200000047683716, PetalLengthCm = 1.2999999523162842, PetalWidthCm = 0.20000000298023224, Species = Iris-setosa;
UPDATE MyIrisTable SET Id = 4, SepalLengthCm = 4.599999904632568, SepalWidthCm = 3.0999999046325684, PetalLengthCm = 1.5, PetalWidthCm = 0.20000000298023224, Species = Iris-setosa;
SQL Inserts
iris.head(4).toSQLInserts('MyIrisTable', /* optional file name */)
INSERT INTO MyIrisTable (Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species) VALUES (1, 5.099999904632568, 3.5, 1.399999976158142, 0.20000000298023224, Iris-setosa);
INSERT INTO MyIrisTable (Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species) VALUES (2, 4.900000095367432, 3, 1.399999976158142, 0.20000000298023224, Iris-setosa);
INSERT INTO MyIrisTable (Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species) VALUES (3, 4.699999809265137, 3.200000047683716, 1.2999999523162842, 0.20000000298023224, Iris-setosa);
INSERT INTO MyIrisTable (Id, SepalLengthCm, SepalWidthCm, PetalLengthCm, PetalWidthCm, Species) VALUES (4, 4.599999904632568, 3.0999999046325684, 1.5, 0.20000000298023224, Iris-setosa);
Settings
Option | Default | Sensible Alternatives | Description |
---|---|---|---|
`PRINT_PREC` | 2 | 3, 4, 5, 6, 7, 8 | how many float digits after the radix point to print |
`FLOAT_PREC` | 32 | 64 | - |
`MIN_COL_WIDTH` | 10 | 12, 15, 20 | constrain width of columns when printing |
`HEAD_LEN` | 5 | 7, 10, 20 | by default print this number of rows when running `.head()`, `.tail()` etc. |
To set:
DF.opts.OPTION = VALUE;
More Advanced Examples
Fix Column Names With Spaces
const args = df.colNames
// replace spaces with '_'
.map(c => [c, c.replace(/\s+/, '_')])
// flatten
.reduce((pair1, pair2) => pair1.concat(pair2), []);
df = df.rename(...args)
Matrix of Normalized Differences Between Means of Columns
This would normally take a lot of code:
iris.normalize()
.matrix(
(col1, col2) => Math.abs(col1.mean() - col2.mean()),
true, // show cols
true, // halves the computation time when f(c2, c1) == f(c1, c2)
0) // saves computation on the diagonal, when f(c, c) == id
Save Memory
df = df.labelEncode() // string cols => unsigned int
.kBins(null) // f64, f32, ... => unsigned int
.downcast() // optimize
// see memory
df.memory()
// see dtypes
df.dtype()
// megabytes
B = df.memory() // mem for each col
.add() // add up
.val(0, 1) // get total
MB = B / 1e6
Column Human-Friendly API
TODO
Disclaimer
- I am not a statistician
- Unit tests for
DataFrame
are not done yet - Alpha-stage
- I would not use it in production (yet)
- This isn't supposed to be an exact copy of pandas
- In some places it's not efficient
- Date columns / mixed data types not supported. Every column must be either
numeric OR string. A single
DataFrame
may have a combination of numeric and string columns. - I am a student.
License
MIT