Merge branch 'master' of https://github.com/sl-solution/InMemoryDatasets.jl

Author: sl-solution

CHANGELOG.md

@@ -1,3 +1,13 @@
+# Dev
+
+## New features
+
+* `byrow(ds, t::DataType, col)` convert values of `col` to `t`. 
+
+## Fixes
+
+* Fix an issue in `flatten/!` - columns with type `Any`.
+
 # Version 0.7.3
 
 ## New features

docs/src/man/basics.md

@@ -44,7 +44,7 @@ The first line of the output provides the general information about the data set
 The following example shows how to create a data set by providing a range of values.
 
 ```jldoctest
-julia> Dataset(A=1:3, B=5:7, fixed=1)
+julia> Dataset(A = 1:3, B = 5:7, fixed = 1)
 3×3 Dataset
  Row │ A         B         fixed
      │ identity  identity  identity
@@ -99,7 +99,7 @@ julia> Dataset([1 0; 2 0], :auto)
    1 │        1         0
    2 │        2         0
 
-julia> Dataset([[1 ,2], [0, 0]], :auto)
+julia> Dataset([[1, 2], [0, 0]], :auto)
 2×2 Dataset
  Row │ x1        x2
      │ identity  identity
@@ -369,6 +369,91 @@ julia> insertcols!(ds, :var3 => [3.5, 4.6, 32.0])
    3 │        3  val3            32.0
 ```
 
+### Converting the columns' type
+
+To convert the values of a column to another type, user can use the following syntax:
+
+`modify!(ds, col => byrow(T))`
+
+where `ds` is the input data set, `col` is the column which its values' type is going to be converted and `T` is the new type (the `byrow` function is discussed in [Row-wise operations](https://sl-solution.github.io/InMemoryDatasets.jl/stable/man/byrow/), and the `modify!` function is discussed in [Transforming datasets](https://sl-solution.github.io/InMemoryDatasets.jl/stable/man/modify/#Transforming-data-sets)). This functionality must be used in cases where each individual value needed to be converted. For scenarios that the convertion process needs the information of all values in a column, the `byrow` function must be dropped, e.g. `modify!(ds, col => PooledArray)`. Additionally, user may allow `Julia` to find the most suitable type of a column by calling `modify!(ds, col => byrow(identity))`. In the following example we are using `modify!` to correct the type of columns in `ds`.
+
+> Note that in the following example calling `byrow(identity)` on `:y` convert type `Any` to `Integer`. However, note that `Integer` is an abstract type and it will slow down the performance of operations on `ds`. To improve the performance of calculations, user may use `modify!(ds, :y => byrow(Int))` instead.
+
+```jldoctest
+julia> using PooledArrays
+
+julia> ds = Dataset(x = [missing,2,3,4], y = Any[1,missing,-1,true], z = ["a", "bc", "a", missing])
+4×3 Dataset
+ Row │ x         y         z        
+     │ identity  identity  identity 
+     │ Int64?    Any       String?  
+─────┼──────────────────────────────
+   1 │  missing  1         a
+   2 │        2  missing   bc
+   3 │        3  -1        a
+   4 │        4  true      missing  
+
+julia> modify!(ds, :x => byrow(Float64), :y => byrow(identity), :z => PooledArray)
+4×3 Dataset
+ Row │ x          y         z        
+     │ identity   identity  identity 
+     │ Float64?   Integer?  String?  
+─────┼───────────────────────────────
+   1 │ missing           1  a
+   2 │       2.0   missing  bc
+   3 │       3.0        -1  a
+   4 │       4.0      true  missing  
+
+julia> ds[:, :x]
+4-element Vector{Union{Missing, Float64}}:
+  missing
+ 2.0
+ 3.0
+ 4.0
+
+julia> ds[:, :y]
+4-element Vector{Union{Missing, Integer}}:
+    1
+     missing
+   -1
+ true
+
+julia> ds[:, :z]
+4-element PooledVector{Union{Missing, String}, UInt32, Vector{UInt32}}:
+ "a"
+ "bc"
+ "a"
+ missing
+```
+
+To convert the type of multiple columns at once, user may use the boradcasting technique:
+
+```jldoctest
+julia> using PooledArrays
+
+julia> ds = Dataset(x = [missing,2,3,4], y = Any[1,missing,-1,true], z = ["a", "bc", "a", missing])
+4×3 Dataset
+ Row │ x         y         z        
+     │ identity  identity  identity 
+     │ Int64?    Any       String?  
+─────┼──────────────────────────────
+   1 │  missing  1         a
+   2 │        2  missing   bc
+   3 │        3  -1        a
+   4 │        4  true      missing  
+
+julia> modify!(ds, [:x, :y] .=> byrow(Float64)) # note "." in ".=>"
+4×3 Dataset
+ Row │ x          y          z        
+     │ identity   identity   identity 
+     │ Float64?   Float64?   String?  
+─────┼────────────────────────────────
+   1 │ missing          1.0  a
+   2 │       2.0  missing    bc
+   3 │       3.0       -1.0  a
+   4 │       4.0        1.0  missing  
+```
+
 ### Some useful functions
 
 The following functions are very handy when working with a data set, for more information look at the package documentation. Note that functions which end with `!` modify the original data set.

src/byrow/byrow.jl

@@ -194,9 +194,6 @@ byrow(ds::AbstractDataset, ::typeof(join), col::MultiColumnIndex; threads = nrow
 
 byrow(ds::AbstractDataset, ::typeof(mapreduce), cols::MultiColumnIndex = names(ds, Union{Missing, Number}); op = .+, f = identity,  init = _missings(mapreduce(eltype, promote_type, view(_columns(ds),index(ds)[cols])), nrow(ds)), kwargs...) = mapreduce(f, op, eachcol(ds[!, cols]), init = init; kwargs...)
 
-# specific path for converting Any to suitable type
-byrow(ds::AbstractDataset, ::typeof(identity), col::ColumnIndex) = identity.(_columns(ds)[index(ds)[col]])
-
 
 function byrow(ds::AbstractDataset, f::Function, cols::MultiColumnIndex; threads = nrow(ds)>1000)
 	colsidx = multiple_getindex(index(ds), cols)
@@ -223,3 +220,10 @@ function byrow(ds::AbstractDataset, f::Function, col::ColumnIndex; threads = nro
 	end
 	res
 end
+
+
+# specific path for converting Any to suitable type
+byrow(ds::AbstractDataset, ::typeof(identity), col::ColumnIndex) = identity.(_columns(ds)[index(ds)[col]])
+
+# special case for converting the type of a column conveniently 
+byrow(ds::AbstractDataset, f::Type, col::ColumnIndex) = convert(Vector{Union{Missing, f}}, _columns(ds)[index(ds)[col]])

src/byrow/doc.jl

@@ -98,6 +98,8 @@ byrow_docs_text = """
 Perform a row-wise operation specified by `fun` on selected columns `cols`. Generally,
 `fun` can be any function that returns a scalar value for each row.
 
+> User can pass a type as `fun` when `cols` is referring to a single column. In this case, `byrow` simply converts the selected column to vector of type `fun`.
+
 `byrow` is fine tuned for the following operations. To get extra help for each of them search help for `byrow(fun)`, e.g. `?byrow(sum)`;
 
 # Reduction operations
@@ -1234,6 +1236,8 @@ Variant of `byrow(stdze!)` which pass a copy of `ds` and leave `ds` untouched.
 
 Return the result of calling `fun` on each row of `ds` selected by `cols`. The `fun` function must accept one argument which contains the values of each row as a vector of values and return a scalar.
 
+When user passes a type as `fun` and a single column as `cols`,  `byrow` convert the corresponding column to the type specified by `fun`.
+
 For generic functions there are two special cases:
 
 * When `cols` is a single column, `byrow(ds, fun, cols)` acts like `fun.(ds[:, cols])`

src/dataset/modify.jl

@@ -528,7 +528,7 @@ function _modify_f_barrier(ds, msfirst, mssecond, mslast)
             end
         catch e
             if e isa MethodError
-                throw(ArgumentError("There is problem in your `byrow`, make sure that the output of `byrow` is a vector"))
+                throw(ArgumentError("There might be a problem in the `byrow` usage, make sure that the output of `byrow` is a vector"))
             end
             rethrow(e)
         end