parallelize arrow memcpy and memset operations - #5

Open

atumanov wants to merge 214 commits into

pcmoritz:masterfrom

atumanov:parallel-arrow-memcpy

Commits on Feb 5, 2016

Initial Commit
jacques-n
committed
d5aa7c4

Commits on Feb 17, 2016

Update readme and add license in root.
jacques-n
committed
cbc56bf
ARROW-1: Initial Arrow Code Commit

StevenMPhillips
authored and
jacques-n
committed
fa5f029
ARROW-3: This patch includes a WIP draft specification document for the physical Arrow memory layout produced over a series of discussions amongst the to-be Arrow committers during late 2015. There…

wesm
authored and
jacques-n
committed
16e44e3
ARROW-4: This provides an partial C++11 implementation of the Apache Arrow data structures along with a cmake-based build system. The codebase generally follows Google C++ style guide, but more cle…

wesm
authored and
jacques-n
committed
23c4b08

Commits on Feb 20, 2016

ARROW-5: Update drill-fmpp-maven-plugin to 1.5.0

lw-lin
authored and
jacques-n
committed
7e76e3a
ARROW-5: Correct Apache Maven repo for maven plugin use
jacques-n
committed
e9cc8ce
ARROW-9: Replace straggler references to Drill

lw-lin
authored and
jacques-n
committed
e6905ef

Commits on Mar 1, 2016

ARROW-8: Add .travis.yml and test script for Arrow C++. OS X build fixes
wesm
committed
a385622

Commits on Mar 3, 2016

Commits on Mar 4, 2016

ARROW-15: Fix a naming typo for memory.AllocationManager.AllocationOutcome

lw-lin
authored and
wesm
committed
307977e
ARROW-10: Fix mismatch of javadoc names and method parameters

lw-lin
authored and
wesm
committed
0c95d3c
ARROW-26: Add instructions for enabling Arrow C++ Parquet adapter build
wesm
committed
3b777c7
ARROW-23: Add a logical Column data structure
wesm
committed
9c2b954

Commits on Mar 5, 2016

ARROW-24: C++: Implement a logical Table container type
wesm
committed
612fbc7

Commits on Mar 7, 2016

Commits on Mar 8, 2016

Commits on Mar 9, 2016

ARROW-17: set some vector fields to package level access for Drill compatibility
StevenMPhillips
committed
1650026
ARROW-18: Fix decimal precision and scale in MapWriters
StevenMPhillips
committed
243ed4e
ARROW-51: Add simple ValueVector tests
StevenMPhillips
committed
31def7d
ARROW-46: ListVector should initialize bits in allocateNew

minji-kim
authored and
StevenMPhillips
committed
e822ea7
ARROW-42: Add Python tests to Travis CI build
wesm
committed
8367527
ARROW-54: [Python] Rename package to "pyarrow"
wesm
committed
6fdcd49

Commits on Mar 16, 2016

ARROW-55: [Python] Fix unit tests in 2.7

danrobinson
authored and
wesm
committed
883c62b
ARROW-64: Add zsh support to C++ build scripts

xhochy
authored and
wesm
committed
5881aac

Commits on Mar 17, 2016

ARROW-68: Better error handling for not fully setup systems

emkornfield
authored and
wesm
committed
c996610

Commits on Mar 21, 2016

ARROW-73: Support older CMake versions

xhochy
authored and
wesm
committed
3a99f39
ARROW-72: Search for alternative parquet-cpp header

xhochy
authored and
wesm
committed
016b92b

Commits on Mar 22, 2016

ARROW-28: Adding google's benchmark library to the toolchain

emkornfield
authored and
wesm
committed
4ec034b
ARROW-75: Fix handling of empty strings

danrobinson
authored and
wesm
committed
093f9bd

Commits on Mar 23, 2016

Commits on Mar 24, 2016

ARROW-77: [C++] Conform bitmap interpretation to ARROW-62; 1 for nulls, 0 for non-nulls
wesm
committed
fbbee3d

Commits on Mar 25, 2016

Commits on Mar 27, 2016

ARROW-22: [C++] Convert flat Parquet schemas to Arrow schemas
wesm
committed
d3cb6b4
ARROW-63: [C++] Enable ctest to work on systems with Python 3 as the default Python
wesm
committed
d6d53b2
ARROW-65: Be less restrictive on PYTHON_LIBRARY search paths

xhochy
authored and
wesm
committed
0171877

Commits on Mar 28, 2016

ARROW-30: [Python] Routines for converting between arrow::Array/Table and pandas.DataFrame
wesm
committed
1fd0668
ARROW-80: Handle len call for pre-init arrays

xhochy
authored and
wesm
committed
ecadd0b
ARROW-79: [Python] Add benchmarks

xhochy
authored and
wesm
committed
80ec2c1
ARROW-88: [C++] Refactor usages of parquet_cpp namespace
wesm
committed
df7726d
ARROW-83: [C++] Add basic test infrastructure for DecimalType

xhochy
authored and
wesm
committed
38897ee
ARROW-87: [C++] Add all four possible ways to encode Decimals in Parquet to schema conversion

xhochy
authored and
wesm
committed
2d8627c

Commits on Mar 31, 2016

ARROW-93: Fix builds when using XCode 7.3

danrobinson
authored and
wesm
committed
5a68f8d

Commits on Apr 1, 2016

ARROW-89: [Python] Add benchmarks for Arrow<->Pandas conversion

xhochy
authored and
wesm
committed
b3ebce1
ARROW-49: [Python] Add Column and Table wrapper interface

xhochy
authored and
wesm
committed
6d31d59
ARROW-90: [C++] Check for SIMD instruction set support

xhochy
authored and
wesm
committed
79fddd1

Commits on Apr 2, 2016

ARROW-71: [C++] Add clang-tidy and clang-format to the the tool chain.

emkornfield
authored and
wesm
committed
5d12999

Commits on Apr 3, 2016

ARROW-86: [Python] Implement zero-copy Arrow-to-Pandas conversion

xhochy
authored and
wesm
committed
9d88a50

Commits on Apr 11, 2016

ARROW-85: memcmp can be avoided in Equal when comparing with the same …

Kai Zheng
authored and
wesm
committed
7b2153b

Commits on Apr 14, 2016

ARROW-94: [Format] Expand list example to clarify null vs empty list

emkornfield
authored and
wesm
committed
37f7271

Commits on Apr 17, 2016

ARROW-103: Add files to gitignore

danrobinson
authored and
wesm
committed
5843e68

Commits on Apr 18, 2016

ARROW-82: Initial IPC support for ListArray

emkornfield
authored and
wesm
committed
0b472d8

Commits on Apr 23, 2016

ARROW-100: [C++] Computing RowBatch size

pcmoritz
authored and
wesm
committed
a541644

Commits on Apr 30, 2016

ARROW-104: [FORMAT] Add alignment and padding requirements + union clarification

emkornfield
authored and
wesm
committed
56514d9

Commits on May 1, 2016

ARROW-92: Arrow to Parquet Schema conversion

xhochy
authored and
wesm
committed
355f7c9

Commits on May 4, 2016

ARROW-188: Add numpy as install requirement

xhochy
authored and
wesm
committed
ad3d01d

Commits on May 9, 2016

ARROW-190: Python: Provide installable sdist builds

xhochy
authored and
wesm
committed
3302257
ARROW-194: C++: Allow read-only memory mapped source

jihoonson
authored and
wesm
committed
c9ffe54
ARROW-193: typos "int his" fix to "in this"

lfzCarlosC
authored and
wesm
committed
1f04f7f

Commits on May 10, 2016

ARROW-91: Basic Parquet read support

xhochy
authored and
wesm
committed
4bd13b8

Commits on May 15, 2016

ARROW-197: Working first draft of a conda recipe for pyarrow
wesm
committed
68b80a8
ARROW-199: [C++] Refine third party dependency

zhangh43
authored and
wesm
committed
6968ec0

Commits on May 17, 2016

ARROW-185: Make padding and alignment for all buffers be 64 bytes

emkornfield
authored and
wesm
committed
9c59158

Commits on May 18, 2016

ARROW-204: Add Travis CI builds that post conda artifacts for Linux and OS X
wesm
committed
978de1a
ARROW-201: [C++] Initial ParquetWriter implementation

xhochy
authored and
wesm
committed
e0fb369

Commits on May 19, 2016

Make BaseValueVector#MAX_ALLOCATION_SIZE configurable

laurentgo
authored and
StevenMPhillips
committed
c0985a4
Fix BaseAllocator.java NPE when assertions are disabled

laurentgo
authored and
StevenMPhillips
committed
e316b3f

Commits on May 20, 2016

Add java support to Travis CI

laurentgo
authored and
StevenMPhillips
committed
7035467

Commits on May 23, 2016

ARROW-206: Expose a C++ api to compare ranges of slots between two arrays

emkornfield
authored and
wesm
committed
cd1d770

Commits on Jun 1, 2016

[Doc] Update Layout.md

ebegoli
authored and
wesm
committed
c8b8078

Commits on Jun 3, 2016

ARROW-209: [C++] Triage builds due to unavailable LLVM apt repo
wesm
committed
6574095

Commits on Jun 7, 2016

ARROW-211: [Format] Fixed typos in layout examples

Smyatkin-Maxim
authored and
wesm
committed
ce2fe7a
ARROW-60: [C++] Struct type builder API

fengguangyuan
authored and
wesm
committed
9ce13a0

Commits on Jun 8, 2016

ARROW-200: [C++/Python] Return error status on string initialization failure

emkornfield
authored and
wesm
committed
bc6c4c8
ARROW-212: Change contract of PrimitiveArray to reflect its abstractness

emkornfield
authored and
wesm
committed
8197f24

Commits on Jun 10, 2016

ARROW-203: Python: Basic filename based Parquet read/write

xhochy
authored and
wesm
committed
ec66ddd

Commits on Jun 15, 2016

ARROW-217: Fix Travis w.r.t conda 4.1.0 changes

xhochy
authored and
wesm
committed
b4e0e93

Commits on Jun 16, 2016

ARROW-218: Add optional API token authentication option to PR merge tool
wesm
committed
790d541
ARROW-210: Cleanup of the string related types in C++ code base

emkornfield
authored and
wesm
committed
27edd25

Commits on Jun 17, 2016

ARROW-219: Preserve CMAKE_CXX_FLAGS, fix compiler warnings
wesm
committed
a3e3849

Commits on Jun 21, 2016

ARROW-223: Do not link against libpython

xhochy
authored and
wesm
committed
f7ade7b

Commits on Jun 24, 2016

ARROW-222: Prototyping an IO interface for Arrow, with initial HDFS target
wesm
committed
ef90830

Commits on Jun 28, 2016

ARROW-215: Support other integer types and strings in Parquet I/O
xhochy
committed
2f52cf4

Commits on Jul 1, 2016

ARROW-234: Build libhdfs IO extension in conda artifacts
wesm
committed
fab4c82

Commits on Jul 10, 2016

ARROW-233: Add visibility macros, add static build option
wesm
committed
77598fa

Commits on Jul 12, 2016

ARROW-237: Implement parquet-cpp's abstract IO interfaces for memory allocation and file reading
wesm
committed
ff6132f

Commits on Jul 13, 2016

ARROW-106: [C++] Add IPC to binary/string types

emkornfield
authored and
wesm
committed
62390d8

Commits on Jul 18, 2016

ARROW-238: Change InternalMemoryPool::Free() to return Status::Invalid when ther…

jihoonson
authored and
wesm
committed
55bfa83
ARROW-236: Bridging IO interfaces under the hood in pyarrow
wesm
committed
59e5f98

Commits on Jul 19, 2016

ARROW-241: Add missing implementation for splitAndTransfer in UnionVector
StevenMPhillips
committed
a2fb756

Commits on Aug 1, 2016

ARROW-244: Some global APIs of IPC module should be visible to the outside

jihoonson
authored and
wesm
committed
dc79ceb
ARROW-240: Provide more detailed installation instructions for pyarrow. Closes

MechCoder
authored and
wesm
committed
356d015
ARROW-101: Fix java compiler warnings

laurentgo
authored and
wesm
committed
3a2dfba

Commits on Aug 3, 2016

ARROW-246: [Java] UnionVector doesn't call allocateNew() when creating it's vectorType

adeneche
authored and
AbdelHakim Deneche
committed
56835c3

Commits on Aug 4, 2016

ARROW-247: Missing explicit destructor in RowBatchReader causes an incomplete type error

jihoonson
authored and
wesm
committed
5df7d4d

Commits on Aug 5, 2016

ARROW-250: Fix for ARROW-246 may cause memory leaks
adeneche
committed
34e7f48

Commits on Aug 12, 2016

Commits on Aug 14, 2016

ARROW-260: Fix flaky oversized tests

jihoonson
authored and
jacques-n
committed
e8724f8

Commits on Aug 15, 2016

ARROW-245: add endianness to RecordBatch
julienledem
committed
689cd27

Commits on Aug 16, 2016

ARROW-251: Expose APIs for getting code and message of the status

jihoonson
authored and
wesm
committed
268e108
ARROW-107: [C++] Implement IPC for structs

emkornfield
authored and
wesm
committed
246a126

Commits on Aug 18, 2016

ARROW-259: Use Flatbuffer Field type instead of MaterializedField
StevenMPhillips
committed
e7e399d
Revert version to 0.1-SNAPSHOT
StevenMPhillips
committed
fd2e524

Commits on Aug 19, 2016

ARROW-265: Pad negative decimal values with1
StevenMPhillips
committed
282fcac

Commits on Aug 20, 2016

Commits on Aug 22, 2016

ARROW-269: Include typeVector buffers UnionVector.getBuffers()
StevenMPhillips
committed
ec51d56

Commits on Aug 26, 2016

ARROW-264: File format
julienledem
committed
803afeb

Commits on Aug 28, 2016

ARROW-262: Start metadata specification document
wesm
committed
907cc5a
ARROW-271: Update Field structure to be more explicit

julienledem
authored and
wesm
committed
e081a4c
ARROW-242: Support Timestamp Data Type

xhochy
authored and
wesm
committed
0a411fd

Commits on Aug 29, 2016

ARROW-279: rename vector module to arrow-vector
julienledem
committed
e197b2d

Commits on Sep 6, 2016

ARROW-274: Add NullableMapVector to support nullable maps
julienledem
committed
2d8ec78

Commits on Sep 7, 2016

ARROW-284: Disable arrow_parquet module in Travis CI to triage builds
wesm
committed
637584b
ARROW-283: [C++] Account for upstream changes in parquet-cpp
wesm
committed
214b861
ARROW-278: [Format] Rename Tuple to Struct_ in flatbuffers IDL

wesm
authored and
julienledem
committed
270ab4e

Commits on Sep 8, 2016

ARROW-285: Optional flatc download

laurentgo
authored and
julienledem
committed
52089d6

Commits on Sep 9, 2016

ARROW-286: Build thirdparty dependencies in parallel

xhochy
authored and
wesm
committed
a5f2861
ARROW-256: [Format] Add a version number to the IPC/RPC metadata
wesm
committed
077c72b

Commits on Sep 13, 2016

ARROW-289: Install test-util.h

xhochy
authored and
wesm
committed
6b8abb4
ARROW-287: Make nullable vectors use a BitVecor instead of UInt1Vector for bits
julienledem
committed
6f99156

Commits on Sep 16, 2016

ARROW-292: [Java] Upgrade Netty to 4.0.41
adeneche
committed
3487c2f
ARROW-290: Specialize alloc() in ArrowBuf
julienledem
committed
17e90e1

Commits on Sep 18, 2016

ARROW-280: [C++] Refactor IPC / memory map IO to use common arrow_io interfaces. Create arrow_ipc leaf library
wesm
committed
559b865

Commits on Sep 19, 2016

ARROW-297: Fix Arrow pom for release
julienledem
committed
5f1556c

Commits on Sep 21, 2016

ARROW-298: create release scripts
julienledem
committed
5358328
ARROW-299: Use absolute namespace in macros

xhochy
authored and
wesm
committed
430bd95
ARROW-267: [C++] Implement file format layout for IPC/RPC
wesm
committed
7e39747

Commits on Sep 25, 2016

ARROW-296: [Python / C++] Remove arrow::parquet, make pyarrow link against parquet_arrow
wesm
committed
32fd692

Commits on Sep 27, 2016

Commits on Sep 28, 2016

ARROW-308: UnionListWriter.setPosition() should not call startList()
adeneche
committed
bd195e3

Commits on Sep 29, 2016

ARROW-306: Add option to pass cmake arguments via environment variable

xhochy
authored and
wesm
committed
bf30235
ARROW-305: Add compression and use_dictionary options to Parquet

xhochy
authored and
wesm
committed
30f6083

Commits on Sep 30, 2016

ARROW-309: Types.getMinorTypeForArrowType() does not work for Union type
julienledem
committed
391ab64

Commits on Oct 3, 2016

ARROW-314: JSONScalar is unnecessary and unused
julienledem
committed
c7b0480
ARROW-301: Add user field metadata to IPC schemas
julienledem
committed
c3930a0

Commits on Oct 4, 2016

ARROW-302: [C++/Python] Implement C++ IO interfaces for interacting with Python file and bytes objects
wesm
committed
c7e6a07
ARROW-313: Build on any version of XCode

Christopher C. Aycock
authored and
julienledem
committed
c3cfa3d
ARROW-315: finalize timestamp
julienledem
committed
7fb4d24

Commits on Oct 5, 2016

ARROW-318: Revise python/README.md given recent changes in codebase

wesm
authored and
julienledem
committed
dd1b95b
ARROW-321: fix arrow licenses
julienledem
committed
04cf874

Commits on Oct 6, 2016

ARROW-320: ComplexCopier.copy(FieldReader, FieldWriter) should not st…

adeneche
authored and
julienledem
committed
f1a4bd1

Commits on Oct 7, 2016

Commits on Oct 8, 2016

Commits on Oct 11, 2016

ARROW-312: Read and write Arrow IPC file format from Python

wesm
authored and
julienledem
committed
a9747ce
ARROW-112: Changed constexprs to kValue naming.

leifwalsh
authored and
wesm
committed
fb799bc
ARROW-326: Include scale and precision when materializing decimal writer
StevenMPhillips
committed
8c8d341
ARROW-189: Build 3rd party with ExternalProject.

leifwalsh
authored and
wesm
committed
994aa5a

Commits on Oct 12, 2016

Commits on Oct 16, 2016

ARROW-191: Python: Provide infrastructure for manylinux1 wheels

xhochy
authored and
wesm
committed
4ecf327
ARROW-336: Run Apache Rat in Travis builds

xhochy
authored and
wesm
committed
8520061
ARROW-97: API documentation via sphinx-apidoc

xhochy
authored and
wesm
committed
8e8b17f

Commits on Oct 17, 2016

ARROW-261: Refactor String/Binary code paths to reflect unnested (non-list-based) structure
wesm
committed
732a205

Commits on Oct 18, 2016

ARROW-317: Add Slice, Copy methods to Buffer
wesm
committed
676c32c
ARROW-327: [Python] Remove conda builds from Travis CI setup
wesm
committed
e2c0a18
ARROW-334: [Python] Remove INSTALL_RPATH_USE_LINK_PATH
wesm
committed
446ec9b

Commits on Oct 21, 2016

ARROW-342: Set Python version on release

xhochy
authored and
wesm
committed
2f84493

Commits on Oct 26, 2016

ARROW-337: UnionListWriter.list() is doing more than it should, this …

adeneche
authored and
julienledem
committed
3d2e4df

Commits on Oct 29, 2016

ARROW-350: Added Kerberos to HDFS client

Christopher C. Aycock
authored and
wesm
committed
6178bf7
ARROW-339: Python 3 compatibility in merge_arrow_pr.py

wesm
authored and
xhochy
committed
da24c1a
ARROW-354: Fix comparison of arrays of empty strings

xhochy
authored and
wesm
committed
d946e79

Commits on Oct 30, 2016

ARROW-349: Add six as a requirement

hoffmann
authored and
xhochy
committed
772bc6e

Commits on Nov 1, 2016

ARROW-339: [Dev] Lingering Python 3 fixes
wesm
committed
ca088dd
ARROW-348: [Python] Add build-type command line option to setup.py, build CMake extensions in a build type subdirectory
wesm
committed
d414875
ARROW-355: Add tests for serialising arrays of empty strings to Parquet

xhochy
authored and
wesm
committed
c7db80e

Commits on Nov 2, 2016

ARROW-358: Add explicit environment variable to locate libhdfs in one's environment

wesm
authored and
xhochy
committed
e70d97d
ARROW-359: Document ARROW_LIBHDFS_DIR

Christopher C. Aycock
authored and
wesm
committed
2a059bd
ARROW-357: Use a single RowGroup for Parquet files as default.

xhochy
authored and
wesm
committed
17c9ae7

Commits on Nov 3, 2016

ARROW-323: [Python] Opt-in to pyarrow.parquet extension rather than attempting and failing silently
wesm
committed
25e0106

Commits on Nov 6, 2016

ARROW-368: Added note for LD_LIBRARY_PATH in Python README

BryanCutler
authored and
xhochy
committed
e8bc1fe
ARROW-361: Python: Support reading a column-selection from Parquet files

xhochy
authored and
wesm
committed
121e826

Commits on Nov 7, 2016

ARROW-362: Fix memory leak in zero-copy arrow to NumPy/pandas conversion
wesm
committed
79344b3

Commits on Nov 8, 2016

ARROW-312: [Java] IPC file round trip tool for integration testing

julienledem
authored and
wesm
committed
6996c17

Commits on Nov 9, 2016

ARROW-372: json vector serialization format
julienledem
committed
4fa7ac4

Commits on Nov 11, 2016

ARROW-356: Add documentation about reading Parquet

xhochy
authored and
wesm
committed
7f048a4
ARROW-375: Fix unicode Python 3 issue in columns argument of parquet.read_table
wesm
committed
48f9780

Commits on Nov 16, 2016

ARROW-371: Handle pandas-nullable types correctly

xhochy
authored and
wesm
committed
78288b5

Commits on Nov 18, 2016

ARROW-367: converter json <=> Arrow file format for Integration tests

julienledem
authored and
wesm
committed
8417096
ARROW-373: [C++] JSON serialization format for testing
wesm
committed
ed6ec3b

Commits on Nov 19, 2016

implement dense unions
pcmoritz
committed
58bd7be

Commits on Nov 20, 2016

Build arrow_io and arrow_ipc as static libraries.
robertnishihara
committed
c88bd70

Commits on Mar 5, 2017

builder: speed up bitsetting for large length
atumanov
committed
bdae7a2
Merge pull request #3 from atumanov/fast-bitsetting
pcmoritz
authored
ae9c5d9

Commits on Mar 8, 2017

Merge pull request #1 from pcmoritz/static
pcmoritz
authored
9caa1d1
upgrade flatbuffers
pcmoritz
committed
90902e9
Merge pull request #4 from pcmoritz/upgrade-flatbuf
pcmoritz
authored
a4a5526

Commits on Mar 9, 2017

Commits on Mar 13, 2017

fully switch arrow memcpy parallelization to c++ threads
atumanov
committed
a2aeb01
parallelize arrow memset: add to memory util
atumanov
committed
59fe77a

Commits on Mar 14, 2017

arrow: adding reusable threadpool for parallel memcpy+memset
atumanov
committed
a61194a