CONTRIBUTING.md

Contributing

Welcome to chdb contributor's guide.

This document focuses on getting any potential contributor familiarized with the development processes, but other kinds of contributions are also appreciated.

If you are new to using git or have never collaborated in a project previously, please have a look at contribution-guide.org. Other resources are also listed in the excellent guide created by FreeCodeCamp ¹.

Please notice, all users and contributors are expected to be open, considerate, reasonable, and respectful. When in doubt, Python Software Foundation's Code of Conduct is a good reference in terms of behavior guidelines.

Issue Reports

If you experience bugs or general issues with chdb, please have a look on the issue tracker. If you don't see anything useful there, please feel free to fire an issue report.

:::{tip} Please don't forget to include the closed issues in your search. Sometimes a solution was already reported, and the problem is considered solved. :::

New issue reports should include information about your programming environment (e.g., operating system, Python version) and steps to reproduce the problem. Please try also to simplify the reproduction steps to a very minimal example that still illustrates the problem you are facing. By removing other factors, you help us to identify the root cause of the issue.

Documentation Improvements

You can help improve chdb docs by making them more readable and coherent, or by adding missing information and correcting mistakes.

chdb documentation uses Sphinx as its main documentation compiler. This means that the docs are kept in the same repository as the project code, and that any documentation update is done in the same way was a code contribution.

    e.g.,  [reStructuredText] or [CommonMark] with [MyST] extensions.

   :::{tip}
      Please notice that the [GitHub web interface] provides a quick way of
      propose changes in `chdb`'s files. While this mechanism can
      be tricky for normal code contributions, it works perfectly fine for
      contributing to the docs, and can be quite handy.

      If you are interested in trying this method out, please navigate to
      the `docs` folder in the source [repository], find which file you
      would like to propose changes and click in the little pencil icon at the
      top, to open [GitHub's code editor]. Once you finish editing the file,
      please write a message in the form at the bottom of the page describing
      which changes have you made and what are the motivations behind them and
      submit your proposal.
   :::

When working on documentation changes in your local machine, you can compile them with:

make docs

A browser will open with the docs site.

Code Contributions

   An architecture description, design principles or at least a summary of the
   main concepts will make it easy for potential contributors to get started
   quickly.

Submit an issue

Before you work on any non-trivial code contribution it's best to first create a report in the issue tracker to start a discussion on the subject. This often provides additional considerations and avoids unnecessary work.

Create an environment

Before you start coding, we recommend creating an isolated virtual environment to avoid any problems with your installed Python packages. This can easily be done via either virtualenv:

virtualenv <PATH TO VENV>
source <PATH TO VENV>/bin/activate

or Miniconda:

conda create -n chdb python=3 six virtualenv pytest pytest-cov
conda activate chdb

Clone the repository

1. Create an user account on GitHub if you do not already have one.

2. Fork the project repository: click on the Fork button near the top of the page. This creates a copy of the code under your account on GitHub.

3. Clone this copy to your local disk:

   git clone git@github.com:YourLogin/chdb.git
   cd chdb
   

Build the project

chdb-core is a c++ library meant to be imported by python.

To build the c++ library, you must have a few prerequisites, amongst others:

• clang: Minimum version can be found in cmake/tools.cmake, search for CLANG_MINIMUM_VERSION
• Rust: Version can be found in rust/vendor.sh, search for TOOLCHAIN
• make
• cmake
• ccache
• patchelf
• yasm

And some python-specific ones:

• pyenv with at least python 3.9 installed (pyenv install 3.9)

Activate your venv & make sure you're running under 3.9:

pyenv shell 3.9
pip install -r requirements-dev.txt

Create the library with:

pyenv shell 3.9

make buildlib

This includes compiling clickhouse itself, which can take upwards of tens of minutes, depending on your processor. The library can be found in buildlib/libchdb.so. A patched version of the library meant to be imported by python is written under the chdb directory; for example in Linux that'll be chdb/_chdb.abi3.so.

To actually use the library, build the wheel:

pyenv shell 3.9

make wheel

# install it locally
pip install dist/chdb_core-*.whl

You can use make test to run all the python unittests:

make test

Implement your changes

1. Create a branch to hold your changes:

   git checkout -b my-feature
   

   and start making changes. Never work on the main branch!

2. Start your work on this branch. Don't forget to add docstrings to new functions, modules and classes, especially if they are part of public APIs.

3. Add yourself to the list of contributors in AUTHORS.rst.

4. When you’re done editing, do:

   git add <MODIFIED FILES>
   git commit
   

   to record your changes in git.

   :::{important} Don't forget to add unit tests and documentation in case your contribution adds an additional feature and is not just a bugfix.

   Moreover, writing a descriptive commit message is highly recommended. In case of doubt, you can check the commit history with:

   git log --graph --decorate --pretty=oneline --abbrev-commit --all
   

   to look for recurring communication patterns. :::

5. Please check that your changes don't break any unit tests with:

   make test
   

Submit your contribution

1. If everything works fine, push your local branch to the remote server with:

   git push -u origin my-feature
   

2. Go to the web page of your fork and click "Create pull request" to send your changes for review.

   
      Find more detailed information in [creating a PR]. You might also want to open
      the PR as a draft first and mark it as ready for review after the feedbacks
      from the continuous integration (CI) system or any required fixes.
   
   

Troubleshooting

The following tips can be used when facing problems to build or test the package:

• Make sure to fetch all the tags from the upstream repository. The command git describe --abbrev=0 --tags should return the version you are expecting. If you are trying to run CI scripts in a fork repository, make sure to push all the tags. You can also try to remove all the egg files or the complete egg folder, i.e., .eggs, as well as the *.egg-info folders in the src folder or potentially in the root of your project.

• Make sure to have a reliable pyenv installation that uses the correct Python version (e.g., 3.9+). When in doubt you can run:

  pyenv --version
  # OR
  which python
  # should point to .pyenv/shims/python
  

• Pytest can drop you in an interactive session in the case an error occurs. In order to do that you need to pass a --pdb option (for example by running pytest -k <NAME OF THE FALLING TEST> --pdb). You can also setup breakpoints manually instead of using the --pdb option.

Maintainer tasks

Releases

   If instead you are using a different/private package index, please update
   the instructions accordingly.

If you are part of the group of maintainers and have correct user permissions on PyPI, the following steps can be used to release a new version for chdb:

1. Make sure all unit tests are successful.
2. Tag the current commit on the main branch with a release tag, e.g., v1.2.3.
3. Push the new tag to the upstream repository, e.g., git push upstream v1.2.3
4. Clean up the dist and build folders with tox -e clean (or rm -rf dist build) to avoid confusion with old builds and Sphinx docs.
5. Run tox -e build and check that the files in dist have the correct version (no .dirty or git hash) according to the git tag. Also check the sizes of the distributions, if they are too big (e.g., > 500KB), unwanted clutter may have been accidentally included.
6. Run tox -e publish -- --repository pypi and check that everything was uploaded to PyPI correctly.

Footnotes

1. Even though, these resources focus on open source projects and communities, the general ideas behind collaborating with other developers to collectively create software are general and can be applied to all sorts of environments, including private companies and proprietary code bases. ↩