As of version 0.1.4 we have not yet implemented the necessary mechanism to speed up the computations taken out by Pywib. Given the need to analyse high volumes of data (as mouse trackers tend to end up in a good bunch of GigaBytes), the following matters should be taken into hands to ensure metrics are computed in a fast and congruent way, leveraging users computers full capacity of computing.
Tasks
- Take into account users GPU memory can be smaller than datasets size
- PySpark includes Apache Spark, which enables real-time, large-scale data processing in a distributed environment using Python.
- Consider practical use before starting implementation
- Consider practical use before implementation, given we could use PySpark instead.
- Check branch features/async
- For some metrics, we validate first the dataframe as to confirm present columns (e.g. dt, dx or dy for speed) but we re-validate this same dataframe again once we segment it, which could slow down the computation process.
- Analyze which ones are required and which ones are not and remove redundant validations.
As of version 0.1.4 we have not yet implemented the necessary mechanism to speed up the computations taken out by Pywib. Given the need to analyse high volumes of data (as mouse trackers tend to end up in a good bunch of GigaBytes), the following matters should be taken into hands to ensure metrics are computed in a fast and congruent way, leveraging users computers full capacity of computing.
Tasks