Project Introduction

This is one-year NLP project, the case study from master study together with another four team members from SRH Heidelberg University and PwC in Germany. The contribution of the coding divided into team and individual works, the purpose of the project is to evaluate the risk level and analysis the news based on NLP technology during the certain period of time.

Entire Data Pipeline

Entire Pipeline:

1. Data crawling for English news
2. Data preprocessing and Doc2Vec
3. PCA
4. K-means clustering and elbow method
5. Aggolomerative clustering [chosen]
6. Danse based algorithm: DBSCAN clustering [chosen]
7. Danse based algorithm: Optics
8. Top2Vec for topic detection
9. Classification: Native Bayes, XgBoost
10. Consolidate to one unioned dataset and save into database
11. Connect database to visulisation