MUT
Market Understanding Tool
About
This project is intended to make a pipeline of data analysis about opportunities for data science career announced at Indeed. However, this pipeline can classify job opportunities of whenever sector, beyond data science.
This pipeline generates a .html file with:
- Clusters 2D Graph
- Clusters Keywords Ranking
- TF-IDF Ranking
Check the "Brazillian Data Science Jobs Market: A Deep Analysis" on the web!
Project Details
Folders
| Folder | Description |
|---|---|
| db/ | Folder where your Scrapy database will be saved |
| output/ | Folder where your graphs and results will be saved |
Files
| ARGS | USAGE |
|---|---|
| [db-title] | It is your Scrapy database title (e. g., datascience_db) |
| [urls-file] | It is your Indeed URL filename (take a look at sample.urls) |
| [toxicwords-file] | It is the filename of list of words for not use in the analysis (take a look at sample.toxicwords) |
| [num-clusters] | Number of clusters to identify, in a range (e. g., 2-8) or single (e. g., 8) |
Requirements
Paraphrasing The Beatles:
" All you need is docker
Install
1. Clone this repo 🍕
git clone https://github.com/HelioNeves/mut.git
cd /mut2. Basic building 🔧
docker build . -t mutRunning this awesome docker image
1. Load ubuntu layer 🌈
docker run -ti --name MUT-env mut /bin/bash2. Once inside ubuntu, run pipeline python scripts 🐍
Scrapy
python3 scraper.py [db-title] [urls-file]Analytics app
python3 app.py [db-title] [toxicwords-file] [num-clusters]