Skip to content

README.md

Latest commit

 

History

History
40 lines (33 loc) · 1.57 KB

README.md

File metadata and controls

40 lines (33 loc) · 1.57 KB

Overview

This project is centered around data mining and parse json. I am using data from propublica, NewsAPI, Opensecrets, Trade.gov, CKan. I make a request to each API create a model from the json response and then create a python object from the parsed Json Response. I will use pandas and numpy to perform calculations on the data and persent the data to the use.

Third Party Libraries

  1. NewsAPI - get news articles
  2. JsonTraverseParser - parse json easier
  3. scrapy - creating spiders
  4. pyfolio - stock risk analysis
  5. requests - REST Calls
  6. gensim - NLP
  7. scapy - NLP
  8. NLTK- NPL toolkit
  9. newspaper - extract articles and information from website
  10. Civic Information - political data
  11. Political - political data
  12. Youtube - videos

Documentation

  1. NewsAPI - https://newsapi.org/docs
  2. UnData - http://data.un.org/ws/
  3. Opensecrets - https://www.opensecrets.org/api/admin/index.php?function=user_api_list
  4. Scrapy - https://docs.scrapy.org/en/latest/
  5. pyfolio - https://quantopian.github.io/pyfolio/
  6. gensim - https://radimrehurek.com/gensim/tutorial.html
  7. scapy - https://spacy.io/api/doc
  8. newspaper - https://github.com/codelucas/newspaper
  9. Civic Information - https://developers.google.com/civic-information/docs/v2/
  10. Youtube - https://developers.google.com/youtube/v3/code_samples/python

Helpful Links

  1. Open Data - https://project-open-data.cio.gov/

Pyenv

source ~/.bashrc set pycharm to use 3.5.2, checking setting in pycharm and make sure it is using pyenv version. check local version of python, which python.