Developing a scraper server with Python and ElasticSearch

The requirements

Looking for a scraper

pip install selenium flask elasticsearch
{
“ORGANIZATIONS”: [],
“BRIEF_PREVIOUS”: [{}],
“TITLE”: “Some title..”,
“COURSES”: [],
“EXPERIENCE”: [{
“DESCRIPTION”: “Software projects architect...”,
“IMG”: null,
“COMPANY_URL”: “",
“COMPANY”: “CC”,
“DATE”: “July 2012 \u2013 Present (4 years)”,
“TITLE_URL”: “"
},
{
“DESCRIPTION”: “”,
“IMG”: null,
“TITLE”: “Software Engineer”,
“COMPANY_URL”: “",
“COMPANY”: “company..”,
“DATE”: “April 2011 \u2013 July 2012 (1 year 4 months)”,
“TITLE_URL”: “"
}],
“GROUPS”: [{...

REST APIs with Python

from flask import Flask
app = Flask(__name__)
# Publish a REST service accessible from "localhost:5000/"
@app.route("/")
def hello():
return "Hello World!"

if __name__ == "__main__":
app.run()

The Datastore

bin\kibana plugin --install elastic/sense

Scoring the profiles

'query': {
'function_score': {
'functions': [
{
'field_value_factor': {
'field': 'RECOMMENDATIONS_NUMBER',
'factor': 1.5
}
}
],
'query': {
'match': {
'EDUCATION.NAME': 'degree university college academy'
}
},
'score_mode': 'avg'
}
}

Conclusion

--

--

--

R&D Group Manager @ Check Point | Software Engineer

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Regev Golan

Regev Golan

R&D Group Manager @ Check Point | Software Engineer

More from Medium

OpenID connect using Python Flask

Connexion: production-grade REST API in 150 lines of Python

How to cache a method of an unhashable type in Python?

Full Installation Guide of MySQL 8.0 on Ubuntu Machine