How to make Python Flask-RESTful API with Selenium Scraping data and deploy in Heroku

What is the API endpoint?

The API endpoint is the address to access your API, in this case, localhost:5000.

What is Flask-RESTful API?

Flask-RESTful API is a framework that allows us to create REST APIs with Python. It offers us a bunch of helper methods and classes that streamline its usage into something pretty simple.

from flask import Flask, jsonify, request, redirect, send_file, escape, json, Responseapp = Flask(__name__)@app.route('/')
@cross_origin(origin='localhost', headers=['Content- Type', 'Authorization'])
def hello():
name = request.args.get("name", "World")
return f'Hello, {escape(name)}!'
def main():
app.run(host='127.0.0.1', debug=True, port=8080, threaded=True)
if __name__ == '__main__':
main()

What is Python Selenium?

Selenium is a Python library for browser automation. It can be used in order to write test cases that check the application’s behavior across multiple browsers and OSes, without having specifically these browsers or OSs installed locally on your computer. You can find more information about it at https://docs.python-selenium.org/, including installation instructions here: https://docs.python-selenium.org/installation/. We will use Selenium in order to scrape data

By default, Flask does not include features for database abstraction, form validation, or others. Instead, it uses extensions to add functionality as if the features were part of Flask itself. Extensions provide options for databases as well as forms. It can also handle uploads and certain authentication technologies

Selenium will help you to return the latest data you need.

We will use Selenium in order to scrape data. By default, Flask does not include features for database abstraction, form validation, or others. Instead, it uses extensions to add functionality as if the features were part of Flask itself. Extensions provide options for databases as well as forms. It can also handle uploads and certain authentication technologies

from selenium import webdriver
import time
def getAIWriting(input):
result = ""
try:
sleep = 5
username = os.environ.get('username')
password = os.environ.get('password')

url = 'https://app.conversion.ai/templates'
# options = Options()
# options.add_argument(
# "user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36")
# options.add_argument("--headless")
# browser = webdriver.Chrome(options=options)
#
browser.get(url)
time.sleep(2.5)
# login
email_text = browser.find_element_by_id('email')
email_text.send_keys(username)

pass_text = browser.find_element_by_id('password')
pass_text.send_keys(password)
submit_xpath: str = '//button[@type="submit"]'
button_xpath: str = '//button[@type="button"]'

login_btns = browser.find_elements_by_xpath(submit_xpath)

for login_btn in login_btns:
if (login_btn.text == "Sign in"):
login_btn.click()

time.sleep(2.5)

# select template

bio_url = 'https://app.conversion.ai/templates/view/2b38fa60-e113-4e82-9c85-0f9e179d80b3'

TOV = 'Witty'
POV = 'First Person'

browser.get(bio_url)
time.sleep(2.5)

# input
# clean input
generate_btns = browser.find_elements_by_xpath(button_xpath)

for clean_input_btn in generate_btns:
if (clean_input_btn.text == "Clear inputs"):
clean_input_btn.click()

time.sleep(0.3)
personalInformation = browser.find_element_by_id('personalInformation')
personalInformation.send_keys(input)
time.sleep(0.3)
tone = browser.find_element_by_id('tone')
tone.send_keys(TOV)
time.sleep(0.3)
pointOfView = browser.find_element_by_id('pointOfView')
pointOfView.send_keys(POV)
time.sleep(0.3)
# generate
generate_btns = browser.find_elements_by_xpath(submit_xpath)

for generate_btn in generate_btns:
if (generate_btn.text == "Generate"):
generate_btn.click()
time.sleep(sleep)
time.sleep(4)
# time.sleep(sleep)
# output
outputs = browser.find_elements_by_css_selector('pre')

for output in outputs:
print(output.text)
result += output.text + '\n\n'

browser.close()
except Exception as e:
browser.quit()
print("Exception occurred:{}".format(e))

return result

What is Heroku?

Heroku is a cloud Platform-as-a-Service (PaaS) written in Ruby. Heroku allows developers to build, run and manage web applications entirely in the cloud. We will use Heroku to deploy our Python Flask API app in free-tier.

Prepare to deploy to Heroku

Generate: requirement.txt

pip freeze > requirements.txt

Install gunicorn as a web server and create Profile

web: gunicorn main:app

How to deploy in Heroku?

You will need to provide a Git repository and a name for the application. Once you have done this, you can push your application to Heroku.

I have used GitHub to deploy in this case.

P.S.: 30-second timeout limit for Heroku server

Conclusion

In this article, you learned how to create a Python Flask REST API with Selenium Scraping data and deploy it in Heroku. You also learned about the different pieces of the API framework that will allow for simpler access to your API. If you’re looking for help building out an application like this one, we would love to work with you!

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

A hands-on guide for creating a production-ready React app

Day 43— Everyone Must Know This Sorting Technique

Firmware flashing on TI CC2538 using OpenOCD and UART

Modeling blockchains as a Deterministic Finite Automata

Increasing your AppSec velocity with ShiftLeft

An Overview of MongoDB & Mongoose

Laravel | How to extend Notifications and use your custom fields

The First Pass at a Metapass

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
King Chun O

King Chun O

More from Medium

Python vs Other Programming Languages.

My First Programming Project: Text-based Hangman Game using Python

USING DATETIME IN PYTHON

DBeaver, Just Build the Dam Thing