{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## **Web Scraping** at **dev.to**\n", "```\n", "Link Web Scraping :\n", "https://dev.to\n", "\n", "Code By Natasya \n", "medium.com/@liontin\n", "```" ] }, { "cell_type": "code", "execution_count": 505, "metadata": {}, "outputs": [], "source": [ "import requests\n", "import pandas as pd\n", "from urllib.parse import urlencode" ] }, { "cell_type": "code", "execution_count": 506, "metadata": {}, "outputs": [], "source": [ "key = input(' Tags Keyword :') # your tags search\n", "start = input('Number Star of Page') \n", "stop = input('Number Stop of Page') \n", "\n", "startpage = int(start) #validation number of start page\n", "stoppage = int(stop) #validation number of stop page\n", "\n", "devto = []\n", "\n", "url = 'https://dev.to/search/feed_content?'\n", "for page in range(startpage,stoppage+1):\n", " params = {\n", " 'page': page,\n", " 'tag': key,\n", " 'sort_by': 'hotness_score',\n", " 'sort_direction': 'desc',\n", " 'tag_names[]': key,\n", " 'approved': '',\n", " 'class_name': 'Article'\n", " }\n", "\n", " geturl = url+urlencode(params)\n", " r = requests.get(geturl).json()\n", "\n", " dev_article = r['result']\n", " for to in dev_article:\n", " title = to['title']\n", " author = to['user']['username']\n", " link = to['path']\n", " date = to['readable_publish_date']\n", " comments = to['comments_count']\n", " reactions = to['public_reactions_count']\n", " tags_list = to['tag_list']\n", " devto.append([author, title, reactions, comments, tags_list, date, str('https://dev.to%s' % link)])\n" ] }, { "cell_type": "code", "execution_count": 507, "metadata": {}, "outputs": [], "source": [ "dev_to = pd.DataFrame(devto,columns=['author', 'title','reactions','comments', 'tags', 'date', 'link'])" ] }, { "cell_type": "code", "execution_count": 508, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | author | \n", "title | \n", "reactions | \n", "comments | \n", "tags | \n", "date | \n", "link | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "changach | \n", "Python 101:Ultimate Guide To Python | \n", "7 | \n", "0 | \n", "[programming, python] | \n", "Feb 13 | \n", "https://dev.to/changach/python-101ultimate-gui... | \n", "
1 | \n", "brayan_kai | \n", "Python 101: Introduction to Modern Python | \n", "10 | \n", "1 | \n", "[python, programming, datascience, machinelear... | \n", "Feb 13 | \n", "https://dev.to/brayan_kai/python-101-introduct... | \n", "
2 | \n", "jit01 | \n", "Django Internationalization | \n", "1 | \n", "0 | \n", "[djangointernationalization, webpagetranslatio... | \n", "Feb 9 | \n", "https://dev.to/epamindia/django-internationali... | \n", "
3 | \n", "fridex | \n", "Why PyPI Doesn't Know Your Projects Dependenci... | \n", "4 | \n", "0 | \n", "[python, programming, opensource, security] | \n", "Feb 13 | \n", "https://dev.to/fridex/why-pypi-doesnt-know-you... | \n", "
4 | \n", "elliot_brenyasarfo_18749 | \n", "5 Strange Things You Can Do With Python | \n", "4 | \n", "0 | \n", "[javascript, programming, beginners, python] | \n", "Feb 9 | \n", "https://dev.to/elliot_brenyasarfo_18749/5-stra... | \n", "
5 | \n", "0xbolt | \n", "Web3 authentication on Solana using Python⚡ | \n", "2 | \n", "0 | \n", "[solana, web3, python, beginners] | \n", "Feb 13 | \n", "https://dev.to/0xbolt/web3-authentication-on-s... | \n", "
6 | \n", "adachime | \n", "Introduction to modern python: Getting started... | \n", "11 | \n", "2 | \n", "[python, luxtechacademy, datascience, programm... | \n", "Feb 13 | \n", "https://dev.to/adachime/introduction-to-python... | \n", "
7 | \n", "wariarandiritu | \n", "PYTHON 101 | \n", "2 | \n", "0 | \n", "[programming, python] | \n", "Feb 13 | \n", "https://dev.to/wariarandiritu/python-101-3o57 | \n", "
8 | \n", "hannahmwende | \n", "Introduction to modern python | \n", "2 | \n", "0 | \n", "[beginners, programming, python] | \n", "Feb 13 | \n", "https://dev.to/hannahmwende/introduction-to-mo... | \n", "
9 | \n", "baransel | \n", "Python comment line | \n", "5 | \n", "0 | \n", "[python, beginners, tutorial] | \n", "Feb 9 | \n", "https://dev.to/baransel/python-comment-line-1i5p | \n", "