-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathblog.html
More file actions
57 lines (57 loc) · 84.8 KB
/
Copy pathblog.html
File metadata and controls
57 lines (57 loc) · 84.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
<!doctype html><html lang=en dir=ltr class="blog-wrapper blog-list-page plugin-blog plugin-id-default" data-has-hydrated=false><meta charset=UTF-8><meta name=generator content="Docusaurus v3.7.0"><title data-rh=true>Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast.</title><meta data-rh=true name=viewport content="width=device-width, initial-scale=1.0"><meta data-rh=true name=twitter:card content=summary_large_image><meta data-rh=true property=og:image content=https://crawlee.dev/img/crawlee-og.png><meta data-rh=true name=twitter:image content=https://crawlee.dev/img/crawlee-og.png><meta data-rh=true property=og:url content=https://crawlee.dev/blog><meta data-rh=true property=og:locale content=en><meta data-rh=true name=docusaurus_locale content=en><meta data-rh=true name=docsearch:language content=en><meta data-rh=true name=og:description content="Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living."><meta data-rh=true property=og:title content="Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast."><meta data-rh=true name=description content="Guides and tutorials on using Crawlee, the most reliable open-source web scraping and browser automation library for JavaScript and Node.js developers."><meta data-rh=true property=og:description content="Guides and tutorials on using Crawlee, the most reliable open-source web scraping and browser automation library for JavaScript and Node.js developers."><meta data-rh=true name=docusaurus_tag content=blog_posts_list><meta data-rh=true name=docsearch:docusaurus_tag content=blog_posts_list><link data-rh=true rel=icon href=/img/favicon.ico><link data-rh=true rel=canonical href=https://crawlee.dev/blog><link data-rh=true rel=alternate href=https://crawlee.dev/blog hreflang=en><link data-rh=true rel=alternate href=https://crawlee.dev/blog hreflang=x-default><script data-rh=true type=application/ld+json>{"@context":"https://schema.org","@id":"https://crawlee.dev/blog","@type":"Blog","blogPost":[{"@id":"https://crawlee.dev/blog/superscraper-with-crawlee","@type":"BlogPosting","author":[{"@type":"Person","description":"Developer Community Manager","image":"https://avatars.githubusercontent.com/u/53312820?v=4","name":"Saurav Jain","url":"https://github.com/souravjain540"},{"@type":"Person","description":"Web Automation Engineer","image":"https://ca.slack-edge.com/T0KRMEKK6-U04MGU11VUK-7f59c4a9343b-512","name":"Radoslav Chudovský","url":"https://github.com/chudovskyr"}],"datePublished":"2025-03-05T00:00:00.000Z","description":"This article explains how SuperScraper works, highlights its implementation details, and provides code snippets to demonstrate its core functionality.","headline":"Inside implementing SuperScraper with Crawlee","image":{"@id":"https://crawlee.dev/assets/images/superscraper-8d24da63227f97df70998e8900b3a901.webp","@type":"ImageObject","caption":"title image for the blog post: Inside implementing SuperScraper with Crawlee","contentUrl":"https://crawlee.dev/assets/images/superscraper-8d24da63227f97df70998e8900b3a901.webp","url":"https://crawlee.dev/assets/images/superscraper-8d24da63227f97df70998e8900b3a901.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/superscraper-with-crawlee","name":"Inside implementing SuperScraper with Crawlee","url":"https://crawlee.dev/blog/superscraper-with-crawlee"},{"@id":"https://crawlee.dev/blog/crawlee-for-python-v05","@type":"BlogPosting","author":{"@type":"Person","description":"Developer of Crawlee for Python","image":"https://avatars.githubusercontent.com/u/25082181?v=4","name":"Vlada Dusek","url":"https://github.com/vdusek"},"datePublished":"2025-01-10T00:00:00.000Z","description":"Announcing the Crawlee for Python v0.5 release.","headline":"Crawlee for Python v0.5","keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/crawlee-for-python-v05","name":"Crawlee for Python v0.5","url":"https://crawlee.dev/blog/crawlee-for-python-v05"},{"@id":"https://crawlee.dev/blog/scrape-crunchbase-python","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee and web scraping expert","image":"https://avatars.githubusercontent.com/u/34358312?v=4","name":"Max","url":"https://github.com/Mantisus"},"datePublished":"2025-01-03T00:00:00.000Z","description":"Learn how to scrape Crunchbase using Crawlee for Python","headline":"How to scrape Crunchbase using Python in 2024 (Easy Guide)","image":{"@id":"https://crawlee.dev/assets/images/scrape_crunchbase-28a71b5380492fe6618bbd9c90989543.webp","@type":"ImageObject","caption":"title image for the blog post: How to scrape Crunchbase using Python in 2024 (Easy Guide)","contentUrl":"https://crawlee.dev/assets/images/scrape_crunchbase-28a71b5380492fe6618bbd9c90989543.webp","url":"https://crawlee.dev/assets/images/scrape_crunchbase-28a71b5380492fe6618bbd9c90989543.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/scrape-crunchbase-python","name":"How to scrape Crunchbase using Python in 2024 (Easy Guide)","url":"https://crawlee.dev/blog/scrape-crunchbase-python"},{"@id":"https://crawlee.dev/blog/scrape-google-maps","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee","image":"https://avatars.githubusercontent.com/u/69134468?v=4","name":"Satyam Tripathi","url":"https://github.com/triposat"},"datePublished":"2024-12-13T00:00:00.000Z","description":"Learn how to scrape google maps data using Crawlee for Python","headline":"How to scrape Google Maps data using Python","image":{"@id":"https://crawlee.dev/assets/images/google-maps-fb70c1ab3b5c3032b0291db3392eb32d.webp","@type":"ImageObject","caption":"title image for the blog post: How to scrape Google Maps data using Python","contentUrl":"https://crawlee.dev/assets/images/google-maps-fb70c1ab3b5c3032b0291db3392eb32d.webp","url":"https://crawlee.dev/assets/images/google-maps-fb70c1ab3b5c3032b0291db3392eb32d.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/scrape-google-maps","name":"How to scrape Google Maps data using Python","url":"https://crawlee.dev/blog/scrape-google-maps"},{"@id":"https://crawlee.dev/blog/scrape-google-search","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee and web scraping expert","image":"https://avatars.githubusercontent.com/u/34358312?v=4","name":"Max","url":"https://github.com/Mantisus"},"datePublished":"2024-12-02T00:00:00.000Z","description":"Learn how to scrape google search results using Crawlee for Python","headline":"How to scrape Google search results with Python","image":{"@id":"https://crawlee.dev/assets/images/google-search-a91bfdf17a4c2860798444b1be56f625.webp","@type":"ImageObject","caption":"title image for the blog post: How to scrape Google search results with Python","contentUrl":"https://crawlee.dev/assets/images/google-search-a91bfdf17a4c2860798444b1be56f625.webp","url":"https://crawlee.dev/assets/images/google-search-a91bfdf17a4c2860798444b1be56f625.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/scrape-google-search","name":"How to scrape Google search results with Python","url":"https://crawlee.dev/blog/scrape-google-search"},{"@id":"https://crawlee.dev/blog/graphql-persisted-query","@type":"BlogPosting","author":[{"@type":"Person","description":"Developer Community Manager","image":"https://avatars.githubusercontent.com/u/53312820?v=4","name":"Saurav Jain","url":"https://github.com/souravjain540"},{"@type":"Person","description":"Web Automation Engineer","image":"https://avatars.githubusercontent.com/u/31281386?v=4","name":"Matěj Volf","url":"https://github.com/mvolfik"}],"datePublished":"2024-11-15T00:00:00.000Z","description":"Learn how to do reverse engineering on persistedQuery extension by GraphQL and reveal the query hash needed for scraping.","headline":"Reverse engineering GraphQL persistedQuery extension","image":{"@id":"https://crawlee.dev/assets/images/graphql-a3962ed441b2a078e43c8158ad64336a.webp","@type":"ImageObject","caption":"title image for the blog post: Reverse engineering GraphQL persistedQuery extension","contentUrl":"https://crawlee.dev/assets/images/graphql-a3962ed441b2a078e43c8158ad64336a.webp","url":"https://crawlee.dev/assets/images/graphql-a3962ed441b2a078e43c8158ad64336a.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/graphql-persisted-query","name":"Reverse engineering GraphQL persistedQuery extension","url":"https://crawlee.dev/blog/graphql-persisted-query"},{"@id":"https://crawlee.dev/blog/web-scraping-tips","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee and web scraping expert","image":"https://avatars.githubusercontent.com/u/34358312?v=4","name":"Max","url":"https://github.com/Mantisus"},"datePublished":"2024-11-10T00:00:00.000Z","description":"Learn how to think and scrape like a web scraping expert.","headline":"12 tips on how to think like a web scraping expert","image":{"@id":"https://crawlee.dev/assets/images/scraping-tips-8c538d5ae19dc1737b083169ad2a203b.webp","@type":"ImageObject","caption":"title image for the blog post: 12 tips on how to think like a web scraping expert","contentUrl":"https://crawlee.dev/assets/images/scraping-tips-8c538d5ae19dc1737b083169ad2a203b.webp","url":"https://crawlee.dev/assets/images/scraping-tips-8c538d5ae19dc1737b083169ad2a203b.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/web-scraping-tips","name":"12 tips on how to think like a web scraping expert","url":"https://crawlee.dev/blog/web-scraping-tips"},{"@id":"https://crawlee.dev/blog/linkedin-job-scraper-python","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee","image":"https://avatars.githubusercontent.com/u/109217591?v=4","name":"Arindam Majumder","url":"https://github.com/Arindam200"},"datePublished":"2024-10-14T00:00:00.000Z","description":"Learn how to scrape LinkedIn jobs and save it into a CSV file using Python.","headline":"How to create a LinkedIn job scraper in Python with Crawlee","image":{"@id":"https://crawlee.dev/assets/images/linkedin-job-scraper-3490f5b2fd530a5ee7a74cda78c6f0e7.webp","@type":"ImageObject","caption":"title image for the blog post: How to create a LinkedIn job scraper in Python with Crawlee","contentUrl":"https://crawlee.dev/assets/images/linkedin-job-scraper-3490f5b2fd530a5ee7a74cda78c6f0e7.webp","url":"https://crawlee.dev/assets/images/linkedin-job-scraper-3490f5b2fd530a5ee7a74cda78c6f0e7.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/linkedin-job-scraper-python","name":"How to create a LinkedIn job scraper in Python with Crawlee","url":"https://crawlee.dev/blog/linkedin-job-scraper-python"},{"@id":"https://crawlee.dev/blog/scrape-using-jsdom","@type":"BlogPosting","author":{"@type":"Person","description":"Developer Community Manager","image":"https://avatars.githubusercontent.com/u/53312820?v=4","name":"Saurav Jain","url":"https://github.com/souravjain540"},"datePublished":"2024-09-30T00:00:00.000Z","description":"Learn how to scrape using JSDOM, alternative to Cheerio and browser based scraping.","headline":"Optimizing web scraping: Scraping auth data using JSDOM","image":{"@id":"https://crawlee.dev/assets/images/jsdom-12876619cd7871bcebaa64d74adefd4c.webp","@type":"ImageObject","caption":"title image for the blog post: Optimizing web scraping: Scraping auth data using JSDOM","contentUrl":"https://crawlee.dev/assets/images/jsdom-12876619cd7871bcebaa64d74adefd4c.webp","url":"https://crawlee.dev/assets/images/jsdom-12876619cd7871bcebaa64d74adefd4c.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/scrape-using-jsdom","name":"Optimizing web scraping: Scraping auth data using JSDOM","url":"https://crawlee.dev/blog/scrape-using-jsdom"},{"@id":"https://crawlee.dev/blog/scraping-dynamic-websites-using-python","@type":"BlogPosting","author":{"@type":"Person","description":"Community Member of Crawlee and web scraping expert","image":"https://avatars.githubusercontent.com/u/34358312?v=4","name":"Max","url":"https://github.com/Mantisus"},"datePublished":"2024-09-12T00:00:00.000Z","description":"Learn how to scrape dynamic websites using Crawlee for Python with HTTP client.","headline":"Web scraping of a dynamic website using Python with HTTP Client","image":{"@id":"https://crawlee.dev/assets/images/dynamic-websites-d9a83deff0729330b2d3de2d1481cd6a.webp","@type":"ImageObject","caption":"title image for the blog post: Web scraping of a dynamic website using Python with HTTP Client","contentUrl":"https://crawlee.dev/assets/images/dynamic-websites-d9a83deff0729330b2d3de2d1481cd6a.webp","url":"https://crawlee.dev/assets/images/dynamic-websites-d9a83deff0729330b2d3de2d1481cd6a.webp"},"keywords":[],"mainEntityOfPage":"https://crawlee.dev/blog/scraping-dynamic-websites-using-python","name":"Web scraping of a dynamic website using Python with HTTP Client","url":"https://crawlee.dev/blog/scraping-dynamic-websites-using-python"}],"description":"Guides and tutorials on using Crawlee, the most reliable open-source web scraping and browser automation library for JavaScript and Node.js developers.","headline":"Crawlee Blog - learn how to build better scrapers","mainEntityOfPage":"https://crawlee.dev/blog"}</script><link rel=alternate type=application/rss+xml href=/blog/rss.xml title="Crawlee for JavaScript · Build reliable crawlers. Fast. RSS Feed"><link rel=alternate type=application/atom+xml href=/blog/atom.xml title="Crawlee for JavaScript · Build reliable crawlers. Fast. Atom Feed"><link rel=search type=application/opensearchdescription+xml title="Crawlee for JavaScript · Build reliable crawlers. Fast." href=/opensearch.xml><!-- Google Tag Manager --><script>!function(e,t,a,n,r){e[n]=e[n]||[],e[n].push({"gtm.start":new Date().getTime(),event:"gtm.js"});var g=t.getElementsByTagName(a)[0],m=t.createElement(a);m.async=!0,m.src="https://www.googletagmanager.com/gtm.js?id="+r+("dataLayer"!=n?"&l="+n:""),g.parentNode.insertBefore(m,g)}(window,document,"script","dataLayer","GTM-5P7MCS7")</script><!-- End Google Tag Manager --><script>window.intercomSettings={api_base:"https://api-iam.intercom.io",app_id:"kod1r788"},function(){var t=window,e=t.Intercom;if("function"==typeof e)e("reattach_activator"),e("update",t.intercomSettings);else{var n=document,a=function(){a.c(arguments)};a.q=[],a.c=function(t){a.q.push(t)},t.Intercom=a;var i=function(){var t=n.createElement("script");t.type="text/javascript",t.async=!0,t.src="https://widget.intercom.io/widget/kod1r788";var e=n.getElementsByTagName("script")[0];e.parentNode.insertBefore(t,e)};"complete"===document.readyState?i():t.attachEvent?t.attachEvent("onload",i):t.addEventListener("load",i,!1)}}()</script><script src=/js/custom.js></script><link rel=stylesheet href=/assets/css/styles.3744b2b5.css><script src=/assets/js/runtime~main.1e84cbe3.js defer></script><script src=/assets/js/main.d619d1ee.js defer></script><body class=navigation-with-keyboard><script>!function(){var t,e=function(){try{return new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return window.localStorage.getItem("theme")}catch(t){}}();t=null!==e?e:window.matchMedia("(prefers-color-scheme: dark)").matches?"dark":(window.matchMedia("(prefers-color-scheme: light)").matches,"light"),document.documentElement.setAttribute("data-theme",t)}(),function(){try{for(var[t,e]of new URLSearchParams(window.location.search).entries())if(t.startsWith("docusaurus-data-")){var a=t.replace("docusaurus-data-","data-");document.documentElement.setAttribute(a,e)}}catch(t){}}()</script><div id=__docusaurus><link rel=preload as=image href=/img/crawlee-light.svg><link rel=preload as=image href=/img/crawlee-dark.svg><link rel=preload as=image href=/img/crawlee-javascript-light.svg><link rel=preload as=image href=/img/crawlee-javascript-dark.svg><link rel=preload as=image href=/img/crawlee-python-light.svg><link rel=preload as=image href=/img/crawlee-python-dark.svg><link rel=preload as=image href="https://avatars.githubusercontent.com/u/53312820?v=4"><link rel=preload as=image href=https://ca.slack-edge.com/T0KRMEKK6-U04MGU11VUK-7f59c4a9343b-512><link rel=preload as=image href="https://avatars.githubusercontent.com/u/25082181?v=4"><link rel=preload as=image href="https://avatars.githubusercontent.com/u/34358312?v=4"><link rel=preload as=image href="https://avatars.githubusercontent.com/u/69134468?v=4"><link rel=preload as=image href="https://avatars.githubusercontent.com/u/31281386?v=4"><link rel=preload as=image href="https://avatars.githubusercontent.com/u/109217591?v=4"><div role=region aria-label="Skip to main content"><a class=skipToContent_fXgn href=#__docusaurus_skipToContent_fallback>Skip to main content</a></div><nav aria-label=Main class="navbar navbar--fixed-top navbarHideable_m1mJ"><div class=navbar__inner><div class=navbar__items><div class=navbarLogo__GY_><div class=logoWithArrows_cB9I><a class="logoImage_a28E sidebarLogo" href=/><img src=/img/crawlee-light.svg alt=Crawlee class="themedComponent_mlkZ themedComponent--light_NVdE"><img src=/img/crawlee-dark.svg alt=Crawlee class="themedComponent_mlkZ themedComponent--dark_xIcU"></a><svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16"><g stroke=#B2B8CC stroke-linecap=round stroke-linejoin=round><path d="M5.334 6 8 3.333 10.667 6M10.667 10 8 12.667 5.334 10"/></g></svg></div><div class=menuWrapper_zUZ_><div class=menu_CQ3p><a class=menuItem_QSny href=/js><img src=/img/crawlee-javascript-light.svg alt="Crawlee JavaScript" class="themedComponent_mlkZ themedComponent--light_NVdE"><img src=/img/crawlee-javascript-dark.svg alt="Crawlee JavaScript" class="themedComponent_mlkZ themedComponent--dark_xIcU"></a><a href=https://crawlee.dev/python target=_self rel=dofollow class=menuItem_QSny><img src=/img/crawlee-python-light.svg alt="Crawlee Python" class="themedComponent_mlkZ themedComponent--light_NVdE"><img src=/img/crawlee-python-dark.svg alt="Crawlee Python" class="themedComponent_mlkZ themedComponent--dark_xIcU"></a><a class=menuItem_QSny href=/><img src=/img/crawlee-light.svg alt=Crawlee class="themedComponent_mlkZ themedComponent--light_NVdE"><img src=/img/crawlee-dark.svg alt=Crawlee class="themedComponent_mlkZ themedComponent--dark_xIcU"><svg xmlns=http://www.w3.org/2000/svg width=17 height=16 fill=none viewBox="0 0 17 16"><path fill=#555D76 d="M14.742 4.22a.75.75 0 0 1 0 1.06l-7.25 7.25a.75.75 0 0 1-1.06 0l-3.25-3.25a.751.751 0 0 1 1.06-1.06l2.72 2.72 6.72-6.72a.75.75 0 0 1 1.06 0"/></svg></a></div></div></div><div class="navbarItems_fVuF navbarItems__leftMargin_FZoS"><a class="navbar__item navbar__link" href=/js>JavaScript</a><a href=https://crawlee.dev/python target=_self rel=dofollow class="navbar__item navbar__link">Python</a><a aria-current=page class="navbar__item navbar__link navbar__link--active" href=/blog>Blog</a></div></div><div class="navbar__items navbar__items--right"><button aria-label="Toggle navigation bar" aria-expanded=false class="navbar__toggle clean-btn" type=button><svg width=30 height=30 viewBox="0 0 30 30" aria-hidden=true><path stroke=currentColor stroke-linecap=round stroke-miterlimit=10 stroke-width=2 d="M4 7h22M4 15h22M4 23h22"/></svg></button></div></div><div role=presentation class=navbar-sidebar__backdrop></div></nav><div id=__docusaurus_skipToContent_fallback class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class=row><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">All posts</div><div role=group><h3 class=yearGroupHeading_rMGB>2025</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/superscraper-with-crawlee>Inside implementing SuperScraper with Crawlee</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/crawlee-for-python-v05>Crawlee for Python v0.5</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scrape-crunchbase-python>How to scrape Crunchbase using Python in 2024 (Easy Guide)</a></ul></div><div role=group><h3 class=yearGroupHeading_rMGB>2024</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scrape-google-maps>How to scrape Google Maps data using Python</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scrape-google-search>How to scrape Google search results with Python</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/graphql-persisted-query>Reverse engineering GraphQL persistedQuery extension</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/web-scraping-tips>12 tips on how to think like a web scraping expert</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/linkedin-job-scraper-python>How to create a LinkedIn job scraper in Python with Crawlee</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scrape-using-jsdom>Optimizing web scraping: Scraping auth data using JSDOM</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scraping-dynamic-websites-using-python>Web scraping of a dynamic website using Python with HTTP Client</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/infinite-scroll-using-python>How to scrape infinite scrolling webpages with Python</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/common-problems-in-web-scraping>Current problems and mistakes of web scraping in Python and tricks to solve them!</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/launching-crawlee-python>Announcing Crawlee for Python: Now you can use Python to build reliable web crawlers</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/proxy-management-in-crawlee>How Crawlee uses tiered proxies to avoid getting blocked</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/netflix-show-recommender>Building a Netflix show recommender using Crawlee and React</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/scrapy-vs-crawlee>Scrapy vs. Crawlee</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/how-to-scrape-amazon>How to scrape Amazon products</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/blog/crawlee-blog-launch>Launching Crawlee Blog</a></ul></div></nav></aside><main class="col col--7"><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/superscraper-with-crawlee>Inside implementing SuperScraper with Crawlee</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2025-03-05T00:00:00.000Z>March 5, 2025</time> · <!-- -->6 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/53312820?v=4" alt="Saurav Jain"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer"><span>Saurav Jain</span></a></div><small class=avatar__subtitle>Developer Community Manager</small></div></div></div><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/chudovskyr target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src=https://ca.slack-edge.com/T0KRMEKK6-U04MGU11VUK-7f59c4a9343b-512 alt="Radoslav Chudovský"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/chudovskyr target=_blank rel="noopener noreferrer"><span>Radoslav Chudovský</span></a></div><small class=avatar__subtitle>Web Automation Engineer</small></div></div></div></div></header><div class=markdown><p><a href=https://github.com/apify/super-scraper target=_blank rel="noopener noreferrer">SuperScraper</a> is an open-source <a href=https://docs.apify.com/platform/actors target=_blank rel="noopener noreferrer">Actor</a> that combines features from various web scraping services, including <a href=https://www.scrapingbee.com/ target=_blank rel="noopener noreferrer">ScrapingBee</a>, <a href=https://scrapingant.com/ target=_blank rel="noopener noreferrer">ScrapingAnt</a>, and <a href=https://www.scraperapi.com/ target=_blank rel="noopener noreferrer">ScraperAPI</a>.</p>
<p>A key capability is its standby mode, which runs the Actor as a persistent API server. This removes the usual start-up times - a common pain point in many systems - and lets users make direct API calls to interact with the system immediately.</p>
<p>This blog explains how SuperScraper works, highlights its implementation details, and provides code snippets to demonstrate its core functionality.</p>
<p><img decoding=async loading=lazy alt="Google Maps Data Screenshot" src=/assets/images/superscraper-8d24da63227f97df70998e8900b3a901.webp width=1152 height=649 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col text--right"><a aria-label="Read more about Inside implementing SuperScraper with Crawlee" href=/blog/superscraper-with-crawlee><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/crawlee-for-python-v05>Crawlee for Python v0.5</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2025-01-10T00:00:00.000Z>January 10, 2025</time> · <!-- -->6 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/vdusek target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/25082181?v=4" alt="Vlada Dusek"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/vdusek target=_blank rel="noopener noreferrer"><span>Vlada Dusek</span></a></div><small class=avatar__subtitle>Developer of Crawlee for Python</small></div></div></div></div></header><div class=markdown><p>Crawlee for Python v0.5 is now available! This is our biggest release to date, bringing new ported functionality from the <a href=https://github.com/apify/crawlee target=_blank rel="noopener noreferrer">Crawlee for JavaScript</a>, brand-new features that are exclusive to the Python library (for now), a new consolidated package structure, and a bunch of bug fixes and further improvements.</div><footer class="row docusaurus-mt-lg"><div class="col text--right"><a aria-label="Read more about Crawlee for Python v0.5" href=/blog/crawlee-for-python-v05><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/scrape-crunchbase-python>How to scrape Crunchbase using Python in 2024 (Easy Guide)</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2025-01-03T00:00:00.000Z>January 3, 2025</time> · <!-- -->11 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/34358312?v=4" alt=Max></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer"><span>Max</span></a></div><small class=avatar__subtitle>Community Member of Crawlee and web scraping expert</small></div></div></div></div></header><div class=markdown><p>Python developers know the drill: you need reliable company data, and Crunchbase has it. This guide shows you how to build an effective <a href=https://www.crunchbase.com/ target=_blank rel="noopener noreferrer">Crunchbase</a> scraper in Python that gets you the data you need.</p>
<p>Crunchbase tracks details that matter: locations, business focus, founders, and investment histories. Manual extraction from such a large dataset isn't practical -automation is essential for transforming this information into an analyzable format.</p>
<p>By the end of this blog, we'll explore three different ways to extract data from Crunchbase using <a href=https://github.com/apify/crawlee-python target=_blank rel="noopener noreferrer"><code>Crawlee for Python</code></a>. We'll fully implement two of them and discuss the specifics and challenges of the third. This will help us better understand how important it is to properly <a href=https://www.crawlee.dev/blog/web-scraping-tips#1-choosing-a-data-source-for-the-project target=_blank rel="noopener noreferrer">choose the right data source</a>.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>This guide comes from a developer in our growing community. Have you built interesting projects with Crawlee? Join us on <a href=https://discord.com/invite/jyEM2PRvMU target=_blank rel="noopener noreferrer">Discord</a> to share your experiences and blog ideas - we value these contributions from developers like you.</div></div>
<p><img decoding=async loading=lazy alt="How to Scrape Crunchbase Using Python" src=/assets/images/scrape_crunchbase-28a71b5380492fe6618bbd9c90989543.webp width=1152 height=649 class=img_ev3q></p>
<p>Key steps we'll cover:</p>
<ol>
<li>Project setup</li>
<li>Choosing the data source</li>
<li>Implementing sitemap-based crawler</li>
<li>Analysis of search-based approach and its limitations</li>
<li>Implementing the official API crawler</li>
<li>Conclusion and repository access</li>
</ol></div><footer class="row docusaurus-mt-lg"><div class="col col--9"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class=tag_QGVx><a class="tag_zVej tagRegular_sFm0" href=/blog/tags/community>community</a></ul></div><div class="col text--right col--3"><a aria-label="Read more about How to scrape Crunchbase using Python in 2024 (Easy Guide)" href=/blog/scrape-crunchbase-python><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/scrape-google-maps>How to scrape Google Maps data using Python</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-12-13T00:00:00.000Z>December 13, 2024</time> · <!-- -->11 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/triposat target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/69134468?v=4" alt="Satyam Tripathi"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/triposat target=_blank rel="noopener noreferrer"><span>Satyam Tripathi</span></a></div><small class=avatar__subtitle>Community Member of Crawlee</small></div></div></div></div></header><div class=markdown><p>Millions of people use Google Maps daily, leaving behind a goldmine of data just waiting to be analyzed. In this guide, I'll show you how to build a reliable scraper using Crawlee and Python to extract locations, ratings, and reviews from Google Maps, all while handling its dynamic content challenges.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>One of our community members wrote this blog as a contribution to the Crawlee Blog. If you would like to contribute blogs like these to Crawlee Blog, please reach out to us on our <a href=https://apify.com/discord target=_blank rel="noopener noreferrer">discord channel</a>.</div></div>
<h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id=what-data-will-we-extract-from-google-maps>What data will we extract from Google Maps?<a href=#what-data-will-we-extract-from-google-maps class=hash-link aria-label="Direct link to What data will we extract from Google Maps?" title="Direct link to What data will we extract from Google Maps?"></a></h2>
<p>We’ll collect information about hotels in a specific city. You can also customize your search to meet your requirements. For example, you might search for "hotels near me", "5-star hotels in Bombay", or other similar queries.</p>
<p><img decoding=async loading=lazy alt="Google Maps Data Screenshot" src=/assets/images/scrape-google-maps-with-crawlee-screenshot-data-to-scrape-00e7e4e3498679b8a7611eafd0a1bfbe.webp width=1906 height=879 class=img_ev3q></p>
<p>We’ll extract important data, including the hotel name, rating, review count, price, a link to the hotel page on Google Maps, and all available amenities. Here’s an example of what the extracted data will look like:</p>
<div class="language-json codeBlockContainer_Ckt0 theme-code-block" style=--prism-color:#393A34;--prism-background-color:#f6f8fa><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-json codeBlock_bY9V thin-scrollbar" style=color:#393A34;background-color:#f6f8fa><code class=codeBlockLines_e6Vv><span class=token-line style=color:#393A34><span class="token punctuation" style=color:#393A34>{</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"name"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token string" style=color:#e3116c>"Vividus Hotels, Bangalore"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"rating"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token string" style=color:#e3116c>"4.3"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"reviews"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token string" style=color:#e3116c>"633"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"price"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token string" style=color:#e3116c>"₹3,667"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"amenities"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token punctuation" style=color:#393A34>[</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token string" style=color:#e3116c>"Pool available"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token string" style=color:#e3116c>"Free breakfast available"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token string" style=color:#e3116c>"Free Wi-Fi available"</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token string" style=color:#e3116c>"Free parking available"</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token punctuation" style=color:#393A34>]</span><span class="token punctuation" style=color:#393A34>,</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"> </span><span class="token property" style=color:#36acaa>"link"</span><span class="token operator" style=color:#393A34>:</span><span class="token plain"> </span><span class="token string" style=color:#e3116c>"https://www.google.com/maps/place/Vividus+Hotels+,+Bangalore/..."</span><span class="token plain"></span><br></span><span class=token-line style=color:#393A34><span class="token plain"></span><span class="token punctuation" style=color:#393A34>}</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div></div><footer class="row docusaurus-mt-lg"><div class="col col--9"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class=tag_QGVx><a class="tag_zVej tagRegular_sFm0" href=/blog/tags/community>community</a></ul></div><div class="col text--right col--3"><a aria-label="Read more about How to scrape Google Maps data using Python" href=/blog/scrape-google-maps><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/scrape-google-search>How to scrape Google search results with Python</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-12-02T00:00:00.000Z>December 2, 2024</time> · <!-- -->7 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/34358312?v=4" alt=Max></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer"><span>Max</span></a></div><small class=avatar__subtitle>Community Member of Crawlee and web scraping expert</small></div></div></div></div></header><div class=markdown><p>Scraping <code>Google Search</code> delivers essential <code>SERP analysis</code>, SEO optimization, and data collection capabilities. Modern scraping tools make this process faster and more reliable.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>One of our community members wrote this blog as a contribution to the Crawlee Blog. If you would like to contribute blogs like these to Crawlee Blog, please reach out to us on our <a href=https://apify.com/discord target=_blank rel="noopener noreferrer">discord channel</a>.</div></div>
<p>In this guide, we'll create a Google Search scraper using <a href=https://github.com/apify/crawlee-python target=_blank rel="noopener noreferrer"><code>Crawlee for Python</code></a> that can handle result ranking and pagination.</p>
<p>We'll create a scraper that:</p>
<ul>
<li>Extracts titles, URLs, and descriptions from search results</li>
<li>Handles multiple search queries</li>
<li>Tracks ranking positions</li>
<li>Processes multiple result pages</li>
<li>Saves data in a structured format</li>
</ul>
<p><img decoding=async loading=lazy alt="How to scrape Google search results with Python" src=/assets/images/google-search-a91bfdf17a4c2860798444b1be56f625.webp width=1152 height=649 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col col--9"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class=tag_QGVx><a class="tag_zVej tagRegular_sFm0" href=/blog/tags/community>community</a></ul></div><div class="col text--right col--3"><a aria-label="Read more about How to scrape Google search results with Python" href=/blog/scrape-google-search><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/graphql-persisted-query>Reverse engineering GraphQL persistedQuery extension</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-11-15T00:00:00.000Z>November 15, 2024</time> · <!-- -->5 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/53312820?v=4" alt="Saurav Jain"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer"><span>Saurav Jain</span></a></div><small class=avatar__subtitle>Developer Community Manager</small></div></div></div><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/mvolfik target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/31281386?v=4" alt="Matěj Volf"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/mvolfik target=_blank rel="noopener noreferrer"><span>Matěj Volf</span></a></div><small class=avatar__subtitle>Web Automation Engineer</small></div></div></div></div></header><div class=markdown><p>GraphQL is a query language for getting deeply nested structured data from a website's backend, similar to MongoDB queries.</p>
<p>The request is usually a POST to some general <code>/graphql</code> endpoint with a body like this:</p>
<p><img decoding=async loading=lazy alt="GraphQL Query" src=/assets/images/graphql-a3962ed441b2a078e43c8158ad64336a.webp width=853 height=275 class=img_ev3q></p>
<p>When scraping data from websites using GraphQL, it’s common to inspect the network requests in developer tools to find the exact queries being used. However, on some websites, you might notice that the GraphQL query itself isn’t visible in the request. Instead, you only see a cryptic hash value. This can be confusing and makes it harder to understand how data is being requested from the server.</p>
<p>This is because some websites use a feature called <a href=https://www.apollographql.com/docs/apollo-server/performance/apq/ target=_blank rel="noopener noreferrer">"persisted queries.</a> It's a performance optimization that reduces the amount of data sent with each request by replacing the full query text with a precomputed hash. While this improves website speed and efficiency, it introduces challenges for scraping because the query text isn’t readily available.</p>
<p><img decoding=async loading=lazy alt="Persisted Query Reverse Engineering" src=/assets/images/graphql-persisted-query-6e36e61d76503e617fe4e7651bdf53a3.webp width=1152 height=649 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col text--right"><a aria-label="Read more about Reverse engineering GraphQL persistedQuery extension" href=/blog/graphql-persisted-query><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/web-scraping-tips>12 tips on how to think like a web scraping expert</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-11-10T00:00:00.000Z>November 10, 2024</time> · <!-- -->12 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/34358312?v=4" alt=Max></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer"><span>Max</span></a></div><small class=avatar__subtitle>Community Member of Crawlee and web scraping expert</small></div></div></div></div></header><div class=markdown><p>Typically, tutorials focus on the technical aspects, on what you can replicate: "Start here, follow this path, and you'll end up here." This is great for learning a particular technology, but it's sometimes difficult to understand why the author decided to do things a certain way or what guides their development process.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>One of our community members wrote this blog as a contribution to Crawlee Blog. If you want to contribute blogs like these to Crawlee Blog, please reach out to us on our <a href=https://apify.com/discord target=_blank rel="noopener noreferrer">discord channel</a>.</div></div>
<p>In this blog, I'll discuss the general rules and principles that guide me when I work on web scraping projects and allow me to achieve great results.</p>
<p>So, let's explore the mindset of a web scraping developer.</p>
<p><img decoding=async loading=lazy alt="How to think like a web scraping expert" src=/assets/images/scraping-tips-8c538d5ae19dc1737b083169ad2a203b.webp width=1152 height=649 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col col--9"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class=tag_QGVx><a class="tag_zVej tagRegular_sFm0" href=/blog/tags/community>community</a></ul></div><div class="col text--right col--3"><a aria-label="Read more about 12 tips on how to think like a web scraping expert" href=/blog/web-scraping-tips><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/linkedin-job-scraper-python>How to create a LinkedIn job scraper in Python with Crawlee</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-10-14T00:00:00.000Z>October 14, 2024</time> · <!-- -->7 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/Arindam200 target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/109217591?v=4" alt="Arindam Majumder"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/Arindam200 target=_blank rel="noopener noreferrer"><span>Arindam Majumder</span></a></div><small class=avatar__subtitle>Community Member of Crawlee</small></div></div></div></div></header><div class=markdown><h2 class="anchor anchorWithHideOnScrollNavbar_WYt5" id=introduction>Introduction<a href=#introduction class=hash-link aria-label="Direct link to Introduction" title="Direct link to Introduction"></a></h2>
<p>In this article, we will build a web application that scrapes LinkedIn for job postings using Crawlee and Streamlit.</p>
<p>We will create a LinkedIn job scraper in Python using Crawlee for Python to extract the company name, job title, time of posting, and link to the job posting from dynamically received user input through the web application.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>One of our community members wrote this blog as a contribution to Crawlee Blog. If you want to contribute blogs like these to Crawlee Blog, please reach out to us on our <a href=https://apify.com/discord target=_blank rel="noopener noreferrer">discord channel</a>.</div></div>
<p>By the end of this tutorial, you’ll have a fully functional web application that you can use to scrape job postings from LinkedIn.</p>
<p><img decoding=async loading=lazy alt="Linkedin Job Scraper" src="data:image/webp;base64,UklGRpgcAABXRUJQVlA4IIwcAAAQfAGdASpcBt0CPpFIoEylpCaioJN4QNASCWlu+F8+/k/AuBCbOFzUYLOjUh0O8tefHp3bdRuonqp/z31K/PL9aj/jZM75b/xP42eEf+Z8N/IZ799v+U91X5q/y375fwPNPwB+IGoR+V/0L/U72OAD8t/r368+Ob/l+jH2Q9gLzB8EGgN5Nf+j5Tvr/2EOmGB9D/YD8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfM+Yq8OTW3tvbe29t7b23tvbe22iZO/M+Z8z5nzPmfM+Z8z5nzPmfM+Z8z5nzPmfwiTklmXH4pWwfHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4b/YoBGhwWLFixYsWLFixYsWLFixYsWLFixYsWLFixYsWLFixYsWLFixYsWLFimOzM6zIyDLIC6inMYBQN4AdJ+xanZv1IVvZErjAbqWdBIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRLgLnNevXr169evXr169evXr169evXr169evXr169evXr169evXr169evXr169e69y5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly6R48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48iCtWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atW8tkyZMmTJkyZMmTHkuiL1UwcSehb4g6f+/kZqrDYDuhB5g1lF0nZC9nZ+vVNIKgtI4eyNTRiq4pxF9r8+2qMYQezlaAB+FZCmqk8DlC4jj+06dOnTp06dOnTp06dOnTp06dOnT3kiRIkSJEiRIkSJC/a91YpfY9LY7LHPlSIuGOyl8hLfCadJCMOqQf2nFwzjL0sCEFlwSmiGpay2JlSr+iwgbQ/pr3/Ynau1aSaw0G/XGdFiZN0U3g9p2vfcqW19E3mXGJv0k2NLrU4el8Rntj/mUStqrFnao8rs/wEEb5SvrG/EAEcf8WauHMhiN9lYgl2y/WuvXS47Ad0sCQXpQ5Nij8jEYp+5SOO5My++pwpNPhyQp6DiawABYsWLFixYsWLFixYsWLFixYsWLPB48ePHjx48ePHjwyCazIpuZ2oqIwtbNyUGmr/YqnQDGqKoAqAOC+AcbvAl6wa8o9xi6LIPKbbUi45jqAXa2j2Gj95zrxLBMiuH02BvEolPoFjYmD0ZKtzDSDllWy8+Flzd+B+OzJt/0oD3bUj4YEAiLSXdQ+r41F0F9sCCT3Ev3MTUyqF7F08v9d3zuBQwIC8skNs/43cnF0y20CAU2fPnz58+fPnz58+fPnz58+fPn0AQkSJEiRIkSJEiRIk0lONsiRIkVM0IoWkPA0COSa9evXr169evXr169evXr169evde5cuXLly5cuXLlybh1/cSC+LPV6vV6vV6vV6vV6vV6vV6vV6vV6vV6HatiOGNmzZs2bNmzZs2bNuGBAgQIECBAgQIEBsEIXEFaMWp2OMZCEqZYsWLFixYsWLFixYsWLFixYs8Hjx48ePHjx48ePHpLVq1atWrVq1atWrGVxgdKMwI6H/Ym0ZEun08JZg6tAPH4t+yZMmTJkyZMmTJkyZMmTJkyZQDw4cOHDhw4cOHDhxYHxw4cOHDhw4cOHCIWNkEvOOewez7OeSlCkjWWuuTa9XtfwJxUszla8lJ/crS4InNevXr169evXr16917ly5cuXLly5cuXKjLb+Vq1atWrVq1atWrVq1atWrVq1bBpsRIkSJEiRIkSJEiSG+fPnz58+fPnz587tyCt9HPAEWaQwGAwGAwGAwGAwGAwGAwGAwGAwGAwF/sqMz6biw4cOHDhw4cOHDhxFLZs2bNmzZs2bNmy1x9llmnoZJ6a7f4Qo9Rdv8IUeou3+EKPUXb/CFHqLt/hCj1FxJLVq1atWrVq1atWrWbyZMmTJkyZMmTJkwObpZg7BbwG3kXgHT+Ifw9We5EAuFPbxx2Nz+SJEiRIkSJEiRIkSJEiRIkSQ3z58+fPnz58+fPn1BIkSJEiRIkSJEiRFbm9gAP3Yig/dVuq8k79Vbvl+gB1E4kVDD3La9Xq9Xq9Xq9Xq9Xq9Xq9Xq9Xq9Wj91BRIkSJEiRIkSJEiROGfPnz58+fPnz58+PjNKh8L8OYiTklmXH4pZ6i7f4Qo9Rdv8IUeou3+EKPUXb/CFG8B1yuF2bNmzZs2bNmzZs24YECBAgQIECBAgQGwQjnYwyZMmTJkyZMmTJkyZMmTJkyZMocqXLly5cuXLly5cuXSPHjx48ePHjx48eO0d82+2TRbe4zjoOiFHqLt/hCj1F2/whR6i7f4Qo9Rdv8IUeou3+EKOf+PTb4Hxw4cOHDhw4cOHDmQiRIkSJEiRIkSJEVuc9x5l7TVdAzwpeAZ4UvAM8KXgGeFLwDPCl4BnhS8AzuuQgQIECBAgQIECBAjOCxYsWLFixYsWLFaZan17VjsKGW4mfhbr4uDWNmMlU5/DYpzUFDKudkJXnBM87dLMZhwLuBdANKzlFVJjrxPyBw4cOHDhw4cOHDhw4cOHDiwPjhw4cOHDhw4cOHMhEiRIkSJEiRIkSIrc31ectMWdFKFIYDAYDAYDAYDAYDAYDAYDAX9HO5dz+vXr169evXr169ews1atWrVq1atWrVqxlcxua1atWrVq1atWrVq1atWrVq1at7J4cOHDhw4cOHDhw4sD44cOHDhw4cOHDhELGz8ho0ggQIECBAgQIECBAgQIECBAgQFB2mEOa1atWrVq1atWrVq3lsmTJkyZMmTJkyZGeus1CmLLs9Rdv8IUeou3+EKPUXb/CFHqLt/hCj1F2/whR6i7f4QlQUougsWLFixYsWLFixZ4PHjx48ePHjx48eLwut/K2BZaHwJ/1E7OH/169evXr169evXr169evXsLNWrVq1atWrVq1atZvJkyZMmTJkyZMmTA5vfI1UIFuBXQxuyTahDxzn+3naEaFCUHjKr7Z0cAY25W4DZ4BCiMzT8bgjHr169evXr169evXr169evfNatWrVq1atWrVq1by2TJkyZMmTJkyZMjPXBCbVB4EhjESWdcDRjdynfp2vU3Lp06dOnTp06dOnTp06dOnWFEiRIkSJEiRIkSJE4Z8+fPnz58+fPnz4+R9TGVTFByOR1er1er1er1er1er1er1er1er1erzzKO2I2gcOHDhw4cOHDhw4cWB8cOHDhw4cOHDhw6SHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOZCJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJFzmvXr169evXr169evXr169evXr169evXr169evXr169evXr169evXr169evde5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5dI8ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48ePHjx48eRBWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWreWyZMmTJkyZMmTJkyZMmTJkyZMmTJkyZMmTJkyZMmTJkyZMmTJkyZMmTJkyZ0Bw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw4cOHDhw7PPnz58+fPnz58+fPnz58+fPnz58+fPnz58+fPnz58+fPnz58+fPnz58+fPoAhIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIkSJEiRIlLNWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWrVq1atWvbLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLly5cuXLkYAP6AO7hn23LIvWT3M3vMst7B+vrx5e/ZlBrK08CPxP+D/oZWo2jCcFgsFgsFgsFjqxIDI/q0JC2E6Q6uQNNk+PZFrddrzPSK0FGCRVC1EavogH9xSgVxasOrkksNwPYjioVCoWodXHir8xLkjqxIBGNBdEp+ta4LBcqQdXGbxGr3lLDiCLxAo2l6DHkxJSoVC1wy0zVQ4tVYtVW0LzAnginiPvpXWx/eOlaivY/mfa8FgsFgsFguXAOrjLiUaViPxgV600NYu3cBGcQLNnHMCLd6t0Fm1HqWPeCKNMtyAAAAX9w7UT+nbjLsWFoAAAG0aKFHOqwbxgPSUn6VD01qVkVGzxq9DOalKLLfFdrCKMGk7Ri1lMwtOcMzf01s2OduQRBx/mYRhMNNDKvgrWKatcolqXGWPgEreD0n/tgs3WiHzwSspoRTkcz5zogVz21uiELFwRzBqFqw7kpZ74PK/6L73212+vDZSVYmytUmEz2pEOCPLTb0FmPxgFCygOKse06osrNc6X8siWviWy17381t9GwO6wzroCRVlbbpFydOnqWK3QnPbr2LY180iBGBKGn21nmp9E1ICTJj255HWH5H6PmnWWhdE7wSv/jrL4OlAA7b5BJBNMA2+i+QAAAAAAAAAAAAAAAAAAAAAAAFKwHSLbvIIml3X9mx7PjoUS30gxAQRlx7lfW7KBGB7v3LaMKXRiEBO3FNhEWH5DJHyzfg098I5D0+n8oBt6iFQHta9JXQ8agvoMYB3vFfgPyYmRfwBNFD3mZ69AI0mbVx8bcozR0lbRv6V++Me39KY/DDNJ5ikQiIA+em3gL1nIl5Rj7KcwFX9/q10mQbVxo+tboWxDo4s3iwu8djqrMkJmDuIN9r0f6eRkPbO0o2GoPSIjKnB2wKihBTFED/YUmnam69VhOXvGQNe6EbQ9SsozxE+Ax0PfeaTOmhYTRRyLmtSWJVP7U6YJcE7EOwTWTbOs/9zQibPYftSrMAzKEFAC2vvC7BSkNVQ0R6i6GrmtNXBV281fx0i5JEk1QgEH/vSnPEtVoCbTFAgIZ+DotsYbjvr218F4UspZu8Az+0XLs3VE7B4/UaDG1OoQ96GAAD8CZexB5fHSOqilCDQU8NRKtiujsjJHvNNPePGf6HdyZno/gnUzPNA1AALXmEzozCBneXJguHb9SkUznQxFKb+zRsVY3bVqa7eQrIqNDZzBn6k8M1M4aMwO/Rr60fvC7xxtJMChb2D0qWBjQzV840aI0yrDphsWsfdRJ58RGfSGthmJQ44QW7GMfRW0DczfQpX4NUYYudCC9kg3cH2KLaGGzr1wKQjUi1uP0XWGtC1GribJrlrgpWfGK3WibAT5AbVHhRaMLyeuo9JQk3yUJthoJQWKcEtyBEfTAigGu64QtlQFjC3MWBCULmu9R1slHE5mA7dPPBVWYE6wDCwX8nXWcnuEhBp6xTQnaIBBg665KMfjOWwYTkjXexBkzjNkgKUxtLTYO6VnNKU81flbNhdbod8+YoeBWMHnru94FhgYjNu30wugcG7sR6EYSUKfTQAuG6WG5vfff4zx9QDFflwYA+lQo4gZSoIgoJAwc1LdypG+cVr7Ht4xViZiibBUebK32Yg3gwP4ywZrD1ftYKRVmGi+YYmlTbcq+DwurGdc3OAujqiHYezOb/JzvkE1cQvizN1o2OCwR1SnvW6U9lcOUNZiMBB9gxZHE0cvnM7IWJ1yjooW6KJ/IIdj5OkS7Vc1l+pw2rgOv4sAmV4uzX7twZZtP3Ejgch6itSsKUAcFi4yZ71ctpXqpipsyrOCdKxvqiiF5MbkAD5iPm1IUC4fW/MmYxGleN2VtCx2eW/ciSQtMQYHBYINjqujpxf/V4HUTTZaIlG2WTv9tcM/rvpuoiWDMWEOZbN7vWeyvUvpzY/peveh5oubVK6r2Wjh8KdMV/TgpzF5IMHK2Ocuwz/INqWzuvPXPD8FQbF3pzIPPbDP5lR1zgWNmEjhV4+nTfCUfUrriHBqRc/HtuCDdA7oDq6BomPCW+RNi1Zl3Fwo31w2xTHlws8VuelJUJVj0IvSYrBcp0pV8YoluCWTxxrEXKwq9pufG+oRm6CWxDTrtkHy4Bf42chCHTHBsxRBxSfQlDoD2OXPap40JwDjKKwej7l8WF3Hnfk0IMTZ5iVLwjneExnfimEuhNfj4smBGf1nQK+2irInWJzco+sp6PQl3jeczs4RlP9fZsCXzU9dqHxyMMNs3jYbMOCUxnJAk8IRoxN3ul436EnVeoRf9r7Fnakpd+KFMgNHs62+Kf3eFW83gvbbu/jZaMI/76k3VJbAZcezIMY8/tFr+rlLPEJUCN0xd/wtKTu+A0sn8b+S/aGoef0nVx03u2Yar3v3spduyfAixL8MM6nxOr5LAEVX/Awz80nDfviaZCMCgEQ63zch+xVJdR+aakXy7qat1qq1SvPbnjZpmtKwp05S6yoFfotiuAADG+7lE7rBa0aDn8vkC4fQymz61TcdKYty0IFt3C+Qxl+orJXeh/zZDFvdR/LxUFVM6pvHrfsBibZvliCYvB0mcGyNi4sWg/3Ao3U+M3OkP8TfCleFBCZOd8r7i00/fdp/otgaZLjZ8Bysc/xp2iyhT4LQmJHC/pKrHAeNhJGhbrpsYyVPc9NSNs4iOIerR5LzUMmL+an5aNSC5SbR5z727Ep1SlKocfyWZjEQQ4E0lOMgCKSCk5j0dHX8huFwRjSpCszqcM5nX8lpe4WjACdF+7kyOa26FiCPySLR8Iy4W+yS30OGoWqbvKjn/AH3bXaZvYwsTQAIA1TzudC8+7Rvcdb302oZyy9PvIwsr+ZsN2hFZmSuFdwATVOPHlwrlnUXbmrpRhg8SOHA7UOTZ2f0Tig1wIJ9nwaeO2b1JOVYH4Qc4jiLQSjwxk4cjbolKN+V2RzbMfQwlbFM6kXoHnAYZC8qv0Q3lyczdaccWf4Ka0oUZRMEU3zC7USdguTkzM0DK4n/OCNbubEPUrEPWEzS3PsLLK/XmXpZGKSuYlpdD0mF3H5OpbRJQRt5Pp4EvmeLSAYNOXP6FtvbRDDhHyTY91bfX04xsLLLnWk/uxUYg7HPiCNGHoy6v11s2fVJzKqtF0MkvZF6S9SqE+aplINFLSM+biUMHZq58vo6vm2OauUZkgYYBEfcLw+Q9oRMgEn+LVmDSkgTf5uPXZKmnRcXHaHH22y5BlHmV/biXKZAECO6SBgSE5sx0V/mP0WDcifW+KQIyKNbS2AYpreXVuvqjwziBaDJkVBTCoH9BwMsukgbczmIFYS1n/E88nleas2TCZj469Fwjrokr6GAAAAAAAAAUuj4L1wuzBr4ANeUEty1rRAWQAA98OKegz+OXSrc5LLfMIXjYdL1roEsPB8dO4D1eaeuN09cQZM1STJuU4Xev4XoaNwHzuryum3loNEdkwfgvJTTiAzgR2ABY+d+AaZ9eocK6s5S8L3w31jnkXWrHFGBML+4chZ6/k4HXYtk5lM9bBg7VeiBLbD99lVtzzDChneeJx+X+CQfWXuCqpQtUx8p+KB+D0hlCt4PDLt6jO+1owc+7aG+s49Ceq/tjGd3FJBA7uPy944jY8xb45hN/QEqioGF7mf/daYg9LuZUZtDDs98qUmrby76oKJ0PW1d/VypjDysJqe+926kU9mIvVPIAAADVVwjytyvACT+XHfug3t+IuN3PZZ9SFwXG7nss+pC4Ljdz2WfUhcFxu57LPqQuC43c9ln1IXBcbueyz6kLguN3PZZ9SEWABU/J4AAAAAAJ6GAAAAAAAABlVk1PjFjG9jKp8I8CAcdo81w+rJ0QzZdTsHjlIMTCIivF6+54ecBYjxNB+hDXPuUmIPX3DOZrscbjH9laqLeXobhiNV4wh9eCYSz8ffq4biO/L38voUVHkP4Z0ynmn8ZNB0hHf4zEOOZgxdcx15DtyHnwq0C7QxB5FH8deOx8KXzg+Gh1yOA6SQkydlG6BtYQkUGXLYZx0in0TAzVSGIFWUZxUxWFJWc91aOhD1hlq1ijqU5bETt16pY8/bXr4A8sqFVdS90VdbNDlvO2LRRXDEWkoCqaCTa2meY0QKmy/63mqG3dMcRQvKmO6XlQN4EDGAAAB0Zr7v83mc/84feuyZY6XEFtYM7HEY6SdMBTZyPsKq9jvMD3hQgpX2CeSZAS+9Ilh6RWup7PRD80o1xMUDC64q64x2FA0Y4xGWpkeM/pz0Zq0d+fRIrNfacyVig6yRCy8Sxw0BoCwYywAGIZeCCI8PAGYLAA2qaBgiwAF/BYHL4+AoxYAaCSDwJvZhwAAAIQlFTpUxgpS1A8nlPjZkgsDAJH5qukhvXydIed994fOggH8EMhTVDVtu7v7wlKjXi9WEEezbIGNnhgEVtfMvv85OMTHSvGFteDoST2J1nuWhm2fGSadUB0v1d+ugd5MtH7xL7X1gBWY0p/cGFi5S7ufPrZXXw1/ng5xGxUXRb6sIaeouXLb4zm2QfUSc1O5CUucu5e84lRA13hIJziA74lz3tvG0Avx5uOTx4dA+dHAvLCUFAeAODYh+QdcvkVMjbhI0cTcGgz2UXcJi0zYoe0rfhA4ndDRQiqquIrVnGmRjq+XCQVS1jT89F8RwrwWS1bnbbZFjMvwyxZwh4ks0aBiZIXfc/M8N0Du2CTPWclmraBcsfG+OpLXr9Tc+Yq0ZP/5t5Q1gv+dpwtxLLDt9fT3pgy5jpfXsoNcC9+qM7dLW3+2GDYmJR4fiWuoHYo9QdaXf8iAisso6PAXcrwBDRUXuiSAWJGD46WAU2vZQAGCisYsv1P3rSeNoSlIuPe92HFNXAMP3Sp8t1+wAAAHiiyh3+kb8VQWPWI9M7DOqTydZyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTyeTydLBYAgOJAAAAcYTt24AAAA9YIAAAAAAD5LZIz1V5aIR5JR2TuI9wgAn7dWuPUAAABHLkZArjelrbGpqDoLo2qFqqVcEfqpk5QfNOQoCsYgOBLEBPXYOQnUXknc4jGqAyC//gdXbDUdDjwyNHufwY4lVjsvX2yQxB2VCeVhr22pxq9yWnbhhWX/iPpM0BfbLGe5I9Z08Xdamn8/8xen0W8cfzwVrKogWHMfJruMKL0dR8T0czWzc3s/XAmlBKWdmzAoRe8Fgn1RzlVnsDLN40nvgfvJh9OY6uewTufqhgI5HqvK9sK9k1undUY97F+Lyr3M4XEIL2QyawItQwg1ODjsSmTWtNd5YxWkTXfYv7bCuAyNGyDF3hvKa83ZnDpD5j4JYlA6kk+enAE4RTGgol40rRDWp2ZN8LIhXWs1hc7aPagXnEOdDrxlqWPwyb8G2/h5ufuO3oFcOiysQOYyzaLEVfPfsa5oMdrxpsISO66msXWSmfVdZeUmbFkEFfP/CbyYcW3LwDMJYN8i5Bf3BW3m3VZNr6yw7fHT8MDPTYB3NjukSN5cVRdaNK7YBk3/vu5TpvDyEB6pmdBl/RwzZTtYfvYCP80JWxW/GLTKGF6Rhl2TFAjsAAABreajb8jK/nT2a6PaZiGc1u46XXk7PWyY3vAAAAFo7TNu+VTSE0ADz3VxwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" width=1628 height=733 class=img_ev3q></p>
<p>Let's begin.</div><footer class="row docusaurus-mt-lg"><div class="col text--right"><a aria-label="Read more about How to create a LinkedIn job scraper in Python with Crawlee" href=/blog/linkedin-job-scraper-python><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/scrape-using-jsdom>Optimizing web scraping: Scraping auth data using JSDOM</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-09-30T00:00:00.000Z>September 30, 2024</time> · <!-- -->8 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/53312820?v=4" alt="Saurav Jain"></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/souravjain540 target=_blank rel="noopener noreferrer"><span>Saurav Jain</span></a></div><small class=avatar__subtitle>Developer Community Manager</small></div></div></div></div></header><div class=markdown><p>As scraping developers, we sometimes need to extract authentication data like temporary keys to perform our tasks. However, it is not as simple as that. Usually, it is in HTML or XHR network requests, but sometimes, the auth data is computed. In that case, we can either reverse-engineer the computation, which takes a lot of time to deobfuscate scripts or run the JavaScript that computes it. Normally, we use a browser, but that is expensive. Crawlee provides support for running browser scraper and Cheerio Scraper in parallel, but that is very complex and expensive in terms of compute resource usage. JSDOM helps us run page JavaScript with fewer resources than a browser and slightly higher than Cheerio.</p>
<p>This article will discuss a new approach that we use in one of our Actors to obtain the authentication data from TikTok ads creative center generated by browser web applications without actually running the browser but instead of it, using JSDOM.</p>
<p><img decoding=async loading=lazy alt="JSDOM based approach from scraping" src="data:image/webp;base64,UklGRoYlAABXRUJQVlA4IHolAAAwdwGdASqABIgCPpFIoEwlpCalIROpKNASCWlu8hrke3XPLgHb9D1Wds7uP6XZF5P/t89k/4y9YOjyKelv5OR1wufOzmP5V+Z/rP7afDhc37F+Dvl1Ym/Ef2383dR/zBv0Z/33WF8wH7U+r35xPqAf2vqT/QA8un2R/3M9EP//6076W/3vrK81nzvcRey/PmfHth4ATxe0OwP8JP5HWyzxX/B5Uf23fH/vGHpLm3ht0692Id2Me7EO7qpM1Mx7EXjVX3Z+P0sNbmhaq+7Px+lhrc0LVSiIMc5QmllZTyQ6/UOl+HeivFrbDtSRugEP6BUI/Io7XtSExQKhH5FHa9qQmKBT9v3rKxJiZwlqpq2aTj/S243PsvniQmt2Me7EO7GPdiHdjHuxDuyVHuVr//+RRytVe5Xp42cEzTTpdQSRoMelzbw26de7EO7GPdiHdjHwUEH5RB5AgN7KtoB81ZAWCumRCeI3xh7nviHdjHuxDuxj3Yh3Yx7sQ79q6AzCmFssXeWHQihJQMLSDfUzVv8EH1Ps7sQ7sY92Id2Me7EO7GPdkIZfaen6Z1VcL5TTSj0cqLr1u+4OtYk6ViED5t4bdOvdiHdjHuxDvm1QJX5PR8lvcQmLDQlSVaSOyXUOS/TMD5t4bdOvdiHdjHuxDuxldSJ2EKAemmKo1uaCyZt46Gb5NF6sqSMPXuxDuxj3Yh3Yx7sQ7sY9/J2udY4xJydhkoMkml2RqBya2890xTSMqaTuxDuxj3Yh3Yx7sQ7sZVJ3XYB/hj4x16Cm99+WZUQdKpzXUP2cG6x2GPjHYc9lFTmMAcXVRZfYQeZXe1sTviHc30Wxu5gV6mbU609u7GV4boVQJeArWbeG3Ttfr1TZeVXjT4U9hRTGGyBJIoxrwpJKpweRHnFxFeQTf8w8he8rkWTNH5sca9IyHPYvJF0152hdyDaZvwU043TndjHofS90evVmMIWlSxyvnQ3HQWYTB3HQWf8oPF/K2CDePonCcndiH0L+/RR2NymLtMHOM5iJSX+iSgVYOHOzVVuGDXqwnw5Kex0udSaJPKhnatlyRm+zKn2d2G7FzF6m2YX9R7gmf5n8dePVSRoF92txlovLwtoxJ77P2Vtw6Bnr792IfRGc24UVNve6j3cXAxcJSdl6VK/+uvkGZD4O6SMPXuw4XOEJoc19BltrDUhMUCoQrEn8cT/Okrh+SYbdP8JsZ42ar/8kKP/tHILN/HizKI0b3Nz3xDue/jGqxXt1/kN/29L18Tc2JBXfOt1P4g9nUSgY5ht0/yjoDtBWxvM2sbdRoOElJylR03Av7+lUbO87rP92IdX36n00Hqj/TrpJ3Be5xiKftRbGHW97wIGOYbdP9YxipMOa3+D7sCoaigpQRxBxBNBCt7zEpICIQPlfJEgIjLd3UxVAmhlyqWYHGaSSZQxn5DDjJe7TBigXjnr/Pv0rEINL3QvSfoRmlBJ6uD/9150ca7asaQOFeBJIGkz6M4d+J7Ew26cm8oYI8NNLq6oIy8PmiT5pPg2NNaKBKC9Em+ffpWIQapo9KHsuGt/lRrxFyQCL9+zkp1MsiIw/WSSWcsRzmJPMHLDJvlH2nrb4pmwsSoXtD7TimKNyssrKaQf5vFMgaRK5gPmw0hDFUpb5BBUiaLyf7yYEk569D2mT+QHiHdrat3zD6LEuihoQVvkC++dkNjpZ8xmVFIC4dSjR5FnALGaFnjLnh48CBmJ+0p+yvSZ8aE0q4rIX7Kyt8j4xCQL9bjhgLVVIX+2ha9CFYx342uQKS93RSV+3Tr3YlFVabmIJRU9643pKXWswn9vuaznftXMRbdS2P0rA1Fmag1SgoQUsGizXtsvHqz3E2w6BYZfJwUaQhiqSQpjRhUUpJk6KIapxeeEHn4Nq+Bj0RFBEPte7opLQUgQPm+HmBPyguBszs23OXDabCM6AR9iqpK7fTKb1fq05zou1NLEeW9BvMshQFgg26HeopgJ1pkhkaAIBddmwa2IQL9ekhttDlwnUn6lR3VQIBJ3RMUj0XuzBufWvwzrKmobJaNW5dIbcNRWlemrWNjFZCCAFwZmi2SSwbcZ4nDSZsH2cJjMjJ70m+KJYuzjyD7YMarAJrG2Zl9FgfEwelUcJEULEukZz4WjqATFJ6UkCWIghjHVUOo4AGZc5DzQIJkY0s5BMyfnp/ZTP/DhDNww/ar7oCnfwLnouFAuJ9qpfESiusYDedgzq5A1Oj4R/+vbFxvgkD5uf2jjmBt3Wf7ryeaj7QqOnKykB0o/mBKz+6cGgOF1nDBOCsW5sxBcmD9hvwV6S0yBA72Cu8i8csiV0e2a5aq7+K7EO7IAjwCT/z2gW354UuFWvbCrky5qSFV92fz7bksyxXIt6Y8fe00B+yR8nQnTI5LioG+hUxhF6iz4g7qiXOWZBCnKu0Nd0x7tdRJzUkvS7wQesdJtrcrKNy3OwPpJP9Q/XmxK48psHdgf9G9UoY1bAQbNXOnBbeftaiWo94GPRxZfiobF+VRa9UWZTAcW95Qu3fJIbYXKsFStMOOUEqS7F9SXMxnsOQLM+f6ifNQb01gjMqy/V6o7Z1HPKF3Y9hYkYbhs/jhXNwToPGIryKeaku/9LlPUD/nWgvhsNRzhA+bDPQSPvNxhKWiui6oBdGffmgl+ejplEWb9ogEw0edwPBhp+3214zZM1iWo/IY3WnYZ4bi2/Nt3BgpSYFo9BSKBnZ3xzXp1YZ7iI7Lrp4BU0v0HEDVv8j0fn/n5IWquhoKnZ/QY1Q0IEuWP4FzcxzYIOYrEiAblenlIt0pchSkomdSaJP92txlqk1XLCF/iSocc7oEm9LcAY9UZsepi9s+qymtS3eRN1QNKChNZ/vhTrU5t3XpRDCXnAVQklQsDYTd2OBcbckhm7+dlQxRWc8gyNq8J7Mp5DVcVrNt50MxZNB/QDMzZPEPHOATnAOE5iphg4c7SW6XK7yljOJN2QonixK09Q/m1gj1V8WITGLPqq7lg7ROYa1yNLs7I3swpdpa7QZRpCoUbKN/0YCfikbvfnCpVOKItMzKHU/d7OlfzpSULSdZIVMTYT8cAvN4FcKGQCEelgbZxvC6PTMSyHX3Lf+hHXwzH5xki2XsbJ2fB54SanofQ9+Vx4Xm8n3Dqivym9ewr9wceYzOQselcvpZk5P2xp3z0WrhUQrHlJBnRhjf4v+dDviKUT0G2fKWd+IxH001O3N38cpowikOEWxqHAPVdnmLZa/3Oxm7HFTT+Yl+SyYWuy12jhwzAQK3NC1WGvEa5ZL7u5eOjinRkbIYzP1MoCPWp+ifZdyc+dmDm1F9DqI1S955dgBb08sNxpVm2Kuu7l2YzNp8573sRlV27fi3PGKeq0B4lYWd12IMmrQ4LceRgdLLvG9qr1kGX0sNQt3lflHrarYhx2MwqQ5nmrc+nhM/PK7vnBnkZdVmGllOMzQhEp3AmUDMOngPIYP7Upmkc8ky8yBPi+i21mZ/A4Euos0XHnU44+7PzWUyGVPEjK2zaOJa9mA6NhIgeKkI8c6xfuV0RCh00UUseY8C65NXkP7QzH6M7ta9OTqtv7jwrG2Y0C+GHG3F0wX5N+slhBAO92Id2vole9+wzoxP7GN7uWG/xQK9+FDEa7l9J44r0WcMcHHpUrOM3di1z6IF0dKsKSDsO5PKmzcrIwBoi9xNPg8poRRUbY13nZ0rW7pV48xxai0JnqHNNSPWodFDEoZaaFA0bfvhjpj0hH+q0gQjJzvjMoBdIEDUWp+Xae1EYROK+CcMxhbpmSJ9787oR7UsW8z1VZvrt4dAaVkMb30qVHuSW4hCA33LfVQscWL3wZeLSJKoRND0b2qStKuWa/fiY/SpP8B9osXO+tmgJ6AEqVCG4shzfJdos392fb13MPz4DlmojevdiHdq3JKMVshhVEHLIsALp4D50t/LEN+7sc3WFn/JpEkndiH0yGikF35Soeoj2pfhZpc6kjQLn6zivOyomdqXj1UkaBc/c1XPXqU9iBc6k0Sf7IAP70qt+Fw3yknAYA7LVKgOvTmhueEKfCy4Bt60SjxBgq/nyMoMIFeUqAIbvDD5idTJMUs6k2/U4apNwXa1R0vZcc/hbckkBk4CE5CEmNQTTn71Tx5eA1SqJvM2jQs8Th+R4xZ4nD8jxizxOH5HjFnicPyPGLPE4fkeMWeJw/I8Ys8Th+R4xZ4nD8jxizxOH5HjFnicPyPGLPE4fkeMWfk1JaYSL2sBIPU4lijrV83LuXubZbWNMbtYlJLyunBqwxhiXPhSHECu2B7Whe3OaN1B+AAAACgZSXguc4Z7wrczIZWB1dN4az8IAMYBTFGnsQFxatGRm3E+X5qKKmdtvbSFpnvAAAAAFAv7WwYxOZjj4JZ2xNprcvMngNbZC0yXcZbCWf+Ub36jieerr1o6RKjq4H/c4t9JUCBFW7PT4AAAABGc+sdqd/R2v/yRSMINAdmonAerozQP2B2tKdTLGRb6jWgUNFh7cFEjAFGqI6AKVcBAAAABDwZCJaZuPJuE+kF8Z8OA8b2KoIQmNZBKB21js8MxLaTXHWdQ4rASXyKrW3OBHWwdIAAAAFAUIX4IEmthvVUdZ19K2AbHS/1B0ThS65THEhBrfG05eiKuw9kj2PwpVeNkzgzqi3xSNiAAAACezDFfm01s9DrZ4vobHKFkWGP2DnB0SHl9B/C5qd8iMxRRJldNJw+IV7ifAPJmrY0oTKkUW3YROS0wAAAACe+wsV7E06xgCYcnN+dtfIsjMF+a5bBP4ANJshTFG4bJTYDIkv4M5UkIJgzEBXcHaIicsqAAAAAE9l9jzY8o5MT7/nINttrv5BV5BxXCYEJ1muPR8Eln+KvCWKefQ/L06OYOAAAAAise67IcdeSlQkU4pf8mnbi7Fn9q4zMy2sB005Zx71UVo2px4Fx3U6VUjFskwPToe2v66Bysmma/t4ndDrQWyZIJ7HTqsmcpmcpFkGoFig6aHacJg83flHnbbABkn3UAmueMH9Y00cW9lTLUfOY/YJsEIvnVcAdYJ8HlEIGfKE7GmtcRytQwd6SBo0/wR9zh0zeBvn8/A0K1nxq9njkTczwZ5Gsgpt379mp6QYzX+L/ZXyAoqJPP9DItNcsUTx1Q8cvUbzemTP0nkugv1WqyvTSWhlBWsdb2RIegwnl1iEz2i+ReqjjFop761Vrh3s0ixRuOrVjUAByHHe6lkvzRC3Z7sfxIdCNTw85wGXsV8LBvct4zP1ANxA36kCJtxd2oNdxoIs9+GSV0L/hU07CPm1eg1T/gWi5aPuz5QZjAnJjSZrPQuI2+14FE3EiVVK8o738CeuVsWYZV0G5uBAdKxks0CBjnTyTZfxbvjfl0ZRw489XB6RLPlmMA+eLREaM/jfbiWNSLS1QpqaFguiNcqLt+0mlwpY43mE4TMoB91tcSc0JynT1Fr/66aayPSu7MKlR6Sb+/f2Mtc+/enAcrnEqmtOt8xpPWcR2cwp7ItGs61dxpHie/OxEGnj+M/79bbj/pUEVZlSpqJhOKFxi3ideQ0goKlPy/DWQUTXzHnMEu7DxrobLcthlbblJm64fOshLTVyXrnJOWmdSG5fWItHNyAx6BjYReIiZJhcZviw/vkPNOVYMRjdUHFuPRByQpbGsfSf6ncUJi7oZKSMTU4lWKl4EXTWoaHAG7LmPb9V9d8B066Ap+OO4jpuwLBxajh+ngn3Trjm/pOy8fJchJT/JbcclJ2L7TeLeSbSxw8XG2IYnNu/gWtXA6fFtKMgMSvAStWv+0NO8l2/Oi7pEjTmPJBOPLUPuQKF/jDMn+iM/W09qEBe9Sxw/2ZcSi+FyC8fDw9gnkn0CRqwKcarOPf2EtfMpzC1SC97iVj+FquFqdQgacW51kXs+L+KvcxdHdiZ0oklhBtItEYFhMWtJ1ZSMbneoape4iSBh8Nl8n0XAwIzWE8bG9wY2yt6tCLFBbfoQE38A9KukJ0ueJpMRYmCcSlUXhAdX59oGlkxWRHZ72ilX6zOPE6JA/JshqGTqIxO/BDFckaFSQlCYX+ey0+u2tAIXc/IgbfF77X2256/s5xCJyylXXwDZMHbCW8fp8Lj/ZHyOXKka8GbjnpHkjFV3esGUsN3mDt5bOmc36OHS3dtzo9787ECrKVLjj2Xj/7/oFJSUNEi2S9vmZmJRT5D+dGI85GGVuQvrEbfaaCZGDUMD7ZAv3MIaxYxnYBn0ER+pLRYTFyLuSIY5fSWg4kCcnuupFX/i9MAEj93r1TrUQP3VCfK4EN3FpC3XNySvO+hxo1ehcSAFZrQGaH5MWQDKwyGtv+ljyhObmHMBcTj+LxW6iiQBxIHBVe7/fXCnLkngDsNvMMv7Ve7z0Ek6ZBBX7pNdXDOpByGkutTe9Sra/40XxiCIk+nPCcJPWbBTjYpr9ruFb5o1SuU0Qv5n8HYKv/nRd8MOrHI3/8qviuR73GgDBKD46Wq0dKGEhPe5cg2Cl+M17WRDitU7eI5nWPRGX2QZgkypLG4NGYHmxDCubFyhmABvF4rb8NdJ0fD6azOW/PtZ3tSkhrolI+gGXKCDUuZKwpNmZeLNR64tKsINJv5iYYp8XqeAwZQymTT9TcyquM+57bA8QOaZa8zvx3pI0VHF+LNrXex8R7yf2xGYEJlkEdY9aXGW2BC0qE/K/UrAoa0xmvGHDq5op/syk36LHKP2ybAjyvULOvYfA+4MkgsB+w4W4eVk+ZdY9TdBx8YRcDwoqCP3ngrBipb4ZbAl2HLZtKhXl2U3FABEiUcyUtvps9GuykLj1pAn45sZsymBkQ6Dzj1fAH8VSj+b0T7bCZpfKE3Tj7myMW5J9DuGdZ+QBR+UA01hqVrroOOgSRnEDyc9AhoqdbAVfv0A6b6ho2JPs4Hp3QjJvmaDH8XR1aibcTFe32+5LOpAi75dBPs5ljubeb3//TWtSLiPmlE6VBBGamhh41AZTEsNrYZWeHGDSGbw//LkD6lkp6QaK/4yG8Fl4voW+snkwonBoAIDKhjudqXFPLtBt0T+4lP5T+wZ/9mVeCq7CdS3syGmi383vVS3ast0nnpmp2OUYConOWp4a6TXtQEJQPk06Z1xPsvAMjNslqCn+y8numhXEpwL1z4fSAz1IcZ14C7GHnp82Rci4A0SNhqOXoH3DPRQZTOBxEUlb7I8XqAbz3bktt1BT9dkNcQ9P8oD3iqzXS4tyJo2Pj3WQ1RSl0f1XcsPNv/QFFw1oi7DRTzpqb/90wp5anJWIt5ZocLda6GN3d39ujh2X6/LY8fj5+q3sX4FYvW+oEcZ+Uv9Jg96dtbdJdqP582wDp869YIEgT+CxNCxNCxNC08a5zqvqU1vpVcY8JpdANT2/dDnLZuVaI31h26KszeFcmwJQYiqvnjuQdbLB6X8KnY9JsVYEikVA3MDZETgKGB2lE86NSdVwHokfN2p+RDrChCCj8v4zbu4ouQ+1NJ0Z2zIXc3ASoZ4SQz6sKAl0VgyPQMMXx2hpD+0b+R+KLLke1s8owlN4VYRE5iqCXx1PVBIv10eSFeY10Dc27kvNBHHPWdYSR8eZ5CNtjvh9/MNJ6u2YToBN/T1dOnX0VgPsO4yKHbaVSTWfJfU/JALE3dTnrNWLnWQz2WbMG4Wdt8cyO3m34Ah4nZBz19j/X5s36I3FWnhKQlBLE7RwcTFzi+huZoVvLELYjO8Z5lFp02279SCW9jD10ZXH0agtoxawa8T3Xsg/o5CHafdQv1FnEDqMwS/GaiADDipfU1wagstKLbhCeZrga8jVuZ0ZrVxY0WIb/RSoM14pmW8E6vZahRGg7oemlhGSFdC9HGuOljaYoaVXaiWKKzeGGeDE2dwmYZGDwG5OLB+YgWo9MiKomEqcmq+ykuvtQziB+JVAtgpc7czSq7O8L73B2Q9Fre8bC1sqsW5qzfHU2/10dJhcZnXM812+4Q4WAyvFPiPjFRs4MPkyGGwQ1bn08XmxEB6Cc1Yut4noTDL4EoCjYaNgWOzkF8gN8yEKiPNPLPls1j69bSTI61YlBrart2YHcKRqAN3loZOjzTJXY60+yrdO4oPMqNQKEFLY/3d+Udftrj8jV32V1PzNviY8IKUbCxgF/30KqHefjD/OhQ4Y3qeaVN8U3V9ced23AyPldtjn6rwt9+F/X/JvVj8z1EwNM/t7JVR91nSlejns609ZOBsilQGCDWbipid8RVKxgb0wADr1rcnWTKVcr0rObCAlUJQko7FugBsncvg7g1hYGyBPsQ7cCCOCWgtMOguCeU9rk2M2H+xnqW3cChfe9jFImpHcrREOmwf7Rl/hPldHY/1J8ol9y1UhCtNm1RYnFvC1YxQGk/tFxyGBrS86qwpcrKzYYYlt087ttysxy2QSbNUBESAEk+YNAvfDD2FddBUC1xFnARYI92nHS+lcyexLkTIwhYR/MPf4EHbI472EluNATMZNbxaszNlchIW+TQu7BlD3SLvkIX0UVRAUCp0lbt33zzNgUD2vXSTzejc3GFtdwdtpyXs3fPZvj4CSDwA30LTMRZH1HKQoMDKTHmn1s8TRhe+3bYTvFwl+rpbHDr42GhjZWGhkXsw6+XAkOuHt+863o+nMY7yEI6cMVLPJ8xd/pg/er8aAgC24pfTwHBFmOQbMw+gTeeU7kXx4dWEBS5vcBHr4nUd1gAg1BwAQf3skpo9EKRUjakytqwPWeNN8fl4gS1nAP6UnxKN+t6zOYaJkf4Ins7OcAJljs6VU0/UlXXBK5OS7a9ept6JYorNtheoez6IjQ49NSlRET9POE59rEcvbjlkWeAwuaQ0gYQuCBq0LsFspq7XKPhLnbmE1Ie0qzc4m1TTOX6QJ8Et9HGXBujGoN5DeZAQ+AfT8CJWYtGReUGf6wAPUb4N0negvGJoHFfEpeTNWCeMbAhTvMqcbkPzEAUEpF9AkwsR9BFOnzoZD8QfKG0BvHHGwc93fGLCIz0M+hzqC4D+AeDbq0Knzb771+FLhOgRT38DVYbznTIGLRbSSlEcLfvimOqJIpRCLwcUi4Adh9jyVnPUO9p/ZZphvtZ0wqPN8OGXqPohtMn4v6WawiTy5LW9TLoxKmbK7RTcP/e5xPfaoAwOuEgJMWNFgEJx/emx6kT/8sXxfBk+rqoMonvmynxkXvbq5Jighwe7STRAoqJagdBQoLQdyUjisWNHpzLUWKMUypATAgNvenjVdGP9XL4Ihui5IArtRT1L7Z/rkiW1Sqdz56+9v2ik7mtgRNeyyRUPa6VTcefdbxIi9BRsoHx+yRsfJp55pdjXJU4tfcAb8QvaT1ibKNupF10KlvbAteUVrO7todbx/qPp7Xdp2gCuu273OVENYbhdDC4jJBfe7kWVcWHn/sdN9QEu51NYYtUqLqs0FnRAsAMU/FgyYGr2I9ixne5VpWEsEzEwt9BX8uNmcKSGvL/iVKoc5/d+WQi6ToX2KM1eLoJG8z5hG8odHMAALXr3I5vwXviu+7ZeReJpb/BKiJeX0/Wozv5u3HyeGzSov3UxgsDJj8vqIpi8eSlllb8ObgXa3mNCX4y1GiZ5kL4ve8IN/4DgRQSis2ujmhHKEHeeL32jzopGde0AuYX/GeQtqLuvo9PORQm4q6JZt6cIoMC2yGx3WpkXR6OGybNeX8hxeUooojNzeTZfTDoNRLfitCMtukF4gsvMXTVcemhUkdddPDyBICua6Vdcrw7BIT5J1fSsD66eCDHnM5cQQoveFUNnOeiGVG4338+IX1Dmn05vw1Sa+O5fK2V7V4p3UDIlm0VdgeeaLB4dqBer6mCOXeHuz+uBQfeBcfLK4DWaZfhI+jfQ1ZSJ8Db4h5zu/j5j3NGpwTDAUeSeY/HY8L8vmyQsTqZrH2kS4dSQGzUltko4sKs0zeUTqA3ZjOVkoXrnasKNWxbm5gs1W/10tiX7s+dU0VQwSd1SBDDdhHH89VJFChvw2gj7iYF/fjFnCTdMNi962mdTe+5282iskn/SvDUPXfsqQbg8suRWdBUvK/WvTbI8qcLuPx/gUI/G1bUAPWeQp/4iGIpdikW9hJ5RxVhgUPdBcXz7Yi8mYl8Irc1T2OkJP0nQGI+moI0c9pWBoG3NhIEVVqV8Z8h6/wEM4Vxhq7AxpOm2URCLwVsFA7vBhXegJDXwx8AOAfloXAYRAUnd/94bY0U82hH5EwnQVvFtzlclIWT8YBieFHc+eV9C/a3xnjT4DCEuuHUR/BEjOVvT/2dznReyGOL11jFYIJzOwkRNoiUIF5UbnBe4258Vc93Cl/YWYa3pS1g0OHrobaOWESgvAQmCq3Ug+c6Zk8aNNN6wcsUP+nW8z4OZaYR3mrMcIXsIrKl4NPP+MuyqS5dxRsVJiV5JCkZgjeSxnGNBHSbr2Y8ExR3W1Yl5vAGY3i3O75KyZi5R8VUtNGjG86JlhxebOKsmHcjBRlwxwsY54s4eMLxn6EO4L7QuOsOL/HB9f6CJ3W/LjAX8pSMkOQ7BuKMy1eSRBfQsavOHMz0OAZbeSi1aLSFGo4g08tPL3uIRswxqVPI/p75nE/PGps/ItImobbbQbfdNYmp5CE6NaGbf0bGqzaJmzGZ+Ty9C1i9RTmlTHGtiypNeWbpQg8hQM/gu+xWu8+IzdXyqR+SrIN50xFANydpLWxL5Bp03Nh/NiZtIJ+ULz7q1BisqyQ1v0WG2ZWnn5zrmm7TWlKQVeBDoCjZjI0RATnBazmbQlCJEsiFBSktaPdnNlkFjwjagJdAEiulancMu67iosEnDprowzKoc8FZHARaqXdHgwzaAcVEyRgJD+o5Ncdi5UU+yKjhuhFjGwfAAzzmsGfvnOWGOUGdeF38RTDNpD+GYlyjXaPj3ZKHN9N2OSjDoMKm+DIm0R7ElCtHcPgTQhaXm5YuBwVfgY4J2NxpOy7IWrQzqED3zXMDhlso9OFUdLVLP2yii+tQ4IIxE1ddKFXpUealUbNTjMrm9kjeWRi1TX9E4mCrQfexWa9nqgdI98nHp6+9sx+ZEZDkLzzTFNkhUUOzSWlNGhc02H87G1ZvOE5WN/AwutrCAFmVM+917ex+ilY7sqFTK6Y8psgLiChJiqxh+TRi/XGDVdK1cT0Td5Aggu+wmMJPlIAxCfWhFjxJCgIz/v938tSqygvglsZ6maEeWYyGCg8OwqY459ATfDLA1vaUNg5aH8JUKo4w4+GvuEu9C1lJjrjQaW1g6Fa2U9jdtMTPyxNUxEunpCgk4KdA9teViVBhVJ1kMDP1tOMNDUHvVO6WlUlSw9O7ofsg+H8/wdFGVieKOjHkEpClyjlc8HxUre66KyrX1+NaoS9Eskg94XASK2+fT7wsQbC81P3P/zaSsTMM2dCzvRcRYuYHqakiHnZwnB1x70qEJaazTBGCbEFPJ/KGojkE1DlSI2xBkQF7COEMpBP26WnW/f63impb483ZLZSJGJ10FOQ5hFh7AXrDgrwUiEOiT5OoNyVRvkUKmR1cgOeQyFY9REftEZEYqtPJurl0LzekQV/k71/TUuPW7z7RMTAbZGqB93pru1wl+lqTrQmPeRxRxNcxsJQuXjjlqa8CEcxEWEhPHCWm2dJwlptmcskqythSYCYmx2ouPLXfy2Z2EVU0d6Lb3kMVl5AWce0eE3/ou4Ir7sW8okWwRKqBUbYF/AXrIqJzJT6s/7rBSFJb2o5RyGMxlihNHOc89REmYTdSs19CtC9PlEfXXPJBkLDo+NdaSMZf/v6UG4FqyRyKGtjEHtYA/6Md5r1bHys/+2V7INVo8UBqC6Ebnqpi6xCB9wXq6EGOhadcdOjISVdqzovg93PVPA3C/kjhVNcprG+qfVVMSsp5+F69HPsEABj0wBgZQ0kAFNWwmlB/oR5Kc4Yck2pjKYcPYRJiK9uYI/rsa8WeuUMsY8OowF5VdjuEYJ0peImt1VTBmbs1n3Z0gQfAAzgvA/qAAez+1j57asGP3wSXFokIsT6y6f/YOeaKlgPaFQPDx6Ptf4BVjH0mGfLqJN0UK2Gx3fyfyPlGYZBgUHTSdNUcgUY7Gr/K7NI4Xs5vMmG6cptvsQjWqpSqcHsAKDEAT9rMFlUu/N8xyyMgkHcADe7Pl18uuGi/7sbj1fJU+o+eAibl2YZkylocZCWrBQe/h10Nl2Kv2kYiRSeMK/vbp+hVtNuOy20D0yBR3LyIyIrya0JL2zo9cDhHPDBic92UfoytHDTFFAo7v6LOZgwHzmz8iAJOR9MG/qHjdFWRqqH9zBap7bZKfuRm3KpEOcIoLaS672lVzwEzMC75ABdEjtg2OjUQ4SYpcC+1azsFAeAJMK2Zi1T7iNXB1LgxcyblWAwiNL6gLouXLMxlj+1r2wr+MGDFGvWLwyJ+obzAYRBhO7rqrkt6+dcaHBpwdjvFvAYR6wg4Ciuj5+aGlOmfYCFweMWKRgMBhHrCLMnFLayVGqhQnhvAeHJ0fGS9FGhZEITOBWCfrGr7sDLhdpI8YBHFNd5IMwLyoZ7AgdEeGoGbvbQCvU7bzhHJ8joc7pfD/qcyIFoJ/4ADzmBXuPOXlt580YtHtyK864KXro/ogKyzC3GbNw9uKfCprIStLLKgVoxuTk6f2BQYPOToPpxSy0Z2g2qjA9fgtJZmUSQUDnpR9s1pNwq3To11J4W2gzzEWzVfAccvB0E77GGBAC/QAAAABYtF5UgQd+5iBPX/r7w7aRkWAzqRkHkiDj+cAAAAbEw3Sxc55BkJ/yQCl8XZiiu9tnkC6Qldg21zcIyev7dv8nwoPQMSjYChAA89sfRhAZD1PEJE/mFwQJQhYTCK6aBfHtU04PwEAAA=" width=1152 height=648 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col text--right"><a aria-label="Read more about Optimizing web scraping: Scraping auth data using JSDOM" href=/blog/scrape-using-jsdom><b>Read More</b></a></div></footer></article><article class=margin-bottom--xl><header><h2 class=title_xvU1><a href=/blog/scraping-dynamic-websites-using-python>Web scraping of a dynamic website using Python with HTTP Client</a></h2><div class="container_iJTo margin-vert--md"><time datetime=2024-09-12T00:00:00.000Z>September 12, 2024</time> · <!-- -->13 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--6 authorCol_q4o9"><div class="avatar margin-bottom--sm"><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer" class=avatar__photo-link><img class=avatar__photo src="https://avatars.githubusercontent.com/u/34358312?v=4" alt=Max></a><div class=avatar__intro><div class=avatar__name><a href=https://github.com/Mantisus target=_blank rel="noopener noreferrer"><span>Max</span></a></div><small class=avatar__subtitle>Community Member of Crawlee and web scraping expert</small></div></div></div></div></header><div class=markdown><p>Dynamic websites that use JavaScript for content rendering and backend interaction often create challenges for web scraping. The traditional approach to solving this problem is browser emulation, but it's not very efficient in terms of resource consumption.</p>
<div class="theme-admonition theme-admonition-note admonition_xJq3 alert alert--secondary"><div class=admonitionHeading_Gvgb><span class=admonitionIcon_Rf37><svg viewBox="0 0 14 16"><path fill-rule=evenodd d="M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"/></svg></span>note</div><div class=admonitionContent_BuS1><p>One of our community members wrote this blog as a contribution to Crawlee Blog. If you want to contribute blogs like these to Crawlee Blog, please reach out to us on our <a href=https://apify.com/discord target=_blank rel="noopener noreferrer">discord channel</a>.</div></div>
<p>In this article, we'll explore an alternative method based on in-depth site analysis and the use of an HTTP client. We'll go through the entire process from analyzing a dynamic website to implementing an efficient web crawler using the <a href=https://www.crawlee.dev/python/ target=_blank rel="noopener noreferrer"><code>Crawlee for Python</code></a> framework.</p>
<p><img decoding=async loading=lazy alt="How to scrape dynamic websites in Python" src=/assets/images/dynamic-websites-d9a83deff0729330b2d3de2d1481cd6a.webp width=1152 height=649 class=img_ev3q></div><footer class="row docusaurus-mt-lg"><div class="col col--9"><b>Tags:</b><ul class="tags_jXut padding--none margin-left--sm"><li class=tag_QGVx><a class="tag_zVej tagRegular_sFm0" href=/blog/tags/community>community</a></ul></div><div class="col text--right col--3"><a aria-label="Read more about Web scraping of a dynamic website using Python with HTTP Client" href=/blog/scraping-dynamic-websites-using-python><b>Read More</b></a></div></footer></article><nav class=pagination-nav aria-label="Blog list page navigation"><a class="pagination-nav__link pagination-nav__link--next" href=/blog/page/2><div class=pagination-nav__label>Older entries</div></a></nav></main></div></div></div><footer class="footer_J6tW light"><div class=footerTop_DGOU><div class=footerTopRow_xbYo><div class=footerTopRowLeft__jwX><a width=120 class=footerLogo_Jxa4 href=/><img src=/img/crawlee-light.svg alt="Docusaurus themed image" class="themedComponent_mlkZ themedComponent--light_NVdE" width=120><img src=/img/crawlee-dark.svg alt="Docusaurus themed image" class="themedComponent_mlkZ themedComponent--dark_xIcU" width=120></a><div><button class="clean-btn toggleButton_yw5v toggleButtonDisabled_BJd7" type=button disabled title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)"><svg xmlns=http://www.w3.org/2000/svg width=20 height=20 fill=none viewBox="0 0 20 20" class=""><path stroke=#555D76 stroke-linecap=round stroke-linejoin=round stroke-width=1.5 d="M6.666 10a3.333 3.333 0 1 0 6.666 0 3.333 3.333 0 0 0-6.666 0"/><path stroke=#555D76 stroke-linecap=round stroke-linejoin=round stroke-width=2 d="M2.5 10h.833M10 2.5v.833M16.667 10h.833M10 16.667v.833M4.667 4.667l.583.583m10.083-.583-.583.583m0 9.5.583.583M5.25 14.75l-.583.583"/></svg><svg xmlns=http://www.w3.org/2000/svg width=20 height=20 fill=none viewBox="0 0 20 20" class=""><path stroke=#555D76 stroke-linecap=round stroke-linejoin=round stroke-width=1.5 d="M10 2.5h.328a6.25 6.25 0 0 0 6.6 10.371A7.5 7.5 0 1 1 10 2.493zM14.167 3.333A1.667 1.667 0 0 0 15.833 5a1.667 1.667 0 0 0-1.666 1.666A1.667 1.667 0 0 0 12.5 5a1.667 1.667 0 0 0 1.667-1.667M15.833 9.166H17.5m-.834-.833V10"/></svg><span></span></button></div></div><div class=footerTopRowRight_dEd9><div><div class=footerTitle_tbhc>Docs</div><ul class="footerList_lK9E clean-list"><li><a class="footer__link-item footerLink_YJ8n" href=/js/docs/guides>Guides</a><li><a class="footer__link-item footerLink_YJ8n" href=/js/docs/examples>Examples</a><li><a class="footer__link-item footerLink_YJ8n" href=/js/api/core>API reference</a><li><a class="footer__link-item footerLink_YJ8n" href=/js/api/core/changelog>Changelog</a></ul></div><div><div class=footerTitle_tbhc>Product</div><ul class="footerList_lK9E clean-list"><li><a href=https://discord.com/invite/jyEM2PRvMU target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">Discord<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a><li><a href=https://stackoverflow.com/questions/tagged/crawlee target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">Stack Overflow<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a><li><a href=https://twitter.com/apify target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">Twitter<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a><li><a href=https://www.youtube.com/apify target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">YouTube<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a></ul></div><div><div class=footerTitle_tbhc>More</div><ul class="footerList_lK9E clean-list"><li><a href=https://apify.com target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">Apify platform<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a><li><a href=https://docusaurus.io target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">Docusaurus<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a><li><a href=https://github.com/apify/crawlee target=_blank rel="noopener noreferrer" class="footer__link-item footerLink_YJ8n">GitHub<svg xmlns=http://www.w3.org/2000/svg width=16 height=16 fill=none viewBox="0 0 16 16" class=externalLinkIcon_gEsj><path fill=#fff fill-rule=evenodd d="M2 1.5a.5.5 0 0 1 .5-.5h4a.5.5 0 0 1 .5.5v4a.5.5 0 0 1-1 0V2.707L1.854 6.854a.5.5 0 1 1-.708-.708L5.293 2H2.5a.5.5 0 0 1-.5-.5" clip-rule=evenodd /></svg></a></ul></div></div></div></div><div class=footerBottom_VaxJ><div class=footerBottomRow_vynO><div><svg xmlns=http://www.w3.org/2000/svg width=14 height=12 fill=none viewBox="0 0 14 12" class=hearthIcon_zliG><path fill=#B2B8CC d="M3.653.049a4 4 0 0 1 3.325.95l.025.022.022-.02a4 4 0 0 1 3.156-.96l.164.024a4 4 0 0 1 2.242 6.672l-.12.123-.032.028-4.966 4.919a.67.67 0 0 1-.876.055l-.062-.055-4.996-4.948A4 4 0 0 1 3.653.049"/></svg>Crawlee is forever free and open source</div><div>© <!-- -->2025<!-- --> Apify</div></div></div></footer></div>
<!-- Google Tag Manager (noscript) -->
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5P7MCS7" height=0 width=0 style=display:none;visibility:hidden></iframe></noscript>
<!-- End Google Tag Manager (noscript) -->