In this article, we will look at the five best mobile and 4G proxy providers for web scraping. We will not only look at the different features they offer but also perform a real-world test that includes the performance, speed, success and error rate on some of the most popular websites: Instagram, Google, Amazon and the top 1,000 Alexa rank (the list of the most visited domains in the world).
You can use many types of proxies for web scraping. Depending on the website you are extracting data from, it can be interesting or even mandatory to use mobile or 4G proxies.
Mobile proxies are great for bypassing rate-limits because the IP blocks assigned to mobile network operators generally have a great reputation. Using 3G/4G proxies minimizes the risk of getting blocked or encountering CAPTCHAS while extracting data from third-party websites.
Also, mobile and 4G proxies can be mandatory when scraping social media websites or mobile-first websites.
There are lots of players in the mobile and 4G proxy field, and it can be hard to choose one. I looked at these main features for the list:
- Price
- Performance (speed and reliability)
- Success rate on different websites
1) ScrapingBee
Yes... I know it sounds like an ad, and if you looked at our previous blog post about free proxies, it sounds like déjà-vu. But even with an unbiased opinion we're the best service on this list :)
ScrapingBee is a simple API, and you can get the raw HTML of any website with a simple API call. We offer different types of proxies — residential, data center, and mobile so that you will never get blocked again while scraping the web.
With ScrapingBee you don't have to deal with proxy management or headless browsers.
Our pricing starts at $29/mo. We have also have a $99/mo plan for more volume , and of course we can offer custom pricings for higher needs.
Results (full benchmark & methodology)
| Website | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| 45 | 0 | 955 | 3.35 | |
| 80 | 0 | 920 | 8.30 | |
| Amazon | 22 | 0 | 978 | 3.34 |
| Top 1000 Alexa | 5 | 0 | 995 | 3.34 |
2) AirProxy
AirProxy is an interesting provider on this list. They are very transparent about where their mobile proxies come from, as you can see in this video:
Basically, they rent sim cards from Italian mobile carriers and let you use them. The authentication is user/password based, which is great when you are deploying your scrapers on cloud providers without a fixed IP address. The cost is high, it's 87€ (~$100) per month per IP. The great thing about them is that you can request an IP change every 15mn.
While these proxies are great for social media scraping and automation, they're not that great for other use cases. For example, these proxies are blocked by default on everything that isn't social media or Google. On Google the rate limits are a big problem. In my test, the IP was blocked after a few thousands requests. And you can only request a new IP after 15 min, it's hard to work with if you need high volumes.
On the other hand, we found their proxies reliable, and the speed is awesome.
Results (full benchmark)
1,000 requests were sent for each website:
| Website | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| 10 | 15 | 975 | 2.66 | |
| 0 | 1000 | 0 | -- | |
| Amazon | 1000 | 0 | 0 | -- |
| Top 1000 Alexa | 1000 | 0 | 0 | -- |
3) Luminati
Luminati is the biggest player in the proxy industry. They have many different proxy types -- data center, residential and, of course, mobile proxies. They get their mobile proxies with this "brilliant" idea:
Luminati offers an SDK (Software development kit) for mobile developers to monetize apps. Instead of running ads, they monetize their mobile apps by installing the Luminati SDK and sharing a small part of users' bandwitdth. They claim to have access to 72 million mobile IP addresses around the world.
The different options with Luminati mobile proxies are insane:
- Country-level targeting
- City-level targeting
- Carrier (AT&T, Verizon...)
Luminati also offers an extensive dashboard, and a great documentation. The major problem is the price. You will have to pay $42/GB of bandwidth, and it goes very quickly. If you're scraping a website with a 3 MB pages, that consume 1 MB of bandwidth (after Gzip/Brotli compression), it will cost you $42 per 1,000 pages.
Overall the results are good, but we didn't expect so many errors on Amazon. We suspect that they don't assign the same IP pools on every website.
Results (full benchmark)
1,000 requests were sent for each website:
| Website | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| 17 | 16 | 967 | 5.66 | |
| 32 | 32 | 968 | 1.32 | |
| Amazon | 113 | 0 | 887 | 8.62 |
| Top 1000 Alexa | 47 | 0 | 953 | 5.50 |
4) Smartproxy
Smartproxy is another big player in the mobile proxy game. They don't have a pure mobile proxy offering, but most of their proxies are from mobile devices. Here are the different options Smart proxy offers:
- Country-level targeting
- City-level targeting in the US
- Sticky sessions
- Great customer service (they often answer in less than three minutes)
The bandwidth cost is much lower than Luminati and the performance is similar in our benchmark.
Results (full benchmark)
1000 requests were sent for each website:
| Website | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| 8 | 62 | 931 | 3.57 | |
| 99 | 0 | 901 | 5.09 | |
| Amazon | 82 | 4 | 914 | 5.65 |
| Top 1000 Alexa | 139 | 0 | 861 | 5.31 |
5) Oxylabs
Oxylabs is one of the biggest a residential proxy provider. They have residential proxies in most countries. As with Smartproxy, they claim that most of their residential proxies are mobile IPs, and the other part comes directly from Internet service providers.
They are aware that the residential proxy acquisition process is a sensitive subject, and they set explicit contractual obligations with their proxy providers? This ensures that the end-users are aware and that consent is documented.
Here are some features they offer:
- Residential IPs (mostly mobile)
- City and country-level targeting
Oxylab is blocks Google by default so we couldn't measure it. For Instagram, we were disappointed by the results, about 30% of requests were blocked.
Results (full benchmark)
1000 requests were sent for each website:
| Website | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| 4 | 332 | 664 | 5.04 | |
| - | - | - | -- | |
| Amazon | 47 | 17 | 936 | 5.75 |
| Top 1000 Alexa | 120 | 4 | 876 | 4.47 |
Benchmark
Now that we have our list of mobile and 4G proxy providers, it's time to benchmark against real websites. Generally, mobile proxies are great for extracting data from social media, so we have included Instagram in the benchmark. We used a few Instagram profiles from well-known celebrities. We also tested Google, Amazon, and the top 1,000 Alexa rank.
To benchmark against these websites, we made a small script that uses one of the proxy providers and performed 1,000 requests on the target domain. For each response, we looked at the HTTP codes on the proxy server (5**), but also HTTP codes that indicate the request was blocked (403, 429). We also checked for known block messages. For example Amazon sends 200 HTTP codes with a CAPTCHA inside the HTML, Instagram redirects to the login page when they block a request.
You can find the full benchmark code here: https://github.com/ScrapingBee/freeproxylist-blogpost/blob/master/benchmark_mobile_proxies.py
| Proxy provider | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| Airproxy | 10 | 15 | 975 | 2.66 |
| Luminati | 17 | 16 | 967 | 5.66 |
| Oxylabs | 4 | 332 | 664 | 5.04 |
| Smartproxy | 8 | 62 | 931 | 3.57 |
| ScrapingBee | 45 | 0 | 955 | 3.3 |
| Proxy provider | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| Airproxy | - | - | - | -- |
| Luminati | 32 | 32 | 968 | 1.32 |
| Oxylabs | - | - | - | -- |
| Smartproxy | 99 | 0 | 901 | 5.09 |
| ScrapingBee | 80 | 0 | 920 | 8.30 |
*Using ScrapingBee Google API
Amazon
| Proxy provider | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| Airproxy | - | - | - | -- |
| Luminati | 113 | 0 | 887 | 8.62 |
| Oxylabs | 47 | 17 | 936 | 5.75 |
| Smartproxy | 82 | 4 | 914 | 5.65 |
| ScrapingBee | 22 | 0 | 978 | 3.34 |
Top 1,000 Alexa Rank
| Proxy provider | Errors | Blocked | Success | Average Time |
|---|---|---|---|---|
| Airproxy | - | - | - | -- |
| Luminati | 47 | 0 | 953 | 5.50 |
| Oxylabs | 120 | 4 | 876 | 4.47 |
| Smartproxy | 139 | 0 | 861 | 5.31 |
| ScrapingBee | 5 | 0 | 995 | 3.34 |
Analysis
This was a quick benchmark of the major mobile providers on the market.
One of the difficult things when choosing a mobile proxy provider is that many are built for social media automation and not web scraping. When you scrape the web, you want as many clean IP addresses as possible plus a reasonable bandwidth price.
With social media automation/bots, 4G proxy providers tend to offer only one IP address, that doesn't necessarily change.
This will quickly get blocked on websites with IP rate-limit restrictions like Google or Amazon. That's why you need to be careful when choosing a provider. The first 1,000 requests can work but then you start getting blocked.
The other big issue is price. The providers with the largest number of proxies, like Luminati and Oxylabs charge a high price for bandwidth. Although they have the best options and are flexible (especially Luminati), the price can be prohibitive.
As you can see in the benchmark, there are significant differences between those providers, especially regarding the success rate, error rate, speed, and price. These are all of the things you will need to consider before making a decision.
If you liked this article, don't hesitate to check out our article about the best mobile proxies for web scraping.
Happy Scrapping :)
Kevin worked in the web scraping industry for 10 years before co-founding ScrapingBee. He is also the author of the Java Web Scraping Handbook.