ProxyProviders Quick Start Documentation

ProxyProviders

The Unified Python Proxy API for managing proxies with support for BrightData, WebShare, and more.

Table of Contents

Documentation
Getting Started
- How to Support The Project
- Installing
Quick Start Guide
- Examples

Documentation

You can find the full documentation here

Note: If you want to learn how to web scrape websites check my free and open-source course for learning everything web scraping

How to Support The Project

Star the repo 😎
Consider sponsoring me on GitHub
Send me an email or a LinkedIn message about what you’re doing with it :D
Submit PRs for issues or any new providers/features :)

Getting Started

To get started using this package follow the instructions below.

Installation

This package is installable via pip.

python -m pip install proxyproviders

Quick Start Guide

Here’s a quick bit of code to get you started! There’s also a few examples in the folder.

Note: If you want to learn how to web scrape websites check my free and open-source course for learning everything web scraping

Choose a Proxy Provider

If you already haven’t, choose which proxy provider to use. You can find a list in the documentation. After choosing, look at the documentation around the specific provider you’ve choosen. Where needed, we’ve laid out steps to get api keys in the documentation. These steps will vary slightly by provider. For this example I’ll be using the Webshare Provider because it’s both easy to setup and they give you 10 free data center proxies to test out.

You can create an account on webshare here (affiliate link), then head into the API tab on the side and generate a new token. Keep this API token safe and don’t post it publically.

Basic Example

After you can list out your proxies with

from proxyproviders import Webshare

proxy_provider = Webshare(api_key="your-api-key")

proxies = proxy_provider.list_proxies()

print(proxies)

Each provider has their own custom options, the Webshare class lets you specify url params according to their api spec, here’s an example which will only return proxies that are based in the US.

proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"})

Using ProxyConfig

For any shared logic across all types of proxy providers, we use the ProxyConfig data class to configure them. The full docs for ProxyConfig are here. In this example we will configure it to use a shorter refresh_interval than default.

from proxyproviders import Webshare, ProxyConfig
import time

config = ProxyConfig(refresh_interval=3)
ws = Webshare(api_key="your-api-token", config=config)

proxies = ws.list_proxies() # calls API

ws.list_proxies() # cached

time.sleep(5)
ws.list_proxies() # calls API since it's more than 3s later

Function Using Generic ProxyProvider

Since all proxy providers implement the same interface, we can make a function that allows us to easily swap out and utilize different providers. This is the main appeal of having a unified interface. It allows other modules to be provider agnostic, like my TikTokAPI package.

from proxyproviders import Webshare, BrightData, ProxyProvider, ProxyConfig

def some_function(provider: ProxyProvider):
    proxies = provider.list_proxies()
    print(proxies)

webshare = Webshare(api_key="your_api_key")
brightdata = BrightData(api_key="your_api_key", zone="my_zone")

some_function(webshare)
some_function(brightdata)

Here’s a more meaningful example that takes the Proxy class and uses it to create a python requests http proxy.

from proxyproviders import Webshare, BrightData, ProxyProvider
import requests
import os


def request_with_proxy(provider: ProxyProvider):
    requests_proxy = None

    if provider:
        proxies = provider.list_proxies()

        requests_proxy = {
            "http": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
            "https": f"http://{proxies[0].username}:{proxies[0].password}@{proxies[0].proxy_address}:{proxies[0].port}",
        }

    r = requests.get("https://httpbin.org/ip", proxies=requests_proxy)
    return r.json()

webshare = Webshare(api_key="your_api_key")
brightdata = BrightData(api_key="your_api_key", zone="your_zone")

print(f"Your IP: {request_with_proxy(None)}")
print(f"Webshare: {request_with_proxy(webshare)}")
print(f"BrightData: {request_with_proxy(brightdata)}")

Making Your Own Proxy Provider

Here’s a skeleton of how you can make your very own ProxyProvider class. You’ll need to implemenet all the required functions of the ProxyProvider which may be more than what’s here at the time of writing.

If you do find yourself making one of these, consider contributing it back to the repository so everyone can use them :D

from proxyproviders import ProxyProvider, ProxyConfig, Proxy
from typing import List, Optional

class MyProxyProvider(ProxyProvider):
    def __init__(self, config: Optional[ProxyConfig] = None):
        super().__init__(config=config)

    def _fetch_proxies(self):
        proxies: List[Proxy] = []

        for i in range(10):
            # TODO: your real proxy fetching logic

            # There are required fields on the Proxy class, be sure that these are filled out properly
            # especially if you're using it with another library.
            proxy = Proxy(
                id=str(i),
                username="username",
                password="password",
                proxy_address="192.168.1.1",
                port=80,
            )

            proxies.append(proxy)

        return proxies

def some_function(provider: ProxyProvider):
    proxies = provider.list_proxies()
    for proxy in proxies:
        print(proxy)

provider = MyProxyProvider()
some_function(provider) # calls the function with the provider

ProxyProvider Shared Interface

class proxyproviders.proxy_provider.ProxyConfig(refresh_interval: float = 600)[source]

Bases: object

Provides standard shared configuration options for all proxy providers.

Example:

from proxyproviders import Webshare, ProxyConfig

config = ProxyConfig(refresh_interval=5)
proxy_provider = Webshare(api_key="your-api-key", config=config)

refresh_interval: float = 600: Duration in seconds after which the proxy list is refreshed to fetch new proxies. Set to 0 to disable automatic refreshing after the initial fetch.

class proxyproviders.proxy_provider.ProxyProvider(config: ProxyConfig | None = None)[source]

Bases: ABC

Base class for all proxy providers that provides shared functionality across all providers.

You’ll need to use a specific provider to fetch proxies and this shouldn’t directly be initialized. However, this class is useful for type hinting and for creating custom providers that are not included in the library, although consider contributing them to the library if you are making them.

Parameters:: config – Configuration for the proxy provider. View the ProxyConfig class docs for more information.

list_proxies(force_refresh: bool = False) → List[Proxy][source]

Returns the stored proxies. If a format function is provided, it applies that function to each proxy.

Parameters:: force_refresh – If True, fetches new proxies even if the list is not considered stale.

should_refresh() → bool[source]: Returns True if the proxy list should be refreshed based on the refresh_interval.

ProxyProviders Supported Providers

Current Providers Supported:

Webshare.io (referral link) - try out 10 free datacenter proxies
BrightData (affiliate link)

Don’t see a provider you need? Open an issue or submit a PR!

class proxyproviders.providers.webshare.Webshare(api_key: str, search_params: dict | None = None, config: ProxyConfig | None = None)[source]

Bases: ProxyProvider

Webshare is a proxy provider that offers residential and datacenter proxies.

Create an account for webshare here (affiliate link) to get started with 10 free data center proxies.

You can find your API key in the Webshare dashboard here

You can find a list of all available parameters in the Webshare API documentation here

Parameters:

api_key – Your Webshare API key
search_params – Optional parameters to include in the API requests
config – Configuration for the proxy provider. View the ProxyConfig class docs for more information.

Example:

from proxyproviders import Webshare

# Initialize the Webshare API client with an API key and optional parameters
proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"})

# Fetch proxies
proxies = proxy_provider.list_proxies()

# With config
from proxyproviders import ProxyConfig

config = ProxyConfig(refresh_interval=60)
proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"}, config=config)

# Fetch proxies
proxies = proxy_provider.list_proxies()

class proxyproviders.providers.brightdata.BrightData(api_key: str, zone: str, username_suffix: str | None = None, use_super_proxy: bool | None = True, config: ProxyConfig | None = None)[source]

Bases: ProxyProvider

BrightData (formerly luminati) is a proxy provider that offers residential and datacenter proxies.

Create an account here (affiliate link).

You can find your API key in the account settings here then create “add token” with scope “limit” (BrightData article for more info)

The BrightData API documentation is here

Parameters:

api_key – Your BrightData API key
zone – The zone ID/name to fetch proxies for (you can get this from the BrightData dashboard)
username_suffix – Optional suffix to append to the username more info allows you to target region, city, etc. (requires use_super_proxy=True)
use_super_proxy – Optional flag to use super proxy instead of targeting specific IPs, this is enabled by default. If you want to target specific IPs or have consistent IPs for a session, set this to False.
config – Configuration for the proxy provider. View the ProxyConfig class docs for more information.

Example:

from proxyproviders import BrightData

# Initialize the BrightData API client with an API key and a zone
proxy_provider = BrightData(api_key="your-api-key", zone="my_zone")

# Fetch proxies
proxies = proxy_provider.list_proxies() # returns one proxy for super proxy by default

# If you want to manage specific IPs
proxy_provider = BrightData(api_key="your-api-key", zone="my_zone", use_super_proxy=False)
proxies = proxy_provider.list_proxies() # returns multiple proxies for each IP in the zone (potentially thousands)

get_active_zones() → Dict[source]

Fetches active zones from BrightData API.

Response:

[
    {
        "name": "zone1",
        "type": "dc",
    }
]

get_zone_username(zone: str) → str[source]

Fetches zone username for the given zone ID from BrightData API.

Note: this isn’t directly an API endpoint, I’m sort of reconstructing some things here and it seems to behave a little weird.

Parameters:: zone – The zone ID to fetch username for

get_zone_passwords(zone: str) → Dict[source]

Fetches zone passwords from BrightData API.

Parameters:: zone – The zone ID to fetch passwords for

Response:

{
    "passwords": [
        "password1",
        "password2",
    ]
}

list_all_ips_in_zone(zone: str, country: str | None = None) → Dict[source]

Fetches all IPs in a zone from BrightData API.

Parameters:

zone – The zone ID to fetch IPs for
country – Optional 2-letter country code to filter IPs by

Response:

[
    {
        "ip": "192.168.1.1",
        "country": "US",
    }
]

ProxyProviders Models

class proxyproviders.models.proxy.Proxy(id: str, username: str, password: str, proxy_address: str, port: int, country_code: str | None = None, city_name: str | None = None, created_at: datetime | None = None, protocols: List[str] | None = None)[source]

Bases: object

Our shared data model for a proxy object across all providers.

id: str: A unique identifier for the proxy

username: str: The username required for authenticating with the proxy

password: str: The password required for authenticating with the proxy

proxy_address: str: The IP address or domain name of the proxy

port: int: The port number through which the proxy connection is established

country_code: str | None = None: The country code where the proxy is located, e.g., ‘US’, ‘FR’. Optional

city_name: str | None = None: The city name where the proxy is located, e.g., ‘New York’, ‘Paris’. Optional

created_at: datetime | None = None: The timestamp when the proxy was created. Optional

protocols: List[str] | None = None: A list of connection protocols supported by the proxy, e.g., [‘http’, ‘https’]

ProxyProviders Exceptions

exception proxyproviders.exceptions.ProxyProviderException[source]

Bases: Exception

Base class for all ProxyProvider errors.

exception proxyproviders.exceptions.ProxyFetchException(message: str, status_code: int | None = None)[source]

Bases: ProxyProviderException

Raised when there is an error fetching proxies from the provider.

exception proxyproviders.exceptions.ProxyConversionException[source]

Bases: ProxyProviderException

Raised when there is an error converting proxy data to a standardized format.

exception proxyproviders.exceptions.ProxyInvalidResponseException(response: str)[source]

Bases: ProxyProviderException

Raised when the provider returns an invalid response.