User-Agent Randomizer

Till can randomly generate a random combination of millions of user-agent headers when your scrapers send HTTP requests via Till to the target websites.

User-agent randomization is useful in avoiding detection by anti-scrapers that some websites use.

Important: This feature is disabled by default. To turn it on, you need to set both of the following configurations:

Note: The User-Agent header will change to a random value on every HTTP request that goes through Till. If you wish to have one user-agent "stick" to a certain set of HTTP requests, please consider using the Sticky Sessions feature.

Configuration Steps

Step 1: Configure Till

There are two ways you can configure Till, either through the CLI or by setting it in the config file.

CLI

To configure Till via the CLI, run the command with the --force-user-agent and --ua-type flag.

$ till serve --force-user-agent --ua-type desktop 

Config File

If you have already created a config.yaml file, you can add a configuration like so:

# User agent settings.
# When set to true, it will override all user-agent with a randomly generated one
force-user-agent: true
# specify user agent type to generate randomly. 
ua-type: desktop

# specify the user agent config file to customize the user-agent generation. See "Customizing User-Agent Generation" section.
# Note : This is a premium feature.
ua-config-file: /path/to/your/ua-config-file.json

Step 2: Verify Till

Now, you just need to verify that your Till configuration is working.

To verify it using curl you can do the following command:

$ curl 'https://fetchtest.datahen.com/echo/request' -H 'X-DH-Cache-Freshness: now' -k --proxy http://localhost:2933

You should now be able to see that the User-Agent header will change on every request.

Customizing User-Agent Generation

Till allows you to customize the generated user agents by providing a custom user agent configuration file with --ua-config-file flag or by setting ua-config-file on till's config file.

Note: This is a Premium feature. If you've already upgraded your plan, you can restart Till and it will be turned on.

Our user agent configuration file is based on usage probabilities that are used by our user agent randomizer to correctly distribute the user agents usage according to the market OS+Browser usage.

It is recommended to use our default user agent configuration file as base for your customizations and add/remove the OSs, browsers, and variants as you need.

Here is an example of a custom user agent configuration file to explain it's structure:

{
    "desktop": { // ua-type-id, user agent type name, this is what you use on ua-type flag
        "id": "desktop", // ua-type-id
        "oses": [ // supported operating systems list
            {
                "id": "windows", // os-id, identifies the operating system brand as uniq
                "probability": 0.7712, // randomizer probability
                "variants": [ // OS variants list
                    {
                        "id": "windows7", // os-variant-id, must be uniq within the same OS
                        "signatures": ["Windows NT 6.1; Win64; x64"], // user agent signature
                        "probability": 0.1856, // randomizer probability
                        "browser_ids": [ // OS variant's supported browser ID list
                            "ie",
                            "chrome",
                            "firefox"
                        ],
                        "data": { // variables to be used on the browser's ua_fromat definition
                            "kernel": ["Windows NT 6.1"]
                        }
                    }
                ]
            }
        ],
        "browsers": { // supported browser list
            "ie": {  // browser-id, must be uniq, usually a reference to the browser brand
                "id": "ie", // browser-id
                "probability": 0.0119, //randomizer probability

                // user agent format is the template the randomizer will use to
                // create a user agent by replacing the variables inside "<>", like the OS
                // signature and data variables, or the browser's version and variants data
                "ua_format": "Mozilla/5.0 (<os:kernel>; Trident/7.0; rv:11.0) like Gecko",
                "variants": [ // browser variants list, useful on chromium based browsers
                    {
                        "id": "ie", // variant-id, must be uniq within the same browser
                        "probability": 1, // randomizer probability
                        "data": {} // variables to be used on the browser's ua_fromat definition
                    }
                ]
            },
            "chrome": {
                "id": "chrome",
                "probability": 0.6630,
                "ua_format": "Mozilla/5.0 (<os:signature>) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<browser:version> Safari/537.36",
                "variants": [
                    {
                        "id": "chrome",
                        "probability": 1,
                        "data": {
                            "version": [
                                "85.0.4183",
                                "86.0.4240"
                            ]
                        }
                    }
                ]
            },
            "firefox": {
                "id": "firefox",
                "probability": 0.0408,
                "ua_format": "Mozilla/5.0 (<os:signature>; rv:<browser:version>) Gecko/20100101 Firefox/<browser:version>",
                "variants": [
                    {
                        "id": "firefox",
                        "probability": 1,
                        "data": {
                            "version": ["80", "81"]
                        }
                    }
                ]
            }
        }
    },
    "tablet": { /* ... */ },
    "my_ua_type": { /* ... */ }