Integrating Till with Go

Till can be easily integrated with your Go scraper without much code changes.
The following is an example of using Go's net/http standard library to integrate with Till.

Please follow the steps below.

Step 1: Install Till

Follow the instructions to install Till

Step 2: Modify your Go script

Next, you need to modify your existing Go script to integrate with Till.

The following is an example of a GET request:

func main(){
  // setup the proxy connection to Till
  proxyUrl, err := url.Parse("http://localhost:2933")
  myClient := &http.Client{Transport: &http.Transport{
    Proxy:           http.ProxyURL(proxyUrl),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }}


  // create a new GET request
  greq, err := http.NewRequest("GET", "https://fetchtest.datahen.com/echo/request", nil)
  if err != nil {
    log.Fatal(err)
  }

  // Add the header to force a Cache Miss on Till
  greq.Header.Add("X-DH-Cache-Freshness", "now")

  // Do the actual request
  gresp, err := myClient.Do(greq)
  if err != nil {
    log.Fatal(err)
  }

  // print out the response
  grout, err := httputil.DumpResponse(gresp, true)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println("-------------------")
  fmt.Println("GET RESPONSE:")
  fmt.Println("-------------------")
  fmt.Println(string(grout))
}

The following is an example of a POST request:

func main(){
  // setup the proxy connection to Till
  proxyUrl, err := url.Parse("http://localhost:2933")
  myClient := &http.Client{Transport: &http.Transport{
    Proxy:           http.ProxyURL(proxyUrl),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
  }}


  // create a new POST request
  jsonData := `{"hello":"world"}`

  preq, err := http.NewRequest("POST", "https://postman-echo.com/post", bytes.NewBuffer([]byte(jsonData)))
  if err != nil {
    log.Fatal(err)
  }

  // Add the header to force a Cache Miss on Till
  preq.Header.Add("X-DH-Cache-Freshness", "now")

  preq.Header.Set("Content-Type", "application/json")
  preq.Header.Set("Accept", "*/*")

  // Do the actual request
  presp, err := myClient.Do(preq)
  if err != nil {
    log.Fatal(err)
  }

  // print out the response
  prout, err := httputil.DumpResponse(presp, true)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println("-------------------")
  fmt.Println("POST RESPONSE:")
  fmt.Println("-------------------")
  fmt.Println(string(prout))
}

Note: To see a working example, you can visit this link.

Step 3: Run your script

Next, run your Go script like you normally would.

Note: If you don't have an existing Go script to try with Till, you can try our working example here.

Step 4: Verify that it works

Visit the Till UI at http://localhost:2980/requests to see that your new requests are shown.