Python Docs

Avoiding 502 (Bad Gateway) Errors IMPORTANT

If you use Curl (pycurl) with Python, you need to set unset the Accept Header. You can do so like:

pycurl_connect.setopt(pycurl.HTTPHEADER, ['Accept:'])

If you do not, you will get 502 Bad Gateway errors with your curl requests. The errors won't occur when your POST payload is small, but medium and large POST payloads will always get this 502 error.

If you follow the examples below you can ignore this suggestion, as these examples use Python without 3rd party libraries (not using Curl!).

Post request content: (request.json)

In the following examples, we use Python to submit POST requests, sending a request.json file as the content. This file follows either the UserRequest or PageRequest specification, which can be viewed from our HTTP Endpoint docs. The default parameters (for pageRequest parameters you do not explicitly set) can be viewed here.

Minimal Example: Capture a Page as PlainText

This Python example captures the text from a page and prints to the command-line.

request.json

(in pageRequest format)

{
	"url":"http://example.com",
	"renderType":"plainText"
}

python code using builtin requests code

import requests
import json

data ={
"url":"http://example.com",
"renderType":"plainText"
}

url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/';
req = requests.post(url, data=json.dumps(data))
results = req.content

print(results)

python code using urllib2

import urllib2

with open('request.json', 'r') as myfile:
    data=myfile.read()

url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/'
headers = {'content-type':'application/json'}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
results = response.read()

print(results)

Minimal Example: Verbose output JSON

This Python example Captures the text from the page and PhantomJsCloud metadata about the request and response, then saves to a .json file.

request.json

(in pageRequest format)

{
	"url":"http://example.com",
	"renderType":"plainText",
	"outputAsJson":true
}

python code

import urllib2

with open('request.json', 'r') as requestFile:
    data=requestFile.read()

url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/'
headers = {'content-type':'application/json'}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
results = response.read()

with open('response.json', 'w') as responseFile:
	responseFile.write(results)

Minimal Example: Capture as JPEG

This Python example will generate a screenshot of a webpage (including rendering AJAX), then saves to a .jpg file.

request.json

(in pageRequest format)

{
	"url":"http://www.highcharts.com/stock/demo/intraday-area",
	"renderType":"jpeg"
}

python code

import urllib2
with open('request.json', 'r') as requestFile:
    data=requestFile.read()
url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/'
headers = {'content-type':'application/json'}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
results = response.read()
print('\nresponse status code')
print(response.code)
print('\nresponse headers (pay attention to pjsc-* headers)')
print(response.headers)
with open('content.jpg', 'wb') as responseFile:
	responseFile.write(results)

Minimal Example: Capture as PDF

This Python example will generate a PDF of Amazon.com's main page (the live, current contents), then saves to a .pdf file.

request.json

(in pageRequest format)

{
	"url":"https://amazon.com",
	"renderType":"pdf"
}

python code

import urllib2
with open('request.json', 'r') as requestFile:
    data=requestFile.read()
url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/'
headers = {'content-type':'application/json'}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
results = response.read()
print('\nresponse status code')
print(response.code)
print('\nresponse headers (pay attention to pjsc-* headers)')
print(response.headers)
with open('content.pdf', 'wb') as responseFile:
	responseFile.write(results)

Minimal Example: All parameters

This Python example shows using all PhantomJsCloud parameters in a request, capturing the page as a .jpg image, returning metadata (json) and the image (base64 encoded), and then saves the image to a file. Most the parameters used are the defaults, but you can see a list of the most up-to-date default values here.

request.json

(in userRequest format)

{
	"pages":[
		{
			"url": "http://example.com",
			"content": null,
			"urlSettings": {
				"operation": "GET",
				"encoding": "utf8",
				"headers": {},
				"data": null
			},
			"renderType": "jpg",
			"outputAsJson": true,
			"requestSettings": {
				"ignoreImages": false,
				"disableJavascript": false,
				"userAgent": "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) Safari/534.34 PhantomJS/2.0.0 (PhantomJsCloud.com/2.0.1)",
				"authentication": {
					"userName": "guest",
					"password": "guest"
				},
				"xssAuditingEnabled": false,
				"webSecurityEnabled": false,
				"resourceWait": 15000,
				"resourceTimeout": 35000,
				"maxWait": 35000,
				"waitInterval": 1000,
				"stopOnError": false,
				"resourceModifier": [],
				"customHeaders": {},
				"clearCache": false,
				"clearCookies": false,
				"cookies": [],
				"deleteCookies": []
			},
			"suppressJson": [
				"events.value.resourceRequest.headers",
				"events.value.resourceResponse.headers",
				"frameData.content",
				"frameData.childFrames"
			],
			"renderSettings": {
				"quality": 70,
				"pdfOptions": {
					"border": null,
					"footer": {
						"firstPage": null,
						"height": "1cm",
						"lastPage": null,
						"onePage": null,
						"repeating": "%pageNum%/%numPages%"
					},
					"format": "letter",
					"header": null,
					"height": null,
					"orientation": "portrait",
					"width": null
				},
				"clipRectangle": null,
				"renderIFrame": null,
				"viewport": {
					"height": 1280,
					"width": 1280
				},
				"zoomFactor": 1,
				"passThroughHeaders": false
			},
			"scripts": {
				"domReady": [],
				"loadFinished": []
			}
		}
	],
	"proxy":false
}

python code

import urllib2
import json
import pprint
pp = pprint.PrettyPrinter(indent=4)
with open('request.json', 'r') as requestFile:
    data=requestFile.read()
url = 'http://PhantomJScloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/'
headers = {'content-type':'application/json'}
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
results = response.read()
obj = json.loads(results)
print('\nresponse status code')
print(response.code)
print('\nrPhantomJsCloud metadata')
pp.pprint(obj['meta'])
with open('content.jpg', 'wb') as responseFile:
	responseFile.write(obj['content']['data'].decode('base64'))