Pharo-Chrome

Pharo-Chrome provides a simple API for Google Chrome's DevTools protocol (https://chromedevtools.github.io/devtools-protocol/).

The are three main public classes in the package:

• GoogleChrome is the interface to the Chrome Browser
• ChromeTabPage is the interface to a single tab / page in the browser
• ChromeNode is a single DOM node within a page, with a similar interface to Soup

A ZnEasy like interface is provided for retrieving pages:

GoogleChrome get: 'http://pharo.org'

retrieves the contents in a headless browser and returns the root node.

An example of more detailed operations are available in GoogleChrome class>>exampleNavigation.

Features

• A ZnEasy like feature for retrieving pages: #get:
• Extraction of tabular data (html tables): #extractTables, #tableData
• Screenshot capture: #captureScreenshot
• Headless mode

Installation

Unixy platforms (Linux & MacOS) require OSProcess or OSSubprocess to be installed prior to loading Pharo-Chrome. To load OSSubprocess in 32 bit images:

Metacello new
	configuration: 'OSSubprocess';
	repository: 'github://marianopeck/OSSubprocess:master/repository';
	version: #stable;
	load.

On Windows the OSWindows package is required:

Metacello new
	configuration: 'OSWindows';
	repository: 'http://smalltalkhub.com/mc/Pharo/MetaRepoForPharo60/main/';
	load.

Once the platform dependent packages have been loaded, load the main Pharo-Chrome package:

Metacello new
	baseline: 'Chrome';
	repository: 'github://akgrant43/Pharo-Chrome:master/repository';
	load.

To keep the original Monticello idea of having #stable and #development versions, Pharo-Chrome is mostly using the GitFlow workflow:

• The master branch is #stable
• The development branch is #development

Pharo 7 Installation

Pharo-Chrome relies on ZnWebSocket, which doesn't appear to be loaded in Pharo 7.
If you are missing ZnWebSocket, please run:

Gofer new
   smalltalkhubUser: 'SvenVanCaekenberghe' project: 'ZincHTTPComponents';
   package: 'ConfigurationOfZincHTTPComponents';
   load.
(Smalltalk globals at: #ConfigurationOfZincHTTPComponents) project latestVersion load: 'WebSocket'.

For more details, visit the Pharo WebSockets implementation or send a message to the pharo-users mailing list for help.

A Short Demo

Stock price retrieval

As a demonstration of Pharo-Chrome, we'll retrieve some stock prices from the Australian ASX S&P 200 stock index:

rootNode := GoogleChrome get: 'https://finance.yahoo.com/quote/%5EAXJO/history?p=%5EAXJO'

The messages I use most for navigating the DOM are:

#children

Answer an array of the child nodes

#parent

Answer the parent node of the receiver

#findAllTags:

Find all tags matching the supplied criteria (see below).

#findAllStrings:

Find all strings matching the supplied criteria.

#allSelect:

Answer all nodes matching the supplied criteria.

Both #findAllTags: and #findAllStrings: can take a string, a collection or a boolean as the parameter. E.g. to find the h1 and h2 headings:

headings := rootNode findAllTags: #('h1' 'h2').

More interesting is extracting all the data from a table, e.g. historical prices:

tables := rootNode extractTables.

At the time of writing, two tables were returned. The larger table is the one we're interested in. Most of the rows will be actual data, smaller rows typically contain comments, so we'll make the data uniform by simply rejecting smaller rows. It can be nicely inspected using:

historicalData := (tables sorted: #size ascending) last.
dataFrame := DataFrame fromRows: (historicalData select: [ :each | each size = 7 ]).
dataFrame asStringTable.

"
     |  1             2         3         4         5         6            7       
-----+-----------------------------------------------------------------------------
1    |  Date          Open      High      Low       Close*    Adj Close**  Volume  
2    |  Nov 14, 2017  6,021.80  6,021.80  5,957.10  5,966.00  5,966.00     -       
3    |  Nov 13, 2017  6,029.40  6,029.40  6,010.70  6,021.80  6,021.80     -       
4    |  Nov 10, 2017  6,049.40  6,049.40  6,020.70  6,029.40  6,029.40     -       
etc.
"

Note that the example above assumes that DataFrame has been installed in the image:

Metacello new
  baseline: 'DataFrame';
  repository: 'github://PolyMathOrg/DataFrame';
  load.

pharo.org screenshot

To retrieve a png image of the pharo.org front page in headless mode:

| browser page image |

browser := GoogleChrome new.
browser open.
page := browser firstTab.
page get: 'http://pharo.org'.
image := page captureScreenshot.
browser closeAndExit.
image