Merge pull request #9 from HyperionGray/new_api

New API

Author: mehaase

.gitignore

@@ -1,5 +1,6 @@
 .coverage
 *.egg-info
+.ipynb_checkpoints
 .mypy_cache
 .vscode
 __pycache__

.travis.yml

@@ -13,4 +13,4 @@ install:
   - pip install -r requirements.txt
 
 script:
-  - make test
+  - make

Makefile

@@ -1,9 +1,29 @@
-.PHONY: test
+.PHONY: docs
 
-publish: test
+default: mypy-generate test-generate generate test-import mypy-cdp test-cdp
+
+docs:
+	$(MAKE) -C docs html
+
+generate:
+	python generator/generate.py
+
+mypy-cdp:
+	mypy trio_cdp/
+
+mypy-generate:
+	mypy generator/
+
+publish: test-import mypy-cdp test-cdp
 	rm -fr dist trio_chrome_devtools_protocol.egg-info
 	python setup.py sdist
 	twine upload dist/*
 
-test:
+test-cdp:
 	pytest tests/ --cov=trio_cdp --cov-report=term-missing
+
+test-generate:
+	pytest generator/
+
+test-import:
+	python -c 'import trio_cdp; print(trio_cdp.accessibility)'

README.md

@@ -4,6 +4,7 @@
 ![Python Versions](https://img.shields.io/pypi/pyversions/trio-chrome-devtools-protocol)
 ![MIT License](https://img.shields.io/github/license/HyperionGray/trio-chrome-devtools-protocol.svg)
 [![Build Status](https://img.shields.io/travis/com/HyperionGray/trio-chrome-devtools-protocol.svg?branch=master)](https://travis-ci.com/HyperionGray/trio-chrome-devtools-protocol)
+[![Read the Docs](https://img.shields.io/readthedocs/trio-cdp.svg)](https://trio-cdp.readthedocs.io)
 
 This Python library performs remote control of any web browser that implements
 the Chrome DevTools Protocol. It is built using the type wrappers in
@@ -12,180 +13,32 @@ I/O using [Trio](https://trio.readthedocs.io/). This library handles the
 WebSocket negotiation and session management, allowing you to transparently
 multiplex commands, responses, and events over a single connection.
 
-The example demonstrates the salient features of the library.
+The example below demonstrates the salient features of the library by navigating to a
+web page and extracting the document title.
 
 ```python
+from trio_cdp import open_cdp, page, dom
+
 async with open_cdp(cdp_url) as conn:
     # Find the first available target (usually a browser tab).
-    targets = await conn.execute(target.get_targets())
+    targets = await target.get_targets()
     target_id = targets[0].id
 
     # Create a new session with the chosen target.
-    session = await conn.open_session(target_id)
-
-    # Navigate to a website.
-    await session.execute(page.enable())
-    async with session.wait_for(page.LoadEventFired):
-        await session.execute(page.navigate(target_url))
-
-    # Extract the page title.
-    root_node = await session.execute(dom.get_document())
-    title_node_id = await session.execute(dom.query_selector(root_node.node_id,
-        'title'))
-    html = await session.execute(dom.get_outer_html(title_node_id))
-    print(html)
-```
-
-We'll go through this example bit by bit. First, it starts with a context
-manager:
-
-```python
-async with open_cdp(cdp_url) as conn:
-```
-
-This context manager opens a connection to the browser when the block is entered
-and closes the connection automatically when the block exits. Now we have a
-connection to the browser, but the browser has multiple targets that can be
-operated independently. For example, each browser tab is a separate target. In
-order to interact with one of them, we have to create a session for it.
-
-```python
-targets = await conn.execute(target.get_targets())
-target_id = targets[0].id
-```
-
-The first line here executes the `get_targets()` command in the browser. Note
-the form of the command: `await conn.execute(...)` will send a command to the
-browser, parse its response, and return a value (if any). The command is one of
-the methods in the PyCDP package. Trio CDP multiplexes commands and responses on
-a single connection, so we can send commands concurrently if we want, and the
-responses will be routed back to the correct task.
-
-In this case, the command is `target.get_targets()`, which returns a list of
-`TargetInfo` objects. We grab the first object and extract its `target_id`.
-
-```python
-session = await conn.open_session(target_id)
-```
-
-In order to connect to a target, we open a session based on the target ID.
-
-```python
-await session.execute(page.enable())
-async with session.wait_for(page.LoadEventFired):
-    await session.execute(page.navigate(target_url))
-```
-
-Here we use the session (remember, it corresponds to a tab in the browser) to
-navigate to the target URL. Just like the connection object, the session object
-has an `execute(...)` method that sends a command to the target, parses the
-response, and returns a value (if any).
-
-This snippet also introduces another concept: events. When we ask the browser to
-navigate to a URL, it acknowledges our request with a response, then starts the
-navigation process. How do we know when the page is actually loaded, though?
-Easy: the browser can send us an event!
-
-We first have to enable page-level events by calling `page.enable()`. Then we
-use `session.wait_for(...)` to wait for an event of the desired type. In this
-example, the script will suspend until it receives a `page.LoadEventFired`
-event. (After this block finishes executing, you can run `page.disable()` to
-turn off page-level events if you want to save some bandwidth and processing
-power, or you can use the context manager `async with session.page_enable(): ...`
-to automatically enable page-level events just for a specific block.)
-
-Note that we wait for the event inside an `async with` block, and we do this
-_before_ executing the command that will trigger this event. This order of
-operations may be surprising, but it avoids race conditions. If we executed a
-command and then tried to listen for an event, the browser might fire the event
-very quickly before we have had a chance to set up our event listener, and then
-we would miss it! The `async with` block sets up the listener before we run the
-command, so that no matter how fast the event fires, we are guaranteed to catch
-it.
-
-```python
-root_node = await session.execute(dom.get_document())
-title_node_id = await session.execute(
-    dom.query_selector(root_node.node_id, 'title'))
-html = await session.execute(dom.get_outer_html(title_node_id))
-print(html)
-```
-
-The last part of the script navigates the DOM to find the `<title>` element.
-First we get the document's root node, then we query for a CSS selector, then
-we get the outer HTML of the node. This snippet shows some new APIs, but the
-mechanics of sending commands and getting responses are the same as the previous
-snippets.
+    async with conn.open_session(target_id) as session:
 
-A more complete version of this example can be found in `examples/get_title.py`.
-There is also a screenshot example in `examples/screenshot.py`. The unit tests
-in `tests/` also provide more examples.
+        # Navigate to a website.
+        async with session.page_enable()
+        async with session.wait_for(page.LoadEventFired):
+            await session.execute(page.navigate(target_url))
 
-To run the examples, you need a Chrome binary in your system. You can get one like this:
-
-## Running Examples on MacOS
-
-**Terminal 1**
-
-This sets up the chrome browser in a specific version, and runs it in debug mode with Tor proxy for network traffic.
-
-```
-wget https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Mac%2F678035%2Fchrome-mac.zip?generation=1563322360871926&alt=media
-unzip chrome-mac.zip && rm chrome-mac.zip
-./chrome-mac/Chromium.app/Contents/MacOS/Chromium --remote-debugging-port=9000
-> DevTools listening on ws://127.0.0.1:9000/devtools/browser/<DEV_SESSION_GUID>
-```
-
-**Terminal 2**
-
-This runs the example browser automation script on the instantiated browser window.
-
-```bash
-python examples/get_title.py ws://127.0.0.1:9000/devtools/browser/<DEV_SESSION_GUID> https://hyperiongray.com
-```
-
-## Running Examples on Linux
-
-**Terminal 1**
-
-This sets up the chrome browser in a specific version, and runs it in debug mode with Tor proxy for network traffic.
-
-```
-wget https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/678025/chrome-linux.zip
-unzip chrome-linux.zip && rm chrome-linux.zip
-./chrome-linux/chrome --remote-debugging-port=9000
-> DevTools listening on ws://127.0.0.1:9000/devtools/browser/<DEV_SESSION_GUID>
+        # Extract the page title.
+        root_node = await session.execute(dom.get_document())
+        title_node_id = await session.execute(dom.query_selector(root_node.node_id,
+            'title'))
+        html = await session.execute(dom.get_outer_html(title_node_id))
+        print(html)
 ```
 
-**Terminal 2**
-
-This runs the example browser automation script on the instantiated browser window.
-
-```bash
-python examples/get_title.py ws://127.0.0.1:9000/devtools/browser/<DEV_SESSION_GUID> https://hyperiongray.com
-```
-
-## Changelog
-
-### 0.5.0
-
-* **Backwards Compability Break:** Rename `open_cdp_connection()` to `open_cdp()`.
-* Fix `ConnectionClosed` bug.
-
-### 0.4.0
-
-* Add support for passing in a nursery. (Supports usage in Jupyter notebook.)
-
-### 0.3.0
-
-* New APIs for enabling DOM events and Page events.
-
-### 0.2.0
-
-* Restructure event listeners.
-
-### 0.1.0
-
-* Initial version
-
-<a href="https://www.hyperiongray.com/?pk_campaign=github&pk_kwd=trio-cdp"><img alt="define hyperion gray" width="500px" src="https://hyperiongray.s3.amazonaws.com/define-hg.svg"></a>
+This example code is explained [in the documentation](https://trio-cdp.readthedocs.io)
+and more example code can be found in the `examples/` directory of this repository.

docs/.gitignore

@@ -0,0 +1 @@
+_build

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/api.rst

@@ -0,0 +1,81 @@
+API
+===
+
+Trio CDP replicates the entire API of PyCDP. For example, PyCDP has a ``cdp.dom`` module,
+while in Trio CDP, we have ``trio_cdp.dom``. The Trio CDP version has all of the same data
+types, commands, and events as the PyCDP version. 
+
+The only difference between the two libraries is that the Trio CDP commands are ``async
+def`` functions that can be called from Trio, whereas the PyCDP commands are generators
+that must be executed in a special way. This document explains both flavors and how to
+use them.
+
+You should consult the `PyCDP documentation
+<https://py-cdp.readthedocs.io/en/latest/api.html>`_ for a complete reference of the
+data types, commands, and events that are available.
+
+Simplified API
+--------------
+
+.. highlight: python
+
+The simplified API allows you to await CDP commands directly. For example, to run a CSS
+query to get a blockquote element, the code is:
+
+.. code::
+
+    from trio_cdp import dom
+    
+    ...other code...
+ 
+    node_id = await dom.query(root, 'blockquote')
+
+As you can see in `the PyCDP documentation
+<https://py-cdp.readthedocs.io/en/latest/api/dom.html#cdp.dom.query_selector>`_, the
+command takes a ``NodeId`` as its first argument and a string as its second argument.
+The arguments in Trio CDP are always the same as in PyCDP.
+
+The return types in Trio CDP are different from PyCDP, however. The PyCDP command is a
+generator, where as the Trio CDP command is an ``async def``. The PyCDP command
+signature shows ``Generator[Dict[str, Any], Dict[str, Any], NodeId]]``. The generator
+contains 3 parts. The first two parts are always the same: ``Dict[str, Any]``. The third
+part indicates the real return type, and that is the type that the Trio CDP command will
+return. In this case, it returns a ``NodeId``.
+
+If you have code completion in your Python IDE, it will help you see what the return
+type is for each Trio CDP command.
+
+.. note::
+
+    In order for this calling style to work, you must be inside a "session
+    context", i.e. your code must be nested in (or called from inside of) an ``async with
+    conn.open_session()`` block. If you try calling this from outside of a session context,
+    you will get an exception.
+
+Low-level API
+-------------
+
+The low-level API is a bit more verbose, but you may find it necessary or preferable in
+some situations. With this API, you import commands from PyCDP and pass them into the
+session ``execute()`` method. Taking the same example as the previous section, here is
+how you would execute a CSS query:
+
+.. code::
+
+    from cdp import dom
+    
+    ...other code...
+ 
+    node_id = await session.execute(dom.query(root, 'blockquote'))
+
+If you compare this example to the example in the previous section, there are two big changes.
+First, ``dom`` is imported from PyCDP instead of Trio CDP. This means that is a generator,
+not an ``async def``.
+
+Second, in order to run the command on a given session, we have to call that session's
+``execute()`` method and pass in the PyCDP generator.
+
+Other than being a little more verbose (calling ``session.execute(...)`` for every CDP
+command), the low-level API is otherwise very similar to the simplified API described in
+the previous section. It still takes the same arguments and returns the same type (here
+a ``NodeId``).

docs/changelog.rst

@@ -0,0 +1,33 @@
+Changelog
+=========
+
+0.6.0
+-----
+
+* Simplified calling convention for CDP commands.
+
+0.5.0
+-----
+
+* **Backwards Compability Break:** Rename `open_cdp_connection()` to `open_cdp()`.
+* Fix `ConnectionClosed` bug.
+
+0.4.0
+-----
+
+* Add support for passing in a nursery. (Supports usage in Jupyter notebook.)
+
+0.3.0
+-----
+
+* New APIs for enabling DOM events and Page events.
+
+0.2.0
+-----
+
+* Restructure event listeners.
+
+0.1.0
+-----
+
+* Initial version