Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emscripten support #3330

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open

Emscripten support #3330

wants to merge 49 commits into from

Conversation

joemarshall
Copy link
Contributor

@joemarshall joemarshall commented Oct 2, 2024

Summary

I added a discussion for this ages back but there's been no input, so I've written it (because I was contracted to do the work anyway, so I might as well contribute it upstream). This PR adds support for running in emscripten / webassembly platforms, where all network connections go via the browser.

Currently in progress, but tests okay locally, so I've opened this to check the CI changes, I've got to update docs also.

Checklist

  • [X ] I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

@tomchristie
Copy link
Member

This is really interesting, thanks... ☺️

I've taken a bit of a look at the ecosystem here, tho am going to need a bit more orientation... Would it make sense to document an example of how to write an HTML page that includes a Python REPL with httpx imported and available?

@hoodmane
Copy link

hoodmane commented Oct 8, 2024

>>> from js import fetch
>>> resp = await fetch("console.html")
>>> text = await resp.text()
>>> print(text[:100])
<!doctype html>
<html>
  <head>
    <meta charset="UTF-8" />
    <meta
      http-equiv="origin-tria

@tomchristie
Copy link
Member

Ah yep, okay...

>>> r = await js.fetch('https://cdn.jsdelivr.net/pyodide/v0.23.4/full/repodata.json')
>>> t = await r.text()
>>> t[:100]
'{"info": {"arch": "wasm32", "platform": "emscripten_3_1_32", "version": "0.23.4", "python": "3.11.2"'

@tomchristie
Copy link
Member

tomchristie commented Oct 8, 2024

Okay, well this is neat.

Open the pyodide console, then...

>>> import micropip, ssl, js
>>> await micropip.install('httpx')
>>> import httpx
>>> class JSTransport(httpx.AsyncBaseTransport):
    async def handle_async_request(self, request):
        url = str(request.url)
        options = {
            'method': request.method,
            'headers': dict(request.headers),
            'body': await request.aread(),
        }
        fetch_response = await js.fetch(url, options)
        status_code = fetch_response.status
        headers = dict(fetch_response.headers)
        buffer = await fetch_response.arrayBuffer()
        content = buffer.to_bytes()
        return httpx.Response(status_code=status_code, headers=headers, content=content)
>>> client = httpx.AsyncClient(transport=JSTransport())
>>> r = await client.get('https://cdn.jsdelivr.net/pyodide/v0.23.4/full/repodata.json')
>>> r
<Response [200 OK]>
>>> r.json()
{'info': {'arch': 'wasm32', 'platform': 'emscripten_3_1_32', 'version': '0.23.4', 'python': '3.11.2'}, 'packages': {'asciitree': {'n
ame': 'asciitree', 'version': '0.3.3', ...

@tomchristie
Copy link
Member

tomchristie commented Oct 9, 2024

Dealing with this incrementally, here’s some isolated PRs that I think we should address first…

  • Refactor the import of httpcore so that it’s only loaded if HTTPTransport/AsyncHTTPTransport is instantiated.
  • Refactor the import of certifi in _config.py so it’s only loaded if SSLContext is instantiated.
  • Refactor imports of ssl so that it’s only loaded if SSLContext is instantiated, or is behind a TYPE_CHECKING guard.

(If anyone’s up for tackling these, currently ought to be against the version-1.0 branch, until that’s merged)

@tomchristie
Copy link
Member

Thanks again for your work here @joemarshall.
Here's where I think we're at on this...

  • Lazy load certifi & httpcore
  • Import ssl under typechecking branches.
  • Consider introducing JSFetchTransport().

@joemarshall
Copy link
Contributor Author

@tomchristie I put in the PR that makes import ssl optional now (#3385 )

I updated this PR so it follows on from that PR.

How this PR works now is it moves _transports.default into _transports.httpcore, which defines [Async]HTTPCoreTransport, adds an extra module _transports.jsfetch file which defines [Async]JavascriptFetchTransport. Then in _transports/__init__.py it adds an alias of HTTPTransport which goes to whichever HTTP backend is in use (i.e. httpcore by default, JS fetch on emscripten)

@joemarshall
Copy link
Contributor Author

@tomchristie I updated this to follow the changes in master - I think #3385 is redundant now, as the ssl changes are minimal at this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants