Replace psf/requests with an unofficial fork

[Ousret]

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

• OS truststore by default, no more certifi!
• Object-oriented headers. Could bring additional features!
• Fully type-annotated!
• HTTP/3 over QUIC.
• HTTP/2 by default.
• Exit Python http.client! in favor of h11.
• Timeout by default.
• Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
• All the features you expose are available in all three protocols.
• Python 3.7+, no sacrifice needed.
• Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

• Stricter on emitted requests per RFCs
• Young project but based on solid bases, knowledge, and experiences.
• Need to publish new packages to distro. Easy but time-consuming
• Exit pyopenssl, not supported. more of a pro to me
• Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549
• Close System for meta information output options (timing, SSL info, …) #1023
• Close Include SSL/TLS and other metadata information in --meta/-vv #826
• Close "SSL: CERTIFICATE_VERIFY_FAILED" should show cert at least with --debug #800
• Close Display used server IP in verbose mode #1495
• Close SSL/TLS server certificate validation #1202
• Close Add equivalent cURL --resolve flag #99
• Close binding to an IP #1422
• Close Add -4 and -6 switches to force IP protocol version #94
• Close I would like the option to disable the DNS Cache and do name resolution on every request #1480
• Close Unable to install httpie via Chocolatey #1522
• Close Multiple response headers with same name combined to a single comma-separated list #1413
• Close Digest authentication with qop=auth-int fails "TypeError: expected string or bytes-like object"  #1446
• Close Should use the user-provided Host header for SNI #414
• Close HTTPie could be lead to believe data was passed in stdin when it was not #1551
• Close add local-port flag #1456
• Close can't install chocolatey issues with python error code 1603 #1338
• Close Bad filename when having non ASCII characters #1401
• Close Request is fired to packages.httpie.io on every invocation of http #1527
• Close Cookies with "Domain=localhost" aren't getting stored in session file #602
• Close Change insecure SSL/TLS to deprecated #962
• Close httpie -d does not work with gzip compressed content #1554
• Close --download reports an "incomplete download" with Content-Encoding: gzip and content-length. #423
• Close Deprecate PROTOCOL_SSLv23 #1400
• Close Feature Request: Support special-use domain names like localhost. #1458
• Close --ssl — TLS 1.3 & Python 3.7 compatibility  #722
• Close httpie - support for manipulating hostheaders? #579
• Close Installation using chocolatey fails installing to system Python environment #1580
• Close Unable to access HTTPS URLs on Windows securely #1581
• Close Does not handle HTTP informational responses correctly #752
• Close Request to server is very slow #1599
• Close werkzeug compatibility #1530

---

4.0.0 (unreleased)

• Switched from the requests library to the compatible niquests. (#1531)
• Added support for HTTP/2, and HTTP/3 protocols. (#523, #692, #1531)
• Added support for early (informational) responses. (#752) (#1531)
• Added support for Happy Eyeballs algorithm via --heb flag (disabled by default). #1599 #1531
• Added support for IPv4/IPv6 enforcement with -6 and -4. (#94, #1531)
• Added support for alternative DNS resolvers via --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99, #1531)
• Added support for binding to a specific network adapter with --interface. (#1422, #1531)
• Added support for specifying the local port with --local-port. (#1456, #1531)
• Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495, #1023, #826, #1531)
• Added support to load the operating system trust store for the peer certificate validation. (#480, #1531)
• Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
• Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023, #1531)
• Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458, #1527)
• Fixed the case when multiple headers where concatenated in the response output. (#1413, #1531)
• Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551, #1531)
  This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
• Improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
• Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
• Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
• Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
• Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554, #423, #1527)
• Fixed downloader yielding an incorrect speed when the remote is using Content-Encoding aka. compressed body. (#1554, #423, #1527)
• Removed support for preserving the original casing of HTTP headers. This comes as a constraint of newer protocols, namely HTTP/2+ that normalize header keys by default. From the HTTPie user perspective, they are "prettified" in the output by default. e.g. x-hello-world is displayed as X-Hello-World.
• Removed support for pyopenssl. (#1531)
• Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
• Removed dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
• Removed dependency on multidict in favor of an internal one due to often missing pre-built wheels. (#1522, #1531)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579).
Report is 370 commits behind head on master.

❗ Current head 7cd6579 differs from pull request most recent head 55aea9c

Please upload reports for the commit 55aea9c to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dwt]

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

[dwt]

@jkbrzt Aren't you the maintainer here? Could you give an update on this?

[techknowlogick]

@Ousret, like the others in the thread, I also wanted to thank you for your work on this PR. I have been building it with my NixOS flake https://gitea.com/techknowlogick/tklk-flake as I run the PR version daily.

[markstos]

@Ousret At what point do you go ahead and publish your work as a fork?

[markstos]

@Ousret Thank you for all your work on this.

As a httpie user, I'm also interested in "correctness over performance". I would rather be able to test HTTP/2 and HTTP/3 connections a fraction of a second slower than to not be able to test them at all.

I arrived at this pull request because I was affected by one of the amount two dozen issues that it closes. If the original maintainer doesn't merge it, I hope it's packaged as a fork. Though, my own open source plate is also full as well, and I'm not volunteering at the moment, either.

[jkbrzt]

Sorry to see this. I expressed concerns over performance as with Niquests, the most common use case of fetching a JSON response is 50% slower (including over HTTP/1). My last words were that I'll do more testing. For the record, we have been in touch over email (as well as sponsoring Ahmed) for over half a year. I wish all the best to both Ahmed and his project.