download: Fix to send URL path rather than absolute URL in HTTP GET requests

davidfstr · davidfstr · commit dd3981c73ba8 · 2022-07-24T17:13:27.000-07:00
This behavior conforms to RFC 2616 (HTTP/1.1) §5.1.2: > The most common form of Request-URI is that used to identify a > resource on an origin server or gateway. In this case the absolute > path of the URI MUST be transmitted (see section 3.2.1, abs_path) as > the Request-URI, and the network location of the URI (authority) MUST > be transmitted in a Host header field. Helps download & serve: http://blog.animeworld.com/#/#/ Probably also helps download WordPress sites in general.
diff --git a/README.md b/README.md
@@ -178,6 +178,9 @@ Release Notes ⋮
 
 * Downloading improvements
     * Can force redownload of older URLs using the `--stale-before` CLI option.
+    * Fix to send URL path rather than absolute URL in HTTP GET requests,
+      improving conformance to RFC 2616 (HTTP/1.1).
+        * This helps download WordPress sites successfully.
 
 * Parsing improvements
     * Can identify URL references inside Atom feeds and RSS feeds.
diff --git a/src/crystal/download.py b/src/crystal/download.py
@@ -119,6 +119,7 @@ def __call__(self):
         url_parts = urlparse(self.url)
         scheme = url_parts.scheme
         host_and_port = url_parts.netloc
+        path = url_parts.path or '/'
         
         if scheme == 'http':
             conn = HTTPConnection(host_and_port)
@@ -138,7 +139,7 @@ def __call__(self):
         for (k, v) in _EXTRA_HEADERS.items():
             headers[k] = v
         
-        conn.request('GET', self.url, headers=headers)
+        conn.request('GET', path, headers=headers)
         response = conn.getresponse()
         
         metadata = ResourceRevisionMetadata({
