There is an HTTP redirection loop when file_get_contents, curl or
Guzzle is used to fetch a page of Le Soir website, but not with wget.
We so implement a fallback solution to download with this software.
Details
Details
- Reviewers
dereckson xcombelle - Maniphest Tasks
- T697: Le Soir - Redirection limit reached
- Commits
- rSTGf443c0ca9ba8: Allow to download with wget
Diff Detail
Diff Detail
- Repository
- rSTG Source templates generator
- Lint
No Lint Coverage - Unit
No Test Coverage - Branch
- lesoir-wget
- Build Status
Buildable 398 Build 479: arc lint + arc unit
Event Timeline
Comment Actions
Regression This commit introduces the following issue when the page IS NOT downloaded through this new method:
Warning: array_key_exists() expects parameter 2 to be array, null given in /usr/home/dereckson/dev/nasqueron/tools/3rdparty/source-templates-generator/page.php on line 237
The Page::meta_tags member isn't set as Page::analyse isn't called anymore.
Thanks to Scoopfinder to have noticed the issue.
| page.php | ||
|---|---|---|
| 124 | Add $data = $this->data;. | |
Comment Actions
@xcombelle confirmed on #wikipedia-fr the code is safe as far as security is concerned.
They also noted curl works with a wget user-agent, so maybe Le Soir has a whitelist: once a request is correctly done, further requests aren't filtered in one of the step.
We should maybe contact Le Soir operations team to inquire about this issue.