Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

peru_compras: update spider to use new endpoint #1120

Open
sentry-io bot opened this issue Nov 1, 2024 · 2 comments
Open

peru_compras: update spider to use new endpoint #1120

sentry-io bot opened this issue Nov 1, 2024 · 2 comments
Assignees

Comments

@sentry-io
Copy link

sentry-io bot commented Nov 1, 2024

Hopefully it's temporary, as we need their data :)

Sentry Issue: REGISTRY-KINGFISHER-COLLECT-1

Gave up retrying <GET https://www.catalogos.perucompras.gob.pe/ConsultaOrdenesPub/DescargaJsonOCDS?pAcuerdo=151&pFechaIni=2017-01-01&pFechaFin=2024-11-01> (failed 3 times): 500 Internal Server Error
@sentry-io sentry-io bot added existing spider unavailable The data source is entirely unavailable labels Nov 1, 2024
@yolile
Copy link
Member

yolile commented Nov 4, 2024

From https://www.catalogos.perucompras.gob.pe/ConsultaOrdenesPub

The new endpoint seems to be
a POST to 'https://www.catalogos.perucompras.gob.pe/ConsultaOrdenesPub/getListaDescargaMasiva'
--data-raw 'Anio=&Mes='

And then, from the response, download the files

[
  {
    "C_Anio": "2023",
    "CodMes": "01",
    "C_Mes": "Enero",
    "C_Ruta": "contproveedor/DescargaMasiva",
    "C_FileJson": "Datos_Abiertos01022023034435.json",
    "C_FileExcel": "Datos_Abiertos01022023034435.xlsx",
    "C_FileCsv": ""
  },
...
]

And the full URL is, for example: https://saeusceprod01.blob.core.windows.net/contproveedor/DescargaMasiva/Datos_Abiertos01022023034435.json

@yolile yolile removed the unavailable The data source is entirely unavailable label Nov 4, 2024
@yolile yolile changed the title peru_compras: (2024-10) Gave up retrying <GET https://www.catalogos.perucompras.gob.pe/ConsultaOrdenesPub/DescargaJsonOCD... peru_compras: update spider to use new endpoint Nov 4, 2024
@jpmckinney jpmckinney assigned jpmckinney and unassigned yolile Nov 4, 2024
@jpmckinney
Copy link
Member

jpmckinney commented Nov 4, 2024

Hmm, the original spider was able to get data up to Oct 2024: https://data.open-contracting.org/en/publication/78

~collect/scrapyd/logs/kingfisher/peru_compras/3dd7eab597e411ef82b6a036bccb3328.log

Edit: The original issue I reported should not have been unavailable, since it did get 84 200 responses, with 34 500s.

Edit2: I suppose we can revert to the original, and keep the new code as a peru_compras_bulk, with the caveat that it is lagging the API.

@jpmckinney jpmckinney reopened this Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants