Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multithreading/async for all requests #304

Closed
SlowMo24 opened this issue Apr 25, 2022 · 4 comments
Closed

Add multithreading/async for all requests #304

SlowMo24 opened this issue Apr 25, 2022 · 4 comments
Assignees
Labels
brainstorming Idea for a potential new feature or adaption that still needs further discussion comments welcome Indicates that the creator of this PR is open for early review comments enhancement New feature or request user requirement

Comments

@SlowMo24
Copy link
Contributor

Requirement

The user wants to request one or more indicators for one ore more layers for multiple geometries.

Current Situation

The user has to do all optimisation at her end (e.g. asynchronous requests). Most users will use a very ineffective loop approach:

for indicator in indicators:
    for layer in layers:
        for geometry in geometries:
            request(oqt)

For an example see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/ideal-vgi/osm-element-vectorisation/-/blob/6a65893fdaaa5cad97f4b1d7cf7fdc1701e89287/src/osm_element_vectorisation/indicators/surrounding/ohsome/oqt.py

Possible better (more performant) solution

Use group-by(geometry/indicator/layer) or for async(geometry/indicator/layer) approach to allow users to request multiple geometries at once. Multiple geometries are currently possible, what is the approach here?

This would require user friendly error handling, where the request only fails, if all geometries fail but returns partial responses, whenever possible.

Related: #151 and GIScience/ohsome-py#37

@SlowMo24 SlowMo24 added brainstorming Idea for a potential new feature or adaption that still needs further discussion user requirement labels Apr 25, 2022
@joker234 joker234 added enhancement New feature or request comments welcome Indicates that the creator of this PR is open for early review comments labels Apr 25, 2022
@matthiasschaub
Copy link
Collaborator

matthiasschaub commented Apr 25, 2022

Thanks for creating this detailed issue! Some short notes from our discussion:

Currently possible:

Utilizing groupByBoundary queries to the ohsome API for a FeatureCollection input to OQT is not easily implemented. Also it is not clear if this will lead to big performance improvements.

For errors see this issue: #306

@matthiasschaub
Copy link
Collaborator

matthiasschaub commented Jul 7, 2022

Asynchronous calculation of Indicators is now implemented. See previous comment.

@SlowMo24
Copy link
Contributor Author

SlowMo24 commented Jul 8, 2022

Great solution, thank you. One question: Are reports available via the API as JSON object and do they also handle FeatureCollections?

@matthiasschaub
Copy link
Collaborator

You are welcome!

In short the answer to your question is Yes and Yes.

Reports are also available via the API. The response is a GeoJSON representing the input GeoJSON. If the input is just a Feature the response will also be a Feature. If the input is a FeatureCollection the response will also be a FeatureCollection. The overall result and also the result of every Indicator will be part of the properties.
See also: https://oqt.ohsome.org/api/docs#/report/post_report_report_post

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
brainstorming Idea for a potential new feature or adaption that still needs further discussion comments welcome Indicates that the creator of this PR is open for early review comments enhancement New feature or request user requirement
Projects
None yet
Development

No branches or pull requests

3 participants