Issues related to product version update in #238 #265

ranghetti · 2023-08-14T13:34:20Z

I have updated MODIStsp_prodopts.xml.zip with MCD64A1 and deleted old .RData and launched load_prodops. MODIStsp_prodopts.RData is properly updated.

But: I get "Warning: Error in MODIStsp_get_prodlayers: Version 006 is not available for product Burned_Monthly_500m (MCD64A1)." Modistsp seems to assume that 006 should be available for all products. Something else needs to be fixed.

006 has been depreciated for almost all products (even the download folders have been deleted), but MODIStsp seems to be expecting "006" version product information to exist too. MODIStsp should be updated accordingly. 061 should be default.

If you fix this, I can update the .xml for all products.

Originally posted by @pkautio in #238 (comment)

ranghetti · 2023-08-14T13:37:08Z

@pkautio I have fixed that in 81c3c6b, please update MODIStsp from branch devel (and make PR to this branch).
For any other issue, please open this thread.

pkautio · 2023-08-15T09:12:01Z

I updated MODIStsp to the latest development version. Now the error above has gone away, but when I try to download "061" version, I get:
"Error in MODIStsp_get_prodlayers(o$selprod) : Version 006 is not available for product Burned_Monthly_500m (MCD64A1)"

There is still an assumption somewhere that "006" should exist.

"MODIStsp_get_prodlayers.R" line 31. Should the value be "061" instead of "006"?

ranghetti · 2023-08-16T07:08:27Z

I am correctly downloading and processing them.
Please, post your MODIStsp_ProdOpts.xml.zip content (from tag <MCD64A1> to </MCD64A1>).

ranghetti · 2023-08-16T07:09:24Z

"MODIStsp_get_prodlayers.R" line 31. Should the value be "061" instead of "006"?

No, this is only the default version, but the selected version is passed to the function when called.

pkautio · 2023-08-16T08:17:06Z

MODIStsp(gui             = FALSE,
         out_folder      = "FireData",
         out_folder_mod  = "FireDataMod",
         selprod         = "Burned_Monthly_500m (MCD64A1)",
         prod_version    = "061",
         bandsel         = "Burn_Date",
         user            = "removed",
         password        = "removed",
         start_date      = "2001.01.01", 
         end_date        = "2023.12.31", 
         verbose         = TRUE,
         spatmeth        = "file",
         spafile         = spatial_filepath,
         out_format      = "GTiff")

GDAL version in use: 3.5.2
Error in MODIStsp_get_prodlayers(o$selprod) : 
  Version 006 is not available for product Burned_Monthly_500m (MCD64A1).
If you think this is an error, please open a new GitHub issue at 'https://github.com/ropensci/MODIStsp/issues' and report it.

pkautio · 2023-08-16T08:21:46Z

I basically updated all "006"-related references to "061" within MCD64A1:

                <v_number>061</v_number>
                <prod_fullname>MCD64A1: Burned Area Monthly L3 Global 500m V061</prod_fullname>
                <main_out_folder>Burned_Monthly_500m_v61</main_out_folder>
                <native_res>463.3127</native_res>
                <http_terra>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_terra>
                <http_aqua>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_aqua>

It makes no sense to keep "006" default, since it is no longer available for any product except for MOD44W.

ranghetti · 2023-08-16T08:47:51Z

Ok, there was a bug fixed in 80efcb2. Now it may work also in manual mode.

I basically updated all "006"-related references to "061" within MCD64A1:

This is ok in products for which band structure did not change from version 006 to 061.
For subsequent products, it is strategic to remember to check this assumption, by comparing layers from version 006 (e.g. https://lpdaac.usgs.gov/products/mcd64a1v006, section "Layers") to 061 (e.g. https://lpdaac.usgs.gov/products/mcd64a1v061/).
You can find an example o diverging structures in the xml file, comparing code of the two versions of product MOD09A1.

It makes no sense to keep "006" default, since it is no longer available for any product except for MOD44W.

This is true, I changed that in f8ea3ac. According to this change, tests will probably fail, so it will be necessary to test them before releasing the new master.

Thank you

pkautio · 2023-08-16T08:51:58Z

I tested .xml file with both "006" and "061" versions of data. Now it works - I can download "061" version of data. If I remove "006", I will get the error above.

Yes, I need to double-check all the layers too, but this was only to test the principle. The .xml file should work without any reference to "006".

ranghetti · 2023-08-16T08:57:18Z

I tested .xml file with both "006" and "061" versions of data. Now it works - I can download "061" version of data. If I remove "006", I will get the error above.

Do you mean if you remove <version> <v_number>061</v_number> ... </version> from the xml file? Actually I am leaving only 061 and all is working (using your chunk and changing only spatfile).

hakimabdi · 2023-09-04T09:18:14Z

I've modified the XML file as suggested here, but still cannot download v061 of MCD64A1. Any further suggestions how to remedy this?

<MCD64A1>
            <name>Burned_Monthly_500m (MCD64A1)</name> 
            <cat_01>Land Cover Characteristics</cat_01> 
            <cat_02>Thermal Anomalies and Fire</cat_02>
            <file_prefix_terra>MCD64A1</file_prefix_terra>
            <file_prefix_aqua>MCD64A1</file_prefix_aqua>
            <versions>
                <version>
                    <v_number>061</v_number>
                    <prod_fullname>MCD64A1: Burned Area Monthly L3 Global 500m V061</prod_fullname>
                    <main_out_folder>Burned_Monthly_500m_v61</main_out_folder>
                    <native_res>463.3127</native_res>
                    <http_terra>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_terra>
                    <http_aqua>https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/</http_aqua>
                    <combined>1</combined>
                    <tiled>1</tiled>
                    <www>https://lpdaac.usgs.gov/products/mcd64a1v061</www>

The command I tried was like this:

MODIStsp(gui = FALSE,
out_folder = "C:/Fire",
out_folder_mod = "C:/Fire",
selprod = "Burned_Monthly_500m (MCD64A1)",
prod_version = "061",
sensor = "both",
download_server = "https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/",
bandsel = "Burn_Date",
user = "user",
password = "password",
start_x = 18,
end_x = 18,
start_y = 4,
end_y = 4,
start_date = "2005-01-01",
end_date = "2005-06-30",
verbose = TRUE,
spatmeth = "tiles",
out_projsel = "User Defined",
output_proj = "4326",
out_res_sel = "Native",
out_res = 463.3127,
resampling = "near",
reprocess = TRUE,
delete_hdf = TRUE,
nodata_change = FALSE,
scale_val = FALSE,
compress = "LZW",
out_format = "GTiff")

And this is the error message I got:

GDAL version in use: 3.6.2
Error: assert_that: missing values present in assertion

There was no option to select version 061 in the GUI even after updating the XML file.

pkautio · 2023-09-05T08:58:24Z

@hakimabdi I got it working specifically with MCD64A1. You need to download the latest dev version of MODISTSP before the change.

I try to allocate time during next weekend to update all or at least most .xml entries.

hakimabdi · 2023-09-05T09:39:50Z

Ok, will do, thanks for your help!

hamzamukhtar786 · 2023-09-05T11:09:22Z

Can anybody help me installation of this version. I don't know why i am getting error. Even i am following each step.

ranghetti · 2023-09-08T21:09:19Z

I try to allocate time during next weekend to update all or at least most .xml entries.

Thank you @pkautio, when you will have updated the xml please create a PR to the devel branch.

For the others, please do not use this thread for different issues. I am sorry ont to be available to provide assistance, as you would have seen. Thank you for your comprehension.

pkautio · 2023-09-22T13:53:34Z

I have now updated .xml file, which works fine in my environment. Please advice how I can update it?

ranghetti · 2023-09-22T20:25:12Z

If you have experience with GitHub you can open a PR to the develop branch; otherwise, attach here the file.

pkautio · 2023-09-24T19:31:08Z

Here is the full updated .xml version. I have not checked if there are some new products, but the existing one's should be up-to-date. All 060 versions are removed and 061 version added if it was missing.

I personally work with MCD64A1 and the downloading of data was tested with that product.

MODIStsp_ProdOpts.zip

pkautio · 2023-10-01T08:58:05Z

Pull request has been made including the .xml updates.

vferng · 2023-10-05T18:07:04Z

Dear developers and community,

I am trying to download MCD64A1 data (version 061) - see the code below- and I get similar errors (Error in if (prod_opts$tiled == 0) { : argument is of length zero).

MODIStsp(gui = FALSE,
out_folder = 'MODIS',
out_folder_mod = 'MODIS',
selprod = 'Burned_Monthly_500m (MCD64A1)',
prod_version = '061',
bandsel = 'Burn_Date',
sensor = 'both',
user = 'user' ,
password = 'password',
start_date = '2022.01.01',
end_date = '2022.12.31',
verbose = TRUE,
bbox = c(18, -36, 55, -10),
spatmeth = 'bbox',
out_format = 'GTiff',
compress = 'GTiff',
out_projsel = 'User Defined',
output_proj = '+proj=longlat +ellps=WGS84 +datum=WGS84',
delete_hdf = TRUE,
parallel = TRUE
)

I've replaced my MODIStsp_ProdOpts.zip file by the one provided here, and also updated the version and still gives me back the same error.
Do you know some solution?

Best wishes

pkautio · 2023-10-05T18:33:04Z

Hi,

Install latest development version of modiststp: install_github("ropensci/MODIStsp", ref="devel")
Delete file "C:\Users%user%\AppData\Local\R\win-library\4.3\MODIStsp\ExtData\MODIStsp_ProdOpts.RData"
Replace file "C:\Users%user%\AppData\Local\R\win-library\4.3\MODIStsp\ExtData\MODIStsp_ProdOpts.xml.zip"
Start MODIStsp and it downloads the data from xml file. Otherwise it will use the configuration from .RData file and that's the old content (not working)

vferng · 2023-10-05T18:50:33Z

Thank you so much!

ranghetti · 2023-10-06T06:55:07Z

Thank you @pkautio , I will update the package as soon as possible.

ranghetti · 2023-10-07T21:38:50Z

Fixed in github master version 2.0.10.9000 (6a9eba2).

github-actions bot closed this as completed Aug 14, 2023

ranghetti reopened this Aug 14, 2023

ropensci deleted a comment from github-actions bot Aug 14, 2023

ranghetti added the bug Issues due to a code bug label Aug 14, 2023

ranghetti mentioned this issue Aug 14, 2023

Update products #238

Open

3 tasks

ranghetti closed this as completed Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues related to product version update in #238 #265

Issues related to product version update in #238 #265

ranghetti commented Aug 14, 2023 •

edited

Loading

ranghetti commented Aug 14, 2023

pkautio commented Aug 15, 2023 •

edited

Loading

ranghetti commented Aug 16, 2023

ranghetti commented Aug 16, 2023

pkautio commented Aug 16, 2023 •

edited by ranghetti

Loading

pkautio commented Aug 16, 2023 •

edited

Loading

ranghetti commented Aug 16, 2023

pkautio commented Aug 16, 2023 •

edited

Loading

ranghetti commented Aug 16, 2023

hakimabdi commented Sep 4, 2023 •

edited

Loading

pkautio commented Sep 5, 2023

hakimabdi commented Sep 5, 2023

hamzamukhtar786 commented Sep 5, 2023

ranghetti commented Sep 8, 2023

pkautio commented Sep 22, 2023

ranghetti commented Sep 22, 2023

pkautio commented Sep 24, 2023 •

edited

Loading

pkautio commented Oct 1, 2023

vferng commented Oct 5, 2023 •

edited

Loading

pkautio commented Oct 5, 2023

vferng commented Oct 5, 2023

ranghetti commented Oct 6, 2023

ranghetti commented Oct 7, 2023

Issues related to product version update in #238 #265

Issues related to product version update in #238 #265

Comments

ranghetti commented Aug 14, 2023 • edited Loading

ranghetti commented Aug 14, 2023

pkautio commented Aug 15, 2023 • edited Loading

ranghetti commented Aug 16, 2023

ranghetti commented Aug 16, 2023

pkautio commented Aug 16, 2023 • edited by ranghetti Loading

pkautio commented Aug 16, 2023 • edited Loading

ranghetti commented Aug 16, 2023

pkautio commented Aug 16, 2023 • edited Loading

ranghetti commented Aug 16, 2023

hakimabdi commented Sep 4, 2023 • edited Loading

pkautio commented Sep 5, 2023

hakimabdi commented Sep 5, 2023

hamzamukhtar786 commented Sep 5, 2023

ranghetti commented Sep 8, 2023

pkautio commented Sep 22, 2023

ranghetti commented Sep 22, 2023

pkautio commented Sep 24, 2023 • edited Loading

pkautio commented Oct 1, 2023

vferng commented Oct 5, 2023 • edited Loading

pkautio commented Oct 5, 2023

vferng commented Oct 5, 2023

ranghetti commented Oct 6, 2023

ranghetti commented Oct 7, 2023

ranghetti commented Aug 14, 2023 •

edited

Loading

pkautio commented Aug 15, 2023 •

edited

Loading

pkautio commented Aug 16, 2023 •

edited by ranghetti

Loading

pkautio commented Aug 16, 2023 •

edited

Loading

pkautio commented Aug 16, 2023 •

edited

Loading

hakimabdi commented Sep 4, 2023 •

edited

Loading

pkautio commented Sep 24, 2023 •

edited

Loading

vferng commented Oct 5, 2023 •

edited

Loading