Running MDCS on ArcGIS Linux Server

ArcGIS Enterprise Marketplace Linux image comes with ArcPy. Thus, it is possible to run MDCS script (in theory).

Step-by-step guide for Table/Raster Catalog

  1. Copy the entire contents of https://github.com/Esri/mdcs-py/blob/master/scripts/ to working directory.
  2. Modify raster_type section of MDCS.xml.
  3. <!-- AR - Add Rasters-->
      <addrasters>
        <addraster>
              <dataset_id>power</dataset_id>
              <raster_type>Table / Raster Catalog</raster_type>
    </addraster></addrasters>
  4. Specify table (in CSV) location.
  5. <sources>
                <data_path>Z:\home\arcgis\data\LC08.csv</data_path>
              </sources>
    
  6. Edit the Raster column of input table .csv file. It must be local and a raster proxy. You can test a sample raster proxy file from [2]. You can use VRT file here if you'd like to use MRF in S3.
  7. OBJECTID,Variable,StdTime,StdTime_Max,StdZ,Raster
    0,ALL_SFC_SW_DN,2018-01-01 00:00:00,2018-01-01 00:59:59,2,"Z:\home\arcgis\data\LC08_L1TP_160043_20180326_20180404_01_T1\LC08_L1TP_160043_20180326_20180404_01_T1_B1.TIF"
    
  8. If you want to visualize mosaic dataset with ArcGIS Pro on windows machine, set the path that exists in both Windows Server and Linux Server. You can keep adding more VRT files as long as object id and time is unique.
  9. OBJECTID,Variable,StdTime,StdTime_Max,StdZ,Raster
    0,ALL_SFC_SW_DN,2018-01-01 00:00:00,2018-01-01 00:59:59,2,"C:\Users\hyoklee\Downloads\test1.vrt"
    1,ALL_SFC_SW_DN,2018-01-02 00:00:00,2018-01-02 00:59:59,2,"C:\Users\hyoklee\Downloads\test2.vrt"
    
  10. To make visualization work, make sure that the Windows Server has S3FullAccess role attached in the EC2 instance first. Then, check that Source value in VRT file starts with vsis3/.
  11. <simplesource>
          <sourcefilename relativetovrt="1">/vsis3/sdt-data/ceres/syn1deg/terra-aqua-mo\
    dis_edition4a/init_all_sfc_sw_dn/mrf/CER_SYN1deg_1Hour_Terra_Aqua_MODIS_Edition4A_4\
    06406.20180101.INIT_ALL_SFC_SW_DN.mrf:MRF:Z0</sourcefilename>
          <sourceband>1</sourceband>
          <sourceproperties rasterxsize="360" rasterysize="180" datatype="Float32" bloc\="" kxsize="360" blockysize="180">
          <srcrect xoff="0" yoff="0" xsize="360" ysize="180">
          <dstrect xoff="0" yoff="0" xsize="360" ysize="180">
        </dstrect></srcrect></sourceproperties></simplesource>
    

Step-by-step guide for Raster Dataset

This is useful for creating a mosaic set from a set of TIF files that resides in a directory.

  1. Copy the entire contents of https://github.com/Esri/mdcs-py/blob/master/scripts/ to working directory.
  2. If you use Raster Dataset for rasterType, you can create mosaic from MRFs. Put MRF files under the data_path.
  3. <!-- AR - Add Rasters-->
          <addrasters>
            <addraster>
              <dataset_id>power</dataset_id>
              <raster_type>Raster Dataset</raster_type>
              <sources>
                <data_path>Z:\home\arcgis\data\cermrf</data_path>
              </sources>
              <filter>*</filter>
            </addraster>
          </addrasters>
    
  4. Disable DF and AI in MDCS.xml if you don't use Table / Raster Catalog for rasterType.
  5. <application>
      <name>CERES_Hourly_Solar</name>                                               
      <!-- Command>CM+AR+SP+AF+CV+CC+BF+DF+CS+AI</Command -->                       
      <command></command>CM+AR+SP+AF+CV+CC+BF+CS
    </application>
Information Icon MDTools_Setup.exe cannot be installed. MDTools is a set of command line tools that simplifies some common management tasks when working with rasters in a mosaic dataset.

Error 999999

Arcpy throws an error when it adds an item. This is a known issue for Linux platform according to Abhijit at Esri. It doesn't happen on Windows.

2020-01-28T17:26:58.118: Synchronizing items associated with raster type instan\
ce 'Table / Raster Catalog' [ID: 1].
2020-01-28T17:26:58.124: Error: 80042019: Could not build mosaic dataset item. \
[ID: 1, URI: 'Z:\home\arcgis\data\CERES.csv|0']
2020-01-28T17:26:58.127: Error: 80042019: Could not build mosaic dataset item. \
[ID: 2, URI: 'Z:\home\arcgis\data\CERES.csv|1']
2020-01-28T17:26:58.129: Error: 8004205e: 2 mosaic dataset items could not be b\
uilt.
...
log-msg: Command:AR->Add rasters/data to a mosaic dataset.
log-msg: Adding rasters:
log-msg:        Using mosaic dataset/ID:CERES_Hourly_Solar/power
log-msg:        Using ART for CERES_HOURLY_SOLAR: Table / Raster Catalog
log-msg: Adding items..
log-warning: Args less than required, filling with default (#)
log-msg: Calling (arcpy.AddRastersToMosaicDataset_management)
log-critical: ERROR 999999: Something unexpected caused the tool to fail. Conta\
ct Esri Technical Support (http://esriurl.com/support) to Report a Bug, and ref\
er to the error help for potential solutions or workarounds.
Failed to execute (AddRastersToMosaicDataset).

Installing Missing Packages

  1. conda install pip
  2. "z:\home\arcgis\bin\Python\envs\test\Scripts\pip.exe" install logger

Use of py cache (.pyc) files and Python version

Use of solutionsLib.pyc is not compatible with Python3 [1]. Always use source https://github.com/Esri/mdcs-py/blob/master/scripts/solutionsLib.py.

Z:\home\arcgis\src\SDT-56>"z:\home\arcgis\bin\Python\envs\test\python.exe" MDCS.py 
Traceback (most recent call last):
  File "MDCS.py", line 34, in <module>
    import solutionsLib  # import Raster Solutions library
ImportError: bad magic number in 'solutionsLib': b'\x03\xf3\r\n'
</module>

Troubleshooting

If MDCS hangs, try to restart server using the startup scripts under /arcgis/server/.

References

  1. https://github.com/Miserlou/Zappa/issues/854
  2. https://www.esri.com/arcgis-blog/products/arcgis-pro/imagery/optimizerasters-2-released-available-for-download

Page Last Updated: Jul 9, 2021 at 12:59 PM EDT