Match Symbols to a Style in ArcMap

Here is an interesting article about using a drawing option in ArcMap called Match To Symbols.  By naming a feature attribute with the same symbol name that is in a style, you can quickly symbolize them for feature categories.  Click below to read.

matchtosymbolarcmap

This is just like when us old school GIS types used a field to store a symbol number to symbolize our points, lines, and polygons!

Spider Diagrams

Recently I had a need to generate a spider diagram for my map.  Spider diagrams, also called desire lines for business scenarios, are a series of rays drawn from a central point to related points.  The result shows you the actual area of influence for each central point.

spiderdiagram

Some good examples would be store locations and customers that visit them, library locations and patrons’ home locations, or in my case fire station locations and incidents they responded to.  Continue reading

Parcel Center XY: Using ArcGIS Server REST API

We currently have a business need to get at parcel center points using parcel numbers.  If some of our business systems can communicate using HTTP, I thought using the ArcGIS Server REST API with our map service would be ideal.  But first I needed to figure out how to do it.  What better way to test the concept by building a Python script! Continue reading

15 Minutes With ArcGIS Earth 1.0

ArcGIS Earth 1.0 is ESRI’s newest app that has been marketed as a replacement for users of the soon to be retired Google Earth Enterprise.  ArcGIS Earth is a desktop based interactive globe which allows you to visualize and explore 3D and 2D spatial data.  ArcGIS Earth software is free.

My goal for this post was not to have a comprehensive review of ArcGIS Earth, but to see how fast I could get it up and running with my data without reading any install or user guide.  The less fuss the better.

You first need to download the compressed setup executable from ESRI’s ArcGIS Earth site.

agsearth1

After that, you run the setup executable which wants to “unzip” the needed install files, so pick a directory to place them in.  Next, it will start the install which will ask you where to install the software.  You can take the default location or place it somewhere else that you like.  Once installed, let it start up and you are greeted with a nice globe that spins as it zooms in with a starry background, similar to Google Earth.

agsearth2

I do like the black framed window with minimal, muted controls that you can hide away.

So I just want to add a shapefile to this puppy.  In the upper left on the tool strip I see a “+” icon which I guess is to add data, so I click on it.  I was right.  Doing so I am presented with public data from ArcGIS Online.

agsearth3

I don’t want that but I see an “Add Files” option as well as an “Enter a URL” option.  Cool!  I click on “Add Files” and a “Select files” button appears, which I click on and then a file browser window opens for me to select either a .KML, .KMZ, or .SHP file.  I select my elevation contours shapefile and the globe zooms to the data.

agsearth4

The first thing I notice is that it is a little slow drawing the data.  Maybe too many contour lines?  The other thing I notice is that the vector data is being rasterized for the display on the globe.  Maybe that is what takes some time?  I don’t know, but as you zoom in and out, you see the re-rasterization of the data.

agsearth5

To navigate around, I use my scroll wheel on the mouse to zoom in/out and to pan you left click on the globe and drag it.  To tilt the globe, right click and drag as well.

agsearth6

I also find if you double-left click it also zooms in.  A double-right click seems to center that location in the window.

I did try my 3D SpaceMouse with ArcGIS Earth, but it only would zoom in and out, no pan or tilt was working.  My 3D mouse worked right out of the box with Google Earth and also ArcGlobe, so I think ESRI will need to add the ability to work with other input devices.

I then notice a little blue dot on the list icon in the upper left.

agsearth7

I bet that is telling me I have a layer being drawn on the earth.  Clicking on the icon reveals a list of my data.

agsearth8

I can turn the layer on or off by clicking on the check box.  If I double click on the layer name, the globe zooms to the layer.  If I right click on the layer name, I am presented with “Remove” or “Properties”.  Selecting “Properties” presents me with a Properties window that allows me to change the symbol type, color, weight, and opacity of the contour lines.

agsearth9

If I click on “Summary” in the Properties menu, it lists the name, feature type, file source, any layer description, and coordinate system.  That’s handy!

Now I want to add a KMZ file.  I click on the “+” icon and add my fire hazard zones for LA County.

agsearth10

Again the globe zooms to the data and again I notice slowness.  Why would this be slow with data on my local drive?  Zooming in and out is slow too.  Maybe the background basemap is taking a while to stream to ArcGIS Earth?

One thing I did not mention is that you can click on any of your features to bring up their attributes.  That comes up fast at least.

agsearth11

The last thing I want to do is add one of our ArcGIS Server map services to the globe.  Again I click on the “+” icon and this time select “Enter a URL”.  Here you can enter the full path to your map service, like ” http://server_name/arcgis/rest/services/layer_name/MapServer “.  I added our 2014 orthophotography.

agsearth12

The speed of panning and zooming seem about the same.

agsearth13

Other tools that are available to you are draw tools that allow you to draw points, lines, and polygons; Measure tools for point, line, path, circle, and polygon; Save image to create an image file of your view; Email an image of your view; Print your view; Settings; and Basemap changes.

I do want to mention that under Settings, you can change the startup view to continue where you left off when you open ArcGIS Earth again, and also you can change the default spatial reference from Web Mercator to WGS 84.  Other settings include an atmospheric effect, mouse navigation speed (which did not fix my slowness), and an advanced setting for using a proxy server.

One last thing, in the upper right of the window, you can enter an address to zoom to.  Also you can reorient your globe to north up by clicking on the north arrow icon and click on the home icon to zoom out to the globe.

Overall ArcGIS Earth is a handy tool to display your KML, shapefile, and map service data on an interactive 3D globe.  I would have liked to see it work with data in a geodatabase, but maybe that is for a later release?  I did notice that at the top of the interface there was an option to “Sign in” to your ArcGIS Online or Portal accounts.  Clicking on it presents a sign in window.

agsearth14

This would then give you access to more data that you have in your ArcGIS Online and/or Portal accounts.  Which brings up this observation … I do like the fact that I can use ArcGIS Earth without forcing me to login.  I think that was wise on ESRI’s part.  Not everyone wants an online account, especially if they only deal with local data.  I think this will help the software be more accepted.  We will see.

Should you need ArcGIS Earth help, you can find it here.

I do not know why ArcGIS Earth was slow for me.  I have a pretty powerful workstation.  I found myself becoming very impatient waiting for the globe to keep up with my movements.  If it does not get resolved, I don’t see myself using ArcGIS Earth that much.  ArcGlobe and the 3D in ArcGIS Pro work a lot faster.  Your mileage may vary.

Try ArcGIS Earth out for yourself and see if the tool works out for you. -mike

Updating ArcGIS Server License and PostgreSQL

Our ArcGIS Server licenses were going to expire soon, so I went ahead and updated them with new license files.  All was well … for a while.

A week later one of our ArcGIS Server installations started having issues with the map services.  Basically the map services, though running, stopped displaying the features that were in them.  Looking into the error logs, we found stuff like this:

The Layer:’Parcel’ in Map:’Layers’ is invalid. The base table definition string “bur.ags.parcels” is invalid. No ArcGIS server license found. Contact your geodatabase administrator to update license

What?  I updated the ArcGIS Server license!  What is going on?  Reading the error messages again, the part about “the base table definition string is invalid” reminded me about when I setup the Enterprise Geodatabase.

For this particular ArcGIS Server installation, I decided to setup the Enterprise Geodatabase inside PostgreSQL, store GIS data in it, and use the data for the map services.  When I was setting it up I used the Create Enterprise Geodatabase tool.  One of the parameters for the tool required me to provide a path to the keycodes file that was created when I authorized ArcGIS Server.  I wonder if it is PostgreSQL?

Using ArcCatalog I tried to connect to the Enterprise Geodatabase in PostgreSQL, and here is what I got:

ags-server-postgresql

Aha!  I guess PostgreSQL does not automatically get updated when I update the ArcGIS Server license.  I am running ArcGIS Server 10.2, so the keycodes file (named keycodes) is located under C:\Program Files\ESRI\License10.2\sysgen.  After entering that and pressing OK, I had access to the Enterprise Geodatabase again.  I then restarted the map services that access the Enterprise Geodatabase and all was well again!

Surprisingly my other ArcGIS Server installation’s map services were working fine.  However, they access an Enterprise Geodatabase in Oracle.  I guess Oracle was updated automatically when I updated the ArcGIS Server license file?  I don’t know, but if something similar happens, I will check the Enterprise Geodatabase first!

If you use ArcGIS Server with PostgreSQL, this little fix should help you if you encounter similar problems.  -mike

ODOT Workaround for FILESTREAM and Collector

Do you use SQL Server and are thinking about using Collector to store potentially large amounts and data and pictures?  Check out this article.  Experts from the Ohio Department of Transportation provide a detailed workflow example, with step by step instruction, for using FILESTREAM with Collector for ArcGIS to gather, manage and store extremely large datasets in the best ways possible for performance, management and scalability, in line with current industry best practices.

Extracting Features from Map Services

ArcGIS Server map services are a pretty cool thing.  Someone else is providing GIS data for you to consume in a map or application.  And if they keep it up to date over time, even better.

However, there are some times when you just need the data instead so you can edit/manipulate it for your GIS analysis needs.  If you cannot acquire a shapefile or geodatabase, there is a way to extract the features from a map service.

For you to extract the features, there must be three things in place:

  • You must have access to the REST interface for the map service
  • The layer in the map service that you want must be a feature layer
  • The layer must have a query option with JSON format supported

To demonstrate this, let’s head on over to the map services provided by Southern California Association of Governments (SCAG).  Specifically, the one for city boundaries:

http://maps.scag.ca.gov/scaggis/rest/services/City_Boundaries/MapServer

Since we have access to the REST interface, we can check out the different settings of the map service.  Note there are two layers, City Boundaries (layer 0) and County Boundaries (layer 1).

extract1

We are interested in the City Boundaries layer, so click on it.  This brings up detailed information about the data.

extract2

They were good to include when the data was updated in the Description.  Also note that it is a feature layer and we can query it.  If you scroll down to the bottom you will see the query option in the Supported Operations section.

extract3

Also listed at the bottom are the fields associated with the features.  We should take a look at these next to see what fields we want.  To do this, go back to the previous page and then click on the ArcGIS.com Map option in the View In section at the top:

extract4

This will open up a default map on ArcGIS Online and display the map service layers.

extract5

Click on City Boundaries to view the two layers, then click on the icon that looks like a spreadsheet just below the City Boundaries layer.

extract6

This will bring up the attribute table for the city boundaries.

extract7

After taking a look at the data, all I want are the CITY, ACRES, and COUNTY fields.

Now let’s go back to the REST interface for the City Boundaries layer.  Take a look at the MaxRecordCount setting.  It is set to 1000.  This means that only 1000 records (features) can be returned for each query.  This can be extra work for you if this map service has more than 1000 features.  We need to find out how many features there are.  To do this, scroll to the bottom and click on the Query link in the Supported Operations section.  This will bring up the query interface for the map service.  This allows you to do a lot with the map service and get different results.  I will not go into the details of each parameter.  You can find out more info about them from the ArcGIS REST API reference.  For now, just enter “1=1” (without the quotes) in the Where input field.

extract8

This is a little trick to return as many records as we can.  Also make sure to set Return Geometry to False and Return Count only to True.  With Return Count only set to True, we will actually get all records (an exeption to the MaxRecordCount setting).  Once those are set, click the Query (GET) button at the bottom of the query form.  It looks like nothing happened, but scroll down to the bottom to see the result.

extract9

Note the count is 191.  So we don’t have to be concerned with the 1000 record limit.  We will be able to grab all the features at once.

NOTE: For those of you that are curious about the query, take a look at the URL in your web browser.  This is the REST query syntax used to get the record count.  We will be using something similar in a Python script to extract the actual geometry of all features.  You can mess around with the query form to see what happens and the URL that is created to make that happen.

We will be using a Python script to extract the features from the map service and create a featureset in a geodatabase.  Here is the script:

import arcpy

arcpy.env.overwriteOutput = True

baseURL = "http://maps.scag.ca.gov/scaggis/rest/services/City_Boundaries/MapServer/0/query"
where = "1=1"
fields = "CITY,ACRES,COUNTY"

query = "?where={}&outFields={}&returnGeometry=true&f=json".format(where, fields)

fsURL = baseURL + query

fs = arcpy.FeatureSet()
fs.load(fsURL)

arcpy.CopyFeatures_management(fs, "H:/scag/data.gdb/city_boundaries")

So what does this do?  First it loads the arcpy module and sets the overwrite option to True in case we want to run it again and overwrite the featureset.  A baseURL variable is set to the query URL of the city boundaries map service, a where variable for our where clause for the query, and a fields variable with a comma delimited list of the fields that we want.

Next a query variable is set with the actual query needed to extract the features.  Note the query is in quotes with {} for values that are plugged in by the variables listed in format() at the end.  Also note the returnGeometry parameter is set to true and f (for response format) is set to json.  What this query will do is extract all the records with just the fields we want, return all the geometry (the polygons), and output the whole thing in JSON format.  With the baseURL and query variables set, we merge them together for one long query URL as the fsURL variable.

Finally, we create an empty featureset (the fs variable), load the returned JSON data from the map service query into it, then copy it as a featureset to our geodatabase.

It takes a second or two to run the script.  Check out the result!

extract10

And here are the attributes we wanted for all 191 records.

extract11

I do want to point out that the data you get will be in the same projection as the map service.  You can find out what projection the map service is in by looking at the Spatial Reference setting in the REST interface.  Note this map service is using 26911:

extract12

What the heck is that?  Go over to spatialreference.org and enter the number in the search field.  It will tell you this is the NAD83 UTM zone 11N projection.  If you wanted something else, like NAD83 California State Plane zone 5 US feet (2229), you would edit the Python script to include the output spatial reference parameter in your query, like this: “&outSR=2229”.  The data is projected on the fly for you and saved to your geodatabase!

I hope this helps you out when you are looking for data in map services.  -mike