Geofoolery

 

Like many other local government geospatial types, a good deal of my recent time has been devoted to obtaining, processing, and presenting 2010 Census Redistricting Data for my jursdiction. If it's possible for an experience to be both vexing and rewarding, this has been a textbook case.

Others have detailed how to obtain Census data and the often fustrating experience of simply trying to find relevant data, but let's face it, getting Census data is just plain complicated and convoluted. After experiencing frustrations of my own and uttering a number of Munchian screams among other things, I finally got the data I was looking for. Questions from the public and elected officials served as a clarion call for getting the data into a format that was much easier for potential users to digest. Even more importantly, with an eye toward redistricting, appropriate data needed to be mapped and visualized. This was yet again a job for a simple web mapping application.

My vision for an application was one that allowed users to compare 2000 and 2010 figures for every level of Census geography in my county, from blocks on up to the entire county. I eventually came to realize that at the block level, decennial comparisons are often not worth the paper they're written on because of shifting block boundaries. This was especially problematic in parts of the county where significant levels of growth had occurred since 2000. After coming to that realization, I looked at the next level up, block groups, and found that while boundaries had shifted, the amount of change wasn't drastic. The same was with true with tracts. Changed boundaries weren't a concern with the remaining levels of geography since they were all tied to political subdivisions.

Now that I had my eye on the (moving) goalpost, the question was how to marry the 2000 data with the 2010 boundaries. Certainly this could be done using ArcGIS geoprocessing tools, but I also wanted to test some different approaches while being mindful of data accuracy and consistency. Enter SpatiaLite. Pounding out a little SQL and seeing your results within seconds versus running a collection of GUI-based tools or a model, firing up ArcMap, and then loading the resultant data set, was a big timesaver. 

After testing a number of different approaches in SpatiaLite, I opted to use block data as the basis for my block group and tract data sets. My approach consisted of nothing more than a series of spatial joins. However, because of the changes in boundaries, I couldn't just take 2000 block data and cram it into 2010 block groups and tracts. After visually inspecting areas where block group and tract boundaries had changed, it appeared that spatial joins between the centroids of the 2000 blocks and the 2010 geographies' polygons would produce reasonably accurate 2000-2010 comparisons.

Five or six lines of pretty ordinary SQL in SpatiaLite, most of it just renaming fields, and mere seconds were all it took per dataset created. I can only imagine how many tools and time would have been required to achieve the same result with ArcGIS. Although I've used SpatiaLite and its cousin PostGIS to perform queries on numerous occasions, I was once again reminded that opening a GUI and clicking buttons often becomes a process that occurs with little thought given to the hamster running under the hood. Relationships between the datasets being processed are often not considered. It's good to peel back the lid and see a bit more of the whole picture, not to mention simply having more control over geoprocessing operations by tweaking queries.

Now that I had my data, the question was what platform to use to get it on the web. Feeling that I needed to get more experience with the ArcGIS Server JavaScript API, that was my choice. This was my first fairly significant foray into it. Having OpenLayers experience was helpful, but I had to hit my head on my desk and splash cold water on my face on several occasions to snap out of an OpenLayers mindset. I exported my geometry tables out of SpatiaLite as shapefiles and then loaded them into a file geodatabase for my ArcGIS Server service.

The result:

Some lessons were learned, or in some cases, reinforced:

• A hybrid proprietary-open source model is awfully compelling. Even Esri thinks so.
• Shapefiles...shapefiles...shapefiles...sigh...why are we still dealing with them and their limitations in the second decade of the 21st century? Granted, I opted for an approach where I had to use them, but still...
• The old mantra of loading everything into ArcSDE continues to ring ever more hollow for me. For small datasets, local file geodatabases clearly perform well in ArcGIS Server.
• Caching isn't everything. In this case, the time and resources involved in creating and maintaining a cache weren't worth what would likely be a minimal boost in performance.
• Google Chart Tools provide an absurdly simple way to deploy whiz-bang interactive charts in your JavaScript application.
• Having some jQuery experience, I didn't know what to make of Dojo at first. I warmed up to it, but jQuery would still be my first choice.
• The JS API is a fine option when you're working with ArcGIS Server, but it strikes me as missing a level of flexibility offered by OpenLayers. There were a few OpenLayers properties and functions I would have liked to have used, but I found no obvious equivalents in the JS API. They weren't dealbreakers though.

A few items of interest related to open source GIS in local government crossed my proverbial desk recently.

From Government Technology, March 15, 2011:

 http://www.govtech.com/e-government/Open-Source-Software-Oregon-Transportation.html ​

Open source software is providing functions for the district that McHugh previously doubted it could handle — most notably, GIS

 Farther down the page in the Government Technology article:

 ​Beginning in 2007, the department took a larger bite, switching out several major Esri GIS products with open source alternatives. McHugh replaced Esri ArcSDE, ArcIMS and MapObjects with open source products GeoServer, OpenLayers and PostGIS. At the time of the switch, the agency was considering whether or not to add Esri’s ArcServer. 

“ArcServer was going to be a big investment for us, and it didn’t really align with our standards. The existing platform, ArcIMS, wasn’t really meeting our needs,” Bibiana McHugh said. “After laying everything out, GeoServer immediately popped to the top. It was clear this was an option that could meet all of our needs, save a lot of money — and it aligned with all of our Internet software standards.”​​

 

From Regina Obe: 

http://www.bostongis.com/blog/index.php?/archives/167-Norht-Carolina-GIS-Conference-2011-Impressions.html

One thing that really stroke me was how deeply PostGIS had penetrated into government. I guess I don't get out enough. In one of the talks we were in -- the basic message was We chose PostGIS, not because it was cheaper, but because it was better. and this statement was made after having failed getting ArcSDE to perform to the level required by end-users. That seems to be a new milestone that open source has achieved that it's no longer just competing on price, but it can outperform on performance and functionality as well. I think for us we originally chose PostGIS because it was all we could afford, but even when we could afford the others, we settled on PostGIS, because it was just better even when comparing on features, performance, ease of maintainability, cross platform. Price is just icing on the cake. 

 

​A job listing for a GIS Programmer with the City of Cedar Rapids, Iowa:     

http://www.gjc.org/gjc-cgi/showjob.pl?id=1300293489

​Interest in Open Source products and Open Standards, specifically those relating to geospatial Mapserver, OpenLayers, GML, WCS, WFS, WMS.

 

Particularly noteworthy to me is that there is a clear undercurrent in the TriMet article and Regina Obe's blog post that open source tools are sometimes just simply better fits for an agency's mission and tasks. Although the usual financial arguments are mentioned, they are somewhat secondary in both of these pieces. While open source tools are certainly no panacea and have their own shortcomings, arguments that they are at times better options are very much in line with what I find in my work. It's all about selecting the right tool for the mission or job.

Granted, three items that refer to use of open source GIS in local government are certainly not a definitive sign of a wholesale movement, but they're still very interesting. Since so many agencies are heavily invested in Esri and other proprietary stacks, I don't think we're at the point where those of us in local government need to acquire absolutely mad skills in many facets of GIS to stay relevant. But, it is becoming clearer that at the very least, exposure to and willingness to experiment with tools outside of a traditional proprietary stack are good notches to have on your belt. Never-ending budgetary pressures pushing against agencies from all sides will probably only serve to bolster this.

That proprietary stack your agency invested in may not be going away tomorrow, six months from now, two years from now, or even ever, but having increasingly viable alternatives available is a very reassuring thing in tough times.   

 

Like many geospatial people, I'm familiar with OpenStreetMap (OSM) and have put its offerings to occasional use, not to mention contributing periodic edits. Despite my experience with OSM, I would classify myself as an intermediate user. Although I find the project fascinating and valuable in so many ways, I have just never delved as deeply I should into a number of significant aspects of the project. A new book, OpenStreetMap by Jonathan Bennett, provided me with a needed golden ticket to fill gaps in my OSM knowledge.

The book begins with a discussion of the project's history, structure, and benefits. Bennett then goes on to point the reader to valuable online resources such as wikis, mailing lists, and the OSM community's social media presence. He rightly urges project volunteers to take full advantage of these resources and clearly demonstrates their value. 

Although I wouldn't have expected it when I first started reading the book, Chapter 3, Gathering Data using GPS, may have been the most worthwhile chapter of the book from my perspective. The chapter provides a comprehensive view of what is involved in collecting data for OSM with GPS and discusses many aspects of this approach I hadn't ever considered. Bennett's ideas on documentation associated with GPS data collection and his discussion on collection techniques themselves caused a number of light bulbs to go off in my head.

The editing discussion begins by covering OSM's data model and then moves into fairly comprehensive coverage of various editors, e.g. Potlatch, JOSM, and Merkaartor. My previous awareness of editors other than Potlatch notwithstanding, because Potlatch is fine for many tasks and quick and easy, I've used it almost exclusively to make edits to OSM. Chapter 5's discussion of alternatives should help dislodge myself from my overreliance on Potlatch. I do wish this chapter had mentioned other editing options such as the QGIS OSM plugin and the ArcGIS Editor for OpenStreetMap, but as these editing options are geared more toward GIS users, they may have fallen outside of the general audience that I believe the book intends to reach. Chapter 6, Mapping and Editing Techniques, ends the editing discussion with a number of real world examples and advice on making edits appropriate for the examples. Bennett concludes his treatment of the editing process by covering a number of tools available for checking for problems. This portion of the book was also helpful to me since it covered ground quite new to me.

Chapters 8 and 9 show the reader how to produce customized maps with the openstreetmap.org exporter and Kosmos, as well as getting raw OSM data by downloading planet files, using the OSM REST API, or using the extended API (Xapi). Chapter 10's discussion of Osmosis topped off a group of three chapters that will serve as a very handy reference for me in better leveraging these tools. Again, these are OSM capabilities I was aware of, but only marginally so before reading the book.  

The book concludes with Chapter 11's brief discussion of the future of the project. The reader is undoubtedly shown that OSM is continuing to grow and evolve with changes to the project's licensing and new tools and capabilities being discussed.

Bennett succeeds in presenting a comprehensive overview of OSM while remaining pragmatic and not overly technical. Those new to OSM should be able to pick up the book and get off the ground fairly quickly, while more advanced users who feel like they still have much to learn about the project, such as myself, should find the book full of good advice and a helpful pointer to new resources. The book is a very good read for anyone interested in OSM and a good reference to have on hand.

The state of location-based social networking (LBSN) has been a topic I've been chewing on and spitting out a lot lately thanks to some recent reading like this and this, as well as some Twitter insight like this and this. Wednesday's announcement of Facebook Places, surely the worst kept secret in this arena, just threw even more gasoline on the fire and got my mind running a million miles a second on the topic.

The recent ascent of Foursquare seems to have created a perception that LBSN is somehow new, sometimes even among people who should probably know better. With services like Brightkite, Gowalla, and Loopt having been in existence for a number of years before Foursquare, LBSN is of course hardly new. Somehow I'd like to think that after a few years of existence, LBSN can give us something more than tweets announcing that someone has just become the mayor of Bunghole Liquors, or being reduced to yet another way to bombard users with marketing messages. I agree with those who expressed in the tweets linked above that LBSN being very much tied to a check-in model is probably detrimental to it in the long run. 

I may be a Foursquare and Brightkite user, but I'm to the point where I want LBSN to be more meaty and substantial. Badges, pins, and mayorships have run their course and check-in fatigue often sets in for me. Questioning why I'm checking in at a particular venue often crosses my mind. Is sharing that I'm at Kroger because I need to grab some bread and milk really that compelling and valuable to those following me? It's highly doubtful. I know that I'm not alone in feeling this kind of ambivalence toward LBSN.

I've been trying to think of other applications of LBSN, but I'm coming up a bit empty on ideas. Maybe it's me, maybe it's something inherent to the check-in model - I don't know. What I am fairly confident of is that certainly someone has to be able to come up with some compelling use cases for LBSN beyond plain old check-ins.

Geofencing certainly has some interesting potential. Brightkite's long-forgotten Wall feature is interesting and seems like it could be put to good use in some way. For all the flak it's been getting over just about everything lately, Google should probably get some credit for Latitude even though no one seems to use it. LBSN running in the background is far more interesting to me and strikes me as far more useful than the check-in model, although there are certainly privacy implications users need to be aware of. Microsoft Vine never got much attention and seems to have been completely forgotten, which is a shame since it's an LBSN service that has an important purpose - providing a means for people to stay connected in case of emergency.  Lastly, although it's not exactly a check-in service, Twitter's Geolocation API also seems like it has potential just waiting under the hood, although the API seems to be completely underutilized.

The question remains for me though - what can we really do with LBSN beyond announcing our location to the world and trying to sell stuff? Maybe there's a role for it in emergency response and public safety, maybe simple fleet management and logistics types of applications. This is the stuff I'm thinking about right now, but until I see some more substantial uses of LBSN, I'm afraid I'm stuck asking the same question Ms. Clara Peller made famous a good 25 years ago.

 

By way of Slashdot, I stumbled across the following use of geospatial technology in municipal code enforcement...

It goes without saying (although I'm still saying it here) that I'm all for incorporating geospatial techologies into government workflows, not to mention fair and equitable enforcement and application of codes and laws. Geospatial technologies certainly play an already large, and increasing, role in enforcing codes and laws. Still, I'm not sure the old adage about there being no such thing as bad press is quite right here. I certainly wouldn't want my agency's GIS efforts to receive this kind of publicity when the use of geospatial technologies in local government results in many good things that enhance civic life.

No matter what, I have a hunch the 'Eye in the Sky' qualities of this story would do Alan Parsons and his bandmates proud.

I recently finished reading 'On the Grid' by Scott Huler after hearing about the book thanks to an interview with Huler on the public radio program 'Here and Now'. The book revolves around a question that most of us have probably kicked around at some point in time - how does the infrastructure that is essential to modern life work, and why does it even work at all? Although cable companies often do their best to prove otherwise, and we certainly don't like it when our power goes out, our infrastructure is remarkably reliable. Why is this, and why doesn't it simply collapse under the immense demand its users often place on it? Having worked a bit with infrastructure earlier in my career, collecting sewer and water system data in the field with GPS as part of mapping these systems, and of course having pondered the same kind of questions as Huler, I was hooked after listening to the interview. The book clearly sounded like a fascinating read. I wasn't disappointed. As much as I appreciate engineers, I found it very compelling and appealing that the book was written not by an engineer, but just an ordinary guy just trying to understand how all this stuff works.

Huler takes what I found to be a surprisingly through look at the infrastructure in his own locale of Raleigh, NC. His journey started in his backyard, eventually leading him through sewers and trying to trace the routes that various pipes and wires take from their origins to his house. As part of his journey, Huler obtained access to a panoply of services and facilities where few laypeople probably ever set foot, for example the control room of a nuclear plant and a cable TV headend. He was able to ask many questions of many different people behind the operation and maintenance of these systems, a chance many of us would surely jump at.  

As for those of us who are geospatially inclined, Huler throws us a pretty big bone. On several occasions, Huler pointed out the role that GIS now plays in the workings of major portions of modern infrastructure. He described GIS as the "systems that make it all happen" in the 'Here and Now' interview. In addition to mentioning GIS in relation to several types of infrastructure, Huler visited the City of Raleigh's GIS Department. Huler may not be a geospatial professional, but in the span of about three pages he sums up very well how GIS in local government often operates, as well as how crucial it has become to the operation of services and infrastructure provided by local government. I was surprised and pleased to see that he even touched on the question of charging for governmental GIS data. In Raleigh's case, the GIS department, as Huler puts it, "spent too much time gatekeeping" when it was charging for data. Being admittedly biased, I was enthusiastically applauding the GIS references in the book, to the point of being suprised that my wife didn't shove me out the front door out of sheer annoyance. Congratulations are in order for Colleen Sharpe, Raleigh's GIS Manager, and her staff for such great exposure to an audience not familiar with GIS, as well as clearly doing a good job of explaining GIS in a nutshell to Huler.

 

Parts of northern and central Illinois were raked by a severe weather outbreak that produced a number of tornadoes during the evening of Saturday, June 5. My county, Kankakee County, didn't entirely escape the severe weather, but we were fortunate in that there were only minor injuries and that the damage that occurred here wasn't as extensive in other areas. Because of the widespread nature of the event, my department put together a simple interactive map of local storm reports received by the National Weather Service's Chicago and Central Illinois offices. The map is embedded below, but because Posterous seems to have some limitations on what can be done with html in posts, it's best experienced by clicking on the link.

Despite my increasing level of experience with OpenLayers, I hadn't used OpenStreetMap as an OpenLayers base layer up to this point. This was a good chance to try that. The Kankakee County data is our own being served up by GeoServer. We grabbed a shapefile of the local storm reports from the Iowa Envrionmental Mesonet LSR app and loaded it into PostGIS. By the way, as a geospatial and weather nerd, the Iowa Environmental Mesonet in general has become one of my favorite destinations for weather information and data. With a number of downloadable datasets in common GIS formats, some nice Google Maps mashups, and some nice OGC web services, there's a lot there to keep geospatial types happy. Check it out sometime when you have a minute.

Maybe it's the slight case of social media burnout and information overload I feel like I just shook off talking, but mustering up much excitement over Google Buzz is taking some effort. In all fairness to Google, I don't have Buzz yet and have not tried it, but I find myself wondering if I have the time and desire to possibly keep on top of yet another social networking platform (aka the dreaded YASN). My lukewarm interest in Buzz probably stems from several things.

Few topics seem to have geospatial people on Twitter drawing battle lines more than the GISP. I'm sure it goes without saying to many reading this that the topic came up again last week and was the subject of some pointed discussion.

I have been quite critical and skeptical of the GISP in the past, a point of view that I voiced in a very similar Twitter discussion in perhaps April or May 2009. At the time I said that I felt that in the absence of an exam, the GISP's value was very much in doubt for me. Looking at the GISP through the eyes of a manager, I said that it was at best an extremely minor factor in making hiring decisions and that experience and education would always trump the GISP in evaluating a candidate's background. I also felt that after working in GIS for ten years, that the GISP would likely have little benefit to me as a means of establishing or certifying my level of experience, nor would it do much for me in terms of career advancement.

My opinion has morphed since then into what I would call "indifferent skepticism", which I'll explain a bit more later. I even began the process of applying for the GISP in July 2009, but have yet to finish since many other things I consider to be more important, both personal and professional, seem to be consuming my time. My main motivation for applying, something I still plan to do, is that in my field of local government I have noticed that a significant percentage of my counterparts across my state have their GISPs. These are people for whom I have nothing but respect and admiration, and whose relationships have been immensely valuable to me as a GIS practitioner working in local government. If these people see enough value in the GISP to get theirs, I thought that perhaps I needed to take a step back and rethink the issue.

Whether my anecdotal observation is a sign that the GISP is gaining traction amongst GIS practitioners working in local government is still an unanswered question for me, but it has me thinking that it may only be a matter of time before the GISP could become a factor in hiring decisions, at least at management levels. It is not at all unusual in local government for H.R. staff unfamiliar with GIS, much less the GISP, to make hiring decisions for management level positions. I can quite easily see why such H.R. staff might come to think that because the last person holding a particular GIS management position held the GISP, that the certification is a necessary requirement for the position and for working in GIS, or at least is a very desirable quality in an applicant. That being said, I have not seen any job postings,save for one or two, that explicitly require a GISP, but it is my understanding that this is happening with more frequency in parts of North America, particularly in the public sector in British Columbia.

As for just what this notion of "indifferent skepticism" is, I'll be the first to admit that it sounds self-contradictory, but I can't think of any other way  to express my ambivalence about the GISP. "Skepticism" comes from the fact that many of my doubts and questions regarding the GISP's value haven't necessarily diminished, but the "indifferent" portion is due to the fact that although I'm still not convinced the GISP is as advantageous as some of its proponents argue, I also don't see it as particularly harmful. Perhaps it could be useful to someone relatively new to the field in establishing deeper professional credibility, but of course anyone thinking about it from that perspective would do well to look at some of the criticism leveled against the GISP before putting too many eggs in that basket. At the very least, having a GISP can't hurt when balanced with other compelling attributes in a geospatial professional's background.

One theme in the Twitter debate I found interesting was that several people have allowing their GISPs to lapse because they feel they are well enough established enough in their career where the certification does not lend them any additional advantage. Re-certification may be the pudding where we'll find the proof of the GISP's value to those in the profession. If many GISPs decide not to renew once their original certifications expire, GISP proponents should probably be asking themselves some questions about the current structure of the certification. Conversely, a healthy number of renewals combined with some growth may be a sign of increased acceptance and value within the field. With many original certifications coming due now or in the very near future, we may have some idea soon enough.