[Rdap] Next Generation Data.gov

[Joe Hourcle]

Yesterday, I attended a workshop at the GSA on the 'next generation 
data.gov'.  After a survey in the breakout session, I realized I was the 
only non-manager who attended, and it was for the session on APIs that 
they said was the 'more technical' session.  (I admit, I wasn't actually 
invited directly, I was forwarded the invite by my ATR (the civil servant 
who directs my work ... pretty much my boss))

Anyway, the basic summary is this:

   *  data.gov is being upgraded.

   *  they're using a platform from Socrata : http://www.socrata.com/
      It's basically a shared dataset hosting platform, with some built in
      tools for interaction & visualization.

   *  people will be able to interact with the data, not just download
      files, provided that as part of the submission process, you actually
      describe the columns, etc.  You can also pre-define common filters,
      sorting, views, visualizations, etc.

   *  there's also a 'social' component (one of those things that we
      mentioned in the RDAP 'Future of Digital Libraries').  They said that
      the owner for each dataset could define what level of social
      interaction was allowed.  I'm not sure all of what was allowed (I
      think they mentioned commenting and defining views for other people
      to use), but they could be moderated or disallowed.

   *  you can run a local server to expose your data using their API, and
      then just register it with them, and it'll make calls to your server
      to get the data.

   *  application developers can register to get an ID to use the API.
      it'll automatically rate throttle any that are being too abusive, but
      it also allows for dataset owners to see who's using their data, or
      for end users to see what tools have been built to use the data.
      (there was also a request for data owners to be able to send a
      message to all of the developers using their data, so they could warn
      of possible upcoming changes)


Anyway ...

They showed off a lot of cool features that'd probably be useful for most 
tabular data.  They showed an import screen that had options for 
'dataset', 'chart', 'calendar', and there might've been a forth, but I got 
up too early, and can't remember.  I told 'em I had a few million images, 
and it didn't sound like they were really geared towards that ... maybe to 
serve the catalog of the data, but not the data itself)

I asked about putting NSF research data in there, and I got a kinda 
roundabout answer about how it'd have to be approved through the 'normal 
agency channels', and I don't know if NSF would want us mixing this type 
of research 'data' in with their other 'data'.

Due to the nature of what's being done, I don't think it'd qualify under 
TRAC, so you'd likely want a separate archival copy of the data, but I 
could be mistaken.

They offer a generic API for serving tabular data, (Socrata Open Data 
API) so it's possible that other people could implement it, even if you 
don't want to license their product, or you could write something to 
harvest the various data sources.  I haven't looked into the spec, so I 
don't know how hard it'd be to try to translate between something like 
IVOA TAP (http://www.ivoa.net/Documents/TAP/).

...

It looks like Socrata's updating their website right now, but I wasn't 
given any sort of an NDA to sign, and I'm assuming they'd need some load 
testing, etc, so once it's back up:

Beta of the new site.  (you have to register ... it let me in immediately, 
but I used a '.gov' address, and it's down right now, so I can't test with 
one of my other addresses):

 	http://datagov.socrata.com/

Documentation (for data submission, API usage, etc)

 	http://dev.socrata.com/

-Joe



---------- Forwarded message ----------
Date: Thu, 14 Apr 2011 21:05:22 -0500
From: "hyon.kim at gsa.gov" <hyon.kim at gsa.gov>
To: "marion.royal at gsa.gov" <marion.royal at gsa.gov>
Cc: "chris.metcalf at socrata.com" <chris.metcalf at socrata.com>,
     "charles at socrata.com" <charles at socrata.com>,
     "saf.rabah at socrata.com" <saf.rabah at socrata.com>
Subject: Next Generation Data.gov Platform - Link to Workshop Materials

Thank you for your interest in the Next Generation Data.gov Platform.  We have posted the agenda, presentations and the Getting
Started Guide at the following link:

http://www.socrata.com/datagov/workshop/presentations/

We will be following up with those of you who expressed interest in participating in the new platform.

We will keep you informed of our progress as we move toward the launch of the Next Generation Data.gov Platform.

Thank you.

Hyon Kim
Deputy Program Director
Data.gov
(202) 694 8148