When you think of iterative development, you’re unlikely to also think “government agency.” You’re even less likely to think “MTA.”

But that might not be the case in the future, as today the MTA showed a strong sign that the situation is changing: They listened to feedback and responded. Quickly.

In my post yesterday, I identified several aspects of the new developer resources page that could be improved. On Thursday and today I had more communications with representatives from the agency and provided some additional feedback.

This afternoon — less than 48 hours after my post went online — they addressed several of my suggestions for improvement. First, they’ve significantly streamlined the download process, replacing a form requiring lots of personal information with an optional one that simply asks for your email address — so you can be notified of data updates — and what you plan to use the data for. This information is entirely optional as there is also now a link to take you directly to the download page.

Second, they explicitly added information about how frequently the schedules are updated, as well as upload dates for each of the datasets.

Lastly, they’ve posted GTFS data for the MTA Bus Company. This is another big milestone, as it means that now every single MTA transit agency has its data online, for free, in GTFS.

These changes — along with the rapidity with which they were made — show that the MTA is serious about open data and proactively working with the developer community.

For an agency that gets a lot of flack (both unwarranted and warranted), they deserve credit where credit is due. Bravo, MTA!

It’s here: The MTA has officially launched its redesigned website, complete with a spiffy new look, access to multiple trip planners, and a convenient way to quickly check on the status of subway and bus lines.

While there’s much to say about the site’s new design, what excites us most is the developer center, which I think is the most important announcement of the day. Here’s a quick rundown of the news, starting with…

…What’s good

Before talking about the new stuff, it’s useful to think about just how far things have come in a relatively short amount of time. Just five months ago, developers were being threatened with legal action by the MTA, the only way to get any raw schedule data was to submit a formal FOIL request (and get a CD weeks later with data in an undocumented and cryptic format), and there was no good avenue for developers to positively engage with the agency.

Contrast that with the situation today: The MTA has released GTFS data for the entire subway system, NYCT buses, Metro-North Railroad, Long Island Rail Road, and Long Island Bus. This is a tremendous and welcome step forward. With the click of a button, developers now have access to the majority of the schedule data for NYC trains and buses, all in a standard format.

Also encouraging is that the MTA has expanded its Twitter (and Facebook) presence. Follow @MTAInsider for updates from inside MTA HQ. Here’s hoping the agency uses this as an opportunity to listen as well as talk to the transit riders.

…What needs to be improved

Personal information shouldn’t be required just to download the GTFS. I understand why the agency has an interest in collecting developers’ email addresses, but I think the download process should be as simple and straightforward as possible. Right now, the process is overly complicated, requiring everything from a street address to your phone number to the IP address range where your application will be used. This information should all be optional. You should only need to enter your address if you want to receive emails when new schedules are posted.

There is no MTA developer mailing list. Such a list would be a fantastic resource and would greatly improve communications between the MTA and the developer community. This could be easily implemented and hopefully will be soon.  Other agencies, such as MassDOT, have seen great success using a public mailing list to engage with developers.

Another issue is that there’s still not a clear path for applications to freely and easily use the standard route markers to properly identify lines. A clear, click-through license that explicitly grants usage of these symbols for such purposes — for both commercial and non-commercial use — would be another important step forward. It would also further the social goal of trademarks: reducing consumer confusion in the marketplace, making it easier for riders to get information on their preferred bus and subway routes.

Missing from the data sets released today is schedule data for the MTA Bus Company, which operates a significant portion of the buses in New York City, and the Staten Island Railway (Update: Looks like the NYCT Subway GTFS includes the SIR schedule!). However, my guess is that it is only a matter of time until this data is also released. Other datasets — from ridership numbers to greater facility information — would also be welcomed. Hopefully these and other datasets are on the way.

One such dataset that’s of particular interest to me is subway entrance and geometry data. The MTA has previously made the argument that releasing such data would pose a security threat, but this seems far fetched, particularly since all of this data has already been released, just in a manner that’s less useful for developers. The neighborhood subway maps, which are displayed prominently in subway stations throughout the city, show the station entrances and geometries, and the NYCityMap operated by NYC DoITT has a GIS layer with the location of every single subway entrance.

The final big omission is the lack of real-time data. Since such data doesn’t exist for most of the system, it’s unreasonable to expect it to be released today. However, there is some real-time transit data in New York — namely for the L train and the 34th st buses in Manhattan — and this information should be released for developers to build applications on top of.

…What’s unknown

One big unknown is how weekly service advisories will be handled. With so much weekend track work going on, everybody knows that subway service can change dramatically come Friday nights. Weekly service alerts and schedule changes, provided in a well structured and machine-readable format, would be immensely useful and help ensure that transit apps provide riders with the most accurate information possible. With the MTA now defaulting to Google’s trip planner on its homepage, the need to get updated GTFS data out on a regular basis is particularly acute.

The second big unknown is how responsive and open to developer feedback the agency will be. The announcement today suggest they are serious about leveraging the outside development community, and I have witnessed a palpable shift in the agency’s willingness to engage over the past several months. Here’s hoping that things will continue moving in the right direction.

…What’s next

These improvements did not come about easily. There were substantial logistical, legal, and political hurdles to overcome, and I know many people, both inside and outside the agency, worked hard to bring about these changes.

The NY Open Transit Data group has long been advocating for these changes and working to constructively engage with the MTA. I’m pleased to say that today’s announcements incorporate some of the core recommendations we’ve made to the agency on open data policy. Our group’s next meetup is Wednesday, January 20 at 6:30pm. Come join us to talk about what these developments mean, what’s next, and, most importantly, how we can use this newly opened data to improve transit in New York.

I’m proud  that the city I call home has joined the ranks of those providing open transit data, and I can’t wait to see what comes next, both from the the MTA and the New York tech community. It’s going to be another great year for open data.

Update 1/15/2010: Two days after writing this post, the MTA has already addressed several of these issues!

The quest for open transit data in New York continues, but the Times’ coverage today of the upcoming launch of the MTA’s new website gives cause to be optimistic. As the Times reports, the MTA is set to launch a redesign of its website this Wednesday, giving the agency’s site a much needed — and appreciated — overhaul. The overall design of the site looks to be greatly improved, and the subway service status on the front page is alone reason to celebrate, as anyone who’s been bitten by weekend service changes will surely understand.

Another welcome change is the addition of the trip planner to the front page. Interestingly, the default option now uses Google’s transit planner, though the screenshots reveal that you’ll also be able to plan trips using either Trips 1-2-3 or the in-house MTA trip planner.

The most exciting part for open data geeks though is this promising morsel:

The new site will also make it easier for outside software designers to get free access to system timetables and routes.

The article contains no further information about what this means, though the screenshot does show a “Developer Resources” link on the lower right-hand corner of the page.

The MTA has hinted for a while at changes to its developer and licensing policies, but beyond the cessation of legal threats last August, there’s been virtually no public announcements on the topic. Many people, including those of us here at TOPP who founded the NY Open Transit Data group, have long advocated and worked to open up New York’s transit data. We’ve had increasingly positive interactions with the MTA, particularly since the arrival of chairman and CEO Jay Walder last October, but are still waiting to see results.

It’s unlikely that the launch on Wednesday will be perfect, but I think it will prove to be a significant step toward the goal so many of us share: universal access to free, complete, and up-to-date transit data for New York.

It looks like it’s going to be a good week for open data.

Last week the first NYC BigApps meetup was held at our office and later this month on November 21st we’ll also serve as the venue for the NYC BigApps DevCamp (please RSVP).

The meetup last week was a good opportunity for app developers to learn more about the process and have a chance to talk to the people facilitating the contest. Brandon Kessler from ChallengePost led the meetup and helped to better explain the contest. Peter Robinson from NYC EDC was there to help explain the objectives and background of BigApps. Sam Litt and Murugan Kanpa from the NYC Department of Information Technology and Telecommunications (DoITT) were there to help answer a number of questions and field a lot of feedback about city data and NYC DataMine. Lou Klepner helped live-stream the meetup and the video is now available on Vimeo:

NYC BigApps – Nov 2, 2009 – Final from Lou Klepner on Vimeo.

Tim O’Reilly often describes the government as a platform, John Geraci provided us with the The Four Pillars of an Open Civic System, and Micah Sifry offered the Three Branches of We.gov. Here I present The Root, Branches, and Fruit of Government as an Open Platform.

The recent Gov 2.0 Summit was primarily focused around “Government as a Platform” and this theme was interpreted in a variety of ways. Many of the talks at both the Expo and the Summit used terms like “2.0,” “platform,” and “open” ambiguously. I personally use the label “open government” interchangeably with what I understand to be “government 2.0,” but what does “open” really mean within the context of government and technology platforms?

The web and democracy as open platforms.

(more…)