Last night’s New York Public Transit Data Summit was a resounding success. We had a room packed with passionate and thoughtful people eager to help make public transit in New York more efficient, accessible, and and easy to use.

For more than two hours, the group — comprised of over two dozen transit advocates, mobile and web developers, urban planners, lawyers, and open government supporters — discussed the current climate for developing transit applications, its shortcomings, and how the community can work with the MTA to improve things. (more…)

The effort to open municipal data is an initiative with momentum. Inspired in part by the transparency mandate on the federal level that gave us the first ever White House CIO and data.gov, cities across the country are opening up. One city in particular set the scene before this all hit the national stage: Washington D.C. has delivered precedents like the first online city data directory, the first open API for 311 service requests, and open invitations for developers to produce apps with city data using initiatives like Apps for Democracy.  D.C. has laid the groundwork for a national model by delivering some killer apps. Previously the CTO for D.C., Vivek Kundra is now the national CIO. This model has not yet been fully embraced by New York City, but it’s getting close. In fact the New York State Senate Office of the CIO has been paving the way for opening government data on the state level.

(more…)

We’re very excited about the new version of the Orton Family Foundation’s Community Almanac that we’ve been working on at TOPP Labs. In this post I’ll share some of the design decisions that went into that project and some of the reasons why things have changed.

The Opening Line

In order to make it easier for users to contribute to the site, we decided to ditch the previous workflow—an impeding multi-step wizard that walked you through registering and adding your story—and we replaced it with a more unobtrusive workflow. Now users can jump right in and start adding whatever content they want in any order they like, even before they’ve registered or signed in.

Below are some of the wireframe mockups outlining this new just-in-time workflow:

Note: There are a few great ideas in these mockups that didn’t make it into this iteration: namely user profiles and an achievement rewards system. Maybe they’ll make it into a future release?

The Well-Worn Page

In addition to improving the workflow, we wanted to push a new book/page metaphor throughout the site. What was previously referred to as a story is now a page. Users add pages to their community’s almanac, or book, with the site as a whole is presented more as a collection of almanacs or library of books.

Not only was the term “story” inaccurate, as users can publish whatever type of content they like—text, images, maps, audio, video, or PDFs, none of which are necessarily narrative—but the term “page” is also more straightforward and familiar. Users are accustomed to viewing and adding “pages” on a website. Additionally, extending the “page” metaphor to include books and a library provided a more cohesive aesthetic direction and was also just more fun.

The Carefully Considered Cover

Once our ideas for the new user workflow and bibliophilic metaphors were established, I kicked off the visual design with a mood board in order to set the general tone for the look & feel:

The mood board sought to create a warm, inviting, and familiar vibe based that reflects the “heart & soul” sentiments of the Orton Foundation by referencing traditional, tactile imagery.

With this mood board and the book page metaphor in mind, I began sketching ideas for the basic structure and layout. The final sketch depicts an off-centered book where the recto page contains the primary content and the verso page acts as a sidebar for secondary content.  The charm of this layout resides in its flexible width. Although it behaves as a fixed-width layout, it’s not. In larger windows the sidebar items float beside each other, forming columns across the left-hand page. If your monitor is ridiculously large you can even see the whole book!

This fluid layout inspired quite a few bells and whistles, not to mention a couple of fun Easter eggs. The informational tour has been moved to home page as a little slideshow-style presentation that directs users to the map or signup. There’s also a cool slide-down login area and shuffling stack of almanacs on the home page. Check out the parallax scrolling that happens in the header illustration while you resize the width of the browser window. And we didn’t stop there! Who knows what else you’ll find… But hopefully you’ll never see these witty error pages in action: 400 (Client Error), 500 (Server Error).

Finding Your Community: The Map Workflow

While we were really excited to launch all these new features,we knew that the current map workflow wasn’t quite right and needed to be changed. In order to solicit user feedback on the other new features, we decided to launch in iterative stages.

In order to find a community in the current map workflow we perform two geocode requests. First, we geocode based on what the user types. Then, if there’s a valid result, we find the latitude and longitude given back by the first result and reverse geocode it. With this method, we only get cities or towns as the result of searching for neighborhoods or smaller communities and can thus limit the fragmenting of communities by creating a canonical, publicly-owned almanac for each location.

Now almanacs are now publicly owned, rather than moderated by a particular user. On the old Community Almanac site the user that starts an almanac would personally own that almanac. If, for example, the user “Hatfield” started a Tug Fork River almanac he could moderate out the user “McCoy”. Then “McCoy” could in turn start another warring Tug Fork River almanac. While this new map workflow is great for creating canonical almanacs, it does expose another problem…

Finding Your Community: Neighborhoods

Sometimes prominent communities are geographically contained within the cities/towns that a map search would return. For instance, a community such as Third Ward, Texas—a neighborhood within Houston, Texas—will geocode simply as “Houston.”

So, in our first iteration, we’ve included a help link under the map for users having trouble finding their community. As a temporary solution, if a user cannot find their community on the map, they can contact us and have their almanac created for them manually. This method isn’t really optimal. What’s needed is a search method that will return canonical results within varied levels of specificity. In our next iteration, we plan to make the map workflow more robust and limit manual intervention.

Finding Your Community: Polishing the User Experience

We had to step back and take a look at our options, and among the miscellaneous fields returned by Google are four gems:

1. Locality (city/town)
2. State
3. AddressLine
4. Accuracy

We’re currently only using the first two, Locality and State but what is potentially useful is AddressLine combined with Accuracy. An AddressLine is what google recognizes as a feature of a specific point. This can be very specific (like “Columbus Circle”), a little more general (like “Central Park”), on up to the city, state, or country that the point is contained in. Accuracy is an approximation of the size of a feature, and helps us clue in to what neighborhoods the point is referring to. It’s not perfect, but it gives us some choices we can present to the user to let them decide.

To use a search for “third ward houston texas” as an example, if we discard any AddressLine results with an Accuracy greater than 5 or less than 3 (to pick limits based on this one search), we can offer up the following choices to the user:

• Greater Third Ward (with an accuracy of 4)
• South Central (with an accuracy of 4)
• Houston (with an accuracy of 4)
• Greater Houston (with an accuracy of 3)

Almanac moderation may still be needed since AddressLine is far less predictable than current city/town results. But if we check them against existing almanac titles, we can hopefully remove any manual intervention for all users, offering neighborhood-level specificity to almanac search and creation.

So, Where’s Your Community?

Dig in. Find your community on the map and start adding to its almanac. Share your heart & soul! We think this redesign turned out great and we’d love to get your feedback. Let us know what you think.

Access to transportation schedules has been a hot topic lately. Most recently, iPhone developer Chris Schoenfeld has  come in conflict with the MTA over schedule data. Chris wrote an app that uses schedules for Metro-North, and the MTA wants him to pay royalties for his use of the data from both past and future sales of his application. Chris has refused, noting that data under US law is not copyrightable and thus he is legally free to use and distribute it as he sees fit.

A couple of us at TOPP Labs have also run into issues trying to get MTA schedule data. We’re always experimenting with transit-related applications (from bus trackers to trip planners) and public data is integral to many of them. This past March, we requested MTA bus schedule data via a FOIL request. Within a month, the MTA had responded by sending us a CD containing bus route and schedule information for all the New York City Transit bus lines.  This data was in an undocumented format, and we set out reverse engineering it with the goal of writing a parser to generate GTFS data (we’ve got the parser working and released it as free software and you can download the GTFS).

Not long into the process, sections of Broadway were closed down for vehicle traffic. As a result, several bus routes were changed, making some of the data the MTA had sent us obsolete.

We wrote to the MTA to try to figure out how best to keep our data up to date. They told us that we must file a FOIL request every month or two and that there was no way to know when the schedules would be updated. But having one or two month old data isn’t of much value to us; we tried sending FOIL requests more frequently but quickly found that this angered the MTA, since their process for fielding a FOIL request is somewhat laborious. Since we’re not interested in making the MTA’s life harder, we stopped and started pursuing other avenues for getting up to date data.

Over the past months we’ve gone back and forth with the MTA several times, trying to find a way to get up to date data. So far, we haven’t found anything mutually satisfactory. But we’re still hopeful that we can find a good solution.

Everyone involved wants the same things. We all want better public transit. We all benefit when reliable data is easy to come by. The MTA benefits because independent developers write applications that make the infrastructure the MTA maintains more valuable for its riders. And this means the MTA doesn’t need to pay to develop apps for the iPhone, Android, Pre, Blackberry, and whatever comes next. Riders benefit because they can access schedules in the way that’s easiest and most convenient for them.

New York has a vast transportation network with complicated schedule data, and it’s inevitable for some errors to slip by. We’ve noticed a few oversights in the data we’ve received (e.g, the S55 shape data is out of date, the schedule data doesn’t distinguish between the M14A and M14D) and would love to help fix them. The net is full of examples of crowdsourcing for correcting errors (“given enough eyeballs, all bugs are shallow”), but there’s not currently a process in place for citizens to submit corrections.

The MTA is understandably concerned about inaccurate or out of date data giving them a bad name. They don’t want riders getting bad data and then blaming them for it, and neither do we. And the MTA doesn’t want to spend all their time responding to FOIL requests. It’s in nobody’s interest to make the MTA’s already tough job harder; as taxpayers, we want to help the MTA be as efficient as possible. As New Yorkers, we want the city to stay on the cutting edge of public transportation.

We think these problems are solvable. That’s why we propose a meeting of the minds. We think progress is made when people come together, honestly discuss their goals, and work cooperatively to reach mutually beneficial solutions.

And that’s why we’re hosting a New York Transporation Data Summit. With beer. While the event will be open to the public, we’ve specifically reached out to MTA employees, open government advocates, application developers, and transit enthusiasts.

Here’s the scoop:

WHERE: 148 Lafayette St, NY, New York, 12th floor (map)
WHEN: Tuesday, August 25 at 6pm
WHAT: Meetup to discuss how the MTA and the developer community can best collaborate.

Please RSVP here.

Please come join us for pizza, beer, and a friendly discussion. There’s also a stunning view of the city we all love.

(This post was written by David Turner and Nicholas Bergson-Shilcock)