Staff: Committee on Technology

Jeffrey Baker, Counsel

Colleen Pagter, Policy Analyst
John Russell, Financial Analyst

THE COUNCIL

Report of the Infrastructure Division

Robert Newman, Legislative Director

COMMITTEE ON TECHNOLOGY
Hon. Daniel R. Garodnick, Chair

June 21, 2010

Int. No. 29: By Council Members Brewer, Fidler, Gentile,

Gonzalez, James, Lander, Palma, Nelson, Lappin, Dromm and Garodnick

Title: A Local Law to amend the administrative code of

the city of New York, in relation to creating open data standards.

Administrative Code: Amends Title 23 of the Administrative Code of the

City of New York by adding a new chapter 3.
1. INTRODUCTION

On Monday, June 21, the Committee on Technology, chaired by Council Member

Daniel R. Garodnick will hold a hearing on Int. No. 29. Int. No. 29 creates an open data

policy for the City of New York.

2. OPEN GOVERNMENT DATA

“Open government data” is the concept that data generated by public sector

entities that is legally accessible should be available to the public electronically via the

Internet in “open raw formats.”1 “Open,” refers to the use of non-proprietary software or

systems used to encode the data. For example, XML (extensible markup language) is a

non-proprietary language used to aid information systems in sharing structured data,

especially via the Internet.2 XML is available to anyone, and information that is stored in

an XML format is may be accessed by anyone. “Raw,” refers to data that has not been

processed. For example, the demographic data collected by the United States Census

Bureau would be raw, but the reports generated by the United States Census Bureau

analyzing such data would be, by comparison “cooked.”

Adopting open government data standards would promote inclusion of more

individuals into the governmental process by making access to information easy and

affordable; it would promote transparency and accountability by providing data in its raw

form, enabling members of the public to perform their own data analysis and draw their

1
Improving Access to Government through Better Use of the Web, W3C, 10 March 2009.
2
Extensible Markup Language (XML) 1.0 (Fourth Edition), W3C, 16 August 2006, edited 29 September
2006.

2
own conclusions; and it permits deeper and more varied analysis of government data by

enabling two or more data sets to be integrated together (commonly known as a

“mashup”), often by aggregating data and displaying the information visually on a map.

For example, housingmaps.com is a mashup that integrates GoogleMaps with housing

listings obtained from Craigslist to visually represent the available housing in a city

within a user selected price range.

The ingenuity of the public to create data mashups is limited only by the

availability of useful data. In November, 2008, the District of Columbia Chief

Technology Officer (CTO) announced the Applications for Democracy program whereby

private citizens compete for $50,000 in total prize money by creating mashups that utilize

any of the 200 data feeds available through the office of the CTO website.3 Within thirty

days, the Office of the CTO received 47 completed applications valued at $2,300,000.4

Applications included a carpool matchmaker, and a real time alert notification that would

send alerts (crime reports, building permits, etc.) based on a user’s location if they have a

GPS-enabled device.

On June 29, 2009, the Economic Development Corporation and the Department

of Information Technology and Telecommunication announced the NYC BigApps

Competition, modeled on the Applications for Democracy program in Washington, DC.5

To use in the competition, the City created NYC Data Mine, an online repository of over

80 public data sets from 32 agencies. The competition received 85 eligible applications

3
47 Applications in 30 Days for $50K, Government Technology, November 13, 2008.
4
http://www.appsfordemocracy.org/
5
Mayor Bloomberg Announces Five Technology Initiatives to Accessibility, Transparency, and
Accountability across City Government. Press Release June 29, 2009.

3
by December 15, 2009, and on February 4, 2010, $20,000 in prizes were awarded to

thirteen different categories. The Best Overall Application Grand Prize was awarded for

the application WayFinder NYC, which helps users locate the nearest subway and PATH

stations. The Investor’s Choice and Popular Choice Grand Prizes were awarded for the

application NYC Way, a bundle of over thirty features that allow users to locate nearby

restaurants, bars, events, restrooms, wifi hotspots and more.

On December 8, 2007, a group of 30 open government advocates developed eight

principles that define open government data.6 While these principles are not binding,

they are instructive. They are as follows:

1. Complete – All public data is made available. Public data is data that is not
subject to valid privacy, security or privilege limitations.

2. Primary – Data is as collected at the source, with the highest possible level of
granularity, not in aggregate or modified forms.

3. Timely – Data is made available as quickly as necessary to preserve the value

of the data.

4. Accessible – Data is available to the widest range of users for the widest range
of purposes.

5. Machine processable – Data is reasonably structured to allow automated

processing.

6. Non-discriminatory – Data is available to anyone, with no requirement of

registration.

7. Non-proprietary – Data is available in a format over which no entity has

exclusive control.

8. License-free – Data is not subject to any copyright, patent, trademark or trade

secret regulation. Reasonable privacy, security and privilege restrictions may
be allowed.

6
http://resource.org/8_principles.html

4
 The City of New York currently makes an enormous amount of public data

available on NYC.gov, from the Building Information System (BIS) operated by the

Department of Buildings, to the Automated City Register Information System (ACRIS)

operated by the Department of Finance, NYC Stat, operated by the Mayor’s Office of

Operations, which consolidates information form the Citywide Performance Report

(CPR), neighborhood statistics, and the American Recovery and Reinvestment Act of

2009 (ARRA), and NYC Data Mine, which is the repository for the data sets made

available for the NYC BigApps contest.7 Unfortunately, much of this data is available

only in proprietary and/or visual formats, and not all of it is raw data. Some of the more

extensive databases, such as the Building Information System (BIS), specifically forbid

automated machine processing. Combined, these factors limit the usefulness of the data

currently provided by the City.

3. INT. NO. 29

Section 1 of Int. No. 29 amends Title 23 of the Administrative Code of the City of

New York by adding a new Chapter 3, titled “Accessibility to Public Data Sets”.

Section 23-301 defines the terms “agency,” “consensus,” “data,” “data set,”

“department,” “determination,” “measurement,” “public data set,” “technical standard,”

“transaction,” “voluntary consensus standards,” “voluntary consensus standards bodies”

and “web application programming interface.”

Section 23-302 requires that all public data sets maintained by City agencies shall

be made available on the Internet through a single web portal, formatted to enable

7
http://www.nyc.gov/

5
viewing by web browsers and mobile devices and also in their raw or unprocessed form.

These public data sets shall be updated online as often as necessary to preserve the

integrity and usefulness of the data sets using web syndication technology, which is a

service that would notify the public of all updates for each public data set and for each

agency. These public data sets shall be made available without any registration

requirement, license requirement or restrictions on their use, and shall be presented and

structured in a format that permits automated processing. Furthermore the City shall

implement a web application programming interface which would allow third-party

programs to request and receive information from the City’s website in real time and pass

that information to the users of their application.

Section 23-303 requires the Department of Information Technology and

Telecommunications to promulgate rules establishing an Internet data set policy and a

technical standards manual for the publishing of public data sets on the Internet by City

agencies for the City of New York by July 5, 2010. The Department of Information

Technology and Telecommunications is also required to consult with voluntary

consensus standards bodies, such as the World Wide Web Consortium, and to participate

with such bodies when such participation is feasible and in the public interest.

Section 23-304 requires each agency, in consultation with the Department of

Information Technology and Telecommunications to review the public data sets under its

control and to develop and submit an agency compliance plan to the Mayor and the

Council no later than July 5, 2010. Such plan shall describe how such agency intends to

achieve full compliance with this chapter by December 2, 2013, and shall include an

accounting of all public data sets under the control of the agency. As a part of such plan,

6
agencies shall classify the public data sets under their control as “immediate”, “priority”

and “legacy”. “Immediate” includes any public data set that can be made available on the

Internet within thirty days of the agency acquiring or creating such data set. “Legacy”

includes any public data set that, due to its size or complexity, or due to technology

constraints, cannot be made available on the Internet by January 2, 2012. All other public

data sets shall be classified as “priority”.

Section 23-305 exempts public data sets from the requirements of Section 23-302

for a limited period of time, based on which classification they were assigned in the

agency compliance plan created pursuant to Section 23-304. “Immediate” public data

sets are exempt for thirty days after enactment of the law. “Priority” public data sets are

exempt until January 2, 2012. “Legacy” public data sets are exempt until December 2,

2013.

Section 2 of this bill states that the bill becomes effective immediately upon

enactment.

7
8
 Int. No. 29

By Council Members Brewer, Fidler, Gentile, Gonzalez, James, Lander, Palma, Nelson,
Lappin, Dromm and Garodnick

A LOCAL LAW

To amend the administrative code of the city of New York, in relation to creating open
data standards.

Be it enacted by the Council as follows:

Section 1. Title 23 of the administrative code of the city of New York is amended

by adding a new chapter 3 to read as follows:

CHAPTER 3

ACCESSIBILITY TO PUBLIC DATA SETS

§ 23-301 Definitions.
§ 23-302 Public data sets availability.
§ 23-303 Internet data set policy and technical standards.
§ 23-304 Agency compliance plan.
§ 23-305 Exemptions and timetable.

§23-301 Definitions. As used in this chapter: a. “Agency” means an office,

administration, department, division, bureau, board, commission, authority, corporation,

advisory committee or other governmental entity performing a governmental function of

the city of New York.

b. “Consensus” means general agreement, but not necessarily unanimity, and

includes a process for attempting to resolve objections by interested parties, as long as all

comments have been fairly considered.

c. “Data” means any factual information collected, produced or reproduced by,

with or for an agency that records any aspect of a measurement, a transaction, or a

determination.

9
 d. “Data set” means a comprehensive collection of interrelated data on any subject

that is maintained on a computer system owned or operated by an agency. Such term

shall not refer to data collected and stored on computing devices where such data has not

been added to any data set or used to create a new data set, provided that the data stored

on such computing devices is routinely added to a data set. Such term shall not refer to

copies of the data set, or portions thereof, that may be stored on computing devices.

e. “Department” means the department of information technology and

telecommunications or any successor agency.

f. “Determination” means any final decision made by an agency with respect to a

person, including, but not limited to: (1) eligibility for services or benefits; (2) issuing a

permit; (3) registration, certification and licensing; and (4) liability for civil and criminal

penalties.

g. “Measurement” means to quantify by means of comparison to a reference

standard any characteristic of a phenomenon or object.

h. “Public data set” means any data set that is maintained by an agency that must

be accessible for inspection by the public in accordance with any provision of law or that

an agency shall decide to make accessible, excluding any data to which an agency may

deny access pursuant to section eighty-seven of the public officers law.

j. “Technical standard” means (1) the common and repeated use of rules,

conditions, guidelines or characteristics for products or related processes and production

methods, and related management systems practices; and (2) the definition of terms;

classification of components, delineation of procedures; specifications of dimensions,

materials, performance, designs or operations; measurement of quality and quantity in

10
describing materials, processes, products, systems, services or practices; test methods and

sampling procedures; or descriptions of fit and measurements of size or strength.

j. “Transaction” means any interaction between an agency and any person,

including employees.

k. “Voluntary consensus standards” means standards developed or adopted by

voluntary consensus standards bodies, both domestic and international. These standards

include provisions requiring that owners of relevant intellectual property agree to make

such intellectual property available on a non-discriminatory, royalty-free or reasonable

royalty basis to all interested parties.

l. “Voluntary consensus standards bodies” means domestic or international

organizations which plan, develop, establish, or coordinate voluntary consensus standards

using agreed-upon procedures.

m. “Web application programming interface” means an interface through which

an application program may request services or information from a web page or library

through the Internet.

§23-302 Public data set availability. a. All public data sets maintained by city

agencies shall be made available for inspection by the public on the Internet through a

single web portal that is linked to nyc.gov or any successor website maintained by or on

behalf of the city of New York.

b. All public data sets shall be formatted to enable viewing by web browsers and,

where practicable, mobile devices. All public data sets shall also be made available in

their raw or unprocessed form.

c. All public data sets shall be updated as often as necessary to preserve the

11
integrity and usefulness of the data sets. The web portal hosting the public data sets shall

make use of web syndication technology to notify the public of all updates for each

public data set and for each agency.

d. All public data sets shall be presented and structured in a format that permits

automated processing.

e. All public data sets shall be made available without any registration

requirement, license requirement or restrictions on their use.

f. All public data sets shall be accessible to external search capabilities.

g. The department shall create or adopt and utilize a web application

programming interface that shall permit application programs to request and receive

public data sets directly from the web portal.

§23-303 Internet data set policy and technical standards. a. The department shall

promulgate rules establishing an Internet data set policy for the city of New York no later

than July 5, 2010, for the purposes of ensuring agency compliance with the provisions

and open data principles set forth in section 23-302 of this chapter. Such policy shall

include, but not be limited to, guidelines to assist agencies in classifying their public data

sets either as immediate, priority and legacy pursuant to section 23-304 of this chapter.

b. No later than July 5, 2010, the department shall prepare and publish a technical

standards manual for the publishing of public data sets on the Internet by city agencies

for the purpose of making public data available to the greatest number of users and for

the greatest number of applications and shall, whenever practicable, use non-proprietary

technical standards for web publishing and e-government that have been developed or

adopted by voluntary consensus standards bodies. Such manual shall identify the reasons

12
why each technical standard was selected and for which types of data it is applicable and

may recommend or require that data be published in more than one technical standard.

Such manual shall be updated by the department as necessary.

c. The department shall consult with voluntary consensus standards bodies and

shall, when such participation is feasible, in the public interest and is compatible with

agency and departmental missions, authorities and priorities, participate with such bodies

in the development of technical standards.

§23-304 Agency compliance plan. a. Each agency, in consultation with the

department, shall review all public data sets under its control and shall classify them as

immediate, priority and legacy, in accordance with subdivision c of this section and

pursuant to guidelines promulgated by the department under section 23-303 of this

chapter.

b. Each agency, in consultation with the department, shall submit an agency

compliance plan to the mayor and the council no later than July 5, 2010. Such plan shall

describe how such agency intends to achieve full compliance with this chapter by

December 2, 2013, and shall include an accounting of all public data sets under the

control of the agency, how such data sets have been classified pursuant to subdivision a

of this section, and the rationale for each classification. No later than the first Monday in

January of each year beginning in 2011, each agency shall submit an annual update to its

compliance plan to the mayor and the council until such agency reports full compliance

with this chapter.

c. Public data sets shall be classified as follows:

1. Immediate. Any public data set in possession of an agency as of the effective

13
date of the local law that added this chapter and any public data set such agency acquires

or creates after such effective date through July 5, 2010, that can be made available on

the Internet pursuant to section 23-302 of this chapter within thirty days of the agency

acquiring or creating such public data set shall be classified as immediate. Agencies

should make best efforts to classify as many public data sets as immediate as is possible.

Any public data set acquired or created after July 5, 2010, shall be classified as

immediate.

2. Legacy. Any public data set that, due to its size or complexity, or due to

technology constraints, cannot be made available on the Internet pursuant to section 23-

302 of this chapter by January 2, 2012, shall be classified as legacy. Agencies shall make

best efforts to avoid placing public data sets into the legacy classification.

3. Priority. Any public data set that does not meet the conditions of immediate or

legacy shall be classified as priority.

§23-305 Exemptions and timetable. a. Public data sets classified as immediate

pursuant to section 23-304 of this chapter shall be exempt from the provisions of section

23-302 of this chapter and from applicable rules promulgated by the department pursuant

to such section until January 3, 2011.

b. Public data sets classified as priority pursuant to section 23-304 of this chapter

shall be exempt from the provisions of section 23-302 of this chapter and from applicable

rules promulgated by the department pursuant to such section until January 2, 2012.

c. Public data sets classified as legacy pursuant to section 23-304 of this chapter

shall be exempt from the provisions of section 23-302 of this chapter and from applicable

rules promulgated by the department pursuant to such section until December 2, 2013.

14
§2. This local law shall take effect immediately.

15