The Dirty World of Ranking Cities

BY SAM SALKIN

Cities differ vastly from one another.

That’s probably too obvious of a statement.

During the summer of 2015, I worked for a non-profit consulting firm advising cities around the world on how to solve the nasty issues plaguing their citizen’s quality of life. One of our clients, a major European city, came to us with a seemingly simple question. The city believed it ranked among the world’s best cities at managing municipal solid waste—known to most of us as garbage or trash. The city not only wanted to see how they stacked up to other cities but also how to improve and tell people about their success.

Thus began a summer of investigating and evaluating how cities around the world manage their waste. I spent hours scouring websites of cities seeking all kinds of data. My hunt led me to open data portals, annual reports, and vendor audits to determine metrics such as the amount of waste a city generated and how they treated it. Google Translate simultaneously became by best friend and most lamented foe as I tried to decipher words such as incineration, plastics, and diversion in languages like French, Norwegian, and Japanese.

In the end, I couldn’t tell our client they were the best.

I also couldn’t tell them who was.

Cities vary in dizzying ways. Density, depth of transit networks, gross metropolitan product, and countless other measures vary widely. In trying to understand who was the “best in trash,” I became convinced of the futility of the exercise.

As time progressed, I became even more skeptical of the usefulness of ranking cities. It’s not just limited to solid waste, either. Cities obsess about who’s above them and who’s nipping at their heels in many different performance rankings. Instead, cities should focus on improving and benchmarking against themselves. The only place a city should compare itself to is . . . itself.

Perhaps our client had asked the wrong question from the start.

However, until I could persuade my client, I had to answer their original question: How did they stack up to other cities in solid waste performance?

Who Takes What from Whom

Take New York City. Its Department of Sanitation provides trash collection services for over eight million residents and all its public buildings three times a week using city vehicles. Recycling is collected once a week, but paper and cardboard must be separated from other recyclables.[i] Municipal taxes cover the costs. However, all private businesses must hire a company to haul away their trash because the city does not collect it[ii].

Compare this to Seattle. A city-owned public utility is not only responsible for garbage but also water, sewage, and drainage.[iii] For waste collection, households choose a specific size trash bin and pay a monthly bill depending on the bin’s size. Recycling is free and there is a small fee to recycle organic waste like food or lawn clippings. City vehicles collect trash, recycling, and organic waste only once a week. Businesses and commercial property must purchase service from the city at set rates. Then, Seattle contracts with two private companies to collect waste for the city.

These differences are not unique to the United States. Berlin owns a private company to collect household waste.[iv] Commercial customers can contract with this company, or they can seek better prices from competitors. Each of London’s thirty-three boroughs has its own waste collection regime. Toronto will collect trash from all households, but only businesses under a certain size.[v] Singapore’s government does not collect any waste. Instead, private contractors handle all of it.

Beyond who collects what and when, how residents prepare their waste for collection also varies widely.

North American cities almost exclusively enjoy door-to-door collection where trucks roam the streets, picking up waste from our front or back doors. Conversely, in rural areas, some municipalities only operate a transfer station and citizens bring trash to the dump themselves.

In Europe, door-to-door collection is far from the norm, though some cities offer it. Many cities use shared bins located around neighborhoods for residents to deposit trash. Rome is currently transitioning from a shared-collection system to door-to-door. Brussels collects all trash and recycling door-to-door—except for glass.[vi] Zurich only collects paper from individual customers and everything else goes in neighborhood bins.[vii]

If a theme of inconsistency is emerging, it only goes deeper.

Think about how you bring your trash out for collection. Do you separate recycling from other waste? To what extent do you separate different kinds of recyclables? Are plastics in a separate bin from metals or paper? Does your city accept organics or compost such as yard or food waste? Houston collects all trash in a single bin that is later sorted for treatment or recycling and resale.[viii] Compare that with San Francisco where customers have three bins—one for trash, one for recycling, and one for organics.[ix] The other extreme is a city like Helsinki with individual bins for each kind of waste and recyclable where papers, plastics, metals, and glass all have their own containers.

Definitions of important terms also vary between cities. In the waste management realm, “diversion rate” is a popular term to describe waste not going to a landfill. New York City’s Department of Sanitation defines diverted waste as trash “sent to reuse, recycling, or composting.”[x] Across the country, Los Angeles works from a different definition that counts diverted waste as “recycling, composting, waste-to-energy (the technological successor to incineration), transformation, and source reduction.”[xi] To clarify, municipalities in California can count a reduction in waste generated by their citizens as “diversion.” Therefore, if a city decreases the amount of waste generated, their diversion rate goes up despite the waste never actually existing.

To be fair, California has abandoned “diversion rate” as a key statistic in favor of the “total amount disposed.” Still, this made me wonder how you can possibly compare two cities’ statistics when there is disagreement over what simple terms mean.

All of these complications occur before a city even decides what to do with waste once its collected. Some cities send trash to a landfill while others sell to a recycling company or treat it and burn it for energy or heat.

Spreading the Word

Another challenge arises when looking for good relevant information about each city’s waste management system. As with everything else, it is an exercise in inconsistency. Some cities provide regularly updated and detailed reports breaking down waste collected from specific areas on a monthly basis. Other cities tie information into an open data portal that publishes the information in a variety of formats for deeper analysis.

Unfortunately, this isn’t the norm. Aside from translation and website navigation issues, finding reliable and comprehensive information creates challenges. Some cities have not updated their website in years. Others provide copious information on how much waste they collect but provide no information about how they treat that waste. Many cities offer strategic plans on how to reduce their waste output or achieve higher rates of composting, but provide no information on their current activities.

My other big concern stemmed from identifying data sources. Since a city’s “customers” vary widely, it is often unclear whether a city’s data represents waste generated by all its citizens and businesses or only the residents from whom the agency collects. In a city like Singapore, where private companies handle all collection, the city conducts rigorous audits to present a comprehensive picture of the city’s waste profile.

As someone trying to answer a simple question, all of these differences became exhausting.

A simple apples-to-apples comparison was completely out of reach. Even when I tried to normalize data by converting it to a per capita measure, it did not seem fair because of underlying differences between how cities operate, define, and report on trash.

Defining “Best”

Halfway through the summer, after poring through the data and reports, I developed a new take on the question at hand. On some level, a city’s waste performance has far more to do with citizens than the agency responsible for collecting and processing waste.

It occurred to me the amount of waste a city processes is a function of what its population consumes and generates. The amount a city recycles certainly stems from policy decisions, but ultimately reflects citizen compliance with those rules. Culture dictates how much waste is generated both in aggregate and on a per capita basis. If a city has a particularly high recycling rate and low per capita waste generation, that is obviously something to celebrate, but is largely a consequence of citizens’ actions, not a city’s efficacy at waste management.

This challenge mirrors the metrics used to judge cities. It doesn’t matter if it is museum attendance or traffic safety – both are a consequence of policy and culture. They can’t be completely attributed to the actions of a city government.

Aside from how much waste a city generates, there are innumerable ways to measure a solid waste agency’s operational performance. Customer service and satisfaction are easily measured through metrics such as missed pickups per thousand or how long it takes to resolve customer complaints. I am less knowledgeable on industrial methods, but I am certain a city’s efficiency at processing waste and money received for recyclables is measurable. Unlike other metrics we’ve examined, these are helpful for comparing cities.

All this information must be gathered somewhere, but throughout my research, I saw very little reported. Perhaps cities like to keep this data confidential, or few people care about the information— aside from the trash-obsessed. However, considering how inconsistently cities report on their most basic activities, this lack of information was no surprise.

Best for Whom? Best How?

Throughout this discussion, one question loomed large. I started to wonder who cities are trying to impress. I questioned whether citizens care that their city is the “best at trash.” While it might be a source of pride for the wonks and urbanists among us, I’d imagine most people just care about trash being picked up when its supposed to, having clean streets, and not bearing the environmental hazards of their trash.

In asking legitimate questions about how much citizens really care, the importance of comprehensive reporting became undeniable because it signals a city’s commitment to delivering quality services and continuously improving.

The idea of “comprehensive” reporting might seem nebulous, but after seeing how cities around the world vary, the recipe is simple. First, transparency is key. With definitions of seemingly straightforward terms varying widely, it’s important to explain exactly what “diversion” means in your city. Beyond definitions, cities should report as much data as they can—the good and the bad. The second key ingredient is frequency. It isn’t enough to post statistics to a website once every few years and claim they are “reporting.” Rather, cities should provide updates on a predictable schedule. They could provide deeper temporal details by going beyond annual stats and giving monthly or quarterly breakdowns. Finally, the last piece is accessibility. It isn’t enough to bury an annual report on an obscure web page that few will find. Cities should be up front and meet citizens where they get information. In many places, the web makes the most sense, but in areas without stable Internet access, waste agencies could deliver report cards to customers or disseminate information via the press. Beyond being comprehensive, cities must make reporting relevant—the right information going to the right people through the right channels.

If you’re a city like my client, and your goal is demonstrating superiority in managing waste, provide a frequent comprehensive account of your activities to the people who pay attention. Perhaps your target audience is an industry group like the International Solid Waste Association or C40, a network of cities banding together to tackle climate change. Regularly pushing out detailed information makes cities relevant and demonstrates leadership.

However, some cities should simply prove the strength of services to its citizens. This is where transparency is paramount—even if the news isn’t good. Provide updates on customer service metrics. Show levels of street cleanliness. Make sure local elected officials understand the quality of service in the areas they represent. When agencies consistently provide updates and put their cards on the table, their incentives to improve become inherent.

For cities looking to encourage behavior change and reduce waste, reporting should be citizen-centric and focused on helping residents understand their role in waste management. Present relevant data in non-technical language. If a city wants to increase composting, help explain composting to citizens, its benefits, and show how their neighborhood compares to others.

It’s Between You and You

With all the pitfalls of comparing cities to each other, the better exercise is for cities to compare themselves to themselves. The underlying differences between Berlin and London and Toronto make it practically impossible to conduct meaningful analysis. An econometrician might disagree but no matter what formula and weights one comes up with, the nuance is so deep that the rankings or scores lose meaning. After this summer, I suppose I am slightly impressed that Los Angeles recycles more than Rome. Maybe. But they are vastly different cities with their own waste management regimes, too dissimilar for an objective comparison.

It would be impressive to see if a city’s recycling rate has increased over the past five years and an explanation of how that was accomplished. The vast stakeholders ranging from ordinary citizens to elected officials and journalists, would probably see far more value in a positive trend than rankings of dissimilar cities. If a city is trending downwards or middling, they now have accountability to the public. It might be awkward but it puts pressure on to turn things around.

This is the challenge and opportunity in comprehensive and relevant reporting. Cities need to define their goals and implement an effective public communications strategy.

Beyond Waste

This article is about trash and my odyssey helping a city understand how it stacks up. But really, it’s not about garbage.

Easily, I could have written the same 2,500 words about arts funding or public health. Our culture obsesses with rankings. These days, everywhere you turn there’s a new list of the “ten healthiest cities” or “the most livable neighborhoods.” While these are helpful heuristics, it is unclear how much they actually improve the quality of life for citizens. As much as we try to control for certain variables or normalize values, I’m not convinced knowing how much better City A is than City B helps anyone aside from people looking for bragging rights. Knowing how much better City A is than it was six months ago provides true public value.

When a city commits to honest appraisals of its performance and includes the public through comprehensive reporting, everyone wins. City employees see their work is relevant and citizens understand how their actions fit into the bigger picture. Think about this outside of the context of garbage and waste. Think about education or traffic congestion rankings. Striving for similarity is futile.

There is one city every city is similar to—itself. Why waste time chasing the pack when cities should commit to improving itself for its citizens? Instead of being number one, why not just be . . . better? Improve consistently. Tell your citizens and the world about that.

It took me all summer to realize it, but our obsession with ranking cities might just be garbage.

Sam Salkin is a 2016 graduate of the Master of Public Policy Program at the Harvard Kennedy School. Sam worked as a sustainable transportation consultant, advising local governments and developers on strategies to promote multi-modal travel options that reduce congestion. He also worked as a communications consultant and project manager for public education efforts in the US Department of Veterans Affairs’ initiative to end homelessness among veterans. Post-HKS, Sam is working as an analyst with the New York City Subway focusing on projects to recover from Superstorm Sandy and protect the system from future climate-related impacts.

Image Sources: Isado, Flickr with Creative Commons License.