What’s the scope?
This part of the miniseries is actually the heart of the whole series. It describes why and how our approach differs AFAIK from all other approaches of geocoding news, including the approaches taken recently by Google News with it’s local news extension ( see here and here) and Yahoo with it’s Newsglobe.It is an approach that augments these approaches by not only assigning the places in the news but in addition also assigning places of the news, especially one kind of a place of a news story, we call scope of a news story that describes the geographic area(s) of relevance of a news story.Scope are in our opinion at least as important as the places in a news story. As the examples below show determining the geographic scope of a story is a task that reporters and editors have done for ages.Since determining the geographic scope of a news story is much more difficult than recognizing places mentioned in a story it is also much less likely to be suspect to automatic identification via named entity recognition and the likes. (BTW. Holovaty calls everyblocks variant of named entity recognition “geoparsing” in his recent OJR interview, Google calls their variant “but instead we analyze every word in every story to understand what location the news is about and where the source is located.”) .Hence determining the scopes of news stories is something where the news provider at the source has a real advantage / USP.
In the remainder of this post i’m trying to convince you that distinguishing between places in stories and places of stories makes sense and that adding metadata about the scopes of news stories is useful and important for news providers. If you disagree or have comments i’m eager to hear these either as comments or via mail to relations at ka2 dot de.
Earlier Parts of this miniseries: Part I: Adding geographic metadata to news at the source, Part II: Great news from Adrian Holovaty, Part III: Early Experiments, …)
Use-cases for geocoded news
In order to motivate this argument it makes very much sense to step back for a moment and ask: “What are the various use-cases for geocoded news that should be supported?”
Use case 1: Putting the places in news stories on a map
Ever since the advent of Google Maps the one use case that typically is cited in conjunction with geocoding news is putting the news on map, i.e. for every news story a pin or a number of pins is virtually pushed into a map. Every pin in the map denotes a location IN the news story. This is the use case i mostly talked about until now and that everybody knows from all the various approaches to geocoding the news.
There is a variant of this use case where not an overview map of a number of stories is shown, but a map only detailing the locations in a single news story. This use case is very common in TV and broadcast news, but rarely seen online.
Use case 2: Defining the geographic area of relevance
But actually there is one other use case that in our opinion is as least as important as the first one. This use case is about defining the geographic area(s) of relevance of a news story. IMHO this is a use case that editors and reporters are already used to since ages. This second use case is important even in cases where putting news stories on a map is not the focus.
And it is a use case that provider of purely technical solutions for geocoding news stories cannot easily automize.
What do i mean with “This is a use case that editors are used to since ages?” Well, ever since the inception of the newsroom editors have thought of news stories in terms of:
- Is this story newsworthy nationwide or only on a regional or maybe just a local level?
- Do i print this story n the first section of the newspaper or do i print it in the local section?
- Do i dispatch this story into the national newswire or only a regional newswire?
- …
By attaching metadata describing the geographical area(s) of relevance to the news stories, our approach takes this kind of thinking to the logical next level (and into the digital age).
In order to describe why this use case is important evene without mapping applications i want to give you an example from our own operations:
Right now dpa-infocom is delivering 12 regional news wires. Their coverage corresponds to the 16 german states (“Bundesländer”) where 4 news wires cover the area of two german states. A newspaper customer that wants to add local news from our regional news wires to his web site had until recently only the possibility to add these news items on a wire level.
This means that a newspaper located in a city on the border of two states. had to add two news wires. But within theses news wires there were not only stories deemed covering this city but also cities in the opposite corner of the respective states, cities sometime a couple of hundred of kilometers away. News stories definitely not suitable to the local pages of his website.
With the addition of the geographic area(s) of relevance to the news, our customers now can identify the news stories that are deemed relevant on a national, state-wide, county-wide or city/town/locality-wide level (for select bigger cities even on a borough or district-level). This enables the customer to filter the wires accordingly, not only ignoring stories attached to counties that are irrelevant to him but also sorting the relevant stories e.g. into it’s county-wide local editions and websites.
It also allows us to realize specializes wires defined purely by a filter on the metadata of the news items, aggregating only the news stories a certain customer is interested in.
What is a scope of a news item?
This question is best illustrated with the series of screenshots below. They show the hierachical administrative partition of germany.
Germany is on the first level divided into 16 states (“Länder”). These are themselves divided into a total of forty-something second-level units called “Regierungsbezirke” and a total of 439 third-level regions called “Kreise” (roughly equivalent to counties). On the fourth level there are some 12000 localities (cities, towns, etc.). depending on the size of the city, there might be one or two administrative levels within the city (boroughs and districts).
The scope of a news story in our regional news wires is a set of administrative regions form this selection. I’m going to explain why we’ve chosen the administrative regions as the basis for defining the scope of a news item in the next post when i’m going to formalize our approach
Hence adding geographic areas of relevance (either districts, boroughs, cities, counties or states) is IMHO an important way of “designating granular locations” of news stories.
Examples
Unfortunately our regional news wires are in german, hence including concrete examples in this post wouldn’t help too much. Hence i tried to abstract from the concrete examples to some typical news stories regularily found on our wires. I’m sure you come up with examples of your own in no time.
I also try to illustrate why i think that automisation of designating scopes of stories is difficult by giving alternative scopes, depending on sowm twists within the story.
- A story about new legislation in a certain state will be assigned a state-wide area of relevance, although the dateline typically show the states capitol.
- A story about a car accident in city X with the driver of the car coming from the town Y, the news story will most likely be assigned two city/town-wide areas of relevance X, and Y. If it is a serious car accident with multiple dead this might be changed to a single county- or state-wide area of relevance.
- A story about a soccer game where teams from district A and district B are playing against each other will most likely be assigned two geographic areas of relevance district A and B. If the game is the final of city-wide cup, this most likely will be changed to the whole city. If the two teams happen to play in a professional sports league this news story most-likely will get a nation/country-wide area of relevance, unless they are playing a friendly or benefit game against each other.
- A story about a series of burglaries in districts A,B,and C in town X and districts D and E in town Y, that is on trial at a court in city Z, is most likely to be assigned the geographic areas of of X,Y (maybe also Z), unless the burglaries itself were exceptional, e.g burglaries of famous artworks. This most likely results in a state-wide or nation-wide geographic area of relevance.
What next?
In the next post of this miniseries i’m going to present a formalization of what i’ve described informally in this post. I’m also going to present who we are representing the meta-data within our wire feeds, and sowm ideas about representing them within “standard” feedformats like ATOM and RSS2.0.
In another blog post i’m then going to cover our approach to the first use case mentioned above: Designating places in news stories.
I’m probably then going to write about the problems with getting access to the geometries of the administrative region and preparing the data. This most likely will end up in a rant about government hindering innovation by not making public information public. I might also include some ideas how to circumnavigate these issues.