Thoughts on Analyzing and Visualizing Text Messages

Nairobi was shut down during the peak of the pandemic in 2020. The city was closed off from the outside world and a curfew was imposed, limiting movement within. Everyone had experienced this lockdown bogged down in their own estate or community. 

During this time, we deployed an application called Mtaa, a mobile application that enables users to send geolocated text messages with an “emotion” attached to them. We deployed the app to keep in touch with our friends and team members and to collect data on people’s experiences of the pandemic and the measures imposed.

Our friends sent more than 700 messages in a period of only a few weeks. The messages ranged from day-to-day interactions, reports of extreme kindness and sacrifice, to small acts of daily life like painting graffiti and making music videos, to talking about crime and lack of essential services, such as access to water and food. The app gave us a sense of purpose during trying times, but most importantly,  it kept us close and connected through the sharing of experiences. The exercise also gave us an opportunity to see whether we can make these narratives into actionable data. 

Analyzing more than 700 messages posted in several languages represented a unique challenge. There are a few standard ways to analyze such data, either geospatially or statistically, each with its own strengths and shortcomings. I’ll touch on a few in this blog post.   

I’ll start with the most obvious and easiest one: “putting points on the map”. Each message is geolocated, so the most straightforward way is to display them on a map. Points on the map are visually appealing but not particularly useful, especially when points start to cluster in the same area. Once the number of data increases, the map itself becomes crowded and hard to read.

Text messages sent from Mlango Kubwa, Nairobi

Second, our messaging platform allows people to express their ‘feelings’ about a situation in form of emotions. Users can tag each message based on whether they think it has a more positive or negative connotation associated with it. These so-called ‘feelings’ can be displayed as an aggregate in an area and represented as such. We used hexagons to represent an ‘average emotion’ associated with an area (see the image below). In this way, we can identify areas that have more positive or negative opinions associated with them. However, by displaying opinions as a ‘vibe’ we lose the granularity or detail that makes these opinions so interesting in the first place.  

Hexagons represent an ‘average emotion’ associated with an area

Third, we can tag each message based on the topic it depicts. For example, if a message talks about water, we tag it as ‘water,’ if it talks about crime, we tag it as ‘crime,’ etc. To do this we have to read through all of the messages, create categories, and manually place each message into a relevant category. Naturally, this is time-consuming but it allows us to look at the statistical and spatial distribution of topics. For example, we can see how many messages there are about the lack of food, security, development, access to services, or even people having a good time, and where they are located. This process could be sped up using language processing techniques but they would have to work on several languages at once. Tagging messages in this way does have its merit in that it allows for easier analysis, however, as in the previous point, we lose the detail and importance of each message as a unique story.

Chart representing topics of text messages

Further, we can experiment with turning people’s experiences into more traditional, social-media-driven narratives by merging visuals with messages. This is a very powerful storytelling technique fit for modern-day social media consumption (see image below), but apart from being visually striking, there’s not a lot there. Plus, it adds human interference through selecting and preparing each ‘experience,’ which interferes with the free-flowing communication about a place that people occupy.

Instagram post of one of the messages

Finally, we can display messages on a timeline. This looks cool but is only useful if we’re interested in the temporal distribution of certain events at a certain location. We can see in the image below that May 8 was a particularly dark day in our collective family. 

Timeline of the messages sent

I’m sure there are more ways of analyzing qualitative data in the form of text messages. However one goes about it, it is important to note that peoples’ narratives and stories are essential in learning about a place. Telling a story through data needs to be a multilayered process, it has to rely on more than just ‘points on the map’ or raw statistics. We learn the most when we listen with curiosity and when we listen to understand. 

Please follow and like us:

Nungwi, Chwaka, and Makunduchi Mapping

https://twitter.com/Fundi_Films

In 2020 and 2021, Spatial Collective provided mapping services for Zanzibar’s Nungwi, Chwaka, and Makunduchi settlements. The project fell under the broad framework of the Tanzania Urban Resilience Program and was meant to showcase how locally accessible, low-cost technologies can support resilience building and urban development planning by producing high-quality spatial information. Specifically, the assignment was meant to build the capacity of university students through an industrial training program and generate up-to-date spatial information to support urban planning in Nungwi, Chwaka, and Makunduchi.

On two separate occasions in 2020 and 2021, ninety students from the State University of Zanzibar participated in an eight-week-long industrial training program. During this period, the students learned about the theoretical and practical aspects of community mapping, including a wide range of soft skills such as communication and community management which are required in a realistic work setting.

In Nunwgi, using smartphones, handheld GPS devices, and high-resolution drone imagery, the students mapped over 600 businesses, 465 amenities, 4,000 trees, and over 38 km of roads. To update and improve the quality of the existing building dataset in Nungwi, they manually added 500 buildings to the digitized dataset of about 4,800, and also captured 62 tourism-related facilities, 99 restaurants, including entertainment areas, and 47 craft workshops. In the end, working with local historians in the area the students mapped 23 places of cultural significance.

Similarly, in Chwaka and Makunduchi, using the same technology and approach, the students mapped over 300 businesses, 500 amenities, 884 electrical infrastructure features, and over 50 km of roads. To update and improve the quality of the existing building dataset in Chwaka and Makunduchi, they added building attributes to over 7,500 buildings and captured 846 additional buildings.

https://twitter.com/Fundi_Films

During the program, the students gained fundamental knowledge in setting up mapping projects. At the same time, the datasets produced will benefit the Zanzibar Urban Services Project, the Department of Urban and Regional Planning, and other stakeholders as they undertake various urban improvement initiatives in the three areas. 

Finally, the project showcased how the potential of innovative technologies paired with the local capacity can provide stakeholders with timely and accurate information.

Please follow and like us:

Community Mapping Urban Risks in Mwanza

In late 2020 and early 2021, Spatial Collective (SC) carried out the Community Mapping of Urban Risks in Mwanza under the broad framework of the Tanzania Urban Resilience Program (TURP). To complete the project, the company partnered with the Humanitarian OpenStreetMap Team (HOT) and OpenMap Development Tanzania (OMDTZ). This assignment intended to showcase how locally accessible, low-cost technologies can support resilience building and urban development planning through the production of high-quality spatial information. Specifically, the assignment’s goal was to train university students and government authorities in community mapping by coordinating several community mapping engagements in pre-selected wards in Mwanza and producing up-to-date, open, and accurate data on exposure to floods and rockfalls.

Seventy-nine students from the Institute of Rural Development Planning (IRDP) and the Saint Augustine University of Tanzania (SAUT) participated in a two-month-long industrial training program. During this period, they learned about the theoretical and practical aspects of community mapping. By putting theory to practice, they, together with hundreds of community members, collected a series of disaster-related data touching on flood-prone areas, historical rockfall incidents, urban exposure, and drainage infrastructure. The work focused on two municipalities: Nyamagana and Ilemela.

Students participating in the training

Using a mixed-method approach which consisted of digitizing the satellite imagery, mobile and GPS field data collection, community consultations, and stakeholder meetings, the students collected more than 300,000 buildings using satellite imagery, 15,000 locations of flood-prone areas, 800 locations exposed to rockfalls, and more than 80,000 points on urban exposure. In addition, more than 1,500 drain segments with 177 km of drainage lines, 3,113 drainage-related points, and 1,328 elevation points were mapped using a combination of mobile phones and Do It Yourself Real-Time Kinematic (DIY RTK) GPS.

A sample of the data collected

After several months of fieldwork, the students gained fundamental knowledge in setting up community mapping projects and producing datasets that benefit Mwanza’s local authorities and other stakeholders in infrastructure improvement projects. Finally, the project highlighted the potential of pairing innovative technologies with local capacity to provide stakeholders with timely and accurate information.

Please follow and like us:

Mtaa and COVID-19

Informal settlements are often missing from the geographic and statistical representation of countries. Nairobi’s informal settlements are no exception. Very little information is available on the quality and quantity of public institutions and amenities, public services, or on the population itself living in the areas. They are often not even represented on official maps.

This invisibility became even more pronounced with the onset of Covid-19 crisis when movement in and around Nairobi, and Kenya, was significantly limited. This lack of free movement introduced yet another problem: due to the travel restrictions, the information flow from these areas also got restricted to an extent.

Social media, such as WhatsApp, Facebook, and Twitter filled in the gap of missing information but, while these tools do a great job of enabling easier public sharing, the information itself remains scattered through the interest groups. If you’re not a part of the group, you simply miss out.

In order to understand what our friends and co-workers, their families, and communities are going through, we needed to consolidate all the information in one spot in an easy, accessible way.

***

Mtaa is an application we developed to conduct a quick scan of the area where we plan to work. Users send texts, together with a location and an “emotion” attached to the location. Everyone is free to send whatever message they wish. The messages are open, without censorship or rules of what should be posted. We keep it this way because at the moment we rely on trusted sources for information and we don’t see the need to censor anything. We love such qualitative data because we love stories and stories of what is going on in informal settlements are often lacking.

Then, the pandemic hit.

***

As a company focused on community data collection, we had to find a way to support our colleagues, enable safe data collection, and still center community experiences.

Mtaa enabled us to do so.

Our friends utilize the platform to send messages – ​ safely from their homes – about experiences they and their communities are facing. These friends are trusted reporters, many of whom have worked with or been trained by us before, sharing experiences of the neighborhoods that are otherwise often missing from representation.

Gerry sending a message

Currently, they’ve sent about 500 messages from Mathare, Kibera, Kariobangi, Kawangware, City Centre, Ngong Road, Mukuru, and Umoja neighborhoods in Nairobi. The messages range from day-to-day interactions, reports of extreme kindness and sacrifice, small acts of daily life like painting graffiti and making of music videos, to talking about crime and lack of essential services, such as access to water and food.

Water situation is very dire here. We have gone so long without water services and where one gets some, the community scrambles and struggles with high cost and long queues in order to access the commodity.

Thievery activities taking place due to lack of finaces and very limited job opportunities. Especially now it is really bad along the roads.

Hii Covid imekuja sana waaaaah!!!

Woke up, all good! Good morning Mlango!

We must emphasize that no one is asked to, nor needs to, go and collect data outside the safety of their homes. The only rule is that they send whatever they want whenever they want it: rumors, feelings, and other relevant messages as and when they see fit.

We are now working on various analyses and visualizations of data to identify hot-spots, priority areas, and main issues in each community based on this incoming information. One of the visualizations – an interactive map of messages received and their locations – can be seen below. Now, our job is to analyze and share the information with relevant actors providing services on the ground. In the end, the goal is to make this information useful.

Geolocated messages from Mlango Kubwa, Mathare

Please follow and like us:

Household Survey

This is part of a series on the Industrial Training of Students on Zanzibar by Spatial Collective. You can read parts one, two, three, four, five, and six here.

One of the topics covered during the training was on using mobile phones to collect data on households. The goal was to introduce the students to mobile data collection and to teach them how to interact with community members while conducting a survey.

KOBO software was used for data collection and some of the data collected were:

  • building type, material, structure, roof, building levels, condition, and age
  • how many people – adults and children – live in the household
  • does the household have a drainage connection
  • does the household have a sanitation connection
  • does the household have an electricity
  • does the household have a water connection
  • how does the household get rid of the waste
  • has the community experienced flooding recently
  • has the household experience flooding recently
  • what were the main causes of flooding

The survey was carried out in one of the most flood-affected areas of Zanzibar Urban West. A few visualizations of the 2,300 households surveyed are displayed below.

Please follow and like us:

Mapping Drainage

This is part of a series on the Industrial Training of Students on Zanzibar by Spatial Collective. You can read parts one, two, three, four, and five here.

As mentioned in previous blog posts, at the start of the training, the students met with the stakeholders from the civil society, government and the private sector to determine what are their key data needs. During these meetings, the following themes continuously came up in the conversation: creating a base map, mapping roads and drainage, and conducting a household survey in the Zanzibar City’s Urban West.

Drainage mapping was likely the most complicated task performed during the training. This is because the building materials, shapes, and sizes of drainage lines and accompanied infrastructure often change rapidly as they make their way downstream towards the sea. For example, a concrete, rectangular, closed drainage of 1-meter in width and depth can suddenly transition into an open, informal, 25 cm deep ditch made of dirt.

To make sure everybody collects the same data, in the same way, we set up precise data collection protocols and spent a lot of time practicing with the students in the field.

We decided to collect the following data and attributes on drainage infrastructure:

  • Type of drainage: [ditch, drain, undergroun]
  • If underground: [culvert]; Profile: [round, boxed]
  • Built from the material: [concrete, steel, asphalt, sand, plants, trees, gravel, dirt]
  • If covered, by material: [concrete, grating, metal, wood]
  • If blockage: [dirt, concrete, rubish or solid waste, grass or plants[
  • Width: [top, bottom]
  • Depth
  • Profile: [open rectangular, tabulated, trapezoid, elliptical, trapezoid eliptical]
  • Infrastructure: manholes, drainage outlets, and inlets, bridges, etc.

In less than two weeks, the students collected about 35 kilometers of drainage lines and more than 650 drainage related points of interest. A map and the list of some of the data collected can be found below.

Please follow and like us:

Half a million buildings mapped

This is part of a series on the Industrial Training of Students on Zanzibar by Spatial Collective. You can read parts one, two, three, and four here.

Building upon previous efforts (read here and here), the State University of Zanzibar students finalized digitization of buildings on the Zanzibar Archipelago (Unguja and Pemba Islands).

Fifty students participated in the digitization effort. They added about 100,000 buildings to the pre-existing dataset of almost 400,000 buildings. The entire dataset was re-checked for errors and building reference numbers were assigned following the nomenclature put forth by the Zanzibar’s Commission for Lands.

Finally, once the mapping was complete, Spatial Collective secured an agreement with the Commission for Lands, who is the rightful owner of the data, to release the buildings and make them available in OpenStreetMap.

Below is an image of Unguja and Pemba Islands presented only through buildings.

Please follow and like us:

Creating a Base Map of Zanzibar

This is part of a series on the Industrial Training of Students on Zanzibar by Spatial Collective. You can read part one, two and three here.

Geo-services are becoming an essential part of the fabric of society and geographical information is now interwoven with many aspects of life. Key to that service is accurate, up to date, extensive base map.

Recent laudable efforts to create crowd-sourced open data on Zanzibar have shown that the quality of the data produced by the students and others, using innovative tools and methods, can achieve the quality standards required to fulfill the essential geo-data needs on the Islands.

The goal of creating a base map of Zanzibar Urban West was to teach the students about OpenStreetMap, open-source software, GPS and mobile data collection, and other technical aspects of mapping, data collection, and open data.

During this effort, the students mapped 42 shehias and added almost 15,000 points of interest, 100,000 buildings, and more than 200 kilometers of roads to the base map of Zanzibar City Urban West. An excerpt of the data is displayed below.

Please follow and like us:

Zanzibar Industrial Training Themes

Between August and October, we carried out the Resilience Academy Industrial Training during which we taught 50 students from the State University of Zanzibar on community mapping. The project was supported by the World Bank’s Open Cities Initiative and was also a part of the larger Tanzania Resilience Academy Initiative that targeted four Universities across Tanzania.

At the start of the exercise in August, the students met with the stakeholders from the civil society, government and the private sector to determine what are their key data needs. During the workshop, several themes continuously came up in the conversations:

  • A need for a base map of Zanzibar Urban West
  • A need for the road network of Zanzibar Urban West
  • A need for a drainage map of Zanzibar Urban West
  • A household survey of one of the most flood-affected areas in Zanzibar Urban West

Another goal was to use the Industrial Training to complete the digitization of Pemba Island, finalizing the mapping of buildings on the entire Archipelago in the process.

The training commenced on August 20 during which the students learned about project design, budget allocation, equipment set-up, basically, all the nitty-gritty of how to carry out an independent mapping of communities based on the stakeholders’ needs.

The specific topics covered were:

  • How to design a mapping project
  • How to conduct GPS data collection
  • Introduction to OpenStreetMap
  • How to map drainage
  • How to digitize drone imagery using QuantumGIS
  • How to use mobile phones for data collection
  • How to set up a household survey using KOBO software

The students then used this newly acquired knowledge to collect approximately 15,000 GPS points of interest, 35 kilometers of drainage lines, 200 kilometers of roads, digitized about 100,000 buildings on Pemba Island, gathered 2,000 household surveys and hundreds of public opinions.

In the next blog posts, we’ll explore each individual topic in detail.

Please follow and like us:

Summary of the work completed so far on Zanzibar

In June, we signed the contract for the work on Tanzania Resilience Academy. Shortly after that, on June 16, we visited Zanzibar where we met the Resilience Academy team, reviewed the venue and equipment, discussed the plans for the training, and organized logistics for our team.

In July, we prepared a Curriculum for the Industrial Training and wrote the Inception Report. We again visited Zanzibar on July 18 to review the documents with the team on Zanzibar and worked out the logistics for the training.

After that, the team took a short summer break for two weeks but early by August, we were back in the office. We purchased all the necessary equipment and Justus and I relocated to Zanzibar for at least a couple of months.

The Industrial Training started on August 20. Approximately 50 students were introduced to mapping theory and best practices during the first week of training, and on Thursday, August 22, we organized a stakeholder meeting to determine the mapping themes and the scope of the mapping exercise.

After the initial week of theory, the students spent the last three weeks in the field collecting data and updating OpenStreetMap of the Zanzibar Urban West. By the end of week four, we will have mapped (collect information on basic public amenities) almost 40 shehias (wards) in the city.

The work continues…

Please follow and like us: