1. Load libraries

library(jsonlite)
library(ggmap)
library(dplyr)
library(stringr)
library(tidyr)

2. Get a Google API console key

You need this to be able to use the geocoding functions in ggmap, as well as to be able to plot. I learned how to do this from Rebecca Henderson’s R-Ladies workshop, in which she pointed us to this Medium article explaining how and why you need an API. Note that you need to enable billing (i.e., with a credit card), but don’t need to pay to use the service.

Once you have successfully obtained an API console key, you need to enable it in your R session:

ggmap::register_google("your-secret-api-key")

2. Download your location history from your Google Account

You can download your own Google data, including from Maps, via Google takeout. Be sure to only select “Location History.” This can take a little while (it took about 30 minutes for me to download the last 4 years, which is my entire Maps timeline). The final output is a json file called Location History.json. You can also do this from within Google Maps by going to “Your Data in Maps” >> “Download your Maps Data.”

3. Parse the location history file

The jsonlite R package allows you to read in json files into R for further manipulation. I moved my .json output file to my R project folder and ran the following. This took a couple minutes.

file <- "Location History.json"
lh <- read_json(file, simplifyVector = TRUE)

Next you need to extract the location data and convert the time and date into a human-readable format. The output contains more columns of information (including velcocity, activity, and estimated accuracy), which I’ve excluded in the preview.

locations <- lh$locations
locations$time = as.POSIXct(as.numeric(locations$timestampMs)/1000, 
                            origin = "1970-01-01")

locations %>% select(1:3,time) %>% head()

##     timestampMs latitudeE7 longitudeE7                time
## 1 1437674692250  455027169  -735712951 2015-07-23 14:04:52
## 2 1437674752999  455028758  -735716985 2015-07-23 14:05:52
## 3 1437674813904  455028758  -735716985 2015-07-23 14:06:53
## 4 1437675197645  455026308  -735713548 2015-07-23 14:13:17
## 5 1437675258697  455026081  -735713646 2015-07-23 14:14:18
## 6 1437675326946  455026395  -735714586 2015-07-23 14:15:26

You can check your earliest and most recent maps data time stamps using min() and max() functions.

min(locations$time)

## [1] "2015-07-23 14:04:52 EDT"

max(locations$time)

## [1] "2020-01-02 10:32:05 EST"

My location tracking has been on since July 2015. The most recent point reflects the last time I downloaded my Maps data.

4. Tidy the location history

Latitude and longitude are recorded in Google in E7 format. You need to divide the original longitude/latitude values by 10^7 to get standard coordinate formatting.

I also found it helpful to round my final coordinates. The more decimal places you include, the more precise your location identification will be. In my case, all I really cared about was whether I could reliably capture border crossings. Some light Wikipediaing informed me that rounding coordinates to the tens gives you a resolution of 11.1 km, roughly allowing you to distinguish major cities. Zero decimal places, i.e., rounding to the nearest integer, allows for 111.1 km precision, which can unambiguously capture large regions and countries (see also here). Rounding greatly reduces the amount of data you have to work with, which speeds processing up tremendously later on. It’s also simply not necessary, IMHO, at least for this project, to know your location down to the meter.

To prepare for further reduction, I also added a coordinates column (coord).

loc <- loc %>% mutate(
     date = str_sub(time,1,10),
     time = str_sub(time,12,19),
     lon = round((loc$longitudeE7 / 1e7), 0),
     lat = round((loc$latitudeE7 / 1e7), 0),
     coord = paste(lon,lat,sep="_")
) 

I became a PR on November 22, 2015. I’ve filtered out dates prior to this (however, days within the 5 year period prior to obtaining PR status do count as half days, but for now I’m going to omit them).

loc <- locations %>% filter(date >= "2015-11-22")

I grouped the location data by date and coordinates - that is, one row for each unique date-coordinate combination. Since I had rounded my coordinates to 0 decimals earlier, this now gave me one row for approximately every 111 km. Importantly, I was able to collapse over days that I drove around a single city or over shorter distances. This allowed me to speed up further parsing below.

loc <- loc %>% 
  group_by(date,coord) %>% 
  summarize(n=n()) %>% 
  separate(lonlat, c("lon","lat"), sep="_", remove=FALSE) %>%
  mutate(lon = as.numeric(lon),
         lat = as.numeric(lat)) %>%
  ungroup()

To test whether rounding to the nearest integer was sufficient, I made a temporary data frame of 1 day’s worth of location history on a date that I know I crossed from Ontario to New York. This allowed me to confirm that rounding to 0 decimal places was more or less enough to capture what I wanted, knowing that this may mean I don’t capture every single day with 100% accuracy (for example, this runs the risk of days that I spent near a border being counted as crossing a border - a risk I felt was worth the time saved by the data reduction). You could just as easily a month or other specified time span.

tmp_day <- loc %>% 
  filter(date == "2019-10-20") # Look just at Oct 20 2019
  
tmp_day %>% select(1:6)

## # A tibble: 3 x 6
##   date       lonlat   lon   lat     n country
##   <date>     <chr>  <dbl> <dbl> <int> <chr>  
## 1 2019-10-20 -79_43   -79    43   423 USA    
## 2 2019-10-20 -80_43   -80    43   325 Canada 
## 3 2019-10-20 -81_43   -81    43   294 Canada

The n column reflects how many data points were logged for that coordinate on that day.

5. Coordinates 👉 Country: Reverse geocoding

Geocoding allows you to convert addresses into geographic coordinates. Reverse geocoding allows you to do the opposite: convert coordinates to addresses. The ggmap package provides a function revgeocode() to do just this.

I created a small function to use revgeocode() to get the address from my location data. I used the stringr package to extract the country I was in for each observation - the country of the address is always(?) the last element of the list produced by revgeocode(). This function returns either just the country, or the full address retrieved from Google.

get_country <- function(lon,lat, out=c("country","address")){
  address <- suppressMessages(
    revgeocode(
      as.vector(
        c(as.numeric(lon),
          as.numeric(lat)))))
  
  # Split address by commas
  address_split <- str_split(address, pattern = ", ")
  # Country is always the last element
  l <- length(address_split[[1]]) 
  country <- address_split[[1]][l]
  
  # Convert NAs to characters to make life easier later
  if(is.na(country)){country <- "NA"}
  if(is.na(address)){address <- "NA"}
    
  if(out=="country"){
    return(country)
  }else if(out=="address"){
    return(address)
  }
}

I then created another function to determine, for a given data point, whether I was or was not in a given country (e.g., Canada). This function returns 1 if “country” equals the string “Canada”, and 0 if not. This allows me to eventually sum the days spent in Canada.

visited_country <- function(x, country = "Canada"){
  if(x==country){
    visit <- 1
  }else{visit <- 0}
}

Next I used mapply() to iterate over each observation in my location history data, applying my get_country() and visited_country() functions to each row.

# Get country
loc$country <- mapply(get_country, 
                      loc$lon,loc$lat, 
                      out = "country")
# Get address
loc$address <- mapply(get_country,
                      loc$lon,loc$lat,
                      out = "address")

# Log whether I visited Canada
loc$visited <- mapply(visited_country, 
                      loc$countries, 
                      "Canada")

head(loc) # roads were renamed for privacy

## # A tibble: 6 x 8
##   date       lonlat   lon   lat     n country address               visited
##   <date>     <chr>  <dbl> <dbl> <int> <chr>   <chr>                   <dbl>
## 1 2015-12-01 -81_43   -81    43   442 Canada  2111 Swan Rd, Dorche…       1
## 2 2015-12-02 -81_43   -81    43   403 Canada  2111 Swan Rd, Dorche…       1
## 3 2015-12-03 -81_43   -81    43   446 Canada  2111 Swan Rd, Dorche…       1
## 4 2015-12-04 -79_44   -79    44     2 Canada  815 Martin Rd W, Whi…       1
## 5 2015-12-04 -81_43   -81    43   330 Canada  2111 Swan Rd, Dorche…       1
## 6 2015-12-04 -83_42   -83    42    35 Canada  717 McCormick Rd, Ha…       1

Note that I modfied the street names so as not to show actual addresses in this post. Because of the coordinate rounding, the addresses also likely do not reflect my exact location (just a location within 111km).

Finally, I computed how many days I was in Canada by grouping my data by date and summing my visits.

days_in_country <- loc %>%
  group_by(date) %>%
  summarize(n_in_canada = sum(visited),
            countries = paste(country,collapse=", ")) %>%
  ungroup() %>%
  mutate(in_canada = ifelse(n_in_canada > 0,1,0))

sum(days_in_country$in_canada)

## [1] 1101

Looks like, since November 22, 2015, I’ve spent 1101 days in Canada, just over the required 1095! Since my pre-PR days count for some to use as padding (and since I do still go back and forth to Canada periodically), this is good news. I’ll be eligible to apply for another year.

6. Logging Trips

I could also now collapse over countries and days for a log of the trips that I took. I only care about trips outside of Canada, so I’ll filter Canada out first, then set up columns for trip “start” and “end” dates.

trips <- loc %>%
  filter(country!="Canada")%>%
  group_by(date, country) %>%
  summarize(start = min(date),
            end = max(date)) %>%
  ungroup()

head(trips)

## # A tibble: 6 x 4
##   date       country start      end       
##   <date>     <chr>   <date>     <date>    
## 1 2015-12-04 USA     2015-12-04 2015-12-04
## 2 2015-12-05 USA     2015-12-05 2015-12-05
## 3 2015-12-06 USA     2015-12-06 2015-12-06
## 4 2015-12-09 USA     2015-12-09 2015-12-09
## 5 2015-12-10 USA     2015-12-10 2015-12-10
## 6 2015-12-11 USA     2015-12-11 2015-12-11

Next, I “rolled up” the dates of each trip by using the lag() function (thank you to this helpful StackOverflow post because I had no clue what I was doing here). I imagine the lubridate package would also be super handy here, but I didn’t go down that route.

trips <- trips %>%
  mutate(gr = cumsum(start-lag(end, default=1) != 1)) %>%
  group_by(gr, country) %>%
  summarise(FromDate = min(start), 
            ToDate   = max(end),
            dur = as.Date(ToDate) - as.Date(FromDate)) %>%
  ungroup()

This glimpse of the earliest part of my trip log captures a couple short trips to the US, a trip to Isreal, a layover in Belgium, and a return to the US.

head(trips,5)

## # A tibble: 5 x 5
##      gr country FromDate   ToDate     dur   
##   <int> <chr>   <date>     <date>     <drtn>
## 1     1 USA     2015-12-04 2015-12-06 2 days
## 2     2 USA     2015-12-09 2015-12-13 4 days
## 3     3 Israel  2015-12-18 2015-12-22 4 days
## 4     4 Belgium 2015-12-31 2015-12-31 0 days
## 5     4 USA     2015-12-24 2015-12-30 6 days