Monday, September 21, 2015

A Tool for Mapping Drunk Tweets and Police Shootings

How do we map the density of a set of events? For example, we might want to map locations of tweets supporting Bernie Sanders as opposed to Hillary Clinton or locations of housing evictions or locations of police shootings. I confronted this problem recently while writing a post for Quartz (which they split into two) about where people tweet more about wine and where they tweet more about beer. Here’s a finished map (you can see more in the Quartz posts); color shows the fraction of #beer or #wine tweets which are about #wine, with red denoting pro-#wine areas.
I built a tool which lets you make maps like this, and because I think this problem is often useful to solve and often solved badly, I provide some thoughts on how to make maps below. If you’re not interested in details, you should probably just look at the maps in the Quartz posts, but if you keep reading you’ll at least get to see me make a lot of bad maps.
One simple thing to do is just to make a state-by-state map where each state’s color corresponds to the density of events. This has a few problems. It requires us to map all the latitude, longitude pairs to their states, and if we want to look a country besides America, we need to adapt our method; more fundamentally, it’s not very high-resolution, and there are often interesting patterns at the sub-state level.


To get better resolution, people often just plot the exact latitude, longitude location of the tweets, so you get visualizations that look like this:
Screen Shot 2015-09-03 at 6.16.55 PM.png
Which is pretty but not very useful (like your momma!) because it basically just shows us where the cities are. (I have lost track of the number of data analyses I have read that can be summarized as, “when you have more people, more things happen”).


What I think you usually want to do is plot the density of events relative to some background. For example, I don’t care about the absolute density of wine tweets, which will be heavily correlated with population density; I care about the fraction of beer/wine tweets which are wine tweets.
So one thing we can do is estimate the density of wine tweets, estimate the density of beer tweets and then plot the difference: densitywine - densitybeer. (We can estimate density using a method called, appropriately, kernel density estimation). Here’s what happens when we do that; red denotes areas with more wine, blue with more beer.
overdensity_problem
The problem is that the reddest areas aren’t necessarily the areas where 90% of people are tweeting about wine; they might also just be the areas with a ton of tweeters (which will also have larger differences between densitywine and densitybeer). So maybe we really want something like densitywine / (densitybeer + densitywine), which we can interpret as the fraction of tweets which are about wine. Here’s what that looks like.
too_variable_problem
The problem, basically, is that the ratio of two things is unstable when the denominator gets small, which often happens. I tried various ways of getting around this but they were finicky.


So here’s an alternate solution: for every point which you want to color, look at the closest 10 beer/wine tweets and see how many of them are about beer. It seems like this will take a really long time if we have, say, 10,000 points and 50,000 tweets. Luckily, computer scientists have devised an efficient way of doing this which takes about two lines of code and a second to run (#MyFieldIsCoolerThanYourField) [1].
hemorrhaging_problem
This doesn’t look so good because California appears to be hemorrhaging, but once we mask off the oceans we get a nice map:
okay_thats_better


I’ve posted the code to make maps like these on GitHub so you can make maps of your own (and let me know if you find anything cool)! Keep in mind that the map will be less reliable in areas with little data. You can use it on any data (not just from Twitter) that has latitude and longitude. It requires knowledge of Python, so shoot me an email if you get stuck. If I get enough complaints from people who want to make maps but can’t use Python, I’ll just build a web tool.


Also, I am not a mapmaking expert, so feel free to tell me how I could’ve used CartoDB or whatever to do all this! (My problem with CartoDB is that the free version won’t keep data private and limits the size of your datasets to 50 MB. I’m not really a 50 MB kind of girl.)


Notes:
[1] We just train a k-nearest neighbors classifier and plot its classification surface. There’s also the question of how you choose the number of nearest neighbors to look at. If you choose too small a number, the map gets very splotchy:
too_small_bandwidth_problem

And if you choose too large a number, you lose real details. I’m not exactly sure how best to choose, so hit me up if you have thoughts.  Do not tell me to use cross-validation. Seriously, we’re mapping drunk tweets here.

A final note: this tool obviously relies on you having latitude / longitude data. Many datasets are not in this form (eg, they might include addresses instead) and I have not found a great way to rapidly convert between addresses and latitudes / longitudes, because many APIs are rate limited. Let me know if you have a good solution to this problem.

49 comments:

  1. Very cool, and interesting. Public health / epi Mike from NC here ... reminds me of ggplot2's viz asthetic, which I love. I'll be taking a look at this for some other R projects, and given the commonalities in python and R, I expect this will be a great learning tool. Thanks for sharing.

    ReplyDelete
    Replies
    1. Thanks, I'm glad it was useful to someone :) I don't like R, but I love ggplot2.

      Delete
  2. I have an unusual amount of data in the form of "Long/Lat, {0,1}", and I've always wanted to make maps like this with them but never found the time, so I'm excited to see your code! How well do you think it scales?

    ReplyDelete
    Replies
    1. Cool! How many points do you have?

      Delete
    2. ~100 million, but dramatically downsampling is just a fact of life. How many tweets were in there?

      Delete
  3. Ever wanted to get free Twitter Followers?
    Did you know you can get them AUTOMATICALLY AND TOTALLY FREE by using Like 4 Like?

    ReplyDelete
  4. Hey emma ,
    Your doing a great job, delivering such valuable content is not easy. Keep doing !

    Academic Editing and Proofreading Services
    Academic Proofreading Services

    ReplyDelete

  5. Thanks for writing a informative post for us. We also share a post on Corrugated, Printed and carton boxes Manufacturers in India. Durable packaging is the leading Manufacturers Supplier and exporter of quality carton box.
    Contact durable packaging team using service . Dial Toll-Free Number:- 0130-6544658 for help all type of problems.

    ReplyDelete
  6. The information you shared is nice. We also share a post on Name change services Name change is available professional services in name change industry. For any assistance talk to us or call us:- 9871075159 and find free consulting.
    NAME CHANGE IN GAZETTE
    NAME CHANGE AFTER MARRIAGE
    NAME CHANGE DOCUMENT
    NAME CHANGE AFTER DIVOCE
    NAME CHANGE AFTER INTER RELIGION MARRIAGE

    ReplyDelete
  7. Thanku for sharing an informative post. I also share a post on fire station .A-onefire is the best fire saftey service provider in delhi . For any help please call us:-9811057544 then we provide a better service.

    For more information visit my website given below :
    website : https://www.a-onefire.in

    Firestation

    Fireservices

    No 1 Fire AMC company india

    Fire fighting maintenance

    ReplyDelete
  8. I love this. It is soo informative. Are you also searching for cheap assignment writers we are the best solution for you. We are best known for delivering the best services to students without having to break the bank

    ReplyDelete
  9. If you found any common issues in your Quickbooks like login issues, connectivity issues, Printing errors, PDF related issues, and Performance issues, you can simply download QB Tool Hub which is the combination of all essential tools to fix common issues.

    Quickbooks tool hub download

    ReplyDelete
  10. We do not charge a single penny if our technicians fail to repair your device. No one other than Geek squad can provide instant & same services at the same reasonable prices.

    For more information visit site :- Geek Squad tech support

    ReplyDelete
  11. Your blogs are amazing. Keep sharing. I love them Are you also searching for urgent assignment help? we are the best solution for you. We are best known for delivering the best urgent assignment help.

    ReplyDelete
  12. I love this. It is soo informative. Are you also searching for cheap assignment help we are the best solution for you. We are best known for delivering the best services to students without having to break the bank

    ReplyDelete
  13. I love it here. Keep sharing your good vibes. I love them Are you also searching for assignment help UK? we are the best solution for you. We are best known for delivering cheap assignments to students without having to break the bank

    ReplyDelete
  14. I love this. It is soo informative. Are you also searching for cheap assignment writing help we are the best solution for you. We are best known for delivering the best services to students without having to break the bank

    ReplyDelete
  15. Good blog. Keep sharing. I love them Are you also searching for Cheap assignment help? we are the best solution for you. We are best known for delivering cheap assignments to students without having to break the bank

    ReplyDelete
  16. I want to testify on how I got cured from Herpes. I have been living with this disease for the past 11 months, i have done all i can to cure this disease but all my efforts proved abortive until i met an old friend of mine who told me about a herbal doctor called Dr Oniha, she told me that Dr Oniha have cure for all kinds of diseases, though i never believed in it because my believe was that there is no cure for Herpes disease. But I decided to give it a try when I contacted Dr Oniha, he told me he has a cure for herpes that he cured with herbal medicine. I order for the herbal medicine, which Dr Oniha sent to me through a courier service which i make use of and now behold the herpes is gone and i now have my life back,if you are out there living with this disease i will like you to also contact Dr Oniha and get this disease cured out of your body. I am a living testimony of Dr Oniha herbal cure. Thanks once again Dr Oniha for you are God sent. contact Dr oniha through his contact information.
    Email:driyaseherblahoe@gmail.com
    Whatsapp number: +2347057052206

    ReplyDelete
  17. Thank you for posting such a great article. Keep it up mate.

    task club app download

    ReplyDelete
  18. Thank you for posting such a great article. Keep it up mate.

    RTPS Bihar

    ReplyDelete
  19. After exploring a handful of the blog posts on your blog, I seriously like your way of writing a blog 카지노사이트

    ReplyDelete
  20. I’m very pleased to discover this site. I want to to thank you for ones time for this particularly wonderful read!! I definitely savored every part of it and i also have you saved as a favorite to see new information on your blog. 토토

    ReplyDelete
  21. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site. Feel free to visit my website; 토토사이트

    ReplyDelete
  22. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with some pics to drive the message home a little bit, but other than that, this is magnificent blog. A great read. I’ll certainly be back. Feel free to visit my website; 카지노사이트

    ReplyDelete
  23. I have to thank you for the efforts you’ve put in penning this blog. I’m hoping to check out the same high-grade blog posts by you later on as well. In truth, your creative writing abilities has encouraged me to get my own blog now Feel free to visit my website; 카지노사이트

    ReplyDelete
  24. This really is an incredibly amazing powerful resource that you’re offering and you just provide it away cost-free!! I comparable to discovering websites that view the particular price of providing you beautiful learning resource for zero cost. We truly dearly loved examining this web site. Be thankful! Feel free to visit my website; 배트맨토토

    ReplyDelete
  25. I think this is one of the best blog for me because this is really helpful for me. Thanks for sharing this valuable information for free Feel free to visit my website; 카지노사이트

    ReplyDelete
  26. Thank you for posting such a great article. Keep it up mate.

    Haryana Rozgar Mela 2021

    ReplyDelete
  27. If you are looking for online grocery shopping in durgapur, download Bazaar Durgapur app from playstore. Avail discounts on combo offers.

    ReplyDelete
  28. Such an amazing and helpful post. I really really love it.
    바카라사이트

    ReplyDelete
  29. Luxury Home Builders Melbourne - Looking for luxury home builders in Melbourne? Bullseye offers signature custom homes plus contemporary, modern and affordable new homes.

    ReplyDelete