I’ve been working on mapping the internet using a multi threaded python program that scans for open port 80 across the entire internet. I’ve built this front end to the system here.
I also built a screen scrapper for a website that geolocates IP addresses to use in my mapping but after a few requests the website blocks me (Screen scrapper code is in the same GitHub.)
I did some research and found about 7 websites that sell these geolocation data-sets for $50-$5000 depending what all you want. Two of the sites have free data-sets. Free is all I can afford so I’ve downloaded them and started working on uploading them to my webhost Dreamhost.
It would be simple if Dreamhost would all you to “Load Data INFILE” but they will not. Because their mysql servers are separate form their file hosts. Meaning mysql does not have access to your sites files. In order to bypass this issue we have to covert the csv from IP2LOCATION to a .sql insert file. Then run it from command prompt to get the best import speed.
Here you will find the steps and code involved in converting the csv file and upload it to your databases.
I hope this helps someone else with dreamhost issues also.