Writing A Data Extraction To Web Page Program
-
In my area, there are few different law enforcement agencies that post real time data on car accidents. One is http://www.flhsmv.gov/fhp/traffic/crs_h501.htm. They post the accidents by county, and then in the location heading, they add the intersection and the city. For most of these counties and cities, our website, http://www.kempruge.com/personal-injury/auto-and-car-accidents/ has city and county specific pages. I need to figure out a way to pull the information from the FHP site and other real time crash sites so that it will automatically post on our pages. For example, if there's an accident in Hillsborough County on I-275 in Tampa, I'd like to have that immediately post on our "Hillsborough county car accident attorney" page and our "Tampa car accident attorney" page.
I want our pages to have something comparable to a stock ticker widget, but for car accidents specific to each pages location AND combines all the info from the various law enforcement agencies. Any thoughts on how to go about creating this?
As always, thank you all for taking time out of your work to assist me with whatever information or ideas you have. I really appreciate it.
-
-
Write a Perl program (or other language script) that will: a) read the target webpage, b) extract the data relevant for your geographic locations, c) write a small html file to your server that formats the data into a table that will fit on the webpage where you want it published.
-
Save that Perl program in your /cgi-bin/ folder. (you will need to change file permissions to allow the perl program to execute and the small html file to be overwritten)
-
Most servers allow you to execute files from your /cgi-bin/ on a schedule such as hourly or daily. These are usually called "cron jobs". Find this in your server's control panel. Set up a cron job that will execute your Perl program automatically.
-
Place a server-side include the size and shape of your data table on the webpage where you want the information to appear.
This set-up will work until the URL or format of the target webpage changes. Then your script will produce errors or write garbage. When that happens you will need to change the URL in the script and/or the format that it is read in.
-
-
You need to get a developer who understands a lot about http requests. You will need to have one that knows how to basically run a spidering program to ping the website and look for changes and scrape data off of those sites. You will also need to have the program check and see if the coding on the page changes, as if it does, then the scraping program will need to be re-written to account for this.
Ideally, those sites would have some sort of data API or XML feed etc to pull off of, but odds are they do not. It would be worth asking, as then the programming/programmer would have a much easier time. It looks like the site is using CMS software from http://www.cts-america.com/ - they may be the better group to talk to about this as you would potentially be interfacing with the software they develop vs some minion at the help desk for the dept of motor vehicles.
Good luck and please do produce a post here or a YouMoz post to show the finished product - it should be pretty cool!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should i be using shortcodes for my my page content.
Hello, I have a question. Sorry if this is been answered before. Recently I decided to do a little face lift to my main website pages. I wanted to make my testimonials more pretty. Found this great plugin for testimonials which creates shortcodes. I love how it looks like, but just realised that when I use images in shortcodes, these are not picked up by search engines 😞 only text is. Image search ability is pretty important for me and I'm not sure if I should stick with my plain design and upload images manually with all alt tags and title tags or there is a way to adjust shortcode so it shows images to search engines. You can see example here. https://a-fotografy.co.uk/maternity-photographer-edinburgh/ Let me know your thoughts guys. Regards, Armands
Web Design | | A_Fotografy1 -
Joomla Core Pages Delisted
Hey Everyone, This query may be more for a Joomla developer or someone that has had a similar issue. I'm not really looking for answers like, "check Google Search Console" or anything like that. We have a client who recently had all of their core pages delisted in Google but the blog is still being displayed in search results. For example, if you search "company name" they have a blog post that ranks #13 or so organically. I tested Google Search Console and Google is saying that the site is temporarily unavailable. We haven't made any changes or updates to Joomla's core structure so I'm unsure as to where this change is coming from. Here are some items we've checked: 1. Site searches within Google, resulted in seeing core pages are not indexed but blog pages are 2. Google Search Console - looked for manual actions (none found), looked at sitemap errors (nothing mentioned), looked at robots.txt (no issues here), attempted to fetch the site as Google (temporarily unavailable). 3. Called the hosting company (Rackspace) to discuss potential issues. They were extremely helpful but we were unable to find anything. The blog is actually a Module that was added so I'm thinking something has changed to block Google bots from the core Joomla structure but it hasn't blocked them from the blog structure. Without putting the company name or url on blast, has anyone heard of or experienced anything like this? Any help or insights would be much appreciated!
Web Design | | Leadhub0 -
Link colour on page?
I always thought that the link colour has to be different from text colour? I have come across a site http://www.printandpackaging.co.uk/ and it has made me question this belief, they seem to only have bolded the link which would be very nice if this is fine.
Web Design | | BobAnderson0 -
Page Content
What is the minimum amount of content a page should have to be seo friendly? What is the maximum amount of content a page should have to be seo friendly?
Web Design | | bronxpad0 -
Page Redirection solution
Page Redirection solution needs, there are 2 sites in the same folder and one page of old site is bxxxxxseo.com/products.php new site bxxxxxseo.com/product_list.php .there are many old page indexed i wanna redirect all old pages to relevant pages of new site using SEO friendly way .Any help really appropriate. Thank you
Web Design | | innofidelity0 -
Google Penalizing Websites that Have Contact Forms at Top of Website Page?
Has anyone else heard of Google penalizing websites for having their contact forms located at the top of the website? For example http://www.austintenantadvisors.com/ Look forward to hearing other thoughts on this.
Web Design | | webestate1 -
Landing Page - is this one a good landing page?
Hello Everyone, I want to ask about this landing page: http://www.rpgdicas.com.br/builds/diablo-2/build-necromancer-summoner-diablo-2.html It's in portuguese-brazil. What should I improve? Any tipps will be apreciated keywords: Build Necromancer Summoner, build necromancer summoner diablo 2, so on... thanks, everyone.
Web Design | | augustos0 -
Where to find high quality (affordable) web designers?
Hi everyone, I am looking for find high quality web designers that are affordable. I am open to many options. There are several things I have looked into. 1. I have looked for designers via CSS galleries, but I don't really know how to get in touch with designers or find them. Rand recently talked about this in a webinar, but if anyone has specific insights on how to find people this way, please let me know. 2. I have also looked into website design contests from sites such as: DesignCrowd.com 99designs.com CrowdSpring.com DesignContest.com I haven't used these services and I was wondering if anyone has experience with design contests. 3. I have looked into the option of hiring a freelancer on oDesk or a similar freelancer site. I don't really know the cost, how to find a good designer, how to avoid inexperienced but cheap designers and all the other such roadblocks that come along with freelancers. If anyone could provide insight into this, it would be greatly appreciated.
Web Design | | alexhoug0