New handwriting recognition study compares usage and performance of OCR, ICR and manual data entry

Companies still collect much of their critical information via pen and paper, yet they ultimately need this information to be available in their digital systems. How much do companies rely on this handwritten data, and how do they then convert it to the digital data they ultimately need? This recent study by the Association for Information and Image Management  (AIIM) gives the answers:

Companies rely on handwritten data…

From AIIM

Of the companies surveyed for the study,  50% identified handwritten information as important to their business processes and a full 25% identified it as playing a key role for them. This data could be generated internally, through employee evaluations, culture surveys, site inspections, invoices, walk sheets, etc. It could also come from current or potential clients, in the form of newsletter signup sheets, registration forms, raffle tickets, satisfaction surveys, comment cards, mail-orders forms, purchase orders, and even signed contracts.

…But they struggle to convert handwriting to digital data

data extraction

While companies rely on handwriting to collect data, they need that data entered into their computerized systems quickly and accurately. How are they bridging the paper and digital worlds? The reality is that most companies live with a painful disconnect between their data collection methods and digital data needs. More than half of those surveyed enter the data by hand, while another third rely on OCR, and another 12% use ICR (intelligent character recognition). Before Captricity, there really were no other options available.

Manual Entry, OCR & ICR are Woefully Inadequate:

Unfortunately, all three options – OCR, ICR, and manual entry – come with significant trade-offs in terms of flexibility, turnaround time, and/or quality.

    All of us at some point have dealt with manual data entry. It’s slow, often expensive, not always accurate, and can lead to significant lag times in getting your data. Many companies tell us they have backlogs of months or even years that their manual data entry staff just have not been able to deal with.

    OCR (Optical Character Recognition) converts images of text into a digital, machine-readable format. While it tends to work adequately for well-scanned and printed text, it is extremely inaccurate for handwriting, and yields only a “bag of text”, not structured data. In other words, if you start with a scanned, typed form, OCR will give you a .txt file, not a data set. And if you start with a form filled in by hand, OCR will give you very little useful data at all.

    ICR (Intelligent Character Recognition) was created to more accurately read handwriting. If you have ever filled out a driver’s license application or customs form, you’re already familiar with the highly-regulated ICR-ready forms, where boxes or “combs” (small vertical lines) separate each letter. While this system can read hand-printed text a bit better, it’s as limiting as a Scantron bubblesheet is to teachers who want to ask open-ended questions. ICR makes free-form text and short answers almost impossible. Furthermore, setting up ICR-compatible forms takes time and expertise, requiring significant up-front investment. For the vast majority of those organizations that rely on handwritten data, this is not a practical solution.  They are in a tough spot.

Enter Captricity for REAL handwriting recognition.

Our unique data capture technology was created specifically to turn any handwritten form, no matter the format, into digital data quickly and accurately. Multiple-choice, numerical response, likert scale, short answers and long answers are no problem! There is minimal set-up and no software to install.  Take your completed forms, scan or photograph them, and upload the images to Captricity. Our special mix of computer algorithms and human intelligence extracts data faster than manual re-keying and more accurately than OCR, with more flexibility than ICR.

Source:http://captricity.com/handwriting-recognition-study-on-ocr-icr-manual-data-entry/

Tips For Easier Product Uploads In Magento

Uploading products into Magento can be a very time consuming task, especially when you need to upload several hundred or even several thousand products.  Fortunately there are some shortcuts that you can take to make this a quicker process.  This guide will provide a high level overview as to the most effective methods for achieving quicker bulk uploads into Magento.  We also suggest you refer to the Magento user guide for detailed tutorials on performing some of these tasks.

Utilize Magento’s Bulk Product Import Feature:

In addition to giving you the ability to add products individually, Magento also provides bulk import capabilities.  By utilizing Magento’s bulk import feature a user can import a large number of products from a single CSV file.  While the formatting requirements for bulk product uploads into Magento are very specific, you can easily get an understanding as to how your product must be formatted by exporting existing products out of Magento into a CSV file.  It’s important to note that since some of your products may be configurable or have multiple variations the CSV  formatting requirements for these products will differ from those without.

Upload Images In Bulk:

Instead of individually uploading images at the product level you can upload images in bulk.  Bulk image uploads in Magento are always performed via FTP.  Additionally, images must be appropriately labeled according to the SKU and or SKU variation of the product they are associated with.

Add Categories and Attributes In Bulk:

While this does not directly correlate to uploading individual products, taking this step will make the entire import process far more efficient.  Preparing all of your attributes and categories for a bulk import before uploading any products will allow you to easily associate products with categories and attributes that you will have already created.

Limit File Size For Bulk Imports:

By keeping the number of products you are uploading to Magento at one time to a reasonable number you will prevent any potential server time outs or delays in successfully uploading all of your product data.  Dividing a CSV file with 500 + products into two or three files and running separate uploads will require a little more time on the front end but will invariably prevent future headaches.

Updating Existing Products:

You can also utilize Magento’s bulk import feature to update existing products on your storefront.  To make bulk updates to existing products via a CSV file you only need to import the SKU field as well as the field(s) which you would like to update.  Once successfully uploaded into Magento your changes will take effect on your storefront.

If you’re looking for a faster and easier way to upload and manage both product data and images in Magento, ClaraStream provides a web based application that will integrate directly with your Magento storefront, allowing you to quickly and easily upload new products or make updates to existing products, and avoid tedious data formatting in spreadsheets and hours of manual data entry.  Take a quick TOUR now to learn how ClaraStream can help you save time uploading and managing your product data.

Source:https://www.clarastream.com/2013/06/tips-for-easier-product-uploads-in-magento/

Data cleaning service to clean ways to retrieve

Data cleansing or data scrubbing and one or to identify an act of fraud or false proof. Is dataset table to the right? Many companies, business sales and sales by the maid service to provide data to the database. Data cleaning company helps to set the date and error free.

After cleaning removes all consistencies Dataset is consistent with other similar systems. Data validation is the process of separating and removing some typos. Data transformation, statistical methods, parsing, syntax errors and eliminate duplicate data as known technique is used for cleaning. Nice and clean data needed to meet the criteria listed below:

Accuracy: density, integrity and stability.

Completion of missing data must be corrected.

Density: Data released and the price in proportion to the number of values must be well known.

Consistency: challenges and sense deals with the differences.

Uniformity: focused on irregularities or indiscretions.

Integrity: a combined value of wholeness and soundness criteria.

Unique: It is related to the number of duplicate data.

Data cleaning services are offered by companies:

Remove Duplicate ideas.

Tagging and Identification of a record or facts.

Remove duplicate or fake and false evidence.

Data verification.

Delete old records.

Opt-in and opt facts as third parties in order to remove the list

Data cleansing, aggregation and organization.

Identify incomplete or inaccurate facts or figures.

Product specifications, order and establish the facts, including metaphors, improved.

Duplicate data or data which records seem to be as many as received.

Common problems of data cleaning applications:

Sometimes there is a loss of information in the data. No doubt, invalid and duplicate entries are removed, but often the information is limited and is insufficient for a number of entries. It also leads to a loss of information is removed. Data cleansing is very expensive and time consuming. Thus it is important to maintain effectively.

Fortunately, the benefits worth more and more challenges.

And most companies these days, depending on the existence and quality of the data that have business continuity. Data mainly customer information, customer profiles, different products, addresses and important people and market research, etc. This information is mainly collected from various databases and phone numbers are on the technical details. Since these databases use different formats or styles, the data collected are very clumsy and sometimes incomprehensible, but which we cannot control the way data is stored in the database.

So, the best solution for us to organize data is to implement a data called cleaning process. there are various software available in the market that can help clean data are applied.

It is a very important process for their business activities depending on the quality of the data.

Which in turn will lead to losses?

Ways to clean data to retrieve:

1) When importing data, make sure that there is a common format for applying anywhere it is stored, this will ensure consistency.

2) Dictionary software or use MS Word to check for spelling mistakes or grammatical errors frequently. This must be done manually, it can be very time consuming for the entire above amount of information.

3) When copying to an external source of data is always copied into the notepad so that all types of formatting are done.

Source:http://www.tampabaycleaning.com/172-data-cleaning-service-to-clean-ways-to-retrieve-2

Website data scraping is not an easy

Website data scraping is not an easy task and it takes tremendous time when it comes to analysis and restructuring of the data. It is for these reasons that, you should visit us as we make this process look simple. We have a team of skilled and experienced data scrapers who will make the result from the project that you present future proof and flexible enough to fit into as many situations as you may think of or you may be finding solutions for.

Indeed, our website data scraping experts are knowledgeable and they will use their experienced hands to deliver the best data to you and within a short duration. In Web Data Scraping process input source will be web resource and most common output formats are xls, csv, XML, notepad, word file etc. Website Data Scraping having excellence to scrape database from HTML, XML, text, word file, images, reports, PDF files etc.

As world is growing fast every businesses having higher value of time so values of manual work is going rapidly down day by day. Imagine how many days it will take to scrape millions of records manually, may be over years. As world is rising extremely fast so we have to upgrade ourselves with time and its necessities.

Website Data Scraping introducing ourselves as world’s most preferred and reliable data scraping service provider. Website Data Scraping equipped with latest tools, techniques, technology and experienced manpower. We upgrade our tools, technology as per client’s necessity after certain interval to convey tremendous quality to our worldwide clients.

We are capable to deal with composite type of web scraping requirement and deliver world class quality before expected time. Our outstanding quality, time duration and previous client’s feedback force us to self-importance on ourselves as one of consistent and high quality web scraping service provider. High quality, time duration to complete work and price quote is matters a lot for any client and we try to fulfill all these needs.

We always prefer our entire client as priority customer’s weather we are getting business of only $10 from them. Website Data Scraping never compromise in quality and delivery time and due to these reasons you can try us for your Web Data Scraping requirement.

Web Data Scraping

Can you imagine to get thousands, lacks or millions of web based database in usable format only in 2-10 days? Yes, now its possible with Website Data Scraping. Get over thousands of web based database scraped only in few days and reuse those database for various purposes.

Business Directory Scraping

Online business directories are the best sources to explore the contact details of required service provider. We can help out to build your own niche business directory or in email marketing campaign by collection validated email ids. Don’t hesitate and contact us with business directory link in order to start working.

Web Research and Data Collection

Website Data Scraping having experienced team for internet searching, web research and data collection to satisfy our client’s requirement and make some profit for organization. Our primary key is to satisfy our customers’ needs at lowest price quote.

- Business directory scraping – yellow pages, yell, yelp, scoot, manta, lawyers, b2bindex etc.
- Report mining, document data scraping, PDF and scanned images scraping.
- Metadata scraping, web crawling, text corpus, weather data mining, stock data scraping.
- Job wrapping, resume scraping, students email id scraping, school and university data scraping.
- Web research, web data mash up, internet searching and data collection.
- Product scraping, image scraping, online price comparison and comparison of feed aggregates.
- Data scraping from LinkedIn, twitter, face book and other social networking sites.
- Product scraping from eBay, Amazon, eCommerce and online shopping websites.

Source:http://www.bharatbhasha.net/finance-and-business.php/404654

Product Feed Integration and Scraping Products From Supplier Web Sites

This is an old post. The information it contains is probably out of date or innacurate

This is a post that was written a long time ago and is only being kept here for posterity. You should probably look up more recent blog posts related to the subject you are researching

One of the big tasks that any ecommerce retail business must undertake is the continual updating and inserting of products into the catalogue. Done one by one, this task can take a ridiculous amount of time. In some instances there is no better option, but in the vast majority of cases there is!

Product Feed

The ideal scenario is that your supplier makes available an up to date product feed which is regularly updated and contains all of the information you need to insert those products into your catalogue. The challenge with this is that it is highly unlikely that you will literally be able to upload this data as is. The reason being that each ecommerce system has its own quirks and separate ways of doing things. Before you can upload this data into your feed, it is highly likely that it will need to be altered and prepared for insertion.

You could do updating by hand – but that brings us back to our first point. Doing things by hand can take a ridiculously large amount of time. Instead – we recommend that you have a script which does all this preparation for you.

In fact this task is something that Edmonds Commerce specialises in. Not least because it is something that we have done plenty of and so we have a good understanding of how to do the job. Furthermore – we understand how to do the job well.

Spidering and Scraping Products from Supplier Web Site

If your supplier does not provide a feed, or if the feed they supply does not have all of the information that you want, you might think you are stuck. You are not!

It is perfectly possible to build a system which will visit every product on your suppliers web site and grab all of the information and pictures and then save them into a format that you can insert into your catalogue system. It is even possible to extend the scraping system so that it goes all the way and inserts the products into your site for you.

Again this is something that Edmonds Commerce specialises in.

Conclusion

If you find that you or your staff are spending large amounts of time manually copying and pasting information from supplier web sites – you need to ask yourself if that is really cost effective. Whilst developing a script to process a feed or scrape a supplier web site might involve a significant initial outlay – the humongous saving in staff time ensures that you will quickly recoup this cost and then will be straight into a profitable scenario. Furthermore your catalogue will be absolutely up to date with the latest pictures, information and prices meaning that you have the best chance to sell those products.

If you want to discuss how Edmonds Commerce could help you achieve these great goals of cost reduction and totally up to date catalogue – please do get in touch.

Source:http://edmondscommerce.github.io/spidering/ecommerce/product%20catalogue/product%20feed/scraping/product-feed-integration-and-scraping-products-from-supplier-web-sites.html

The Importance of Mindfulness in Data Entry

Data entry is fast becoming a lucrative field of employment. While not necessarily the most glamorous job in the world, data entry is still a vital job in any field. There are all kinds of data entry jobs on the market, from working as an independent contractor and entering data for different companies, to being the data entry personnel in a general company, to being full-time data entry staff at a company that specializes in data management. The skills required are quite specific, and people that have the right skills can excel in data entry. People who can excel at data entry are also likely to enjoy doing data entry, and take satisfaction in seeing the work completed. The skill set for people who enter data is small but specific, with mindfulness playing a key role.

What is Mindfulness?

Mindfulness is, in a general sense, being present and attentive to the task at hand. When we are mindful, we are aware of what we are doing and what is going on around us. Mindfulness involves paying attention. Mindfulness actually means to not be caught up in the mind, rather, to be caught up in the experience. Our minds can charge around with thoughts, worries, anxieties and other distractions; being mindful is a practice of stepping away from this racing mind and into awareness of what we are doing at any given moment.

Why is it Important to Data Entry?

Data entry personnel would do well to practice mindfulness in their work. Data entry is, in almost all cases, a tedious and repetitive task. Generally, human beings faced with repetitive tasks have a tendency to stop thinking and turn on auto-pilot rather than paying attention to the work. It takes a certain level of energy and brain power to remain mindful when entering data.

The most obvious benefit to entering data mindfully is the improvement of accuracy and lessening of the rate of mistakes, however large or small. Mindfulness while entering data can also contribute to the work being done faster.

How to be Mindful while Entering Data

If you have been into data entry for a long time and have your patterns and systems, beginning to practice mindfulness may involve a bit of a learning curve. Humans are creatures of habit and we attach to patterns, however good or bad for us. However, patterns can be unlearned and new patterns can be picked up.

Adding a mindfulness practice to your life will contribute to your skillset as data entry personnel, which could lead to advancing your career. A simple mindfulness practice to do outside of work is a short meditation each day. While at work, try to deliberately slowdown every once in a while and pay close attention to the data you are entering. Over time, this practice will contribute to your work becoming more accurate and you will find you are turning out more of it at a greater pace.

Data entry is becoming a competitive field for personnel. All manner of company need their data entered, whether in-house, outsourced to a data management company, or done by a private data entry contractor. Being mindful and attentive will put you in the lead for getting those contracts and doing the best work you can in the field.

Source:http://www.dataentryoutsourced.com/blog/the-importance-of-mindfulness-in-data-entry/

Top Programs for Online Data Entry

Data can be entered online and offline. To be proficient in data entry, it is helpful to build a skill set that can work with online or offline programs and techniques. To be a leading-edge data management company to whom people outsource their data entry, it is vital to know all the software and programs available for online and offline data entry.

For companies in the business of managing data, it is helpful to have a wide range of software and programs as well as trained staff available. This will contribute to greater client satisfaction. If a client presents a job to the data management company, a savvy company will know exactly which program to use that will best fit the job. Here is a general look at some of the most commonly used software and programs for online data entry.

Entering Data for Analysis

One version of online data entry is data that needs to be entered, cleaned and analyzed. Sharing Excel spreadsheets in Google Docs, DropBox, or any of the other cloud-type applications are also user-friendly and efficient ways to enter and store your data. Excel spreadsheets in particular are useful for entering data that will further be analyzed. Most online data entry programs are simple for the user to learn, use and share with.

Like anything, there are also more sophisticated programs that make data cleansing and analysis much easier, and therefore entering data directly into these is more efficient.

Managing your Online Database

The other common mode of data entry is entering data directly into a company database. This is commonly used in large companies to keep contact information for their clients or suppliers, or even staff files.

There are many database software options available in the market. Choosing the best one for you depends upon the needs of your company. If you are outsourcing your online data entry, it is always worth asking about the database software the data management company uses.

Database software and programs come in all different types and can be compatible with many different operating systems. Mainframe databases are generally larger and used by big companies generating vast amounts of data. Smaller companies or people on the go often choose to house their database on a cloud or desktop. If you are creating the database for your own company, there are a host of options available online to buy and get you started. If you are outsourcing your data management, communicate your database needs to your data management company so that they work with you to find the best program for you.

There are also programs out there that let you build and manage your own custom database, which may be a more cost effective route for your company.

Whether you are entering data to be cleaned and analyzed, or entering data that will be stored in a database, there are myriad programs on the market designed to help you manage your data easily and effectively.

Source: http://www.dataentryoutsourced.com/blog/top-programs-for-online-data-entry/

Dice Scraping LinkedIn for Post-Mortem Recruiter Spam

If you know me, you know that I’m not a big fan of recruiters. Particularly, recruiters who take the shotgun approach to finding candidates by sending the same copied-and-pasted email to hundreds of potential applicants. I know that these are copied-and-pasted, because my various email accounts will get the exact same email sent minutes apart.

recruiter-1 recruiter-2

I was getting some more recruiter spam today, so I asked the recruiter where he had gotten my information from (since I had deleted my LinkedIn account a week earlier). He send me a screenshot of this page on Dice:

dice-scraper

Dice used to be a huge name in the hiring market. And I suppose it still is (look at all those tabs!) but it has since fallen in popularity thanks to LinkedIn. Anyway, turns out Dice scrapes popular social media networks, runs some heuristics on them, and figures out which profiles on various sites belong together (many of the links between sites could have been determined based on what I had entered into the sites, but not all). Most of the information above came from my LinkedIn profile, and was cached (e.g. scraped and stored in their database) since the LinkedIn profile no longer exists.

I’m fine with people being able to find my information on various sites, but not really a big fan of Dice tying all this stuff together (also, not sure whose MySpace profile my account is linked to on Dice…). I figured the next thing to do would be to ask Dice to delete this page about me. I sure didn’t ask Dice to aggregate this data on my behalf.

dice-remove

It’s almost creepy if you think about it, sites cyber-stalking you and aggregating it in once place. What if this site had information about forum posts and buying habits and dating site profiles of mine?

I haven’t heard back from Dice yet, but we’ll see.

Source:https://thomashunter.name/blog/dice-scraping-linkedin-for-post-mortem-recruiter-spam/

Social Engineering – Scraping Data from Linkedin

Foundation:

I was performing a relatively unique task for a social engineering engagement for a client. Normally I’ll just receive a list of email accounts and/or phone numbers of specific users the client wishes to test. In this case they didn’t want to provide ANY information at all. They wanted to see what I would be able to find and then target those users.

I started with the usual google searches looking for pertinent data and found a little. Used metagoofil and theHarvester as well, which turned up about 20 valid accounts. During my googling I found a very interesting portal page that allowed users to reset their domain passwords. I wasn’t interested in brute forcing any accounts (yet), but was able to use the functionality to test for valid accounts. I browsed to a webpage detailing some of the executives at the company and tried varying combinations of their names to find the format they used to create accounts. It turned out to be first initial last name, not surprisingly.

I then turned my attention to Linkedin and found over 1800 existing employees. If I could just grab all the names of employees and then format them I could then fire this list of usernames at the portal page to get a large list of valid user accounts. How best to do this?

Unfortunately Linkedin is one of the worst designed websites for automating this. If I were able to change the number of results per page I could simply do this manually and it wouldn’t take long. For example if I could return 100 results per page that would only be 18 pages to save manually and then grep out the profile names. That wouldn’t be so bad to save 18 pages manually. Unfortunately Linkedin has it hardcoded that you can only view 10 results per page. It looks like the limit might be 25 results for the Linkedin API, but the actual website appears to be limited to 10 per page. That means I’d have to browse to and save 180 pages manually, too much work. Thus trying to automate it with a script to crawl through each page, saving the output, looked like the best option.

To do this I used the intruder module of Burp Suite. I also needed a paid account for Linkedin, otherwise you would just see their first name and last initial. I borrowed an account (legitimately) from a friend and logged into Linkedin. This captured the request using the proxy intercept feature. I found the request for the search results page in history, right clicked and chose ‘send to intruder’.

On the positions tab for intruder you can see the HTTP request from the client. There are many variables as part of this GET request so the first step is to remove all of them with the ‘Clear §’ button. This removes all the variables that intruder will manipulate. Next select the page_num variable and select the ‘Add §’button.

Note that I changed the variable Keyword=ORG_NAME to protect the client, in reality it was just the organizations name. The attack type doesn’t necessarily matter for this test because we’re only manipulating a single variable, for the difference between the attack types check out the portswigger website.

Now select the payloads tab and choose numbers in the payload set dropdown. This section is pretty self explanatory. We want the page numbers to walk through every number from 1 to 180 and the step defines how much it increments each time. Once you’re ready click Intruder -> Start Attack.

Once the attack has completed you can highlight all the requests, right click and choose ‘save selected items’. Choose a folder and all the contents of the requests will be saved in one file. This works perfectly for what we’re trying to do as we can simply grep out the first and last name.

There were a few locations in the html file that had the persons first and last name. The one that seemed like it would be the easiest to manipulate were part of a link that looked like this

<a href="http://www.linkedin.com/people/invite?from=profile&key=2540332&firstName=Tyler&lastName=Wrightson&amp&#8230;

So to grab all the names I grepped for that page with the command;

grep "people\/invite" burpsuite.txt | cut -d";" -f3,4 >> names.txt

Then I put them in a nice format using search and replace in vi
vi
%s/firstName=//g
%s/&lastName=/ /g
%s/&amp//g

I also noticed that some usernames had plus and percent sign symbols for additional information like using a nickname in quotes for their title. Because there were only a few I went through and manually removed them to make sure that it didn’t interfere with their last name.

Now we have a perfectly constructed list of first name and last name separated by a space.

#grab first initials with
cat names.txt | cut -c1 >> first_initial.txt
#grab last names with
cat names.txt | cut -d " " -f2 >> last_names.txt

Because there was an even amount of items I simply pasted each of these files manually into an excel spreadsheet, saved the spreadsheet as a csv and then I could manipulate them as needed. In this case I simply did a search and replace on the comma to replace with nothing and voila, 485 account names to test in a perfect newline delimited file.

Next I created a perl script to use each name in the file to construct a large post data file to be used by curl. The name of the file was $USERNAME.post

I then used curl to hit the password page with each of the usernames like so

curl --data $_.post https://client.com/reset_password.aspx >> $_.response\n";

*Note that this is also not the actual name of the client’s password reset page.

This took the data in the USERNAME.post file, sent the request to the password reset page and saved the output as USERNAME.response. Thus I could then grep for a string that indicated a correct login in all of the .response files and grep would give me the name of the user that was valid as part of the file name. I did this in part because the username was not spit back at the user in the html of the page.

When this was all complete I had 110 valid domain usernames! Not bad for a start. I then wanted to remove those legitimate names from the larger list so that I could target the valid credentials differently than the potentially valid account names.

Source:http://twrightson.wordpress.com/2012/08/05/social-engineering-scraping-data-from-linkedin/

Yellow pages scraper for USA and Canada Shareware

Our yellow pages scrapers are versatile, lightweight and powerful YP extractor tools. Our programs are designed to extract business information from the yellow pages directory with the latest and up to date information as per your customized requirement. Currently we are offering yellow pages scraper for 3 different countries. Each country are distribute with seperate installer. The software is available on many editions (USA, Canada, USAand Canada and Australia alone).

Yellow pages extarctors features

- Number of records you can extract (Unlimited)
- Search/ Extract by keyword (One or many keywords)
- Search/ Extract by URL (One or many URLs)
- Search by keyword / by categories / by custom categories
- Support Proxy (Hide your IP Address)
- Save search results (CSV, Excel, html, PDF)
- Support many types of output format

*If you need other features to satisfy your requirements. Please contact us

Source:http://www.soft2download.in/yellow-pages-scraper-for-usa-and-canada-1-0-0-4/

How to scrape Yellow Pages with ScreenScraper Chrome Extension

Recently I was asked to help with the job of scraping company information from the Yellow Pages website using the ScreenScraper Chrome Extension. After working with this simple scraper, I decided to create a tutorial on how to use this Google Chrome Extension for scraping pages similar to this one. Hopefully, it will be useful to many of you.

1. Install the Chrome Extension

You can get the extension here. After installation you should see a small monitor icon in the top right corner of your Chrome browser.

2. Open the source page

Let’s open the page from which you want to scrape the company information:

Yellow Pages for Scraping

3. Determine the parent element (row)

The first thing you need to do for the scraping is to determine which HTML element will be the parent element. A parent element is the smallest HTML element that contains all the information items you need to scrape (in our case they are Company Name, Company Address and Contact Phone).  To some extent a parent element defines a data row in the resulting table.

To determine it, open Google Chrome Developer Tools (by pressing Ctrl+Shift+I), click the magnifying class (at the bottom of the window) and select the parent element on the page. I selected this one:

Selecting the parent element

As soon as you have selected it, look into the developer tools window and you will see the HTML code related to this element:

The parent element HTML code

As is seen from the highlighted HTML line, you can easily define a parent element by its class: listingInfoAndLogo.

5. Determine the information elements (columns)

After you have learned how to determine the parent element, it should be easy to specify the information elements that contain the information you want to scrape (they represent columns in the resultant table).

Just do this in the same way that you did it for the parent element -  by selecting it on the page:

The data element

and looking at the highlighted HTML code below:

The data element HTML code

As you can see, the company name is defined by businessName class.

6. Tune the ScreenScraper itself

After all the data elements you want to scrape are found, open the ScreenScraper by clicking the small monitor icon in the top-right corner of your browser. Then do the following:

    Enter the parent element class name (listingInfoAndLogo in our case) into the Selector field, preceding it with a dot (*see below for why)

    Click the Add Column button

    Enter a field’s name (any) into the Field text box

    Enter the information item class into the Selector text box, preceding it with a dot

    Repeat steps 2-4 for each information item element you want to be scraped

*You need to put a dot before the class name because the ScreenScraper requires element definition in CSS Selector format only (with a dot before it)

After you enter all these definitions you should see the preview of the scraped data at the bottom of the extension’s window:

ScreenScraper Chrome Extension window

If the result is satisfactory you can download it in JSON or CSV format by pressing the corresponding button.

That’s it! I hope the tutorial is clear enough. But if not, feel free to write your comments below and I’ll give additional explanations.

Source:http://scraping.pro/how-to-scrape-yellow-pages-with-screenscraper-chrome-extension/

PHP for Web scraping and bot development

Web scraping is a computer science technique for extracting information and data from web sites. In data mining research scraping and analysing of information is discussed. Practically web scraping is necessary if you want to develop a web application where you want to show customised information from various websites.  For this you’ve to first scrap data from the sites and then apply some logic to filter the information.

Practically you can use different languages to write the program that will automatically search and collect the information. But if you’re PHP experts and want to use PHP for this kind of stuff here I am referring a book with PHP library. Practically I found this book is very helpful to learn the topic and their library is easy to use.

Source: http://scrapingwebsite.blogspot.in/2013/05/php-for-web-scraping-and-bot-development.html

PHP HTTP Screen-Scraping Class with Caching

Class_http.php is a "screen-scraping" utility that makes it easy to scrape content and cache scraped content for any number of seconds desired before hitting the live source again. Caching makes you a good neighbor!

The class has 2 static methods that make it easy to extract individual tables of data out of web pages. The class even comes with a companion script that makes it easy to use and cache external images directly within img elements.

The class cloaks itself as the User Agent of the user making the request to your script. It also sends your script as the Referer, since in essence, it is the referrer. This means you should be able to screen-scrape sites that normally block screen-scraping. This class is not meant to help you break any company's usage policies. Be a good neighbor, and always use caching when you can.

Need to access protected content? The class can do basic authentication. However, a lot of sites that require login do not use basic authentication.

Most current information and documentation and downloads found at
http://www.troywolf.com/articles/php/class_http.

There are three complete PHP files listed below. First is the class file, class_http.php. The second is example.php to show you how to use the class. The third file is image_cache.php--a companion script to cache images for use within the src attribute of img elements.

Troy Wolf operates ShinySolutions Webhosting, and is the author of SnippetEdit--a PHP application providing browser-based website editing that even non-technical people can use. "Website editing as easy as it gets." Troy has been a professional Internet and database application developer for over 10 years. He has many years' experience with ASP, VBScript, PHP, Javascript, DHTML, CSS, SQL, and XML on Windows and Linux platforms.

Source: http://scrapingwebsite.blogspot.in/2013/05/php-http-screen-scraping-class-with.html