How to extract text from HTML using Ruby/Hpricot

To blog Using Gmail at 04/25/2008 04:12 PM

78 views

I found this while solving my own problem. ( That is always a best way to learn).

Requirement: Extract text from HTML body which includes ignoring large white spaces between tags and words.

Solution : Use Hpricot to do the magic

Assumption : Only HTML body is used here

One liner : Hpricot(html).inner_text.gsub("\r"," ").gsub("\n"," ").split(" ").join(" ")

Above line gets the inner text (which is a very convenient method to get the actual meat out of HTML)
Replaces line returns into spaces.
Do a split/join which eliminates multiple spaces between tags and words. This trims it down to single space

Now if want scan thru the whole HTML right from <html> tags, then you will have to strip out script, link, meta, style tags as well. to do that just do following

hpricot = Hpricot(html)

hpricot.search("script").remove

hpricot.search("link").remove

hpricot.search("meta").remove

hpricot.search("style").remove

Possibly related messages

« Previous Message

Next Message »

Tags: ruby ror rails hpricot html parsing

COMMENTS

Close X

Share last 10 tweets from legalizenet

Watching "Little Miss Sunshine" , its a very well made happy-sad movie. Every character is perfectly designed. at 01/05/2009 03:29 AM

[PHOTO] Papaya Tequini @ El Torito http://tinyurl.com/a6fb6x at 01/03/2009 04:01 AM

[PHOTO] Found MessageDance Twitter fatbird in Singapore On my recent trip, Boat Quay waterfront area http://tinyurl.com/ax3vdn at 01/02/2009 01:28 AM

[PHOTO] New Year drinking begins. With Fat tire http://tinyurl.com/9ulzld at 01/01/2009 04:17 AM

Looks like it is going to be a TAKE IT EASY NY this year.. relax, drink with friends. at 12/31/2008 11:40 PM

[PHOTO] At Seoul airport heading to San francisco http://tinyurl.com/7ulyfm at 12/25/2008 08:01 AM

[PHOTO] Airlift - new impressive shared taxi service @ Bangalore airport. http://tinyurl.com/9trus5 at 12/25/2008 07:59 AM

[PHOTO] Funny way to get customers buying Using Recession as a tool http://tinyurl.com/5wz2t9 at 12/16/2008 05:28 PM

[PHOTO] Bangalore's own Metro under construction http://tinyurl.com/64r82g at 12/16/2008 05:25 PM

In Udupi right now. [u] Very refreshing. http://tinyurl.com/5sd38m at 12/09/2008 03:13 PM

Messages Sent	522
Messages Received	635
Comments	164
Preferred Email	Gmail
Preferred Network	http://www...
Friends	47