A community in which webmasters can ask for help with topics such as PHP coding , MySQL , IT jobs, web design, IT security.
Current location:homephp forumphp talk in 2009 yearScreenscraping the ugliest HTML you've ever seen in your life - page 1
User InfoPosts
Screenscraping the ugliest HTML you've ever seen in your life#1
I(m using PHP and libtidy to attempt to screen scrape what might possibly be the most horrendous and malformed use of HTML tables in history. The site closes few table, tr, td, font, or bold tags and consistently nests many different layers of tables within tables.

Example snippet:

<center>
<table border="1" bordercolor="#000000" cellspacing="0" cellpadding="0">
<tr>
<td width="50%">
<center>
Home Team - <b>Wildcats<td>
<center>
Away Team - <b>Polar Bears<tr>
<td colspan="2">
<center>
<b><font size="+1">Rosters<tr>
<td valign="top">
<center>
<table border="0" cellspacing="0">
<tr>
<td>
<font size="2">1&nbsp;<td>
<font size="2">Baird, T<tr>
<td>
<font size="2">2&nbsp;<td>
<font size="2">Knight, P<tr>
<td>
<font size="2">8&nbsp;<td>
<font size="2">Miller, B<tr>
<td>
<font size="2">9&nbsp;<td>
<font size="2">Huebsch, B<tr>
<td>
<font size="2">11&nbsp;<td>
<font size="2">Buschmann, C<tr>
<td>
<font size="2">12&nbsp;<td>
<font size="2">Reding, J<tr>
<td>
<font size="2">14&nbsp;<td>
<font size="2">Simpson, S<tr>
<td>
<font size="2">27&nbsp;<td>
<font size="2">Kupferschmidt, M<tr>
<td>
<font size="2">28&nbsp;<td>
<font size="2">Anderson, D<tr>
<td>
<font size="2">31&nbsp;<td>
<font size="2">Gehrts, J<tr>
<td>
<font size="2">39&nbsp;<td>
<font size="2">McGinnis, G<tr>
<td>
<font size="2">42&nbsp;<td>
<font size="2">Temple, B<tr>
<td>
<font size="2">44&nbsp;<td>
<font size="2">Kemplin, A<tr>
<td>
<font size="2">77&nbsp;<td>
<font size="2">Weiner, B<tr>
<td>
<font size="2">95&nbsp;<td>
<font size="2">
Zytkoskie, D</table>
<td valign="top">
<center>
<table border="0" cellspacing="0">
<tr>
<td>
<font size="2">5&nbsp;<td>
<font size="2">Mack, A<tr>
<td>
<font size="2">8&nbsp;<td>
<font size="2">Foucault, R<tr>
<td>
<font size="2">11&nbsp;<td>
<font size="2">Oberpriller, D *<tr>
<td>
<font size="2">12&nbsp;<td>
<font size="2">Underwood, J<tr>
<td>
<font size="2">15&nbsp;<td>
<font size="2">Oberpriller, M<tr>
<td>
<font size="2">19&nbsp;<td>
<font size="2">Langfus, B<tr>
<td>
<font size="2">25&nbsp;<td>
<font size="2">Carroll, R<tr>
<td>
<font size="2">30&nbsp;<td>
<font size="2">Hirdler, T<tr>
<td>
<font size="2">33&nbsp;<td>
<font size="2">Gibson, S<tr>
<td>
<font size="2">35&nbsp;<td>
<font size="2">Marthaler, C<tr>
<td>
<font size="2">44&nbsp;<td>
<font size="2">Yurik, J<tr>
<td>
<font size="2">58&nbsp;<td>
<font size="2">
Gronemeyer, S</table>
<tr>
<td colspan="2">
<center>
<b><font size="+1">Goals<tr>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<b><font size="2">Period<td>
<b><font size="2">Time<td>
<b><font size="2">Assist 1<td>
<b><font size="2">Assist 2<td>
<b><font size="2">SH<td>
<b><font size="2">PP<tr>
<td nowrap>
<font size="2">Kupferschmidt,&nbsp;M<td>
<font size="2">1<td>
<font size="2">12:51<td nowrap>
<font size="2">Kemplin,&nbsp;A<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">McGinnis,&nbsp;G<td>
<font size="2">1<td>
<font size="2">12:33<td nowrap>
<font size="2">Huebsch,&nbsp;B<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">Kupferschmidt,&nbsp;M<td>
<font size="2">2<td>
<font size="2">16:01<td nowrap>
<font size="2">None<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">Buschmann,&nbsp;C<td>
<font size="2">3<td>
<font size="2">00:38<td nowrap>
<font size="2">None<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
</table>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<b><font size="2">Period<td>
<b><font size="2">Time<td>
<b><font size="2">Assist 1<td>
<b><font size="2">Assist 2<td>
<b><font size="2">SH<td>
<b><font size="2">PP<tr>
<td nowrap>
<font size="2">Oberpriller,&nbsp;D *<td>
<font size="2">3<td>
<font size="2">12:31<td nowrap>
<font size="2">Gronemeyer,&nbsp;S<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
</table>
<tr>
<td colspan="2">
<center>
<b><font size="+1">Penalties<tr>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<font size="2"><b>Period<td>
<font size="2"><b>Minutes<td>
<font size="2"><b>Offense<td>
<font size="2"><b>Start<td>
<font size="2"><b>Expired<tr>
<td nowrap>
<font size="2">Buschmann,&nbsp;C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">11:11<td>
<font size="2">09:11<tr>
<td nowrap>
<font size="2">Buschmann,&nbsp;C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Unsportmanlike Conduct<td>
<font size="2">03:26<td>
<font size="2">01:26<tr>
<td nowrap>
<font size="2">Bench<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Too Many Men<td>
<font size="2">01:46<td>
<font size="2">
00:00</table>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<font size="2"><b>Period<td>
<font size="2"><b>Minutes<td>
<font size="2"><b>Offense<td>
<font size="2"><b>Start<td>
<font size="2"><b>Expired<tr>
<td nowrap>
<font size="2">Marthaler,&nbsp;C<td>
<font size="2">
<center>
1<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">01:19<td>
<font size="2">16:19<tr>
<td nowrap>
<font size="2">Underwood,&nbsp;J<td>
<font size="2">
<center>
2<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">12:32<td>
<font size="2">10:32<tr>
<td nowrap>
<font size="2">Marthaler,&nbsp;C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">11:39<td>
<font size="2">
09:39</table>
<tr>
<td colspan="2">
<center>
<font size="+1"><b>Goalies<tr>
<td>
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Name<td>
<font size="2"><b>Shots<td>
<font size="2"><b>Goals<tr>
<td>
<font size="2">Baird,&nbsp;T<td>
<font size="2">20<td>
<font size="2">1<tr>
<td>
<font size="2"><b>Open Net<td>
<td>
<font size="2">
0</table>
<td>
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Name<td>
<font size="2"><b>Shots<td>
<font size="2"><b>Goals<tr>
<td>
<font size="2">Hirdler,&nbsp;T<td>
<font size="2">42<td>
<font size="2">


Magically, all browsers seem to render this just fine.
PHPTidy manages to do a good sense of it all, but the tables are nested so deeply and almost randomly that it(s really hard to traverse this using DOM XPath.

Does anyone have any recommendations for other approaches for taking this on?

POST-MORTEM: After way too many Belgian wheat beers and dirtying up my code real good I got great results by removing all tags via strip_tags() except table, tr, and td, then running it through libtidy. It(s now formatted beautifully and very easily traversed. Seems like it just needed a little massaging before sending it in to the parser.

posted date: 2009-04-08 17:54:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#2
I had made out the solution of this problem. click to view my topic...

hope that hepls.

posted date: 2009-04-08 17:54:01


Re: Screenscraping the ugliest HTML you've ever seen in your life#3
Maybe you(d have better luck scraping the results you need using regular expressions instead of parsing it as XML.

posted date: 2009-04-08 17:58:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#4
There(s a few tricks you can use to clean up highly-predictable structures like tables. Before running HTML tidy, you can use Regex or something to search for <tr>(s and <td>(s which are followed by another <tr> or <td>, and insert the corresponding closer immediately before it. There(s some added trickery for accommodating tables inside of a <td> but nothing that(s impossible to handle. Just start by locating the innermost structure and moving outward from there.The real puzzle is things like unclosed <div>(s and <p>(s, which can be much harder to match up with their corresponding (or lacking) closers.

posted date: 2009-04-08 17:59:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#5
If you are open to other languages such as Python, Beautiful Soup is great at reconstructing poorly written HTML. I just tried running your HTML through the following snippet, and it(s now quite readable.#!/usr/bin/env pythonfrom BeautifulSoup import BeautifulSouphtml = "long string of html"soup = BeautifulSoup(html)print soup.prettify()

posted date: 2009-04-08 18:02:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#6
The really sad thing is, I've seen worse!!!

posted date: 2009-04-08 18:03:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#7
Yes - for the record I'm not necessarily dependent upon PHP to get this done.

posted date: 2009-04-08 18:04:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#8
Definitely check out Beautiful Soup then. It's by far the best screen scraper I've used. There's also a Ruby version available, in case you prefer it to Python.

posted date: 2009-04-08 18:11:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#9
Seconded. It's worse when you see code like this with PHP intermixed with it.

posted date: 2009-04-08 18:17:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#10
the SO prettify script can't parse this html

posted date: 2009-04-08 18:19:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#11
If you are looking for data, I would just remove all html and handle it as line-by-line raw input. You can use the strip_tags function.$clean = strip_tags($input);// example: <p>Test paragraph.</p> <a href="#fragment">Other text</a>// returns: Test paragraph. Other text

posted date: 2009-04-08 20:19:00


Re: Screenscraping the ugliest HTML you've ever seen in your life#12
I used xpath with Python(s lxml library to parse IMDB Top 250 page. View the source for yourself to see how bad it is.The following code parses a saved IMDB Top 250 page (top250.html) and stores the extracted information in a sqlite database (top250.db)import sqlite3from lxml import htmltree = html.parse((top250.html()class TopMovie(object): base_xpath = "/html/body/div/div[2]/layer/div[3]/table/tr/td[3]/div/table/tr/td/table/tr[%d]" def __init__(self, num): self.rank = num self.xpath = self.base_xpath % (self.rank + 1) def rating(self): return tree.xpath(self.xpath + (/td[2]/font()[0].text def link(self): return tree.xpath(self.xpath + (/td[3]/font/a()[0].values()[0] def title(self): return tree.xpath(self.xpath + (/td[3]/font()[0].text_content() def votes(self): return tree.xpath(self.xpath + (/td[4]/font()[0].textdef main(): conn = sqlite3.connect((top250.db() conn.execute("""DROP TABLE IF EXISTS movies""") conn.execute(""" CREATE TABLE movies ( id INTEGER PRIMARY KEY, title TEXT, link TEXT, rating TEXT, votes INTEGER )""") for n in xrange(1, 251): m = TopMovie(n) query = r(INSERT INTO movies VALUES (%d, "%s", "%s", "%s", "%s")( \ % (n, m.title(), m.link(), m.rating(), m.votes().replace((,(, (()) conn.execute(query) conn.commit() conn.close()if __name__ == "__main__": main()

posted date: 2009-04-08 20:37:00


select page: « 1 2 »
Copyright ©2008-2017 www.momige.com, all rights reserved.