TransWikia.com

Unable to extract table content from webpage using selenium

Software Quality Assurance & Testing Asked on October 25, 2021

I am trying to extract the tables generated by selecting "Branches", a city and a district from this site: https://www.acb.com.vn/wps/portal/en/atm

So far, I have been able to write the code to parse through each city and district:

from selenium.webdriver.support.ui import Select
from selenium.webdriver import Chrome
import pandas as pd
import time

webdriver = "chromedriver.exe"

driver = Chrome(webdriver)
driver.get('https://www.acb.com.vn/wps/portal/en/atm')
    
branch_selector = driver.find_element_by_xpath('//*[@id="branch"]')
branch_selector.click()

city = Select(driver.find_element_by_id('cityId'))

for i in range(len(city.options)):
    city.select_by_index(i)
    time.sleep(1)
    
    district = Select(driver.find_element_by_id('districtId'))

    for j in range(len(district.options)):
        district.select_by_index(j)
        time.sleep(1)

        try:
            find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
            find_btn.click()
            time.sleep(1)
            
        except:
            close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
            close_btn.click()
            time.sleep(1)

Now, I want to extract the table that’s displayed in each iteration of the 2 loops. However, if you look at the HTML for the table, it does not make use of the "table" tag:

enter image description here

So, how do I extract the table for each city-district pair?

I tried the following:

    try:
        click_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        click_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            print ([r.text for r in row])
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But it prints a list of blank elements for each city-district pair, the length of the list being as many addresses are present in the table for the corresponding city-district pair:

['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '']
['', '']
['', '']

I also tried to access each element in each row of the table individually:

    try:
        find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        find_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            
            for element in row:
                time.sleep(1)
                
                Type.append(element.find_element_by_class_name('col type'))
                Address.append(element.find_element_by_class_name('col address'))
                District.append(element.find_element_by_class_name('col district'))
                Tel_Fax.append(element.find_element_by_class_name('col tel-fax'))
                Hours.append(element.find_element_by_class_name('col hours'))
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But this gives the following error:

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
<ipython-input-41-2d73f0dc931c> in <module>
     39 
---> 40                     Type.append(element.find_element_by_class_name('col type'))
     41                     Address.append(element.find_element_by_class_name('col address'))

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="col type"]"}

Since it says css selector in the error, I tried the following:

element.find_element_by_css_selector('div.col.type').text

This outputs a blank string, ''.

So, how do I do this?

EDIT: The HTML of the table, for one district-city selection, is:

        <div class="tbody">
            
            <div class="row" id="row1">
                <div class="col stt">1</div>
                <div class="col type">
                
                PGD Hai Bà Trưng</div>
                <div class="col address">56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 6291 3690<br>(028) 6291 3691</div>
                <div class="col hours">      07:00-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh', '10.77714,106.704325', 1); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row2">
                <div class="col stt">2</div>
                <div class="col type">
                
                PGD Đa Kao</div>
                <div class="col address">45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 6290 5980<br>(028) 6290 5981</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh', '10.790715,106.69486', 2); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row3">
                <div class="col stt">3</div>
                <div class="col type">
                
                PGD Nguyễn Công Trứ</div>
                <div class="col address">74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3914 4470 <br>(028) 3914 4471</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh', '10.76972,106.703142', 3); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row4">
                <div class="col stt">4</div>
                <div class="col type">
                
                PGD Lê Lợi</div>
                <div class="col address">72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3821 4619<br>(028) 3821 4618</div>
                <div class="col hours">    07:00-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh', '10.773541,106.699635', 4); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row5">
                <div class="col stt">5</div>
                <div class="col type">
                
                CN Sài Gòn</div>
                <div class="col address">41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3824 3770<br>(028) 3824 3946</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh', '10.786191,106.697818', 5); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row6">
                <div class="col stt">6</div>
                <div class="col type">
                
                PGD Nguyễn Thái Bình</div>
                <div class="col address">176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3915 1310<br>(028) 3915 1311</div>
                <div class="col hours">  07:30 – 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh', '10.768917,106.696863', 6); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row7">
                <div class="col stt">7</div>
                <div class="col type">
                
                PGD Bến Chương Dương</div>
                <div class="col address">328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3837 0586<br>(028) 3837 0584</div>
                <div class="col hours">   7h30-16h30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM', '10.76161,106.695998', 7); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row8">
                <div class="col stt">8</div>
                <div class="col type">
                
                PGD Trần Khắc Chân</div>
                <div class="col address">48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3820 9990<br>(028) 3526 7738</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM', '10.790724, 106.690976', 8); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row9">
                <div class="col stt">9</div>
                <div class="col type">
                
                PGD Cống Quỳnh</div>
                <div class="col address">106  108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 38385464<br>(028) 3925 6645</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('106  108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1', '10.764772,106.687505', 9); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row10">
                <div class="col stt">10</div>
                <div class="col type">
                
                CN Bến Thành</div>
                <div class="col address">96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3825 7949<br>(028) 3825 7950</div>
                <div class="col hours"> 07:30-16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh', '10.774379, 106.697395', 10); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row11">
                <div class="col stt">11</div>
                <div class="col type">
                
                PGD Tân Định   </div>
                <div class="col address">261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 3848 0520<br></div>
                <div class="col hours"> 07:30 - 16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM', '10.791284, 106.688080', 11); return false;">Direction</a></div>
            </div>
            
            <div class="row" id="row12">
                <div class="col stt">12</div>
                <div class="col type">
                
                PGD Nguyễn Du</div>
                <div class="col address">Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh</div>
                <div class="col district">1</div>
                <div class="col tel-fax">(028) 35218626<br>(028) 35218627</div>
                <div class="col hours"> 07:30 -16:30</div>
                <div class="col control"><a href="#" title="Direction" class="btn-direction" onclick="showDialogDirection('Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh', '10.777328,106.698459', 12); return false;">Direction</a></div>
            </div>
            
        </div>

One Answer

Your problem might be sames as https://stackoverflow.com/questions/20888592/gettext-method-of-selenium-chrome-driver-sometimes-returns-an-empty-string

But upon initial checking on the attributes of your element, I would suggest to use "outerText" attribute. You can check your element's properties in developer's tool Console by typing $x("your_xpath").

enter image description here

Answered by Millie Anne Volante on October 25, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP