Focused crawling strategies based on ontologies and simulated annealing methods for rainstorm disaster domain knowledge

Jingfa LIU; Fan LI; Ruoyao DING; Zi'ang LIU

doi:10.1631/FITEE.2100360

Your Location：

Home >

Browse articles >

Focused crawling strategies based on ontologies and simulated annealing methods for rainstorm disaster domain knowledge

Regular Papers | Updated：2022-09-08

- Focused crawling strategies based on ontologies and simulated annealing methods for rainstorm disaster domain knowledge
- 基于本体和模拟退火算法的暴雨灾害主题爬虫策略
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 8, Pages: 1189-1204(2022)
- Affiliations：
  
  1.Guangzhou Key Laboratory of Multilingual Intelligent Processing, Guangdong University of Foreign Studies, Guangzhou510006, China
  2.School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou510006, China
  3.School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing210044, China
  4.Faculty of Science, University of Alberta, Edmonton T6G2H6, Canada
- Author bio：
  
  †E-mail: jfliu@nuist.edu.cn;
  ‡Corresponding authors
- Funds：
- DOI：10.1631/FITEE.2100360
  CLC： TP39
- Received：23 July 2021，
  
  Revised：2022-03-23，
  
  Published：23 August 2022
- Accepted：
Scan QR Code
Jingfa LIU, Fan LI, Ruoyao DING, et al. Focused crawling strategies based on ontologies and simulated annealing methods for rainstorm disaster domain knowledge[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(8): 1189-1204.
DOI：

Jingfa LIU, Fan LI, Ruoyao DING, et al. Focused crawling strategies based on ontologies and simulated annealing methods for rainstorm disaster domain knowledge[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(8): 1189-1204. DOI： 10.1631/FITEE.2100360.

摘要

目前，主题爬虫是从海量异构网络中获取有效领域知识的重要方法。目前大多数主题爬虫技术难以获得高质量爬行结果。主要难点包括主题基准模型的建立、超链接主题相关性的评估和爬行策略的设计。本文采用领域本体为特定主题构建主题基准模型，并提出一种新的基于局部本体和全局本体的多重筛选策略（MFSLG）。为提高待访问超链接主题相关性计算精度，提出一种基于网页文本和链接结构的综合优先度评估方法（CPEM），同时，采用模拟退火（SA）算法避免主题爬虫陷入局部最优搜索。本文首次设计融合SA算法、MFSLG策略和CPEM策略实现主题爬虫，提出两种新的基于本体和SA主题爬虫策略（FCOSA），包括基于全局本体的FCOSA策略（FCOSA_G）和基于局部本体和全局本体的FCOSA策略（FCOSA_LG），以从网络中获取与暴雨灾害主题相关的网页。实验结果表明，针对不同性能指标，所提爬虫策略优于其他主题爬虫策略。

Abstract

At present

focused crawler is a crucial method for obtaining effective domain knowledge from massive heterogeneous networks. For most current focused crawling technologies

there are some difficulties in obtaining high-quality crawling results. The main difficulties are the establishment of topic benchmark models

the assessment of topic relevance of hyperlinks

and the design of crawling strategies. In this paper

we use domain ontology to build a topic benchmark model for a specific topic

and propose a novel multiple-filtering strategy based on local ontology and global ontology (MFSLG). A comprehensive priority evaluation method (CPEM) based on the web text and link structure is introduced to improve the computation precision of topic relevance for unvisited hyperlinks

and a simulated annealing (SA) method is used to avoid the focused crawler falling into local optima of the search. By incorporating SA into the focused crawler with MFSLG and CPEM for the first time

two novel focused crawler strategies based on ontology and SA (FCOSA)

including FCOSA with only global ontology (FCOSA_G) and FCOSA with both local ontology and global ontology (FCOSA_LG)

are proposed to obtain topic-relevant webpages about rainstorm disasters from the network. Experimental results show that the proposed crawlers outperform the other focused crawling strategies on different performance metric indices.

关键词

Keywords

references

Bajpai N , Arora D , 2018 . Domain-based search engine evaluation . In: Saeed K , Chaki N , Pati B , et al . (Eds.), Progress in Advanced Computing and Intelligent Engineering. Advances in Intelligent Systems and Computing, volume 564 . Springer , Singapore , p. 711 - 720 . https://doi.org/10.1007/978-981-10-6875-1_69 https://doi.org/10.1007/978-981-10-6875-1_69

Boukadi K , Rekik M , Rekik M , et al. , 2018 . FC4CD: a new SOA-based focused crawler for cloud service discovery . Computing , 100 ( 10 ): 1081 - 1107 . https://doi.org/10.1007/s00607-018-0600-2 https://doi.org/10.1007/s00607-018-0600-2

Capuano A , Rinaldi AM , Russo C , 2020 . An ontology-driven multimedia focused crawler based on linked open data and deep learning techniques . Multim Tools Appl , 79 ( 11 ): 7577 - 7598 . https://doi.org/10.1007/s11042-019-08252-2 https://doi.org/10.1007/s11042-019-08252-2

Chen YB , Zhang Z , Zhang T , 2011 . A searching strategy in topic crawler using ant colony algorithm . Microcomput Appl , 30 ( 1 ): 53 - 56 (in Chinese) . https://doi.org/10.19358/j.issn.1674-7720.2011.01.018 https://doi.org/10.19358/j.issn.1674-7720.2011.01.018

Cheng YK , Liao WJ , Cheng G , 2018 . Strategy of focused crawler with word embedding clustering weighted in shark-search algorithm . Comput Dig Eng , 46 ( 1 ):‍ 144 - 148 (in Chinese) . https://doi.org/10.3969/j.issn.1672-9722.2018.01.031 https://doi.org/10.3969/j.issn.1672-9722.2018.01.031

Colazzo D , Ghelli G , Pardini L , et al. , 2013 . Almost-linear inclusion for XML regular expression types . ACM Trans Database Syst , 38 ( 3 ): 15 . https://doi.org/10.1145/2508020.2508022 https://doi.org/10.1145/2508020.2508022

Derrac J , García S , Molina D , et al. , 2011 . A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms . Swarm Evol Comput , 1 ( 1 ): 3 - 18 . https://doi.org/10.1016/j.swevo.2011.02.002 https://doi.org/10.1016/j.swevo.2011.02.002

Dong Y , Liu JF , Liu WJ , 2020 . Focused crawler strategy based on multi-objective ant colony algorithm . Comput Eng , 46 ( 9 ): 274 - 282 (in Chinese) . https://doi.org/10.19678/j.issn.1000-3428.0055967 https://doi.org/10.19678/j.issn.1000-3428.0055967

Du YJ , Pen QQ , Gao ZQ , 2013 . A topic-specific crawling strategy based on semantics similarity . Data Knowl Eng , 88 : 75 - 93 . https://doi.org/10.1016/j.datak.2013.09.003 https://doi.org/10.1016/j.datak.2013.09.003

Du YJ , Hai YF , Xie CZ , et al. , 2014 . An approach for selecting seed URLs of focused crawler based on user-interest ontology . Appl Soft Comput , 14 : 663 - 676 . https://doi.org/10.1016/j.asoc.2013.09.007 https://doi.org/10.1016/j.asoc.2013.09.007

Du YJ , Li CX , Hu Q , et al. , 2017 . Ranking webpages using a path trust knowledge graph . Neurocomputing , 269 : 58 - 72 . https://doi.org/10.1016/j.neucom.2016.08.142 https://doi.org/10.1016/j.neucom.2016.08.142

Farag MMG , Lee S , Fox EA , 2018 . Focused crawler for events . Int J Dig Libr , 19 ( 1 ): 3 - 19 . https://doi.org/10.1007/s00799-016-0207-1 https://doi.org/10.1007/s00799-016-0207-1

Gruber TR , 1995 . Toward principles for the design of ontologies used for knowledge sharing? Int J Human-Comput Stud , 43 ( 5-6 ): 907 - 928 . https://doi.org/10.1006/ijhc.1995.1081 https://doi.org/10.1006/ijhc.1995.1081

Guan WG , Luo YC , 2016 . Design and implementation of focused crawler based on concept context graph . Comput Eng Des , 37 ( 10 ): 2679 - 2684 (in Chinese) . https://doi.org/10.16208/j.issn1000-7024.2016.10.019 https://doi.org/10.16208/j.issn1000-7024.2016.10.019

He S , Cheng JX , Cai XB , 2009 . Focused crawler based on simulated anneal algorithm . Comput Technol Dev , 19 ( 12 ): 55 - 58, 62 (in Chinese) . https://doi.org/10.3969/j.issn.1673-629X.2009.12.015 https://doi.org/10.3969/j.issn.1673-629X.2009.12.015

Jia JF , Tumanian V , Li GQ , 2021 . Discovering semantically related technical terms and web resources in Q&A discussions . Front Inform Technol Electron Eng , 22 ( 7 ):‍ 969 - 985 . https://doi.org/10.1631/FITEE.2000186 https://doi.org/10.1631/FITEE.2000186

Jing WP , Wang YJ , Dong WW , 2016 . Research on adaptive genetic algorithm in application of focused crawler search strategy . Comput Sci , 43 ( 8 ): 254 - 257 (in Chinese) . https://doi.org/10.11896/j.issn.1002-137X.2016.8.051 https://doi.org/10.11896/j.issn.1002-137X.2016.8.051

Khadir AC , Aliane H , Guessoum A , 2021 . Ontology learning: grand tour and challenges . Comput Sci Rev , 39 : 100339 . https://doi.org/10.1016/j.cosrev.2020.100339 https://doi.org/10.1016/j.cosrev.2020.100339

Lakzaei B , Shamsfard M , 2021 . Ontology learning from relational databases . Inform Sci , 577 : 280 - 297 . https://doi.org/10.1016/j.ins.2021.06.074 https://doi.org/10.1016/j.ins.2021.06.074

Liu B , Jiang SY , Zou Q , 2020 . HITS-PR-HHblits: protein remote homology detection by combining PageRank and hyperlink-induced topic search . Brief Bioinform , 21 ( 1 ): 298 - 308 . https://doi.org/10.1093/bib/bby104 https://doi.org/10.1093/bib/bby104

Liu JF , Li G , Chen DB , et al , 2010 . Two-dimensional equilibrium constraint layout using simulated annealing . Comput Ind Eng , 59 ( 4 ): 530 - 536 . https://doi.org/10.1016/j.cie.2010.06.009 https://doi.org/10.1016/j.cie.2010.06.009

Liu JF , Li F , Jiang SY , 2019a . Focused annealing crawler algorithm for rainstorm disasters based on comprehensive priority and host information . Comput Sci , 46 ( 2 ):‍ 215 - 222 (in Chinese) . https://doi.org/10.11896/j.issn.1002-137X.2019.02.033 https://doi.org/10.11896/j.issn.1002-137X.2019.02.033

Liu JF , Li X , Jiang SY , 2019b . Focused crawler for rainstorm disaster strategy based on web space evolutionary algorithm . Comput Eng , 45 ( 2 ): 184 - 190 (in Chinese) . https://doi.org/10.19678/j.issn.1000-3428.0052035 https://doi.org/10.19678/j.issn.1000-3428.0052035

Liu JF , Gu YP , Liu WJ , 2020 . Focused crawler method combining ontology and improved Tabu search for meteorological disaster . J Comput Appl , 40 ( 8 ):‍ 2255 - 2261 (in Chinese) .

Liu WJ , Du YJ , 2014 . A novel focused crawler based on cell-like membrane computing optimization algorithm . Neurocomputing , 123 : 266 - 280 . https://doi.org/10.1016/j.neucom.2013.06.039 https://doi.org/10.1016/j.neucom.2013.06.039

Patel A , Schmidt N , 2011 . Application of structured document parsing to focused web crawling . Comput Stand Inter , 33 ( 3 ): 325 - 331 . https://doi.org/10.1016/j.csi.2010.08.002 https://doi.org/10.1016/j.csi.2010.08.002

Prakash J , Kumar R , 2015 . Web crawling through shark-search using PageRank . Proc Comput Sci , 48 : 210 - 216 . https://doi.org/10.1016/j.procs.2015.04.172 https://doi.org/10.1016/j.procs.2015.04.172

Rawat S , Patil DR , 2013 . Efficient focused crawling based on best first search . Proc 3 rd IEEE Int Advance Computing Conf , p. 908 - 911 . https://doi.org/10.1109/IAdCC.2013.6514347 https://doi.org/10.1109/IAdCC.2013.6514347

Rios-Alvarado AB , Lopez-Arevalo I , Sosa-Sosa VJ , 2013 . Learning concept hierarchies from textual resources for ontologies construction . Expert Syst Appl , 40 ( 15 ):‍ 5907 - 5915 . https://doi.org/10.1016/j.eswa.2013.05.005 https://doi.org/10.1016/j.eswa.2013.05.005

Tong YL , 2008 . Application of focused crawler using adaptive dynamical evolutional particle swarm optimization . Geom Inform Sci Wuhan Univ , 33 ( 12 ):‍ 1296 - 1299 (in Chinese) .

Tsikrika T , Moumtzidou A , Vrochidis S , et al. , 2016 . Focussed crawling of environmental web resources based on the combination of multimedia evidence . Multim Tools Appl , 75 ( 3 ): 1563 - 1587 . https://doi.org/10.1007/s11042-015-2624-3 https://doi.org/10.1007/s11042-015-2624-3

Vidal MLA , da Silva AS , de Moura ES , et al. , 2006 . Structure-driven crawler generation by example . Proc 29 th Annual Int ACM SIGIR Conf on Research and Development in Information Retrieval , p. 292 - 299 . https://doi.org/10.1145/1148170.1148223 https://doi.org/10.1145/1148170.1148223

Wang ZG , Meng BJ , 2014 . A comparison of approaches to Chinese word segmentation in Hadoop . Proc IEEE Int Conf on Data Mining Workshop , p. 844 - 850 . https://doi.org/10.1109/ICDMW.2014.43 https://doi.org/10.1109/ICDMW.2014.43

Yang YK , Du YJ , Sun JY , et al. , 2008 . A topic-specific web crawler with concept similarity context graph based on FCA . Proc 4 th Int Conf on Intelligent Computing , p.‍ 840 - 847 . https://doi.org/10.1007/978-3-540-85984-0_101 https://doi.org/10.1007/978-3-540-85984-0_101

Zhu G , Yang JY , Wu XH , et al. , 2017 . Research on construction of hierarchy relationship and ontology of meteorological disaster based on FCA . Mod Inform , 37 ( 5 ):‍ 79 - 88 (in Chinese) . https://doi.org/10.3969/j.issn.1008-0821.2017.05.014 https://doi.org/10.3969/j.issn.1008-0821.2017.05.014

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

A new focused crawler using an improved tabu search algorithm incorporating ontology and host information

OntoCSD: an ontology-based security model for an integrated solution of cyberspace defense

Friendship-aware task planning in mobile crowdsourcing

Knowledge extraction from Chinese wiki encyclopedias

Related Author

Zhihe YANG

Guo ZHONG

Zhen WANG

Jingfa LIU

Ke CHEN

Ruiyun XIE

Jie CHEN

Dandan WU

Related Institution

China Unicom Central South Research Institute

School of Computer Science, Chengdu College of University of Electronic Science and Technology of China

School of Cybersecurity, Northwestern Polytechnical University

China Electronics Technology Cyber Security Co., Ltd.

State Key Laboratory of Software Development Environment, School of Computer Science, Beihang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰