How S&P is using deep web scraping, ensemble learning and Snowflake architecture to collect 5X more data on SMEs

The investment world has a significant problem when it comes to information on small and medium-sized enterprises (SMEs). This is nothing to do with the quality or accuracy of information – this is the lack of any information.

COME CREDIT Capability Assessment, small enterprise financial information is not open, so it is very difficult to access.

S & P Global Market ZakaS & P has solved the problem of all sides of global and separate credit ratings and criteria, this long time. The company’s technical staff was built RiskyA crawling platform of inconvenient data from more than 200 million websites, processes through a large number of algorithms and creates risk scores.

The platform, built in the snowflake architecture, increased the scope of SME by 5x S & P.

“Our goal was to expand and efficiency,” said Moody Hadi, S & P developing the new product of the s /e s / s. “Project was improved by the accuracy and coverage of information, benefits.”

Basic architecture of RiskGauge

The opposite loan management assesses the credit capacity and risk of a company, including several factors, including financial, standard and risk appetite. The S & P global market intelligence presents these concepts to institutional investors, banks, insurance companies, wealth managers and others.

“Great and financial corporate entities lend to suppliers, but they need to know how much it will lend, how often they watch them, and know what the loan process will be.” “They trust third parties to get to know a valid credit account.”

However, it has long been a gap in SME coverage. Hadi, IBM, Microsoft, Google, Google, Google and relaxation, are required to disclose quarterly financial documents, SMEs do not have this obligation, thus limiting financial transparency. In terms of investor, there are about 10 million flax in the United States compared to about 60,000 public companies.

The S & P global market intelligence has all this covers: the company had about 2 million of the company, but the risk was expanded to 10 million.

The platform, which enters the production of the production in January, is based on a system built by Hadi’s team, which provides solid web content, connects it with an anonymous third party database and machine learning (ML) and applied Detailed algorithms to create credit scores.

The company uses Avalanche Take the company’s pages into the mine pages and then to the firm-fed firmographic drivers (market segments).

The platform information pipeline consists of:

• Crawlers / Web Newspapers
• Pre-processing layer
• Miners
• Curator
• RiskGauge goal

In particular, the Come team uses snowpark container services in the middle of processing, mining and treatment steps before Snowflake’s data warehouse and snowpark container services.

At the end of this process, the SMEs are based on the combination of financial, business and market risk; 1 highest, 100est lowest. Investors also receive reports on the risks of detailed awareness of financial, firms, business loans, historical performance and basic developments. They can also compare companies with their peers.

S & P collects how valuable company data

Case, RickGauge uses a large fracture process that draws various details from the web domain of a company, such as the main “Contact Us” and Landscaping Pages and News. Miners go down to several URL layers to break the relevant information.

“As you can imagine, a person can’t do it,” he said. “It will take a lot of time for a person, especially when you are engaged in 200 million websites.” This is the result of several terabits of website data that he noted.

The next step after the data is collected is to operate algorithms that eliminate everything without text; Come noted that the system is not interested in JavaScript or even HTML labels. The information is being cleaned, so man is unreadable, not a code. Then uploaded Avalanche And several data miners continue against pages.

Ensemble algorithms are important for the forecasting process; These types of algorithms combine several individual models (basic models or weak learners), a few individual models (basic models or weak learners ‘phrases (main models or weak learners’. The system is in any pole factors in any pole.

“After a site crawling, algorithms hit various components of the pages taken and vote and return with recommendations,” he said. “In this process, there is no human being in the loop, algorithms are mainly competing. This helps efficiency to increase our coverage.”

This is followed by the initial load, the system automatically tracks the site. This information about this is not updated on weekly; Come added while revealing only one change. When performing subsequent scans, a hash key follows the previous scanning start page and the system creates another key; If it is the same, no changes were made and no action is required. However, if the hash keys do not match, the system will be launched to update the company’s data.

This continuous scrap is important to ensure that the system is as modern as possible. “If you update the site often, it says they are alive, right?” Is true. “Come noted.

Processing speed, giant information, problems with nasty sites

There were difficulties to eliminate the system, especially during the establishment of a system, especially in the system of database and fast processing. The Come team was forced to trade to balance accuracy and speed.

“We have optimized different algorithms to escape faster,” he said. “And bathe; some algorithms were really good, high accuracy, high accuracy, high accuracy, highly, but very expensive with calculation.”

Websites do not always match standard formats that require flexible itching methods.

“You hear a lot about designing websites like this, because we first thought when we started, ‘Hey, every website must fit the Sitemap or XML.” And guess what? ” No one will follow it. “

The sites did not want to enter the system (RPA) system (RPA) system (RPA) System (RPA), for which they have changed so much, then cleanses the necessary components, then cleans the text and cancel the code and any JavaScript.

Come noted that “the biggest challenges were the cleansing of websites with performance and regulation and design.”