Publishers Unblock OpenAI’s Crawler As TikTok’s Parent ByteDance Boasts A 25x Faster Web Scraper

 


OpenAI has been in the spotlight recently, not just for its impressive $6.6 billion funding round, but also for significant changes to its business model. The investment round, backed by major names like Microsoft, NVIDIA, and Thrive Capital, has brought OpenAI's valuation to a staggering $157 billion. Despite these achievements, OpenAI is facing substantial financial losses this year, projected to be over $5 billion. Nevertheless, the company remains confident in its growth, forecasting a remarkable $11 billion in revenue for next year. This growth is expected to come from its AI products, which rely heavily on data gathered from the web.

Central to this data-gathering process is OpenAI’s web crawler, GPTBot, which scrapes online content to train its generative AI models like ChatGPT. While many websites initially resisted this practice by blocking the crawler, recent reports suggest a shift in sentiment. Initially, over 33% of websites had barred GPTBot, but that figure has now dropped to about 25%. Moreover, OpenAI has been able to bring down the block rate among major news outlets from 90% to 50%.

This change can be partially attributed to new partnerships OpenAI has struck with key publishers, including TIME, NewsCorp, Reddit, and Condé Nast. These collaborations suggest a growing willingness among some content providers to work with OpenAI, seeing the potential value in allowing AI to access their content. However, not all publishers have unblocked GPTBot as part of a partnership. For instance, The Onion, a satirical news outlet, unintentionally unblocked the crawler due to a technical glitch. Its CEO, Ben Collins, emphasized that the company had no intention of doing business with OpenAI, referring to its AI tools as a “Plagiarism Machine.”

The dilemma of whether or not to allow web crawlers like GPTBot access to content is a complex one. For AI companies like OpenAI, the data gathered through these crawlers is critical for improving products like ChatGPT. On the other hand, publishers are rightfully concerned about the potential risks to data privacy, content ownership, and copyright infringement. OpenAI’s practices have sparked a larger debate, especially in light of the fact that compliance with robots.txt files—used to block web crawlers—is not legally required.

ByteDance, the parent company of TikTok, has further escalated concerns with its new web scraper, Bytespider. It’s reportedly much faster than GPTBot and ignores robots.txt files entirely. The fact that it can bypass publisher restrictions has caused a stir, particularly given the current tensions between the U.S. and China regarding data security and technology sharing. The growing unease surrounding Chinese companies like ByteDance accessing U.S. data through web scraping tools has raised new questions about how far these technologies should be allowed to go.

In light of these developments, news outlets and publishers face a tough choice. Should they allow AI web crawlers to access their content, potentially benefiting from AI partnerships? Or should they hold firm, insisting on better safeguards and clearer legal frameworks before opening the door to AI companies? The future of AI and content publishing may hinge on the answers to these questions.

Read More

Comments

Post a Comment

Popular posts from this blog

Into The World Of Questionable AI Practices

Image
  Artificial Intelligence (AI) and Generative AI (GenAI) have ushered in a new era of innovation, reshaping industries and captivating the interest of businesses, investors, and consumers. Recent trends in Europe illustrate this shift, as Accel's Euroscape 2024 report reveals that 40% of venture capital funding for cloud startups now targets GenAI companies. While this shift has reduced available funding for other cloud technologies, it has also created new opportunities, revitalizing the tech ecosystem. However, these advancements come with challenges, such as job displacement and the restructuring of traditional roles. Industry leaders, including NVIDIA CEO Jensen Huang, have even suggested that AI could eventually replace human coders, making certain skills obsolete. But beyond the opportunities, concerns around privacy and security are growing. The rise of AI-generated deepfakes, for example, has raised fears about personal safety and identity misuse. Researchers from the Uni...
Read more

Marvel Fusion And CSU Break Ground On $150m Laser Facility

Image
  On October 17, 2024, a significant milestone was achieved in the realm of fusion energy with the groundbreaking of the Advanced Technology Lasers for Applications and Science Facility, or ATLAS, at Colorado State University (CSU). This groundbreaking event followed a major announcement made on August 7, 2023, when German startup Marvel Fusion and CSU unveiled a public-private partnership worth $150 million to establish a state-of-the-art laser facility. Located on CSU's Foothills Campus in Fort Collins, Colorado, the ATLAS facility aims to revolutionize research in fusion energy and various other scientific fields. The ATLAS Facility will be a sophisticated assembly of high-intensity, high-repetition rate lasers designed to fire simultaneously at a fusion target. It is set to utilize three ultra-high-intensity lasers that can deliver an astounding 7 petawatts of power—more than 5,000 times the electrical generation capacity of the entire United States. Each laser will target a f...
Read more

Hevo Data Now Available On Google Cloud Marketplace

Image
Hevo Data , a prominent cloud-based data integration platform, has announced its availability on Google Cloud Marketplace as of October 22, 2024. This development offers organizations a streamlined way to synchronize their data across a wide array of sources into Google Cloud's BigQuery, greatly simplifying data management and analytics processes. With Hevo Data's integration into the Google Cloud ecosystem, businesses can now seamlessly sync data from over 150 pre-built connectors, including popular services like Google Analytics , Google Ads, Google Cloud Storage, and Google Drive, directly into BigQuery. This integration enables Google Cloud users to quickly discover and purchase Hevo Data through Google Cloud Marketplace. Additionally, companies that have unused Google Cloud committed spend can apply it toward acquiring Hevo Data's services, making it easier for them to optimize their existing cloud investments. Manish Jethani, CEO and Co-Founder of Hevo Data, expressed...
Read more