Deep Dive: Addressing Bot Traffic and Enhancing Conversion Rates

Written By: Hussain Abbas, Director of Technology at Squaredance

‍

Over the last few months, Squaredance has conducted an extensive investigation into why our users have been reporting possible declines in conversion rates. After pouring through thousands of data points and running numerous tests and validations, our inquiry has pinpointed the culprit: inflated leads caused by undetected bot traffic.

‍

Previous Protection Layers

Historically, our primary method to safeguard the integrity of clicks and sales was filtering traffic through user agent signatures. This was particularly applied to platforms like Facebook, which provides documentation on its user agent profile to help platforms like ours to account for bot-based traffic.

However, as we learned, these platforms have updated their operations but not their documentation, rendering our old filters ineffective. After review, however, these outdated methods accounted for a mere 0.5% of our total traffic—a stark underrepresentation of bot activity.

‍

Research and Findings

The process to evaluate any potential concerns around our conversion rate tracking began within our Slack Community, as some valuable members highlighted anomalies related to disproportionate lead reports. This prompted a deeper examination that led us to an astonishing revelation: although Facebook was the initial channel that was flagged, other channels were also sending a significant volume of bot traffic adversely impacting our conversion metrics.

To start, we looked at traffic on Squaredance over the past 6 months, used an actively-maintained, open-source list of crawlers by user agent to block traffic, and evaluated its impact. Our analysis revealed that this method only sidelined 0.5% of traffic, confirming that while there is opportunity in continuing to invest in user agent bot signatures, there can be a much larger impact by coupling that with IP-based bot identification. And given Squaredance’s goal is to increase customer acquisition, this is a real concern for our business and its integrity.

We then looked at end-user sessions based on IP addresses, specifically targeting those with high session counts but resulting in zero sales conversions. This criteria helped us identify that up to ~1% of the traffic we analyzed can be identified as bot traffic.

Surprisingly, the majority of the identified bot traffic originated from tech giants such as Facebook, Google, Amazon, and Microsoft. The distribution of crawler traffic showed a significant increase, particularly from Facebook, which doubled its share since the previous year. This analysis underscores the continuous evolution of bot traffic sources, necessitating ongoing vigilance and adaptation in our traffic quality assurance efforts.

‍

‍

Impact on Our Network

Our network includes a wide variety of partners, and we've found that the issue of bot traffic can affect them differently. One striking instance involves a partner who, in April 2024, encountered a staggering 12% of their total traffic attributed to bots—this segment of traffic is precisely what we will filter out moving forward. Another partner, for example, reported only 0.33% of bot traffic.

‍

It’s also important to highlight that certain partners experience noticeable surges in bot traffic at specific times, significantly affecting short-term conversion rates.

For instance, Partner A observed a dramatic surge of bot traffic of 70.06% from Facebook in a span of three days. While these surges appear significant in short intervals, they are not reflective of CVR in the long run.

Despite this variance, our previous analysis reassures us that bot traffic constitutes around 1% of our total traffic across the network. This minimal percentage affirms our efficiency in capturing and representing authentic user engagement, although, as exemplified, the impact on specific partners can be significant. That’s why we’re refining our detection and filtering techniques. We are confident that our new filtering measures will remove these distortions and provide the most accurate representation of actual traffic to our partners.

‍

Implementation Coming July, 2024

During July, Squaredance will be filtering out bot traffic related to Facebook, Google, Amazon, and Microsoft by implementing dual layers of protection: both IP signature and user agent filtering. Based on our findings, this has the potential to impact ~1% of all reported leads on Squaredance.

All users will receive an email notification about the change before its implementation. Additionally, we will update this blog post with the confirmed date once it has been determined.

As for an existing user’s reporting in Squaredance, past data will not be impacted, but all future data from the implementation date onwards will take these changes into effect. To easily display this difference, we’ve changed the colour of historical traffic from within your Reports tab.

‍

Future Plans

Looking forward, we will be taking a much more proactive approach to monitoring the impact of potential bot traffic and other abnormalities that might be impacting campaign performance. We also want to be more transparent with our reporting of our findings, past and present. Some of the ways we’ll be doing this include:

• Providing public access to our reporting and findings in the future.
• Access to our actively maintained, open-source list of identified crawlers.
• Providing quarterly updates to reporting data on any additional bot-based traffic sources we uncover from our active monitoring.

We are committed to updating our filtration processes to stay ahead of evolving bot tactics, as well as staying on top of broader changes that impact the industry as a whole. Moreover, we are developing live documentation systems to keep our community informed about our methods and results in real time.

I want to extend my gratitude to our vigilant community members who initially spotlighted the discrepancies in lead data. Your observations have been instrumental in spearheading these vital changes. Squaredance is now well-positioned to ensure that our platform reflects true user engagement.

Please know that we always welcome any suggestions or feature requests here. Together, let’s continue to refine our systems to represent and serve our genuine user base faithfully.

As we progress, expect to see more updates on how we are actively filtering out more traffic each quarter, refining our approach to keep pace with the digital landscape's rapid evolution.