A Step for Open Sharing: AddThis Integrates Web Intents

May 1, 2012 | written by Will Meyer | Using AddThis, What's New

AddThis has long been committed to standardizing the way in which content is shared across the web. Today we’re excited to announce core product support for another emerging building block in the open sharing ecosystem. It’s called Web Intents.

What the heck is it?

Web Intents is an open specification being developed within the W3C process. It allows web pages to more easily and securely invoke external services on behalf of users. In plain English, this spec is a way for a web browser to help users connect to useful services on the web for performing certain common actions — things like save, share, and print.

For example, a web page can indicate that it contains some content a user might want to save. A user’s browser can detect that, help the user choose a suitable service, and then actually save the content. Same goes for printing, even for the most basic type of sharing — sending a URL to some other service.

You can learn more about Web Intents and what they can do at webintents.org or the W3C site.

How does AddThis work with Web Intents?

We’re announcing 3 things today. All of the bits are either already deployed or will be rolling out over the coming days.

1. AddThis sharing tools can invoke Web Intents.

AddThis social tools are the easiest way to integrate sharing functionality into your site. That’s not just true for deploying Pinterest, Facebook, or over 300 other services and custom interfaces — it’s also true for access to basic Web Intents. When a developer includes AddThis in a page, users immediately get access to any intent handlers that are registered with their browser. No handlers registered? That’s ok too, it won’t hurt anything and you’ll still get AddThis-powered sharing and analytics as usual.

2. AddThis makes Web Intents work, even if the browser doesn’t support them.

By adding AddThis to a page, a site developer can feel free to use Web Intents to initiate sharing actions. AddThis makes sure the handlers are there, and lets users invoke the tools and share content, even without any browser support. Site developers can relax and not worry about figuring out a complex matrix of browser support for their pages.

3. AddThis sharing tools can handle sharing actions.

If a page includes a share intent, AddThis can handle it. End users can share content to any of our 300+ services. AddThis ensures that web intents retains all the flexibility of current sharing tools from day one.

What next?

We’re extremely excited to help take another step toward open sharing by supporting and participating in the work on this emerging specification. We’ll be continuing to add support to our products and talking more about implementation details here and in our docs. We’ll also be talking quite a bit more about Web Intents to users, site operators, and the over 300 service providers we’ve integrated with across the globe. We’ll be releasing tools to make Web Intents support easier and more valuable for all of them.

The more progress the web is able to make on the mechanics of sending content from point A to point B, the more AddThis can stay focused on the optimization, analytics, and unique social experiences that you’ve come to expect from us and our tools. Check back on this blog to stay up-to-date on our support and ongoing product enhancements around Web Intents.

To the Open Web,

Team AddThis

Text Copy Tracking for All

April 19, 2012 | written by Greg Cypes | Updates

Last month, we turned on Text Copy Tracking for WordPress and Blogger publishers; we will be turning on Text Copy Tracking for all AddThis publishers in the next few weeks. Text Copy Tracking gives you insight into the keywords your audience is copying to their clipboards.

What is Text Copy Tracking?
Text Copy Tracking captures the text users copy to their clipboards from your site. We do not track what users do with these copied keywords, but it is likely that many are put into a search engine. These keywords should give you a better idea of your audience’s specific interests and reveal opportunities to keep users engaged with your site. For privacy reasons, we omit any copies that appear to be a URL, an email address, or a phone number. You can use it on any page, but it will only be active for site visitors who are using a supported browser.

Do I need to upgrade my AddThis code to get text copy tracking?
No, we will push this change automatically; all you have to do is sit back and start watching your analytics live view. If you want to turn off text copy tracking, you will have to update to the latest AddThis code.

What happens if I want to turn off copy text tracking?
We understand some users may not want this feature–so you can easily disable text copy tracking. To do so, edit your AddThis code to set the data_track_textcopy property to false on the addthis_config object. You can get more details here.

As always feedback is appreciated via comments below.

Probabilistic Counting

March 26, 2012 | written by Matt Abrams | Data Science, Engineering

At Clearspring we like to count things. Counting the number of distinct elements (the cardinality) of a set is challenge when the cardinality is large.

To better understand the challenge of determining the cardinality of large sets let’s imagine that you have a 16 character ID and you’d like to count the number of distinct IDs that you’ve seen in your logs. Here is an example:

4f67bfc603106cb2

These 16 characters represent 128 bits. 65k IDs would require 1 megabyte of space. We receive over 3 billion events per day and each event has an ID. Those IDs require 384,000,000,000 bytes or 45 gigabytes of storage. And that is just the space the ID field requires! To get the cardinality of IDs in our daily events we could take a simplistic approach. The most straightforward idea is to use an in memory hash set that contains the unique list of IDs seen in the input files. Even if we assume only 1 in 3 records is unique the hash set would still take 119 gigs of RAM not including the overhead Java requires to store objects in memory. You would need a machine with several hundred gigs of memory to count distinct elements this way and that is only to count a single day’s unique IDs. We certainly don’t have a single machine with several hundred gigs of free memory sitting around so we need a better solution.

One common approach to this problem is the use of bitmaps. Bitmaps can be used to quickly and accurately get the cardinality of a given input. While bitmaps drastically reduce the space requirements from the naive set implementation described above they are still problematic when the cardinality is very high and/or you have a very large number of different sets to count. If we want to count to one hundred million using a bitmap you will need one hundred million bits or roughly 12 megabytes for each counter. Sparse bitmaps can be compressed in order to gain space efficiency but our bitmaps are not sparse. It is also not uncommon for our jobs to contain hundreds or even thousands of counters so we require a more space efficient solution.

Luckily for us cardinality estimation is a popular area of research. We’ve leveraged this research and implemented, and open sourced, several distributed probabilistic cardinality estimation algorithms. We will not go into the details of how these algorithms work in this post but if you are interested you can find references to the relevant research papers in our GitHub project. In general these algorithms provide a space efficient mechanism for estimating the cardinality of a set but with less than perfect accuracy. They do this by representing the count as a set of bit fields and use some hashing function to determine the relevant bucket and bit for the element being added to the set. One algorithm, the LogLog cardinality estimation, is amazingly space efficient. Using LogLog you can count up to 100 million distinct elements with 4% error using just 640 bytes.

So now that we can count the number of distinct elements in huge sets using probabilistic cardinality estimators we have to solve for another problem. Clearspring has close to 2 petabytes of storage spread across several hundred servers. We store partitions of our data on each server so in order to get the cardinality of the full data set we need to combine the results from each of the servers. Let’s look at an example. Imagine that we have 3 servers and the table below represents the cardinality for the data on each of those servers:

Machine	Cardinality
1	1000
2	1500
3	2500

We cannot simply add up the cardinality from each server and say that our total cardinality is 5000. That would assume that the data is partitioned in such a way that guarantees that no ID appears on more than one machine. If that condition is not true, as is often the case, we would be over estimating the cardinality by summing the results. This is where the true power of cardinality estimators really shines through. In addition to being space efficient, the cardinality estimators we use are also mergeable. This means that we can take the cardinality estimate, which is represented as an array of bits, from multiple machines and merge them together to get a cardinality estimate for the global set.

If estimating the cardinality of large sets is a problem you have then we recommend using probabilistic counters and if you happen to use Java then our stream-lib project is great place to start.

Want to work on problems like these? We are hiring.