When I first heard about Share Cluj, the project was yet an idea, the team was still discussing about the purpose of the project, thinking how to involve more people and how to put Cluj on world map. Initially, my role in the project was to be an ambassador but, by participating to a team meeting having as main subject the technical part, I immediately became a member of the team.
The technical side was to build a platform to support the contest within the project: any person, from Cluj or not, should promote Cluj while traveling for business or pleasure by sharing their stories about Cluj to those who don’t know the city yet, and then upload on the social networks the “proof”: a photo with the message “#sharecluj” at the location. The technical challenge I saw was the aggregation of the Facebook, Twitter, Instagram feeds based on the “#sharecluj” hashtag, and then present them on the project site: www.sharecluj.com.
I wasn’t too thrilled when I’ve discovered that there are already sites/platforms which can do the targeted aggregation, but at least I had the opportunity to study them and pick the most suitable to the project needs: one that exposes an API which contains the feed with the contestants’ photos from the social networks. I’ve analyzed the offerings of Rignite, Hashtagr, TINT, TagBord; none of these were exposing the API I wished for, nevertheless, being paid services, I assumed that I can discuss with the people maintaining the platforms and we can figure out a solution to integrate as fast as possible, and launch the project as soon as possible.
It was already June - we were aiming to start the contest by July. While waiting for answers from Rignite and TINT, I started doing tests for certain hashtags I kept on seeing in my Facebook feed. The things were not great at all, I was hardly finding anything of what I was expecting, and under any circumstances I couldn’t find all the content - how could you organize a contest when some contestants are registered and others aren’t? Therefore, besides the questionable integration with the site sharecluj.com, another, more difficult problem arose: we couldn’t use any of the existent services, thus I had to make everything from ground zero!
When the deadline was getting closer, and the technical side was far from being solved, you can imagine that I wasn’t too happy or too confident that I would be able to finish everything in such a short term. Sad as I was, I went to work, and distrustfully, I’ve asked my colleagues, Calin, Dan, and Razvan, if they care to do some technical volunteering. I don’t know if my mood convinced them or if they saw the same challenge I saw initially, but in only 10 minutes we had setup a meeting at Dan’s place, with a glass of wine, to make a plan on how to implement sharecluj.com. By the end of the day we got also the functional specifications from Alina, who, from the project manager, has transformed instantly into a product owner.
Although the temptation to experiment with new technologies is big when you have at hand a green field project, we quickly decided to use Java as base-technology, and known frameworks: Spring MVC, JDO. The cost of the project needed to be small, so it was off the table to rent a server, and we’ve decided to use something new: Google AppEngine; with a free quota big enough for data storage, database operations, instance-hours, indexes, deploys, etc., AppEngine proved to be the best for the project. The integration with the social networks was supposed to be done by writing clients for the APIs exposed by each of the networks to get the posts’ metadata and store them in a database.
To make sure that we were going to finish everything that our product owner, Alina, had requested us, we needed to move the deadline towards the beginning of August. The implementation of each point was done based on priority, according to the risk we thought it can represent.
Web applications based on Java and Spring MVC were nothing new for the freshly-formed team, so the first step was to investigate the GraphAPI from Facebook - Facebook, being the most used social network in Romania (thus the most relevant), but also the least present platform while testing the hashtag aggregators, was presenting the biggest risk. While there was the possibility to search in the application by a hashtag, I had high expectations from GraphAPI, to offer me exactly what I wished for: the posts with the hashtag “#sharecluj” – no chance of that! I learnt about the alternative to use the search API for keywords by using the hashtag; the search was working by bizarre rules: some posts we showing up, others weren’t. I made experiments on my personal account: I posted test messages containing #sharecity (I didn’t want to ruin the release); I’ve waited a day, a week, nothing; besides, in a short time the API was becoming obsolete and retired (it’s no longer available at the moment). While searching for explanations, I stumbled upon the FAQ section of TINT where it was explained that in order to have the entire feed from Facebook a workaround was required: either post on a Facebook page or tag a user on the photo. Not the most professional solution, but we all agree that it was the only one.
The good news from Facebook was that for authentication they had implemented OAuth2 and it was also working offline, from a cron job - as we were planning. On top of GraphAPI, plenty of clients were written, from which we have chosen RestFB - easy to use and extensible.
After the experience with Facebook, the general expectations towards any API were low; however, Twitter (the next one on the risk scale) had what we wished for: a simple API, based on hashtags. According to the description of the endpoint, Twitter was not guaranteeing that all the tweets with a certain hashtag will be returned as there is an automatic filtering based on popularity/relevancy; the alternative is to connect and listen to a public stream. In this case, another difficulty has appeared: if we were going with the stream, then another AppEngine instance was required - which would have passed the limits of the quota. Taking into account the costs, but also a possible consistency in procedure, we chose to use the mentions API (one user mentioned it in a tweet). Fortunately, the integration was simple: we used a Java client for the Twitter API: Twitter4j.
Instagram was integrated later than the other networks, and the solutions did not require any concessions. I was happy to notice that Instagram has an API per hashtag, unfiltered, the only limitations being on the number of retrieved results - at each call, the newest results are retrieved, and the full state of the post is not held (through the API). For Instagram API, there aren’t too many clients, but the one picked, jInstagram, was good enough. About authentication, Instagram also implements OAuth2, but it cannot be used offline (for a cron job), but exact the used API (tags/{tag}/media/recent) can be used only with a client_id.
I very much enjoyed to get acquainted with the whole Java AppEngine ecosystem - I was able to use all the frameworks I wanted, without any issues. Firstly, I didn’t had to worry to have the code public before it was valuable - to each application there is a Git repo, you cannot clone it directly, but via the SDK. Then, for Maven integration, there is a management plugin, and also an archetype for web projects. For the database integration, you can JPA or JDO (I chose JDO). Also, the support for cron jobs is included, having as prerequisites the existence of a REST service and of a configuration file.
The web application built with Spring MVC exposes REST services both for getting data from the social networks, and also to make access to the data for the web interface. All the shown data is brought asynchronously, via Ajax calls.
After all the bad surprises from unexpected places and the help from respectable tools, we got to the following design, which was released (successfully!) on the 4th of August, exactly on time.
Immediately after the release, almost the entire development team went on holiday. Nothing bad happened; the application was stable (although no official tester was in the team). However, in time, Facebook hit again! While the project was getting more known, and photos with “#sharecluj” were uploaded in different manners, I’ve discovered more problems and lacunas in Facebook’s API: some photos, although they were following the entire (complicated) procedure, for no reason, weren’t retrieved by the service. I have declared a bug in August which, only at the end of October, was solved, but not completely, there are still some contestants not registered - thus, I’ve declared a new bug.
After three months since the project was released, the application is still stable. Of course, minor adjustments were done in order to correct some glitches or to add more information. It was, and it will continue to be until October 2015 a very interesting experience; I never thought I’ll get to know so well the social networks, or that I would declare defects at Facebook, or that I would do deployments to production from my couch at home.
To sum up, I’m happy that I have contributed to the Share Cluj project and that we have managed to put Cluj on the map in 36 countries already, some really exotic: Senegal, Singapore, Puerto Rico, Australia. Let’s all continue to “Share Cluj”!
by Mircea Vădan
by Călin Lupo
