Under the Hood: The Technical Setup of Upverter

Editor’s note: This is a cross post from the Upverter blog written by Zak Homuth (LinkedIn, @zakhomuth, Github). Follow him on Twitter @zakhomuth. This post was originally published on August 1, 2011, I was just negligent in posting it.

Who doesn’t love tech porn? And what’s better than an inside look at the architecture and tools that power a startup? That’s right, nothing. So we thought, why not put up our own little behind the scenes, and try and share a little bit about how we do what we do?

At Upverter, we’ve built the first ever web-based, the first ever collaborative, and the first ever community and reuse focused EDA tools. This meant re-thinking a lot of assumptions that went into building the existing tools. For example, clients and servers weren’t an afterthought, but instead a core part of our architecture. Collaboration was baked in from the start which also meant a whole new stack – borrowed heavily from guys like Google Wave, and Etherpad.

http://en.wikipedia.org/wiki/Apache_Wave
http://code.google.com/p/etherpad/
http://techblog.gomockingbird.com/archive/5/2010

On the front-end, our pride and joy is what we call the sketch tool. Its more or less where we have spent the bulk of our development time over the last year – a large compiled javascript application that uses long polling to communicate with the API and Design Servers. When we started out to move these tools to the web, we knew that we would be building a big Javascript app. But we didn’t quite know what the app itself would look like and our choice of tech for the app itself has changed quite a bit over time… more on this later!

On the back-end, we run a slew of servers. When it comes to our servers, there was a bit of a grand plan when we started, but in reality they all came about very organically. As we needed to solve new problems and fill voids, we built new servers into the architecture. As it stands right now, we have the following:

Front-end web servers, which serve most of our pages and community content;
API & Design servers, which do most of the heavy lifting and allow for collaboration;
DB servers, which hold the datums; and
Background workers, which handle our background processing and batch jobs.

So let’s talk tech…

We use a lot of Linux (ub) (arch), both on our development workstations and all over our servers.
We use Python on the server side; but when we started out we did take a serious look at using Node.js () and Javascript. But at the time both Node and javascript just wern’t ready yet… But things have come a tremendously long way, and we might have made a different choice if we were beginning today.
We use nginx (http://nginx.org/) for our reverse proxy, load balancing and SSL termination.
We use Flask (http://flask.pocoo.org/) (which is a like Sinatra) for our Community and Front-end web servers. We started with Django, but it was just too full blown and we found ourselves rewriting it enough that it made sense to step a rung lower.
We use Tornado () for our API and design servers. We chose Tornado because it is amazingly good at serving these type of requests at break neck speed.
We built our background workers on Node.js so that we can run copies of the javascript client in the cloud saving us a ton of code duplication.
We do our internal communication through ZMQ (www.zeromq.org) on top of Google Protocol Buffers
Our external communication is also done through our custom RPC javascript again mapped onto Protocol Buffers. http://code.google.com/apis/protocolbuffers/docs/overview.html/
We used MySQL () for both relational and KV data through a set of abstracted custom datastore procedures until very recently, when we switched our KV data over to Kyoto Tycoon ().
Our primary client the sketch tool is built in Javascript with the Google Closure Library () and Compiler ().
The client communicates with the servers via long polling through custom built RPC functions and server-side protocol buffers.
We draw the user interface with HTML5 and canvas (), through a custom drawing library which handles collisions and does damage based redrawing.
And we use soy templates for all of our DOM UI dialogs, prompts, pop-ups, etc.
We host on EC2 and handle our deployment through puppet master ().
Monitoring is done through a collection of OpsView/nagios, PingDom and Collectd.

Our development environment is very much a point of pride for us. We have a spent a lot of time making it possible for us to do some of the things we are trying to do from both the client and server sides and putting together a dev environment that allows our team to work efficiently within our architecture. We value testing, and we are fascists about clean and maintainable code.

We use git (obviously).
We have a headless Javascript unit test infrastructure built on top of QUnit () and Node.js
We have python unit tests built on top of nose ().
We run closure linting () and compiling set to the “CODE FACIEST” mode
We run a full suite of checks within buildbot () on every push to master
We also do code reviews on every push using Rietveld ().
We are 4-3-1 VIM vs. Text Edit vs. Text Mate.
We are 4-2-2 Linux vs. OSX vs. Windows 7.
We are 5-2-1 Android vs. iPhone vs. dumb phone.

If any of this sounds like we are on the right path, you should drop us a line. We are in Toronto, we’re solving very real-world, wicked problems, and we’re always hiring smart developers.

Reference

Under the Hood – Lymbix

We continue the Under the Hood series with a Q&A with Lymbix CTO and Hot Sh!t List member Josh Merchant (@joshmerchant, LinkedIn). (Disclosure: I sit on the Board of Directors for Lymbix and helped them with their application/acceptance to the Microsoft BizSpark One program). Lymbix has raised approximately $3.8MM in funding from GrowthWorks and other angel investors. The Lymbix team is 18 people based in Moncton, NB and continues to grow.

Lymbix Sentiment Intelligence measures the tone and emotional impact of words in everyday written language. As a global leader in sentiment analysis technology, applications powered by Lymbix provide a more definitive look at specific emotions like friendliness, enjoyment, amusement, contentment, sadness, anger, fear, and shame and give insight to the true meaning of what brings positive and negative results. In short, Lymbix delivers incredibly fast sentiment analysis and can identify the real emotion in any domain of text exposing clarity and confidence on an individual message level.

Product Breakdown

An engine that analyzes emotion in text. Simply put, we’ve built an emotional spell check that we call ToneCheck, which looks into the emotions written in email communications, lifting out how someone may feel – or rather, the “tone”, they’ll perceive when they read the message. This technology is built off our core engine, which is available as an API for partners to understand more user expression style sentiment analysis. As a business, Lymbix is building better business communication tools and reporting for companies to analyze communication in sales, human resources, customer support. Think of it like an insurance package fitting nicely into your risk management profile.

How the Technology works

We use an array of techniques to training our systems to better understand the emotional interactions in common day communication. We analyze streams of data, whether it be from Facebook, Twitter, emails, blogs, or the news, and dissect elements of “emotive context”, meaning a snippet of text that can cause an emotional arousal in an individual. This is our linguistics component of our system. We believe in human powered insight, so we then take a slew of emotive context, and blast it through our own crowd-sourced network called ToneADay.com. We have just shy of 10k raters who give us their opinions of both “real” and “fake” emotive context to gauge the levels of emotion that can occur based on parameters such as frequencies, demographics, 8 primary emotions and so forth. We then build emotional lexicons which give us the power to test any incoming queries to detect emotional relevancy. We then apply our “emotional reaction algorithms” to come up with how different emotions play a part in determining the degrees of emotion in the query. When the system ever detects something that it has never heard of it, it quickly takes action and tries to learn it. In effect, the system gets smarter the more that its used.

Technical Details

We’re hosted on Rackspace, as well as Azure. With Rackspace we have a cloud and private hosted solution giving us the elastic scalability that we need to service this type of NLP on a massive scale. We’re a nice blend of Ruby, Java, and C#. Sounds gross, but for us, the solution fits quite nicely.

For horizontal scaling efforts (our API, and freemium ToneCheck users) we use multiple nodes replicated as our “workers”, sitting on Redhat using served by apache. Sinatra is used to handle the REST calls (essentially the wrapper) harnessing java – linking through sockets to provide really fast linguistic calculations on requests. We persist resident data through redis, and pull sync jobs to migrate up to the master datastore. These ‘nodes’ effectively are spawned up and down as we predict traffic congestion. We take full advantage of Rackspace load balancers to handle distribution of these requests. We monitor this bad boy with CloudKick – probably the best monitoring and performance analytics tool we’ve come across.

For ToneCheck (pro/business), we’re deployed on Azure. Works well for our business customers to give better piece of mind of no data persistence, enterprise integration (on a domain level), and security. Essentially we’ve built a RESTful service on a Web role that wraps the same Java logic as in our cloud. We have worker roles to do some of the heavy lifting, but we try to keep things in the Web Role for high priority, super fast response times.

As our system is ever evolving, in terms of understanding new emotive context, we use our own sync services to deploy lexicons across all our worker nodes (Azure & Rackspace). To build the lexicons, we need massive power, so we use a big hypervisor that performs all our “secret sauce” algorithms from our datastore. We have 3 layers of databases in our system, which seems crazy, but each has a niche. MySQL is basic user data for our apps and all the boring data to keep. Mongo is our dynamic datastore thats used for all our linguistic data and everything we need to build our lexicons, which is sharded for optimization and running our Map Reduce jobs. We also keep a Hadoop datastore for all the new language we’re processing for reporting and running massive queries on for some of our “in the making” linguistic calculations/improvements.

Our development practises are pretty neat. We use continuous integration to achieve higher standards of quality for all our apps. We’re a little old school, still using some SVN repos to manage our data (Beanstalk rocks), but now we’re starting to migrate more to the Git. The team is divided up into sub teams, which are all managed independently, and constantly on two week (global) dev cycles. We do all our project management through Pivotal Tracker, and have wicked fun demo days at the end of every cycle showcasing each teams improvements and brainiac innovations to everyone (while consuming beer and pizza). Our team is very passionate about the problem we’re trying to solve, technology, and code. We’re split about 50/50 Android & iPhone, so that pretty much says it all!

If you’re running a mail client (Outlook or GMail or Lotus Notes) you can try ToneCheck and to minimize the “cost” of dealing with misunderstandings.

Interested in being profiled in our Under the Hood series, we are actively looking for Canadian startups building “interesting” technologies and solving “interesting” problems. Contact me by completing your initial Under the Hood submission.

Under the Hood – Chango

I am always curious at what startups are using for the technology under the hood. It helps define the technologies that frontend and backend developers looking for jobs might start using. It helps understand what skills are being developed in the ecosystem and where competition or coopetition for talent may exist. And I am always curious at the emerging trends of technology adoption by local firms.

I have enjoyed the coverage of the technology at Toronto-born startups like BackType and the work of the team at BostonInno (in particular Kevin McCarthy’s The Tech Behind) and Phil Whelan in Vancouver. Dan Morel (@dpmorel) has started looking at the development processes used at Canadian startups (see: Supersize your Startup Dev Productivity & One is the Loneliest Number). If you are a Canadian startup that is building “interesting” technology, we’re actively looking for startups to profile.

We start the series with Chango and technical Q&A with Chris Sukornyk (LinkedIn, @sukornyk) and Mazdak Rezvani (LinkedIn, @mazdak) looking at the technology and the evolution of Chango platform. Chango closed $4.25MM in a Series B financing round led by Rho Canada and included iNovia Capital, Metamorphic Ventures, Extreme Venture Partners and others. The Chango team has doubled over the last 6 months, they are roughly 25 people (including Hot Sh!t List member Josh Davey).

The online advertising landscape is changing rapidly, Chango is taking advantage of Real-Time Bidding to allow marketers to buy ads on the web in real-time. It means big data where time is critical because small fluctuations in price or users can have big impact on the overall effectiveness or cost. The best analogy is a stock market, where instead of stocks, marketers can buy billions of ads each day based on how they think that websites ad slots will perform for them. The cool part about RTB is that you can combine data to put the right ad in front of the right user. This requires huge data processing skills and Chango processes information on about 60,000 ads a second.

What does Chango do?

We are an ad-buying platform that is built for online marketers looking to find new customers using display (banner ads). Our main goal is to eliminate wasted marketing dollars through smarter targeting. Specifically we show ads to people based on what they’ve searched for in the past – a technique we are pioneering called “Search Retargeting“.

Unlike traditional ad networks we exclusively buy ads using real-time bidding, a new technology that is rapidly changing the Display advertising industry and allowing marketers to layer data on top of their media buys.

Chango is unique in that it has access to billions of search terms, billions of incoming ad impressions, and has devised machine learning techniques for combining data from different sources to deliver more efficient and better-targeted campaigns for our advertisers.

What does your product architecture look like?

Software development that deals with “Big Data” typically has two types of challenges: big volume, or low latency. Chango has to deal with both of these issues simultaneously.

We receive over 90,000 requests per second (and growing monthly) from our ad exchange and data partners at peak times of the day. To make matters more interesting, we have approximately 80ms to respond to any incoming bid requests. Unfortunately 30-50ms of this limit is used up because of unavoidable network latency, this leaves us only 30ms to process each bid request. As a result, we have to constantly optimize our bidder subsystem around this challenge.

What are some of the tools and technologies you use?

Open Source is at the heart of our technology stack. Python is the common thread that binds all of our subsystems. Our entire infrastructure is written in Python. We use a (modified version of) the Tornado Web Server for our realtime servers, and Django for the front-end.

When dealing with super fast response times it’s critical to have a super-fast datastore at your disposal. That’s why we use a NoSQL database technology based on the proven memcached server.

The unsung open source hero of our infrastructure is HAProxy. HAProxy handles our load-balancing across the board. We use the “keepalived” feature of Linux to keep these servers in high-availability mode.

As far as our architecture is concerned we try to not have any major dependencies on any specific features of the third-party systems we choose. They just happen to work well for our current environment.

How did you get here?

Scalability is as much an art as it’s a science, but most importantly, it’s about keeping things simple. It took years of hard work, and careful tuning and measurement to arrive at where we are.

Originally we used Java for all server-side development and Ruby on Rails for front end development. The thinking was that we needed a rock-solid language for server architecture and a rapid development environment for front-end work. This concept served us well for a little while; however, in early 2010 we realized that Java was drastically slowing down our ability to iterate quickly and effectively. A single feature was taking days to build, test and deploy.

We bit the bullet and rebuilt the platform entirely in Python and it was probably the best decision we ever made. Not only do we have a consistent language across front end and server side development but it has enabled us to rapidly add features or test new ideas. We are fortunate to have access to the fantastic ecosystem created by the Python community.

Where do you host?

Our first approach back in 2009 was to leverage Amazon Web Services EC2 as a scaleable and cheap way to prototype the platform. That served us well for a while; however, the shared virtual environment meant that we had wildly variable server resources.

We shifted to Hosting.com knowing that we ultimately needed our own equipment and if we wanted a VM environment we would need to set one up ourselves. While Hosting.com provided good support at the time we wanted the rapid provisioning we were used to at EC2 with the power of dedicated hosting.

Ultimately we chose SoftLayer as our hosting provider of choice. SoftLayer offers a VM environment in addition with their “express” service that allows us to get 10s of new servers provisioned in about 3-4 hours! They have been extremely good about allowing us to occasionally provision a whole new parallel cluster as we do capacity planning.

How do you monitor your systems?

Monitoring is done through a combination of Nagios, PagerDuty, and Circonus among others. We have also built a real-time data visualization system that let’s us monitor both infrastructure, and campaign performance. We use this dashboard as our own NOC system hooked up to a TV that is mounted right in the engineering area!

What are your biggest development challenges?

We’ve got two distinct challenges. Our real-time, data processing, and systems engineering teams deal with problems of scalability and big data. As our business continues to grow, we need to re-examine our infrastructure and design choices. We have a very healthy culture of team collaboration, code-review, and refactoring.

Our Dashboard team has a different set of challenges. Our self-serve ad platform is very much like Adwords in that marketers can put down a credit card and launch a campaign themselves. We need to make this an extremely user-friendly system, while keeping it powerful enough to enable people to perform sophisticated reporting and campaign optimization.

How do you win?

The Chango business is all about putting the right ad in front of the right user at the right time. We made an early decision that data you know about a user (ie. search data) is only effective if it is combined with a proprietary bidding engine that can make decisions in real time.

Almost every DSP, data provider or ad network out there today does this by storing information about users in a client-browser “cookie”. They call this a user segment. The problem with user segments is that they are pre-computed and stored within the users browser. If you decide half way through your campaign that you need to adjust your audience (due to lack of performance) there is no way to do so since the information is hard coded in the users browser. The only option is to continue serving ads to this under-performing group of users and wait for that cookie to expire (typically 30 days).

At Chango we’ve decided to make everything real-time, including our decision about who to bid on, and how much to bid. Nothing about the user is pre-computed. So the Chango ‘cookie’ contains nothing more than a unique identifier that anonymously points to all the raw data we know about that person in our database.

Chango is hiring!

Python Developers! There are multiple open positions in Toronto, New York & San Francisco. But if I could have anything it would be talented developers that either know python or want to learn Python.