Tales of a RTBkit adventure.


RTB gave you the power so ... now what ?

Power comes with a price, in this case the price you pay looks like a pretty complex distributed system. Developing one of these systems will chew up resources, a lot of time, and if you are not experienced in such systems you will probably sink in dark waters. This is where Datacratic jumped in and opened up RTBkit (thanks, btw).

RTBkit will drastically reduce your implementation time, and by implementation time I mean the time it takes to be running a production DSP. RTBkit will solve a lot of your problems, but not all of them. So in this post you'll find what took us at Motrixi to get to that point.

The first step as usual is finding the right technical resources and two you need to get started are a sysadmin that knows his way around linux networks and a C++ on linux developer that knows about distributed systems. We deployed our infrastructure on AWS, so knowing how AWS works is a big plus. Here is a small list of tasks that the sysadmin did at the begining:

  •  Created a virtual private cloud (VPC) at amazon
  •  Created all needed nodes inside the VPC
  •  Set up a VPN to access the VPC
  •  Set up Zabbix to monitor the VPC
  •  Tuned some kernel parameters like max open files and some others for TCP

In order to be in production in a shorter time we picked an OpenRTB based exchange (Rubicon) and just started with one. Our first goal was taking 5k QPS and bidding on 1k.

Our first version for the infrastructure had :

  • One 2 core node for the bid requests entry point reverse proxy (NGINX)
  • One 4 core node for the post auction events entry point reverse proxy (NGINX), it's bigger than the other one for a reason, read ahead.
  • One 8 core node with 16GB of RAM to run the rubicon exchange connector and the rest of the singleton processes
  • One 4 core node to run the bidding agents

We then started mounting RTBkit on the relevant nodes and for that we had to :

  • Create all the init.d scripts to handle the services
  • Based on the sample bidding agent we developed a simple agent to do fixed price bidding with configurable filters and a minimum pacing logic.
  • Have some kind of watchdog to look over the main processes, monit was our choice.

Then came the need to abstract the RTBkit based stack form our web application and that is a different story but essentially it came down to building a REST based application that abstracted the outside world (mainly our controlling web app) from the RTBkit interface providing a unified API to do any RTBkit related operation, from starting to stopping and updating agents to setting daily budgets on banking accounts to ad tag generation. We called this API the Campaign Manager. Do not underestimate this, it requires a fair amount of design time and decision making. Our conclusion was that the CM did not have to know any business rules other than the ones that affect the tag generation. It was just a the first level of abstraction from RTBkit.

Finally we had to implement the post auction event loop. This module is the entry point for post auction events such as wins and clicks. We did this by balancing the load among several uwsgi processes using NGINX. The uwsgi processes do :

  • Click redirection to landing page using 302
  • Forward all events to RTBkits ad server

Also this module listens on the PAL for events in order to receive an error filtered stream of events in order to load real time campaign information into a Redis db. This data is then exploited by other modules.

You can say that if you get to this point you have all the main parts for the RTB side of things. This took us (the sys admin and myself "the developer") around 2 months. As a disclaimer I have to say that this was my second RTBkit based production environment so in case you do not know anything about RTBkit it might take 3 to 4 months.

Of course, there’s a lot more to building a DSP. You will also have to develop reporting and analytics, a user interface and data integration. But RTBkit provided us with a solid foundation using which, in less than a year, we have evolved our platform to:

  • Handle 25k QPS of mobile requests
  • Support Rubicon, Nexage and Mopub, with ADX on the way
  • Support segment data augmentors based on advertisers beacons
  • Support segment data augmentors based on external data providers
  • Support latitude and longitude area based campaigns
  • Support many filters based on request data
  • Support different pacing strategies
  • Real time reporting

and a lot more coming ...

About Nic Emiliani, RTB Technology Leader at Motrixi

Designing, developing and integrating distributed systems is what I do as a software engineer. I'm the RTB Technology Leader at Motrixi Media Group.  Two of my biggest interests are Distributed Systems and Machine Learning. I am currently getting a Master degree in Datamining and Machine Learning at the University of Buenos Aires. Big fan of all things Sci-Fi and a rock climber.