Under the Hood with Patrik Potocki

blog post header image

“We want to deliver a top-performing platform that supports partners of all sizes: scalability is extremely important, both from a tech and a cost perspective. In addition to being scalable, the platform should be self-healing and have the flexibility to allow us to try new things and continue to evolve. Part of delivering a great platform is having great visibility, and we want proactive monitoring and alerting to ensure we always know exactly what’s happening.”

My Story in short: Games, shoes, planes and platforms

I have been interested in tech since I was very young. It started with my father’s interest in computers, which we got very early on. In the beginning I was mostly just playing games but (as anyone who grew up in the early days of computer games knows) nothing ever just “worked” and I quickly learned how to debug and troubleshoot.

Later on I got introduced to programming and loved it. I began with ASP Classic, quickly followed by the .NET ecosystem.

I spent nearly 10 years in the Microsoft /.NET world, starting my career as part of the team handling e-commerce for one of Sweden’s largest shoe companies. I then moved to the airline industry, which was really fun and challenging: during promotional periods there would be enormous spikes in users which gave us interesting scaling challenges to think about.

To be a successful developer you really need to be willing to continuously learn. If that’s not the case, then your knowledge is going to get outdated pretty quickly.

Personally, I read a lot of blog posts. It’s a great way to get an overview of different solutions and new ideas. I don’t get deep into the details until I need it, but when we are faced with a challenge I can say “OK, I read about something that could be just what we need.” Then we can do a deep-dive and really see what will work for us.

Four years ago, I joined Fast Track as a Tech Lead. I have been part of almost the entire journey of the company; from building white label casino sites in PHP to launching Fast Track CRM using Golang.

Working with tech that makes me really excited

I have had the privilege of being here from the very beginning, designing the entire Fast Track infrastructure from scratch. I was very curious about distributed systems, CI/CD (continuous integration and continuous deployment) and automation in general, and how they could be used to build super scalable systems. When you are a developer, you want to set up the most optimal and fun environment to work in. I wanted to create an environment where the team could move really fast whilst keeping stability as a top priority, and be able to innovate and roll out new features at a rapid pace. I also wanted to ensure our development team could focus on development, and not be caught up with releases, or configuring virtual machines.

"I am incredibly motivated by Fast Track’s willingness and ability to quickly adapt to change."

When I joined, the company was working on a PHP foundation, with Apache on virtual machines and managed hosting through Rackspace. Everything was managed manually, and we were using Rackspace in a click and play fashion.

Every client has a single tenancy. With the set-up we had, this meant that we would need to manually duplicate any work across each individual tenancy. It might be manageable when you have 3-4 clients, but as we grew we saw this would quickly become very labour intensive.

We decided to redraw and push the organisation in a new direction. After running a few proofs of concept, we came up with a new stack:

  • Moved from PHP to GoLang as back-end
  • Introduced Vue.js as front-end framework
  • Moved from Rackspace to AWS as cloud hosting
  • Introduced Terraform for infrastructure as code
  • Introduced Docker / Kubernetes for container orchestration
  • Continued with MySQL as a general purpose database
  • Continued with RabbitMQ as messaging system
  • Bitbucket / Github for source control
  • Redis for cache
  • Elastic / Kibana / Grafana for logging
 

This stack was chosen with creating new opportunities for efficiency at the forefront of our minds. Of course, we had to consider practicalities: best practices for budget; choosing Golang over .NET Core (it was a close race) because we did not want to be tied to Microsoft products which didn’t serve all our needs as an organisation, etc.

I am incredibly motivated by Fast Track’s willingness and ability to quickly adapt to change and this, combined with the excellent tech team we have in Fast Track, enabled us to rebuild the entire platform from scratch and transition to this stack in less than a year. Overall we are very happy with what we have built. There is always room for improvement, for example we would like to move everything currently on Bitbucket to Github for greater efficiency, but right now this stack enables us to scale with ease.

Learning: Golang is elegant and super stable

I have worked with a lot of different languages. Some of them are quick and messy (ASP / Python), some of them are comprehensive but less resource-efficient (.NET / Java / Swift), and Golang is just right in the middle. It is clean, simple and fast, has great built-in tooling, and the syntax is very nice.

Golang enforces a code structure so that all the code will look the same. Take an extremely big project like kubernetes, for example. Even something as big as this is still easy to understand and get up to speed with if you review the code. This is super beneficial for other developers doing code reviews.

Running Golang in production for 4 years now, I can say that it is by far the most stable and lightweight stack I have ever worked with, and the one that required the least amount of configuration / set up from a server perspective to get it into a “performant” state.

I think it’s relevant to note that very few of our team members had any prior experience working with Golang. Even so, the transition has been easy and everyone is very happy with it.

I strongly recommend taking a look at Golang if you haven't yet.

So what are we building? Fast Track CRM: Let's take a peek under the hood

One of the most common questions my team and I get asked is “how does Fast Track CRM work?”

Like much of the best tech in the world, Fast Track is built with a very simple idea and design at core. In basic terms, we have built an engine that can process and evaluate incoming events and distribute instructions to other services on the back of it (orchestration). The platform runs entirely in real time: we can receive, process and take action on any event in about 100ms (or even faster).

A rule engine evaluates incoming messages. It is connected to a data model, which is used to evaluate eligibility, and creates a consistent structure for instructions that are passed on to other services. Together we refer to it as an Activity, which consists of a trigger, a segment (a group of eligible players) and one or multiple actions (like sending an email or crediting bonus).

At this level, it has nothing to do with iGaming in particular, and the focus is to build in a truly scalable way. What makes our platform a powerful solution for iGaming operators is the layers we put on top of and around this foundation.

"The industry moves fast, and we are well-positioned to adapt to every new challenge our partners might face."

One example of this is our in-house built real-time data platform, FT RTD. The platform consolidates incoming data at rapid speed, and constructs a real-time data warehouse for the operator, which is also the database that FT CRM connects to.

What makes our platform specific to the iGaming industry is, first and foremost, that we built it solely for iGaming operators. We have first-hand experience of the challenges of running CRM and player engagement initiatives for online gambling and we have worked continuously to put the right process on top of the right technology to tackle these.

Everything we do is with our partners front and center; whether it’s building Lifecycles to orchestrate complex engagements, or integrating their in-house bonus engines. The industry moves fast, and we are well-positioned to adapt to every new challenge our partners might face.

Excellent fit for service-oriented architecture

We use a service-oriented architecture for our platform. This means the platform is built up from lots of different services; for example, an SMS distribution service handles the sending of SMS and a bonus service handles the crediting of rewards.

There are many services that make up the whole platform, ranging from integration APIs to email distributors.

A service-oriented architecture allows you to move fast, easily test new technology and try new things in a contained environment. You can select one service to try something out, rather than having to implement it across the whole system. In this way, you can test and rollout ‘riskier’ features by trying them on less critical services first. A perfect example of this is how we rolled out Kafka as a compliment to RabbitMQ in just a matter of weeks.

This structure also makes it easier to debug when you encounter any issues: if an email queue stops then you know that the most likely explanation is that there is something wrong with the email distributor and you can check there.

The most important thing to remember when building services is that each one should be self-contained and “own their data”. If this data needs to be shared it can be done via API or by exposing the relevant info and allowing other services to “subscribe” to messages.

We run the whole platform on Kubernetes and all our services are docker images, which makes it extremely scalable.

But service-oriented architecture has its challenges

"We have rebuilt core parts of the platform numerous times."

Since I started at Fast Track, I have seen the company and our platform continuously evolve, the reality is that we have rebuilt core parts of the platform numerous times. I am really proud of what the team has managed to accomplish in such a short period of time.

One of the changes that I believe had the biggest impact was moving from working with virtual machines to working with Kubernetes.

Kubernetes enables us to achieve a level of efficiency and automation that we couldn’t previously. Instead of manually setting up all our services, we can effectively tell kubernetes what we want and how we want it to behave, and it will make sure to run what we ask for. It also automatically tries to restart services if something goes wrong, and auto-scales as needed. It saves us a lot of time.

But it's not all sunshine and lollipops. There are two things about a service-oriented architecture that are very painful, especially when running single tenants - traceability and deployments.

We are constantly improving how we deal with these.

When it comes to deployments, we decided to invest in building in-house deployment tools. We now build release packages, do infrastructure changes and roll out all our environments using a single portal. This works really well now, but it took time to overcome.

We have invested in comprehensive logging through elastic and kibana which gives us visibility and now there are only a few occasions where the movement of messages and instructions between different services can still be tricky.

What I read late at night

As I mentioned, the key to success is being open to learning continuously. These are some of the resources I use to stay up to date:

 

We are hiring

Fast Track has the best tech team I have ever worked with, and I believe our culture has played a key part in our ability to attract the very best.

We have a lot of open roles in tech at the moment. If you are interested in working with us please have a look below:

Work at Fast Track