Creative CTOR Prediction and Optimization

By Mayssam Naji; Randy Harnarinesingh ; Moiz F; Alkesh Y

Problem Statement

“Provide an easy to use tool for getting Click Through Open Rates (CTOR) for desired email creatives along with an optimizer to increase the CTOR by making changes to the email creative.”  It has the following two components :

• CTOR Prediction Model: A model that is able to predict the CTOR of an email creative.
• Optimizer: A tool that makes changes to the email creative to generate novel creatives with higher CTOR. 

Architecture

Data Collection

Data used are creative files, published date, CTOR score. Then, we engineered many features based on the creative i.e. laplacian, sobelx, colorfulness, keywords, bestselling and many more visual and textual features. 

CTOR prediction Model

The CTOR prediction model aims to predict the CTOR of a given creative. As it can be seen from the diagram before, the problem appears to be a regression problem on a set of features extracted from the creative. After experimenting with different regression approaches, gradient boosting on decision trees (Catboost) proved to be the most efficient and effective approach. 

Feature Importance

Feature engineering played a major role in modeling CTOR prediction. We designed a diverse set of features, around 400, that can give the model a good understanding of the CTOR of a creative. We separated our features in the following feature families:  

• Textual  features : Features based on text inside the creative e.g. presence of word discount etc. 
• Visual features: Global Features of the creative like colorfulness etc. Local visual features are obtained by dividing the creative into 3 by 3 grid and getting the features for each grid. 
• Pose and Face Features : Features based on the pose and facial features of the model in the creative. 

We analyzed the importance of these features towards CTOR prediction using SHAP, a game theoretic to explain the output of any machine learning model . 

Based on this feature importance analysis we picked twenty seven features and combined them using catboost to form the full CTOR prediction model.  We also constructed a small model by combining thirteen features using the Light Gradient Boosting Machine (LGBM). 

CTOR Predictions Results

LightGBM is used for small models and Catboost is used for full models.

‍

The above figure shows the error distribution of the prediction model on the test set. We can see that the model does fairly well on most of the creatives although there are a few outliers. 

Optimizer

We perform two kinds of optimizations to increase the CTOR score of a creative: 

Layout optimization :   We segment the creative into several rectangular regions containing images, text boxes etc. We then perform a search to get the best possible configuration of these rectangles in the creative. 

Image replacement and optimization : We replace the existing images in the creative with images in an asset library that we created to increase the CTOR score of the creative. This process requires three steps:

1. Automatic detection of the images in the creative : We use a U-NET to segment the creative into image and background and output the image mask. 
2. Creating a new creative from the given creative by cropping the image in the creative with a new image. 
3. Search over these new creatives and output the creative with the highest CTOR.
   

The CTOR prediction model is a very costly function to evaluate, it takes roughly 5-10 minutes for the full model and 3-4 minutes for the small model to run on a single creative. This makes the optimization procedure highly time consuming therefore we use embeddings, both text and visual, for getting the CTOR prediction for the purpose of optimization. 

In order to search efficiently we use various optimization strategies namely Grid Search,  Random Search and  Bayesian Search. Due to the large search space and costly CTOR prediction function evaluation, Bayesian optimizer is the right choice to get results in finite time and with less computation expenses.  The basic idea is not to be completely random in choice in search space  but instead use the information from the prior runs to choose better points in the search space. 

Present state-of-the-art optimization algorithm is the Tree-Structured Parzen Estimator (TPE). TPE is an iterative process that uses a prioris of evaluated search space to create a probabilistic model, which is used to suggest the next set of points in the search space. Few advantages of TPE are the following:

1. TPE supports a wide variety of variables in parameter search space e.g., uniform, log-uniform, quantized log-uniform, normally-distributed real value, categorical.
2. Extremely computationally efficient than conventional methods.

We choose HyperOpt as our choice of framework for optimization mainly because it provides TPE  and prune unpromising trials for faster results. Both of these features helped in reaching our own objectives of providing faster and computationally cheaper solutions to the large search space problems. During the development of the algorithms we had several challenges in terms of time complexities and search space complexities. 

Application:

1. Home Screen: When you first launch the application. From here you can upload new creative to work with or see results from recent creative.

2. Model Dashboard: When you upload a creative you see the model dashboard. Here you can choose if you want to get the CTOR score or get the optimized creative.

2a. To optimize creatives…

1. Click “Add line” to add lines to divide your creative
2. Click “Add rect” which will automatically detect all the images from the creative. 

If you have added the rectangles then “Select image to replace” and select image based on the color presented. Also, you can upload your own assets by clicking on the “Upload your own assets” button or you can leave this to use the built-in asset gallery.

‍

‍

2b. To get just the CTOR score, you can directly click on the “Get CTOR” button.

Conclusion

We were able to develop a tool that can help the user to improve CTOR of a given creative by  breaking down the problem into a CTOR prediction model and an optimizer. The tool has an intuitive and easy to use interface for a business team while internally  using state of the art algorithms to make predictions and do the optimization. The current approach can be extended for new kinds of creative modification different from  the currently implemented layout and image replacement optimization, such as text improvement in the creative.